From fgsch at lodoss.net  Mon Feb  1 18:35:37 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 01 Feb 2016 19:35:37 +0100
Subject: [master] 6694bea Cast time_t to work with 32 bit time_t
Message-ID: <E1aQJK5-0005zP-K3@project.varnish-software.com>


commit 6694bea3b2c0c4e6ef0cc723f03e07b1d9731502
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Feb 1 18:16:22 2016 +0000

    Cast time_t to work with 32 bit time_t
    
    Fixes #1851.

diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c
index 174df57..484d369 100644
--- a/bin/varnishstat/varnishstat_curses.c
+++ b/bin/varnishstat/varnishstat_curses.c
@@ -540,8 +540,9 @@ static void
 print_duration(WINDOW *w, time_t t)
 {
 
-	wprintw(w, "%4lu+%02lu:%02lu:%02lu",
-	    t / 86400, (t % 86400) / 3600, (t % 3600) / 60, t % 60);
+	wprintw(w, "%4jd+%02jd:%02jd:%02jd",
+	    (intmax_t)t / 86400, (intmax_t)(t % 86400) / 3600,
+	    (intmax_t)(t % 3600) / 60, (intmax_t)t % 60);
 }
 
 static void



From phk at FreeBSD.org  Mon Feb  1 21:52:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 01 Feb 2016 22:52:11 +0100
Subject: [master] 94c5efc Birthdays are happening
Message-ID: <E1aQMOJ-0006x9-70@project.varnish-software.com>


commit 94c5efc8a522c3f2604f2a6e952b9375941bd066
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 1 21:48:05 2016 +0000

    Birthdays are happening

diff --git a/doc/sphinx/phk/10goingon50.rst b/doc/sphinx/phk/10goingon50.rst
new file mode 100644
index 0000000..03e6c8e
--- /dev/null
+++ b/doc/sphinx/phk/10goingon50.rst
@@ -0,0 +1,281 @@
+.. _phk_10goingon50:
+
+========================
+Varnish - 10 going on 50
+========================
+
+Ten years ago, Dag-Erling and I were busy hashing out the big lines
+of Varnish.
+
+Hashing out had started on a blackboard at University of Basel
+during the `EuroBSDcon 2005 <http://2005.eurobsdcon.org/>`_ conference,
+and had continued in email and IRC ever since.
+
+At some point in February 2006 Dag-Erling laid down the foundations of
+our Subversion and source tree.
+
+The earliest fragment which have survived the conversion to Git is
+subversion commit number 9::
+
+    commit 523166ad2dd3a65e3987f13bc54f571f98453976
+    Author: Dag Erling Sm?rgrav <des at des.no>
+    Date:   Wed Feb 22 14:31:39 2006 +0000
+    
+        Additional subdivisions.
+
+We consider this the official birth-certificate of the Varnish Cache
+FOSS project, and therefore we will celebrate the 10 year birthday
+of Varnish in a couple of weeks.
+
+We're not sure exactly how and where we will celebrate this, but
+follow Twitter user `@varnishcache <https://twitter.com/varnishcache>`_
+if you want don't want to miss the partying.
+
+--------
+VCLOCC1
+--------
+
+One part of the celebration, somehow, sometime, will be the "VCL
+Obfuscated Code Contest #1" in the same spirit as the `International
+Obfuscated C Code Contest <http://www.ioccc.org/>`_.
+
+True afficionados of Obfuscated Code will also appreciate this
+amazing `Obfuscated PL/1 <http://www.multicians.org/proc-proc.html>`_.
+
+The official VCLOCC1 contest rules are simple:
+
+* VCL code must work with Varnish 4.1.1
+* As many Varnishd instances as you'd like. 
+* No inline-C allowed
+* Any VMOD you want is OK
+* You get to choose the request(s) to send to Varnishd
+* If you need backends, they must be simulated by varnishd (4.1.1) instances.
+* *We* get to publish the winning entry on the Varnish project home-page.
+
+The *only* thing which counts is the amazing/funny/brilliant
+VCL code *you* write and what it does.  VMODs and backends are just
+scaffolding which the judges will ignore.
+
+We will announce the submission deadline one month ahead of time, but
+you are more than welcome to start already now.
+
+--------
+Releases
+--------
+
+Our 10 year aniversary was a good excuse to take stock and look at
+the way we work, and changes are and will be happening.
+
+Like any respectable FOSS project, the Varnish project has never been
+accused, or guilty, of releasing on the promised date.
+
+Not even close.
+
+With 4.1 not even close to close.
+
+Having been around that block a couple of times, (*cough* FreeBSD 5.0 *cough*)
+I think I know why and I have decided to put a stop to it.
+
+Come hell or high water [#f1]_, Varnish 5.0 will be released September
+15th 2016.
+
+And the next big release, whatever we call it, will be middle of
+march 2017, and until we change our mind, you can trust a major
+release of Varnish to happen every six months.
+
+Minor releases, typically bugfixes, will be released as need arise,
+and these should just be installable with no configuration changes.
+
+Sounds wonderful, doesn't it ?  Now you can plan your upgrades.
+
+But nothing comes free:  Until we are near September, we won't be able
+to tell you what Varnish 5 contains.
+
+We have plans and ideas for what *should* be there, and we will work
+to reach those milestones, but we will not hold the release for "just this
+one more feature" if they are not ready.
+
+If it is in on september 15th, it will be in the release, if not, it wont.
+
+And since the next release is guaranteed to come six months later,
+it's not a catastrophe to miss the deadline.
+
+So what's the problem and why is this draconian solution better ?
+
+Usually, when FOSS projects start, they are started by "devops",
+Varnish certainly did:  Dag-Erling ran a couple of sites
+with Varnish, as did Kristian, and obviously Anders and Aydun of
+VG did as well, so finding out if you improved or broke things
+during development didn't take long.
+
+But as a project grows, people gravitate from "devops" to "dev",
+and suddenly we have to ask somebody else to "please test -trunk"
+and these people have their own calendars, and are not sure why
+they should test, or even if they should test, much less what they
+should be looking for while they test, because they have not been
+part of the development process.
+
+In all honesty, going from Varnish1 to Varnish4 the amount of
+real-life testing our releases have received *before* being released
+has consistently dropped [#f2]_.
+
+So we're moving the testing on the other side of the release date,
+because the people who *can* live-test Varnish prefer to have a
+release to test.
+
+We'll run all the tests we can in our development environments and
+we'll beg and cajole people with real sites into testing also, but
+we won't wait for weeks and months for it to happen, like we did
+with the 4.1 release.
+
+All this obviously changes the dynamics of the project, and it we
+find out it is a disaster, we'll change our mind.
+
+But until then:  Two major releases a year, as clock-work, mid-september
+and mid-march.
+
+----------------
+Moving to github
+----------------
+
+We're also moving the project to github.  We're trying to find out
+a good way to preserve the old Trac contents, and once we've
+figured that out, we'll pull the handle on the transition.
+
+Trac is starting to creak in the joints and in particular we're
+sick and tired of defending it against spammers.  Moving to github
+makes that Somebody Elses Problem.
+
+We also want to overhaul the project home-page and try to get
+a/the wiki working better.
+
+We'll keep you posted about all this when and as it happens.
+
+--------------------------------------------
+We were hip before it was hip to be hipsters
+--------------------------------------------
+
+Moving to github also means moving into a different culture.
+
+Githubs statistics are neat, but whenever you start to measure
+something, it becomes a parameter for optimization and competition,
+and there are people out there who compete on github statistics.
+
+In one instance the "game" is simply to submit changes, no matter
+how trivial, to as many different projects as you can manage in
+order to claim that you "contribute to a lot of FOSS projects".
+
+There is a similar culture of "trophy hunting" amongst so-called
+"security-researchers" - who has most CVE's to their name?  It
+doesn't seem to matter to them how vacuous the charge or how
+theoretical the "vulnerability" is, a CVE is a CVE to them.
+
+I don't want to play that game.
+
+If you are a contributor to Varnish, you should already have the
+nice blue T-shirt and the mug to prove it.  (Thanks Varnish-Software!)
+
+If you merely stumble over a spelling mistake, you merely
+stumbled over a spelling mistake, and we will happily 
+correct it, and put your name in the commit message.
+
+But it takes a lot more that fixing a spelling mistake to
+become recognized as "a Varnish contributor".
+
+Yeah, we're old and boring.
+
+Speaking of which...
+
+----------------------------
+Where does 50 come into it ?
+----------------------------
+
+On january 20th I celebrated my 50 year birthday, and this was a
+much more serious affair than I had anticipated:  For the first
+time in my life I have received a basket with wine and flowers on
+my birthday.
+
+I also received books and music from certain Varnish users,
+much appreciated guys!
+
+Despite numerically growing older I will insist, until the day I
+die, that I'm a man of my best age.
+
+That doesn't mean I'm not changing.
+
+To be honest, being middle-aged sucks.
+
+Your body starts creaking and you get frustrated seeing people make
+mistakes you warned them against.
+
+But growing older also absolutely rulez, because your age allows
+you to appreciate that you live in a fantastic future with a lot
+of amazing changes - even if it will take a long time before
+progress goes too far.
+
+There does seem to be increasing tendency to want the kids of your
+lawn, but I think I can control that.
+
+But if not I hereby give them permission to steal my apples and
+yell back at me, because I've seen a lot of men, in particular in
+the technical world, grow into bitter old men who preface every
+utterance with "As *I* already said *MANY* years ago...", totally
+oblivious to how different the world has become, how wrong their
+diagnosis is and how utterly useless their advice is.
+
+I don't want to end up like that.
+
+From now on my basic assumption is that I'm an old ass who is part
+of the problem, and that being part of the solution is something I
+have to work hard for, rather than the other way around.
+
+In my case, the two primary physiological symptoms of middle age is
+that after 5-6 hours my eyes tire from focusing on the monitor and
+that my mental context-switching for big contexts is slower than
+it used to be.
+
+A couple of years ago I started taking "eye-breaks" after lunch.
+Get away from the screen, preferably outside where I could rest my
+eyes on stuff further away than 40cm, then later in the day
+come back and continue hacking.
+
+Going forward, this pattern will become more pronounced.  The amount
+of hours I work will be the same, but I will be splitting the workday
+into two halves.
+
+You can expect me to be at my keyboard morning (08-12-ish EU time)
+and evening (20-24-ish EU time) but I may be doing other stuff,
+away from the keyboard and screen, during the afternoon.
+
+Starting this year I have also changed my calendar.
+
+Rather than working on various projects and for various customers
+in increments of half days, I'm lumping things together in bigger
+units of days and weeks.
+
+Anybody who knows anything about process scheduling can see that
+this will increase throughput at the cost of latency.
+
+The major latency impact is that one of the middle weeks of each
+month I will not be doing Varnish.  On the other hand, all
+the weeks I do work on Varnish will now be full weeks.
+
+And with those small adjustments, the Varnish project and I are
+ready to tackle the next ten years.
+
+Let me conclude with a big THANK YOU! to all Contributors and Users
+of Varnish, for making the first 10 years more amazing than I ever
+thought FOSS development could be.
+
+Much Appreciated!
+
+*phk*
+
+.. rubric:: Footnotes
+
+.. [#f1] I've always wondered about that expression.  Is the assumption that
+   if *both* hell *and* high water arrives at the same time they will cancel
+   out ?
+
+.. [#f2] I've seriously considered if I should start a porn-site, just to
+   test Varnish, but the WAF of that idea was well below zero.
diff --git a/doc/sphinx/phk/index.rst b/doc/sphinx/phk/index.rst
index 7dae4e4..60a3375 100644
--- a/doc/sphinx/phk/index.rst
+++ b/doc/sphinx/phk/index.rst
@@ -8,6 +8,7 @@ You may or may not want to know what Poul-Henning thinks.
 .. toctree::
 	:maxdepth: 1
 
+	10goingon50.rst
 	brinch-hansens-arrows.rst
 	ssl_again.rst
 	persistent.rst



From phk at FreeBSD.org  Mon Feb  1 22:09:47 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 01 Feb 2016 23:09:47 +0100
Subject: [master] cace2d5 Spelling fix
Message-ID: <E1aQMfL-0002MZ-7b@project.varnish-software.com>


commit cace2d5b703b44c79264a12165c39a408828a415
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 1 22:09:39 2016 +0000

    Spelling fix

diff --git a/doc/sphinx/phk/10goingon50.rst b/doc/sphinx/phk/10goingon50.rst
index 03e6c8e..6960e1f 100644
--- a/doc/sphinx/phk/10goingon50.rst
+++ b/doc/sphinx/phk/10goingon50.rst
@@ -104,7 +104,7 @@ So what's the problem and why is this draconian solution better ?
 
 Usually, when FOSS projects start, they are started by "devops",
 Varnish certainly did:  Dag-Erling ran a couple of sites
-with Varnish, as did Kristian, and obviously Anders and Aydun of
+with Varnish, as did Kristian, and obviously Anders and Audun of
 VG did as well, so finding out if you improved or broke things
 during development didn't take long.
 



From phk at FreeBSD.org  Tue Feb  2 00:23:24 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 01:23:24 +0100
Subject: [master] f185f0a Don't set the VAS handler until the test-executing
 child process.
Message-ID: <E1aQOke-0008Mn-Pn@project.varnish-software.com>


commit f185f0a455e3e4157ab193c05b4774e66270643f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 1 23:46:00 2016 +0000

    Don't set the VAS handler until the test-executing child process.

diff --git a/bin/varnishtest/vtc_log.c b/bin/varnishtest/vtc_log.c
index 30ae6b8..9b1ddf0 100644
--- a/bin/varnishtest/vtc_log.c
+++ b/bin/varnishtest/vtc_log.c
@@ -55,19 +55,6 @@ static double t0;
 
 /**********************************************************************/
 
-void
-vtc_loginit(char *buf, unsigned buflen)
-{
-
-	t0 = VTIM_mono();
-	vtclog_buf = buf;
-	vtclog_left = buflen;
-	AZ(pthread_mutex_init(&vtclog_mtx, NULL));
-	AZ(pthread_key_create(&log_key, NULL));
-}
-
-/**********************************************************************/
-
 
 struct vtclog *
 vtc_logopen(const char *id)
@@ -305,4 +292,17 @@ vtc_log_VAS_Fail(const char *func, const char *file, int line,
 	abort();
 }
 
-vas_f *VAS_Fail __attribute__((__noreturn__)) = vtc_log_VAS_Fail;
+/**********************************************************************/
+
+void
+vtc_loginit(char *buf, unsigned buflen)
+{
+
+	VAS_Fail = vtc_log_VAS_Fail;
+	t0 = VTIM_mono();
+	vtclog_buf = buf;
+	vtclog_left = buflen;
+	AZ(pthread_mutex_init(&vtclog_mtx, NULL));
+	AZ(pthread_key_create(&log_key, NULL));
+}
+



From phk at FreeBSD.org  Tue Feb  2 00:23:24 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 01:23:24 +0100
Subject: [master] cee7823 We had two vtc_dur variables.  Should be harmless.
Message-ID: <E1aQOke-0008Mt-RV@project.varnish-software.com>


commit cee782317680e7a1df2be6b9089a7da629d73e7c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 1 23:46:28 2016 +0000

    We had two vtc_dur variables.  Should be harmless.

diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index 33b56e1..97ade30 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -56,7 +56,6 @@ volatile sig_atomic_t	vtc_error;	/* Error encountered */
 int			vtc_stop;	/* Stops current test without error */
 pthread_t		vtc_thread;
 static struct vtclog	*vltop;
-unsigned		vtc_maxdur;	/* Max duration of any test */
 
 /**********************************************************************
  * Macro facility



From phk at FreeBSD.org  Tue Feb  2 00:23:24 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 01:23:24 +0100
Subject: [master] 4463fbe Polish things found by address-sanitizer.
Message-ID: <E1aQOke-0008N2-TC@project.varnish-software.com>


commit 4463fbef1aa61f9e120acfe96cd373cb28ff86ab
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 00:09:22 2016 +0000

    Polish things found by address-sanitizer.
    
    None of it was important or troubling.

diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index 97ade30..0ba0c0e 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -62,6 +62,8 @@ static struct vtclog	*vltop;
  */
 
 struct macro {
+	unsigned		magic;
+#define MACRO_MAGIC		0x803423e3
 	VTAILQ_ENTRY(macro)	list;
 	char			*name;
 	char			*val;
@@ -98,18 +100,16 @@ macro_def(struct vtclog *vl, const char *instance, const char *name,
 		if (!strcmp(name, m->name))
 			break;
 	if (m == NULL) {
-		m = calloc(sizeof *m, 1);
+		ALLOC_OBJ(m, MACRO_MAGIC);
 		AN(m);
 		REPLACE(m->name, name);
 		VTAILQ_INSERT_TAIL(&macro_list, m, list);
 	}
 	AN(m);
 	va_start(ap, fmt);
-	free(m->val);
-	m->val = NULL;
 	vbprintf(buf2, fmt, ap);
 	va_end(ap);
-	m->val = strdup(buf2);
+	REPLACE(m->val, buf2);
 	AN(m->val);
 	vtc_log(vl, 4, "macro def %s=%s", name, m->val);
 	AZ(pthread_mutex_unlock(&macro_mtx));
@@ -147,6 +147,7 @@ macro_get(const char *b, const char *e)
 	int l;
 	char *retval = NULL;
 
+
 	l = e - b;
 
 	if (l == 4 && !memcmp(b, "date", l)) {
@@ -158,9 +159,11 @@ macro_get(const char *b, const char *e)
 	}
 
 	AZ(pthread_mutex_lock(&macro_mtx));
-	VTAILQ_FOREACH(m, &macro_list, list)
-		if (!memcmp(b, m->name, l) && m->name[l] == '\0')
+	VTAILQ_FOREACH(m, &macro_list, list) {
+		CHECK_OBJ_NOTNULL(m, MACRO_MAGIC);
+		if (!strncmp(b, m->name, l) && m->name[l] == '\0')
 			break;
+}
 	if (m != NULL)
 		retval = strdup(m->val);
 	AZ(pthread_mutex_unlock(&macro_mtx));
@@ -207,13 +210,16 @@ macro_expand(struct vtclog *vl, const char *text)
 	return (vsb);
 }
 
-/* extmacro is a list of macro's that are defined from the
-   command line and are applied to the macro list of each test
-   instance. No locking is required as they are set before any tests
-   are started.
-*/
+/*
+ * extmacro is a list of macro's that are defined from the
+ * command line and are applied to the macro list of each test
+ * instance. No locking is required as they are set before any tests
+ *  are started.
+ */
 
 struct extmacro {
+	unsigned		magic;
+#define EXTMACRO_MAGIC		0x51019ded
 	VTAILQ_ENTRY(extmacro)	list;
 	char			*name;
 	char			*val;
@@ -233,24 +239,24 @@ extmacro_def(const char *name, const char *fmt, ...)
 		if (!strcmp(name, m->name))
 			break;
 	if (m == NULL && fmt != NULL) {
-		m = calloc(sizeof *m, 1);
+		ALLOC_OBJ(m, EXTMACRO_MAGIC);
 		AN(m);
 		REPLACE(m->name, name);
+		AN(m->name);
 		VTAILQ_INSERT_TAIL(&extmacro_list, m, list);
 	}
 	if (fmt != NULL) {
 		AN(m);
 		va_start(ap, fmt);
-		free(m->val);
 		vbprintf(buf, fmt, ap);
 		va_end(ap);
-		m->val = strdup(buf);
+		REPLACE(m->val, buf);
 		AN(m->val);
 	} else if (m != NULL) {
 		VTAILQ_REMOVE(&extmacro_list, m, list);
-		free(m->name);
-		free(m->val);
-		free(m);
+		REPLACE(m->name, NULL);
+		REPLACE(m->val, NULL);
+		FREE_OBJ(m);
 	}
 }
 
@@ -652,8 +658,10 @@ exec_file(const char *fn, const char *script, const char *tmpdir,
 	init_server();
 
 	/* Apply extmacro definitions */
-	VTAILQ_FOREACH(m, &extmacro_list, list)
+	VTAILQ_FOREACH(m, &extmacro_list, list) {
+		CHECK_OBJ_NOTNULL(m, EXTMACRO_MAGIC);
 		macro_def(vltop, NULL, m->name, "%s", m->val);
+	}
 
 	/*
 	 * We need an IP number which will not repond, ever, and that is a
diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index 0440542..7822ebc 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -223,10 +223,10 @@ cmd_var_resolve(struct http *hp, char *spec)
 		return(hp->bodylen);
 	if (!strcmp(spec, "resp.body"))
 		return(hp->body != NULL ? hp->body : spec);
-	if (!memcmp(spec, "req.http.", 9)) {
+	if (!strncmp(spec, "req.http.", 9)) {
 		hh = hp->req;
 		hdr = spec + 9;
-	} else if (!memcmp(spec, "resp.http.", 10)) {
+	} else if (!strncmp(spec, "resp.http.", 10)) {
 		hh = hp->resp;
 		hdr = spec + 10;
 	} else



From phk at FreeBSD.org  Tue Feb  2 00:23:24 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 01:23:24 +0100
Subject: [master] 776c838 Add a missing VDP flush operation after ESI:includes.
Message-ID: <E1aQOke-0008NC-Ux@project.varnish-software.com>


commit 776c8386b063b53ef40317a3d8a98ef495ff709f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 00:22:04 2016 +0000

    Add a missing VDP flush operation after ESI:includes.
    
    In incredibly seldom circumstances this bug would cause ESI:include
    data to be bad.
    
    Fixes:	#1852

diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index 703109b..efa6fb2 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -775,6 +775,7 @@ VED_Deliver(struct req *req, struct boc *boc, int wantbody)
 		else
 			VDP_push(req, ved_vdp_bytes, ecx->preq, 1);
 		(void)VDP_DeliverObj(req);
+		(void)VDP_bytes(req, VDP_FLUSH, NULL, 0);
 	}
 	VDP_close(req);
 }



From phk at FreeBSD.org  Tue Feb  2 07:52:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 08:52:11 +0100
Subject: [master] b2bbb50 Neuter another case where AddressSanitizer chokes on
 FreeBSD's hyperwide memcmp()
Message-ID: <E1aQVkx-0008UG-AF@project.varnish-software.com>


commit b2bbb5051c691eeed3373af0eeff3448a94b2867
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 00:26:39 2016 +0000

    Neuter another case where AddressSanitizer chokes on FreeBSD's
    hyperwide memcmp()

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 2b71d94..2838914 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -143,10 +143,10 @@ ban_equal(const uint8_t *bs1, const uint8_t *bs2)
 
 	/*
 	 * Compare two ban-strings.
-	 * The memcmp() is safe because the first field we compare is the
-	 * length and that is part of the fixed header structure.
 	 */
 	u = vbe32dec(bs1 + BANS_LENGTH);
+	if (u != vbe32dec(bs2 + BANS_LENGTH))
+		return (0);
 	return (!memcmp(bs1 + BANS_LENGTH, bs2 + BANS_LENGTH, u - BANS_LENGTH));
 }
 



From phk at FreeBSD.org  Tue Feb  2 07:52:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 08:52:11 +0100
Subject: [master] 23a0536 Make NEEDLESS_RETURN a blank macro so it can also be
 used without a return value.
Message-ID: <E1aQVkx-0008UN-Bg@project.varnish-software.com>


commit 23a053635cd3970e6147d626050066702becbcee
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 07:44:06 2016 +0000

    Make NEEDLESS_RETURN a blank macro so it can also be used without
    a return value.

diff --git a/bin/varnishd/flint.lnt b/bin/varnishd/flint.lnt
index 200e37b..cba5eb1 100644
--- a/bin/varnishd/flint.lnt
+++ b/bin/varnishd/flint.lnt
@@ -104,8 +104,6 @@
 
 +libh mgt_event.h
 
--emacro(527, NEEDLESS_RETURN)	// unreachable code
-
 -sem(BAN_Free, custodial(1))
 -sem(EXP_Inject, custodial(1))
 -sem(HSH_Insert, custodial(3))
diff --git a/flint.lnt b/flint.lnt
index c6824f5..1dcb478 100644
--- a/flint.lnt
+++ b/flint.lnt
@@ -93,6 +93,11 @@
 -emacro(740, VTAILQ_PREV) // Unusual pointer cast (incompatible indirect types)
 
 ///////////////////////////////////////////////////////////////////////
+// <vdef.h>
+-emacro(527, NEEDLESS_RETURN)	// unreachable code
+
+///////////////////////////////////////////////////////////////////////
+
 
 -esym(785,VSL_tags)				// Sparse array
 
diff --git a/include/vdef.h b/include/vdef.h
index a015b05..0790e43 100644
--- a/include/vdef.h
+++ b/include/vdef.h
@@ -78,6 +78,6 @@
  */
 #define __match_proto__(xxx)		/*lint -e{818} */
 
-#define NEEDLESS_RETURN(foo)	return (foo)
+#define NEEDLESS_RETURN		return
 
 #endif /* VDEF_H_INCLUDED */



From phk at FreeBSD.org  Tue Feb  2 07:52:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 08:52:11 +0100
Subject: [master] 6f16ca3 Simplify the macro code,
 no need for two different kinds of macros.
Message-ID: <E1aQVkx-0008UW-Cr@project.varnish-software.com>


commit 6f16ca39f37e697b79b3ae587c6b167f9c4f60fe
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 07:49:00 2016 +0000

    Simplify the macro code, no need for two different kinds of macros.

diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index 0ba0c0e..b60deb4 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -71,6 +71,51 @@ struct macro {
 
 static VTAILQ_HEAD(,macro) macro_list = VTAILQ_HEAD_INITIALIZER(macro_list);
 
+/**********************************************************************/
+
+static struct macro *
+macro_def_int(const char *name, const char *fmt, va_list ap)
+{
+	struct macro *m;
+	char buf[256];
+
+	VTAILQ_FOREACH(m, &macro_list, list)
+		if (!strcmp(name, m->name))
+			break;
+	if (m == NULL) {
+		ALLOC_OBJ(m, MACRO_MAGIC);
+		AN(m);
+		REPLACE(m->name, name);
+		AN(m->name);
+		VTAILQ_INSERT_TAIL(&macro_list, m, list);
+	}
+	AN(m);
+	vbprintf(buf, fmt, ap);
+	REPLACE(m->val, buf);
+	AN(m->val);
+	return (m);
+}
+
+
+/**********************************************************************
+ * This is for defining macros before we fork the child process which
+ * runs the test-case.
+ */
+
+void
+extmacro_def(const char *name, const char *fmt, ...)
+{
+	va_list ap;
+
+	va_start(ap, fmt);
+	(void)macro_def_int(name, fmt, ap);
+	va_end(ap);
+}
+
+/**********************************************************************
+ * Below this point is run inside the testing child-process.
+ */
+
 static pthread_mutex_t		macro_mtx;
 
 static void
@@ -84,7 +129,6 @@ macro_def(struct vtclog *vl, const char *instance, const char *name,
     const char *fmt, ...)
 {
 	char buf1[256];
-	char buf2[256];
 	struct macro *m;
 	va_list ap;
 
@@ -96,21 +140,9 @@ macro_def(struct vtclog *vl, const char *instance, const char *name,
 	}
 
 	AZ(pthread_mutex_lock(&macro_mtx));
-	VTAILQ_FOREACH(m, &macro_list, list)
-		if (!strcmp(name, m->name))
-			break;
-	if (m == NULL) {
-		ALLOC_OBJ(m, MACRO_MAGIC);
-		AN(m);
-		REPLACE(m->name, name);
-		VTAILQ_INSERT_TAIL(&macro_list, m, list);
-	}
-	AN(m);
 	va_start(ap, fmt);
-	vbprintf(buf2, fmt, ap);
+	m = macro_def_int(name, fmt, ap);
 	va_end(ap);
-	REPLACE(m->val, buf2);
-	AN(m->val);
 	vtc_log(vl, 4, "macro def %s=%s", name, m->val);
 	AZ(pthread_mutex_unlock(&macro_mtx));
 }
@@ -147,7 +179,6 @@ macro_get(const char *b, const char *e)
 	int l;
 	char *retval = NULL;
 
-
 	l = e - b;
 
 	if (l == 4 && !memcmp(b, "date", l)) {
@@ -163,7 +194,7 @@ macro_get(const char *b, const char *e)
 		CHECK_OBJ_NOTNULL(m, MACRO_MAGIC);
 		if (!strncmp(b, m->name, l) && m->name[l] == '\0')
 			break;
-}
+	}
 	if (m != NULL)
 		retval = strdup(m->val);
 	AZ(pthread_mutex_unlock(&macro_mtx));
@@ -200,7 +231,7 @@ macro_expand(struct vtclog *vl, const char *text)
 			VSB_delete(vsb);
 			vtc_log(vl, 0, "Macro ${%.*s} not found", (int)(q - p),
 			    p);
-			return (NULL);
+			NEEDLESS_RETURN (NULL);
 		}
 		VSB_printf(vsb, "%s", m);
 		free(m);
@@ -210,56 +241,6 @@ macro_expand(struct vtclog *vl, const char *text)
 	return (vsb);
 }
 
-/*
- * extmacro is a list of macro's that are defined from the
- * command line and are applied to the macro list of each test
- * instance. No locking is required as they are set before any tests
- *  are started.
- */
-
-struct extmacro {
-	unsigned		magic;
-#define EXTMACRO_MAGIC		0x51019ded
-	VTAILQ_ENTRY(extmacro)	list;
-	char			*name;
-	char			*val;
-};
-
-static VTAILQ_HEAD(, extmacro) extmacro_list =
-	VTAILQ_HEAD_INITIALIZER(extmacro_list);
-
-void
-extmacro_def(const char *name, const char *fmt, ...)
-{
-	char buf[256];
-	struct extmacro *m;
-	va_list ap;
-
-	VTAILQ_FOREACH(m, &extmacro_list, list)
-		if (!strcmp(name, m->name))
-			break;
-	if (m == NULL && fmt != NULL) {
-		ALLOC_OBJ(m, EXTMACRO_MAGIC);
-		AN(m);
-		REPLACE(m->name, name);
-		AN(m->name);
-		VTAILQ_INSERT_TAIL(&extmacro_list, m, list);
-	}
-	if (fmt != NULL) {
-		AN(m);
-		va_start(ap, fmt);
-		vbprintf(buf, fmt, ap);
-		va_end(ap);
-		REPLACE(m->val, buf);
-		AN(m->val);
-	} else if (m != NULL) {
-		VTAILQ_REMOVE(&extmacro_list, m, list);
-		REPLACE(m->name, NULL);
-		REPLACE(m->val, NULL);
-		FREE_OBJ(m);
-	}
-}
-
 /**********************************************************************
  * Parse a string
  *
@@ -385,7 +366,7 @@ parse_string(const char *spec, const struct cmds *cmd, void *priv,
 				break;
 		if (cp->name == NULL) {
 			vtc_log(vl, 0, "Unknown command: \"%s\"", token_s[0]);
-			return;
+			NEEDLESS_RETURN;
 		}
 
 		assert(cp->cmd != NULL);
@@ -645,7 +626,6 @@ exec_file(const char *fn, const char *script, const char *tmpdir,
 {
 	unsigned old_err;
 	FILE *f;
-	struct extmacro *m;
 
 	(void)signal(SIGPIPE, SIG_IGN);
 
@@ -657,12 +637,6 @@ exec_file(const char *fn, const char *script, const char *tmpdir,
 	init_sema();
 	init_server();
 
-	/* Apply extmacro definitions */
-	VTAILQ_FOREACH(m, &extmacro_list, list) {
-		CHECK_OBJ_NOTNULL(m, EXTMACRO_MAGIC);
-		macro_def(vltop, NULL, m->name, "%s", m->val);
-	}
-
 	/*
 	 * We need an IP number which will not repond, ever, and that is a
 	 * lot harder than it sounds.  This IP# is from RFC5737 and a



From phk at FreeBSD.org  Tue Feb  2 08:44:21 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 09:44:21 +0100
Subject: [master] 0168453 Don't use WS_Overflowed() from panic dumps,
 it has asserts which can recurse.
Message-ID: <E1aQWZR-0005kp-MH@project.varnish-software.com>


commit 01684535a0a99175f23536ba0cb228eb99a33f58
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 08:17:06 2016 +0000

    Don't use WS_Overflowed() from panic dumps, it has asserts which
    can recurse.

diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 47421fe..070f00d 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -141,7 +141,7 @@ pan_ws(struct vsb *vsb, const struct ws *ws)
 	if (pan_already(vsb, ws))
 		return;
 	VSB_indent(vsb, 2);
-	if (WS_Overflowed(ws))
+	if (ws->id[0] & 0x20)
 		VSB_printf(vsb, "OVERFLOW ");
 	VSB_printf(vsb, "id = \"%s\",\n",  ws->id);
 	VSB_printf(vsb, "{s,f,r,e} = {%p",  ws->s);
diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c
index 9293d9e..cd63f49 100644
--- a/bin/varnishd/cache/cache_ws.c
+++ b/bin/varnishd/cache/cache_ws.c
@@ -252,6 +252,7 @@ WS_ReleaseP(struct ws *ws, char *ptr)
 	ws->r = NULL;
 	WS_Assert(ws);
 }
+
 int
 WS_Overflowed(const struct ws *ws)
 {



From phk at FreeBSD.org  Tue Feb  2 08:44:21 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 09:44:21 +0100
Subject: [master] a26fdcd Hold a ref on the oc for BACKGROUND fetches until we
 have checked that we can continue.
Message-ID: <E1aQWZR-0005kv-O7@project.varnish-software.com>


commit a26fdcd6071a6938759039f2513ca624a21b0e72
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 08:43:42 2016 +0000

    Hold a ref on the oc for BACKGROUND fetches until we have checked
    that we can continue.
    
    Fixes #1849

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 83d72be..5332e7c 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -1006,8 +1006,7 @@ VBF_Fetch(struct worker *wrk, struct req *req, struct objcore *oc,
 	oc->boc->vary = req->vary_b;
 	req->vary_b = NULL;
 
-	if (mode != VBF_BACKGROUND)
-		HSH_Ref(oc);
+	HSH_Ref(oc);
 	AZ(bo->fetch_objcore);
 	bo->fetch_objcore = oc;
 
@@ -1047,5 +1046,7 @@ VBF_Fetch(struct worker *wrk, struct req *req, struct objcore *oc,
 	VSLb_ts_req(req, "Fetch", W_TIM_real(wrk));
 	assert(oc->boc == boc);
 	HSH_DerefBusy(wrk, oc);
+	if (mode == VBF_BACKGROUND)
+		(void)HSH_DerefObjCore(wrk, &oc);
 	THR_SetBusyobj(NULL);
 }



From phk at FreeBSD.org  Tue Feb  2 09:14:51 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 10:14:51 +0100
Subject: [master] 8c77b5d Get the sign right on the overflow test
Message-ID: <E1aQX2x-00010V-DM@project.varnish-software.com>


commit 8c77b5df134b32abec2c8c5998b7a112303d1e05
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 09:14:27 2016 +0000

    Get the sign right on the overflow test
    
    Spotted by:	fgs

diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 070f00d..d670326 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -141,8 +141,8 @@ pan_ws(struct vsb *vsb, const struct ws *ws)
 	if (pan_already(vsb, ws))
 		return;
 	VSB_indent(vsb, 2);
-	if (ws->id[0] & 0x20)
-		VSB_printf(vsb, "OVERFLOW ");
+	if (!(ws->id[0] & 0x20))
+		VSB_printf(vsb, "OVERFLOWED ");
 	VSB_printf(vsb, "id = \"%s\",\n",  ws->id);
 	VSB_printf(vsb, "{s,f,r,e} = {%p",  ws->s);
 	if (ws->f > ws->s)



From fgsch at lodoss.net  Tue Feb  2 09:22:30 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Tue, 02 Feb 2016 10:22:30 +0100
Subject: [master] 1ca122e Error earlier if we cannot find the probe
Message-ID: <E1aQXAM-0001xy-Q7@project.varnish-software.com>


commit 1ca122e4f9099e2d4b8d3d740783bced1151d78b
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 2 08:04:20 2016 +0000

    Error earlier if we cannot find the probe
    
    We don't have forward declarations in VCL.  OK'd by phk at .
    Fixes #1837.

diff --git a/bin/varnishtest/tests/r01837.vtc b/bin/varnishtest/tests/r01837.vtc
new file mode 100644
index 0000000..4dc3d68
--- /dev/null
+++ b/bin/varnishtest/tests/r01837.vtc
@@ -0,0 +1,10 @@
+varnishtest "Test VCC errors out if probe is used before it is defined"
+
+varnish v1 -errvcl "Probe p not found" {
+	backend b {
+		.host = "127.0.0.1";
+		.probe = p;
+	}
+	probe p {
+	}
+}
diff --git a/bin/varnishtest/tests/v00034.vtc b/bin/varnishtest/tests/v00034.vtc
index f7d4ed0..41626c1 100644
--- a/bin/varnishtest/tests/v00034.vtc
+++ b/bin/varnishtest/tests/v00034.vtc
@@ -20,7 +20,7 @@ varnish v1 -errvcl {Backend s1 redefined} {
 }
 
 varnish v1 -errvcl {Probe p1 redefined} {
-	backend s1 { .host = "127.0.0.1"; .probe = p1;}
 	probe p1 { }
 	probe p1 { }
+	backend s1 { .host = "127.0.0.1"; .probe = p1;}
 }
diff --git a/lib/libvcc/vcc_backend.c b/lib/libvcc/vcc_backend.c
index 8a413c7..3260f65 100644
--- a/lib/libvcc/vcc_backend.c
+++ b/lib/libvcc/vcc_backend.c
@@ -384,6 +384,12 @@ vcc_ParseHostDef(struct vcc *tl, const struct token *t_be, const char *vgcname)
 			Fb(tl, 0, "\t.probe = &%s,\n", p);
 			ERRCHK(tl);
 		} else if (vcc_IdIs(t_field, "probe") && tl->t->tok == ID) {
+			if (VCC_FindSymbol(tl, tl->t, SYM_PROBE) == NULL) {
+				VSB_printf(tl->sb, "Probe %.*s not found\n",
+				    PF(tl->t));
+				vcc_ErrWhere(tl, tl->t);
+				return;
+			}
 			Fb(tl, 0, "\t.probe = &vgc_probe_%.*s,\n", PF(tl->t));
 			vcc_AddRef(tl, tl->t, SYM_PROBE);
 			vcc_NextToken(tl);



From fgsch at lodoss.net  Tue Feb  2 09:22:30 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Tue, 02 Feb 2016 10:22:30 +0100
Subject: [master] bbf5a34 Fail HTTP/1.0 POST and PUT requests without C-L
Message-ID: <E1aQXAM-0001y4-Rh@project.varnish-software.com>


commit bbf5a3409842039a48b5fd04d98d0c136d575777
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 2 08:17:13 2016 +0000

    Fail HTTP/1.0 POST and PUT requests without C-L
    
    It is not allowed by the spec and we incorrectly assumed chunked
    and eventually timed out.  OK'd by phk at .
    Fixes #1843.

diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c
index deb191d..5734fab 100644
--- a/bin/varnishd/http1/cache_http1_proto.c
+++ b/bin/varnishd/http1/cache_http1_proto.c
@@ -283,6 +283,8 @@ http1_body_status(const struct http *hp, struct http_conn *htc)
 	htc->content_length = -1;
 
 	cl = http_GetContentLength(hp);
+	if (cl == -2)
+		return (BS_ERROR);
 	if (http_GetHdr(hp, H_Transfer_Encoding, &b)) {
 		if (strcasecmp(b, "chunked"))
 			return (BS_ERROR);
@@ -295,8 +297,6 @@ http1_body_status(const struct http *hp, struct http_conn *htc)
 		}
 		return (BS_CHUNKED);
 	}
-	if (cl == -2)
-		return (BS_ERROR);
 	if (cl >= 0) {
 		htc->content_length = cl;
 		return (cl == 0 ? BS_NONE : BS_LENGTH);
@@ -376,10 +376,13 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp)
 	p = http_GetMethod(hp);
 	AN(p);
 
-	/* We handle EOF bodies only for PUT and POST */
-	if (htc->body_status == BS_EOF &&
-	    strcasecmp(p, "put") && strcasecmp(p, "post"))
+	if (htc->body_status == BS_EOF) {
+		assert(hp->protover == 10);
+		/* RFC1945 8.3 p32 and D.1.1 p58 */
+		if (!strcasecmp(p, "post") || !strcasecmp(p, "put"))
+			return (400);
 		htc->body_status = BS_NONE;
+	}
 
 	/* HEAD with a body is a hard error */
 	if (htc->body_status != BS_NONE && !strcasecmp(p, "head"))
diff --git a/bin/varnishtest/tests/r01843.vtc b/bin/varnishtest/tests/r01843.vtc
new file mode 100644
index 0000000..20d37c9
--- /dev/null
+++ b/bin/varnishtest/tests/r01843.vtc
@@ -0,0 +1,17 @@
+varnishtest "HTTP/1.0 POST and PUT need a valid Content-Length"
+
+server s1 { } -start
+
+varnish v1 -vcl+backend {} -start
+
+client c1 {
+	txreq -proto HTTP/1.0 -req "POST" -nolen
+	rxresp
+	expect resp.status == 400
+} -run
+
+client c1 {
+	txreq -proto HTTP/1.0 -req "PUT" -nolen
+	rxresp
+	expect resp.status == 400
+} -run



From fgsch at lodoss.net  Tue Feb  2 09:22:30 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Tue, 02 Feb 2016 10:22:30 +0100
Subject: [master] 1f1d487 Add REAL add/sub/mul/div REAL support
Message-ID: <E1aQXAM-0001yD-TU@project.varnish-software.com>


commit 1f1d4872f38f7f55d61edc9d7b80246fdcfda3a8
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 2 08:43:29 2016 +0000

    Add REAL add/sub/mul/div REAL support
    
    Prompted by someone on #varnish.  OK'd by phk.

diff --git a/bin/varnishtest/tests/v00020.vtc b/bin/varnishtest/tests/v00020.vtc
index a9e9d3b..263b883 100644
--- a/bin/varnishtest/tests/v00020.vtc
+++ b/bin/varnishtest/tests/v00020.vtc
@@ -102,10 +102,18 @@ varnish v1 -vcl {
 
 		set req.http.foo = now + 1s;
 		set req.http.foo = now - 1s;
+
 		set req.http.foo = now - now;
 
 		set req.http.foo = 1 + 1;
 		set req.http.foo = 1 - 1;
+		set req.http.foo = 3 * 2;
+		set req.http.foo = 3 / 2;
+
+		set req.http.foo = 3.6 + 1.4;
+		set req.http.foo = 3.6 - 1.4;
+		set req.http.foo = 3.6 * 1.4;
+		set req.http.foo = 3.6 / 1.4;
 
 		set req.ttl = 1s;
 	}
diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c
index 5989f07..af3b3e7 100644
--- a/lib/libvcc/vcc_expr.c
+++ b/lib/libvcc/vcc_expr.c
@@ -942,6 +942,7 @@ vcc_expr_mul(struct vcc *tl, struct expr **e, enum var_type fmt)
 	case INT:	f2 = INT; break;
 	case DURATION:	f2 = REAL; break;
 	case BYTES:	f2 = REAL; break;
+	case REAL:	f2 = REAL; break;
 	default:
 		if (tl->t->tok != '*' && tl->t->tok != '/')
 			return;
@@ -1049,6 +1050,8 @@ vcc_expr_add(struct vcc *tl, struct expr **e, enum var_type fmt)
 			/* OK */
 		} else if ((*e)->fmt == INT && e2->fmt == INT) {
 			/* OK */
+		} else if ((*e)->fmt == REAL && e2->fmt == REAL) {
+			/* OK */
 		} else if ((*e)->fmt == DURATION && e2->fmt == DURATION) {
 			/* OK */
 		} else if (tk->tok == '+' &&



From phk at FreeBSD.org  Tue Feb  2 10:07:38 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 11:07:38 +0100
Subject: [master] 1c13787 Don't leak a filename in error exits,
 it upsets Coverity.
Message-ID: <E1aQXs2-0007XC-5B@project.varnish-software.com>


commit 1c137872f417c0bcf2e9ef1462c06d555dd15782
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 10:06:50 2016 +0000

    Don't leak a filename in error exits, it upsets Coverity.

diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c
index 87331ff..bd524c5 100644
--- a/lib/libvcc/vcc_compile.c
+++ b/lib/libvcc/vcc_compile.c
@@ -534,6 +534,7 @@ vcc_file_source(const struct vcp * const vcp, struct vsb *sb, const char *fn)
 	if (VFIL_searchpath(vcp->vcl_path, NULL, &f, fn, &fnp) || f == NULL) {
 		VSB_printf(sb, "Cannot read file '%s' (%s)\n",
 		    fnp != NULL ? fnp : fn, strerror(errno));
+		REPLACE(fnp, NULL);
 		return (NULL);
 	}
 	sp = vcc_new_source(f, NULL, fnp);
diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c
index 3a522a5..62f39fc 100644
--- a/lib/libvcc/vcc_vmod.c
+++ b/lib/libvcc/vcc_vmod.c
@@ -61,7 +61,7 @@ void
 vcc_ParseImport(struct vcc *tl)
 {
 	void *hdl;
-	char fn[1024], *fnp;
+	char fn[1024], *fnp, *fnpx;
 	char buf[256];
 	struct token *mod, *t1;
 	struct inifin *ifp;
@@ -130,14 +130,19 @@ vcc_ParseImport(struct vcc *tl)
 	SkipToken(tl, ';');
 
 	if (VFIL_searchpath(tl->param->vmod_path,
-	    vcc_path_dlopen, &hdl, fn, &fnp)) {
+	    vcc_path_dlopen, &hdl, fn, &fnpx)) {
 		VSB_printf(tl->sb, "Could not load VMOD %.*s\n", PF(mod));
-		VSB_printf(tl->sb, "\tFile name: %s\n", fnp != NULL ? fnp : fn);
+		VSB_printf(tl->sb, "\tFile name: %s\n",
+		    fnpx != NULL ? fnpx : fn);
 		VSB_printf(tl->sb, "\tdlerror: %s\n", dlerror());
 		vcc_ErrWhere(tl, mod);
+		REPLACE(fnpx, NULL);
 		return;
 	}
 
+	fnp = TlDup(tl, fnpx);
+	REPLACE(fnpx, NULL);
+
 	bprintf(buf, "Vmod_%.*s_Data", PF(mod));
 	vmd = dlsym(hdl, buf);
 	if (vmd == NULL) {
@@ -195,7 +200,6 @@ vcc_ParseImport(struct vcc *tl)
 	VSB_printf(ifp->ini, "\t    \"%.*s\",\n", PF(mod));
 	VSB_printf(ifp->ini, "\t    ");
 	EncString(ifp->ini, fnp, NULL, 0);
-	free(fnp);
 	VSB_printf(ifp->ini, ",\n");
 	AN(vmd);
 	AN(vmd->file_id);



From phk at FreeBSD.org  Tue Feb  2 10:25:08 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 11:25:08 +0100
Subject: [master] 1cf8057 Cave into to gcc49's paranoia...
Message-ID: <E1aQY8y-000173-Cd@project.varnish-software.com>


commit 1cf8057f687c74012f8a808647ca075a8effeed9
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 10:24:50 2016 +0000

    Cave into to gcc49's paranoia...

diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c
index bd524c5..5db516f 100644
--- a/lib/libvcc/vcc_compile.c
+++ b/lib/libvcc/vcc_compile.c
@@ -534,7 +534,7 @@ vcc_file_source(const struct vcp * const vcp, struct vsb *sb, const char *fn)
 	if (VFIL_searchpath(vcp->vcl_path, NULL, &f, fn, &fnp) || f == NULL) {
 		VSB_printf(sb, "Cannot read file '%s' (%s)\n",
 		    fnp != NULL ? fnp : fn, strerror(errno));
-		REPLACE(fnp, NULL);
+		free(fnp);
 		return (NULL);
 	}
 	sp = vcc_new_source(f, NULL, fnp);
diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c
index 62f39fc..38515b2 100644
--- a/lib/libvcc/vcc_vmod.c
+++ b/lib/libvcc/vcc_vmod.c
@@ -136,12 +136,13 @@ vcc_ParseImport(struct vcc *tl)
 		    fnpx != NULL ? fnpx : fn);
 		VSB_printf(tl->sb, "\tdlerror: %s\n", dlerror());
 		vcc_ErrWhere(tl, mod);
-		REPLACE(fnpx, NULL);
+		free(fnpx);
 		return;
 	}
 
+	AN(fnpx);
 	fnp = TlDup(tl, fnpx);
-	REPLACE(fnpx, NULL);
+	free(fnpx);
 
 	bprintf(buf, "Vmod_%.*s_Data", PF(mod));
 	vmd = dlsym(hdl, buf);



From phk at FreeBSD.org  Tue Feb  2 11:57:55 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 12:57:55 +0100
Subject: [master] 0311c78 Replace alien FD's with /dev/null rather than just
 closing them
Message-ID: <E1aQZal-0002Ol-0q@project.varnish-software.com>


commit 0311c78353cbacbec73a155807d3777749dd0802
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 11:51:14 2016 +0000

    Replace alien FD's with /dev/null rather than just closing them
    
    When we fork the worker process, we close all filedescriptors we
    have not explictly marked for it to inherit, for security reasons.
    
    Operating system libraries may have open filedescriptors (see
    end*ent(3)) and there is no way to chase these down.
    
    At least on OSX something related to DNS lookups leaves such
    a FD around, and when that code later discovers the FD doesn't
    work, it closes it, even though it no longer owns it.
    
    In ticket 1841, that happens to be FD7 which is one of our kqueue FDs.
    
    Normally such library code should set 'close-on-exec' status with
    fcntl(2) but that doesn't seem to be the case here, and this bit
    of wisdom seems neglegted about 50/50, so it probably wouldn't
    help us to examine this.
    
    The fix here is to close the FDs, and replace them with a FD open
    to /dev/null, so that there is no risk of information leak, but we
    don't reuse the FD for something else until the library has properly
    closed it.
    
    Fixes #1841

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index ae01a91..8cedeaf 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -295,7 +295,7 @@ mgt_launch_child(struct cli *cli)
 	unsigned u;
 	char *p;
 	struct vev *e;
-	int i, cp[2];
+	int i, j, k, cp[2];
 	struct sigaction sa;
 
 	if (child_state != CH_STOPPED && child_state != CH_DIED)
@@ -351,13 +351,29 @@ mgt_launch_child(struct cli *cli)
 		assert(dup2(heritage.std_fd, STDOUT_FILENO) == STDOUT_FILENO);
 		assert(dup2(heritage.std_fd, STDERR_FILENO) == STDERR_FILENO);
 
-		/* Close anything we shouldn't know about */
+		/*
+		 * Close all FDs the child shouldn't know about
+		 *
+		 * We cannot just close these filedescriptors, some random
+		 * library routine might miss it later on and wantonly close
+		 * a FD we use at that point in time. (See bug #1841).
+		 * We close the FD and replace it with /dev/null instead,
+		 * That prevents security leakage, and gives the library
+		 * code a valid FD to close when it discovers the changed
+		 * circumstances.
+		 */
 		closelog();
 
 		for (i = STDERR_FILENO + 1; i < CLOSE_FD_UP_TO; i++) {
 			if (vbit_test(fd_map, i))
 				continue;
-			(void)(close(i) == 0);
+			if (close(i) == 0) {
+				k = open("/dev/null", O_RDONLY);
+				assert(k >= 0);
+				j = dup2(k, i);
+				assert(j == i);
+				AZ(close(k));
+			}
 		}
 #ifdef HAVE_SETPROCTITLE
 		setproctitle("Varnish-Chld %s", heritage.name);



From phk at FreeBSD.org  Tue Feb  2 22:46:41 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 02 Feb 2016 23:46:41 +0100
Subject: [master] 62932b4 Turn the cnt_vdp() subroutine into a proper step in
 the FSM.
Message-ID: <E1aQjib-0007HR-1P@project.varnish-software.com>


commit 62932b422f311ed1224f14a216169bcdc1b77a2d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 22:46:13 2016 +0000

    Turn the cnt_vdp() subroutine into a proper step in the FSM.

diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 70adcdb..56ecaed 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -47,67 +47,6 @@
 #include "vsha256.h"
 #include "vtim.h"
 
-static void
-cnt_vdp(struct req *req, struct boc *boc)
-{
-	const char *r;
-	uint16_t status;
-	int sendbody;
-	intmax_t resp_len;
-
-	CHECK_OBJ_NOTNULL(req->transport, TRANSPORT_MAGIC);
-
-	resp_len = http_GetContentLength(req->resp);
-	if (boc != NULL)
-		req->resp_len = resp_len;
-	else
-		req->resp_len = ObjGetLen(req->wrk, req->objcore);
-
-	req->res_mode = 0;
-
-	/* RFC 7230, 3.3.3 */
-	status = http_GetStatus(req->resp);
-	if (!strcmp(req->http0->hd[HTTP_HDR_METHOD].b, "HEAD")) {
-		if (req->objcore->flags & OC_F_PASS)
-			sendbody = -1;
-		else
-			sendbody = 0;
-	} else if (status < 200 || status == 204 || status == 304) {
-		req->resp_len = -1;
-		sendbody = 0;
-	} else
-		sendbody = 1;
-
-	if (!req->disable_esi && req->resp_len != 0 &&
-	    ObjGetattr(req->wrk, req->objcore, OA_ESIDATA, NULL) != NULL)
-		VDP_push(req, VDP_ESI, NULL, 0);
-
-	if (cache_param->http_gzip_support &&
-	    ObjCheckFlag(req->wrk, req->objcore, OF_GZIPED) &&
-	    !RFC2616_Req_Gzip(req->http))
-		VDP_push(req, VDP_gunzip, NULL, 1);
-
-	if (cache_param->http_range_support && http_IsStatus(req->resp, 200)) {
-		http_SetHeader(req->resp, "Accept-Ranges: bytes");
-		if (sendbody && http_GetHdr(req->http, H_Range, &r))
-			VRG_dorange(req, r);
-	}
-
-	if (sendbody < 0) {
-		/* Don't touch pass+HEAD C-L */
-		sendbody = 0;
-	} else if (resp_len >= 0 && resp_len == req->resp_len) {
-		/* Reuse C-L header */
-	} else {
-		http_Unset(req->resp, H_Content_Length);
-		if (req->resp_len >= 0 && sendbody)
-			http_PrintfHeader(req->resp,
-			    "Content-Length: %jd", req->resp_len);
-	}
-
-	req->transport->deliver(req, boc, sendbody);
-}
-
 /*--------------------------------------------------------------------
  * Deliver an object to client
  */
@@ -115,7 +54,6 @@ cnt_vdp(struct req *req, struct boc *boc)
 static enum req_fsm_nxt
 cnt_deliver(struct worker *wrk, struct req *req)
 {
-	struct boc *boc;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
@@ -196,30 +134,8 @@ cnt_deliver(struct worker *wrk, struct req *req)
 	    && req->http->conds && RFC2616_Do_Cond(req))
 		http_PutResponse(req->resp, "HTTP/1.1", 304, NULL);
 
-	/* Grab a ref to the bo if there is one, and hand it down */
-	boc = HSH_RefBusy(req->objcore);
-
-	cnt_vdp(req, boc);
-
-	VSLb_ts_req(req, "Resp", W_TIM_real(wrk));
-
-	if (http_HdrIs(req->resp, H_Connection, "close"))
-		req->doclose = SC_RESP_CLOSE;
-
-	if (req->objcore->flags & (OC_F_PRIVATE | OC_F_PASS)) {
-		if (boc != NULL) {
-			HSH_Abandon(req->objcore);
-			ObjWaitState(req->objcore, BOS_FINISHED);
-		}
-		ObjSlim(wrk, req->objcore);
-	}
-
-	if (boc != NULL)
-		HSH_DerefBusy(wrk, req->objcore);
-
-	(void)HSH_DerefObjCore(wrk, &req->objcore);
-	http_Teardown(req->resp);
-	return (REQ_FSM_DONE);
+	req->req_step = R_STP_TRANSMIT;
+	return (REQ_FSM_MORE);
 }
 
 /*--------------------------------------------------------------------
@@ -276,39 +192,128 @@ cnt_synth(struct worker *wrk, struct req *req)
 	}
 	assert(wrk->handling == VCL_RET_DELIVER);
 
-	if (http_HdrIs(req->resp, H_Connection, "close"))
-		req->doclose = SC_RESP_CLOSE;
-
 	req->objcore = HSH_Private(wrk, 0);
 	AZ(req->objcore->boc);
 	CHECK_OBJ_NOTNULL(req->objcore, OBJCORE_MAGIC);
+	szl = -1;
 	if (STV_NewObject(wrk, req->objcore, TRANSIENT_STORAGE, 1024)) {
 		szl = VSB_len(synth_body);
 		assert(szl >= 0);
-	} else
-		szl = -1;
-	if (szl > 0) {
 		sz = szl;
-		if (ObjGetSpace(wrk, req->objcore, &sz, &ptr) && sz >= szl) {
+		if (sz > 0 &&
+		    ObjGetSpace(wrk, req->objcore, &sz, &ptr) && sz >= szl) {
 			memcpy(ptr, VSB_data(synth_body), szl);
 			ObjExtend(wrk, req->objcore, szl);
-		} else
+		} else if (sz > 0) {
 			szl = -1;
+		}
 	}
+
+	VSB_delete(synth_body);
+
 	if (szl < 0) {
 		VSLb(req->vsl, SLT_Error, "Could not get storage");
 		req->doclose = SC_OVERLOAD;
+		VSLb_ts_req(req, "Resp", W_TIM_real(wrk));
+		(void)HSH_DerefObjCore(wrk, &req->objcore);
+		http_Teardown(req->resp);
+		return (REQ_FSM_DONE);
+	}
+
+	req->req_step = R_STP_TRANSMIT;
+	return (REQ_FSM_MORE);
+}
+
+/*--------------------------------------------------------------------
+ * The mechanics of sending a response (from deliver or synth)
+ */
+
+static enum req_fsm_nxt
+cnt_transmit(struct worker *wrk, struct req *req)
+{
+	struct boc *boc;
+	const char *r;
+	uint16_t status;
+	int sendbody;
+	intmax_t resp_len;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+
+	/* Grab a ref to the bo if there is one, and hand it down */
+	boc = HSH_RefBusy(req->objcore);
+
+	CHECK_OBJ_NOTNULL(req->transport, TRANSPORT_MAGIC);
+
+	resp_len = http_GetContentLength(req->resp);
+	if (boc != NULL)
+		req->resp_len = resp_len;
+	else
+		req->resp_len = ObjGetLen(req->wrk, req->objcore);
+
+	req->res_mode = 0;
+
+	/* RFC 7230, 3.3.3 */
+	status = http_GetStatus(req->resp);
+	if (!strcmp(req->http0->hd[HTTP_HDR_METHOD].b, "HEAD")) {
+		if (req->objcore->flags & OC_F_PASS)
+			sendbody = -1;
+		else
+			sendbody = 0;
+	} else if (status < 200 || status == 204 || status == 304) {
+		req->resp_len = -1;
+		sendbody = 0;
+	} else
+		sendbody = 1;
+
+	if (!req->disable_esi && req->resp_len != 0 &&
+	    ObjGetattr(req->wrk, req->objcore, OA_ESIDATA, NULL) != NULL)
+		VDP_push(req, VDP_ESI, NULL, 0);
+
+	if (cache_param->http_gzip_support &&
+	    ObjCheckFlag(req->wrk, req->objcore, OF_GZIPED) &&
+	    !RFC2616_Req_Gzip(req->http))
+		VDP_push(req, VDP_gunzip, NULL, 1);
+
+	if (cache_param->http_range_support && http_IsStatus(req->resp, 200)) {
+		http_SetHeader(req->resp, "Accept-Ranges: bytes");
+		if (sendbody && http_GetHdr(req->http, H_Range, &r))
+			VRG_dorange(req, r);
+	}
+
+	if (sendbody < 0) {
+		/* Don't touch pass+HEAD C-L */
+		sendbody = 0;
+	} else if (resp_len >= 0 && resp_len == req->resp_len) {
+		/* Reuse C-L header */
 	} else {
-		cnt_vdp(req, NULL);
+		http_Unset(req->resp, H_Content_Length);
+		if (req->resp_len >= 0 && sendbody)
+			http_PrintfHeader(req->resp,
+			    "Content-Length: %jd", req->resp_len);
 	}
 
-	(void)HSH_DerefObjCore(wrk, &req->objcore);
-	VSB_delete(synth_body);
+	req->transport->deliver(req, boc, sendbody);
 
 	VSLb_ts_req(req, "Resp", W_TIM_real(wrk));
 
-	req->err_code = 0;
-	req->err_reason = NULL;
+	if (http_HdrIs(req->resp, H_Connection, "close"))
+		req->doclose = SC_RESP_CLOSE;
+
+	if (req->objcore->flags & (OC_F_PRIVATE | OC_F_PASS)) {
+		if (boc != NULL) {
+			HSH_Abandon(req->objcore);
+			ObjWaitState(req->objcore, BOS_FINISHED);
+		}
+		ObjSlim(wrk, req->objcore);
+	}
+
+	if (boc != NULL)
+		HSH_DerefBusy(wrk, req->objcore);
+
+	(void)HSH_DerefObjCore(wrk, &req->objcore);
+	http_Teardown(req->resp);
+
 	return (REQ_FSM_DONE);
 }
 
diff --git a/include/tbl/steps.h b/include/tbl/steps.h
index 4756d44..5407615 100644
--- a/include/tbl/steps.h
+++ b/include/tbl/steps.h
@@ -53,6 +53,7 @@ REQ_STEP(miss,		MISS,		(wrk, req))
 REQ_STEP(fetch,		FETCH,		(wrk, req))
 REQ_STEP(deliver,	DELIVER,	(wrk, req))
 REQ_STEP(synth,		SYNTH,		(wrk, req))
+REQ_STEP(transmit,	TRANSMIT,	(wrk, req))
 #endif
 
 #ifdef FETCH_STEP



From phk at FreeBSD.org  Tue Feb  2 23:50:33 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 00:50:33 +0100
Subject: [master] b2b35a5 Reorganize the OC_F and OC_EF flags to make space
 for a bitmap of OAs.
Message-ID: <E1aQkiP-0004mH-3P@project.varnish-software.com>


commit b2b35a52ea0608a8d9600f79e0c45a38b5527114
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 23:49:32 2016 +0000

    Reorganize the OC_F and OC_EF flags to make space for a bitmap of OAs.
    
    Rename Obj[GS]etattr() to ..Attr
    
    Add ObjHasAttr() for cheap existence check on OAs.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 310d254..7dfb482 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -420,20 +420,22 @@ struct objcore {
 
 	struct exp		exp;
 
-	uint16_t		flags;
+	uint8_t			flags;
 #define OC_F_BUSY		(1<<1)
 #define OC_F_PASS		(1<<2)
 #define OC_F_INCOMPLETE		(1<<3)
 #define OC_F_ABANDON		(1<<4)
-#define OC_F_PRIVATE		(1<<8)
-#define OC_F_FAILED		(1<<9)
+#define OC_F_PRIVATE		(1<<5)
+#define OC_F_FAILED		(1<<6)
 
-	uint16_t		exp_flags;
-#define OC_EF_OFFLRU		(1<<4)
-#define OC_EF_MOVE		(1<<10)
-#define OC_EF_INSERT		(1<<11)
-#define OC_EF_EXP		(1<<12)
-#define OC_EF_DYING		(1<<7)
+	uint8_t			exp_flags;
+#define OC_EF_OFFLRU		(1<<1)
+#define OC_EF_MOVE		(1<<2)
+#define OC_EF_INSERT		(1<<3)
+#define OC_EF_EXP		(1<<4)
+#define OC_EF_DYING		(1<<5)
+
+	uint16_t		oa_present;
 
 	unsigned		timer_idx;
 	VTAILQ_ENTRY(objcore)	list;
@@ -860,9 +862,10 @@ void ObjUpdateMeta(struct worker *, struct objcore *);
 void ObjFreeObj(struct worker *, struct objcore *);
 void ObjSlim(struct worker *, struct objcore *oc);
 struct lru *ObjGetLRU(const struct objcore *);
-void *ObjGetattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
+int ObjHasAttr(struct worker *, struct objcore *, enum obj_attr);
+void *ObjGetAttr(struct worker *, struct objcore *, enum obj_attr,
     ssize_t *len);
-void *ObjSetattr(struct worker *, struct objcore *, enum obj_attr attr,
+void *ObjSetAttr(struct worker *, struct objcore *, enum obj_attr,
     ssize_t len, const void *);
 int ObjCopyAttr(struct worker *, struct objcore *, struct objcore *,
     enum obj_attr attr);
diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index efa6fb2..88ab55c 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -254,7 +254,7 @@ VDP_ESI(struct req *req, enum vdp_action act, void **priv,
 	while (1) {
 		switch (ecx->state) {
 		case 0:
-			ecx->p = ObjGetattr(req->wrk, req->objcore,
+			ecx->p = ObjGetAttr(req->wrk, req->objcore,
 			    OA_ESIDATA, &l);
 			AN(ecx->p);
 			assert(l > 0);
@@ -699,7 +699,7 @@ ved_stripgzip(struct req *req, const struct boc *boc)
 	 * padding it, as necessary, to a byte boundary.
 	 */
 
-	p = ObjGetattr(req->wrk, req->objcore, OA_GZIPBITS, &l);
+	p = ObjGetAttr(req->wrk, req->objcore, OA_GZIPBITS, &l);
 	AN(p);
 	assert(l == 32);
 	foo.start = vbe64dec(p);
diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
index e3a50db..e1b9a56 100644
--- a/bin/varnishd/cache/cache_esi_fetch.c
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -123,7 +123,7 @@ vfp_esi_end(struct vfp_ctx *vc, struct vef_priv *vef,
 		if (retval == VFP_END) {
 			l = VSB_len(vsb);
 			assert(l > 0);
-			p = ObjSetattr(vc->wrk, vc->oc,
+			p = ObjSetAttr(vc->wrk, vc->oc,
 			    OA_ESIDATA, l, VSB_data(vsb));
 			if (p == NULL) {
 				retval = VFP_Error(vc,
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 5332e7c..99b283d 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -129,7 +129,7 @@ vbf_beresp2obj(struct busyobj *bo)
 		return (-1);
 
 	if (vary != NULL) {
-		b = ObjSetattr(bo->wrk, bo->fetch_objcore, OA_VARY, varyl,
+		b = ObjSetAttr(bo->wrk, bo->fetch_objcore, OA_VARY, varyl,
 		    VSB_data(vary));
 		VSB_delete(vary);
 	}
@@ -140,7 +140,7 @@ vbf_beresp2obj(struct busyobj *bo)
 	bo->beresp->logtag = SLT_ObjMethod;
 
 	/* Filter into object */
-	bp = ObjSetattr(bo->wrk, bo->fetch_objcore, OA_HEADERS, l2, NULL);
+	bp = ObjSetAttr(bo->wrk, bo->fetch_objcore, OA_HEADERS, l2, NULL);
 	AN(bp);
 	HTTP_Encode(bo->beresp, bp, l2,
 	    bo->uncacheable ? HTTPH_R_PASS : HTTPH_A_INS);
@@ -727,7 +727,7 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 
 	AZ(vbf_beresp2obj(bo));
 
-	if (ObjGetattr(bo->wrk, bo->stale_oc, OA_ESIDATA, NULL) != NULL)
+	if (ObjHasAttr(bo->wrk, bo->stale_oc, OA_ESIDATA))
 		AZ(ObjCopyAttr(bo->wrk, bo->fetch_objcore, bo->stale_oc,
 		    OA_ESIDATA));
 
diff --git a/bin/varnishd/cache/cache_gzip.c b/bin/varnishd/cache/cache_gzip.c
index 917109a..0b8d3a7 100644
--- a/bin/varnishd/cache/cache_gzip.c
+++ b/bin/varnishd/cache/cache_gzip.c
@@ -310,7 +310,7 @@ VDP_gunzip(struct req *req, enum vdp_action act, void **priv,
 		if (req->objcore->flags & OC_F_INCOMPLETE)
 			return (0);	/* No idea about length */
 
-		p = ObjGetattr(req->wrk, req->objcore, OA_GZIPBITS, &dl);
+		p = ObjGetAttr(req->wrk, req->objcore, OA_GZIPBITS, &dl);
 		if (p == NULL || dl != 32)
 			return (0); /* No OA_GZIPBITS yet */
 
@@ -369,7 +369,7 @@ VGZ_UpdateObj(const struct vfp_ctx *vc, struct vgz *vg, enum vgz_ua_e e)
 	if (e == VUA_UPDATE && ii == vg->bits)
 		return;
 	vg->bits = ii;
-	p = ObjSetattr(vc->wrk, vc->oc, OA_GZIPBITS, 32, NULL);
+	p = ObjSetAttr(vc->wrk, vc->oc, OA_GZIPBITS, 32, NULL);
 	AN(p);
 	vbe64enc(p, vg->vz.start_bit);
 	vbe64enc(p + 8, vg->vz.last_bit);
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 7d22c25..fb1c5db 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -418,10 +418,11 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 		if (BAN_CheckObject(wrk, oc, req))
 			continue;
 
-		vary = ObjGetattr(wrk, oc, OA_VARY, NULL);
-
-		if (vary != NULL && !VRY_Match(req, vary))
-			continue;
+		if (ObjHasAttr(wrk, oc, OA_VARY)) {
+			vary = ObjGetAttr(wrk, oc, OA_VARY, NULL);
+			if (!VRY_Match(req, vary))
+				continue;
+		}
 
 		if (EXP_Ttl(req, &oc->exp) >= req->t_req) {
 			/* If still valid, use it */
diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c
index 8e3fe0f..7caaafd 100644
--- a/bin/varnishd/cache/cache_http.c
+++ b/bin/varnishd/cache/cache_http.c
@@ -896,7 +896,7 @@ uint16_t
 HTTP_GetStatusPack(struct worker *wrk, struct objcore *oc)
 {
 	const char *ptr;
-	ptr = ObjGetattr(wrk, oc, OA_HEADERS, NULL);
+	ptr = ObjGetAttr(wrk, oc, OA_HEADERS, NULL);
 	AN(ptr);
 
 	return(vbe16dec(ptr + 2));
@@ -915,7 +915,7 @@ HTTP_IterHdrPack(struct worker *wrk, struct objcore *oc, const char **p)
 	AN(p);
 
 	if (*p == NULL) {
-		ptr = ObjGetattr(wrk, oc, OA_HEADERS, NULL);
+		ptr = ObjGetAttr(wrk, oc, OA_HEADERS, NULL);
 		AN(ptr);
 		ptr += 4;	/* Skip nhd and status */
 		ptr = strchr(ptr, '\0') + 1;	/* Skip :proto: */
@@ -948,7 +948,7 @@ HTTP_GetHdrPack(struct worker *wrk, struct objcore *oc, const char *hdr)
 
 	if (hdr[0] == ':') {
 		/* Special cases */
-		ptr = ObjGetattr(wrk, oc, OA_HEADERS, NULL);
+		ptr = ObjGetAttr(wrk, oc, OA_HEADERS, NULL);
 		AN(ptr);
 		ptr += 4;	/* Skip nhd and status */
 
@@ -991,7 +991,7 @@ HTTP_Merge(struct worker *wrk, struct objcore *oc, struct http *to)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(to, HTTP_MAGIC);
 
-	ptr = ObjGetattr(wrk, oc, OA_HEADERS, NULL);
+	ptr = ObjGetAttr(wrk, oc, OA_HEADERS, NULL);
 	AN(ptr);
 
 	to->status = vbe16dec(ptr + 2);
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index b6a60c2..a1d2178 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -297,13 +297,33 @@ ObjGetLRU(const struct objcore *oc)
 }
 
 /*====================================================================
- * ObjGetattr()
+ * ObjHasAttr()
+ *
+ * Check if object has this attribute
+ */
+
+int
+ObjHasAttr(struct worker *wrk, struct objcore *oc, enum obj_attr attr)
+{
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+
+	if (oc->oa_present)
+		return (oc->oa_present & (1 << attr));
+
+	/* resurrected persistent objects don't have oa_present set */
+	return (ObjGetAttr(wrk, oc, attr, NULL) != NULL ? 1 : 0);
+}
+
+/*====================================================================
+ * ObjGetAttr()
  *
  * Get an attribute of the object.
  */
 
 void *
-ObjGetattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
+ObjGetAttr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
    ssize_t *len)
 {
 	const struct obj_methods *om = obj_getmethods(oc);
@@ -315,14 +335,14 @@ ObjGetattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
 }
 
 /*====================================================================
- * ObjSetattr()
+ * ObjSetAttr()
  *
  * If ptr is Non-NULL, it points to the new content which is copied into
  * the attribute.  Otherwise the caller will have to do the copying.
  */
 
 void *
-ObjSetattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
+ObjSetAttr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
     ssize_t len, const void *ptr)
 {
 	const struct obj_methods *om = obj_getmethods(oc);
@@ -330,6 +350,8 @@ ObjSetattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
 	AN(om->objsetattr);
+	assert((int)attr < 16);
+	oc->oa_present |= (1 << attr);
 	return (om->objsetattr(wrk, oc, attr, len, ptr));
 }
 
@@ -362,11 +384,11 @@ ObjCopyAttr(struct worker *wrk, struct objcore *oc, struct objcore *ocs,
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(ocs, OBJCORE_MAGIC);
 
-	vps = ObjGetattr(wrk, ocs, attr, &l);
+	vps = ObjGetAttr(wrk, ocs, attr, &l);
 	// XXX: later we want to have zero-length OA's too
 	if (vps == NULL || l <= 0)
 		return (-1);
-	vpd = ObjSetattr(wrk, oc, attr, l, vps);
+	vpd = ObjSetAttr(wrk, oc, attr, l, vps);
 	if (vpd == NULL)
 		return (-1);
 	return (0);
@@ -397,7 +419,7 @@ ObjSetDouble(struct worker *wrk, struct objcore *oc, enum obj_attr a, double t)
 
 	assert(sizeof t == sizeof u);
 	memcpy(&u, &t, sizeof u);
-	vp = ObjSetattr(wrk, oc, a, sizeof u, NULL);
+	vp = ObjSetAttr(wrk, oc, a, sizeof u, NULL);
 	if (vp == NULL)
 		return (-1);
 	vbe64enc(vp, u);
@@ -412,7 +434,7 @@ ObjGetDouble(struct worker *wrk, struct objcore *oc, enum obj_attr a, double *d)
 	ssize_t l;
 
 	assert(sizeof *d == sizeof u);
-	vp = ObjGetattr(wrk, oc, a, &l);
+	vp = ObjGetAttr(wrk, oc, a, &l);
 	if (vp == NULL)
 		return (-1);
 	if (d != NULL) {
@@ -431,7 +453,7 @@ ObjSetU64(struct worker *wrk, struct objcore *oc, enum obj_attr a, uint64_t t)
 {
 	void *vp;
 
-	vp = ObjSetattr(wrk, oc, a, sizeof t, NULL);
+	vp = ObjSetAttr(wrk, oc, a, sizeof t, NULL);
 	if (vp == NULL)
 		return (-1);
 	vbe64enc(vp, t);
@@ -444,7 +466,7 @@ ObjGetU64(struct worker *wrk, struct objcore *oc, enum obj_attr a, uint64_t *d)
 	void *vp;
 	ssize_t l;
 
-	vp = ObjGetattr(wrk, oc, a, &l);
+	vp = ObjGetAttr(wrk, oc, a, &l);
 	if (vp == NULL || l != sizeof *d)
 		return (-1);
 	if (d != NULL)
@@ -457,7 +479,7 @@ ObjSetU32(struct worker *wrk, struct objcore *oc, enum obj_attr a, uint32_t t)
 {
 	void *vp;
 
-	vp = ObjSetattr(wrk, oc, a, sizeof t, NULL);
+	vp = ObjSetAttr(wrk, oc, a, sizeof t, NULL);
 	if (vp == NULL)
 		return (-1);
 	vbe32enc(vp, t);
@@ -470,7 +492,7 @@ ObjGetU32(struct worker *wrk, struct objcore *oc, enum obj_attr a, uint32_t *d)
 	void *vp;
 	ssize_t l;
 
-	vp = ObjGetattr(wrk, oc, a, &l);
+	vp = ObjGetAttr(wrk, oc, a, &l);
 	if (vp == NULL || l != sizeof *d)
 		return (-1);
 	if (d != NULL)
@@ -486,7 +508,7 @@ ObjCheckFlag(struct worker *wrk, struct objcore *oc, enum obj_flags of)
 {
 	uint8_t *fp;
 
-	fp = ObjGetattr(wrk, oc, OA_FLAGS, NULL);
+	fp = ObjGetAttr(wrk, oc, OA_FLAGS, NULL);
 	AN(fp);
 	return ((*fp) & of);
 }
@@ -496,7 +518,7 @@ ObjSetFlag(struct worker *wrk, struct objcore *oc, enum obj_flags of, int val)
 {
 	uint8_t *fp;
 
-	fp = ObjSetattr(wrk, oc, OA_FLAGS, 1, NULL);
+	fp = ObjSetAttr(wrk, oc, OA_FLAGS, 1, NULL);
 	AN(fp);
 	if (val)
 		(*fp) |= of;
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 56ecaed..1926536 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -68,7 +68,7 @@ cnt_deliver(struct worker *wrk, struct req *req)
 
 	HTTP_Setup(req->resp, req->ws, req->vsl, SLT_RespMethod);
 	if (HTTP_Decode(req->resp,
-	    ObjGetattr(req->wrk, req->objcore, OA_HEADERS, NULL))) {
+	    ObjGetAttr(req->wrk, req->objcore, OA_HEADERS, NULL))) {
 		req->err_code = 500;
 		req->req_step = R_STP_SYNTH;
 		return (REQ_FSM_MORE);
@@ -267,7 +267,7 @@ cnt_transmit(struct worker *wrk, struct req *req)
 		sendbody = 1;
 
 	if (!req->disable_esi && req->resp_len != 0 &&
-	    ObjGetattr(req->wrk, req->objcore, OA_ESIDATA, NULL) != NULL)
+	    ObjHasAttr(wrk, req->objcore, OA_ESIDATA))
 		VDP_push(req, VDP_ESI, NULL, 0);
 
 	if (cache_param->http_gzip_support &&



From phk at FreeBSD.org  Wed Feb  3 00:02:57 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 01:02:57 +0100
Subject: [master] e5642b2 Minor polish
Message-ID: <E1aQkuP-00061T-2e@project.varnish-software.com>


commit e5642b2cbc3d20902025bf4ac4cc34e7822ca510
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 00:02:47 2016 +0000

    Minor polish

diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 1926536..a24ca3a 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -235,19 +235,18 @@ cnt_transmit(struct worker *wrk, struct req *req)
 	const char *r;
 	uint16_t status;
 	int sendbody;
-	intmax_t resp_len;
+	intmax_t clval;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	CHECK_OBJ_NOTNULL(req->transport, TRANSPORT_MAGIC);
 
-	/* Grab a ref to the bo if there is one, and hand it down */
+	/* Grab a ref to the bo if there is one */
 	boc = HSH_RefBusy(req->objcore);
 
-	CHECK_OBJ_NOTNULL(req->transport, TRANSPORT_MAGIC);
-
-	resp_len = http_GetContentLength(req->resp);
+	clval = http_GetContentLength(req->resp);
 	if (boc != NULL)
-		req->resp_len = resp_len;
+		req->resp_len = clval;
 	else
 		req->resp_len = ObjGetLen(req->wrk, req->objcore);
 
@@ -266,25 +265,28 @@ cnt_transmit(struct worker *wrk, struct req *req)
 	} else
 		sendbody = 1;
 
-	if (!req->disable_esi && req->resp_len != 0 &&
-	    ObjHasAttr(wrk, req->objcore, OA_ESIDATA))
-		VDP_push(req, VDP_ESI, NULL, 0);
-
-	if (cache_param->http_gzip_support &&
-	    ObjCheckFlag(req->wrk, req->objcore, OF_GZIPED) &&
-	    !RFC2616_Req_Gzip(req->http))
-		VDP_push(req, VDP_gunzip, NULL, 1);
-
-	if (cache_param->http_range_support && http_IsStatus(req->resp, 200)) {
-		http_SetHeader(req->resp, "Accept-Ranges: bytes");
-		if (sendbody && http_GetHdr(req->http, H_Range, &r))
-			VRG_dorange(req, r);
+	if (sendbody >= 0) {
+		if (!req->disable_esi && req->resp_len != 0 &&
+		    ObjHasAttr(wrk, req->objcore, OA_ESIDATA))
+			VDP_push(req, VDP_ESI, NULL, 0);
+
+		if (cache_param->http_gzip_support &&
+		    ObjCheckFlag(req->wrk, req->objcore, OF_GZIPED) &&
+		    !RFC2616_Req_Gzip(req->http))
+			VDP_push(req, VDP_gunzip, NULL, 1);
+
+		if (cache_param->http_range_support &&
+		    http_IsStatus(req->resp, 200)) {
+			http_SetHeader(req->resp, "Accept-Ranges: bytes");
+			if (sendbody && http_GetHdr(req->http, H_Range, &r))
+				VRG_dorange(req, r);
+		}
 	}
 
 	if (sendbody < 0) {
 		/* Don't touch pass+HEAD C-L */
 		sendbody = 0;
-	} else if (resp_len >= 0 && resp_len == req->resp_len) {
+	} else if (clval >= 0 && clval == req->resp_len) {
 		/* Reuse C-L header */
 	} else {
 		http_Unset(req->resp, H_Content_Length);
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 5939a57..c782e25 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -565,7 +565,7 @@ sml_setattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
 		if (o->esidata == NULL)
 			return (NULL);
 		o->esidata->len = len;
-		retval  = o->esidata->ptr;
+		retval = o->esidata->ptr;
 		break;
 	case OA_FLAGS:
 		assert(len == sizeof o->oa_flags);



From fgsch at lodoss.net  Wed Feb  3 08:18:41 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 03 Feb 2016 09:18:41 +0100
Subject: [master] c1ee4c9 Random typos and grammar fixes
Message-ID: <E1aQse9-0003Ev-0K@project.varnish-software.com>


commit c1ee4c9f76da0a27305f5f022c86cad5628a2703
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Feb 3 08:16:54 2016 +0000

    Random typos and grammar fixes
    
    Some submitted by: github::sublimino

diff --git a/doc/changes.rst b/doc/changes.rst
index 154c84d..6194011 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -566,7 +566,7 @@ Changes from 3.0.6 to 3.0.7-rc1 (2015-03-18)
 - Requests with multiple Content-Length headers will now fail.
 
 - Stop recognizing a single CR (\r) as a HTTP line separator.
-  This opened up a possible cache poisioning attack in stacked installations
+  This opened up a possible cache poisoning attack in stacked installations
   where sslterminator/varnish/backend had different CR handling.
 
 - Improved error detection on master-child process communication, leading to
diff --git a/doc/sphinx/phk/10goingon50.rst b/doc/sphinx/phk/10goingon50.rst
index 6960e1f..ffe4349 100644
--- a/doc/sphinx/phk/10goingon50.rst
+++ b/doc/sphinx/phk/10goingon50.rst
@@ -20,7 +20,7 @@ subversion commit number 9::
     commit 523166ad2dd3a65e3987f13bc54f571f98453976
     Author: Dag Erling Sm?rgrav <des at des.no>
     Date:   Wed Feb 22 14:31:39 2006 +0000
-    
+
         Additional subdivisions.
 
 We consider this the official birth-certificate of the Varnish Cache
@@ -39,13 +39,13 @@ One part of the celebration, somehow, sometime, will be the "VCL
 Obfuscated Code Contest #1" in the same spirit as the `International
 Obfuscated C Code Contest <http://www.ioccc.org/>`_.
 
-True afficionados of Obfuscated Code will also appreciate this
+True aficionados of Obfuscated Code will also appreciate this
 amazing `Obfuscated PL/1 <http://www.multicians.org/proc-proc.html>`_.
 
 The official VCLOCC1 contest rules are simple:
 
 * VCL code must work with Varnish 4.1.1
-* As many Varnishd instances as you'd like. 
+* As many Varnishd instances as you'd like.
 * No inline-C allowed
 * Any VMOD you want is OK
 * You get to choose the request(s) to send to Varnishd
@@ -63,7 +63,7 @@ you are more than welcome to start already now.
 Releases
 --------
 
-Our 10 year aniversary was a good excuse to take stock and look at
+Our 10 year anniversary was a good excuse to take stock and look at
 the way we work, and changes are and will be happening.
 
 Like any respectable FOSS project, the Varnish project has never been
@@ -80,7 +80,7 @@ Come hell or high water [#f1]_, Varnish 5.0 will be released September
 15th 2016.
 
 And the next big release, whatever we call it, will be middle of
-march 2017, and until we change our mind, you can trust a major
+March 2017, and until we change our mind, you can trust a major
 release of Varnish to happen every six months.
 
 Minor releases, typically bugfixes, will be released as need arise,
@@ -95,7 +95,7 @@ We have plans and ideas for what *should* be there, and we will work
 to reach those milestones, but we will not hold the release for "just this
 one more feature" if they are not ready.
 
-If it is in on september 15th, it will be in the release, if not, it wont.
+If it is in on September 15th, it will be in the release, if not, it wont.
 
 And since the next release is guaranteed to come six months later,
 it's not a catastrophe to miss the deadline.
@@ -131,8 +131,8 @@ with the 4.1 release.
 All this obviously changes the dynamics of the project, and it we
 find out it is a disaster, we'll change our mind.
 
-But until then:  Two major releases a year, as clock-work, mid-september
-and mid-march.
+But until then:  Two major releases a year, as clock-work, mid-September
+and mid-March.
 
 ----------------
 Moving to github
@@ -176,7 +176,7 @@ If you are a contributor to Varnish, you should already have the
 nice blue T-shirt and the mug to prove it.  (Thanks Varnish-Software!)
 
 If you merely stumble over a spelling mistake, you merely
-stumbled over a spelling mistake, and we will happily 
+stumbled over a spelling mistake, and we will happily
 correct it, and put your name in the commit message.
 
 But it takes a lot more that fixing a spelling mistake to
@@ -190,7 +190,7 @@ Speaking of which...
 Where does 50 come into it ?
 ----------------------------
 
-On january 20th I celebrated my 50 year birthday, and this was a
+On January 20th I celebrated my 50 year birthday, and this was a
 much more serious affair than I had anticipated:  For the first
 time in my life I have received a basket with wine and flowers on
 my birthday.
diff --git a/doc/sphinx/phk/dough.rst b/doc/sphinx/phk/dough.rst
index aef598b..ea38a2e 100644
--- a/doc/sphinx/phk/dough.rst
+++ b/doc/sphinx/phk/dough.rst
@@ -86,7 +86,7 @@ at least at that time, there were no way to prevent people from
 dropping money into it, no matter how much you wanted to stop them.
 
 In the end I managed to yell loud enough and only got overfunded
-a few percent, and I belive that my attempt to deflect the surplus
+a few percent, and I believe that my attempt to deflect the surplus
 to the FreeBSD Foundation gave them a little boost that year.
 
 So about PayPal:  The first thing they did was to shut my account,
@@ -184,7 +184,7 @@ the message to other communities, that Free and Open Source Software
 does not materialize out of empty space, it is written by people.
 
 People who love what we do, which is why I'm sitting here,
-way past midnight on a friday evening, writing this pamphlet.
+way past midnight on a Friday evening, writing this pamphlet.
 
 But software *is* written by people, real people with kids, cars,
 mortgages, leaky roofs, sick pets, infirm parents and all other
@@ -225,25 +225,25 @@ collect money and hire developers.  This is a relatively complex
 thing to do, and it will only be available for larger projects.
 
 The Apache Foundation "adopts" smaller projects inside their field
-of interest, and I belive that works OK, but I'm not sure if it
+of interest, and I believe that works OK, but I'm not sure if it
 can easily be transplanted to different topics.
 
 The final way is to simply throw money a the developers, the
 way the FreeBSD and Varnish communities have done with me.
 
 It is a far more flexible solution with respect to level of
-engangement, national boundaries etc. etc, but in many ways it
+engagement, national boundaries etc. etc, but in many ways it
 demands more from both sides of the deal, in particular
 with respect to paperwork, taxes and so on.
 
 Conclusion
 ==========
 
-I am obiously biased, I derive a large fraction of my relatively
+I am obviously biased, I derive a large fraction of my relatively
 modest income from community funding, for which I am the Varnish
 community deeply grateful.
 
-But biased as I may be, I belive that the Varnish community and I
+But biased as I may be, I believe that the Varnish community and I
 has shown that a tiny investment goes a long way in Free and Open
 Source Software.
 
diff --git a/doc/sphinx/phk/persistent.rst b/doc/sphinx/phk/persistent.rst
index 536e406..6587e67 100644
--- a/doc/sphinx/phk/persistent.rst
+++ b/doc/sphinx/phk/persistent.rst
@@ -52,7 +52,7 @@ this.
 And in a sense I did.
 
 Varnish has the advantage over databases and filesystems that we
-can actually loose objects without it being a catastrophy.  It would
+can actually loose objects without it being a catastrophe.  It would
 be better if we didn't, but we can simply ditch stuff which doesn't
 look consistent and we'll be safe.
 
diff --git a/doc/sphinx/phk/platforms.rst b/doc/sphinx/phk/platforms.rst
index e85f925..047196b 100644
--- a/doc/sphinx/phk/platforms.rst
+++ b/doc/sphinx/phk/platforms.rst
@@ -108,4 +108,4 @@ are right about Oracles intentions, Solaris may get demoted.
 Until next time,
 
 Poul-Henning, 2010-08-03
-Edited Nils, 2014-03-18 with Poul-Hennings concent
+Edited Nils, 2014-03-18 with Poul-Hennings consent
diff --git a/doc/sphinx/phk/ssl_again.rst b/doc/sphinx/phk/ssl_again.rst
index 31b6efe..d0f8323 100644
--- a/doc/sphinx/phk/ssl_again.rst
+++ b/doc/sphinx/phk/ssl_again.rst
@@ -11,7 +11,7 @@ revisit that issue.
 A SSL/TLS library
 ~~~~~~~~~~~~~~~~~
 
-In 2011 I critized OpenSSL's source-code as being a nightmare,
+In 2011 I criticized OpenSSL's source-code as being a nightmare,
 and as much as I Hate To Say I Told You So, I Told You So:  See also
 "HeartBleed".
 
@@ -19,15 +19,15 @@ The good news is that HeartBleed made people realize that FOSS
 maintainers also have mortgages and hungry kids.
 
 Various initiatives have been launched to make prevent critical
-infrastructure software from being maintained sunday evening between
+infrastructure software from being maintained Sunday evening between
 11 and 12PM by a sleep-deprived and overworked parent, worried about
 about being able to pay the bills come the next month.
 
 We're not there yet, but it's certainly getting better.
 
-However, implementing TLS and SSL is stil insanely complex, and
+However, implementing TLS and SSL is still insanely complex, and
 thanks to Edward Snowdens whistle-blowing, we have very good reasons
-to belive that didn't happen by accident.
+to believe that didn't happen by accident.
 
 The issue of finding a good TLS/SSL implementation is still the
 same and I still don't see one I would want my name associated with.
@@ -40,7 +40,7 @@ Handling Certificates
 ~~~~~~~~~~~~~~~~~~~~~
 
 I still don't see a way to do that.  The Varnish worker-process is not
-built to compartementalize bits at a cryptographic level and making it
+built to compartmentalize bits at a cryptographic level and making it
 do that would be a non-trivial undertaking.
 
 But there is new loop-hole here.
@@ -148,7 +148,7 @@ the privacy of every human being with an internet connection, because
 it takes a lot more skill to look into a SSL connection than a
 plaintext HTTP connection.
 
-"Sunshine is said to be the best of disinfectantants" wrote supreme
+"Sunshine is said to be the best of disinfectants" wrote supreme
 court justice Brandeis, SSL Everywhere puts all traffic in the shade.
 
 Poul-Henning, 2015-04-28
diff --git a/doc/sphinx/phk/thetoolsweworkwith.rst b/doc/sphinx/phk/thetoolsweworkwith.rst
index 4ee56db..1ab4fc5 100644
--- a/doc/sphinx/phk/thetoolsweworkwith.rst
+++ b/doc/sphinx/phk/thetoolsweworkwith.rst
@@ -148,7 +148,7 @@ Because it's not like the call is actually guaranteed to return at
 later than that, so you have to wrap the call in a loop.
 
 Whoever defined the select(2) and poll(2) systemcalls knew better
-than the POSIX and ISO-C group-think:  They specifed a maximum
+than the POSIX and ISO-C group-think:  They specified a maximum
 duration for the call, because then it doesn't matter what time
 it is, only how long time has transpired.
 
diff --git a/doc/sphinx/phk/vcl_expr.rst b/doc/sphinx/phk/vcl_expr.rst
index 6ef02dc..254384d 100644
--- a/doc/sphinx/phk/vcl_expr.rst
+++ b/doc/sphinx/phk/vcl_expr.rst
@@ -32,7 +32,7 @@ precedence operators have.
 
 The C programming language is famous for having a couple of gottchas
 in its precedence rules and given our limited and narrow type
-repetoire, blindly importing a set of precedence rules may confuse
+repertoire, blindly importing a set of precedence rules may confuse
 a lot more than it may help.
 
 Here are the precedence rules I have settled on, from highest to
diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst
index 3149b83..55a2528 100644
--- a/doc/sphinx/reference/varnish-cli.rst
+++ b/doc/sphinx/reference/varnish-cli.rst
@@ -15,7 +15,7 @@ Varnish Command Line Interface
 DESCRIPTION
 ===========
 
-Varnish as a command line interface (CLI) which can control and change
+Varnish has a command line interface (CLI) which can control and change
 most of the operational parameters and the configuration of Varnish,
 without interrupting the running service.
 
diff --git a/doc/sphinx/users-guide/performance.rst b/doc/sphinx/users-guide/performance.rst
index cd00ff5..68c20a2 100644
--- a/doc/sphinx/users-guide/performance.rst
+++ b/doc/sphinx/users-guide/performance.rst
@@ -11,7 +11,7 @@ functions of Varnish that you should be aware of. The next subsection focuses
 on the how to purge content out of your cache. Purging of content is
 essential in a performance context because it allows you to extend the
 *time-to-live* (TTL) of your cached objects. Having a long TTL allows
-Varnish to keep the content in cache longer, meaning Varnish will make fewer requests to your relativly slower backend.
+Varnish to keep the content in cache longer, meaning Varnish will make fewer requests to your relatively slower backend.
 
 The final subsection deals with compression of web content. Varnish can
 gzip content when fetching it from the backend and then deliver it
diff --git a/doc/sphinx/users-guide/vcl-actions.rst b/doc/sphinx/users-guide/vcl-actions.rst
index e66250f..57b2ea8 100644
--- a/doc/sphinx/users-guide/vcl-actions.rst
+++ b/doc/sphinx/users-guide/vcl-actions.rst
@@ -14,7 +14,7 @@ The most common actions to return are these:
 
 *hash*
   When you return hash from `vcl_recv` you tell Varnish to deliver content
-  from cache even if the request othervise indicates that the request
+  from cache even if the request otherwise indicates that the request
   should be passed.
 
 *pipe*
diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst
index a1eba28..21e5873 100644
--- a/doc/sphinx/users-guide/vcl-built-in-subs.rst
+++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst
@@ -230,7 +230,7 @@ vcl_synth
 ~~~~~~~~~
 
 Called to deliver a synthetic object. A synthetic object is generated
-in VCL, not fetched from the backend. Its body may be contructed using
+in VCL, not fetched from the backend. Its body may be constructed using
 the ``synthetic()`` function.
 
 A `vcl_synth` defined object never enters the cache, contrary to a
@@ -325,7 +325,7 @@ vcl_backend_error
 This subroutine is called if we fail the backend fetch or if
 *max_retries* has been exceeded.
 
-A synthetic object is generated in VCL, whose body may be contructed
+A synthetic object is generated in VCL, whose body may be constructed
 using the ``synthetic()`` function.
 
 The `vcl_backend_error` subroutine may terminate with calling ``return()``
diff --git a/doc/sphinx/users-guide/vcl-variables.rst b/doc/sphinx/users-guide/vcl-variables.rst
index 4531537..1e0c832 100644
--- a/doc/sphinx/users-guide/vcl-variables.rst
+++ b/doc/sphinx/users-guide/vcl-variables.rst
@@ -14,7 +14,7 @@ objects can be accessed and manipulated using VCL.
  do on or with the `req` object.
 
 *bereq*
- The backend request object. Varnish contructs this before sending it to the
+ The backend request object. Varnish constructs this before sending it to the
  backend. It is based on the `req` object.
 
 .. XXX:in what way? benc



From phk at FreeBSD.org  Wed Feb  3 10:32:10 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 11:32:10 +0100
Subject: [master] f2882d3 Add ObjKill() for sniping an objcore,
 for instance for LRU-kills.
Message-ID: <E1aQujK-0003QC-W4@project.varnish-software.com>


commit f2882d3c402522cef330cd84217f5cab44748768
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 09:02:40 2016 +0000

    Add ObjKill() for sniping an objcore, for instance for LRU-kills.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 7dfb482..e965cc3 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -855,12 +855,13 @@ ssize_t ObjWaitExtend(struct worker *, struct objcore *, ssize_t l);
 void ObjSetState(const struct objcore *, enum boc_state_e next);
 void ObjWaitState(const struct objcore *, enum boc_state_e want);
 void ObjTrimStore(struct worker *, struct objcore *);
-void ObjTouch(struct worker *wrk, struct objcore *oc, double now);
+void ObjTouch(struct worker *, struct objcore *, double now);
+int ObjKill(const struct worker *, struct objcore *);
 unsigned ObjGetXID(struct worker *, struct objcore *);
-uint64_t ObjGetLen(struct worker *, struct objcore *oc);
+uint64_t ObjGetLen(struct worker *, struct objcore *);
 void ObjUpdateMeta(struct worker *, struct objcore *);
 void ObjFreeObj(struct worker *, struct objcore *);
-void ObjSlim(struct worker *, struct objcore *oc);
+void ObjSlim(struct worker *, struct objcore *);
 struct lru *ObjGetLRU(const struct objcore *);
 int ObjHasAttr(struct worker *, struct objcore *, enum obj_attr);
 void *ObjGetAttr(struct worker *, struct objcore *, enum obj_attr,
@@ -879,7 +880,7 @@ int ObjGetDouble(struct worker *, struct objcore *, enum obj_attr, double *);
 int ObjGetU32(struct worker *, struct objcore *, enum obj_attr, uint32_t *);
 int ObjGetU64(struct worker *, struct objcore *, enum obj_attr, uint64_t *);
 
-int ObjCheckFlag(struct worker *, struct objcore *oc, enum obj_flags of);
+int ObjCheckFlag(struct worker *, struct objcore *, enum obj_flags of);
 void ObjSetFlag(struct worker *, struct objcore *, enum obj_flags of, int val);
 
 /* cache_panic.c */
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 119f521..65cf46c 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -279,7 +279,6 @@ int
 EXP_NukeOne(struct worker *wrk, struct lru *lru)
 {
 	struct objcore *oc, *oc2;
-	struct objhead *oh;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
@@ -292,30 +291,12 @@ EXP_NukeOne(struct worker *wrk, struct lru *lru)
 		    oc, oc->flags, oc->refcnt);
 
 		AZ(oc->exp_flags & OC_EF_OFFLRU);
-		AZ(oc->exp_flags & OC_EF_DYING);
-
-		/*
-		 * It wont release any space if we cannot release the last
-		 * reference, besides, if somebody else has a reference,
-		 * it's a bad idea to nuke this object anyway.
-		 */
-		if (oc->refcnt > 1)
-			continue;
-		oh = oc->objhead;
-		CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
-		if (Lck_Trylock(&oh->mtx))
-			continue;
-		if (oc->refcnt == 1) {
-			oc->exp_flags |= OC_EF_DYING | OC_EF_OFFLRU;
-			oc->refcnt++;
+
+		if (ObjKill(wrk, oc)) {
 			VSC_C_main->n_lru_nuked++; // XXX per lru ?
 			VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-		} else {
-			oc = NULL;
-		}
-		Lck_Unlock(&oh->mtx);
-		if (oc != NULL)
 			break;
+		}
 	}
 	Lck_Unlock(&lru->mtx);
 
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index a1d2178..fb4c897 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -370,6 +370,35 @@ ObjTouch(struct worker *wrk, struct objcore *oc, double now)
 }
 
 /*====================================================================
+ * ObjKill()
+ *
+ * If objcore is idle, gain a ref and mark it dead.
+ */
+
+int
+ObjKill(const struct worker *wrk, struct objcore *oc)
+{
+	int retval = 0;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
+
+	AZ(oc->exp_flags & OC_EF_DYING);
+
+	if (oc->refcnt == 1 && !Lck_Trylock(&oc->objhead->mtx)) {
+		if (oc->refcnt == 1) {
+			oc->exp_flags |= OC_EF_DYING;
+			oc->exp_flags |= OC_EF_OFFLRU;	/* XXX */
+			oc->refcnt++;
+			retval = 1;
+		}
+		Lck_Unlock(&oc->objhead->mtx);
+	}
+	return (retval);
+}
+
+/*====================================================================
  * Utility functions which work on top of the previous ones
  */
 



From phk at FreeBSD.org  Wed Feb  3 10:32:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 11:32:11 +0100
Subject: [master] f267941 Divorce the OFFLRU flag from the EXP flags
Message-ID: <E1aQujL-0003QJ-1Z@project.varnish-software.com>


commit f267941ff257d0b2400bf4ba16df2b19336b97ff
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 09:29:51 2016 +0000

    Divorce the OFFLRU flag from the EXP flags

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index e965cc3..e738271 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -429,7 +429,6 @@ struct objcore {
 #define OC_F_FAILED		(1<<6)
 
 	uint8_t			exp_flags;
-#define OC_EF_OFFLRU		(1<<1)
 #define OC_EF_MOVE		(1<<2)
 #define OC_EF_INSERT		(1<<3)
 #define OC_EF_EXP		(1<<4)
@@ -437,10 +436,12 @@ struct objcore {
 
 	uint16_t		oa_present;
 
-	unsigned		timer_idx;
+	unsigned		timer_idx;	// XXX 4Gobj limit
 	VTAILQ_ENTRY(objcore)	list;
 	VTAILQ_ENTRY(objcore)	lru_list;
-	double			last_lru;
+	float			last_lru;
+	unsigned		lru_flags;
+#define OC_LRU_OFFLRU		(1<<1)
 	VTAILQ_ENTRY(objcore)	ban_list;
 	struct ban		*ban;
 };
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 65cf46c..e5865da 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -141,7 +141,7 @@ exp_mail_it(struct objcore *oc)
 {
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
-	AN(oc->exp_flags & OC_EF_OFFLRU);
+	AN(oc->lru_flags & OC_LRU_OFFLRU);
 	Lck_Lock(&exphdl->mtx);
 	if (oc->exp_flags & OC_EF_DYING)
 		VTAILQ_INSERT_HEAD(&exphdl->inbox, oc, lru_list);
@@ -166,14 +166,16 @@ EXP_Inject(struct worker *wrk, struct objcore *oc, struct lru *lru)
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
-	AZ(oc->exp_flags & (OC_EF_OFFLRU | OC_EF_INSERT | OC_EF_MOVE));
+	AZ(oc->lru_flags & OC_LRU_OFFLRU);
+	AZ(oc->exp_flags & (OC_EF_INSERT | OC_EF_MOVE));
 	AZ(oc->exp_flags & OC_EF_DYING);
 	AZ(oc->flags & OC_F_BUSY);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 
 	Lck_Lock(&lru->mtx);
 	lru->n_objcore++;
-	oc->exp_flags |= OC_EF_OFFLRU | OC_EF_INSERT | OC_EF_EXP;
+	oc->lru_flags |= OC_LRU_OFFLRU;
+	oc->exp_flags |= OC_EF_INSERT | OC_EF_EXP;
 	oc->timer_when = EXP_When(&oc->exp);
 	Lck_Unlock(&lru->mtx);
 
@@ -198,7 +200,8 @@ EXP_Insert(struct worker *wrk, struct objcore *oc)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	HSH_Ref(oc);
 
-	AZ(oc->exp_flags & (OC_EF_OFFLRU | OC_EF_INSERT | OC_EF_MOVE));
+	AZ(oc->lru_flags & OC_LRU_OFFLRU);
+	AZ(oc->exp_flags & (OC_EF_INSERT | OC_EF_MOVE));
 	AZ(oc->exp_flags & OC_EF_DYING);
 	AN(oc->flags & OC_F_BUSY);
 
@@ -207,7 +210,8 @@ EXP_Insert(struct worker *wrk, struct objcore *oc)
 
 	Lck_Lock(&lru->mtx);
 	lru->n_objcore++;
-	oc->exp_flags |= OC_EF_OFFLRU | OC_EF_INSERT | OC_EF_EXP;
+	oc->lru_flags |= OC_LRU_OFFLRU;
+	oc->exp_flags |= OC_EF_INSERT | OC_EF_EXP;
 	oc->exp_flags |= OC_EF_MOVE;
 	Lck_Unlock(&lru->mtx);
 
@@ -257,10 +261,10 @@ EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep)
 	else
 		oc->exp_flags |= OC_EF_MOVE;
 
-	if (oc->exp_flags & OC_EF_OFFLRU) {
+	if (oc->lru_flags & OC_LRU_OFFLRU) {
 		oc = NULL;
 	} else {
-		oc->exp_flags |= OC_EF_OFFLRU;
+		oc->lru_flags |= OC_LRU_OFFLRU;
 		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
 	}
 	Lck_Unlock(&lru->mtx);
@@ -290,7 +294,7 @@ EXP_NukeOne(struct worker *wrk, struct lru *lru)
 		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Cand p=%p f=0x%x r=%d",
 		    oc, oc->flags, oc->refcnt);
 
-		AZ(oc->exp_flags & OC_EF_OFFLRU);
+		AZ(oc->lru_flags & OC_LRU_OFFLRU);
 
 		if (ObjKill(wrk, oc)) {
 			VSC_C_main->n_lru_nuked++; // XXX per lru ?
@@ -376,12 +380,12 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now)
 	/* Evacuate our action-flags, and put it back on the LRU list */
 	Lck_Lock(&lru->mtx);
 	flags = oc->exp_flags;
-	AN(flags & OC_EF_OFFLRU);
+	AN(oc->lru_flags & OC_LRU_OFFLRU);
 	oc->exp_flags &= ~(OC_EF_INSERT | OC_EF_MOVE);
 	oc->last_lru = now;
 	if (!(flags & OC_EF_DYING)) {
 		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
-		oc->exp_flags &= ~OC_EF_OFFLRU;
+		oc->lru_flags &= ~OC_LRU_OFFLRU;
 	}
 	Lck_Unlock(&lru->mtx);
 
@@ -454,10 +458,10 @@ exp_expire(struct exp_priv *ep, double now)
 	Lck_Lock(&lru->mtx);
 
 	oc->exp_flags |= OC_EF_DYING;
-	if (oc->exp_flags & OC_EF_OFFLRU)
+	if (oc->lru_flags & OC_LRU_OFFLRU)
 		oc = NULL;
 	else {
-		oc->exp_flags |= OC_EF_OFFLRU;
+		oc->lru_flags |= OC_LRU_OFFLRU;
 		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
 	}
 	Lck_Unlock(&lru->mtx);
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index fb4c897..93d7acf 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -389,7 +389,7 @@ ObjKill(const struct worker *wrk, struct objcore *oc)
 	if (oc->refcnt == 1 && !Lck_Trylock(&oc->objhead->mtx)) {
 		if (oc->refcnt == 1) {
 			oc->exp_flags |= OC_EF_DYING;
-			oc->exp_flags |= OC_EF_OFFLRU;	/* XXX */
+			oc->lru_flags |= OC_LRU_OFFLRU;	/* XXX */
 			oc->refcnt++;
 			retval = 1;
 		}
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index c782e25..181f3c3 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -635,7 +635,7 @@ sml_touch(struct worker *wrk, struct objcore *oc, double now)
 
 	AN(oc->exp_flags & OC_EF_EXP);
 
-	if (!(oc->exp_flags & OC_EF_OFFLRU)) {
+	if (!(oc->lru_flags & OC_LRU_OFFLRU)) {
 		/* Can only touch it while it's actually on the LRU list */
 		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
 		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);



From phk at FreeBSD.org  Wed Feb  3 10:32:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 11:32:11 +0100
Subject: [master] a47c396 Move lruflag setting out of Obj
Message-ID: <E1aQujL-0003QT-35@project.varnish-software.com>


commit a47c39605a339f28f2c14c37f99b9420ae686759
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 09:37:40 2016 +0000

    Move lruflag setting out of Obj

diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index e5865da..5b375d9 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -299,6 +299,7 @@ EXP_NukeOne(struct worker *wrk, struct lru *lru)
 		if (ObjKill(wrk, oc)) {
 			VSC_C_main->n_lru_nuked++; // XXX per lru ?
 			VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
+			oc->lru_flags |= OC_LRU_OFFLRU;
 			break;
 		}
 	}
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 93d7acf..9dda51b 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -389,7 +389,6 @@ ObjKill(const struct worker *wrk, struct objcore *oc)
 	if (oc->refcnt == 1 && !Lck_Trylock(&oc->objhead->mtx)) {
 		if (oc->refcnt == 1) {
 			oc->exp_flags |= OC_EF_DYING;
-			oc->lru_flags |= OC_LRU_OFFLRU;	/* XXX */
 			oc->refcnt++;
 			retval = 1;
 		}



From phk at FreeBSD.org  Wed Feb  3 10:32:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 11:32:11 +0100
Subject: [master] 7ddc595 Add a documentary assert
Message-ID: <E1aQujL-0003Qc-4J@project.varnish-software.com>


commit 7ddc5950d1c6bc77966b1b93654fea864256e40b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 09:54:58 2016 +0000

    Add a documentary assert

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 2838914..b0f53ad 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -509,6 +509,7 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req)
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	Lck_AssertHeld(&oc->objhead->mtx);
 
 	vsl = req->vsl;
 



From phk at FreeBSD.org  Wed Feb  3 10:32:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 11:32:11 +0100
Subject: [master] d225748 ObjSnipe() is a better name
Message-ID: <E1aQujL-0003Ql-5v@project.varnish-software.com>


commit d2257489ecc12f4665e0a88108a9b2653984e6e5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 09:55:08 2016 +0000

    ObjSnipe() is a better name

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index e738271..a1f8eba 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -857,7 +857,7 @@ void ObjSetState(const struct objcore *, enum boc_state_e next);
 void ObjWaitState(const struct objcore *, enum boc_state_e want);
 void ObjTrimStore(struct worker *, struct objcore *);
 void ObjTouch(struct worker *, struct objcore *, double now);
-int ObjKill(const struct worker *, struct objcore *);
+int ObjSnipe(const struct worker *, struct objcore *);
 unsigned ObjGetXID(struct worker *, struct objcore *);
 uint64_t ObjGetLen(struct worker *, struct objcore *);
 void ObjUpdateMeta(struct worker *, struct objcore *);
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 5b375d9..118743d 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -296,7 +296,7 @@ EXP_NukeOne(struct worker *wrk, struct lru *lru)
 
 		AZ(oc->lru_flags & OC_LRU_OFFLRU);
 
-		if (ObjKill(wrk, oc)) {
+		if (ObjSnipe(wrk, oc)) {
 			VSC_C_main->n_lru_nuked++; // XXX per lru ?
 			VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
 			oc->lru_flags |= OC_LRU_OFFLRU;
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 9dda51b..d8cf967 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -370,13 +370,13 @@ ObjTouch(struct worker *wrk, struct objcore *oc, double now)
 }
 
 /*====================================================================
- * ObjKill()
+ * ObjSnipe()
  *
  * If objcore is idle, gain a ref and mark it dead.
  */
 
 int
-ObjKill(const struct worker *wrk, struct objcore *oc)
+ObjSnipe(const struct worker *wrk, struct objcore *oc)
 {
 	int retval = 0;
 



From phk at FreeBSD.org  Wed Feb  3 10:32:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 11:32:11 +0100
Subject: [master] 4ded44b Make DYING a main Obj flag,
 add a function to set it (ObjKill) and end random code mucking about
 with it.
Message-ID: <E1aQujL-0003Qu-7Z@project.varnish-software.com>


commit 4ded44b6c9633d483de1609a97c99fa0b6db153a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 10:17:14 2016 +0000

    Make DYING a main Obj flag, add a function to set it (ObjKill)
    and end random code mucking about with it.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index a1f8eba..599844b 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -427,12 +427,12 @@ struct objcore {
 #define OC_F_ABANDON		(1<<4)
 #define OC_F_PRIVATE		(1<<5)
 #define OC_F_FAILED		(1<<6)
+#define OC_F_DYING		(1<<7)
 
 	uint8_t			exp_flags;
 #define OC_EF_MOVE		(1<<2)
 #define OC_EF_INSERT		(1<<3)
 #define OC_EF_EXP		(1<<4)
-#define OC_EF_DYING		(1<<5)
 
 	uint16_t		oa_present;
 
@@ -858,6 +858,7 @@ void ObjWaitState(const struct objcore *, enum boc_state_e want);
 void ObjTrimStore(struct worker *, struct objcore *);
 void ObjTouch(struct worker *, struct objcore *, double now);
 int ObjSnipe(const struct worker *, struct objcore *);
+void ObjKill(struct objcore *);
 unsigned ObjGetXID(struct worker *, struct objcore *);
 uint64_t ObjGetLen(struct worker *, struct objcore *);
 void ObjUpdateMeta(struct worker *, struct objcore *);
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 118743d..960fba2 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -143,7 +143,7 @@ exp_mail_it(struct objcore *oc)
 
 	AN(oc->lru_flags & OC_LRU_OFFLRU);
 	Lck_Lock(&exphdl->mtx);
-	if (oc->exp_flags & OC_EF_DYING)
+	if (oc->flags & OC_F_DYING)
 		VTAILQ_INSERT_HEAD(&exphdl->inbox, oc, lru_list);
 	else
 		VTAILQ_INSERT_TAIL(&exphdl->inbox, oc, lru_list);
@@ -168,7 +168,7 @@ EXP_Inject(struct worker *wrk, struct objcore *oc, struct lru *lru)
 
 	AZ(oc->lru_flags & OC_LRU_OFFLRU);
 	AZ(oc->exp_flags & (OC_EF_INSERT | OC_EF_MOVE));
-	AZ(oc->exp_flags & OC_EF_DYING);
+	AZ(oc->flags & OC_F_DYING);
 	AZ(oc->flags & OC_F_BUSY);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 
@@ -202,7 +202,7 @@ EXP_Insert(struct worker *wrk, struct objcore *oc)
 
 	AZ(oc->lru_flags & OC_LRU_OFFLRU);
 	AZ(oc->exp_flags & (OC_EF_INSERT | OC_EF_MOVE));
-	AZ(oc->exp_flags & OC_EF_DYING);
+	AZ(oc->flags & OC_F_DYING);
 	AN(oc->flags & OC_F_BUSY);
 
 	lru = ObjGetLRU(oc);
@@ -256,15 +256,11 @@ EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep)
 
 	Lck_Lock(&lru->mtx);
 
-	if (!isnan(now) && when <= now)
-		oc->exp_flags |= OC_EF_DYING;
-	else
-		oc->exp_flags |= OC_EF_MOVE;
-
 	if (oc->lru_flags & OC_LRU_OFFLRU) {
 		oc = NULL;
 	} else {
 		oc->lru_flags |= OC_LRU_OFFLRU;
+		oc->exp_flags |= OC_EF_MOVE;
 		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
 	}
 	Lck_Unlock(&lru->mtx);
@@ -384,13 +380,13 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now)
 	AN(oc->lru_flags & OC_LRU_OFFLRU);
 	oc->exp_flags &= ~(OC_EF_INSERT | OC_EF_MOVE);
 	oc->last_lru = now;
-	if (!(flags & OC_EF_DYING)) {
+	if (!(oc->flags & OC_F_DYING)) {
 		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
 		oc->lru_flags &= ~OC_LRU_OFFLRU;
 	}
 	Lck_Unlock(&lru->mtx);
 
-	if (flags & OC_EF_DYING) {
+	if (oc->flags & OC_F_DYING) {
 		VSLb(&ep->vsl, SLT_ExpKill, "EXP_Kill p=%p e=%.9f f=0x%x", oc,
 		    oc->timer_when, oc->flags);
 		if (!(flags & OC_EF_INSERT)) {
@@ -458,7 +454,8 @@ exp_expire(struct exp_priv *ep, double now)
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 	Lck_Lock(&lru->mtx);
 
-	oc->exp_flags |= OC_EF_DYING;
+	if (!(oc->flags & OC_F_DYING))
+		ObjKill(oc);
 	if (oc->lru_flags & OC_LRU_OFFLRU)
 		oc = NULL;
 	else {
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index fb1c5db..24c2084 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -393,7 +393,7 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 		assert(oc->objhead == oh);
 
-		if (oc->exp_flags & OC_EF_DYING)
+		if (oc->flags & OC_F_DYING)
 			continue;
 		if (oc->flags & OC_F_FAILED)
 			continue;
@@ -415,8 +415,10 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 		if (oc->exp.ttl <= 0.)
 			continue;
 
-		if (BAN_CheckObject(wrk, oc, req))
+		if (BAN_CheckObject(wrk, oc, req)) {
+			oc->flags |= OC_F_DYING;
 			continue;
+		}
 
 		if (ObjHasAttr(wrk, oc, OA_VARY)) {
 			vary = ObjGetAttr(wrk, oc, OA_VARY, NULL);
@@ -585,7 +587,7 @@ double keep)
 				 */
 				continue;
 			}
-			if (oc->exp_flags & OC_EF_DYING)
+			if (oc->flags & OC_F_DYING)
 				continue;
 			if (spc < sizeof *ocp) {
 				/* Iterate if aws is not big enough */
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index d8cf967..3cc195b 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -370,6 +370,26 @@ ObjTouch(struct worker *wrk, struct objcore *oc, double now)
 }
 
 /*====================================================================
+ * ObjKill()
+ *
+ * It's dead Jim, kick it...
+ */
+
+void
+ObjKill(struct objcore *oc)
+{
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
+
+	AZ(oc->flags & OC_F_DYING);	// XXX ?
+
+	Lck_Lock(&oc->objhead->mtx);
+	oc->flags |= OC_F_DYING;
+	Lck_Unlock(&oc->objhead->mtx);
+}
+
+/*====================================================================
  * ObjSnipe()
  *
  * If objcore is idle, gain a ref and mark it dead.
@@ -384,11 +404,11 @@ ObjSnipe(const struct worker *wrk, struct objcore *oc)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
 
-	AZ(oc->exp_flags & OC_EF_DYING);
+	AZ(oc->flags & OC_F_DYING);
 
 	if (oc->refcnt == 1 && !Lck_Trylock(&oc->objhead->mtx)) {
 		if (oc->refcnt == 1) {
-			oc->exp_flags |= OC_EF_DYING;
+			oc->flags |= OC_F_DYING;
 			oc->refcnt++;
 			retval = 1;
 		}



From phk at FreeBSD.org  Wed Feb  3 22:45:28 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 23:45:28 +0100
Subject: [master] 09ea374 Remove the OFFLRU flag, use last_lru=NAN instead
Message-ID: <E1aR6Ay-0000xK-Et@project.varnish-software.com>


commit 09ea374c01f876f799c1558d5f57a81352cf5bca
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 11:00:14 2016 +0000

    Remove the OFFLRU flag, use last_lru=NAN instead

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 599844b..fe9da3c 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -440,8 +440,6 @@ struct objcore {
 	VTAILQ_ENTRY(objcore)	list;
 	VTAILQ_ENTRY(objcore)	lru_list;
 	float			last_lru;
-	unsigned		lru_flags;
-#define OC_LRU_OFFLRU		(1<<1)
 	VTAILQ_ENTRY(objcore)	ban_list;
 	struct ban		*ban;
 };
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 960fba2..6bcb1bd 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -141,7 +141,7 @@ exp_mail_it(struct objcore *oc)
 {
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
-	AN(oc->lru_flags & OC_LRU_OFFLRU);
+	AN(isnan(oc->last_lru));
 	Lck_Lock(&exphdl->mtx);
 	if (oc->flags & OC_F_DYING)
 		VTAILQ_INSERT_HEAD(&exphdl->inbox, oc, lru_list);
@@ -166,7 +166,7 @@ EXP_Inject(struct worker *wrk, struct objcore *oc, struct lru *lru)
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
-	AZ(oc->lru_flags & OC_LRU_OFFLRU);
+	AZ(isnan(oc->last_lru));
 	AZ(oc->exp_flags & (OC_EF_INSERT | OC_EF_MOVE));
 	AZ(oc->flags & OC_F_DYING);
 	AZ(oc->flags & OC_F_BUSY);
@@ -174,7 +174,7 @@ EXP_Inject(struct worker *wrk, struct objcore *oc, struct lru *lru)
 
 	Lck_Lock(&lru->mtx);
 	lru->n_objcore++;
-	oc->lru_flags |= OC_LRU_OFFLRU;
+	oc->last_lru = NAN;
 	oc->exp_flags |= OC_EF_INSERT | OC_EF_EXP;
 	oc->timer_when = EXP_When(&oc->exp);
 	Lck_Unlock(&lru->mtx);
@@ -200,7 +200,7 @@ EXP_Insert(struct worker *wrk, struct objcore *oc)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	HSH_Ref(oc);
 
-	AZ(oc->lru_flags & OC_LRU_OFFLRU);
+	AZ(isnan(oc->last_lru));
 	AZ(oc->exp_flags & (OC_EF_INSERT | OC_EF_MOVE));
 	AZ(oc->flags & OC_F_DYING);
 	AN(oc->flags & OC_F_BUSY);
@@ -210,7 +210,7 @@ EXP_Insert(struct worker *wrk, struct objcore *oc)
 
 	Lck_Lock(&lru->mtx);
 	lru->n_objcore++;
-	oc->lru_flags |= OC_LRU_OFFLRU;
+	oc->last_lru = NAN;
 	oc->exp_flags |= OC_EF_INSERT | OC_EF_EXP;
 	oc->exp_flags |= OC_EF_MOVE;
 	Lck_Unlock(&lru->mtx);
@@ -256,10 +256,10 @@ EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep)
 
 	Lck_Lock(&lru->mtx);
 
-	if (oc->lru_flags & OC_LRU_OFFLRU) {
+	if (isnan(oc->last_lru)) {
 		oc = NULL;
 	} else {
-		oc->lru_flags |= OC_LRU_OFFLRU;
+		oc->last_lru = NAN;
 		oc->exp_flags |= OC_EF_MOVE;
 		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
 	}
@@ -290,12 +290,12 @@ EXP_NukeOne(struct worker *wrk, struct lru *lru)
 		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Cand p=%p f=0x%x r=%d",
 		    oc, oc->flags, oc->refcnt);
 
-		AZ(oc->lru_flags & OC_LRU_OFFLRU);
+		AZ(isnan(oc->last_lru));
 
 		if (ObjSnipe(wrk, oc)) {
 			VSC_C_main->n_lru_nuked++; // XXX per lru ?
 			VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-			oc->lru_flags |= OC_LRU_OFFLRU;
+			oc->last_lru = NAN;
 			break;
 		}
 	}
@@ -377,12 +377,12 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now)
 	/* Evacuate our action-flags, and put it back on the LRU list */
 	Lck_Lock(&lru->mtx);
 	flags = oc->exp_flags;
-	AN(oc->lru_flags & OC_LRU_OFFLRU);
+	AN(isnan(oc->last_lru));
 	oc->exp_flags &= ~(OC_EF_INSERT | OC_EF_MOVE);
 	oc->last_lru = now;
 	if (!(oc->flags & OC_F_DYING)) {
 		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
-		oc->lru_flags &= ~OC_LRU_OFFLRU;
+		oc->last_lru = now;
 	}
 	Lck_Unlock(&lru->mtx);
 
@@ -456,10 +456,10 @@ exp_expire(struct exp_priv *ep, double now)
 
 	if (!(oc->flags & OC_F_DYING))
 		ObjKill(oc);
-	if (oc->lru_flags & OC_LRU_OFFLRU)
+	if (isnan(oc->last_lru)) {
 		oc = NULL;
-	else {
-		oc->lru_flags |= OC_LRU_OFFLRU;
+	} else {
+		oc->last_lru = NAN;
 		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
 	}
 	Lck_Unlock(&lru->mtx);
diff --git a/bin/varnishd/flint.lnt b/bin/varnishd/flint.lnt
index cba5eb1..89bfae9 100644
--- a/bin/varnishd/flint.lnt
+++ b/bin/varnishd/flint.lnt
@@ -1,5 +1,6 @@
 -d__flexelint_v9__=1
 +fan
+// -w4
 
 -printf(3, VSL)
 -printf(2, http_PrintfHeader)
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 181f3c3..605cd58 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -635,13 +635,13 @@ sml_touch(struct worker *wrk, struct objcore *oc, double now)
 
 	AN(oc->exp_flags & OC_EF_EXP);
 
-	if (!(oc->lru_flags & OC_LRU_OFFLRU)) {
+	if (!isnan(oc->last_lru)) {
 		/* Can only touch it while it's actually on the LRU list */
 		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
 		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
 		VSC_C_main->n_lru_moved++;
+		oc->last_lru = now;
 	}
-	oc->last_lru = now;
 	Lck_Unlock(&lru->mtx);
 }
 



From phk at FreeBSD.org  Wed Feb  3 22:45:28 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 23:45:28 +0100
Subject: [master] 33fbcb8 Add a single-linked list to objcore for the EXP
 inbox, rather than overloading the LRUs list.
Message-ID: <E1aR6Ay-0000xR-Gt@project.varnish-software.com>


commit 33fbcb8e52096b30952557bddc2d83a865b6d491
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 11:12:39 2016 +0000

    Add a single-linked list to objcore for the EXP inbox, rather than
    overloading the LRUs list.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index fe9da3c..4a698ce 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -441,6 +441,7 @@ struct objcore {
 	VTAILQ_ENTRY(objcore)	lru_list;
 	float			last_lru;
 	VTAILQ_ENTRY(objcore)	ban_list;
+	VSTAILQ_ENTRY(objcore)	exp_list;
 	struct ban		*ban;
 };
 
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 6bcb1bd..cd3aab0 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -56,7 +56,7 @@ struct exp_priv {
 	struct worker			*wrk;
 	struct vsl_log			vsl;
 
-	VTAILQ_HEAD(,objcore)		inbox;
+	VSTAILQ_HEAD(,objcore)		inbox;
 	struct binheap			*heap;
 	pthread_cond_t			condvar;
 
@@ -144,9 +144,9 @@ exp_mail_it(struct objcore *oc)
 	AN(isnan(oc->last_lru));
 	Lck_Lock(&exphdl->mtx);
 	if (oc->flags & OC_F_DYING)
-		VTAILQ_INSERT_HEAD(&exphdl->inbox, oc, lru_list);
+		VSTAILQ_INSERT_HEAD(&exphdl->inbox, oc, exp_list);
 	else
-		VTAILQ_INSERT_TAIL(&exphdl->inbox, oc, lru_list);
+		VSTAILQ_INSERT_TAIL(&exphdl->inbox, oc, exp_list);
 	VSC_C_main->exp_mailed++;
 	AZ(pthread_cond_signal(&exphdl->condvar));
 	Lck_Unlock(&exphdl->mtx);
@@ -521,9 +521,9 @@ exp_thread(struct worker *wrk, void *priv)
 	while (1) {
 
 		Lck_Lock(&ep->mtx);
-		oc = VTAILQ_FIRST(&ep->inbox);
+		oc = VSTAILQ_FIRST(&ep->inbox);
 		if (oc != NULL) {
-			VTAILQ_REMOVE(&ep->inbox, oc, lru_list);
+			VSTAILQ_REMOVE(&ep->inbox, oc, objcore, exp_list);
 			VSC_C_main->exp_received++;
 			tnext = 0;
 		} else if (tnext > t) {
@@ -556,7 +556,7 @@ EXP_Init(void)
 
 	Lck_New(&ep->mtx, lck_exp);
 	AZ(pthread_cond_init(&ep->condvar, NULL));
-	VTAILQ_INIT(&ep->inbox);
+	VSTAILQ_INIT(&ep->inbox);
 	AZ(pthread_rwlock_init(&ep->cb_rwl, NULL));
 	VTAILQ_INIT(&ep->ecb_list);
 	exphdl = ep;



From phk at FreeBSD.org  Wed Feb  3 22:45:28 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 23:45:28 +0100
Subject: [master] 9c2f867 Protect oc->exp_flags with EXP->mtx
Message-ID: <E1aR6Ay-0000xa-Ij@project.varnish-software.com>


commit 9c2f8677401ef6e1de7552ff40105888bcce9a53
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 14:26:09 2016 +0000

    Protect oc->exp_flags with EXP->mtx

diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index cd3aab0..89b8294 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -137,12 +137,13 @@ EXP_When(const struct exp *e)
  */
 
 static void
-exp_mail_it(struct objcore *oc)
+exp_mail_it(struct objcore *oc, uint8_t cmds)
 {
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
 	AN(isnan(oc->last_lru));
 	Lck_Lock(&exphdl->mtx);
+	oc->exp_flags |= cmds;
 	if (oc->flags & OC_F_DYING)
 		VSTAILQ_INSERT_HEAD(&exphdl->inbox, oc, exp_list);
 	else
@@ -175,13 +176,12 @@ EXP_Inject(struct worker *wrk, struct objcore *oc, struct lru *lru)
 	Lck_Lock(&lru->mtx);
 	lru->n_objcore++;
 	oc->last_lru = NAN;
-	oc->exp_flags |= OC_EF_INSERT | OC_EF_EXP;
-	oc->timer_when = EXP_When(&oc->exp);
 	Lck_Unlock(&lru->mtx);
 
-	exp_event(wrk, oc, EXP_INJECT);
+	oc->timer_when = EXP_When(&oc->exp);
 
-	exp_mail_it(oc);
+	exp_event(wrk, oc, EXP_INJECT);
+	exp_mail_it(oc, OC_EF_INSERT | OC_EF_EXP);
 }
 
 /*--------------------------------------------------------------------
@@ -211,13 +211,10 @@ EXP_Insert(struct worker *wrk, struct objcore *oc)
 	Lck_Lock(&lru->mtx);
 	lru->n_objcore++;
 	oc->last_lru = NAN;
-	oc->exp_flags |= OC_EF_INSERT | OC_EF_EXP;
-	oc->exp_flags |= OC_EF_MOVE;
 	Lck_Unlock(&lru->mtx);
 
 	exp_event(wrk, oc, EXP_INSERT);
-
-	exp_mail_it(oc);
+	exp_mail_it(oc, OC_EF_INSERT | OC_EF_EXP | OC_EF_MOVE);
 }
 
 /*--------------------------------------------------------------------
@@ -255,18 +252,16 @@ EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep)
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 
 	Lck_Lock(&lru->mtx);
-
 	if (isnan(oc->last_lru)) {
 		oc = NULL;
 	} else {
 		oc->last_lru = NAN;
-		oc->exp_flags |= OC_EF_MOVE;
 		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
 	}
 	Lck_Unlock(&lru->mtx);
 
 	if (oc != NULL)
-		exp_mail_it(oc);
+		exp_mail_it(oc, OC_EF_MOVE);
 }
 
 /*--------------------------------------------------------------------
@@ -309,7 +304,7 @@ EXP_NukeOne(struct worker *wrk, struct lru *lru)
 	/* XXX: We could grab and return one storage segment to our caller */
 	ObjSlim(wrk, oc);
 
-	exp_mail_it(oc);
+	exp_mail_it(oc, 0);
 
 	VSLb(wrk->vsl, SLT_ExpKill, "LRU x=%u", ObjGetXID(wrk, oc));
 	(void)HSH_DerefObjCore(wrk, &oc);
@@ -375,10 +370,13 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now)
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 
 	/* Evacuate our action-flags, and put it back on the LRU list */
-	Lck_Lock(&lru->mtx);
+	Lck_Lock(&ep->mtx);
 	flags = oc->exp_flags;
-	AN(isnan(oc->last_lru));
 	oc->exp_flags &= ~(OC_EF_INSERT | OC_EF_MOVE);
+	Lck_Unlock(&ep->mtx);
+
+	Lck_Lock(&lru->mtx);
+	AN(isnan(oc->last_lru));
 	oc->last_lru = now;
 	if (!(oc->flags & OC_F_DYING)) {
 		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
@@ -433,6 +431,7 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now)
 static double
 exp_expire(struct exp_priv *ep, double now)
 {
+
 	struct lru *lru;
 	struct objcore *oc;
 
@@ -450,12 +449,13 @@ exp_expire(struct exp_priv *ep, double now)
 
 	VSC_C_main->n_expired++;
 
+	if (!(oc->flags & OC_F_DYING))
+		ObjKill(oc);
+
 	lru = ObjGetLRU(oc);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 	Lck_Lock(&lru->mtx);
 
-	if (!(oc->flags & OC_F_DYING))
-		ObjKill(oc);
 	if (isnan(oc->last_lru)) {
 		oc = NULL;
 	} else {



From phk at FreeBSD.org  Wed Feb  3 22:45:28 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 23:45:28 +0100
Subject: [master] fd83702 Add a "posted" flag to keep track of oc's already
 mailed to exp
Message-ID: <E1aR6Ay-0000xj-KJ@project.varnish-software.com>


commit fd83702a2d30003b8b0fd7b3c32640a8867403a0
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 14:59:30 2016 +0000

    Add a "posted" flag to keep track of oc's already mailed to exp

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 4a698ce..4d832a7 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -430,6 +430,7 @@ struct objcore {
 #define OC_F_DYING		(1<<7)
 
 	uint8_t			exp_flags;
+#define OC_EF_POSTED		(1<<1)
 #define OC_EF_MOVE		(1<<2)
 #define OC_EF_INSERT		(1<<3)
 #define OC_EF_EXP		(1<<4)
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 89b8294..fd0b67a 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -143,11 +143,13 @@ exp_mail_it(struct objcore *oc, uint8_t cmds)
 
 	AN(isnan(oc->last_lru));
 	Lck_Lock(&exphdl->mtx);
-	oc->exp_flags |= cmds;
-	if (oc->flags & OC_F_DYING)
-		VSTAILQ_INSERT_HEAD(&exphdl->inbox, oc, exp_list);
-	else
-		VSTAILQ_INSERT_TAIL(&exphdl->inbox, oc, exp_list);
+	if (!(oc->exp_flags & OC_EF_POSTED)) {
+		if (oc->flags & OC_F_DYING)
+			VSTAILQ_INSERT_HEAD(&exphdl->inbox, oc, exp_list);
+		else
+			VSTAILQ_INSERT_TAIL(&exphdl->inbox, oc, exp_list);
+	}
+	oc->exp_flags |= cmds | OC_EF_POSTED;
 	VSC_C_main->exp_mailed++;
 	AZ(pthread_cond_signal(&exphdl->condvar));
 	Lck_Unlock(&exphdl->mtx);
@@ -355,9 +357,8 @@ EXP_Deregister_Callback(uintptr_t *handle)
  */
 
 static void
-exp_inbox(struct exp_priv *ep, struct objcore *oc, double now)
+exp_inbox(struct exp_priv *ep, struct objcore *oc, double now, unsigned flags)
 {
-	unsigned flags;
 	struct lru *lru;
 
 	CHECK_OBJ_NOTNULL(ep, EXP_PRIV_MAGIC);
@@ -369,12 +370,6 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now)
 	lru = ObjGetLRU(oc);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 
-	/* Evacuate our action-flags, and put it back on the LRU list */
-	Lck_Lock(&ep->mtx);
-	flags = oc->exp_flags;
-	oc->exp_flags &= ~(OC_EF_INSERT | OC_EF_MOVE);
-	Lck_Unlock(&ep->mtx);
-
 	Lck_Lock(&lru->mtx);
 	AN(isnan(oc->last_lru));
 	oc->last_lru = now;
@@ -512,6 +507,7 @@ exp_thread(struct worker *wrk, void *priv)
 	struct objcore *oc;
 	double t = 0, tnext = 0;
 	struct exp_priv *ep;
+	unsigned flags = 0;
 
 	CAST_OBJ_NOTNULL(ep, priv, EXP_PRIV_MAGIC);
 	ep->wrk = wrk;
@@ -526,6 +522,9 @@ exp_thread(struct worker *wrk, void *priv)
 			VSTAILQ_REMOVE(&ep->inbox, oc, objcore, exp_list);
 			VSC_C_main->exp_received++;
 			tnext = 0;
+			flags = oc->exp_flags;
+			oc->exp_flags &=
+			    ~(OC_EF_INSERT | OC_EF_MOVE | OC_EF_POSTED);
 		} else if (tnext > t) {
 			VSL_Flush(&ep->vsl, 0);
 			Pool_Sumstat(wrk);
@@ -536,7 +535,7 @@ exp_thread(struct worker *wrk, void *priv)
 		t = VTIM_real();
 
 		if (oc != NULL)
-			exp_inbox(ep, oc, t);
+			exp_inbox(ep, oc, t, flags);
 		else
 			tnext = exp_expire(ep, t);
 	}



From phk at FreeBSD.org  Wed Feb  3 22:45:28 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 23:45:28 +0100
Subject: [master] 8697257 Send EXP_NukeOne() over to storage_simple where it
 belongs (more)
Message-ID: <E1aR6Ay-0000xs-Ls@project.varnish-software.com>


commit 8697257d18eaa4d5f3d0c1d066a9565c0242a7c5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 15:09:57 2016 +0000

    Send EXP_NukeOne() over to storage_simple where it belongs (more)

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 4d832a7..b6961ae 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -695,6 +695,7 @@ void EXP_Insert(struct worker *wrk, struct objcore *oc);
 void EXP_Inject(struct worker *wrk, struct objcore *oc, struct lru *lru);
 void EXP_Rearm(struct objcore *, double now, double ttl, double grace,
     double keep);
+void EXP_Poke(struct objcore *);
 int EXP_NukeOne(struct worker *wrk, struct lru *lru);
 
 enum exp_event_e {
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index fd0b67a..1df0fa0 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -156,6 +156,18 @@ exp_mail_it(struct objcore *oc, uint8_t cmds)
 }
 
 /*--------------------------------------------------------------------
+ * Call EXP's attention to a an oc
+ */
+
+void
+EXP_Poke(struct objcore *oc)
+{
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	exp_mail_it(oc, 0);
+}
+
+/*--------------------------------------------------------------------
  * Inject an object with a reference into the lru/binheap.
  *
  * This can either come from a stevedore (persistent) during startup
@@ -266,53 +278,6 @@ EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep)
 		exp_mail_it(oc, OC_EF_MOVE);
 }
 
-/*--------------------------------------------------------------------
- * Attempt to make space by nuking the oldest object on the LRU list
- * which isn't in use.
- * Returns: 1: did, 0: didn't, -1: can't
- */
-
-int
-EXP_NukeOne(struct worker *wrk, struct lru *lru)
-{
-	struct objcore *oc, *oc2;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-	/* Find the first currently unused object on the LRU.  */
-	Lck_Lock(&lru->mtx);
-	VTAILQ_FOREACH_SAFE(oc, &lru->lru_head, lru_list, oc2) {
-		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-
-		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Cand p=%p f=0x%x r=%d",
-		    oc, oc->flags, oc->refcnt);
-
-		AZ(isnan(oc->last_lru));
-
-		if (ObjSnipe(wrk, oc)) {
-			VSC_C_main->n_lru_nuked++; // XXX per lru ?
-			VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-			oc->last_lru = NAN;
-			break;
-		}
-	}
-	Lck_Unlock(&lru->mtx);
-
-	if (oc == NULL) {
-		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Fail");
-		return (-1);
-	}
-
-	/* XXX: We could grab and return one storage segment to our caller */
-	ObjSlim(wrk, oc);
-
-	exp_mail_it(oc, 0);
-
-	VSLb(wrk->vsl, SLT_ExpKill, "LRU x=%u", ObjGetXID(wrk, oc));
-	(void)HSH_DerefObjCore(wrk, &oc);
-	return (1);
-}
-
 /*--------------------------------------------------------------------*/
 
 uintptr_t
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 605cd58..20d20a4 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -39,6 +39,54 @@
 #include "storage/storage.h"
 #include "storage/storage_simple.h"
 
+/*--------------------------------------------------------------------
+ * Attempt to make space by nuking the oldest object on the LRU list
+ * which isn't in use.
+ * Returns: 1: did, 0: didn't, -1: can't
+ */
+
+int
+EXP_NukeOne(struct worker *wrk, struct lru *lru)
+{
+	struct objcore *oc, *oc2;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+	/* Find the first currently unused object on the LRU.  */
+	Lck_Lock(&lru->mtx);
+	VTAILQ_FOREACH_SAFE(oc, &lru->lru_head, lru_list, oc2) {
+		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+
+		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Cand p=%p f=0x%x r=%d",
+		    oc, oc->flags, oc->refcnt);
+
+		AZ(isnan(oc->last_lru));
+
+		if (ObjSnipe(wrk, oc)) {
+			VSC_C_main->n_lru_nuked++; // XXX per lru ?
+			VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
+			oc->last_lru = NAN;
+			break;
+		}
+	}
+	Lck_Unlock(&lru->mtx);
+
+	if (oc == NULL) {
+		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Fail");
+		return (-1);
+	}
+
+	/* XXX: We could grab and return one storage segment to our caller */
+	ObjSlim(wrk, oc);
+
+	EXP_Poke(oc);
+
+	VSLb(wrk->vsl, SLT_ExpKill, "LRU x=%u", ObjGetXID(wrk, oc));
+	(void)HSH_DerefObjCore(wrk, &oc);
+	return (1);
+}
+
+
 /*-------------------------------------------------------------------*/
 
 static struct storage *



From phk at FreeBSD.org  Wed Feb  3 22:45:28 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 23:45:28 +0100
Subject: [master] 60036a1 Don't take OC's of their LRU just to mail them to EXP
Message-ID: <E1aR6Ay-0000y2-Na@project.varnish-software.com>


commit 60036a1ea25cdc6774366fd66873b3fdbe0acd23
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 15:28:01 2016 +0000

    Don't take OC's of their LRU just to mail them to EXP

diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 1df0fa0..2a6ebc8 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -141,7 +141,6 @@ exp_mail_it(struct objcore *oc, uint8_t cmds)
 {
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
-	AN(isnan(oc->last_lru));
 	Lck_Lock(&exphdl->mtx);
 	if (!(oc->exp_flags & OC_EF_POSTED)) {
 		if (oc->flags & OC_F_DYING)
@@ -239,7 +238,6 @@ EXP_Insert(struct worker *wrk, struct objcore *oc)
 void
 EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep)
 {
-	struct lru *lru;
 	double when;
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
@@ -259,22 +257,7 @@ EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep)
 	VSL(SLT_ExpKill, 0, "EXP_Rearm p=%p E=%.9f e=%.9f f=0x%x", oc,
 	    oc->timer_when, when, oc->flags);
 
-	if (when > oc->exp.t_origin && when > oc->timer_when)
-		return;
-
-	lru = ObjGetLRU(oc);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-
-	Lck_Lock(&lru->mtx);
-	if (isnan(oc->last_lru)) {
-		oc = NULL;
-	} else {
-		oc->last_lru = NAN;
-		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-	}
-	Lck_Unlock(&lru->mtx);
-
-	if (oc != NULL)
+	if (when < oc->exp.t_origin || when < oc->timer_when)
 		exp_mail_it(oc, OC_EF_MOVE);
 }
 
@@ -336,11 +319,14 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now, unsigned flags)
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 
 	Lck_Lock(&lru->mtx);
-	AN(isnan(oc->last_lru));
-	oc->last_lru = now;
-	if (!(oc->flags & OC_F_DYING)) {
-		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
-		oc->last_lru = now;
+	if (isnan(oc->last_lru)) {
+		if (!(oc->flags & OC_F_DYING)) {
+			VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
+			oc->last_lru = now;
+		}
+	} else if (oc->flags & OC_F_DYING) {
+		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
+		oc->last_lru = NAN;
 	}
 	Lck_Unlock(&lru->mtx);
 



From phk at FreeBSD.org  Wed Feb  3 22:45:28 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 23:45:28 +0100
Subject: [master] 51a19b0 upd
Message-ID: <E1aR6Ay-0000yB-P1@project.varnish-software.com>


commit 51a19b07379985607e1308cd24f5a9c30fa315d1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 16:09:00 2016 +0000

    upd

diff --git a/doc/sphinx/phk/10goingon50.rst b/doc/sphinx/phk/10goingon50.rst
index ffe4349..2199227 100644
--- a/doc/sphinx/phk/10goingon50.rst
+++ b/doc/sphinx/phk/10goingon50.rst
@@ -213,7 +213,7 @@ you to appreciate that you live in a fantastic future with a lot
 of amazing changes - even if it will take a long time before
 progress goes too far.
 
-There does seem to be increasing tendency to want the kids of your
+There does seem to be increasing tendency to want the kids off your
 lawn, but I think I can control that.
 
 But if not I hereby give them permission to steal my apples and



From phk at FreeBSD.org  Wed Feb  3 22:45:28 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 23:45:28 +0100
Subject: [master] c5b9d70 Further steps in the separation of EXP and LRU
Message-ID: <E1aR6Ay-0000yK-Qn@project.varnish-software.com>


commit c5b9d70ba198954674817b2cff41118b8b3dfd1b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 16:14:04 2016 +0000

    Further steps in the separation of EXP and LRU

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index b6961ae..b95255e 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -361,7 +361,6 @@ struct lru {
 #define LRU_MAGIC		0x3fec7bb0
 	VTAILQ_HEAD(,objcore)	lru_head;
 	struct lock		mtx;
-	unsigned		n_objcore;
 };
 
 /* Stored object -----------------------------------------------------
@@ -692,7 +691,7 @@ void EXP_Clr(struct exp *e);
 double EXP_Ttl(const struct req *, const struct exp*);
 double EXP_When(const struct exp *exp);
 void EXP_Insert(struct worker *wrk, struct objcore *oc);
-void EXP_Inject(struct worker *wrk, struct objcore *oc, struct lru *lru);
+void EXP_Inject(struct worker *wrk, struct objcore *oc);
 void EXP_Rearm(struct objcore *, double now, double ttl, double grace,
     double keep);
 void EXP_Poke(struct objcore *);
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 2a6ebc8..7b5341d 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -174,22 +174,15 @@ EXP_Poke(struct objcore *oc)
  */
 
 void
-EXP_Inject(struct worker *wrk, struct objcore *oc, struct lru *lru)
+EXP_Inject(struct worker *wrk, struct objcore *oc)
 {
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
-	AZ(isnan(oc->last_lru));
 	AZ(oc->exp_flags & (OC_EF_INSERT | OC_EF_MOVE));
 	AZ(oc->flags & OC_F_DYING);
 	AZ(oc->flags & OC_F_BUSY);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-
-	Lck_Lock(&lru->mtx);
-	lru->n_objcore++;
-	oc->last_lru = NAN;
-	Lck_Unlock(&lru->mtx);
 
 	oc->timer_when = EXP_When(&oc->exp);
 
@@ -207,25 +200,15 @@ EXP_Inject(struct worker *wrk, struct objcore *oc, struct lru *lru)
 void
 EXP_Insert(struct worker *wrk, struct objcore *oc)
 {
-	struct lru *lru;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	HSH_Ref(oc);
 
-	AZ(isnan(oc->last_lru));
 	AZ(oc->exp_flags & (OC_EF_INSERT | OC_EF_MOVE));
 	AZ(oc->flags & OC_F_DYING);
 	AN(oc->flags & OC_F_BUSY);
 
-	lru = ObjGetLRU(oc);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-
-	Lck_Lock(&lru->mtx);
-	lru->n_objcore++;
-	oc->last_lru = NAN;
-	Lck_Unlock(&lru->mtx);
-
 	exp_event(wrk, oc, EXP_INSERT);
 	exp_mail_it(oc, OC_EF_INSERT | OC_EF_EXP | OC_EF_MOVE);
 }
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 24c2084..1d594d7 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -76,6 +76,7 @@ hsh_NewObjCore(struct worker *wrk, int boc)
 	AN(oc);
 	wrk->stats->n_objectcore++;
 	oc->flags |= OC_F_BUSY | OC_F_INCOMPLETE;
+	oc->last_lru = NAN;
 	if (boc) {
 		ALLOC_OBJ(oc->boc, BOC_MAGIC);
 		AN(oc->boc);
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index fe0d889..3b125d1 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -169,7 +169,8 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		HSH_Insert(wrk, so->hash, oc);
 		oc->exp = so->exp;
 		sg->nobj++;
-		EXP_Inject(wrk, oc, sg->lru);
+		oc->last_lru = NAN;
+		EXP_Inject(wrk, oc);
 	}
 	Pool_Sumstat(wrk);
 	sg->flags |= SMP_SEG_LOADED;



From phk at FreeBSD.org  Wed Feb  3 22:45:28 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 03 Feb 2016 23:45:28 +0100
Subject: [master] 3e590da LRU is now completely detached from EXP
Message-ID: <E1aR6Ay-0000yT-Se@project.varnish-software.com>


commit 3e590da45c341536f08edcc2d9e8fa39e82344f1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 22:45:02 2016 +0000

    LRU is now completely detached from EXP

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index b95255e..a52434d 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -695,7 +695,6 @@ void EXP_Inject(struct worker *wrk, struct objcore *oc);
 void EXP_Rearm(struct objcore *, double now, double ttl, double grace,
     double keep);
 void EXP_Poke(struct objcore *);
-int EXP_NukeOne(struct worker *wrk, struct lru *lru);
 
 enum exp_event_e {
 	EXP_INSERT,
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 7b5341d..d60900c 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -167,7 +167,7 @@ EXP_Poke(struct objcore *oc)
 }
 
 /*--------------------------------------------------------------------
- * Inject an object with a reference into the lru/binheap.
+ * Inject an object with a reference into the binheap.
  *
  * This can either come from a stevedore (persistent) during startup
  * or from EXP_Insert() below.
@@ -288,9 +288,8 @@ EXP_Deregister_Callback(uintptr_t *handle)
  */
 
 static void
-exp_inbox(struct exp_priv *ep, struct objcore *oc, double now, unsigned flags)
+exp_inbox(struct exp_priv *ep, struct objcore *oc, unsigned flags)
 {
-	struct lru *lru;
 
 	CHECK_OBJ_NOTNULL(ep, EXP_PRIV_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
@@ -298,21 +297,6 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now, unsigned flags)
 	VSLb(&ep->vsl, SLT_ExpKill, "EXP_Inbox p=%p e=%.9f f=0x%x", oc,
 	    oc->timer_when, oc->flags);
 
-	lru = ObjGetLRU(oc);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-
-	Lck_Lock(&lru->mtx);
-	if (isnan(oc->last_lru)) {
-		if (!(oc->flags & OC_F_DYING)) {
-			VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
-			oc->last_lru = now;
-		}
-	} else if (oc->flags & OC_F_DYING) {
-		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-		oc->last_lru = NAN;
-	}
-	Lck_Unlock(&lru->mtx);
-
 	if (oc->flags & OC_F_DYING) {
 		VSLb(&ep->vsl, SLT_ExpKill, "EXP_Kill p=%p e=%.9f f=0x%x", oc,
 		    oc->timer_when, oc->flags);
@@ -360,8 +344,6 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, double now, unsigned flags)
 static double
 exp_expire(struct exp_priv *ep, double now)
 {
-
-	struct lru *lru;
 	struct objcore *oc;
 
 	CHECK_OBJ_NOTNULL(ep, EXP_PRIV_MAGIC);
@@ -369,6 +351,8 @@ exp_expire(struct exp_priv *ep, double now)
 	oc = binheap_root(ep->heap);
 	if (oc == NULL)
 		return (now + 355./113.);
+	VSLb(&ep->vsl, SLT_ExpKill, "EXP_expire p=%p e=%.9f f=0x%x", oc,
+	    oc->timer_when, oc->flags);
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
@@ -381,21 +365,6 @@ exp_expire(struct exp_priv *ep, double now)
 	if (!(oc->flags & OC_F_DYING))
 		ObjKill(oc);
 
-	lru = ObjGetLRU(oc);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-	Lck_Lock(&lru->mtx);
-
-	if (isnan(oc->last_lru)) {
-		oc = NULL;
-	} else {
-		oc->last_lru = NAN;
-		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-	}
-	Lck_Unlock(&lru->mtx);
-
-	if (oc == NULL)
-		return (now + 1e-3);		// XXX ?
-
 	/* Remove from binheap */
 	assert(oc->timer_idx != BINHEAP_NOIDX);
 	binheap_delete(ep->heap, oc->timer_idx);
@@ -469,7 +438,7 @@ exp_thread(struct worker *wrk, void *priv)
 		t = VTIM_real();
 
 		if (oc != NULL)
-			exp_inbox(ep, oc, t, flags);
+			exp_inbox(ep, oc, flags);
 		else
 			tnext = exp_expire(ep, t);
 	}
diff --git a/bin/varnishd/cache/cache_obj.h b/bin/varnishd/cache/cache_obj.h
index 3fba1ef..a981fb2 100644
--- a/bin/varnishd/cache/cache_obj.h
+++ b/bin/varnishd/cache/cache_obj.h
@@ -70,3 +70,4 @@ struct obj_methods {
 	objsetattr_f	*objsetattr;
 	objtouch_f	*objtouch;
 };
+
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index b7dd9a1..ac4b485 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -121,6 +121,7 @@ uintmax_t STV_FileSize(int fd, const char *size, unsigned *granularity,
 
 struct lru *LRU_Alloc(void);
 void LRU_Free(struct lru *lru);
+int EXP_NukeOne(struct worker *wrk, struct lru *lru);
 
 /*--------------------------------------------------------------------*/
 extern const struct stevedore sma_stevedore;
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 20d20a4..35cf916 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -39,6 +39,8 @@
 #include "storage/storage.h"
 #include "storage/storage_simple.h"
 
+#include "vtim.h"
+
 /*--------------------------------------------------------------------
  * Attempt to make space by nuking the oldest object on the LRU list
  * which isn't in use.
@@ -247,6 +249,7 @@ static void __match_proto__(objfree_f)
 sml_objfree(struct worker *wrk, struct objcore *oc)
 {
 	struct object *o;
+	struct lru *lru;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
@@ -254,6 +257,15 @@ sml_objfree(struct worker *wrk, struct objcore *oc)
 	CAST_OBJ_NOTNULL(o, oc->stobj->priv, OBJECT_MAGIC);
 	o->magic = 0;
 
+	lru = ObjGetLRU(oc);
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+	Lck_Lock(&lru->mtx);
+	if (!isnan(oc->last_lru)) {
+		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
+		oc->last_lru = NAN;
+	}
+	Lck_Unlock(&lru->mtx);
+
 	sml_stv_free(oc->stobj->stevedore, o->objstore);
 
 	memset(oc->stobj, 0, sizeof oc->stobj);
@@ -536,19 +548,28 @@ sml_stable(struct worker *wrk, struct objcore *oc, struct boc *boc)
 {
 	const struct stevedore *stv;
 	struct storage *st;
+	struct lru *lru;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(boc, BOC_MAGIC);
 
-	if (boc->stevedore_priv == NULL)
-		return;
-	CAST_OBJ_NOTNULL(st, boc->stevedore_priv, STORAGE_MAGIC);
-	boc->stevedore_priv = 0;
-	CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
-	stv = oc->stobj->stevedore;
-	CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
-	sml_stv_free(stv, st);
+	if (boc->stevedore_priv != NULL) {
+		/* Free any leftovers from Trim */
+		CAST_OBJ_NOTNULL(st, boc->stevedore_priv, STORAGE_MAGIC);
+		boc->stevedore_priv = 0;
+		CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
+		stv = oc->stobj->stevedore;
+		CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
+		sml_stv_free(stv, st);
+	}
+
+	lru = ObjGetLRU(oc);
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+	Lck_Lock(&lru->mtx);
+	VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
+	oc->last_lru = VTIM_real();
+	Lck_Unlock(&lru->mtx);
 }
 
 static void * __match_proto__(objgetattr_f)



From phk at FreeBSD.org  Wed Feb  3 23:20:26 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 04 Feb 2016 00:20:26 +0100
Subject: [master] af2fb99 Collect all the LRU stuff in its own source file.
Message-ID: <E1aR6io-0006g1-UL@project.varnish-software.com>


commit af2fb997eeef79db5f8d09b8a3a73aefe7920dce
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 23:20:02 2016 +0000

    Collect all the LRU stuff in its own source file.

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 6bb046c..5c84034 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -86,6 +86,7 @@ varnishd_SOURCES = \
 	storage/mgt_stevedore.c \
 	storage/stevedore_utils.c \
 	storage/storage_file.c \
+	storage/storage_lru.c \
 	storage/storage_malloc.c \
 	storage/storage_persistent.c \
 	storage/mgt_storage_persistent.c \
diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index a52434d..dc08c24 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -354,15 +354,6 @@ struct worker {
 	uintptr_t		stack_end;
 };
 
-/* LRU ---------------------------------------------------------------*/
-
-struct lru {
-	unsigned		magic;
-#define LRU_MAGIC		0x3fec7bb0
-	VTAILQ_HEAD(,objcore)	lru_head;
-	struct lock		mtx;
-};
-
 /* Stored object -----------------------------------------------------
  * This is just to encapsulate the fields owned by the stevedore
  */
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index fa2d56c..5c20011 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -37,6 +37,7 @@
 #include <stdlib.h>
 
 #include "cache/cache.h"
+#include "hash/hash_slinger.h"
 
 #include "storage/storage.h"
 #include "vrt.h"
@@ -45,29 +46,6 @@
 static const struct stevedore * volatile stv_next;
 
 /*--------------------------------------------------------------------
- */
-
-struct lru *
-LRU_Alloc(void)
-{
-	struct lru *l;
-
-	ALLOC_OBJ(l, LRU_MAGIC);
-	AN(l);
-	VTAILQ_INIT(&l->lru_head);
-	Lck_New(&l->mtx, lck_lru);
-	return (l);
-}
-
-void
-LRU_Free(struct lru *lru)
-{
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-	Lck_Delete(&lru->mtx);
-	FREE_OBJ(lru);
-}
-
-/*--------------------------------------------------------------------
  * XXX: trust pointer writes to be atomic
  */
 
@@ -101,7 +79,6 @@ stv_pick_stevedore(struct vsl_log *vsl, const char **hint)
 	return (stv);
 }
 
-
 /*-------------------------------------------------------------------
  * Allocate storage for an object, based on the header information.
  * XXX: If we know (a hint of) the length, we could allocate space
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index ac4b485..d63b82b 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -120,8 +120,12 @@ uintmax_t STV_FileSize(int fd, const char *size, unsigned *granularity,
     const char *ctx);
 
 struct lru *LRU_Alloc(void);
-void LRU_Free(struct lru *lru);
-int EXP_NukeOne(struct worker *wrk, struct lru *lru);
+void LRU_Free(struct lru *);
+void LRU_Add(struct objcore *);
+void LRU_Remove(struct objcore *);
+int LRU_NukeOne(struct worker *, struct lru *);
+void LRU_Touch(struct worker *, struct objcore *, double now);
+
 
 /*--------------------------------------------------------------------*/
 extern const struct stevedore sma_stevedore;
diff --git a/bin/varnishd/storage/storage_lru.c b/bin/varnishd/storage/storage_lru.c
new file mode 100644
index 0000000..0acb91d
--- /dev/null
+++ b/bin/varnishd/storage/storage_lru.c
@@ -0,0 +1,184 @@
+/*-
+ * Copyright (c) 2007-2015 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Dag-Erling Sm?rgav <des at des.no>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * STEVEDORE: one who works at or is responsible for loading and
+ * unloading ships in port.  Example: "on the wharves, stevedores were
+ * unloading cargo from the far corners of the world." Origin: Spanish
+ * estibador, from estibar to pack.  First Known Use: 1788
+ */
+
+#include "config.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "cache/cache.h"
+#include "hash/hash_slinger.h"
+
+#include "storage/storage.h"
+#include "vtim.h"
+
+struct lru {
+	unsigned		magic;
+#define LRU_MAGIC		0x3fec7bb0
+	VTAILQ_HEAD(,objcore)	lru_head;
+	struct lock		mtx;
+};
+
+struct lru *
+LRU_Alloc(void)
+{
+	struct lru *l;
+
+	ALLOC_OBJ(l, LRU_MAGIC);
+	AN(l);
+	VTAILQ_INIT(&l->lru_head);
+	Lck_New(&l->mtx, lck_lru);
+	return (l);
+}
+
+void
+LRU_Free(struct lru *lru)
+{
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+	Lck_Delete(&lru->mtx);
+	FREE_OBJ(lru);
+}
+
+void
+LRU_Add(struct objcore *oc)
+{
+	struct lru *lru;
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	lru = ObjGetLRU(oc);
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+	Lck_Lock(&lru->mtx);
+	VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
+	oc->last_lru = VTIM_real();
+	Lck_Unlock(&lru->mtx);
+}
+
+void
+LRU_Remove(struct objcore *oc)
+{
+	struct lru *lru;
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	lru = ObjGetLRU(oc);
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+	Lck_Lock(&lru->mtx);
+	if (!isnan(oc->last_lru)) {
+		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
+		oc->last_lru = NAN;
+	}
+	Lck_Unlock(&lru->mtx);
+}
+
+void __match_proto__(objtouch_f)
+LRU_Touch(struct worker *wrk, struct objcore *oc, double now)
+{
+	struct lru *lru;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+
+	/*
+	 * To avoid the exphdl->mtx becoming a hotspot, we only
+	 * attempt to move objects if they have not been moved
+	 * recently and if the lock is available.  This optimization
+	 * obviously leaves the LRU list imperfectly sorted.
+	 */
+
+	if (oc->flags & OC_F_INCOMPLETE)
+		return;
+
+	if (now - oc->last_lru < cache_param->lru_interval)
+		return;
+
+	lru = ObjGetLRU(oc);
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+
+	if (Lck_Trylock(&lru->mtx))
+		return;
+
+	if (!isnan(oc->last_lru)) {
+		/* Can only touch it while it's actually on the LRU list */
+		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
+		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
+		VSC_C_main->n_lru_moved++;
+		oc->last_lru = now;
+	}
+	Lck_Unlock(&lru->mtx);
+}
+
+/*--------------------------------------------------------------------
+ * Attempt to make space by nuking the oldest object on the LRU list
+ * which isn't in use.
+ * Returns: 1: did, 0: didn't, -1: can't
+ */
+
+int
+LRU_NukeOne(struct worker *wrk, struct lru *lru)
+{
+	struct objcore *oc, *oc2;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+	/* Find the first currently unused object on the LRU.  */
+	Lck_Lock(&lru->mtx);
+	VTAILQ_FOREACH_SAFE(oc, &lru->lru_head, lru_list, oc2) {
+		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+
+		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Cand p=%p f=0x%x r=%d",
+		    oc, oc->flags, oc->refcnt);
+
+		AZ(isnan(oc->last_lru));
+
+		if (ObjSnipe(wrk, oc)) {
+			VSC_C_main->n_lru_nuked++; // XXX per lru ?
+			VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
+			oc->last_lru = NAN;
+			break;
+		}
+	}
+	Lck_Unlock(&lru->mtx);
+
+	if (oc == NULL) {
+		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Fail");
+		return (-1);
+	}
+
+	/* XXX: We could grab and return one storage segment to our caller */
+	ObjSlim(wrk, oc);
+
+	EXP_Poke(oc);
+
+	VSLb(wrk->vsl, SLT_ExpKill, "LRU x=%u", ObjGetXID(wrk, oc));
+	(void)HSH_DerefObjCore(wrk, &oc);
+	return (1);
+}
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index c068a44..735c7c5 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -235,7 +235,7 @@ smp_open_segs(struct smp_sc *sc, struct smp_signspace *spc)
 		ALLOC_OBJ(sg, SMP_SEG_MAGIC);
 		AN(sg);
 		sg->lru = LRU_Alloc();
-		CHECK_OBJ_NOTNULL(sg->lru, LRU_MAGIC);
+		AN(sg->lru);
 		sg->p = *ss;
 
 		sg->flags |= SMP_SEG_MUSTLOAD;
@@ -531,7 +531,7 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv,
 
 	while (1) {
 		if (really > 0) {
-			if (EXP_NukeOne(wrk, stv->lru) == -1)
+			if (LRU_NukeOne(wrk, stv->lru) == -1)
 				return (0);
 			really--;
 		}
@@ -608,10 +608,9 @@ const struct stevedore smp_stevedore = {
  */
 
 static void
-debug_report_silo(struct cli *cli, const struct smp_sc *sc, int objs)
+debug_report_silo(struct cli *cli, const struct smp_sc *sc)
 {
 	struct smp_seg *sg;
-	struct objcore *oc;
 
 	VCLI_Out(cli, "Silo: %s (%s)\n",
 	    sc->stevedore->ident, sc->filename);
@@ -626,10 +625,6 @@ debug_report_silo(struct cli *cli, const struct smp_sc *sc, int objs)
 			   (uintmax_t)(sc->next_top - sc->next_bot));
 		VCLI_Out(cli, "    %u nobj, %u alloc, %u lobjlist, %u fixed\n",
 		    sg->nobj, sg->nalloc, sg->p.lobjlist, sg->nfixed);
-		if (objs) {
-			VTAILQ_FOREACH(oc, &sg->lru->lru_head, lru_list)
-				VCLI_Out(cli, "      OC %p\n", oc);
-		}
 	}
 }
 
@@ -642,7 +637,7 @@ debug_persistent(struct cli *cli, const char * const * av, void *priv)
 
 	if (av[2] == NULL) {
 		VTAILQ_FOREACH(sc, &silos, list)
-			debug_report_silo(cli, sc, 0);
+			debug_report_silo(cli, sc);
 		return;
 	}
 	VTAILQ_FOREACH(sc, &silos, list)
@@ -654,7 +649,7 @@ debug_persistent(struct cli *cli, const char * const * av, void *priv)
 		return;
 	}
 	if (av[3] == NULL) {
-		debug_report_silo(cli, sc, 0);
+		debug_report_silo(cli, sc);
 		return;
 	}
 	Lck_Lock(&sc->mtx);
@@ -663,7 +658,7 @@ debug_persistent(struct cli *cli, const char * const * av, void *priv)
 			smp_close_seg(sc, sc->cur_seg);
 		smp_new_seg(sc);
 	} else if (!strcmp(av[3], "dump")) {
-		debug_report_silo(cli, sc, 1);
+		debug_report_silo(cli, sc);
 	} else {
 		VCLI_Out(cli, "Unknown operation\n");
 		VCLI_SetResult(cli, CLIS_PARAM);
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 3b125d1..a0723bd 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -139,7 +139,6 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 	ASSERT_SILO_THREAD(sc);
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(sg, SMP_SEG_MAGIC);
-	CHECK_OBJ_NOTNULL(sg->lru, LRU_MAGIC);
 	assert(sg->flags & SMP_SEG_MUSTLOAD);
 	sg->flags &= ~SMP_SEG_MUSTLOAD;
 	AN(sg->p.offset);
@@ -225,7 +224,7 @@ smp_new_seg(struct smp_sc *sc)
 		return;
 	*sg = tmpsg;
 	sg->lru = LRU_Alloc();
-	CHECK_OBJ_NOTNULL(sg->lru, LRU_MAGIC);
+	AN(sg->lru);
 
 	sg->p.offset = IRNUP(sc, sg->p.offset);
 	sg->p.length -= sg->p.offset - tmpsg.p.offset;
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 35cf916..95f4c05 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -39,56 +39,6 @@
 #include "storage/storage.h"
 #include "storage/storage_simple.h"
 
-#include "vtim.h"
-
-/*--------------------------------------------------------------------
- * Attempt to make space by nuking the oldest object on the LRU list
- * which isn't in use.
- * Returns: 1: did, 0: didn't, -1: can't
- */
-
-int
-EXP_NukeOne(struct worker *wrk, struct lru *lru)
-{
-	struct objcore *oc, *oc2;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-	/* Find the first currently unused object on the LRU.  */
-	Lck_Lock(&lru->mtx);
-	VTAILQ_FOREACH_SAFE(oc, &lru->lru_head, lru_list, oc2) {
-		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-
-		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Cand p=%p f=0x%x r=%d",
-		    oc, oc->flags, oc->refcnt);
-
-		AZ(isnan(oc->last_lru));
-
-		if (ObjSnipe(wrk, oc)) {
-			VSC_C_main->n_lru_nuked++; // XXX per lru ?
-			VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-			oc->last_lru = NAN;
-			break;
-		}
-	}
-	Lck_Unlock(&lru->mtx);
-
-	if (oc == NULL) {
-		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Fail");
-		return (-1);
-	}
-
-	/* XXX: We could grab and return one storage segment to our caller */
-	ObjSlim(wrk, oc);
-
-	EXP_Poke(oc);
-
-	VSLb(wrk->vsl, SLT_ExpKill, "LRU x=%u", ObjGetXID(wrk, oc));
-	(void)HSH_DerefObjCore(wrk, &oc);
-	return (1);
-}
-
-
 /*-------------------------------------------------------------------*/
 
 static struct storage *
@@ -178,7 +128,7 @@ SML_allocobj(struct worker *wrk, const struct stevedore *stv,
 	ltot = sizeof(struct object) + PRNDUP(wsl);
 	while (1) {
 		if (really > 0) {
-			if (EXP_NukeOne(wrk, stv->lru) == -1)
+			if (LRU_NukeOne(wrk, stv->lru) == -1)
 				return (0);
 			really--;
 		}
@@ -249,7 +199,6 @@ static void __match_proto__(objfree_f)
 sml_objfree(struct worker *wrk, struct objcore *oc)
 {
 	struct object *o;
-	struct lru *lru;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
@@ -257,14 +206,7 @@ sml_objfree(struct worker *wrk, struct objcore *oc)
 	CAST_OBJ_NOTNULL(o, oc->stobj->priv, OBJECT_MAGIC);
 	o->magic = 0;
 
-	lru = ObjGetLRU(oc);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-	Lck_Lock(&lru->mtx);
-	if (!isnan(oc->last_lru)) {
-		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-		oc->last_lru = NAN;
-	}
-	Lck_Unlock(&lru->mtx);
+	LRU_Remove(oc);
 
 	sml_stv_free(oc->stobj->stevedore, o->objstore);
 
@@ -397,7 +339,7 @@ objallocwithnuke(struct worker *wrk, const struct stevedore *stv, size_t size)
 
 		/* no luck; try to free some space and keep trying */
 		if (fail < cache_param->nuke_limit &&
-		    EXP_NukeOne(wrk, stv->lru) == -1)
+		    LRU_NukeOne(wrk, stv->lru) == -1)
 			break;
 	}
 	CHECK_OBJ_ORNULL(st, STORAGE_MAGIC);
@@ -548,7 +490,6 @@ sml_stable(struct worker *wrk, struct objcore *oc, struct boc *boc)
 {
 	const struct stevedore *stv;
 	struct storage *st;
-	struct lru *lru;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
@@ -564,12 +505,7 @@ sml_stable(struct worker *wrk, struct objcore *oc, struct boc *boc)
 		sml_stv_free(stv, st);
 	}
 
-	lru = ObjGetLRU(oc);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-	Lck_Lock(&lru->mtx);
-	VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
-	oc->last_lru = VTIM_real();
-	Lck_Unlock(&lru->mtx);
+	LRU_Add(oc);
 }
 
 static void * __match_proto__(objgetattr_f)
@@ -675,45 +611,6 @@ sml_setattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
 	return (retval);
 }
 
-static void __match_proto__(objtouch_f)
-sml_touch(struct worker *wrk, struct objcore *oc, double now)
-{
-	struct lru *lru;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-
-	/*
-	 * To avoid the exphdl->mtx becoming a hotspot, we only
-	 * attempt to move objects if they have not been moved
-	 * recently and if the lock is available.  This optimization
-	 * obviously leaves the LRU list imperfectly sorted.
-	 */
-
-	if (oc->flags & OC_F_INCOMPLETE)
-		return;
-
-	if (now - oc->last_lru < cache_param->lru_interval)
-		return;
-
-	lru = ObjGetLRU(oc);
-	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
-
-	if (Lck_Trylock(&lru->mtx))
-		return;
-
-	AN(oc->exp_flags & OC_EF_EXP);
-
-	if (!isnan(oc->last_lru)) {
-		/* Can only touch it while it's actually on the LRU list */
-		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
-		VSC_C_main->n_lru_moved++;
-		oc->last_lru = now;
-	}
-	Lck_Unlock(&lru->mtx);
-}
-
 const struct obj_methods SML_methods = {
 	.objfree	= sml_objfree,
 	.objgetlru	= sml_objgetlru,
@@ -726,5 +623,5 @@ const struct obj_methods SML_methods = {
 	.objslim	= sml_slim,
 	.objgetattr	= sml_getattr,
 	.objsetattr	= sml_setattr,
-	.objtouch	= sml_touch,
+	.objtouch	= LRU_Touch,
 };



From phk at FreeBSD.org  Thu Feb  4 01:55:26 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 04 Feb 2016 02:55:26 +0100
Subject: [master] 6df4d96 Split mcf_server_startstop() into two separate
 functions
Message-ID: <E1aR98o-0005Ps-Cq@project.varnish-software.com>


commit 6df4d965a9ded8cb50d184fe0435f0c066841b79
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 4 00:20:41 2016 +0000

    Split mcf_server_startstop() into two separate functions

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 8cedeaf..8baaa6d 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -611,17 +611,16 @@ mgt_stop_child(void)
 }
 
 /*=====================================================================
- * CLI command to start/stop child
+ * CLI commands to start/stop child
  */
 
 void __match_proto__(cli_func_t)
-mcf_server_startstop(struct cli *cli, const char * const *av, void *priv)
+mcf_server_start(struct cli *cli, const char * const *av, void *priv)
 {
 
 	(void)av;
-	if (priv != NULL && child_state == CH_RUNNING)
-		mgt_stop_child();
-	else if (priv == NULL && child_state == CH_STOPPED) {
+	(void)priv;
+	if (child_state == CH_STOPPED) {
 		if (mgt_has_vcl()) {
 			mgt_launch_child(cli);
 		} else {
@@ -634,6 +633,20 @@ mcf_server_startstop(struct cli *cli, const char * const *av, void *priv)
 	}
 }
 
+void __match_proto__(cli_func_t)
+mcf_server_stop(struct cli *cli, const char * const *av, void *priv)
+{
+
+	(void)av;
+	(void)priv;
+	if (child_state == CH_RUNNING) {
+		mgt_stop_child();
+	} else {
+		VCLI_SetResult(cli, CLIS_CANT);
+		VCLI_Out(cli, "Child in state %s", ch_state[child_state]);
+	}
+}
+
 /*--------------------------------------------------------------------*/
 
 void
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index fa021d6..2317822 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -89,20 +89,20 @@ mcf_banner(struct cli *cli, const char *const *av, void *priv)
 
 /* XXX: what order should this list be in ? */
 static struct cli_proto cli_proto[] = {
-	{ CLI_BANNER,		"", mcf_banner, NULL },
-	{ CLI_SERVER_STATUS,	"", mcf_server_status, NULL },
-	{ CLI_SERVER_START,	"", mcf_server_startstop, NULL },
-	{ CLI_SERVER_STOP,	"", mcf_server_startstop, cli_proto },
-	{ CLI_VCL_LOAD,		"", mcf_vcl_load, NULL },
-	{ CLI_VCL_INLINE,	"", mcf_vcl_inline, NULL },
-	{ CLI_VCL_USE,		"", mcf_vcl_use, NULL },
-	{ CLI_VCL_STATE,	"", mcf_vcl_state, NULL },
-	{ CLI_VCL_DISCARD,	"", mcf_vcl_discard, NULL },
-	{ CLI_VCL_LIST,		"", mcf_vcl_list, NULL },
-	{ CLI_PARAM_SHOW,	"", mcf_param_show, NULL },
-	{ CLI_PARAM_SET,	"", mcf_param_set, NULL },
-	{ CLI_PANIC_SHOW,	"", mcf_panic_show, NULL },
-	{ CLI_PANIC_CLEAR,	"", mcf_panic_clear, NULL },
+	{ CLI_BANNER,		"", mcf_banner },
+	{ CLI_SERVER_STATUS,	"", mcf_server_status },
+	{ CLI_SERVER_START,	"", mcf_server_start },
+	{ CLI_SERVER_STOP,	"", mcf_server_stop },
+	{ CLI_VCL_LOAD,		"", mcf_vcl_load },
+	{ CLI_VCL_INLINE,	"", mcf_vcl_inline },
+	{ CLI_VCL_USE,		"", mcf_vcl_use },
+	{ CLI_VCL_STATE,	"", mcf_vcl_state },
+	{ CLI_VCL_DISCARD,	"", mcf_vcl_discard },
+	{ CLI_VCL_LIST,		"", mcf_vcl_list },
+	{ CLI_PARAM_SHOW,	"", mcf_param_show },
+	{ CLI_PARAM_SET,	"", mcf_param_set },
+	{ CLI_PANIC_SHOW,	"", mcf_panic_show },
+	{ CLI_PANIC_CLEAR,	"", mcf_panic_clear },
 	{ NULL }
 };
 
diff --git a/bin/varnishd/mgt/mgt_cli.h b/bin/varnishd/mgt/mgt_cli.h
index 0210aa1..49d849a 100644
--- a/bin/varnishd/mgt/mgt_cli.h
+++ b/bin/varnishd/mgt/mgt_cli.h
@@ -29,7 +29,8 @@
  */
 
 /* mgt_child.c */
-cli_func_t mcf_server_startstop;
+cli_func_t mcf_server_start;
+cli_func_t mcf_server_stop;
 cli_func_t mcf_server_status;
 cli_func_t mcf_panic_show;
 cli_func_t mcf_panic_clear;



From phk at FreeBSD.org  Thu Feb  4 01:55:26 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 04 Feb 2016 02:55:26 +0100
Subject: [master] b616a4d Remove initialization of unused priv arguments
Message-ID: <E1aR98o-0005Py-Ei@project.varnish-software.com>


commit b616a4dd34aa6860f535d1a71b10e59143e92829
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 4 00:23:53 2016 +0000

    Remove initialization of unused priv arguments

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index a17351d..de768f2 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -561,7 +561,7 @@ static struct cli_proto vca_cmds[] = {
 	{ CLI_SERVER_START,	"i", ccf_start },
 	{ "debug.listen_address", "debug.listen_address",
 	    "\tReport the actual listen address.", 0, 0,
-	    "d", ccf_listen_address, NULL },
+	    "d", ccf_listen_address },
 	{ NULL }
 };
 
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index 2317822..baae752 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -121,7 +121,7 @@ mcf_panic(struct cli *cli, const char * const *av, void *priv)
 static struct cli_proto cli_debug[] = {
 	{ "debug.panic.master", "debug.panic.master",
 		"\tPanic the master process.",
-		0, 0, "d", mcf_panic, NULL},
+		0, 0, "d", mcf_panic},
 	{ NULL }
 };
 
@@ -177,7 +177,7 @@ mcf_askchild(struct cli *cli, const char * const *av, void *priv)
 
 static struct cli_proto cli_askchild[] = {
 	{ "*", "<wild-card-entry>", "\t<fall through to cacher>\n",
-		0, 9999, "h*", mcf_askchild, NULL},
+		0, 9999, "h*", mcf_askchild},
 	{ NULL }
 };
 
@@ -308,10 +308,10 @@ mcf_auth(struct cli *cli, const char *const *av, void *priv)
 }
 
 static struct cli_proto cli_auth[] = {
-	{ CLI_HELP,		"", VCLS_func_help, NULL },
+	{ CLI_HELP,		"", VCLS_func_help },
 	{ CLI_PING,		"", VCLS_func_ping },
-	{ CLI_AUTH,		"", mcf_auth, NULL },
-	{ CLI_QUIT,		"", VCLS_func_close, NULL},
+	{ CLI_AUTH,		"", mcf_auth },
+	{ CLI_QUIT,		"", VCLS_func_close },
 	{ NULL }
 };
 



From phk at FreeBSD.org  Thu Feb  4 01:55:26 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 04 Feb 2016 02:55:26 +0100
Subject: [master] e0b9077 Add support for JSON CLI responses.
Message-ID: <E1aR98o-0005Q7-GS@project.varnish-software.com>


commit e0b9077c87b683a12af45daaf8c08ab4e7ce4195
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 4 01:51:05 2016 +0000

    Add support for JSON CLI responses.
    
    To get JSON back, the first argument must be "-j".
    
    Currently only "help -j" is implemented, but that
    will helpfully tell you which commands support
    json output.
    
    All JSON output has the form:
    
    	[ version#, [<cli_command>],
    		stuff
    	]
    
    For instance:
    
    	[ 1, ["help", "-j"],
    	  {
    	  "request": "help",
    	  "syntax": "help [<command>]",
    	  "help": "\tShow command/protocol help.",
    	  "minarg": 0, "maxarg": 1, "flags": "", "json": true
    	  },
    	  {
    	  "request": "ping",
    	  "syntax": "ping [<timestamp>]",
    	  "help": "\tKeep connection alive.",
    	  "minarg": 0, "maxarg": 1, "flags": "", "json": false
    	  },
    	  ...
    	]
    
    The string quoting for weird characters in JSON is XXX incomplete.
    
    Prodded for by:	Kristian

diff --git a/bin/varnishd/cache/cache_cli.c b/bin/varnishd/cache/cache_cli.c
index 3ba8cdd..bc2302e 100644
--- a/bin/varnishd/cache/cache_cli.c
+++ b/bin/varnishd/cache/cache_cli.c
@@ -112,7 +112,7 @@ CLI_Run(void)
 
 static struct cli_proto cli_cmds[] = {
 	{ CLI_PING,		"i", VCLS_func_ping },
-	{ CLI_HELP,             "i", VCLS_func_help },
+	{ CLI_HELP,             "i", VCLS_func_help, VCLS_func_help_json, cli_cmds },
 	{ NULL }
 };
 
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index baae752..6df8f9e 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -142,7 +142,9 @@ mcf_askchild(struct cli *cli, const char * const *av, void *priv)
 	 */
 	if (cli_o <= 0) {
 		if (!strcmp(av[1], "help")) {
-			VCLI_Out(cli, "No help from child, (not running).\n");
+			if (av[2] == NULL || strcmp(av[2], "-j"))
+				VCLI_Out(cli,
+				   "No help from child, (not running).\n");
 			return;
 		}
 		VCLI_SetResult(cli, CLIS_UNKNOWN);
@@ -308,7 +310,7 @@ mcf_auth(struct cli *cli, const char *const *av, void *priv)
 }
 
 static struct cli_proto cli_auth[] = {
-	{ CLI_HELP,		"", VCLS_func_help },
+	{ CLI_HELP,		"", VCLS_func_help, VCLS_func_help_json },
 	{ CLI_PING,		"", VCLS_func_ping },
 	{ CLI_AUTH,		"", mcf_auth },
 	{ CLI_QUIT,		"", VCLS_func_close },
diff --git a/bin/varnishtest/tests/b00008.vtc b/bin/varnishtest/tests/b00008.vtc
index 2448aa4..ac18701 100644
--- a/bin/varnishtest/tests/b00008.vtc
+++ b/bin/varnishtest/tests/b00008.vtc
@@ -24,6 +24,8 @@ varnish v1 -start
 
 varnish v1 -cliok "help"
 
+varnish v1 -cliok "help -j"
+
 varnish v1 -clierr 106 "param.set waiter HASH(0x8839c4c)"
 
 varnish v1 -cliok "param.set cli_limit 128"
diff --git a/include/vcli_priv.h b/include/vcli_priv.h
index 2f98d7d..61f5720 100644
--- a/include/vcli_priv.h
+++ b/include/vcli_priv.h
@@ -48,12 +48,14 @@ struct cli_proto {
 
 	/* Dispatch information */
 	cli_func_t		*func;
+	cli_func_t		*jsonfunc;
 	void			*priv;
 };
 
 /* The implementation must provide these functions */
 int VCLI_Overflow(struct cli *cli);
-void VCLI_Out(struct cli *cli, const char *fmt, ...)
-    __v_printflike(2, 3);
+void VCLI_Out(struct cli *cli, const char *fmt, ...) __v_printflike(2, 3);
 void VCLI_Quote(struct cli *cli, const char *str);
+void VCLI_JSON_str(struct cli *cli, const char *str);
+void VCLI_JSON_ver(struct cli *cli, unsigned ver, const char * const * av);
 void VCLI_SetResult(struct cli *cli, unsigned r);
diff --git a/include/vcli_serve.h b/include/vcli_serve.h
index fe240c1..0ff65d0 100644
--- a/include/vcli_serve.h
+++ b/include/vcli_serve.h
@@ -42,4 +42,5 @@ void VCLS_Destroy(struct VCLS **);
 /* From libvarnish/cli.c */
 cli_func_t	VCLS_func_close;
 cli_func_t	VCLS_func_help;
+cli_func_t	VCLS_func_help_json;
 cli_func_t	VCLS_func_ping;
diff --git a/include/vsb.h b/include/vsb.h
index 4b34314..a3bdd3d 100644
--- a/include/vsb.h
+++ b/include/vsb.h
@@ -76,6 +76,7 @@ char		*VSB_data(const struct vsb *);
 ssize_t		 VSB_len(const struct vsb *);
 void		 VSB_delete(struct vsb *);
 #define VSB_QUOTE_NONL	1
+#define VSB_QUOTE_JSON	2
 void		 VSB_quote(struct vsb *s, const char *p, int len, int how);
 void		 VSB_indent(struct vsb *, int);
 #ifdef __cplusplus
diff --git a/lib/libvarnish/cli_common.c b/lib/libvarnish/cli_common.c
index 44a6869..b735d1f 100644
--- a/lib/libvarnish/cli_common.c
+++ b/lib/libvarnish/cli_common.c
@@ -84,6 +84,31 @@ VCLI_Overflow(struct cli *cli)
 
 /*lint -e{818} cli could be const */
 void
+VCLI_JSON_str(struct cli *cli, const char *s)
+{
+
+	CHECK_OBJ_NOTNULL(cli, CLI_MAGIC);
+	VSB_quote(cli->sb, s, -1, VSB_QUOTE_JSON);
+}
+
+/*lint -e{818} cli could be const */
+void
+VCLI_JSON_ver(struct cli *cli, unsigned ver, const char * const * av)
+{
+	int i;
+
+	CHECK_OBJ_NOTNULL(cli, CLI_MAGIC);
+	VCLI_Out(cli, "[ %u, [", ver);
+	for (i = 1; av[i] != NULL; i++) {
+		VCLI_JSON_str(cli, av[i]);
+		if (av[i + 1] != NULL)
+			VCLI_Out(cli, ", ");
+	}
+	VCLI_Out(cli, "]");
+}
+
+/*lint -e{818} cli could be const */
+void
 VCLI_Quote(struct cli *cli, const char *s)
 {
 
diff --git a/lib/libvarnish/cli_serve.c b/lib/libvarnish/cli_serve.c
index 64fb7dd..9861e0c 100644
--- a/lib/libvarnish/cli_serve.c
+++ b/lib/libvarnish/cli_serve.c
@@ -146,7 +146,7 @@ VCLS_func_help(struct cli *cli, const char * const *av, void *priv)
 				}
 				for (u = 0; u < sizeof cp->flags; u++) {
 					if (cp->flags[u] == '*') {
-						cp->func(cli,av,priv);
+						cp->func(cli, av, NULL);
 						return;
 					}
 				}
@@ -179,7 +179,7 @@ VCLS_func_help(struct cli *cli, const char * const *av, void *priv)
 			if (i)
 				continue;
 			if (wc) {
-				cp->func(cli, av, priv);
+				cp->func(cli, av, NULL);
 				continue;
 			}
 			if (debug != d)
@@ -192,6 +192,57 @@ VCLS_func_help(struct cli *cli, const char * const *av, void *priv)
 	}
 }
 
+void
+VCLS_func_help_json(struct cli *cli, const char * const *av, void *priv)
+{
+	struct cli_proto *cp;
+	struct VCLS_func *cfn;
+	struct VCLS *cs;
+	unsigned u, f_wc, f_i;
+
+	(void)priv;
+	cs = cli->cls;
+	CHECK_OBJ_NOTNULL(cs, VCLS_MAGIC);
+
+	if (priv == NULL)
+		VCLI_JSON_ver(cli, 1, av);
+	VTAILQ_FOREACH(cfn, &cs->funcs, list) {
+		if (cfn->auth > cli->auth)
+			continue;
+		for (cp = cfn->clp; cp->request != NULL; cp++) {
+			f_wc = f_i = 0;
+			for (u = 0; u < sizeof cp->flags; u++) {
+				if (cp->flags[u] == '*')
+					f_wc = 1;
+				if (cp->flags[u] == 'i')
+					f_i = 1;
+			}
+			if (f_wc) {
+				cp->func(cli, av, priv);
+				continue;
+			}
+			if (f_i)
+				continue;
+			VCLI_Out(cli, ",\n  {");
+			VCLI_Out(cli, "\n  \"request\": ");
+			VCLI_JSON_str(cli, cp->request);
+			VCLI_Out(cli, ",\n  \"syntax\": ");
+			VCLI_JSON_str(cli, cp->syntax);
+			VCLI_Out(cli, ",\n  \"help\": ");
+			VCLI_JSON_str(cli, cp->help);
+			VCLI_Out(cli, ",\n  \"minarg\": %d", cp->minarg);
+			VCLI_Out(cli, ", \"maxarg\": %d", cp->maxarg);
+			VCLI_Out(cli, ", \"flags\": ");
+			VCLI_JSON_str(cli, cp->flags);
+			VCLI_Out(cli, ", \"json\": %s",
+			    cp->jsonfunc == NULL ? "false" : "true");
+			VCLI_Out(cli, "\n  }");
+		}
+	}
+	if (priv == NULL)
+		VCLI_Out(cli, "\n]\n");
+}
+
 /*--------------------------------------------------------------------
  * Look for a CLI command to execute
  */
@@ -201,6 +252,7 @@ cls_dispatch(struct cli *cli, struct cli_proto *clp, char * const * av,
     unsigned ac)
 {
 	struct cli_proto *cp;
+	int json = 0;
 
 	AN(av);
 	for (cp = clp; cp->request != NULL; cp++) {
@@ -212,19 +264,27 @@ cls_dispatch(struct cli *cli, struct cli_proto *clp, char * const * av,
 	if (cp->request == NULL)
 		return (0);
 
-	if (cp->func == NULL) {
+	if (ac > 1 && !strcmp(av[2], "-j"))
+		json = 1;
+
+	if (cp->func == NULL && !json) {
 		VCLI_Out(cli, "Unimplemented\n");
 		VCLI_SetResult(cli, CLIS_UNIMPL);
 		return(1);
 	}
+	if (cp->jsonfunc == NULL && json) {
+		VCLI_Out(cli, "JSON unimplemented\n");
+		VCLI_SetResult(cli, CLIS_UNIMPL);
+		return(1);
+	}
 
-	if (ac - 1 < cp->minarg) {
+	if (ac - 1 < cp->minarg + json) {
 		VCLI_Out(cli, "Too few parameters\n");
 		VCLI_SetResult(cli, CLIS_TOOFEW);
 		return(1);
 	}
 
-	if (ac - 1> cp->maxarg) {
+	if (ac - 1> cp->maxarg + json) {
 		VCLI_Out(cli, "Too many parameters\n");
 		VCLI_SetResult(cli, CLIS_TOOMANY);
 		return(1);
@@ -232,7 +292,10 @@ cls_dispatch(struct cli *cli, struct cli_proto *clp, char * const * av,
 
 	cli->result = CLIS_OK;
 	VSB_clear(cli->sb);
-	cp->func(cli, (const char * const *)av, cp->priv);
+	if (json)
+		cp->jsonfunc(cli, (const char * const *)av, cp->priv);
+	else
+		cp->func(cli, (const char * const *)av, cp->priv);
 	return (1);
 }
 
diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c
index 4d80b8a..d737ef1 100644
--- a/lib/libvarnish/vsb.c
+++ b/lib/libvarnish/vsb.c
@@ -498,7 +498,6 @@ VSB_quote(struct vsb *s, const char *p, int len, int how)
 	const char *q;
 	int quote = 0;
 
-	(void)how;	/* For future enhancements */
 	if (len == -1)
 		len = strlen(p);
 
@@ -508,7 +507,7 @@ VSB_quote(struct vsb *s, const char *p, int len, int how)
 			break;
 		}
 	}
-	if (!quote) {
+	if (!quote && !(how & VSB_QUOTE_JSON)) {
 		(void)VSB_bcat(s, p, len);
 		return;
 	}
@@ -536,6 +535,7 @@ VSB_quote(struct vsb *s, const char *p, int len, int how)
 			(void)VSB_cat(s, "\\t");
 			break;
 		default:
+			/* XXX: Implement VSB_QUOTE_JSON */
 			if (isgraph(*q))
 				(void)VSB_putc(s, *q);
 			else



From phk at FreeBSD.org  Thu Feb  4 10:35:52 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 04 Feb 2016 11:35:52 +0100
Subject: [master] 18caa69 Unify objcore allocation
Message-ID: <E1aRHGS-0001sB-QY@project.varnish-software.com>


commit 18caa69dad3b78838c2af5d9699d08475c9ce2ae
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 4 10:35:33 2016 +0000

    Unify objcore allocation

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index dc08c24..fde3164 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -837,6 +837,7 @@ void *MPL_Get(struct mempool *mpl, unsigned *size);
 void MPL_Free(struct mempool *mpl, void *item);
 
 /* cache_obj.c */
+struct objcore * ObjNew(struct worker *, int wantboc);
 typedef int objiterate_f(void *priv, int flush, const void *ptr, ssize_t len);
 int ObjIterate(struct worker *, struct objcore *,
     void *priv, objiterate_f *func);
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 1d594d7..6ad54e9 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -67,28 +67,6 @@ static struct objhead *private_oh;
 
 /*---------------------------------------------------------------------*/
 
-static struct objcore *
-hsh_NewObjCore(struct worker *wrk, int boc)
-{
-	struct objcore *oc;
-
-	ALLOC_OBJ(oc, OBJCORE_MAGIC);
-	AN(oc);
-	wrk->stats->n_objectcore++;
-	oc->flags |= OC_F_BUSY | OC_F_INCOMPLETE;
-	oc->last_lru = NAN;
-	if (boc) {
-		ALLOC_OBJ(oc->boc, BOC_MAGIC);
-		AN(oc->boc);
-		Lck_New(&oc->boc->mtx, lck_busyobj);
-		AZ(pthread_cond_init(&oc->boc->cond, NULL));
-		oc->boc->refcount = 1;
-	}
-	return (oc);
-}
-
-/*---------------------------------------------------------------------*/
-
 static struct objhead *
 hsh_newobjhead(void)
 {
@@ -111,8 +89,10 @@ hsh_prealloc(struct worker *wrk)
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
-	if (wrk->nobjcore == NULL)
-		wrk->nobjcore = hsh_NewObjCore(wrk, 1);
+	if (wrk->nobjcore == NULL) {
+		wrk->nobjcore = ObjNew(wrk, 1);
+		wrk->nobjcore->flags |= OC_F_BUSY | OC_F_INCOMPLETE;
+	}
 	CHECK_OBJ_NOTNULL(wrk->nobjcore, OBJCORE_MAGIC);
 
 	if (wrk->nobjhead == NULL) {
@@ -134,11 +114,11 @@ HSH_Private(struct worker *wrk, int wantboc)
 
 	CHECK_OBJ_NOTNULL(private_oh, OBJHEAD_MAGIC);
 
-	oc = hsh_NewObjCore(wrk, wantboc);
+	oc = ObjNew(wrk, wantboc);
 	AN(oc);
 	oc->refcnt = 1;
 	oc->objhead = private_oh;
-	oc->flags |= OC_F_PRIVATE | OC_F_BUSY;
+	oc->flags |= OC_F_PRIVATE | OC_F_BUSY | OC_F_INCOMPLETE;
 	Lck_Lock(&private_oh->mtx);
 	VTAILQ_INSERT_TAIL(&private_oh->objcs, oc, list);
 	private_oh->refcnt++;
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 3cc195b..cd0c85c 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -60,6 +60,32 @@ obj_getmethods(const struct objcore *oc)
 }
 
 /*====================================================================
+ * ObjNew()
+ *
+ */
+
+struct objcore *
+ObjNew(struct worker *wrk, int wantboc)
+{
+	struct objcore *oc;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+
+	ALLOC_OBJ(oc, OBJCORE_MAGIC);
+	AN(oc);
+	wrk->stats->n_objectcore++;
+	oc->last_lru = NAN;
+	if (wantboc) {
+		ALLOC_OBJ(oc->boc, BOC_MAGIC);
+		AN(oc->boc);
+		Lck_New(&oc->boc->mtx, lck_busyobj);
+		AZ(pthread_cond_init(&oc->boc->cond, NULL));
+		oc->boc->refcount = 1;
+	}
+	return (oc);
+}
+
+/*====================================================================
  * ObjIterate()
  *
  */
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index a0723bd..32c598e 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -158,8 +158,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 	for (;no > 0; so++,no--) {
 		if (EXP_When(&so->exp) < t_now)
 			continue;
-		ALLOC_OBJ(oc, OBJCORE_MAGIC);
-		AN(oc);
+		oc = ObjNew(wrk, 0);
 		oc->flags &= ~OC_F_BUSY;
 		oc->stobj->stevedore = sc->parent;
 		smp_init_oc(oc, sg, no);
@@ -168,7 +167,6 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		HSH_Insert(wrk, so->hash, oc);
 		oc->exp = so->exp;
 		sg->nobj++;
-		oc->last_lru = NAN;
 		EXP_Inject(wrk, oc);
 	}
 	Pool_Sumstat(wrk);



From phk at FreeBSD.org  Thu Feb  4 11:04:18 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 04 Feb 2016 12:04:18 +0100
Subject: [master] 175cac7 LRU code polishing
Message-ID: <E1aRHhy-0004iZ-Nw@project.varnish-software.com>


commit 175cac7ef3a1e68d8ed0f4bd41cc2177d43233d6
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 4 11:04:06 2016 +0000

    LRU code polishing

diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index d63b82b..5e0d78f 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -119,14 +119,14 @@ int STV_GetFile(const char *fn, int *fdp, const char **fnp, const char *ctx);
 uintmax_t STV_FileSize(int fd, const char *size, unsigned *granularity,
     const char *ctx);
 
+/*--------------------------------------------------------------------*/
 struct lru *LRU_Alloc(void);
 void LRU_Free(struct lru *);
-void LRU_Add(struct objcore *);
+void LRU_Add(struct objcore *, double now);
 void LRU_Remove(struct objcore *);
 int LRU_NukeOne(struct worker *, struct lru *);
 void LRU_Touch(struct worker *, struct objcore *, double now);
 
-
 /*--------------------------------------------------------------------*/
 extern const struct stevedore sma_stevedore;
 extern const struct stevedore smf_stevedore;
diff --git a/bin/varnishd/storage/storage_lru.c b/bin/varnishd/storage/storage_lru.c
index 0acb91d..0fbfb36 100644
--- a/bin/varnishd/storage/storage_lru.c
+++ b/bin/varnishd/storage/storage_lru.c
@@ -25,22 +25,17 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
- * STEVEDORE: one who works at or is responsible for loading and
- * unloading ships in port.  Example: "on the wharves, stevedores were
- * unloading cargo from the far corners of the world." Origin: Spanish
- * estibador, from estibar to pack.  First Known Use: 1788
+ * Least-Recently-Used logic for freeing space in stevedores.
  */
 
 #include "config.h"
 
-#include <stdio.h>
 #include <stdlib.h>
 
 #include "cache/cache.h"
 #include "hash/hash_slinger.h"
 
 #include "storage/storage.h"
-#include "vtim.h"
 
 struct lru {
 	unsigned		magic;
@@ -70,16 +65,17 @@ LRU_Free(struct lru *lru)
 }
 
 void
-LRU_Add(struct objcore *oc)
+LRU_Add(struct objcore *oc, double now)
 {
 	struct lru *lru;
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AZ(isnan(now));
 	lru = ObjGetLRU(oc);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 	Lck_Lock(&lru->mtx);
 	VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
-	oc->last_lru = VTIM_real();
+	oc->last_lru = now;
 	Lck_Unlock(&lru->mtx);
 }
 
@@ -139,7 +135,7 @@ LRU_Touch(struct worker *wrk, struct objcore *oc, double now)
 /*--------------------------------------------------------------------
  * Attempt to make space by nuking the oldest object on the LRU list
  * which isn't in use.
- * Returns: 1: did, 0: didn't, -1: can't
+ * Returns: 1: did, 0: didn't;
  */
 
 int
@@ -170,7 +166,7 @@ LRU_NukeOne(struct worker *wrk, struct lru *lru)
 
 	if (oc == NULL) {
 		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Fail");
-		return (-1);
+		return (0);
 	}
 
 	/* XXX: We could grab and return one storage segment to our caller */
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index 735c7c5..f405755 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -531,7 +531,7 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv,
 
 	while (1) {
 		if (really > 0) {
-			if (LRU_NukeOne(wrk, stv->lru) == -1)
+			if (!LRU_NukeOne(wrk, stv->lru))
 				return (0);
 			really--;
 		}
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 95f4c05..b02846e 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -128,7 +128,7 @@ SML_allocobj(struct worker *wrk, const struct stevedore *stv,
 	ltot = sizeof(struct object) + PRNDUP(wsl);
 	while (1) {
 		if (really > 0) {
-			if (LRU_NukeOne(wrk, stv->lru) == -1)
+			if (!LRU_NukeOne(wrk, stv->lru))
 				return (0);
 			really--;
 		}
@@ -339,7 +339,7 @@ objallocwithnuke(struct worker *wrk, const struct stevedore *stv, size_t size)
 
 		/* no luck; try to free some space and keep trying */
 		if (fail < cache_param->nuke_limit &&
-		    LRU_NukeOne(wrk, stv->lru) == -1)
+		    !LRU_NukeOne(wrk, stv->lru))
 			break;
 	}
 	CHECK_OBJ_ORNULL(st, STORAGE_MAGIC);
@@ -505,7 +505,7 @@ sml_stable(struct worker *wrk, struct objcore *oc, struct boc *boc)
 		sml_stv_free(stv, st);
 	}
 
-	LRU_Add(oc);
+	LRU_Add(oc, wrk->lastused);	// approx timestamp is OK
 }
 
 static void * __match_proto__(objgetattr_f)



From phk at FreeBSD.org  Thu Feb  4 23:52:57 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 05 Feb 2016 00:52:57 +0100
Subject: [master] 0b5aeb3 More polishing and cleanup of LRU
Message-ID: <E1aRThp-0005AA-IE@project.varnish-software.com>


commit 0b5aeb39b08b60d39df1cca9d496be2c5ddf5970
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 4 23:08:19 2016 +0000

    More polishing and cleanup of LRU

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index fde3164..f7dd1ce 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -855,7 +855,6 @@ uint64_t ObjGetLen(struct worker *, struct objcore *);
 void ObjUpdateMeta(struct worker *, struct objcore *);
 void ObjFreeObj(struct worker *, struct objcore *);
 void ObjSlim(struct worker *, struct objcore *);
-struct lru *ObjGetLRU(const struct objcore *);
 int ObjHasAttr(struct worker *, struct objcore *, enum obj_attr);
 void *ObjGetAttr(struct worker *, struct objcore *, enum obj_attr,
     ssize_t *len);
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index cd0c85c..4968ee4 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -312,17 +312,6 @@ ObjFreeObj(struct worker *wrk, struct objcore *oc)
 }
 
 /*====================================================================
- */
-struct lru *
-ObjGetLRU(const struct objcore *oc)
-{
-	const struct obj_methods *m = obj_getmethods(oc);
-
-	AN(m->objgetlru);
-	return (m->objgetlru(oc));
-}
-
-/*====================================================================
  * ObjHasAttr()
  *
  * Check if object has this attribute
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index a24ca3a..feb41d6 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -63,8 +63,7 @@ cnt_deliver(struct worker *wrk, struct req *req)
 
 	assert(req->objcore->refcnt > 0);
 
-	if (req->objcore->exp_flags & OC_EF_EXP)
-		ObjTouch(req->wrk, req->objcore, req->t_prev);
+	ObjTouch(req->wrk, req->objcore, req->t_prev);
 
 	HTTP_Setup(req->resp, req->ws, req->vsl, SLT_RespMethod);
 	if (HTTP_Decode(req->resp,
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 5c20011..fcb8461 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -133,10 +133,9 @@ STV_open(void)
 			stv->open(stv);
 	}
 	stv = stv_transient;
-	if (stv->open != NULL) {
-		stv->lru = LRU_Alloc();
+	stv->lru = LRU_Alloc();
+	if (stv->open != NULL)
 		stv->open(stv);
-	}
 	stv_next = VTAILQ_FIRST(&stv_stevedores);
 }
 
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index 5e0d78f..d00db91 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -96,13 +96,13 @@ struct stevedore {
 	const struct obj_methods
 				*methods;
 
-	struct lru		*lru;
+	struct lru		*lru;		/* For storage_lru.c */
 
 #define VRTSTVVAR(nm, vtype, ctype, dval) storage_var_##ctype *var_##nm;
 #include "tbl/vrt_stv_var.h"
 #undef VRTSTVVAR
 
-	/* private fields */
+	/* private fields for the stevedore */
 	void			*priv;
 
 	VTAILQ_ENTRY(stevedore)	list;
diff --git a/bin/varnishd/storage/storage_lru.c b/bin/varnishd/storage/storage_lru.c
index 0fbfb36..4e7ae2e 100644
--- a/bin/varnishd/storage/storage_lru.c
+++ b/bin/varnishd/storage/storage_lru.c
@@ -33,6 +33,7 @@
 #include <stdlib.h>
 
 #include "cache/cache.h"
+#include "cache/cache_obj.h"
 #include "hash/hash_slinger.h"
 
 #include "storage/storage.h"
@@ -44,22 +45,36 @@ struct lru {
 	struct lock		mtx;
 };
 
+static struct lru *
+lru_get(const struct objcore *oc)
+{
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->stobj->stevedore, STEVEDORE_MAGIC);
+	AN(oc->stobj->stevedore->methods);
+	const struct obj_methods *m = oc->stobj->stevedore->methods;
+
+	if(m->objgetlru != NULL)
+		return (m->objgetlru(oc));
+	return (oc->stobj->stevedore->lru);
+}
+
 struct lru *
 LRU_Alloc(void)
 {
-	struct lru *l;
+	struct lru *lru;
 
-	ALLOC_OBJ(l, LRU_MAGIC);
-	AN(l);
-	VTAILQ_INIT(&l->lru_head);
-	Lck_New(&l->mtx, lck_lru);
-	return (l);
+	ALLOC_OBJ(lru, LRU_MAGIC);
+	AN(lru);
+	VTAILQ_INIT(&lru->lru_head);
+	Lck_New(&lru->mtx, lck_lru);
+	return (lru);
 }
 
 void
 LRU_Free(struct lru *lru)
 {
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+	AN(VTAILQ_EMPTY(&lru->lru_head));
 	Lck_Delete(&lru->mtx);
 	FREE_OBJ(lru);
 }
@@ -71,7 +86,7 @@ LRU_Add(struct objcore *oc, double now)
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	AZ(isnan(now));
-	lru = ObjGetLRU(oc);
+	lru = lru_get(oc);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 	Lck_Lock(&lru->mtx);
 	VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
@@ -85,7 +100,7 @@ LRU_Remove(struct objcore *oc)
 	struct lru *lru;
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	lru = ObjGetLRU(oc);
+	lru = lru_get(oc);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 	Lck_Lock(&lru->mtx);
 	if (!isnan(oc->last_lru)) {
@@ -103,6 +118,9 @@ LRU_Touch(struct worker *wrk, struct objcore *oc, double now)
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
+	if (isnan(oc->last_lru))
+		return;
+
 	/*
 	 * To avoid the exphdl->mtx becoming a hotspot, we only
 	 * attempt to move objects if they have not been moved
@@ -110,20 +128,16 @@ LRU_Touch(struct worker *wrk, struct objcore *oc, double now)
 	 * obviously leaves the LRU list imperfectly sorted.
 	 */
 
-	if (oc->flags & OC_F_INCOMPLETE)
-		return;
-
 	if (now - oc->last_lru < cache_param->lru_interval)
 		return;
 
-	lru = ObjGetLRU(oc);
+	lru = lru_get(oc);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 
 	if (Lck_Trylock(&lru->mtx))
 		return;
 
 	if (!isnan(oc->last_lru)) {
-		/* Can only touch it while it's actually on the LRU list */
 		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
 		VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
 		VSC_C_main->n_lru_moved++;
@@ -145,16 +159,16 @@ LRU_NukeOne(struct worker *wrk, struct lru *lru)
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+
 	/* Find the first currently unused object on the LRU.  */
 	Lck_Lock(&lru->mtx);
 	VTAILQ_FOREACH_SAFE(oc, &lru->lru_head, lru_list, oc2) {
 		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+		AZ(isnan(oc->last_lru));
 
 		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Cand p=%p f=0x%x r=%d",
 		    oc, oc->flags, oc->refcnt);
 
-		AZ(isnan(oc->last_lru));
-
 		if (ObjSnipe(wrk, oc)) {
 			VSC_C_main->n_lru_nuked++; // XXX per lru ?
 			VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
@@ -175,6 +189,6 @@ LRU_NukeOne(struct worker *wrk, struct lru *lru)
 	EXP_Poke(oc);
 
 	VSLb(wrk->vsl, SLT_ExpKill, "LRU x=%u", ObjGetXID(wrk, oc));
-	(void)HSH_DerefObjCore(wrk, &oc);
+	(void)HSH_DerefObjCore(wrk, &oc);	// Ref from ObjSnipe
 	return (1);
 }
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index b02846e..1280537 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -215,16 +215,6 @@ sml_objfree(struct worker *wrk, struct objcore *oc)
 	wrk->stats->n_object--;
 }
 
-static struct lru * __match_proto__(objgetlru_f)
-sml_objgetlru(const struct objcore *oc)
-{
-	const struct stevedore *stv;
-
-	stv = oc->stobj->stevedore;
-	CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
-	return (stv->lru);
-}
-
 static int __match_proto__(objiterate_f)
 sml_iterator(struct worker *wrk, struct objcore *oc,
     void *priv, objiterate_f *func)
@@ -613,7 +603,6 @@ sml_setattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
 
 const struct obj_methods SML_methods = {
 	.objfree	= sml_objfree,
-	.objgetlru	= sml_objgetlru,
 	.objiterator	= sml_iterator,
 	.objgetspace	= sml_getspace,
 	.objextend	= sml_extend,



From phk at FreeBSD.org  Thu Feb  4 23:52:57 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 05 Feb 2016 00:52:57 +0100
Subject: [master] 8a34320 Turn the object length into a OA
Message-ID: <E1aRThp-0005AH-Jl@project.varnish-software.com>


commit 8a34320c3b5d10638658125943158af3fcd2d5f7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 4 23:40:09 2016 +0000

    Turn the object length into a OA

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index f7dd1ce..24c85cb 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -843,7 +843,7 @@ int ObjIterate(struct worker *, struct objcore *,
     void *priv, objiterate_f *func);
 int ObjGetSpace(struct worker *, struct objcore *, ssize_t *sz, uint8_t **ptr);
 void ObjExtend(struct worker *, struct objcore *, ssize_t l);
-ssize_t ObjWaitExtend(struct worker *, struct objcore *, ssize_t l);
+uint64_t ObjWaitExtend(struct worker *, struct objcore *, uint64_t l);
 void ObjSetState(const struct objcore *, enum boc_state_e next);
 void ObjWaitState(const struct objcore *, enum boc_state_e want);
 void ObjTrimStore(struct worker *, struct objcore *);
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 4968ee4..0ae34ed 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -136,41 +136,47 @@ void
 ObjExtend(struct worker *wrk, struct objcore *oc, ssize_t l)
 {
 	const struct obj_methods *om = obj_getmethods(oc);
+	uint64_t len;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_ORNULL(oc->boc, BOC_MAGIC);
-	AN(om->objextend);
 	assert(l > 0);
 
+	AZ(ObjGetU64(wrk, oc, OA_LEN, &len));
+	len += l;
+
 	if (oc->boc != NULL) {
 		Lck_Lock(&oc->boc->mtx);
+		AN(om->objextend);
 		om->objextend(wrk, oc, l);
-		AZ(pthread_cond_broadcast(&oc->boc->cond));
+		AZ(ObjSetU64(wrk, oc, OA_LEN, len));
 		Lck_Unlock(&oc->boc->mtx);
+		AZ(pthread_cond_broadcast(&oc->boc->cond));
 	} else {
 		om->objextend(wrk, oc, l);
+		AZ(ObjSetU64(wrk, oc, OA_LEN, len));
 	}
 }
 
 /*====================================================================
  */
 
-ssize_t
-ObjWaitExtend(struct worker *wrk, struct objcore *oc, ssize_t l)
+uint64_t
+ObjWaitExtend(struct worker *wrk, struct objcore *oc, uint64_t l)
 {
-	ssize_t rv;
+	uint64_t rv;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 	Lck_Lock(&oc->boc->mtx);
-	rv = ObjGetLen(wrk, oc);
+	AZ(ObjGetU64(wrk, oc, OA_LEN, &rv));
 	while (1) {
 		assert(l <= rv || oc->boc->state == BOS_FAILED);
 		if (rv > l || oc->boc->state >= BOS_FINISHED)
 			break;
 		(void)Lck_CondWait(&oc->boc->cond, &oc->boc->mtx, 0);
-		rv = ObjGetLen(wrk, oc);
+		AZ(ObjGetU64(wrk, oc, OA_LEN, &rv));
 	}
 	Lck_Unlock(&oc->boc->mtx);
 	return (rv);
@@ -221,12 +227,12 @@ ObjWaitState(const struct objcore *oc, enum boc_state_e want)
 uint64_t
 ObjGetLen(struct worker *wrk, struct objcore *oc)
 {
-	const struct obj_methods *om = obj_getmethods(oc);
+	uint64_t len;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
-	AN(om->objgetlen);
-	return (om->objgetlen(wrk, oc));
+	AZ(ObjGetU64(wrk, oc, OA_LEN, &len));
+	return(len);
 }
 
 /*====================================================================
diff --git a/bin/varnishd/cache/cache_obj.h b/bin/varnishd/cache/cache_obj.h
index a981fb2..e61324e 100644
--- a/bin/varnishd/cache/cache_obj.h
+++ b/bin/varnishd/cache/cache_obj.h
@@ -49,7 +49,6 @@ typedef void *objgetattr_f(struct worker *, struct objcore *,
     enum obj_attr attr, ssize_t *len);
 typedef void *objsetattr_f(struct worker *, struct objcore *,
     enum obj_attr attr, ssize_t len, const void *ptr);
-typedef uint64_t objgetlen_f(struct worker *, struct objcore *);
 typedef void objtouch_f(struct worker *, struct objcore *, double now);
 
 struct obj_methods {
@@ -62,7 +61,6 @@ struct obj_methods {
 	objiterator_f	*objiterator;
 	objgetspace_f	*objgetspace;
 	objextend_f	*objextend;
-	objgetlen_f	*objgetlen;
 	objtrimstore_f	*objtrimstore;
 	objstable_f	*objstable;
 	objslim_f	*objslim;
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 32c598e..60ec88e 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -45,6 +45,7 @@
 
 #include "hash/hash_slinger.h"
 #include "vsha256.h"
+#include "vend.h"
 #include "vtim.h"
 
 #include "storage/storage_persistent.h"
@@ -435,7 +436,7 @@ smp_oc_sml_getobj(struct worker *wrk, struct objcore *oc)
 				break;
 			l += st->len;
 		}
-		if (l != o->len)
+		if (l != vbe64dec(o->oa_len))
 			bad |= 0x100;
 
 		if(bad) {
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 1280537..84cf755 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -393,19 +393,6 @@ sml_extend(struct worker *wrk, struct objcore *oc, ssize_t l)
 	CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
 	assert(st->len + l <= st->space);
 	st->len += l;
-	o->len += l;
-}
-
-static uint64_t __match_proto__(objgetlen_f)
-sml_getlen(struct worker *wrk, struct objcore *oc)
-{
-	struct object *o;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-
-	o = sml_getobj(wrk, oc);
-	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	return (o->len);
 }
 
 static void __match_proto__(objtrimstore_f)
@@ -532,6 +519,9 @@ sml_getattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
 	case OA_VARY:
 		*len = 4;			// XXX: hack
 		return (o->oa_vary);
+	case OA_LEN:
+		*len = sizeof o->oa_len;
+		return (o->oa_len);
 	case OA_VXID:
 		*len = sizeof o->oa_vxid;
 		return (o->oa_vxid);
@@ -588,6 +578,10 @@ sml_setattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
 		st->len += len;
 		retval = o->oa_vary;
 		break;
+	case OA_LEN:
+		assert(len == sizeof o->oa_len);
+		retval = o->oa_len;
+		break;
 	case OA_VXID:
 		assert(len == sizeof o->oa_vxid);
 		retval = o->oa_vxid;
@@ -606,7 +600,6 @@ const struct obj_methods SML_methods = {
 	.objiterator	= sml_iterator,
 	.objgetspace	= sml_getspace,
 	.objextend	= sml_extend,
-	.objgetlen	= sml_getlen,
 	.objtrimstore	= sml_trimstore,
 	.objstable	= sml_stable,
 	.objslim	= sml_slim,
diff --git a/bin/varnishd/storage/storage_simple.h b/bin/varnishd/storage/storage_simple.h
index ce4a286..84a3277 100644
--- a/bin/varnishd/storage/storage_simple.h
+++ b/bin/varnishd/storage/storage_simple.h
@@ -40,7 +40,8 @@ struct object {
 #define OBJECT_MAGIC		0x32851d42
 	struct storage		*objstore;
 
-	char			oa_vxid[4];
+	uint8_t			oa_len[8];
+	uint8_t			oa_vxid[4];
 	uint8_t			*oa_vary;
 	uint8_t			*oa_http;
 	uint8_t			oa_flags[1];
@@ -48,7 +49,6 @@ struct object {
 	char			oa_lastmodified[8];
 
 	struct storagehead	list;
-	ssize_t			len;
 
 	struct storage		*esidata;
 };
diff --git a/include/tbl/obj_attr.h b/include/tbl/obj_attr.h
index d0f281e..310ffab 100644
--- a/include/tbl/obj_attr.h
+++ b/include/tbl/obj_attr.h
@@ -31,6 +31,7 @@
 
 /* upper, lower */
 #ifdef OBJ_ATTR
+OBJ_ATTR(LEN,		len)
 OBJ_ATTR(VXID,		vxid)
 OBJ_ATTR(EXP,		exp)
 OBJ_ATTR(VARY,		vary)



From phk at FreeBSD.org  Fri Feb  5 12:31:45 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 05 Feb 2016 13:31:45 +0100
Subject: [master] 8b4036e Further tightening up of obj/stv/lru
Message-ID: <E1aRfY9-00061B-67@project.varnish-software.com>


commit 8b4036e951bb6b7c9fceb4175f452dba6c9d9e0f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 5 12:30:40 2016 +0000

    Further tightening up of obj/stv/lru
    
    All objcores, also for synth and req.body are now born with a boc,
    which must of course be properly disposed of.
    
    More assertive LRU code.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 24c85cb..5e9d30d 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -837,7 +837,7 @@ void *MPL_Get(struct mempool *mpl, unsigned *size);
 void MPL_Free(struct mempool *mpl, void *item);
 
 /* cache_obj.c */
-struct objcore * ObjNew(struct worker *, int wantboc);
+struct objcore * ObjNew(struct worker *);
 typedef int objiterate_f(void *priv, int flush, const void *ptr, ssize_t len);
 int ObjIterate(struct worker *, struct objcore *,
     void *priv, objiterate_f *func);
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 6ad54e9..1a28fbf 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -90,7 +90,7 @@ hsh_prealloc(struct worker *wrk)
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
 	if (wrk->nobjcore == NULL) {
-		wrk->nobjcore = ObjNew(wrk, 1);
+		wrk->nobjcore = ObjNew(wrk);
 		wrk->nobjcore->flags |= OC_F_BUSY | OC_F_INCOMPLETE;
 	}
 	CHECK_OBJ_NOTNULL(wrk->nobjcore, OBJCORE_MAGIC);
@@ -108,13 +108,13 @@ hsh_prealloc(struct worker *wrk)
 /*---------------------------------------------------------------------*/
 
 struct objcore *
-HSH_Private(struct worker *wrk, int wantboc)
+HSH_Private(struct worker *wrk)
 {
 	struct objcore *oc;
 
 	CHECK_OBJ_NOTNULL(private_oh, OBJHEAD_MAGIC);
 
-	oc = ObjNew(wrk, wantboc);
+	oc = ObjNew(wrk);
 	AN(oc);
 	oc->refcnt = 1;
 	oc->objhead = private_oh;
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 0ae34ed..9f0bdf2 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -65,7 +65,7 @@ obj_getmethods(const struct objcore *oc)
  */
 
 struct objcore *
-ObjNew(struct worker *wrk, int wantboc)
+ObjNew(struct worker *wrk)
 {
 	struct objcore *oc;
 
@@ -75,13 +75,11 @@ ObjNew(struct worker *wrk, int wantboc)
 	AN(oc);
 	wrk->stats->n_objectcore++;
 	oc->last_lru = NAN;
-	if (wantboc) {
-		ALLOC_OBJ(oc->boc, BOC_MAGIC);
-		AN(oc->boc);
-		Lck_New(&oc->boc->mtx, lck_busyobj);
-		AZ(pthread_cond_init(&oc->boc->cond, NULL));
-		oc->boc->refcount = 1;
-	}
+	ALLOC_OBJ(oc->boc, BOC_MAGIC);
+	AN(oc->boc);
+	Lck_New(&oc->boc->mtx, lck_busyobj);
+	AZ(pthread_cond_init(&oc->boc->cond, NULL));
+	oc->boc->refcount = 1;
 	return (oc);
 }
 
@@ -117,6 +115,7 @@ ObjGetSpace(struct worker *wrk, struct objcore *oc, ssize_t *sz, uint8_t **ptr)
 	const struct obj_methods *om = obj_getmethods(oc);
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 	AN(sz);
 	AN(ptr);
 	assert(*sz > 0);
@@ -139,7 +138,7 @@ ObjExtend(struct worker *wrk, struct objcore *oc, ssize_t l)
 	uint64_t len;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CHECK_OBJ_ORNULL(oc->boc, BOC_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 	assert(l > 0);
 
 	AZ(ObjGetU64(wrk, oc, OA_LEN, &len));
@@ -369,6 +368,7 @@ ObjSetAttr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
 	const struct obj_methods *om = obj_getmethods(oc);
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 
 	AN(om->objsetattr);
 	assert((int)attr < 16);
@@ -451,6 +451,7 @@ ObjCopyAttr(struct worker *wrk, struct objcore *oc, struct objcore *ocs,
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 	CHECK_OBJ_NOTNULL(ocs, OBJCORE_MAGIC);
 
 	vps = ObjGetAttr(wrk, ocs, attr, &l);
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index d670326..79566af 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -418,6 +418,8 @@ pan_req(struct vsb *vsb, const struct req *req)
 
 	VCL_Panic(vsb, req->vcl);
 
+	if (req->body_oc != NULL)
+		pan_objcore(vsb, "BODY", req->body_oc);
 	if (req->objcore != NULL)
 		pan_objcore(vsb, "REQ", req->objcore);
 
diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c
index c3fd142..90145e4 100644
--- a/bin/varnishd/cache/cache_req_body.c
+++ b/bin/varnishd/cache/cache_req_body.c
@@ -216,16 +216,19 @@ VRB_Cache(struct req *req, ssize_t maxsize)
 		return (-1);
 	}
 
-	req->body_oc = HSH_Private(req->wrk, 0);
+	vfc->http = req->http;
+
+	req->body_oc = HSH_Private(req->wrk);
 	AN(req->body_oc);
 	XXXAN(STV_NewObject(req->wrk, req->body_oc, TRANSIENT_STORAGE, 8));
 
-	vfc->http = req->http;
 	vfc->oc = req->body_oc;
 	V1F_Setup_Fetch(vfc, req->htc);
 
 	if (VFP_Open(vfc) < 0) {
 		req->req_body_status = REQ_BODY_FAIL;
+		HSH_DerefBusy(req->wrk, req->body_oc);
+		AZ(HSH_DerefObjCore(req->wrk, &req->body_oc));
 		return (-1);
 	}
 
@@ -239,6 +242,8 @@ VRB_Cache(struct req *req, ssize_t maxsize)
 		if (req->req_bodybytes > maxsize) {
 			req->req_body_status = REQ_BODY_FAIL;
 			(void)VFP_Error(vfc, "Request body too big to cache");
+			HSH_DerefBusy(req->wrk, req->body_oc);
+			AZ(HSH_DerefObjCore(req->wrk, &req->body_oc));
 			VFP_Close(vfc);
 			return(-1);
 		}
@@ -286,5 +291,6 @@ VRB_Cache(struct req *req, ssize_t maxsize)
 		req->req_body_status = REQ_BODY_FAIL;
 	}
 	VSLb_ts_req(req, "ReqBody", VTIM_real());
+	HSH_DerefBusy(req->wrk, req->body_oc);
 	return (vfps == VFP_END ? req->req_bodybytes : -1);
 }
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index feb41d6..e468aab 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -191,8 +191,7 @@ cnt_synth(struct worker *wrk, struct req *req)
 	}
 	assert(wrk->handling == VCL_RET_DELIVER);
 
-	req->objcore = HSH_Private(wrk, 0);
-	AZ(req->objcore->boc);
+	req->objcore = HSH_Private(wrk);
 	CHECK_OBJ_NOTNULL(req->objcore, OBJCORE_MAGIC);
 	szl = -1;
 	if (STV_NewObject(wrk, req->objcore, TRANSIENT_STORAGE, 1024)) {
@@ -208,6 +207,7 @@ cnt_synth(struct worker *wrk, struct req *req)
 		}
 	}
 
+	HSH_DerefBusy(wrk, req->objcore);
 	VSB_delete(synth_body);
 
 	if (szl < 0) {
@@ -551,7 +551,7 @@ cnt_pass(struct worker *wrk, struct req *req)
 		break;
 	case VCL_RET_FETCH:
 		wrk->stats->s_pass++;
-		req->objcore = HSH_Private(wrk, 1);
+		req->objcore = HSH_Private(wrk);
 		CHECK_OBJ_NOTNULL(req->objcore, OBJCORE_MAGIC);
 		VBF_Fetch(wrk, req, req->objcore, NULL, VBF_PASS);
 		req->req_step = R_STP_FETCH;
diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h
index e9752be..54becbb 100644
--- a/bin/varnishd/hash/hash_slinger.h
+++ b/bin/varnishd/hash/hash_slinger.h
@@ -73,7 +73,7 @@ void HSH_Purge(struct worker *, struct objhead *, double ttl, double grace,
 void HSH_config(const char *h_arg);
 struct boc *HSH_RefBusy(const struct objcore *);
 void HSH_DerefBusy(struct worker *wrk, struct objcore *);
-struct objcore *HSH_Private(struct worker *wrk, int wantboc);
+struct objcore *HSH_Private(struct worker *wrk);
 void HSH_Abandon(struct objcore *oc);
 
 #ifdef VARNISH_CACHE_CHILD
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index d00db91..32b7e47 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -121,7 +121,7 @@ uintmax_t STV_FileSize(int fd, const char *size, unsigned *granularity,
 
 /*--------------------------------------------------------------------*/
 struct lru *LRU_Alloc(void);
-void LRU_Free(struct lru *);
+void LRU_Free(struct lru **);
 void LRU_Add(struct objcore *, double now);
 void LRU_Remove(struct objcore *);
 int LRU_NukeOne(struct worker *, struct lru *);
diff --git a/bin/varnishd/storage/storage_lru.c b/bin/varnishd/storage/storage_lru.c
index 4e7ae2e..54b8619 100644
--- a/bin/varnishd/storage/storage_lru.c
+++ b/bin/varnishd/storage/storage_lru.c
@@ -71,10 +71,17 @@ LRU_Alloc(void)
 }
 
 void
-LRU_Free(struct lru *lru)
+LRU_Free(struct lru **pp)
 {
+	struct lru *lru;
+
+	AN(pp);
+	lru = *pp;
+	*pp = NULL;
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
+	Lck_Lock(&lru->mtx);
 	AN(VTAILQ_EMPTY(&lru->lru_head));
+	Lck_Unlock(&lru->mtx);
 	Lck_Delete(&lru->mtx);
 	FREE_OBJ(lru);
 }
@@ -85,12 +92,15 @@ LRU_Add(struct objcore *oc, double now)
 	struct lru *lru;
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AZ(oc->boc);
+	AN(isnan(oc->last_lru));
 	AZ(isnan(now));
 	lru = lru_get(oc);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 	Lck_Lock(&lru->mtx);
 	VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
 	oc->last_lru = now;
+	AZ(isnan(oc->last_lru));
 	Lck_Unlock(&lru->mtx);
 }
 
@@ -100,13 +110,13 @@ LRU_Remove(struct objcore *oc)
 	struct lru *lru;
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AZ(oc->boc);
 	lru = lru_get(oc);
 	CHECK_OBJ_NOTNULL(lru, LRU_MAGIC);
 	Lck_Lock(&lru->mtx);
-	if (!isnan(oc->last_lru)) {
-		VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-		oc->last_lru = NAN;
-	}
+	AZ(isnan(oc->last_lru));
+	VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
+	oc->last_lru = NAN;
 	Lck_Unlock(&lru->mtx);
 }
 
@@ -172,7 +182,7 @@ LRU_NukeOne(struct worker *wrk, struct lru *lru)
 		if (ObjSnipe(wrk, oc)) {
 			VSC_C_main->n_lru_nuked++; // XXX per lru ?
 			VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
-			oc->last_lru = NAN;
+			VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);
 			break;
 		}
 	}
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 60ec88e..df159e1 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -51,7 +51,7 @@
 #include "storage/storage_persistent.h"
 
 /*
- * We use the low bit to mark objects still needing fixup
+ * We use the top bit to mark objects still needing fixup
  * In theory this may need to be platform dependent
  */
 
@@ -106,7 +106,7 @@ smp_save_segs(struct smp_sc *sc)
 		if (sg == sc->cur_seg)
 			continue;
 		VTAILQ_REMOVE(&sc->segments, sg, list);
-		LRU_Free(sg->lru);
+		LRU_Free(&sg->lru);
 		FREE_OBJ(sg);
 	}
 	smp_save_seg(sc, &sc->seg1);
@@ -159,7 +159,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 	for (;no > 0; so++,no--) {
 		if (EXP_When(&so->exp) < t_now)
 			continue;
-		oc = ObjNew(wrk, 0);
+		oc = ObjNew(wrk);
 		oc->flags &= ~OC_F_BUSY;
 		oc->stobj->stevedore = sc->parent;
 		smp_init_oc(oc, sg, no);
@@ -169,6 +169,9 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		oc->exp = so->exp;
 		sg->nobj++;
 		EXP_Inject(wrk, oc);
+		AN(isnan(oc->last_lru));
+		HSH_DerefBusy(wrk, oc);	// XXX Keep it an stream resurrection?
+		AZ(isnan(oc->last_lru));
 	}
 	Pool_Sumstat(wrk);
 	sg->flags |= SMP_SEG_LOADED;
@@ -272,7 +275,7 @@ smp_close_seg(struct smp_sc *sc, struct smp_seg *sg)
 		assert(sg->p.offset >= sc->ident->stuff[SMP_SPC_STUFF]);
 		assert(sg->p.offset < sc->mediasize);
 		sc->free_offset = sg->p.offset;
-		LRU_Free(sg->lru);
+		LRU_Free(&sg->lru);
 		FREE_OBJ(sg);
 		return;
 	}
@@ -507,6 +510,8 @@ smp_oc_objfree(struct worker *wrk, struct objcore *oc)
 		sg->nfixed--;
 		wrk->stats->n_object--;
 	}
+	AZ(isnan(oc->last_lru));
+	LRU_Remove(oc);
 
 	Lck_Unlock(&sg->sc->mtx);
 	memset(oc->stobj, 0, sizeof oc->stobj);
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 84cf755..f27bc95 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -206,7 +206,8 @@ sml_objfree(struct worker *wrk, struct objcore *oc)
 	CAST_OBJ_NOTNULL(o, oc->stobj->priv, OBJECT_MAGIC);
 	o->magic = 0;
 
-	LRU_Remove(oc);
+	if (oc->boc == NULL)
+		LRU_Remove(oc);
 
 	sml_stv_free(oc->stobj->stevedore, o->objstore);
 



From phk at FreeBSD.org  Fri Feb  5 20:09:46 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 05 Feb 2016 21:09:46 +0100
Subject: [master] 3995e91 Rename ref/rel functions from "busy" to "boc" and
 remove special cases dealing with oc's without a boc duing fill-up.
Message-ID: <E1aRmhO-000500-Il@project.varnish-software.com>


commit 3995e9106edf3acc4fca960554682064c5815371
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 5 20:08:45 2016 +0000

    Rename ref/rel functions from "busy" to "boc" and remove special
    cases dealing with oc's without a boc duing fill-up.

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 99b283d..35a6dd6 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -956,7 +956,7 @@ vbf_fetch_thread(struct worker *wrk, void *priv)
 		(void)HSH_DerefObjCore(wrk, &bo->stale_oc);
 
 	wrk->vsl = NULL;
-	HSH_DerefBusy(wrk, bo->fetch_objcore);
+	HSH_DerefBoc(wrk, bo->fetch_objcore);
 	VBO_ReleaseBusyObj(wrk, &bo);
 	THR_SetBusyobj(NULL);
 }
@@ -989,7 +989,7 @@ VBF_Fetch(struct worker *wrk, struct req *req, struct objcore *oc,
 	bo = VBO_GetBusyObj(wrk, req);
 	CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC);
 
-	boc = HSH_RefBusy(oc);
+	boc = HSH_RefBoc(oc);
 	CHECK_OBJ_NOTNULL(boc, BOC_MAGIC);
 
 	VSLb(bo->vsl, SLT_Begin, "bereq %u %s", VXID(req->vsl->wid), how);
@@ -1028,7 +1028,7 @@ VBF_Fetch(struct worker *wrk, struct req *req, struct objcore *oc,
 		(void)vbf_stp_fail(req->wrk, bo);
 		if (bo->stale_oc != NULL)
 			(void)HSH_DerefObjCore(wrk, &bo->stale_oc);
-		HSH_DerefBusy(wrk, oc);
+		HSH_DerefBoc(wrk, oc);
 	} else {
 		bo = NULL; /* ref transferred to fetch thread */
 		if (mode == VBF_BACKGROUND) {
@@ -1045,7 +1045,7 @@ VBF_Fetch(struct worker *wrk, struct req *req, struct objcore *oc,
 	AZ(bo);
 	VSLb_ts_req(req, "Fetch", W_TIM_real(wrk));
 	assert(oc->boc == boc);
-	HSH_DerefBusy(wrk, oc);
+	HSH_DerefBoc(wrk, oc);
 	if (mode == VBF_BACKGROUND)
 		(void)HSH_DerefObjCore(wrk, &oc);
 	THR_SetBusyobj(NULL);
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 1a28fbf..b9bff87 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -385,7 +385,7 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 			if (req->hash_ignore_busy)
 				continue;
 
-			if (oc->boc != NULL && oc->boc->vary != NULL &&
+			if (oc->boc->vary != NULL &&
 			    !VRY_Match(req, oc->boc->vary))
 				continue;
 
@@ -714,7 +714,7 @@ HSH_Ref(struct objcore *oc)
  */
 
 struct boc *
-HSH_RefBusy(const struct objcore *oc)
+HSH_RefBoc(const struct objcore *oc)
 {
 	struct objhead *oh;
 	struct boc *boc;
@@ -735,7 +735,7 @@ HSH_RefBusy(const struct objcore *oc)
 }
 
 void
-HSH_DerefBusy(struct worker *wrk, struct objcore *oc)
+HSH_DerefBoc(struct worker *wrk, struct objcore *oc)
 {
 	struct boc *boc;
 	unsigned r;
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 9f0bdf2..76f67df 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -144,17 +144,12 @@ ObjExtend(struct worker *wrk, struct objcore *oc, ssize_t l)
 	AZ(ObjGetU64(wrk, oc, OA_LEN, &len));
 	len += l;
 
-	if (oc->boc != NULL) {
-		Lck_Lock(&oc->boc->mtx);
-		AN(om->objextend);
-		om->objextend(wrk, oc, l);
-		AZ(ObjSetU64(wrk, oc, OA_LEN, len));
-		Lck_Unlock(&oc->boc->mtx);
-		AZ(pthread_cond_broadcast(&oc->boc->cond));
-	} else {
-		om->objextend(wrk, oc, l);
-		AZ(ObjSetU64(wrk, oc, OA_LEN, len));
-	}
+	Lck_Lock(&oc->boc->mtx);
+	AN(om->objextend);
+	om->objextend(wrk, oc, l);
+	AZ(ObjSetU64(wrk, oc, OA_LEN, len));
+	Lck_Unlock(&oc->boc->mtx);
+	AZ(pthread_cond_broadcast(&oc->boc->cond));
 }
 
 /*====================================================================
diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c
index 90145e4..06a43fa 100644
--- a/bin/varnishd/cache/cache_req_body.c
+++ b/bin/varnishd/cache/cache_req_body.c
@@ -227,7 +227,7 @@ VRB_Cache(struct req *req, ssize_t maxsize)
 
 	if (VFP_Open(vfc) < 0) {
 		req->req_body_status = REQ_BODY_FAIL;
-		HSH_DerefBusy(req->wrk, req->body_oc);
+		HSH_DerefBoc(req->wrk, req->body_oc);
 		AZ(HSH_DerefObjCore(req->wrk, &req->body_oc));
 		return (-1);
 	}
@@ -242,7 +242,7 @@ VRB_Cache(struct req *req, ssize_t maxsize)
 		if (req->req_bodybytes > maxsize) {
 			req->req_body_status = REQ_BODY_FAIL;
 			(void)VFP_Error(vfc, "Request body too big to cache");
-			HSH_DerefBusy(req->wrk, req->body_oc);
+			HSH_DerefBoc(req->wrk, req->body_oc);
 			AZ(HSH_DerefObjCore(req->wrk, &req->body_oc));
 			VFP_Close(vfc);
 			return(-1);
@@ -291,6 +291,6 @@ VRB_Cache(struct req *req, ssize_t maxsize)
 		req->req_body_status = REQ_BODY_FAIL;
 	}
 	VSLb_ts_req(req, "ReqBody", VTIM_real());
-	HSH_DerefBusy(req->wrk, req->body_oc);
+	HSH_DerefBoc(req->wrk, req->body_oc);
 	return (vfps == VFP_END ? req->req_bodybytes : -1);
 }
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index e468aab..fa8f373 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -207,7 +207,7 @@ cnt_synth(struct worker *wrk, struct req *req)
 		}
 	}
 
-	HSH_DerefBusy(wrk, req->objcore);
+	HSH_DerefBoc(wrk, req->objcore);
 	VSB_delete(synth_body);
 
 	if (szl < 0) {
@@ -241,7 +241,7 @@ cnt_transmit(struct worker *wrk, struct req *req)
 	CHECK_OBJ_NOTNULL(req->transport, TRANSPORT_MAGIC);
 
 	/* Grab a ref to the bo if there is one */
-	boc = HSH_RefBusy(req->objcore);
+	boc = HSH_RefBoc(req->objcore);
 
 	clval = http_GetContentLength(req->resp);
 	if (boc != NULL)
@@ -310,7 +310,7 @@ cnt_transmit(struct worker *wrk, struct req *req)
 	}
 
 	if (boc != NULL)
-		HSH_DerefBusy(wrk, req->objcore);
+		HSH_DerefBoc(wrk, req->objcore);
 
 	(void)HSH_DerefObjCore(wrk, &req->objcore);
 	http_Teardown(req->resp);
diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h
index 54becbb..9a64916 100644
--- a/bin/varnishd/hash/hash_slinger.h
+++ b/bin/varnishd/hash/hash_slinger.h
@@ -71,8 +71,8 @@ void HSH_Insert(struct worker *, const void *hash, struct objcore *);
 void HSH_Purge(struct worker *, struct objhead *, double ttl, double grace,
     double keep);
 void HSH_config(const char *h_arg);
-struct boc *HSH_RefBusy(const struct objcore *);
-void HSH_DerefBusy(struct worker *wrk, struct objcore *);
+struct boc *HSH_RefBoc(const struct objcore *);
+void HSH_DerefBoc(struct worker *wrk, struct objcore *);
 struct objcore *HSH_Private(struct worker *wrk);
 void HSH_Abandon(struct objcore *oc);
 
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index df159e1..0c7c8ec 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -170,7 +170,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		sg->nobj++;
 		EXP_Inject(wrk, oc);
 		AN(isnan(oc->last_lru));
-		HSH_DerefBusy(wrk, oc);	// XXX Keep it an stream resurrection?
+		HSH_DerefBoc(wrk, oc);	// XXX Keep it an stream resurrection?
 		AZ(isnan(oc->last_lru));
 	}
 	Pool_Sumstat(wrk);
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index f27bc95..3fdd242 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -236,7 +236,7 @@ sml_iterator(struct worker *wrk, struct objcore *oc,
 	obj = sml_getobj(wrk, oc);
 	CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
 
-	boc = HSH_RefBusy(oc);
+	boc = HSH_RefBoc(oc);
 
 	if (boc == NULL) {
 		VTAILQ_FOREACH(st, &obj->list, list)
@@ -300,7 +300,7 @@ sml_iterator(struct worker *wrk, struct objcore *oc,
 			break;
 		}
 	}
-	HSH_DerefBusy(wrk, oc);
+	HSH_DerefBoc(wrk, oc);
 	return (ret);
 }
 
@@ -351,6 +351,7 @@ sml_getspace(struct worker *wrk, struct objcore *oc, ssize_t *sz,
 
 	o = sml_getobj(wrk, oc);
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 
 	st = VTAILQ_LAST(&o->list, storagehead);
 	if (st != NULL && st->len < st->space) {
@@ -364,15 +365,11 @@ sml_getspace(struct worker *wrk, struct objcore *oc, ssize_t *sz,
 	if (st == NULL)
 		return (0);
 
-	if (oc->boc != NULL) {
-		CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
-		Lck_Lock(&oc->boc->mtx);
-		VTAILQ_INSERT_TAIL(&o->list, st, list);
-		Lck_Unlock(&oc->boc->mtx);
-	} else {
-		AN(oc->flags & (OC_F_PRIVATE));
-		VTAILQ_INSERT_TAIL(&o->list, st, list);
-	}
+	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
+	Lck_Lock(&oc->boc->mtx);
+	VTAILQ_INSERT_TAIL(&o->list, st, list);
+	Lck_Unlock(&oc->boc->mtx);
+
 	*sz = st->space - st->len;
 	assert (*sz > 0);
 	*ptr = st->ptr + st->len;
@@ -405,6 +402,7 @@ sml_trimstore(struct worker *wrk, struct objcore *oc)
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 
 	stv = oc->stobj->stevedore;
 	CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
@@ -420,13 +418,9 @@ sml_trimstore(struct worker *wrk, struct objcore *oc)
 		return;
 
 	if (st->len == 0) {
-		if (oc->boc != NULL) {
-			Lck_Lock(&oc->boc->mtx);
-			VTAILQ_REMOVE(&o->list, st, list);
-			Lck_Unlock(&oc->boc->mtx);
-		} else {
-			VTAILQ_REMOVE(&o->list, st, list);
-		}
+		Lck_Lock(&oc->boc->mtx);
+		VTAILQ_REMOVE(&o->list, st, list);
+		Lck_Unlock(&oc->boc->mtx);
 		sml_stv_free(stv, st);
 		return;
 	}
@@ -445,22 +439,13 @@ sml_trimstore(struct worker *wrk, struct objcore *oc)
 
 	memcpy(st1->ptr, st->ptr, st->len);
 	st1->len = st->len;
-	if (oc->boc != NULL) {
-		Lck_Lock(&oc->boc->mtx);
-		VTAILQ_REMOVE(&o->list, st, list);
-		VTAILQ_INSERT_TAIL(&o->list, st1, list);
-		Lck_Unlock(&oc->boc->mtx);
-	} else {
-		VTAILQ_REMOVE(&o->list, st, list);
-		VTAILQ_INSERT_TAIL(&o->list, st1, list);
-	}
-	if (oc->boc == NULL) {
-		sml_stv_free(stv, st);
-	} else {
-		/* sml_stable frees this */
-		AZ(oc->boc->stevedore_priv);
-		oc->boc->stevedore_priv = st;
-	}
+	Lck_Lock(&oc->boc->mtx);
+	VTAILQ_REMOVE(&o->list, st, list);
+	VTAILQ_INSERT_TAIL(&o->list, st1, list);
+	Lck_Unlock(&oc->boc->mtx);
+	/* sml_stable frees this */
+	AZ(oc->boc->stevedore_priv);
+	oc->boc->stevedore_priv = st;
 }
 
 static void __match_proto__(objstable_f)



From phk at FreeBSD.org  Fri Feb  5 21:21:09 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 05 Feb 2016 22:21:09 +0100
Subject: [master] cc5380f Reduce the amount of magic for persistent stevedores.
Message-ID: <E1aRnoT-0002fD-EK@project.varnish-software.com>


commit cc5380f2add34e2577d5339470a43e087bb6ed6e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 5 21:20:37 2016 +0000

    Reduce the amount of magic for persistent stevedores.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 5e9d30d..33c40ed 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -682,7 +682,6 @@ void EXP_Clr(struct exp *e);
 double EXP_Ttl(const struct req *, const struct exp*);
 double EXP_When(const struct exp *exp);
 void EXP_Insert(struct worker *wrk, struct objcore *oc);
-void EXP_Inject(struct worker *wrk, struct objcore *oc);
 void EXP_Rearm(struct objcore *, double now, double ttl, double grace,
     double keep);
 void EXP_Poke(struct objcore *);
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index d60900c..06db2d7 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -167,30 +167,6 @@ EXP_Poke(struct objcore *oc)
 }
 
 /*--------------------------------------------------------------------
- * Inject an object with a reference into the binheap.
- *
- * This can either come from a stevedore (persistent) during startup
- * or from EXP_Insert() below.
- */
-
-void
-EXP_Inject(struct worker *wrk, struct objcore *oc)
-{
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-
-	AZ(oc->exp_flags & (OC_EF_INSERT | OC_EF_MOVE));
-	AZ(oc->flags & OC_F_DYING);
-	AZ(oc->flags & OC_F_BUSY);
-
-	oc->timer_when = EXP_When(&oc->exp);
-
-	exp_event(wrk, oc, EXP_INJECT);
-	exp_mail_it(oc, OC_EF_INSERT | OC_EF_EXP);
-}
-
-/*--------------------------------------------------------------------
  * Insert new object.
  *
  * We grab a reference to the object, which will keep it around until
@@ -203,11 +179,11 @@ EXP_Insert(struct worker *wrk, struct objcore *oc)
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	HSH_Ref(oc);
 
 	AZ(oc->exp_flags & (OC_EF_INSERT | OC_EF_MOVE));
 	AZ(oc->flags & OC_F_DYING);
-	AN(oc->flags & OC_F_BUSY);
+
+	HSH_Ref(oc);
 
 	exp_event(wrk, oc, EXP_INSERT);
 	exp_mail_it(oc, OC_EF_INSERT | OC_EF_EXP | OC_EF_MOVE);
@@ -352,7 +328,7 @@ exp_expire(struct exp_priv *ep, double now)
 	if (oc == NULL)
 		return (now + 355./113.);
 	VSLb(&ep->vsl, SLT_ExpKill, "EXP_expire p=%p e=%.9f f=0x%x", oc,
-	    oc->timer_when, oc->flags);
+	    oc->timer_when - now, oc->flags);
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index b9bff87..11714f9 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -273,7 +273,6 @@ HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc)
 	assert(oh->refcnt > 0);
 
 	/* Insert (precreated) objcore in objecthead */
-	oc->refcnt = 1;
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	AZ(oc->flags & OC_F_BUSY);
 
@@ -745,6 +744,7 @@ HSH_DerefBoc(struct worker *wrk, struct objcore *oc)
 	boc = oc->boc;
 	CHECK_OBJ_NOTNULL(boc, BOC_MAGIC);
 	Lck_Lock(&oc->objhead->mtx);
+	assert(oc->refcnt > 0);
 	assert(boc->refcount > 0);
 	r = --boc->refcount;
 	if (r == 0)
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 0c7c8ec..deb1340 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -165,13 +165,15 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		smp_init_oc(oc, sg, no);
 		oc->stobj->priv2 |= NEED_FIXUP;
 		oc->ban = BAN_RefBan(oc, so->ban);
-		HSH_Insert(wrk, so->hash, oc);
 		oc->exp = so->exp;
 		sg->nobj++;
-		EXP_Inject(wrk, oc);
+		oc->refcnt++;
+		HSH_Insert(wrk, so->hash, oc);
+		EXP_Insert(wrk, oc);
 		AN(isnan(oc->last_lru));
 		HSH_DerefBoc(wrk, oc);	// XXX Keep it an stream resurrection?
 		AZ(isnan(oc->last_lru));
+		(void)HSH_DerefObjCore(wrk, &oc);
 	}
 	Pool_Sumstat(wrk);
 	sg->flags |= SMP_SEG_LOADED;
@@ -457,17 +459,14 @@ smp_oc_sml_getobj(struct worker *wrk, struct objcore *oc)
 	return (o);
 }
 
-static void
+static void __match_proto__(objupdatemeta_f)
 smp_oc_objupdatemeta(struct worker *wrk, struct objcore *oc)
 {
-	struct object *o;
 	struct smp_seg *sg;
 	struct smp_object *so;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	o = smp_oc_sml_getobj(wrk, oc);
-	AN(o);
 
 	CAST_OBJ_NOTNULL(sg, oc->stobj->priv, SMP_SEG_MAGIC);
 	CHECK_OBJ_NOTNULL(sg->sc, SMP_SC_MAGIC);
diff --git a/bin/varnishtest/tests/p00005.vtc b/bin/varnishtest/tests/p00005.vtc
index 55bc6c9..de6f3c3 100644
--- a/bin/varnishtest/tests/p00005.vtc
+++ b/bin/varnishtest/tests/p00005.vtc
@@ -52,6 +52,8 @@ varnish v1 -vcl+backend {
 
 delay 5
 
+varnish v1 -expect n_object == 0
+
 logexpect l1 -v v1 -g vxid -q "Begin ~ bereq" {
     expect * 1002 Storage         "persistent s0"
 } -start



From phk at FreeBSD.org  Fri Feb  5 22:24:40 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 05 Feb 2016 23:24:40 +0100
Subject: [master] d297266 Remove unused typedef
Message-ID: <E1aRonw-0008JK-Uh@project.varnish-software.com>


commit d297266422fe31b7a5f855c8b458c9e909711fde
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 5 21:31:18 2016 +0000

    Remove unused typedef

diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index 32b7e47..c3d697d 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -60,7 +60,6 @@ struct storage {
 typedef void storage_init_f(struct stevedore *, int ac, char * const *av);
 typedef void storage_open_f(const struct stevedore *);
 typedef struct storage *storage_alloc_f(const struct stevedore *, size_t size);
-typedef void storage_trim_f(struct storage *, size_t size, int move_ok);
 typedef void storage_free_f(struct storage *);
 typedef int storage_allocobj_f(struct worker *, const struct stevedore *,
     struct objcore *, unsigned ltot, int really);



From phk at FreeBSD.org  Fri Feb  5 22:24:41 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 05 Feb 2016 23:24:41 +0100
Subject: [master] 658932a Rename the stevedore_simple methods to make it clear
 that they are.
Message-ID: <E1aRonx-0008JQ-06@project.varnish-software.com>


commit 658932ab564d2e24bbb25bb6bd043c03d90f2f27
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 5 21:38:58 2016 +0000

    Rename the stevedore_simple methods to make it clear that they are.

diff --git a/bin/varnishd/storage/mgt_stevedore.c b/bin/varnishd/storage/mgt_stevedore.c
index 9852ea6..0f4e2ec 100644
--- a/bin/varnishd/storage/mgt_stevedore.c
+++ b/bin/varnishd/storage/mgt_stevedore.c
@@ -182,7 +182,6 @@ STV_Config(const char *spec)
 	else if (ac != 0)
 		ARGV_ERR("(-s%s) too many arguments\n", stv->name);
 
-	AN(stv->alloc);
 	AN(stv->allocobj);
 	AN(stv->methods);
 
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index c3d697d..fee1206 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -84,8 +84,8 @@ struct stevedore {
 	unsigned		transient;
 	storage_init_f		*init;		/* called by mgt process */
 	storage_open_f		*open;		/* called by cache process */
-	storage_alloc_f		*alloc;		/* --//-- only if SML */
-	storage_free_f		*free;		/* --//-- only if SML */
+	storage_alloc_f		*sml_alloc;	/* --//-- only if SML */
+	storage_free_f		*sml_free;	/* --//-- only if SML */
 	storage_close_f		*close;		/* --//-- */
 	storage_allocobj_f	*allocobj;	/* --//-- */
 	storage_signal_close_f	*signal_close;	/* --//-- */
diff --git a/bin/varnishd/storage/storage_file.c b/bin/varnishd/storage/storage_file.c
index 5985f91..bcd5afa 100644
--- a/bin/varnishd/storage/storage_file.c
+++ b/bin/varnishd/storage/storage_file.c
@@ -475,8 +475,8 @@ const struct stevedore smf_stevedore = {
 	.name		=	"file",
 	.init		=	smf_init,
 	.open		=	smf_open,
-	.alloc		=	smf_alloc,
-	.free		=	smf_free,
+	.sml_alloc	=	smf_alloc,
+	.sml_free	=	smf_free,
 	.allocobj	=	SML_allocobj,
 	.methods	=	&SML_methods,
 };
diff --git a/bin/varnishd/storage/storage_malloc.c b/bin/varnishd/storage/storage_malloc.c
index 2aacd41..19a8ac4 100644
--- a/bin/varnishd/storage/storage_malloc.c
+++ b/bin/varnishd/storage/storage_malloc.c
@@ -220,8 +220,8 @@ const struct stevedore sma_stevedore = {
 	.name		=	"malloc",
 	.init		=	sma_init,
 	.open		=	sma_open,
-	.alloc		=	sma_alloc,
-	.free		=	sma_free,
+	.sml_alloc	=	sma_alloc,
+	.sml_free	=	sma_free,
 	.allocobj	=	SML_allocobj,
 	.methods	=	&SML_methods,
 	.var_free_space =	sma_free_space,
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index f405755..fed7243 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -537,7 +537,7 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv,
 		}
 		st = smp_allocx(stv, ltot, ltot, &so, &objidx, &sg);
 		if (st != NULL && st->space < ltot) {
-			stv->free(st);		// NOP
+			stv->sml_free(st);		// NOP
 			st = NULL;
 		}
 		if (st != NULL)
@@ -588,18 +588,18 @@ smp_alloc(const struct stevedore *st, size_t size)
 /*--------------------------------------------------------------------*/
 
 const struct stevedore smp_stevedore = {
-	.magic	=	STEVEDORE_MAGIC,
-	.name	=	"deprecated_persistent",
-	.init	=	smp_mgt_init,
-	.open	=	smp_open,
-	.close	=	smp_close,
-	.alloc	=	smp_alloc,
-	.allocobj =	smp_allocobj,
-	.free	=	NULL,
-	.signal_close = smp_signal_close,
-	.baninfo =	smp_baninfo,
-	.banexport =	smp_banexport,
-	.methods =	&smp_oc_realmethods,
+	.magic		= STEVEDORE_MAGIC,
+	.name		= "deprecated_persistent",
+	.init		= smp_mgt_init,
+	.open		= smp_open,
+	.close		= smp_close,
+	.sml_alloc	= smp_alloc,
+	.allocobj	= smp_allocobj,
+	.sml_free	= NULL,
+	.signal_close	= smp_signal_close,
+	.baninfo	= smp_baninfo,
+	.banexport	= smp_banexport,
+	.methods	= &smp_oc_realmethods,
 };
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 3fdd242..b588798 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -55,8 +55,8 @@ sml_stv_alloc(const struct stevedore *stv, size_t size)
 
 	for (;;) {
 		/* try to allocate from it */
-		AN(stv->alloc);
-		st = stv->alloc(stv, size);
+		AN(stv->sml_alloc);
+		st = stv->sml_alloc(stv, size);
 		if (st != NULL)
 			break;
 
@@ -75,8 +75,8 @@ sml_stv_free(const struct stevedore *stv, struct storage *st)
 
 	CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
 	CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
-	if (stv->free != NULL)
-		stv->free(st);
+	if (stv->sml_free != NULL)
+		stv->sml_free(st);
 }
 
 /*--------------------------------------------------------------------
@@ -132,9 +132,9 @@ SML_allocobj(struct worker *wrk, const struct stevedore *stv,
 				return (0);
 			really--;
 		}
-		st = stv->alloc(stv, ltot);
+		st = stv->sml_alloc(stv, ltot);
 		if (st != NULL && st->space < ltot) {
-			stv->free(st);
+			stv->sml_free(st);
 			st = NULL;
 		}
 		if (st != NULL)
@@ -323,7 +323,7 @@ objallocwithnuke(struct worker *wrk, const struct stevedore *stv, size_t size)
 
 	for (fail = 0; fail <= cache_param->nuke_limit; fail++) {
 		/* try to allocate from it */
-		AN(stv->alloc);
+		AN(stv->sml_alloc);
 		st = sml_stv_alloc(stv, size);
 		if (st != NULL)
 			break;
@@ -407,7 +407,7 @@ sml_trimstore(struct worker *wrk, struct objcore *oc)
 	stv = oc->stobj->stevedore;
 	CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
 
-	if (stv->free == NULL)
+	if (stv->sml_free == NULL)
 		return;
 
 	o = sml_getobj(wrk, oc);



From phk at FreeBSD.org  Fri Feb  5 22:24:41 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 05 Feb 2016 23:24:41 +0100
Subject: [master] 6ed66f5 Make it possible for persistent to avoid LRU.
Message-ID: <E1aRonx-0008Ja-1f@project.varnish-software.com>


commit 6ed66f50d7753204e2f3b9f86a2abf3adbef548e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 5 22:06:42 2016 +0000

    Make it possible for persistent to avoid LRU.

diff --git a/bin/varnishd/cache/cache_obj.h b/bin/varnishd/cache/cache_obj.h
index e61324e..830efce 100644
--- a/bin/varnishd/cache/cache_obj.h
+++ b/bin/varnishd/cache/cache_obj.h
@@ -32,7 +32,6 @@
 
 typedef void objupdatemeta_f(struct worker *, struct objcore *);
 typedef void objfree_f(struct worker *, struct objcore *);
-typedef struct lru *objgetlru_f(const struct objcore *);
 
 /* This method is only used by SML (...to get to persistent) */
 typedef struct object *sml_getobj_f(struct worker *, struct objcore *);
@@ -53,7 +52,6 @@ typedef void objtouch_f(struct worker *, struct objcore *, double now);
 
 struct obj_methods {
 	objfree_f	*objfree;
-	objgetlru_f	*objgetlru;
 	objupdatemeta_f	*objupdatemeta;
 
 	sml_getobj_f	*sml_getobj;
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index fcb8461..660f511 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -128,12 +128,10 @@ STV_open(void)
 	struct stevedore *stv;
 
 	VTAILQ_FOREACH(stv, &stv_stevedores, list) {
-		stv->lru = LRU_Alloc();
 		if (stv->open != NULL)
 			stv->open(stv);
 	}
 	stv = stv_transient;
-	stv->lru = LRU_Alloc();
 	if (stv->open != NULL)
 		stv->open(stv);
 	stv_next = VTAILQ_FIRST(&stv_stevedores);
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index fee1206..6696b01 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -58,7 +58,7 @@ struct storage {
 /* Prototypes --------------------------------------------------------*/
 
 typedef void storage_init_f(struct stevedore *, int ac, char * const *av);
-typedef void storage_open_f(const struct stevedore *);
+typedef void storage_open_f(struct stevedore *);
 typedef struct storage *storage_alloc_f(const struct stevedore *, size_t size);
 typedef void storage_free_f(struct storage *);
 typedef int storage_allocobj_f(struct worker *, const struct stevedore *,
diff --git a/bin/varnishd/storage/storage_file.c b/bin/varnishd/storage/storage_file.c
index bcd5afa..97f8403 100644
--- a/bin/varnishd/storage/storage_file.c
+++ b/bin/varnishd/storage/storage_file.c
@@ -385,14 +385,15 @@ smf_open_chunk(struct smf_sc *sc, off_t sz, off_t off, off_t *fail, off_t *sum)
 	smf_open_chunk(sc, sz - h, off + h, fail, sum);
 }
 
-static void
-smf_open(const struct stevedore *st)
+static void __match_proto__(storage_open_f)
+smf_open(struct stevedore *st)
 {
 	struct smf_sc *sc;
 	off_t fail = 1 << 30;	/* XXX: where is OFF_T_MAX ? */
 	off_t sum = 0;
 
 	ASSERT_CLI();
+	st->lru = LRU_Alloc();
 	if (lck_smf == NULL)
 		lck_smf = Lck_CreateClass("smf");
 	CAST_OBJ_NOTNULL(sc, st->priv, SMF_SC_MAGIC);
diff --git a/bin/varnishd/storage/storage_lru.c b/bin/varnishd/storage/storage_lru.c
index 54b8619..987a0f1 100644
--- a/bin/varnishd/storage/storage_lru.c
+++ b/bin/varnishd/storage/storage_lru.c
@@ -50,11 +50,7 @@ lru_get(const struct objcore *oc)
 {
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(oc->stobj->stevedore, STEVEDORE_MAGIC);
-	AN(oc->stobj->stevedore->methods);
-	const struct obj_methods *m = oc->stobj->stevedore->methods;
-
-	if(m->objgetlru != NULL)
-		return (m->objgetlru(oc));
+	CHECK_OBJ_NOTNULL(oc->stobj->stevedore->lru, LRU_MAGIC);
 	return (oc->stobj->stevedore->lru);
 }
 
diff --git a/bin/varnishd/storage/storage_malloc.c b/bin/varnishd/storage/storage_malloc.c
index 19a8ac4..7c7efd7 100644
--- a/bin/varnishd/storage/storage_malloc.c
+++ b/bin/varnishd/storage/storage_malloc.c
@@ -198,12 +198,13 @@ sma_init(struct stevedore *parent, int ac, char * const *av)
 	sc->sma_max = u;
 }
 
-static void
-sma_open(const struct stevedore *st)
+static void __match_proto__(storage_open_f)
+sma_open(struct stevedore *st)
 {
 	struct sma_sc *sma_sc;
 
 	ASSERT_CLI();
+	st->lru = LRU_Alloc();
 	if (lck_sma == NULL)
 		lck_sma = Lck_CreateClass("sma");
 	CAST_OBJ_NOTNULL(sma_sc, st->priv, SMA_SC_MAGIC);
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index fed7243..20a4aec 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -234,8 +234,7 @@ smp_open_segs(struct smp_sc *sc, struct smp_signspace *spc)
 	for(; ss <= se; ss++) {
 		ALLOC_OBJ(sg, SMP_SEG_MAGIC);
 		AN(sg);
-		sg->lru = LRU_Alloc();
-		AN(sg->lru);
+		VTAILQ_INIT(&sg->objcores);
 		sg->p = *ss;
 
 		sg->flags |= SMP_SEG_MUSTLOAD;
@@ -320,8 +319,8 @@ smp_thread(struct worker *wrk, void *priv)
  * Open a silo in the worker process
  */
 
-static void
-smp_open(const struct stevedore *st)
+static void __match_proto__(storage_open_f)
+smp_open(struct stevedore *st)
 {
 	struct smp_sc	*sc;
 
@@ -569,6 +568,7 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv,
 
 	smp_init_oc(oc, sg, objidx);
 
+	VTAILQ_INSERT_TAIL(&sg->objcores, oc, lru_list);
 	Lck_Unlock(&sc->mtx);
 	return (1);
 }
@@ -689,7 +689,6 @@ SMP_Init(void)
 	smp_oc_realmethods.sml_getobj = smp_oc_methods.sml_getobj;
 	smp_oc_realmethods.objupdatemeta = smp_oc_methods.objupdatemeta;
 	smp_oc_realmethods.objfree = smp_oc_methods.objfree;
-	smp_oc_realmethods.objgetlru = smp_oc_methods.objgetlru;
 	smp_oc_realmethods.objtouch = NULL;
 }
 
diff --git a/bin/varnishd/storage/storage_persistent.h b/bin/varnishd/storage/storage_persistent.h
index b20ef02..4a28ba6 100644
--- a/bin/varnishd/storage/storage_persistent.h
+++ b/bin/varnishd/storage/storage_persistent.h
@@ -193,7 +193,7 @@ struct smp_seg {
 #define SMP_SEG_MAGIC		0x45c61895
 
 	struct smp_sc		*sc;
-	struct lru		*lru;
+	VTAILQ_HEAD(,objcore)	objcores;
 
 	VTAILQ_ENTRY(smp_seg)	list;		/* on smp_sc.smp_segments */
 
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index deb1340..74b74fe 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -106,7 +106,7 @@ smp_save_segs(struct smp_sc *sc)
 		if (sg == sc->cur_seg)
 			continue;
 		VTAILQ_REMOVE(&sc->segments, sg, list);
-		LRU_Free(&sg->lru);
+		AN(VTAILQ_EMPTY(&sg->objcores));
 		FREE_OBJ(sg);
 	}
 	smp_save_seg(sc, &sc->seg1);
@@ -163,6 +163,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		oc->flags &= ~OC_F_BUSY;
 		oc->stobj->stevedore = sc->parent;
 		smp_init_oc(oc, sg, no);
+		VTAILQ_INSERT_TAIL(&sg->objcores, oc, lru_list);
 		oc->stobj->priv2 |= NEED_FIXUP;
 		oc->ban = BAN_RefBan(oc, so->ban);
 		oc->exp = so->exp;
@@ -170,9 +171,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		oc->refcnt++;
 		HSH_Insert(wrk, so->hash, oc);
 		EXP_Insert(wrk, oc);
-		AN(isnan(oc->last_lru));
 		HSH_DerefBoc(wrk, oc);	// XXX Keep it an stream resurrection?
-		AZ(isnan(oc->last_lru));
 		(void)HSH_DerefObjCore(wrk, &oc);
 	}
 	Pool_Sumstat(wrk);
@@ -224,11 +223,9 @@ smp_new_seg(struct smp_sc *sc)
 
 	ALLOC_OBJ(sg, SMP_SEG_MAGIC);
 	if (sg == NULL)
-		/* Failed allocation */
 		return;
 	*sg = tmpsg;
-	sg->lru = LRU_Alloc();
-	AN(sg->lru);
+	VTAILQ_INIT(&sg->objcores);
 
 	sg->p.offset = IRNUP(sc, sg->p.offset);
 	sg->p.length -= sg->p.offset - tmpsg.p.offset;
@@ -277,7 +274,7 @@ smp_close_seg(struct smp_sc *sc, struct smp_seg *sg)
 		assert(sg->p.offset >= sc->ident->stuff[SMP_SPC_STUFF]);
 		assert(sg->p.offset < sc->mediasize);
 		sc->free_offset = sg->p.offset;
-		LRU_Free(&sg->lru);
+		AN(VTAILQ_EMPTY(&sg->objcores));
 		FREE_OBJ(sg);
 		return;
 	}
@@ -509,31 +506,16 @@ smp_oc_objfree(struct worker *wrk, struct objcore *oc)
 		sg->nfixed--;
 		wrk->stats->n_object--;
 	}
-	AZ(isnan(oc->last_lru));
-	LRU_Remove(oc);
+	VTAILQ_REMOVE(&sg->objcores, oc, lru_list);
 
 	Lck_Unlock(&sg->sc->mtx);
 	memset(oc->stobj, 0, sizeof oc->stobj);
 }
 
-/*--------------------------------------------------------------------
- * Find the per-segment lru list for this object
- */
-
-static struct lru * __match_proto__(objgetlru_f)
-smp_oc_objgetlru(const struct objcore *oc)
-{
-	struct smp_seg *sg;
-
-	CAST_OBJ_NOTNULL(sg, oc->stobj->priv, SMP_SEG_MAGIC);
-	return (sg->lru);
-}
-
 const struct obj_methods smp_oc_methods = {
 	.sml_getobj =		smp_oc_sml_getobj,
 	.objupdatemeta =	smp_oc_objupdatemeta,
 	.objfree =		smp_oc_objfree,
-	.objgetlru =		smp_oc_objgetlru,
 };
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index b588798..42112e7 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -206,7 +206,7 @@ sml_objfree(struct worker *wrk, struct objcore *oc)
 	CAST_OBJ_NOTNULL(o, oc->stobj->priv, OBJECT_MAGIC);
 	o->magic = 0;
 
-	if (oc->boc == NULL)
+	if (oc->boc == NULL && oc->stobj->stevedore->lru != NULL)
 		LRU_Remove(oc);
 
 	sml_stv_free(oc->stobj->stevedore, o->objstore);
@@ -329,6 +329,8 @@ objallocwithnuke(struct worker *wrk, const struct stevedore *stv, size_t size)
 			break;
 
 		/* no luck; try to free some space and keep trying */
+		if (stv->lru == NULL)
+			break;
 		if (fail < cache_param->nuke_limit &&
 		    !LRU_NukeOne(wrk, stv->lru))
 			break;
@@ -457,18 +459,19 @@ sml_stable(struct worker *wrk, struct objcore *oc, struct boc *boc)
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(boc, BOC_MAGIC);
+	stv = oc->stobj->stevedore;
+	CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
 
 	if (boc->stevedore_priv != NULL) {
 		/* Free any leftovers from Trim */
 		CAST_OBJ_NOTNULL(st, boc->stevedore_priv, STORAGE_MAGIC);
 		boc->stevedore_priv = 0;
 		CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
-		stv = oc->stobj->stevedore;
-		CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
 		sml_stv_free(stv, st);
 	}
 
-	LRU_Add(oc, wrk->lastused);	// approx timestamp is OK
+	if (stv->lru != NULL)
+		LRU_Add(oc, wrk->lastused);	// approx timestamp is OK
 }
 
 static void * __match_proto__(objgetattr_f)



From phk at FreeBSD.org  Fri Feb  5 22:24:41 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 05 Feb 2016 23:24:41 +0100
Subject: [master] 4b52825 Move the storage_simples "getobj" helper method from
 oc to stv.
Message-ID: <E1aRonx-0008Jk-2z@project.varnish-software.com>


commit 4b528257611c6568cc70d85c58805d12fb6b65d5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 5 22:17:49 2016 +0000

    Move the storage_simples "getobj" helper method from oc to stv.

diff --git a/bin/varnishd/cache/cache_obj.h b/bin/varnishd/cache/cache_obj.h
index 830efce..41e3ae5 100644
--- a/bin/varnishd/cache/cache_obj.h
+++ b/bin/varnishd/cache/cache_obj.h
@@ -34,7 +34,6 @@ typedef void objupdatemeta_f(struct worker *, struct objcore *);
 typedef void objfree_f(struct worker *, struct objcore *);
 
 /* This method is only used by SML (...to get to persistent) */
-typedef struct object *sml_getobj_f(struct worker *, struct objcore *);
 
 typedef int objiterator_f(struct worker *, struct objcore *,
     void *priv, objiterate_f *func);
@@ -54,8 +53,6 @@ struct obj_methods {
 	objfree_f	*objfree;
 	objupdatemeta_f	*objupdatemeta;
 
-	sml_getobj_f	*sml_getobj;
-
 	objiterator_f	*objiterator;
 	objgetspace_f	*objgetspace;
 	objextend_f	*objextend;
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index 6696b01..f492de2 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -59,8 +59,6 @@ struct storage {
 
 typedef void storage_init_f(struct stevedore *, int ac, char * const *av);
 typedef void storage_open_f(struct stevedore *);
-typedef struct storage *storage_alloc_f(const struct stevedore *, size_t size);
-typedef void storage_free_f(struct storage *);
 typedef int storage_allocobj_f(struct worker *, const struct stevedore *,
     struct objcore *, unsigned ltot, int really);
 typedef void storage_close_f(const struct stevedore *);
@@ -70,6 +68,10 @@ typedef int storage_baninfo_f(const struct stevedore *, enum baninfo event,
 typedef void storage_banexport_f(const struct stevedore *, const uint8_t *bans,
     unsigned len);
 
+typedef struct object *storage_getobj_f(struct worker *, struct objcore *);
+typedef struct storage *storage_alloc_f(const struct stevedore *, size_t size);
+typedef void storage_free_f(struct storage *);
+
 /* Prototypes for VCL variable responders */
 #define VRTSTVTYPE(ct) typedef ct storage_var_##ct(const struct stevedore *);
 #include "tbl/vrt_stv_var.h"
@@ -86,6 +88,7 @@ struct stevedore {
 	storage_open_f		*open;		/* called by cache process */
 	storage_alloc_f		*sml_alloc;	/* --//-- only if SML */
 	storage_free_f		*sml_free;	/* --//-- only if SML */
+	storage_getobj_f	*sml_getobj;	/* --//-- only if SML */
 	storage_close_f		*close;		/* --//-- */
 	storage_allocobj_f	*allocobj;	/* --//-- */
 	storage_signal_close_f	*signal_close;	/* --//-- */
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index 20a4aec..ce67b35 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -593,13 +593,15 @@ const struct stevedore smp_stevedore = {
 	.init		= smp_mgt_init,
 	.open		= smp_open,
 	.close		= smp_close,
-	.sml_alloc	= smp_alloc,
 	.allocobj	= smp_allocobj,
-	.sml_free	= NULL,
 	.signal_close	= smp_signal_close,
 	.baninfo	= smp_baninfo,
 	.banexport	= smp_banexport,
 	.methods	= &smp_oc_realmethods,
+
+	.sml_alloc	= smp_alloc,
+	.sml_free	= NULL,
+	.sml_getobj	= smp_sml_getobj,
 };
 
 /*--------------------------------------------------------------------
@@ -686,7 +688,6 @@ SMP_Init(void)
 	lck_smp = Lck_CreateClass("smp");
 	CLI_AddFuncs(debug_cmds);
 	smp_oc_realmethods = SML_methods;
-	smp_oc_realmethods.sml_getobj = smp_oc_methods.sml_getobj;
 	smp_oc_realmethods.objupdatemeta = smp_oc_methods.objupdatemeta;
 	smp_oc_realmethods.objfree = smp_oc_methods.objfree;
 	smp_oc_realmethods.objtouch = NULL;
diff --git a/bin/varnishd/storage/storage_persistent.h b/bin/varnishd/storage/storage_persistent.h
index 4a28ba6..410bc48 100644
--- a/bin/varnishd/storage/storage_persistent.h
+++ b/bin/varnishd/storage/storage_persistent.h
@@ -308,6 +308,7 @@ void smp_close_seg(struct smp_sc *sc, struct smp_seg *sg);
 void smp_init_oc(struct objcore *oc, struct smp_seg *sg, unsigned objidx);
 void smp_save_segs(struct smp_sc *sc);
 extern const struct obj_methods smp_oc_methods;
+storage_getobj_f smp_sml_getobj;
 
 /* storage_persistent_subr.c */
 
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 74b74fe..87d81e8 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -388,8 +388,8 @@ smp_loaded_st(const struct smp_sc *sc, const struct smp_seg *sg,
  * objcore methods for persistent objects
  */
 
-static struct object *
-smp_oc_sml_getobj(struct worker *wrk, struct objcore *oc)
+struct object *
+smp_sml_getobj(struct worker *wrk, struct objcore *oc)
 {
 	struct object *o;
 	struct smp_seg *sg;
@@ -402,9 +402,6 @@ smp_oc_sml_getobj(struct worker *wrk, struct objcore *oc)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	AN(oc->stobj->stevedore);
 
-	/* Some calls are direct, but they should match anyway */
-	assert(oc->stobj->stevedore->methods->sml_getobj == smp_oc_sml_getobj);
-
 	CAST_OBJ_NOTNULL(sg, oc->stobj->priv, SMP_SEG_MAGIC);
 	so = smp_find_so(sg, oc->stobj->priv2);
 
@@ -513,7 +510,6 @@ smp_oc_objfree(struct worker *wrk, struct objcore *oc)
 }
 
 const struct obj_methods smp_oc_methods = {
-	.sml_getobj =		smp_oc_sml_getobj,
 	.objupdatemeta =	smp_oc_objupdatemeta,
 	.objfree =		smp_oc_objfree,
 };
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 42112e7..40e3b31 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -156,14 +156,15 @@ SML_allocobj(struct worker *wrk, const struct stevedore *stv,
 static struct object *
 sml_getobj(struct worker *wrk, struct objcore *oc)
 {
-	const struct obj_methods *m;
+	const struct stevedore *stv;
 	struct object *o;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	m = oc->stobj->stevedore->methods;
-	if (m->sml_getobj != NULL)
-		return (m->sml_getobj(wrk, oc));
+	stv = oc->stobj->stevedore;
+	CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC);
+	if (stv->sml_getobj != NULL)
+		return (stv->sml_getobj(wrk, oc));
 	if (oc->stobj->priv == NULL)
 		return (NULL);
 	CAST_OBJ_NOTNULL(o, oc->stobj->priv, OBJECT_MAGIC);



From phk at FreeBSD.org  Fri Feb  5 22:24:41 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 05 Feb 2016 23:24:41 +0100
Subject: [master] 13c713a Drop pointless template structure.
Message-ID: <E1aRonx-0008Ju-4L@project.varnish-software.com>


commit 13c713a0fb2ba303c0a49024a9b03df97b3932c8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 5 22:23:46 2016 +0000

    Drop pointless template structure.

diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index ce67b35..61f566d 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -688,9 +688,9 @@ SMP_Init(void)
 	lck_smp = Lck_CreateClass("smp");
 	CLI_AddFuncs(debug_cmds);
 	smp_oc_realmethods = SML_methods;
-	smp_oc_realmethods.objupdatemeta = smp_oc_methods.objupdatemeta;
-	smp_oc_realmethods.objfree = smp_oc_methods.objfree;
 	smp_oc_realmethods.objtouch = NULL;
+	smp_oc_realmethods.objupdatemeta = smp_oc_objupdatemeta;
+	smp_oc_realmethods.objfree = smp_oc_objfree;
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishd/storage/storage_persistent.h b/bin/varnishd/storage/storage_persistent.h
index 410bc48..0716458 100644
--- a/bin/varnishd/storage/storage_persistent.h
+++ b/bin/varnishd/storage/storage_persistent.h
@@ -307,8 +307,9 @@ void smp_new_seg(struct smp_sc *sc);
 void smp_close_seg(struct smp_sc *sc, struct smp_seg *sg);
 void smp_init_oc(struct objcore *oc, struct smp_seg *sg, unsigned objidx);
 void smp_save_segs(struct smp_sc *sc);
-extern const struct obj_methods smp_oc_methods;
 storage_getobj_f smp_sml_getobj;
+void smp_oc_objupdatemeta(struct worker *, struct objcore *);
+void smp_oc_objfree(struct worker *, struct objcore *);
 
 /* storage_persistent_subr.c */
 
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 87d81e8..f46787f 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -453,7 +453,7 @@ smp_sml_getobj(struct worker *wrk, struct objcore *oc)
 	return (o);
 }
 
-static void __match_proto__(objupdatemeta_f)
+void __match_proto__(objupdatemeta_f)
 smp_oc_objupdatemeta(struct worker *wrk, struct objcore *oc)
 {
 	struct smp_seg *sg;
@@ -478,7 +478,7 @@ smp_oc_objupdatemeta(struct worker *wrk, struct objcore *oc)
 	}
 }
 
-static void __match_proto__(objfree_f)
+void __match_proto__(objfree_f)
 smp_oc_objfree(struct worker *wrk, struct objcore *oc)
 {
 	struct smp_seg *sg;
@@ -509,11 +509,6 @@ smp_oc_objfree(struct worker *wrk, struct objcore *oc)
 	memset(oc->stobj, 0, sizeof oc->stobj);
 }
 
-const struct obj_methods smp_oc_methods = {
-	.objupdatemeta =	smp_oc_objupdatemeta,
-	.objfree =		smp_oc_objfree,
-};
-
 /*--------------------------------------------------------------------*/
 
 void



From phk at FreeBSD.org  Sat Feb  6 10:38:59 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sat, 06 Feb 2016 11:38:59 +0100
Subject: [master] 246428b A lot of commenting and some minor cleanups
Message-ID: <E1aS0GZ-0005fv-K8@project.varnish-software.com>


commit 246428b5ae679a0f3825cff91e84603db288eaa7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sat Feb 6 10:38:29 2016 +0000

    A lot of commenting and some minor cleanups

diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 76f67df..b63a54a 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -25,18 +25,60 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
- * Primary API:
- *	ObjNew		Associate stevedore with oc
- *	ObjGetSpace	Add space
- *	ObjExtend	Commit space
- *	ObjDone		Object completed
- *	ObjGetLen	Len of committed space
- *	ObjIterate	Iterate over committed space
- *	ObjReserveAttr	Attr will be set later
- *	ObjSetAttr	Set attr now
- *	ObjGetAttr	Get attr no
- *	ObjRelease	Done with attr ptr
- *	ObjTouch	Object was used
+ * Lifetime of an objcore:
+ *	phase 0	- nonexistent
+ *	phase 1	- created, but no stevedore associated
+ *	phase 2	- stevedore associated, being filled out
+ *	phase 3	- stable, no changes happening
+ *	phase 4	- unavailable, being dismantled
+ *	phase 5	- stevedore disassociated
+ *	phase 6	- nonexistent
+ *
+ * 0->1	ObjNew()	creates objcore
+ *
+ * 1->2	STV_NewObject()	associates a stevedore
+ *
+ * 2	ObjSetState()	sets state
+ * 2	ObjWaitState()	waits for particular state
+ *			INVALID->REQ_DONE->STREAM->FINISHED->FAILED
+ *
+ * 2	ObjGetSpace()	allocates space
+ * 2	ObjExtend()	commits content
+ * 2	ObjWaitExtend()	waits for content - used to implement ObjIterate())
+ * 2	ObjTrimStore()	signals end of content addition
+ *
+ * 2	ObjSetAttr()
+ * 2	  ObjCopyAttr()
+ * 2	  ObjSetFlag()
+ * 2	  ObjSetDouble()
+ * 2	  ObjSetU32()
+ * 2	  ObjSetU64()
+ *
+ * 2->3	ObjStable()	Will no longer be modified
+ *
+ * 23	ObjHasAttr()
+ * 23	ObjGetAttr()
+ * 23	  ObjCheckFlag()
+ * 23	  ObjGetDouble()
+ * 23	  ObjGetU32()
+ * 23	  ObjGetU64()
+ * 23	  ObjGetLen()
+ * 23	  ObjGetXID()
+ *
+ * 23	ObjIterate()	... over body
+ *
+ * 23	ObjTouch()	Signal to LRU(-like) facilities
+ *
+ * 23	ObjUpdateMeta()	ban/ttl/grace/keep changed
+ *
+ * 3->4	ObjSnipe()	kill if not in use
+ * 3->4	ObjKill()	make unavailable
+ *
+ * 234	ObjSlim()	Release body storage (but retain attribute storage)
+ *
+ * 4->5	ObjFreeObj()	disassociates stevedore
+ *
+ * 5->6 FREE_OBJ()	...in HSH_DerefObjCore()
  */
 
 #include "config.h"
diff --git a/bin/varnishd/storage/mgt_stevedore.c b/bin/varnishd/storage/mgt_stevedore.c
index 0f4e2ec..920a4a2 100644
--- a/bin/varnishd/storage/mgt_stevedore.c
+++ b/bin/varnishd/storage/mgt_stevedore.c
@@ -186,7 +186,6 @@ STV_Config(const char *spec)
 	AN(stv->methods);
 
 	if (!strcmp(stv->ident, TRANSIENT_STORAGE)) {
-		stv->transient = 1;
 		AZ(stv_transient);
 		stv_transient = stv;
 	} else {
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index f492de2..4f12d9c 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -68,9 +68,9 @@ typedef int storage_baninfo_f(const struct stevedore *, enum baninfo event,
 typedef void storage_banexport_f(const struct stevedore *, const uint8_t *bans,
     unsigned len);
 
-typedef struct object *storage_getobj_f(struct worker *, struct objcore *);
-typedef struct storage *storage_alloc_f(const struct stevedore *, size_t size);
-typedef void storage_free_f(struct storage *);
+typedef struct object *sml_getobj_f(struct worker *, struct objcore *);
+typedef struct storage *sml_alloc_f(const struct stevedore *, size_t size);
+typedef void sml_free_f(struct storage *);
 
 /* Prototypes for VCL variable responders */
 #define VRTSTVTYPE(ct) typedef ct storage_var_##ct(const struct stevedore *);
@@ -83,22 +83,28 @@ struct stevedore {
 	unsigned		magic;
 #define STEVEDORE_MAGIC		0x4baf43db
 	const char		*name;
-	unsigned		transient;
-	storage_init_f		*init;		/* called by mgt process */
-	storage_open_f		*open;		/* called by cache process */
-	storage_alloc_f		*sml_alloc;	/* --//-- only if SML */
-	storage_free_f		*sml_free;	/* --//-- only if SML */
-	storage_getobj_f	*sml_getobj;	/* --//-- only if SML */
-	storage_close_f		*close;		/* --//-- */
-	storage_allocobj_f	*allocobj;	/* --//-- */
-	storage_signal_close_f	*signal_close;	/* --//-- */
-	storage_baninfo_f	*baninfo;	/* --//-- */
-	storage_banexport_f	*banexport;	/* --//-- */
+
+	/* Called in MGT process */
+	storage_init_f		*init;
+
+	/* Called in cache process */
+	storage_open_f		*open;
+	storage_close_f		*close;
+	storage_allocobj_f	*allocobj;
+	storage_signal_close_f	*signal_close;
+	storage_baninfo_f	*baninfo;
+	storage_banexport_f	*banexport;
+
+	/* Only if SML is used */
+	sml_alloc_f		*sml_alloc;
+	sml_free_f		*sml_free;
+	sml_getobj_f		*sml_getobj;
 
 	const struct obj_methods
 				*methods;
 
-	struct lru		*lru;		/* For storage_lru.c */
+	/* Only if LRU is used */
+	struct lru		*lru;
 
 #define VRTSTVVAR(nm, vtype, ctype, dval) storage_var_##ctype *var_##nm;
 #include "tbl/vrt_stv_var.h"
diff --git a/bin/varnishd/storage/storage_file.c b/bin/varnishd/storage/storage_file.c
index 97f8403..7b85dc6 100644
--- a/bin/varnishd/storage/storage_file.c
+++ b/bin/varnishd/storage/storage_file.c
@@ -415,7 +415,7 @@ smf_open(struct stevedore *st)
 
 /*--------------------------------------------------------------------*/
 
-static struct storage *
+static struct storage * __match_proto__(sml_alloc_f)
 smf_alloc(const struct stevedore *st, size_t size)
 {
 	struct smf *smf;
@@ -451,7 +451,7 @@ smf_alloc(const struct stevedore *st, size_t size)
 
 /*--------------------------------------------------------------------*/
 
-static void __match_proto__(storage_free_f)
+static void __match_proto__(sml_free_f)
 smf_free(struct storage *s)
 {
 	struct smf *smf;
diff --git a/bin/varnishd/storage/storage_malloc.c b/bin/varnishd/storage/storage_malloc.c
index 7c7efd7..d132b1c 100644
--- a/bin/varnishd/storage/storage_malloc.c
+++ b/bin/varnishd/storage/storage_malloc.c
@@ -59,7 +59,7 @@ struct sma {
 
 static struct VSC_C_lck *lck_sma;
 
-static struct storage *
+static struct storage * __match_proto__(sml_alloc_f)
 sma_alloc(const struct stevedore *st, size_t size)
 {
 	struct sma_sc *sma_sc;
@@ -125,7 +125,7 @@ sma_alloc(const struct stevedore *st, size_t size)
 	return (&sma->s);
 }
 
-static void __match_proto__(storage_free_f)
+static void __match_proto__(sml_free_f)
 sma_free(struct storage *s)
 {
 	struct sma_sc *sma_sc;
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index 61f566d..14ee3a7 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -577,7 +577,7 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv,
  * Allocate a bite
  */
 
-static struct storage *
+static struct storage * __match_proto__(sml_alloc_f)
 smp_alloc(const struct stevedore *st, size_t size)
 {
 
diff --git a/bin/varnishd/storage/storage_persistent.h b/bin/varnishd/storage/storage_persistent.h
index 0716458..4ccb014 100644
--- a/bin/varnishd/storage/storage_persistent.h
+++ b/bin/varnishd/storage/storage_persistent.h
@@ -307,7 +307,7 @@ void smp_new_seg(struct smp_sc *sc);
 void smp_close_seg(struct smp_sc *sc, struct smp_seg *sg);
 void smp_init_oc(struct objcore *oc, struct smp_seg *sg, unsigned objidx);
 void smp_save_segs(struct smp_sc *sc);
-storage_getobj_f smp_sml_getobj;
+sml_getobj_f smp_sml_getobj;
 void smp_oc_objupdatemeta(struct worker *, struct objcore *);
 void smp_oc_objfree(struct worker *, struct objcore *);
 



From phk at FreeBSD.org  Sat Feb  6 12:00:56 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sat, 06 Feb 2016 13:00:56 +0100
Subject: [master] c0d6265 Eliminate HSH_Complete(),
 the boc->state takes care of it.
Message-ID: <E1aS1Xs-0005ER-Qr@project.varnish-software.com>


commit c0d62658c7dc535fb1b2db1ed69517b8ec833808
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sat Feb 6 12:00:37 2016 +0000

    Eliminate HSH_Complete(), the boc->state takes care of it.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 33c40ed..cfd17bb 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -413,7 +413,6 @@ struct objcore {
 	uint8_t			flags;
 #define OC_F_BUSY		(1<<1)
 #define OC_F_PASS		(1<<2)
-#define OC_F_INCOMPLETE		(1<<3)
 #define OC_F_ABANDON		(1<<4)
 #define OC_F_PRIVATE		(1<<5)
 #define OC_F_FAILED		(1<<6)
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 35a6dd6..2b2d200 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -946,7 +946,6 @@ vbf_fetch_thread(struct worker *wrk, void *priv)
 
 	if (bo->fetch_objcore->boc->state == BOS_FINISHED) {
 		AZ(bo->fetch_objcore->flags & OC_F_FAILED);
-		HSH_Complete(bo->fetch_objcore);
 		VSLb(bo->vsl, SLT_Length, "%ju",
 		    (uintmax_t)ObjGetLen(bo->wrk, bo->fetch_objcore));
 	}
diff --git a/bin/varnishd/cache/cache_gzip.c b/bin/varnishd/cache/cache_gzip.c
index 0b8d3a7..98523a0 100644
--- a/bin/varnishd/cache/cache_gzip.c
+++ b/bin/varnishd/cache/cache_gzip.c
@@ -307,8 +307,8 @@ VDP_gunzip(struct req *req, enum vdp_action act, void **priv,
 		http_Unset(req->resp, H_Content_Encoding);
 
 		req->resp_len = -1;
-		if (req->objcore->flags & OC_F_INCOMPLETE)
-			return (0);	/* No idea about length */
+		if (req->objcore->boc != NULL)
+			return (0);	/* No idea about length (yet) */
 
 		p = ObjGetAttr(req->wrk, req->objcore, OA_GZIPBITS, &dl);
 		if (p == NULL || dl != 32)
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 11714f9..d58e070 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -91,7 +91,7 @@ hsh_prealloc(struct worker *wrk)
 
 	if (wrk->nobjcore == NULL) {
 		wrk->nobjcore = ObjNew(wrk);
-		wrk->nobjcore->flags |= OC_F_BUSY | OC_F_INCOMPLETE;
+		wrk->nobjcore->flags |= OC_F_BUSY;
 	}
 	CHECK_OBJ_NOTNULL(wrk->nobjcore, OBJCORE_MAGIC);
 
@@ -118,7 +118,7 @@ HSH_Private(struct worker *wrk)
 	AN(oc);
 	oc->refcnt = 1;
 	oc->objhead = private_oh;
-	oc->flags |= OC_F_PRIVATE | OC_F_BUSY | OC_F_INCOMPLETE;
+	oc->flags |= OC_F_PRIVATE | OC_F_BUSY;
 	Lck_Lock(&private_oh->mtx);
 	VTAILQ_INSERT_TAIL(&private_oh->objcs, oc, list);
 	private_oh->refcnt++;
@@ -613,25 +613,6 @@ HSH_Fail(struct objcore *oc)
 
 	Lck_Lock(&oh->mtx);
 	oc->flags |= OC_F_FAILED;
-	oc->flags &= ~OC_F_INCOMPLETE;
-	Lck_Unlock(&oh->mtx);
-}
-
-/*---------------------------------------------------------------------
- * Remove the busyobj from an objcore
- */
-
-void
-HSH_Complete(struct objcore *oc)
-{
-	struct objhead *oh;
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	oh = oc->objhead;
-	CHECK_OBJ(oh, OBJHEAD_MAGIC);
-
-	Lck_Lock(&oh->mtx);
-	oc->flags &= ~OC_F_INCOMPLETE;
 	Lck_Unlock(&oh->mtx);
 }
 
@@ -725,10 +706,13 @@ HSH_RefBoc(const struct objcore *oc)
 	assert(oc->refcnt > 0);
 	boc = oc->boc;
 	CHECK_OBJ_ORNULL(boc, BOC_MAGIC);
-	if (boc != NULL && boc->state < BOS_FINISHED)
-		boc->refcount++;
-	else
-		boc = NULL;
+	if (boc != NULL) {
+		assert(boc->refcount > 0);
+		if (boc->state < BOS_FINISHED)
+			boc->refcount++;
+		else
+			boc = NULL;
+	}
 	Lck_Unlock(&oh->mtx);
 	return (boc);
 }
diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h
index 9a64916..b4b7bcd 100644
--- a/bin/varnishd/hash/hash_slinger.h
+++ b/bin/varnishd/hash/hash_slinger.h
@@ -104,7 +104,6 @@ struct objhead {
 
 void HSH_Fail(struct objcore *);
 void HSH_Unbusy(struct worker *, struct objcore *);
-void HSH_Complete(struct objcore *oc);
 void HSH_DeleteObjHead(struct worker *, struct objhead *oh);
 int HSH_DerefObjHead(struct worker *, struct objhead **poh);
 int HSH_DerefObjCore(struct worker *, struct objcore **ocp);



From phk at FreeBSD.org  Mon Feb  8 08:37:35 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 08 Feb 2016 09:37:35 +0100
Subject: [master] 431b502 Remove overzealous assert
Message-ID: <E1aShKB-0008Lp-MH@project.varnish-software.com>


commit 431b502b2fdb2269ccfed7aa82c13e2948b206c7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 08:37:23 2016 +0000

    Remove overzealous assert

diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index b63a54a..d0ae3db 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -440,8 +440,6 @@ ObjKill(struct objcore *oc)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
 
-	AZ(oc->flags & OC_F_DYING);	// XXX ?
-
 	Lck_Lock(&oc->objhead->mtx);
 	oc->flags |= OC_F_DYING;
 	Lck_Unlock(&oc->objhead->mtx);



From fgsch at lodoss.net  Mon Feb  8 09:24:40 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 08 Feb 2016 10:24:40 +0100
Subject: [master] 1790503 Missing include prompted by -std=c99
Message-ID: <E1aSi3k-0006kc-U3@project.varnish-software.com>


commit 179050374c0f92bac517288c226b514c42d6d2ac
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Feb 8 09:19:29 2016 +0000

    Missing include prompted by -std=c99

diff --git a/lib/libvarnishapi/vsl_query.c b/lib/libvarnishapi/vsl_query.c
index 1741c13..0159429 100644
--- a/lib/libvarnishapi/vsl_query.c
+++ b/lib/libvarnishapi/vsl_query.c
@@ -34,6 +34,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <strings.h>
 #include <unistd.h>
 
 #include "vdef.h"



From phk at FreeBSD.org  Mon Feb  8 09:49:16 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 08 Feb 2016 10:49:16 +0100
Subject: [master] 9450493 We have rather haphazardly been using "127.0.0.1" in
 test-cases,
 asuming that would for ever be the case. This fails on IPv6 only systems and
 FreeBSD jails where localhost is the "real" IP number.
Message-ID: <E1aSiRY-0000pS-1s@project.varnish-software.com>


commit 9450493d2e6a32c1034fe37f79912f720bb40070
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 09:46:51 2016 +0000

    We have rather haphazardly been using "127.0.0.1" in test-cases,
    asuming that would for ever be the case.  This fails on IPv6 only
    systems and FreeBSD jails where localhost is the "real" IP number.

diff --git a/bin/varnishtest/tests/b00031.vtc b/bin/varnishtest/tests/b00031.vtc
index 9dcb83e..485cebb 100644
--- a/bin/varnishtest/tests/b00031.vtc
+++ b/bin/varnishtest/tests/b00031.vtc
@@ -2,10 +2,10 @@ varnishtest "Test X-Forward-For headers"
 
 server s1 {
 	rxreq
-	expect req.http.X-Forwarded-For == "127.0.0.1"
+	expect req.http.X-Forwarded-For == "${localhost}"
 	txresp
 	rxreq
-	expect req.http.X-Forwarded-For == "1.2.3.4, 127.0.0.1"
+	expect req.http.X-Forwarded-For == "1.2.3.4, ${localhost}"
 	txresp
 } -start
 
diff --git a/bin/varnishtest/tests/c00005.vtc b/bin/varnishtest/tests/c00005.vtc
index 846b283..afa103c 100644
--- a/bin/varnishtest/tests/c00005.vtc
+++ b/bin/varnishtest/tests/c00005.vtc
@@ -11,7 +11,7 @@ server s1 {
 
 varnish v1 -arg "-p vsl_mask=+VCL_trace" -vcl+backend {
 	acl acl1 {
-		"localhost";
+		"${localhost}";
 	}
 
 	sub vcl_recv {
diff --git a/bin/varnishtest/tests/d00003.vtc b/bin/varnishtest/tests/d00003.vtc
index e6182d7..970ed2d 100644
--- a/bin/varnishtest/tests/d00003.vtc
+++ b/bin/varnishtest/tests/d00003.vtc
@@ -21,6 +21,7 @@ server s2 {
 } -start
 
 varnish v1 -vcl+backend {
+	import ${vmod_std};
 	import ${vmod_directors};
 
 	sub vcl_init {
@@ -41,7 +42,7 @@ varnish v1 -vcl+backend {
 		} else if (req.url == "/13") {
 			set req.backend_hint = h1.backend(req.http.Void + "" + req.url);
 		} else if (req.url == "/ip") {
-			set req.backend_hint = h1.backend(client.ip);
+			set req.backend_hint = h1.backend(std.ip("127.0.0.1", client.ip));
 		} else {
 			set req.backend_hint = h1.backend(req.url);
 		}
diff --git a/bin/varnishtest/tests/m00011.vtc b/bin/varnishtest/tests/m00011.vtc
index 930ed8e..45528c9 100644
--- a/bin/varnishtest/tests/m00011.vtc
+++ b/bin/varnishtest/tests/m00011.vtc
@@ -21,8 +21,8 @@ varnish v1 -vcl+backend {
 client c1 {
 	txreq
 	rxresp
-	expect resp.http.foo0 == "127.0.0.1"
-	expect resp.http.foo1 == "127.0.0.1"
+	expect resp.http.foo0 == "${localhost}"
+	expect resp.http.foo1 == "${localhost}"
 	expect resp.http.foo2 == "127.0.0.2"
 	expect resp.http.foo3 == "1.2.3.5"
 	expect resp.http.foo4 == "2001:db8::"
diff --git a/bin/varnishtest/tests/o00001.vtc b/bin/varnishtest/tests/o00001.vtc
index 3d4e2f6..ecf025a 100644
--- a/bin/varnishtest/tests/o00001.vtc
+++ b/bin/varnishtest/tests/o00001.vtc
@@ -4,7 +4,7 @@ server s1 {
 	# The server address is part of the hash-key
 	# so we need three responses
 	rxreq
-	expect req.http.x-forwarded-for == "127.0.0.1"
+	expect req.http.x-forwarded-for == "${localhost}"
 	txresp -hdr "Obj: 1"
 	rxreq
 	expect req.http.x-forwarded-for == "1.2.3.4"
@@ -49,7 +49,7 @@ client c1 {
 	expect resp.status == 200
 	expect resp.http.si == "${v1_addr}"
 	expect resp.http.sp == "${v1_port}"
-	expect resp.http.ci == "127.0.0.1"
+	expect resp.http.ci == "${localhost}"
 } -run
 delay .1
 
@@ -71,7 +71,7 @@ client c1 {
 	expect resp.status == 200
 	expect resp.http.si == "${v1_addr}"
 	expect resp.http.sp == "${v1_port}"
-	expect resp.http.ci == "127.0.0.1"
+	expect resp.http.ci == "${localhost}"
 } -run
 delay .1
 
@@ -84,7 +84,7 @@ client c1 {
 	expect resp.status == 200
 	expect resp.http.si == "${v1_addr}"
 	expect resp.http.sp == "${v1_port}"
-	expect resp.http.ci == "127.0.0.1"
+	expect resp.http.ci == "${localhost}"
 } -run
 delay .1
 
@@ -98,7 +98,7 @@ client c1 {
 	expect resp.status == 200
 	expect resp.http.si == "${v1_addr}"
 	expect resp.http.sp == "${v1_port}"
-	expect resp.http.ci == "127.0.0.1"
+	expect resp.http.ci == "${localhost}"
 } -run
 delay .1
 
@@ -116,7 +116,7 @@ client c1 {
 	expect resp.http.fc == false
 	expect resp.http.si == "${v1_addr}"
 	expect resp.http.sp == "${v1_port}"
-	expect resp.http.ci == "127.0.0.1"
+	expect resp.http.ci == "${localhost}"
 } -run
 delay .1
 
diff --git a/bin/varnishtest/tests/r00921.vtc b/bin/varnishtest/tests/r00921.vtc
index e207c18..7eac5bb 100644
--- a/bin/varnishtest/tests/r00921.vtc
+++ b/bin/varnishtest/tests/r00921.vtc
@@ -2,8 +2,8 @@ varnishtest "VCC type issue in regsub arg 1"
 
 server s1 {
 	rxreq
-	expect req.http.foo == "127.0.0.1"
-	expect req.http.bar == "127.0.0.1"
+	expect req.http.foo == "${localhost}"
+	expect req.http.bar == "${localhost}"
 	txresp
 } -start
 
diff --git a/bin/varnishtest/tests/r01312.vtc b/bin/varnishtest/tests/r01312.vtc
index 05003ea..c39dadc 100644
--- a/bin/varnishtest/tests/r01312.vtc
+++ b/bin/varnishtest/tests/r01312.vtc
@@ -6,6 +6,8 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
+	import ${vmod_std};
+
 	acl foo {
 		"127.0.0.2";
 		"127.0.1"/19;
@@ -15,8 +17,8 @@ varnish v1 -vcl+backend {
 		"127.0.1"/19;
 	}
 	sub vcl_deliver {
-		set resp.http.ACLfoo = client.ip ~ foo;
-		set resp.http.ACLbar = client.ip ~ bar;
+		set resp.http.ACLfoo = std.ip("127.0.0.1", client.ip) ~ foo;
+		set resp.http.ACLbar = std.ip("127.0.0.1", client.ip) ~ bar;
 	}
 } -start
 
diff --git a/bin/varnishtest/tests/r01650.vtc b/bin/varnishtest/tests/r01650.vtc
index e21ceba..ec124bb 100644
--- a/bin/varnishtest/tests/r01650.vtc
+++ b/bin/varnishtest/tests/r01650.vtc
@@ -2,7 +2,7 @@ varnishtest "xff handling discards multiple headers"
 
 server s1 {
 	rxreq
-	expect req.http.X-Forwarded-For == "1.2.3.4, 5.6.7.8, 127.0.0.1"
+	expect req.http.X-Forwarded-For == "1.2.3.4, 5.6.7.8, ${localhost}"
 	txresp
 } -start
 
diff --git a/bin/varnishtest/tests/r01693.vtc b/bin/varnishtest/tests/r01693.vtc
index f16d620..5f051d5 100644
--- a/bin/varnishtest/tests/r01693.vtc
+++ b/bin/varnishtest/tests/r01693.vtc
@@ -16,7 +16,7 @@ logexpect l1 -v v1 {
 	expect 0 1001 Hash "bar"
 	expect 0 1001 Hash "3"
 	expect 0 1001 Hash "/"
-	expect 0 1001 Hash "127.0.0.1"
+	expect 0 1001 Hash "${localhost}"
 } -start
 
 client c1 {
diff --git a/bin/varnishtest/tests/v00008.vtc b/bin/varnishtest/tests/v00008.vtc
index add54ef..95a870f 100644
--- a/bin/varnishtest/tests/v00008.vtc
+++ b/bin/varnishtest/tests/v00008.vtc
@@ -8,7 +8,7 @@ server s1 {
 
 	rxreq
 	expect req.url == "/bar"
-	expect req.http.host == "127.0.0.1"
+	expect req.http.host == "${localhost}"
 	txresp -body "foo1"
 } -start
 
diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index b60deb4..8890a62 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -121,6 +121,12 @@ static pthread_mutex_t		macro_mtx;
 static void
 init_macro(void)
 {
+	struct macro *m;
+
+	/* Dump the extmacros for completeness */
+	VTAILQ_FOREACH(m, &macro_list, list)
+		vtc_log(vltop, 4, "extmacro def %s=%s", m->name, m->val);
+
 	AZ(pthread_mutex_init(&macro_mtx, NULL));
 }
 
@@ -637,15 +643,6 @@ exec_file(const char *fn, const char *script, const char *tmpdir,
 	init_sema();
 	init_server();
 
-	/*
-	 * We need an IP number which will not repond, ever, and that is a
-	 * lot harder than it sounds.  This IP# is from RFC5737 and a
-	 * C-class broadcast at that.
-	 * If tests involving ${bad_ip} fails and you run linux, you should
-	 * check your /proc/sys/net/ipv4/ip_nonlocal_bind setting.
-	 */
-	macro_def(vltop, NULL, "bad_ip", "192.0.2.255");
-
 	/* Move into our tmpdir */
 	AZ(chdir(tmpdir));
 	macro_def(vltop, NULL, "tmpdir", "%s", tmpdir);
diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index 601f245..7272e22 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -427,6 +427,41 @@ dns_works(void)
 }
 
 /**********************************************************************
+ * Figure out what IP related magic
+ */
+
+static void
+ip_magic(void)
+{
+	const char *p;
+	int fd;
+	char abuf[VTCP_ADDRBUFSIZE];
+
+	/*
+	 * In FreeBSD jails localhost/127.0.0.1 becomes the jails IP#
+	 * XXX: IPv6-only hosts would have similar issue, but it is not
+	 * XXX: obvious how to cope.  Ideally "127.0.0.1" would be
+	 * XXX: "localhost", but that doesn't work out of the box.
+	 * XXX: Things like "prefer_ipv6" parameter complicates things.
+	 */
+	fd = VTCP_listen_on("127.0.0.1", NULL, 10, &p);
+	assert(fd >= 0);
+	VTCP_myname(fd, abuf, sizeof abuf, NULL, 0);
+	AZ(close(fd));
+	extmacro_def("localhost", "%s", abuf);
+
+	/*
+	 * We need an IP number which will not repond, ever, and that is a
+	 * lot harder than it sounds.  This IP# is from RFC5737 and a
+	 * C-class broadcast at that.
+	 * If tests involving ${bad_ip} fails and you run linux, you should
+	 * check your /proc/sys/net/ipv4/ip_nonlocal_bind setting.
+	 */
+
+	extmacro_def("bad_ip", "%s", "192.0.2.255");
+}
+
+/**********************************************************************
  * Main
  */
 
@@ -538,6 +573,7 @@ main(int argc, char * const *argv)
 	}
 
 	feature_dns = dns_works();
+	ip_magic();
 
 	if (iflg)
 		i_mode();



From phk at FreeBSD.org  Mon Feb  8 10:33:30 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 08 Feb 2016 11:33:30 +0100
Subject: [master] 5ead7c5 Dont panic because a housekeeping thread races
 against orderly shutdown.
Message-ID: <E1aSj8M-0004tV-Rq@project.varnish-software.com>


commit 5ead7c5151f9717f0ab442e544f3fd9bc29a4571
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 10:32:56 2016 +0000

    Dont panic because a housekeeping thread races against orderly
    shutdown.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index de768f2..6a2f1d4 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -506,8 +506,12 @@ vca_acct(void *arg)
 	while (1) {
 		(void)sleep(1);
 		if (vca_tcp_opt_init()) {
-			VTAILQ_FOREACH(ls, &heritage.socks, list)
+			VTAILQ_FOREACH(ls, &heritage.socks, list) {
+				if (ls->sock == -2)
+					continue;	// raced VCA_Shutdown
+				assert (ls->sock > 0);
 				vca_tcp_opt_set(ls->sock, 1);
+			}
 		}
 		now = VTIM_real();
 		VSC_C_main->uptime = (uint64_t)(now - t0);
@@ -581,7 +585,7 @@ VCA_Shutdown(void)
 
 	VTAILQ_FOREACH(ls, &heritage.socks, list) {
 		i = ls->sock;
-		ls->sock = -1;
+		ls->sock = -2;
 		(void)close(i);
 	}
 }



From phk at FreeBSD.org  Mon Feb  8 10:41:01 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 08 Feb 2016 11:41:01 +0100
Subject: [master] e7137ad Another VCA shutdown race.
Message-ID: <E1aSjFd-0005pi-26@project.varnish-software.com>


commit e7137ad3b526218a7ab0847fcac5e44416592b27
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 10:40:44 2016 +0000

    Another VCA shutdown race.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 6a2f1d4..328a64d 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -405,6 +405,12 @@ vca_accept_task(struct worker *wrk, void *arg)
 				   &wa.acceptaddrlen);
 		} while (i < 0 && errno == EAGAIN);
 
+		if (i < 0 && ls->sock == -2) {
+			/* Shut down in progress */
+			sleep(2);
+			continue;
+		}
+
 		if (i < 0) {
 			switch (errno) {
 			case ECONNABORTED:
@@ -416,7 +422,8 @@ vca_accept_task(struct worker *wrk, void *arg)
 			case EBADF:
 				VSL(SLT_Debug, ls->sock, "Accept failed: %s",
 				    strerror(errno));
-				return;
+				vca_pace_bad();
+				break;
 			default:
 				VSL(SLT_Debug, ls->sock, "Accept failed: %s",
 				    strerror(errno));



From phk at FreeBSD.org  Mon Feb  8 11:45:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 08 Feb 2016 12:45:11 +0100
Subject: [master] 2222e6b Move VCA_Shutdown() call to the main code
Message-ID: <E1aSkFj-00033B-FS@project.varnish-software.com>


commit 2222e6bd38c67d2e0ba2b38627cc8bcce6ceb6c2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 11:40:02 2016 +0000

    Move VCA_Shutdown() call to the main code

diff --git a/bin/varnishd/cache/cache_cli.c b/bin/varnishd/cache/cache_cli.c
index bc2302e..da8ee40 100644
--- a/bin/varnishd/cache/cache_cli.c
+++ b/bin/varnishd/cache/cache_cli.c
@@ -105,7 +105,6 @@ CLI_Run(void)
 		i = VCLS_Poll(cls, -1);
 	} while(i > 0);
 	VSL(SLT_CLI, 0, "EOF on CLI connection, worker stops");
-	VCA_Shutdown();
 }
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c
index 8d2e8f0..78ab1d2 100644
--- a/bin/varnishd/cache/cache_main.c
+++ b/bin/varnishd/cache/cache_main.c
@@ -269,6 +269,7 @@ child_main(void)
 
 	CLI_Run();
 
+	VCA_Shutdown();
 	BAN_Shutdown();
 	STV_close();
 



From phk at FreeBSD.org  Mon Feb  8 11:45:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 08 Feb 2016 12:45:11 +0100
Subject: [master] acb59df Use global -t timeout for CLI timeout
Message-ID: <E1aSkFj-00033H-HD@project.varnish-software.com>


commit acb59df64c05e6bb64ec501e2212b6055a0212d0
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 11:44:34 2016 +0000

    Use global -t timeout for CLI timeout

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 91c4072..1ff780f 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -109,7 +109,7 @@ varnish_ask_cli(const struct varnish *v, const char *cmd, char **repl)
 			vtc_log(v->vl, 0, "CLI write failed (%s) = %u %s",
 			    cmd, errno, strerror(errno));
 	}
-	i = VCLI_ReadResult(v->cli_fd, &retval, &r, 30.0);
+	i = VCLI_ReadResult(v->cli_fd, &retval, &r, vtc_maxdur);
 	if (i != 0) {
 		vtc_log(v->vl, 0, "CLI failed (%s) = %d %u %s",
 		    cmd, i, retval, r);



From phk at FreeBSD.org  Mon Feb  8 11:50:21 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 08 Feb 2016 12:50:21 +0100
Subject: [master] 7f26d65 Under circumstances I don't understand,
 test p00006 takes 52 seconds on osx.
Message-ID: <E1aSkKj-0003gQ-2J@project.varnish-software.com>


commit 7f26d6506ba1bcd725e8e40ec9c6adf35fbddb67
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 11:49:14 2016 +0000

    Under circumstances I don't understand, test p00006 takes 52 seconds
    on osx.
    
    Use the cli_timeout to control how long the MGR waits before killing
    the worker process, and give this test 60 seconds.

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 8baaa6d..1484e06 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -483,14 +483,20 @@ mgt_reap_child(void)
 		ev_listen = NULL;
 	}
 
-	/* Wait 10 seconds for child to die */
-	for (i = 0; i < 10; i++) {
+	/* Compose obituary */
+	vsb = VSB_new_auto();
+	XXXAN(vsb);
+
+	/* Wait for child to die */
+	for (i = 0; i < mgt_param.cli_timeout; i++) {
 		r = waitpid(child_pid, &status, WNOHANG);
 		if (r == child_pid)
 			break;
 		(void)sleep(1);
 	}
 	if (r == 0) {
+		VSB_printf(vsb, "Child (%jd) not dying, killing", (intmax_t)r);
+
 		/* Kick it Jim... */
 		if (MGT_FEATURE(FEATURE_NO_COREDUMP))
 			(void)kill(child_pid, SIGKILL);
@@ -504,9 +510,6 @@ mgt_reap_child(void)
 
 	MAC_reopen_sockets(NULL);
 
-	/* Compose obituary */
-	vsb = VSB_new_auto();
-	XXXAN(vsb);
 	VSB_printf(vsb, "Child (%jd) %s", (intmax_t)r,
 	    status ? "died" : "ended");
 	if (WIFEXITED(status) && WEXITSTATUS(status)) {
diff --git a/bin/varnishtest/tests/p00006.vtc b/bin/varnishtest/tests/p00006.vtc
index fb5fcca..9303fdc 100644
--- a/bin/varnishtest/tests/p00006.vtc
+++ b/bin/varnishtest/tests/p00006.vtc
@@ -12,6 +12,7 @@ server s1 {
 
 varnish v1 \
 	-arg "-sdeprecated_persistent,${tmpdir}/_.per,10m" \
+	-arg "-pcli_timeout=60"
 	-arg "-pban_lurker_sleep=0" \
 	-vcl+backend { } -start
 



From phk at FreeBSD.org  Mon Feb  8 12:03:02 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 08 Feb 2016 13:03:02 +0100
Subject: [master] 11988ad forgot a back-slash
Message-ID: <E1aSkX0-0004vV-47@project.varnish-software.com>


commit 11988adda012eb5cb9d606698e5a57f844e69538
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 12:02:50 2016 +0000

    forgot a back-slash

diff --git a/bin/varnishtest/tests/p00006.vtc b/bin/varnishtest/tests/p00006.vtc
index 9303fdc..605a88d 100644
--- a/bin/varnishtest/tests/p00006.vtc
+++ b/bin/varnishtest/tests/p00006.vtc
@@ -12,7 +12,7 @@ server s1 {
 
 varnish v1 \
 	-arg "-sdeprecated_persistent,${tmpdir}/_.per,10m" \
-	-arg "-pcli_timeout=60"
+	-arg "-pcli_timeout=60" \
 	-arg "-pban_lurker_sleep=0" \
 	-vcl+backend { } -start
 



From fgsch at lodoss.net  Mon Feb  8 13:35:32 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 08 Feb 2016 14:35:32 +0100
Subject: [master] 55dfe08 Test esi:included uncompressed synthetic responses
Message-ID: <E1aSlyW-00056R-ES@project.varnish-software.com>


commit 55dfe089607c1743593a9776bc9fafc0f45d5ed3
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Feb 8 09:47:53 2016 +0000

    Test esi:included uncompressed synthetic responses

diff --git a/bin/varnishtest/tests/r01838.vtc b/bin/varnishtest/tests/r01838.vtc
new file mode 100644
index 0000000..ce1f20b
--- /dev/null
+++ b/bin/varnishtest/tests/r01838.vtc
@@ -0,0 +1,32 @@
+varnishtest "Uncompressed synthetic responses as esi includes"
+
+server s1 {
+	rxreq
+	txresp -body {<FOO><esi:include src="/foo"/></FOO>}
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_recv {
+		if (req.url == "/foo") {
+			return (synth(998));
+		}
+	}
+
+	sub vcl_synth {
+		if (resp.status == 998) {
+			synthetic("synthetic body");
+			return (deliver);
+		}
+	}
+
+	sub vcl_backend_response {
+		set beresp.do_esi = true;
+	}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "<FOO>synthetic body</FOO>"
+} -run



From fgsch at lodoss.net  Mon Feb  8 16:42:07 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 08 Feb 2016 17:42:07 +0100
Subject: [master] 539d018 Silence warning
Message-ID: <E1aSot5-0003HT-JL@project.varnish-software.com>


commit 539d01805c140f6652357328834371222decfa7f
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Feb 8 16:36:56 2016 +0000

    Silence warning

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 1484e06..b759dcf 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -455,7 +455,7 @@ mgt_reap_child(void)
 	int i;
 	int status;
 	struct vsb *vsb;
-	pid_t r;
+	pid_t r = 0;
 
 	assert(child_pid != -1);
 



From phk at FreeBSD.org  Tue Feb  9 00:19:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:11 +0100
Subject: [master] 7d31cc7 Set socket non-blocking before closing,
 if we are load-shedding it.
Message-ID: <E1aSw1P-0007Lp-S7@project.varnish-software.com>


commit 7d31cc7708bfd42c982a730a53547c30b61468cb
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 20:29:09 2016 +0000

    Set socket non-blocking before closing, if we are load-shedding it.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 328a64d..5588082 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -313,9 +313,10 @@ vca_make_session(struct worker *wrk, void *arg)
 		 * connection with minimum effort and fuzz, rather than try
 		 * to send an intelligent message back.
 		 */
+		vca_pace_bad();
+		(void)VTCP_nonblocking(wa->acceptsock);
 		AZ(close(wa->acceptsock));
 		wrk->stats->sess_drop++;
-		vca_pace_bad();
 		WS_Release(wrk->aws, 0);
 		return;
 	}



From phk at FreeBSD.org  Tue Feb  9 00:19:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:11 +0100
Subject: [master] f0ff676 Initialize a variable to quieten down compilers and
 FlexeLint
Message-ID: <E1aSw1P-0007Lv-TX@project.varnish-software.com>


commit f0ff676fe738e41ccd9e02abdff6936b83b4ce6f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 20:45:18 2016 +0000

    Initialize a variable to quieten down compilers and FlexeLint

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index b759dcf..e484252 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -453,7 +453,7 @@ static void
 mgt_reap_child(void)
 {
 	int i;
-	int status;
+	int status = 0xffff;
 	struct vsb *vsb;
 	pid_t r = 0;
 



From phk at FreeBSD.org  Tue Feb  9 00:19:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:11 +0100
Subject: [master] 6a3398d Introduce "struct proto" which is opaque to the
 management process.
Message-ID: <E1aSw1P-0007M4-Uv@project.varnish-software.com>


commit 6a3398d58224b5ea8c0860229b0aa8b72335f899
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 20:46:58 2016 +0000

    Introduce "struct proto" which is opaque to the management process.

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 5c84034..d98065b 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -107,6 +107,7 @@ noinst_HEADERS = \
 	cache/cache_esi.h \
 	cache/cache_obj.h \
 	cache/cache_pool.h \
+	cache/cache_proto.h \
 	cache/cache_priv.h \
 	common/heritage.h \
 	hash/hash_slinger.h \
diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index cfd17bb..7d3e2c7 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -618,6 +618,12 @@ enum sess_attr {
 	SA_LAST
 };
 
+enum sess_step {
+#define SESS_STEP(l, u)		S_STP_##u,
+#include "tbl/steps.h"
+#undef SESS_STEP
+};
+
 struct sess {
 	unsigned		magic;
 #define SESS_MAGIC		0x2c2f9c5a
diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 5588082..fa9a5ff 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -41,6 +41,7 @@
 #include <netinet/tcp.h>
 
 #include "cache.h"
+#include "cache_proto.h"
 #include "cache_pool.h"
 #include "common/heritage.h"
 
@@ -329,7 +330,7 @@ vca_make_session(struct worker *wrk, void *arg)
 
 	sp->fd = wa->acceptsock;
 	wa->acceptsock = -1;
-	sp->sess_step = wa->acceptlsock->first_step;
+	sp->sess_step = wa->acceptlsock->proto->first_step;
 
 	assert(wa->acceptaddrlen <= vsa_suckaddr_len);
 	SES_Reserve_remote_addr(sp, &sa);
@@ -494,6 +495,7 @@ vca_acct(void *arg)
 	(void)vca_tcp_opt_init();
 
 	VTAILQ_FOREACH(ls, &heritage.socks, list) {
+		CHECK_OBJ_NOTNULL(ls->proto, PROTO_MAGIC);
 		assert (ls->sock > 0);		// We know where stdin is
 		AZ(listen(ls->sock, cache_param->listen_depth));
 		vca_tcp_opt_set(ls->sock, 1);
diff --git a/bin/varnishd/cache/cache_proto.h b/bin/varnishd/cache/cache_proto.h
new file mode 100644
index 0000000..c432192
--- /dev/null
+++ b/bin/varnishd/cache/cache_proto.h
@@ -0,0 +1,34 @@
+/*-
+ * Copyright (c) 2016 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+struct proto {
+	unsigned		magic;
+#define PROTO_MAGIC		0x711e77c1
+	enum sess_step		first_step;
+};
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index da489fc..0c23ddf 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -62,12 +62,6 @@ enum obj_flags {
 #undef OBJ_FLAG
 };
 
-enum sess_step {
-#define SESS_STEP(l, u)		S_STP_##u,
-#include "tbl/steps.h"
-#undef SESS_STEP
-};
-
 struct cli;
 
 /**********************************************************************
diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h
index 42e9ec7..cbb8f9c 100644
--- a/bin/varnishd/common/heritage.h
+++ b/bin/varnishd/common/heritage.h
@@ -31,6 +31,7 @@
 
 struct vsm_sc;
 struct suckaddr;
+struct proto;
 
 struct listen_sock {
 	unsigned			magic;
@@ -39,10 +40,13 @@ struct listen_sock {
 	int				sock;
 	char				*name;
 	struct suckaddr			*addr;
-	enum sess_step			first_step;
+	const struct proto		*proto;
 	const char			*proto_name;
 };
 
+extern const struct proto VPX_proto;
+extern const struct proto HTTP1_proto;
+
 VTAILQ_HEAD(listen_sock_head, listen_sock);
 
 struct heritage {
diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c
index 210be90..d86ba28 100644
--- a/bin/varnishd/http1/cache_http1_fsm.c
+++ b/bin/varnishd/http1/cache_http1_fsm.c
@@ -38,6 +38,7 @@
 #include <stdlib.h>
 
 #include "cache/cache.h"
+#include "cache/cache_proto.h"
 #include "cache_http1.h"
 #include "hash/hash_slinger.h"
 
@@ -285,3 +286,8 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 
 	}
 }
+
+const struct proto HTTP1_proto = {
+	.magic =		PROTO_MAGIC,
+	.first_step =		S_STP_H1NEWSESS,
+};
diff --git a/bin/varnishd/mgt/mgt_acceptor.c b/bin/varnishd/mgt/mgt_acceptor.c
index 3cf1bbc..63adbac 100644
--- a/bin/varnishd/mgt/mgt_acceptor.c
+++ b/bin/varnishd/mgt/mgt_acceptor.c
@@ -117,7 +117,7 @@ struct mac_help {
 	int			good;
 	const char		*name;
 	const char		*proto_name;
-	enum sess_step		first_step;
+	const struct proto	*proto;
 };
 
 static int __match_proto__(vss_resolved_f)
@@ -141,7 +141,7 @@ mac_callback(void *priv, const struct suckaddr *sa)
 	ls->name = strdup(mh->name);
 	AN(ls->name);
 	ls->proto_name = mh->proto_name;
-	ls->first_step = mh->first_step;
+	ls->proto = mh->proto;
 	VTAILQ_INSERT_TAIL(&heritage.socks, ls, list);
 	mh->good++;
 	return (0);
@@ -197,12 +197,12 @@ MAC_Arg(const char *arg)
 	mh->name = av[1];
 
 	if (av[2] == NULL || !strcmp(av[2], "HTTP/1")) {
-		mh->first_step = S_STP_H1NEWSESS;
+		mh->proto = &HTTP1_proto;
 		mh->proto_name = "HTTP/1";
 		if (av[2] != NULL && av[3] != NULL)
 			ARGV_ERR("Too many sub-arguments to -a(HTTP/1)\n");
 	} else if (!strcmp(av[2], "PROXY")) {
-		mh->first_step = S_STP_PROXYNEWSESS;
+		mh->proto = &VPX_proto;
 		mh->proto_name = "PROXY";
 		if (av[3] != NULL)
 			ARGV_ERR("Too many sub-arguments to -a(PROXY)\n");
diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index 4799e4b..a73f21a 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -39,6 +39,7 @@
 #include <string.h>
 
 #include "../cache/cache.h"
+#include "../cache/cache_proto.h"
 
 #include "vend.h"
 #include "vsa.h"
@@ -379,3 +380,8 @@ VPX_Proto_Sess(struct worker *wrk, void *priv)
 	wrk->task.func = SES_Proto_Req;
 	wrk->task.priv = req;
 }
+
+const struct proto VPX_proto = {
+	.magic =		PROTO_MAGIC,
+	.first_step =		S_STP_PROXYNEWSESS,
+};



From phk at FreeBSD.org  Tue Feb  9 00:19:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:11 +0100
Subject: [master] d8c04ee Move objcore related enums out of mgt view
Message-ID: <E1aSw1Q-0007ME-01@project.varnish-software.com>


commit d8c04ee9075a0f9a7ced08f2ec5862d977ac01fc
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 20:53:18 2016 +0000

    Move objcore related enums out of mgt view

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 7d3e2c7..3cdaca7 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -398,6 +398,18 @@ struct boc {
  * housekeeping fields parts of an object.
  */
 
+enum obj_attr {
+#define OBJ_ATTR(U, l)  OA_##U,
+#include "tbl/obj_attr.h"
+#undef OBJ_ATTR
+};
+
+enum obj_flags {
+#define OBJ_FLAG(U, l, v)       OF_##U = v,
+#include "tbl/obj_attr.h"
+#undef OBJ_FLAG
+};
+
 struct objcore {
 	unsigned		magic;
 #define OBJCORE_MAGIC		0x4d301302
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index 0c23ddf..52bbd90 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -40,28 +40,11 @@
 #include "vsb.h"
 #include "vapi/vsc_int.h"
 
-/*
- * Enums cannot be forward declared (any more...) so put it here
- * to make everybody see it.
- */
-
 enum baninfo {
 	BI_NEW,
 	BI_DROP
 };
 
-enum obj_attr {
-#define OBJ_ATTR(U, l)	OA_##U,
-#include "tbl/obj_attr.h"
-#undef OBJ_ATTR
-};
-
-enum obj_flags {
-#define OBJ_FLAG(U, l, v)	OF_##U = v,
-#include "tbl/obj_attr.h"
-#undef OBJ_FLAG
-};
-
 struct cli;
 
 /**********************************************************************



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] 861d557 Dramatically reduce the scope of enum baninfo: It is
 now only part of the stevedore API.
Message-ID: <E1aSw1Q-0007MN-1S@project.varnish-software.com>


commit 861d5577cb5cc6132d4f199948acc8336c2e7552
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 21:13:41 2016 +0000

    Dramatically reduce the scope of enum baninfo:  It is now only
    part of the stevedore API.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 3cdaca7..701a1e4 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -1061,8 +1061,6 @@ int STV_NewObject(struct worker *, struct objcore *,
     const char *hint, unsigned len);
 void STV_open(void);
 void STV_close(void);
-int STV_BanInfo(enum baninfo event, const uint8_t *ban, unsigned len);
-void STV_BanExport(const uint8_t *bans, unsigned len);
 
 /*
  * A normal pointer difference is signed, but we never want a negative value
diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index b0f53ad..3f70c42 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -301,19 +301,23 @@ ban_export(void)
 	VSC_C_main->bans_persisted_fragmentation = 0;
 }
 
+/*
+ * For both of these we do a full export on info failure to remove
+ * holes in the exported list.
+ * XXX: we should keep track of the size of holes in the last exported list
+ */
 void
-ban_info(enum baninfo event, const uint8_t *ban, unsigned len)
+ban_info_new(const uint8_t *ban, unsigned len)
 {
-	if (STV_BanInfo(event, ban, len)) {
-		/* One or more stevedores reported failure. Export the
-		 * list instead. The exported list should take up less
-		 * space due to drops being purged and completed being
-		 * truncated. */
-		/* XXX: Keep some measure of how much space can be
-		 * saved, and only export if it's worth it. Assert if
-		 * not */
+	if (STV_BanInfoNew(ban, len))
+		ban_export();
+}
+
+void
+ban_info_drop(const uint8_t *ban, unsigned len)
+{
+	if (STV_BanInfoNew(ban, len))
 		ban_export();
-	}
 }
 
 /*--------------------------------------------------------------------
@@ -743,7 +747,7 @@ BAN_Compile(void)
 
 	/* Report the place-holder ban */
 	b = VTAILQ_FIRST(&ban_head);
-	AZ(STV_BanInfo(BI_NEW, b->spec, ban_len(b->spec)));
+	ban_info_new(b->spec, ban_len(b->spec));
 
 	ban_export();
 
diff --git a/bin/varnishd/cache/cache_ban.h b/bin/varnishd/cache/cache_ban.h
index 100acee..f3992bb 100644
--- a/bin/varnishd/cache/cache_ban.h
+++ b/bin/varnishd/cache/cache_ban.h
@@ -109,7 +109,9 @@ extern pthread_cond_t	ban_lurker_cond;
 
 void ban_mark_completed(struct ban *b);
 unsigned ban_len(const uint8_t *banspec);
-void ban_info(enum baninfo event, const uint8_t *ban, unsigned len);
+void ban_info_new(const uint8_t *ban, unsigned len);
+void ban_info_drop(const uint8_t *ban, unsigned len);
+
 int ban_evaluate(struct worker *wrk, const uint8_t *bs, struct objcore *oc,
     const struct http *reqhttp, unsigned *tests);
 double ban_time(const uint8_t *banspec);
diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c
index b3bda03..4dec68b 100644
--- a/bin/varnishd/cache/cache_ban_build.c
+++ b/bin/varnishd/cache/cache_ban_build.c
@@ -306,7 +306,7 @@ BAN_Commit(struct ban_proto *bp)
 		VSC_C_main->bans_req++;
 
 	if (bi != NULL)
-		ban_info(BI_NEW, b->spec, ln);	/* Notify stevedores */
+		ban_info_new(b->spec, ln);	/* Notify stevedores */
 
 	if (cache_param->ban_dups) {
 		/* Hunt down duplicates, and mark them as completed */
diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
index 63371f8..e913fda 100644
--- a/bin/varnishd/cache/cache_ban_lurker.c
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -71,7 +71,7 @@ ban_cleantail(void)
 			VTAILQ_REMOVE(&ban_head, b, list);
 			VSC_C_main->bans_persisted_fragmentation +=
 			    ban_len(b->spec);
-			ban_info(BI_DROP, b->spec, ban_len(b->spec));
+			ban_info_drop(b->spec, ban_len(b->spec));
 		} else {
 			b = NULL;
 		}
diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h
index f3a3aaa..7dd941e 100644
--- a/bin/varnishd/cache/cache_priv.h
+++ b/bin/varnishd/cache/cache_priv.h
@@ -128,6 +128,11 @@ void VRTPRIV_dynamic_kill(struct vrt_privs *privs, uintptr_t id);
 /* cache_vrt_vmod.c */
 void VMOD_Init(void);
 
+/* stevedore.c */
+int STV_BanInfoDrop(const uint8_t *ban, unsigned len);
+int STV_BanInfoNew(const uint8_t *ban, unsigned len);
+void STV_BanExport(const uint8_t *banlist, unsigned len);
+
 /* storage_persistent.c */
 void SMP_Init(void);
 void SMP_Ready(void);
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index 52bbd90..0b494c3 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -40,11 +40,6 @@
 #include "vsb.h"
 #include "vapi/vsc_int.h"
 
-enum baninfo {
-	BI_NEW,
-	BI_DROP
-};
-
 struct cli;
 
 /**********************************************************************
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 660f511..a7648ec 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -166,14 +166,27 @@ STV_close(void)
  */
 
 int
-STV_BanInfo(enum baninfo event, const uint8_t *ban, unsigned len)
+STV_BanInfoDrop(const uint8_t *ban, unsigned len)
 {
 	struct stevedore *stv;
 	int r = 0;
 
 	VTAILQ_FOREACH(stv, &stv_stevedores, list)
 		if (stv->baninfo != NULL)
-			r |= stv->baninfo(stv, event, ban, len);
+			r |= stv->baninfo(stv, BI_DROP, ban, len);
+
+	return (r);
+}
+
+int
+STV_BanInfoNew(const uint8_t *ban, unsigned len)
+{
+	struct stevedore *stv;
+	int r = 0;
+
+	VTAILQ_FOREACH(stv, &stv_stevedores, list)
+		if (stv->baninfo != NULL)
+			r |= stv->baninfo(stv, BI_NEW, ban, len);
 
 	return (r);
 }
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index 4f12d9c..462c74c 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -40,6 +40,11 @@ struct vsl_log;
 struct vfp_ctx;
 struct obj_methods;
 
+enum baninfo {
+	BI_NEW,
+	BI_DROP
+};
+
 /* Storage -----------------------------------------------------------*/
 
 struct storage {



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] a3c3181 Move a bit of FlexeLintery up to vdef.h
Message-ID: <E1aSw1Q-0007MX-2d@project.varnish-software.com>


commit a3c3181cc40a4dc6aba8d0ac3695472ed5e770ca
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 21:21:18 2016 +0000

    Move a bit of FlexeLintery up to vdef.h

diff --git a/bin/varnishd/cache/cache_mempool.c b/bin/varnishd/cache/cache_mempool.c
index 472441c..f4c1c30 100644
--- a/bin/varnishd/cache/cache_mempool.c
+++ b/bin/varnishd/cache/cache_mempool.c
@@ -96,7 +96,7 @@ mpl_guard(void *priv)
 {
 	struct mempool *mpl;
 	struct memitem *mi = NULL;
-	double mpl_slp __state_variable__(mpl_slp);
+	double __state_variable__(mpl_slp);
 	double last = 0;
 
 	CAST_OBJ_NOTNULL(mpl, priv, MEMPOOL_MAGIC);
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index 0b494c3..9f21257 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -42,17 +42,6 @@
 
 struct cli;
 
-/**********************************************************************
- * FlexeLint and compiler shutuppery
- */
-
-/*
- * State variables may change value before we use the last value we
- * set them to.
- * Pass no argument.
- */
-#define __state_variable__(xxx)		/*lint -esym(838,xxx) */
-
 /**********************************************************************/
 
 /* Name of transient storage */
diff --git a/include/vdef.h b/include/vdef.h
index 0790e43..bf5d3f4 100644
--- a/include/vdef.h
+++ b/include/vdef.h
@@ -78,6 +78,12 @@
  */
 #define __match_proto__(xxx)		/*lint -e{818} */
 
+/*
+ * State variables may change value before we have considered the
+ * previous value
+ */
+#define __state_variable__(varname)	varname /*lint -esym(838,varname) */
+
 #define NEEDLESS_RETURN		return
 
 #endif /* VDEF_H_INCLUDED */



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] 289cd3c Shift more general stuff from common.h to vdef.h
Message-ID: <E1aSw1Q-0007Mg-3m@project.varnish-software.com>


commit 289cd3cc7b2a52f2dd0e8f3a591ad7227825ac2b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 21:26:27 2016 +0000

    Shift more general stuff from common.h to vdef.h

diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index 9f21257..e280282 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -40,8 +40,6 @@
 #include "vsb.h"
 #include "vapi/vsc_int.h"
 
-struct cli;
-
 /**********************************************************************/
 
 /* Name of transient storage */
@@ -78,30 +76,3 @@ void VSM_common_delete(struct vsm_sc **sc);
 void VSM_common_copy(struct vsm_sc *to, const struct vsm_sc *from);
 void VSM_common_cleaner(struct vsm_sc *sc, struct VSC_C_main *stats);
 void VSM_common_ageupdate(const struct vsm_sc *sc);
-
-/*---------------------------------------------------------------------
- * Generic power-2 rounding macros
- */
-
-#define PWR2(x)     ((((x)-1UL)&(x))==0)		/* Is a power of two */
-#define RDN2(x, y)  ((x)&(~((uintptr_t)(y)-1UL)))	/* PWR2(y) true */
-#define RUP2(x, y)  (((x)+((y)-1))&(~((uintptr_t)(y)-1UL))) /* PWR2(y) true */
-
-/*--------------------------------------------------------------------
- * Pointer alignment magic
- */
-
-#if defined(__sparc__)
-/* NB: Overbroad test for 32bit userland on 64bit SPARC cpus. */
-#  define PALGN	    (sizeof(double) - 1)	/* size of alignment */
-#else
-#  define PALGN	    (sizeof(void *) - 1)	/* size of alignment */
-#endif
-#define PAOK(p)	    (((uintptr_t)(p) & PALGN) == 0)	/* is aligned */
-#define PRNDDN(p)   ((uintptr_t)(p) & ~PALGN)		/* Round down */
-#define PRNDUP(p)   (((uintptr_t)(p) + PALGN) & ~PALGN)	/* Round up */
-
-/*--------------------------------------------------------------------
- * To be used as little as possible to wash off const/volatile etc.
- */
-#define TRUST_ME(ptr)	((void*)(uintptr_t)(ptr))
diff --git a/include/vdef.h b/include/vdef.h
index bf5d3f4..3ec0608 100644
--- a/include/vdef.h
+++ b/include/vdef.h
@@ -67,6 +67,33 @@
 #  define __v_printflike(f,a)
 #endif
 
+/*********************************************************************
+ * Pointer alignment magic
+ */
+
+#if defined(__sparc__)
+/* NB: Overbroad test for 32bit userland on 64bit SPARC cpus. */
+#  define PALGN	    (sizeof(double) - 1)	/* size of alignment */
+#else
+#  define PALGN	    (sizeof(void *) - 1)	/* size of alignment */
+#endif
+#define PAOK(p)	    (((uintptr_t)(p) & PALGN) == 0)	/* is aligned */
+#define PRNDDN(p)   ((uintptr_t)(p) & ~PALGN)		/* Round down */
+#define PRNDUP(p)   (((uintptr_t)(p) + PALGN) & ~PALGN)	/* Round up */
+
+/*********************************************************************
+ * To be used as little as possible to wash off const/volatile etc.
+ */
+#define TRUST_ME(ptr)	((void*)(uintptr_t)(ptr))
+
+/**********************************************************************
+ * Generic power-2 rounding macros
+ */
+
+#define PWR2(x)     ((((x)-1UL)&(x))==0)		/* Is a power of two */
+#define RDN2(x, y)  ((x)&(~((uintptr_t)(y)-1UL)))	/* PWR2(y) true */
+#define RUP2(x, y)  (((x)+((y)-1))&(~((uintptr_t)(y)-1UL))) /* PWR2(y) true */
+
 /**********************************************************************
  * FlexeLint and compiler shutuppery
  */



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] 756e1d7 fix a comment
Message-ID: <E1aSw1Q-0007Mp-4u@project.varnish-software.com>


commit 756e1d749910951787f84d070649ee141b88d8fc
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 21:30:00 2016 +0000

    fix a comment

diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index e280282..a5674a7 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -65,7 +65,7 @@ void mgt_child_inherit(int fd, const char *what);
 /* cache/cache_vcl.c */
 int VCL_TestLoad(const char *);
 
-/* vsm.c */
+/* common_vsm.c */
 struct vsm_sc;
 struct VSC_C_main;
 struct vsm_sc *VSM_common_new(void *ptr, ssize_t len);



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] 6aca214 white space
Message-ID: <E1aSw1Q-0007My-64@project.varnish-software.com>


commit 6aca2141267bdb19a1e280b550d8067b266b057e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 21:38:09 2016 +0000

    white space

diff --git a/bin/varnishd/storage/mgt_stevedore.c b/bin/varnishd/storage/mgt_stevedore.c
index 920a4a2..b2acaa8 100644
--- a/bin/varnishd/storage/mgt_stevedore.c
+++ b/bin/varnishd/storage/mgt_stevedore.c
@@ -92,14 +92,12 @@ smp_fake_init(struct stevedore *parent, int ac, char * const *av)
 	);
 }
 
-
 static const struct stevedore smp_fake_stevedore = {
 	.magic = STEVEDORE_MAGIC,
 	.name = "deprecated_persistent",
 	.init = smp_fake_init,
 };
 
-
 /*--------------------------------------------------------------------
  * Parse a stevedore argument on the form:
  *	[ name '=' ] strategy [ ',' arg ] *



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] 7375e05 Wean cache_rfc2616.c from struct exp.
Message-ID: <E1aSw1Q-0007N7-7H@project.varnish-software.com>


commit 7375e05ef099d4a6463493e795ad0404d86c3822
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 21:48:31 2016 +0000

    Wean cache_rfc2616.c from struct exp.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 701a1e4..95dde95 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -1050,7 +1050,8 @@ int WS_Overflowed(const struct ws *ws);
 void *WS_Printf(struct ws *ws, const char *fmt, ...) __v_printflike(2, 3);
 
 /* cache_rfc2616.c */
-void RFC2616_Ttl(struct busyobj *, double now);
+void RFC2616_Ttl(struct busyobj *, double now, double *t_origin,
+    float *ttl, float *grace, float *keep);
 unsigned RFC2616_Req_Gzip(const struct http *);
 int RFC2616_Do_Cond(const struct req *sp);
 void RFC2616_Weaken_Etag(struct http *hp);
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 2b2d200..de2a2ca 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -384,8 +384,12 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo)
 	/*
 	 * What does RFC2616 think about TTL ?
 	 */
-	EXP_Clr(&bo->fetch_objcore->exp);
-	RFC2616_Ttl(bo, now);
+	RFC2616_Ttl(bo, now,
+	    &bo->fetch_objcore->exp.t_origin,
+	    &bo->fetch_objcore->exp.ttl,
+	    &bo->fetch_objcore->exp.grace,
+	    &bo->fetch_objcore->exp.keep
+	);
 
 	/* private objects have negative TTL */
 	if (bo->fetch_objcore->flags & OC_F_PRIVATE)
diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c
index cb089cf..6b80371 100644
--- a/bin/varnishd/cache/cache_rfc2616.c
+++ b/bin/varnishd/cache/cache_rfc2616.c
@@ -62,25 +62,27 @@
  */
 
 void
-RFC2616_Ttl(struct busyobj *bo, double now)
+RFC2616_Ttl(struct busyobj *bo, double now, double *t_origin,
+    float *ttl, float *grace, float *keep)
 {
 	unsigned max_age, age;
 	double h_date, h_expires;
 	const char *p;
 	const struct http *hp;
-	struct exp *expp;
 
 	CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC);
-	expp = &bo->fetch_objcore->exp;
-
-	hp = bo->beresp;
-
 	assert(now != 0.0 && !isnan(now));
-	expp->t_origin = now;
+	AN(t_origin);
+	AN(ttl);
+	AN(grace);
+	AN(keep);
 
-	expp->ttl = cache_param->default_ttl;
-	expp->grace = cache_param->default_grace;
-	expp->keep = cache_param->default_keep;
+	*t_origin = now;
+	*ttl = cache_param->default_ttl;
+	*grace = cache_param->default_grace;
+	*keep = cache_param->default_keep;
+
+	hp = bo->beresp;
 
 	max_age = age = 0;
 	h_expires = 0;
@@ -98,7 +100,7 @@ RFC2616_Ttl(struct busyobj *bo, double now)
 		 * compatible with fractional seconds.
 		 */
 		age = strtoul(p, NULL, 10);
-		expp->t_origin -= age;
+		*t_origin -= age;
 	}
 
 	if (http_GetHdr(hp, H_Expires, &p))
@@ -109,7 +111,7 @@ RFC2616_Ttl(struct busyobj *bo, double now)
 
 	switch (http_GetStatus(hp)) {
 	default:
-		expp->ttl = -1.;
+		*ttl = -1.;
 		break;
 	case 302: /* Moved Temporarily */
 	case 307: /* Temporary Redirect */
@@ -119,7 +121,7 @@ RFC2616_Ttl(struct busyobj *bo, double now)
 		 * Do not apply the default ttl, only set a ttl if Cache-Control
 		 * or Expires are present. Uncacheable otherwise.
 		 */
-		expp->ttl = -1.;
+		*ttl = -1.;
 		/* FALL-THROUGH */
 	case 200: /* OK */
 	case 203: /* Non-Authoritative Information */
@@ -144,7 +146,7 @@ RFC2616_Ttl(struct busyobj *bo, double now)
 			else
 				max_age = strtoul(p, NULL, 0);
 
-			expp->ttl = max_age;
+			*ttl = max_age;
 			break;
 		}
 
@@ -155,7 +157,7 @@ RFC2616_Ttl(struct busyobj *bo, double now)
 
 		/* If backend told us it is expired already, don't cache. */
 		if (h_expires < h_date) {
-			expp->ttl = 0;
+			*ttl = 0;
 			break;
 		}
 
@@ -167,9 +169,9 @@ RFC2616_Ttl(struct busyobj *bo, double now)
 			 * trust Expires: relative to our own clock.
 			 */
 			if (h_expires < now)
-				expp->ttl = 0;
+				*ttl = 0;
 			else
-				expp->ttl = h_expires - now;
+				*ttl = h_expires - now;
 			break;
 		} else {
 			/*
@@ -177,7 +179,7 @@ RFC2616_Ttl(struct busyobj *bo, double now)
 			 * derive a relative time from the two headers.
 			 * (the negative ttl case is caught above)
 			 */
-			expp->ttl = (int)(h_expires - h_date);
+			*ttl = (int)(h_expires - h_date);
 		}
 
 	}
@@ -186,19 +188,19 @@ RFC2616_Ttl(struct busyobj *bo, double now)
 	 * RFC5861 outlines a way to control the use of stale responses.
 	 * We use this to initialize the grace period.
 	 */
-	if (expp->ttl >= 0 && http_GetHdrField(hp, H_Cache_Control,
+	if (*ttl >= 0 && http_GetHdrField(hp, H_Cache_Control,
 	    "stale-while-revalidate", &p) && p != NULL) {
 
 		if (*p == '-')
-			expp->grace = 0;
+			*grace = 0;
 		else
-			expp->grace = strtoul(p, NULL, 0);
+			*grace = strtoul(p, NULL, 0);
 	}
 
 	VSLb(bo->vsl, SLT_TTL,
 	    "RFC %.0f %.0f %.0f %.0f %.0f %.0f %.0f %u",
-	    expp->ttl, expp->grace, -1., now,
-	    expp->t_origin, h_date, h_expires, max_age);
+	    *ttl, *grace, -1., now,
+	    *t_origin, h_date, h_expires, max_age);
 }
 
 /*--------------------------------------------------------------------



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] 196e396 Struct exp being 20 bytes long causes alignment
 issues in persistent.
Message-ID: <E1aSw1Q-0007NH-8X@project.varnish-software.com>


commit 196e3965a9b8461ea1136b7d8d10240bf081d8ce
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 22:15:43 2016 +0000

    Struct exp being 20 bytes long causes alignment issues in persistent.
    
    Add macros to work on "disolved" struct exp

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 95dde95..7ca3eb7 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -694,10 +694,34 @@ extern pthread_t cli_thread;
 #define ASSERT_CLI() do {assert(pthread_self() == cli_thread);} while (0)
 
 /* cache_expire.c */
-void EXP_Clr(struct exp *e);
+
+/*
+ * The set of variables which control object expiry are inconveniently
+ * 24 bytes long (double+3*float) and this causes alignment waste if
+ * we put then in a struct.
+ * These three macros operate on the struct we don't use.
+ */
+
+#define EXP_ZERO(xx)							\
+	do {								\
+		(xx)->t_origin = 0.0;					\
+		(xx)->ttl = 0.0;					\
+		(xx)->grace = 0.0;					\
+		(xx)->keep = 0.0;					\
+	} while (0)
+
+#define EXP_COPY(to,fm)							\
+	do {								\
+		(to)->t_origin = (fm)->t_origin;			\
+		(to)->ttl = (fm)->ttl;					\
+		(to)->grace = (fm)->grace;				\
+		(to)->keep = (fm)->keep;				\
+	} while (0)
+
+#define EXP_WHEN(to)							\
+	((to)->t_origin + (to)->ttl + (to)->grace + (to)->keep)
 
 double EXP_Ttl(const struct req *, const struct exp*);
-double EXP_When(const struct exp *exp);
 void EXP_Insert(struct worker *wrk, struct objcore *oc);
 void EXP_Rearm(struct objcore *, double now, double ttl, double grace,
     double keep);
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 06db2d7..10f5f82 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -87,20 +87,6 @@ exp_event(struct worker *wrk, struct objcore *oc, enum exp_event_e e)
 }
 
 /*--------------------------------------------------------------------
- * struct exp manipulations
- */
-
-void
-EXP_Clr(struct exp *e)
-{
-
-	e->ttl = -1;
-	e->grace = 0;
-	e->keep = 0;
-	e->t_origin = 0;
-}
-
-/*--------------------------------------------------------------------
  * Calculate an objects effective ttl time, taking req.ttl into account
  * if it is available.
  */
@@ -120,7 +106,7 @@ EXP_Ttl(const struct req *req, const struct exp *e)
  * Calculate when this object is no longer useful
  */
 
-double
+static double
 EXP_When(const struct exp *e)
 {
 	double when;
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index de2a2ca..57295da 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -798,6 +798,7 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
 	http_TimeHeader(bo->beresp, "Date: ", now);
 	http_SetHeader(bo->beresp, "Server: Varnish");
 
+	bo->fetch_objcore->exp.t_origin = now;
 	if (!VTAILQ_EMPTY(&bo->fetch_objcore->objhead->waitinglist)) {
 		/*
 		 * If there is a waitinglist, it means that there is no
@@ -806,13 +807,13 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
 		 * each objcore on the waiting list sequentially attempt
 		 * to fetch from the backend.
 		 */
-		bo->fetch_objcore->exp.t_origin = now;
 		bo->fetch_objcore->exp.ttl = 1;
 		bo->fetch_objcore->exp.grace = 5;
 		bo->fetch_objcore->exp.keep = 5;
 	} else {
-		EXP_Clr(&bo->fetch_objcore->exp);
-		bo->fetch_objcore->exp.t_origin = now;
+		bo->fetch_objcore->exp.ttl = 0;
+		bo->fetch_objcore->exp.grace = 0;
+		bo->fetch_objcore->exp.keep = 0;
 	}
 
 	synth_body = VSB_new_auto();
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index 14ee3a7..bbc8a83 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -475,7 +475,7 @@ smp_allocx(const struct stevedore *st, size_t min_size, size_t max_size,
 			sc->next_top -= sizeof(**so);
 			*so = (void*)(sc->base + sc->next_top);
 			/* Render this smp_object mostly harmless */
-			EXP_Clr(&(*so)->exp);
+			EXP_ZERO((*so));
 			(*so)->ban = 0.;
 			(*so)->ptr = 0;
 			sg->objs = *so;
@@ -562,7 +562,7 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv,
 	/* We have to do this somewhere, might as well be here... */
 	assert(sizeof so->hash == DIGEST_LEN);
 	memcpy(so->hash, oc->objhead->digest, DIGEST_LEN);
-	so->exp = oc->exp;
+	EXP_COPY(so, &oc->exp);
 	so->ptr = (uint8_t*)o - sc->base;
 	so->ban = BAN_Time(oc->ban);
 
diff --git a/bin/varnishd/storage/storage_persistent.h b/bin/varnishd/storage/storage_persistent.h
index 4ccb014..9edb47b 100644
--- a/bin/varnishd/storage/storage_persistent.h
+++ b/bin/varnishd/storage/storage_persistent.h
@@ -144,7 +144,10 @@ struct smp_segptr {
 
 struct smp_object {
 	uint8_t			hash[32];	/* really: DIGEST_LEN */
-	struct exp		exp;
+	double			t_origin;
+	float			ttl;
+	float			grace;
+	float			keep;
 	uint32_t		__filler__;	/* -> align/8 on 32bit */
 	double			ban;
 	uint64_t		ptr;		/* rel to silo */
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index f46787f..9710dab 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -157,7 +157,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 	/* Clear the bogus "hold" count */
 	sg->nobj = 0;
 	for (;no > 0; so++,no--) {
-		if (EXP_When(&so->exp) < t_now)
+		if (EXP_WHEN(so) < t_now)
 			continue;
 		oc = ObjNew(wrk);
 		oc->flags &= ~OC_F_BUSY;
@@ -166,7 +166,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		VTAILQ_INSERT_TAIL(&sg->objcores, oc, lru_list);
 		oc->stobj->priv2 |= NEED_FIXUP;
 		oc->ban = BAN_RefBan(oc, so->ban);
-		oc->exp = so->exp;
+		EXP_COPY(&oc->exp, so);
 		sg->nobj++;
 		oc->refcnt++;
 		HSH_Insert(wrk, so->hash, oc);
@@ -439,8 +439,8 @@ smp_sml_getobj(struct worker *wrk, struct objcore *oc)
 			bad |= 0x100;
 
 		if(bad) {
-			EXP_Clr(&oc->exp);
-			EXP_Clr(&so->exp);
+			EXP_ZERO(&oc->exp);
+			EXP_ZERO(so);
 		}
 
 		sg->nfixed++;
@@ -470,11 +470,11 @@ smp_oc_objupdatemeta(struct worker *wrk, struct objcore *oc)
 		/* Lock necessary, we might race close_seg */
 		Lck_Lock(&sg->sc->mtx);
 		so->ban = BAN_Time(oc->ban);
-		so->exp = oc->exp;
+		EXP_COPY(so, &oc->exp);
 		Lck_Unlock(&sg->sc->mtx);
 	} else {
 		so->ban = BAN_Time(oc->ban);
-		so->exp = oc->exp;
+		EXP_COPY(so, &oc->exp);
 	}
 }
 
@@ -491,7 +491,7 @@ smp_oc_objfree(struct worker *wrk, struct objcore *oc)
 	so = smp_find_so(sg, oc->stobj->priv2);
 
 	Lck_Lock(&sg->sc->mtx);
-	EXP_Clr(&so->exp);
+	EXP_ZERO(so);
 	so->ptr = 0;
 
 	assert(sg->nobj > 0);



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] da08539 Change EXP_Ttl() to take objcore argument.
Message-ID: <E1aSw1Q-0007NR-9j@project.varnish-software.com>


commit da0853992588103b55df646c453f3592b0cd80ee
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 22:27:19 2016 +0000

    Change EXP_Ttl() to take objcore argument.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 7ca3eb7..387aa01 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -721,7 +721,7 @@ extern pthread_t cli_thread;
 #define EXP_WHEN(to)							\
 	((to)->t_origin + (to)->ttl + (to)->grace + (to)->keep)
 
-double EXP_Ttl(const struct req *, const struct exp*);
+double EXP_Ttl(const struct req *, const struct objcore *);
 void EXP_Insert(struct worker *wrk, struct objcore *oc);
 void EXP_Rearm(struct objcore *, double now, double ttl, double grace,
     double keep);
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 10f5f82..590bc02 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -92,14 +92,16 @@ exp_event(struct worker *wrk, struct objcore *oc, enum exp_event_e e)
  */
 
 double
-EXP_Ttl(const struct req *req, const struct exp *e)
+EXP_Ttl(const struct req *req, const struct objcore *oc)
 {
 	double r;
 
-	r = e->ttl;
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+
+	r = oc->exp.ttl;
 	if (req != NULL && req->d_ttl > 0. && req->d_ttl < r)
 		r = req->d_ttl;
-	return (e->t_origin + r);
+	return (oc->exp.t_origin + r);
 }
 
 /*--------------------------------------------------------------------
@@ -334,7 +336,7 @@ exp_expire(struct exp_priv *ep, double now)
 
 	CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
 	VSLb(&ep->vsl, SLT_ExpKill, "EXP_Expired x=%u t=%.0f",
-	    ObjGetXID(ep->wrk, oc), EXP_Ttl(NULL, &oc->exp) - now);
+	    ObjGetXID(ep->wrk, oc), EXP_Ttl(NULL, oc) - now);
 	exp_event(ep->wrk, oc, EXP_REMOVE);
 	(void)HSH_DerefObjCore(ep->wrk, &oc);
 	return (0);
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index d58e070..a8f438b 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -406,7 +406,7 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 				continue;
 		}
 
-		if (EXP_Ttl(req, &oc->exp) >= req->t_req) {
+		if (EXP_Ttl(req, oc) >= req->t_req) {
 			/* If still valid, use it */
 			assert(oh->refcnt > 1);
 			assert(oc->objhead == oh);



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] dadd878 Get rid of struct exp entirely
Message-ID: <E1aSw1Q-0007Na-B4@project.varnish-software.com>


commit dadd87802ee41822ab3d5856aec494ffcf3f96ba
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 22:42:07 2016 +0000

    Get rid of struct exp entirely

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 387aa01..a9cd4fd 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -273,15 +273,6 @@ struct dstat {
 
 /*--------------------------------------------------------------------*/
 
-struct exp {
-	double			t_origin;
-	float			ttl;
-	float			grace;
-	float			keep;
-};
-
-/*--------------------------------------------------------------------*/
-
 struct vsl_log {
 	uint32_t		*wlb, *wlp, *wle;
 	unsigned		wlr;
@@ -420,7 +411,10 @@ struct objcore {
 	double			timer_when;
 	long			hits;
 
-	struct exp		exp;
+	double			t_origin;
+	float			ttl;
+	float			grace;
+	float			keep;
 
 	uint8_t			flags;
 #define OC_F_BUSY		(1<<1)
diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 3f70c42..d79f75b 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -576,7 +576,7 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req)
 	} else {
 		VSLb(vsl, SLT_ExpBan, "%u banned lookup", ObjGetXID(wrk, oc));
 		VSC_C_main->bans_obj_killed++;
-		EXP_Rearm(oc, oc->exp.t_origin, 0, 0, 0);	// XXX fake now
+		EXP_Rearm(oc, oc->t_origin, 0, 0, 0);	// XXX fake now
 		return (1);
 	}
 }
diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
index e913fda..0c48da1 100644
--- a/bin/varnishd/cache/cache_ban_lurker.c
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -181,7 +181,7 @@ ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt,
 			VSLb(vsl, SLT_ExpBan, "%u banned by lurker",
 			    ObjGetXID(wrk, oc));
 
-			EXP_Rearm(oc, oc->exp.t_origin, 0, 0, 0);
+			EXP_Rearm(oc, oc->t_origin, 0, 0, 0);
 					// XXX ^ fake now
 			VSC_C_main->bans_lurker_obj_killed++;
 		} else {
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 590bc02..74e1bfc 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -98,26 +98,10 @@ EXP_Ttl(const struct req *req, const struct objcore *oc)
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
-	r = oc->exp.ttl;
+	r = oc->ttl;
 	if (req != NULL && req->d_ttl > 0. && req->d_ttl < r)
 		r = req->d_ttl;
-	return (oc->exp.t_origin + r);
-}
-
-/*--------------------------------------------------------------------
- * Calculate when this object is no longer useful
- */
-
-static double
-EXP_When(const struct exp *e)
-{
-	double when;
-
-	if (e->t_origin == 0)
-		return (0.);
-	when = e->t_origin + e->ttl + e->grace + e->keep;
-	AZ(isnan(when));
-	return (when);
+	return (oc->t_origin + r);
 }
 
 /*--------------------------------------------------------------------
@@ -193,18 +177,18 @@ EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep)
 	AN(oc->exp_flags & OC_EF_EXP);
 
 	if (!isnan(ttl))
-		oc->exp.ttl = now + ttl - oc->exp.t_origin;
+		oc->ttl = now + ttl - oc->t_origin;
 	if (!isnan(grace))
-		oc->exp.grace = grace;
+		oc->grace = grace;
 	if (!isnan(keep))
-		oc->exp.keep = keep;
+		oc->keep = keep;
 
-	when = EXP_When(&oc->exp);
+	when = EXP_WHEN(oc);
 
 	VSL(SLT_ExpKill, 0, "EXP_Rearm p=%p E=%.9f e=%.9f f=0x%x", oc,
 	    oc->timer_when, when, oc->flags);
 
-	if (when < oc->exp.t_origin || when < oc->timer_when)
+	if (when < oc->t_origin || when < oc->timer_when)
 		exp_mail_it(oc, OC_EF_MOVE);
 }
 
@@ -275,7 +259,7 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, unsigned flags)
 	}
 
 	if (flags & OC_EF_MOVE) {
-		oc->timer_when = EXP_When(&oc->exp);
+		oc->timer_when = EXP_WHEN(oc);
 		ObjUpdateMeta(ep->wrk, oc);
 	}
 
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 57295da..3b9e705 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -53,7 +53,7 @@ vbf_allocobj(struct busyobj *bo, unsigned l)
 	oc = bo->fetch_objcore;
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 
-	lifetime = oc->exp.ttl + oc->exp.grace + oc->exp.keep;
+	lifetime = oc->ttl + oc->grace + oc->keep;
 
 	if (bo->uncacheable || lifetime < cache_param->shortlived)
 		storage_hint = TRANSIENT_STORAGE;
@@ -73,10 +73,10 @@ vbf_allocobj(struct busyobj *bo, unsigned l)
 	 * on Transient storage.
 	 */
 
-	if (oc->exp.ttl > cache_param->shortlived)
-		oc->exp.ttl = cache_param->shortlived;
-	oc->exp.grace = 0.0;
-	oc->exp.keep = 0.0;
+	if (oc->ttl > cache_param->shortlived)
+		oc->ttl = cache_param->shortlived;
+	oc->grace = 0.0;
+	oc->keep = 0.0;
 	return (STV_NewObject(bo->wrk, bo->fetch_objcore,
 	    TRANSIENT_STORAGE, l));
 }
@@ -150,7 +150,7 @@ vbf_beresp2obj(struct busyobj *bo)
 		    VTIM_parse(b)));
 	else
 		AZ(ObjSetDouble(bo->wrk, bo->fetch_objcore, OA_LASTMODIFIED,
-		    floor(bo->fetch_objcore->exp.t_origin)));
+		    floor(bo->fetch_objcore->t_origin)));
 
 	return (0);
 }
@@ -385,15 +385,15 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo)
 	 * What does RFC2616 think about TTL ?
 	 */
 	RFC2616_Ttl(bo, now,
-	    &bo->fetch_objcore->exp.t_origin,
-	    &bo->fetch_objcore->exp.ttl,
-	    &bo->fetch_objcore->exp.grace,
-	    &bo->fetch_objcore->exp.keep
+	    &bo->fetch_objcore->t_origin,
+	    &bo->fetch_objcore->ttl,
+	    &bo->fetch_objcore->grace,
+	    &bo->fetch_objcore->keep
 	);
 
 	/* private objects have negative TTL */
 	if (bo->fetch_objcore->flags & OC_F_PRIVATE)
-		bo->fetch_objcore->exp.ttl = -1.;
+		bo->fetch_objcore->ttl = -1.;
 
 	AZ(bo->do_esi);
 	AZ(bo->was_304);
@@ -689,7 +689,7 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 	ObjSetState(bo->fetch_objcore, BOS_FINISHED);
 	VSLb_ts_busyobj(bo, "BerespBody", W_TIM_real(wrk));
 	if (bo->stale_oc != NULL)
-		EXP_Rearm(bo->stale_oc, bo->stale_oc->exp.t_origin, 0, 0, 0);
+		EXP_Rearm(bo->stale_oc, bo->stale_oc->t_origin, 0, 0, 0);
 	return (F_STP_DONE);
 }
 
@@ -756,7 +756,7 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 	if (!bo->do_stream)
 		HSH_Unbusy(wrk, bo->fetch_objcore);
 
-	EXP_Rearm(bo->stale_oc, bo->stale_oc->exp.t_origin, 0, 0, 0);
+	EXP_Rearm(bo->stale_oc, bo->stale_oc->t_origin, 0, 0, 0);
 
 	/* Recycle the backend connection before setting BOS_FINISHED to
 	   give predictable backend reuse behavior for varnishtest */
@@ -798,7 +798,7 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
 	http_TimeHeader(bo->beresp, "Date: ", now);
 	http_SetHeader(bo->beresp, "Server: Varnish");
 
-	bo->fetch_objcore->exp.t_origin = now;
+	bo->fetch_objcore->t_origin = now;
 	if (!VTAILQ_EMPTY(&bo->fetch_objcore->objhead->waitinglist)) {
 		/*
 		 * If there is a waitinglist, it means that there is no
@@ -807,13 +807,13 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
 		 * each objcore on the waiting list sequentially attempt
 		 * to fetch from the backend.
 		 */
-		bo->fetch_objcore->exp.ttl = 1;
-		bo->fetch_objcore->exp.grace = 5;
-		bo->fetch_objcore->exp.keep = 5;
+		bo->fetch_objcore->ttl = 1;
+		bo->fetch_objcore->grace = 5;
+		bo->fetch_objcore->keep = 5;
 	} else {
-		bo->fetch_objcore->exp.ttl = 0;
-		bo->fetch_objcore->exp.grace = 0;
-		bo->fetch_objcore->exp.keep = 0;
+		bo->fetch_objcore->ttl = 0;
+		bo->fetch_objcore->grace = 0;
+		bo->fetch_objcore->keep = 0;
 	}
 
 	synth_body = VSB_new_auto();
@@ -883,7 +883,7 @@ vbf_stp_fail(struct worker *wrk, const struct busyobj *bo)
 		/* Already unbusied - expire it */
 		AN(bo->fetch_objcore);
 		EXP_Rearm(bo->fetch_objcore,
-		    bo->fetch_objcore->exp.t_origin, 0, 0, 0);
+		    bo->fetch_objcore->t_origin, 0, 0, 0);
 	}
 	wrk->stats->fetch_failed++;
 	ObjSetState(bo->fetch_objcore, BOS_FAILED);
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index a8f438b..7fbd283 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -392,7 +392,7 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 			continue;
 		}
 
-		if (oc->exp.ttl <= 0.)
+		if (oc->ttl <= 0.)
 			continue;
 
 		if (BAN_CheckObject(wrk, oc, req)) {
@@ -418,10 +418,10 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 			*ocp = oc;
 			return (HSH_HIT);
 		}
-		if (oc->exp.t_origin > exp_t_origin) {
+		if (oc->t_origin > exp_t_origin) {
 			/* record the newest object */
 			exp_oc = oc;
-			exp_t_origin = oc->exp.t_origin;
+			exp_t_origin = oc->t_origin;
 		}
 	}
 
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 79566af..0a85637 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -249,7 +249,7 @@ pan_objcore(struct vsb *vsb, const char *typ, const struct objcore *oc)
 	if (oc->boc != NULL)
 		pan_boc(vsb, oc->boc);
 	VSB_printf(vsb, "exp = { %f, %f, %f, %f }\n",
-	    oc->exp.t_origin, oc->exp.ttl, oc->exp.grace, oc->exp.keep);
+	    oc->t_origin, oc->ttl, oc->grace, oc->keep);
 	VSB_printf(vsb, "objhead = %p,\n", oc->objhead);
 	VSB_printf(vsb, "stevedore = %p", oc->stobj->stevedore);
 	if (oc->stobj->stevedore != NULL) {
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index fa8f373..04c4e7f 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -91,7 +91,7 @@ cnt_deliver(struct worker *wrk, struct req *req)
 	 * age. Truncate to zero in that case).
 	 */
 	http_PrintfHeader(req->resp, "Age: %.0f",
-	    fmax(0., req->t_prev - req->objcore->exp.t_origin));
+	    fmax(0., req->t_prev - req->objcore->t_origin));
 
 	http_SetHeader(req->resp, "Via: 1.1 varnish-v4");
 
diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c
index cf25f67..442bd47 100644
--- a/bin/varnishd/cache/cache_vrt_var.c
+++ b/bin/varnishd/cache/cache_vrt_var.c
@@ -481,65 +481,65 @@ VRT_r_bereq_retries(VRT_CTX)
  *	grace&keep are relative to ttl
  */
 
-#define VRT_DO_EXP_L(which, sexp, fld, offset)			\
+#define VRT_DO_EXP_L(which, oc, fld, offset)			\
 								\
 void								\
-VRT_l_##which##_##fld(VRT_CTX, double a)	\
+VRT_l_##which##_##fld(VRT_CTX, double a)			\
 {								\
 								\
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);			\
 	a += (offset);						\
 	if (a < 0.0)						\
 		a = 0.0;					\
-	sexp.fld = a;						\
+	oc->fld = a;						\
 	VSLb(ctx->vsl, SLT_TTL, "VCL %.0f %.0f %.0f %.0f",	\
-	    sexp.ttl, sexp.grace, sexp.keep, sexp.t_origin);	\
+	    oc->ttl, oc->grace, oc->keep, oc->t_origin);	\
 }
 
-#define VRT_DO_EXP_R(which, sexp, fld, offset)			\
+#define VRT_DO_EXP_R(which, oc, fld, offset)			\
 								\
 double								\
-VRT_r_##which##_##fld(VRT_CTX)		\
+VRT_r_##which##_##fld(VRT_CTX)					\
 {								\
 	double d;						\
 								\
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);			\
-	d = sexp.fld;						\
+	d = oc->fld;						\
 	if (d <= 0.0)						\
 		d = 0.0;					\
 	d -= (offset);						\
 	return(d);						\
 }
 
-VRT_DO_EXP_R(obj, ctx->req->objcore->exp, ttl,
-    ctx->now - ctx->req->objcore->exp.t_origin)
-VRT_DO_EXP_R(obj, ctx->req->objcore->exp, grace, 0)
-VRT_DO_EXP_R(obj, ctx->req->objcore->exp, keep, 0)
+VRT_DO_EXP_R(obj, ctx->req->objcore, ttl,
+    ctx->now - ctx->req->objcore->t_origin)
+VRT_DO_EXP_R(obj, ctx->req->objcore, grace, 0)
+VRT_DO_EXP_R(obj, ctx->req->objcore, keep, 0)
 
-VRT_DO_EXP_L(beresp, ctx->bo->fetch_objcore->exp, ttl,
-    ctx->now - ctx->bo->fetch_objcore->exp.t_origin)
-VRT_DO_EXP_R(beresp, ctx->bo->fetch_objcore->exp, ttl,
-    ctx->now - ctx->bo->fetch_objcore->exp.t_origin)
-VRT_DO_EXP_L(beresp, ctx->bo->fetch_objcore->exp, grace, 0)
-VRT_DO_EXP_R(beresp, ctx->bo->fetch_objcore->exp, grace, 0)
-VRT_DO_EXP_L(beresp, ctx->bo->fetch_objcore->exp, keep, 0)
-VRT_DO_EXP_R(beresp, ctx->bo->fetch_objcore->exp, keep, 0)
+VRT_DO_EXP_L(beresp, ctx->bo->fetch_objcore, ttl,
+    ctx->now - ctx->bo->fetch_objcore->t_origin)
+VRT_DO_EXP_R(beresp, ctx->bo->fetch_objcore, ttl,
+    ctx->now - ctx->bo->fetch_objcore->t_origin)
+VRT_DO_EXP_L(beresp, ctx->bo->fetch_objcore, grace, 0)
+VRT_DO_EXP_R(beresp, ctx->bo->fetch_objcore, grace, 0)
+VRT_DO_EXP_L(beresp, ctx->bo->fetch_objcore, keep, 0)
+VRT_DO_EXP_R(beresp, ctx->bo->fetch_objcore, keep, 0)
 
 /*--------------------------------------------------------------------
  */
 
-#define VRT_DO_AGE_R(which, sexp)				\
+#define VRT_DO_AGE_R(which, oc)					\
 								\
 double								\
 VRT_r_##which##_##age(VRT_CTX)		\
 {								\
 								\
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);			\
-	return(ctx->now - sexp.t_origin);			\
+	return(ctx->now - oc->t_origin);			\
 }
 
-VRT_DO_AGE_R(obj, ctx->req->objcore->exp)
-VRT_DO_AGE_R(beresp, ctx->bo->fetch_objcore->exp)
+VRT_DO_AGE_R(obj, ctx->req->objcore)
+VRT_DO_AGE_R(beresp, ctx->bo->fetch_objcore)
 
 /*--------------------------------------------------------------------
  * [be]req.xid
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index bbc8a83..7ebb747 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -522,7 +522,11 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv,
 	assert(really >= 0);
 
 	/* Don't entertain already dead objects */
-	if ((oc->exp.ttl + oc->exp.grace + oc->exp.keep) <= 0.)
+	if (oc->flags & OC_F_DYING)
+		return (0);
+	if (oc->t_origin <= 0.)
+		return (0);
+	if (oc->ttl + oc->grace + oc->keep <= 0.)
 		return (0);
 
 	ltot = sizeof(struct object) + PRNDUP(wsl);
@@ -562,7 +566,7 @@ smp_allocobj(struct worker *wrk, const struct stevedore *stv,
 	/* We have to do this somewhere, might as well be here... */
 	assert(sizeof so->hash == DIGEST_LEN);
 	memcpy(so->hash, oc->objhead->digest, DIGEST_LEN);
-	EXP_COPY(so, &oc->exp);
+	EXP_COPY(so, oc);
 	so->ptr = (uint8_t*)o - sc->base;
 	so->ban = BAN_Time(oc->ban);
 
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 9710dab..e235c5f 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -166,7 +166,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		VTAILQ_INSERT_TAIL(&sg->objcores, oc, lru_list);
 		oc->stobj->priv2 |= NEED_FIXUP;
 		oc->ban = BAN_RefBan(oc, so->ban);
-		EXP_COPY(&oc->exp, so);
+		EXP_COPY(oc, so);
 		sg->nobj++;
 		oc->refcnt++;
 		HSH_Insert(wrk, so->hash, oc);
@@ -439,7 +439,7 @@ smp_sml_getobj(struct worker *wrk, struct objcore *oc)
 			bad |= 0x100;
 
 		if(bad) {
-			EXP_ZERO(&oc->exp);
+			EXP_ZERO(oc);
 			EXP_ZERO(so);
 		}
 
@@ -470,11 +470,11 @@ smp_oc_objupdatemeta(struct worker *wrk, struct objcore *oc)
 		/* Lock necessary, we might race close_seg */
 		Lck_Lock(&sg->sc->mtx);
 		so->ban = BAN_Time(oc->ban);
-		EXP_COPY(so, &oc->exp);
+		EXP_COPY(so, oc);
 		Lck_Unlock(&sg->sc->mtx);
 	} else {
 		so->ban = BAN_Time(oc->ban);
-		EXP_COPY(so, &oc->exp);
+		EXP_COPY(so, oc);
 	}
 }
 



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] f583b81 Remove unnessary EXP_Arm() call
Message-ID: <E1aSw1Q-0007Nk-CR@project.varnish-software.com>


commit f583b81fe08c01024cbddbdcf2faae790e1c476b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 22:58:34 2016 +0000

    Remove unnessary EXP_Arm() call

diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index e235c5f..e3cbe07 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -449,7 +449,6 @@ smp_sml_getobj(struct worker *wrk, struct objcore *oc)
 		oc->stobj->priv2 &= ~NEED_FIXUP;
 	}
 	Lck_Unlock(&sg->sc->mtx);
-	EXP_Rearm(oc, NAN, NAN, NAN, NAN);	// XXX: Shouldn't be needed
 	return (o);
 }
 



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] f7b57f5 Rename ObjSnipe() and ObjKill() to HSH_ditto since
 they don't actually call into the stevedores methods.
Message-ID: <E1aSw1Q-0007Nt-Dk@project.varnish-software.com>


commit f7b57f54142dff6c4e899569e08e22f56d4123af
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 23:22:57 2016 +0000

    Rename ObjSnipe() and ObjKill() to HSH_ditto since they don't actually
    call into the stevedores methods.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index a9cd4fd..3797f1d 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -882,8 +882,6 @@ void ObjSetState(const struct objcore *, enum boc_state_e next);
 void ObjWaitState(const struct objcore *, enum boc_state_e want);
 void ObjTrimStore(struct worker *, struct objcore *);
 void ObjTouch(struct worker *, struct objcore *, double now);
-int ObjSnipe(const struct worker *, struct objcore *);
-void ObjKill(struct objcore *);
 unsigned ObjGetXID(struct worker *, struct objcore *);
 uint64_t ObjGetLen(struct worker *, struct objcore *);
 void ObjUpdateMeta(struct worker *, struct objcore *);
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 74e1bfc..2e71c83 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -311,7 +311,7 @@ exp_expire(struct exp_priv *ep, double now)
 	VSC_C_main->n_expired++;
 
 	if (!(oc->flags & OC_F_DYING))
-		ObjKill(oc);
+		HSH_Kill(oc);
 
 	/* Remove from binheap */
 	assert(oc->timer_idx != BINHEAP_NOIDX);
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 7fbd283..0e88dab 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -671,6 +671,53 @@ HSH_Unbusy(struct worker *wrk, struct objcore *oc)
 	Lck_Unlock(&oh->mtx);
 }
 
+/*====================================================================
+ * HSH_Kill()
+ *
+ * It's dead Jim, kick it...
+ */
+
+void
+HSH_Kill(struct objcore *oc)
+{
+
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
+
+	Lck_Lock(&oc->objhead->mtx);
+	oc->flags |= OC_F_DYING;
+	Lck_Unlock(&oc->objhead->mtx);
+}
+
+/*====================================================================
+ * HSH_Snipe()
+ *
+ * If objcore is idle, gain a ref and mark it dead.
+ */
+
+int
+HSH_Snipe(const struct worker *wrk, struct objcore *oc)
+{
+	int retval = 0;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
+
+	AZ(oc->flags & OC_F_DYING);
+
+	if (oc->refcnt == 1 && !Lck_Trylock(&oc->objhead->mtx)) {
+		if (oc->refcnt == 1) {
+			oc->flags |= OC_F_DYING;
+			oc->refcnt++;
+			retval = 1;
+		}
+		Lck_Unlock(&oc->objhead->mtx);
+	}
+	return (retval);
+}
+
+
 /*---------------------------------------------------------------------
  * Gain a reference on an objcore
  */
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index d0ae3db..1fa8d01 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -71,8 +71,8 @@
  *
  * 23	ObjUpdateMeta()	ban/ttl/grace/keep changed
  *
- * 3->4	ObjSnipe()	kill if not in use
- * 3->4	ObjKill()	make unavailable
+ * 3->4	HSH_Snipe()	kill if not in use
+ * 3->4	HSH_Kill()	make unavailable
  *
  * 234	ObjSlim()	Release body storage (but retain attribute storage)
  *
@@ -428,52 +428,6 @@ ObjTouch(struct worker *wrk, struct objcore *oc, double now)
 }
 
 /*====================================================================
- * ObjKill()
- *
- * It's dead Jim, kick it...
- */
-
-void
-ObjKill(struct objcore *oc)
-{
-
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
-
-	Lck_Lock(&oc->objhead->mtx);
-	oc->flags |= OC_F_DYING;
-	Lck_Unlock(&oc->objhead->mtx);
-}
-
-/*====================================================================
- * ObjSnipe()
- *
- * If objcore is idle, gain a ref and mark it dead.
- */
-
-int
-ObjSnipe(const struct worker *wrk, struct objcore *oc)
-{
-	int retval = 0;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
-
-	AZ(oc->flags & OC_F_DYING);
-
-	if (oc->refcnt == 1 && !Lck_Trylock(&oc->objhead->mtx)) {
-		if (oc->refcnt == 1) {
-			oc->flags |= OC_F_DYING;
-			oc->refcnt++;
-			retval = 1;
-		}
-		Lck_Unlock(&oc->objhead->mtx);
-	}
-	return (retval);
-}
-
-/*====================================================================
  * Utility functions which work on top of the previous ones
  */
 
diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h
index b4b7bcd..807c83d 100644
--- a/bin/varnishd/hash/hash_slinger.h
+++ b/bin/varnishd/hash/hash_slinger.h
@@ -75,6 +75,8 @@ struct boc *HSH_RefBoc(const struct objcore *);
 void HSH_DerefBoc(struct worker *wrk, struct objcore *);
 struct objcore *HSH_Private(struct worker *wrk);
 void HSH_Abandon(struct objcore *oc);
+int HSH_Snipe(const struct worker *, struct objcore *);
+void HSH_Kill(struct objcore *);
 
 #ifdef VARNISH_CACHE_CHILD
 
diff --git a/bin/varnishd/storage/storage_lru.c b/bin/varnishd/storage/storage_lru.c
index 987a0f1..785e3e1 100644
--- a/bin/varnishd/storage/storage_lru.c
+++ b/bin/varnishd/storage/storage_lru.c
@@ -175,7 +175,7 @@ LRU_NukeOne(struct worker *wrk, struct lru *lru)
 		VSLb(wrk->vsl, SLT_ExpKill, "LRU_Cand p=%p f=0x%x r=%d",
 		    oc, oc->flags, oc->refcnt);
 
-		if (ObjSnipe(wrk, oc)) {
+		if (HSH_Snipe(wrk, oc)) {
 			VSC_C_main->n_lru_nuked++; // XXX per lru ?
 			VTAILQ_REMOVE(&lru->lru_head, oc, lru_list);
 			VTAILQ_INSERT_TAIL(&lru->lru_head, oc, lru_list);



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] 94693bf Properly update the OC_EF_EXP flag and only poke if
 it is set.
Message-ID: <E1aSw1Q-0007O3-F1@project.varnish-software.com>


commit 94693bf97407a990a931827a81cbdd0a5bd2cace
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 23:31:38 2016 +0000

    Properly update the OC_EF_EXP flag and only poke if it is set.

diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 2e71c83..40fe1a1 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -135,7 +135,8 @@ EXP_Poke(struct objcore *oc)
 {
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	exp_mail_it(oc, 0);
+	if (oc->exp_flags & OC_EF_EXP)
+		exp_mail_it(oc, 0);
 }
 
 /*--------------------------------------------------------------------
@@ -246,6 +247,7 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, unsigned flags)
 	    oc->timer_when, oc->flags);
 
 	if (oc->flags & OC_F_DYING) {
+		oc->exp_flags &= OC_EF_EXP;
 		VSLb(&ep->vsl, SLT_ExpKill, "EXP_Kill p=%p e=%.9f f=0x%x", oc,
 		    oc->timer_when, oc->flags);
 		if (!(flags & OC_EF_INSERT)) {
@@ -309,6 +311,7 @@ exp_expire(struct exp_priv *ep, double now)
 		return (oc->timer_when);
 
 	VSC_C_main->n_expired++;
+	oc->exp_flags &= OC_EF_EXP;
 
 	if (!(oc->flags & OC_F_DYING))
 		HSH_Kill(oc);



From phk at FreeBSD.org  Tue Feb  9 00:19:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 01:19:12 +0100
Subject: [master] d584388 Use HSH_Kill() instead of EXP_Rearm().
Message-ID: <E1aSw1Q-0007OC-Gh@project.varnish-software.com>


commit d584388d763ab42a28a15bce82b3af83afea5ac3
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 00:18:49 2016 +0000

    Use HSH_Kill() instead of EXP_Rearm().

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index d79f75b..0b333a9 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -576,7 +576,8 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req)
 	} else {
 		VSLb(vsl, SLT_ExpBan, "%u banned lookup", ObjGetXID(wrk, oc));
 		VSC_C_main->bans_obj_killed++;
-		EXP_Rearm(oc, oc->t_origin, 0, 0, 0);	// XXX fake now
+		oc->flags |= OC_F_DYING;
+		EXP_Poke(oc);
 		return (1);
 	}
 }
diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
index 0c48da1..0d56883 100644
--- a/bin/varnishd/cache/cache_ban_lurker.c
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -181,8 +181,7 @@ ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt,
 			VSLb(vsl, SLT_ExpBan, "%u banned by lurker",
 			    ObjGetXID(wrk, oc));
 
-			EXP_Rearm(oc, oc->t_origin, 0, 0, 0);
-					// XXX ^ fake now
+			HSH_Kill(oc);
 			VSC_C_main->bans_lurker_obj_killed++;
 		} else {
 			if (oc->ban != bd) {
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 40fe1a1..9553841 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -121,6 +121,7 @@ exp_mail_it(struct objcore *oc, uint8_t cmds)
 			VSTAILQ_INSERT_TAIL(&exphdl->inbox, oc, exp_list);
 	}
 	oc->exp_flags |= cmds | OC_EF_POSTED;
+	AN(oc->exp_flags & OC_EF_EXP);
 	VSC_C_main->exp_mailed++;
 	AZ(pthread_cond_signal(&exphdl->condvar));
 	Lck_Unlock(&exphdl->mtx);
@@ -311,7 +312,7 @@ exp_expire(struct exp_priv *ep, double now)
 		return (oc->timer_when);
 
 	VSC_C_main->n_expired++;
-	oc->exp_flags &= OC_EF_EXP;
+	oc->exp_flags &= ~OC_EF_EXP;
 
 	if (!(oc->flags & OC_F_DYING))
 		HSH_Kill(oc);
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 3b9e705..c168108 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -689,7 +689,7 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 	ObjSetState(bo->fetch_objcore, BOS_FINISHED);
 	VSLb_ts_busyobj(bo, "BerespBody", W_TIM_real(wrk));
 	if (bo->stale_oc != NULL)
-		EXP_Rearm(bo->stale_oc, bo->stale_oc->t_origin, 0, 0, 0);
+		HSH_Kill(bo->stale_oc);
 	return (F_STP_DONE);
 }
 
@@ -756,7 +756,7 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 	if (!bo->do_stream)
 		HSH_Unbusy(wrk, bo->fetch_objcore);
 
-	EXP_Rearm(bo->stale_oc, bo->stale_oc->t_origin, 0, 0, 0);
+	HSH_Kill(bo->stale_oc);
 
 	/* Recycle the backend connection before setting BOS_FINISHED to
 	   give predictable backend reuse behavior for varnishtest */
@@ -882,8 +882,7 @@ vbf_stp_fail(struct worker *wrk, const struct busyobj *bo)
 	if (bo->fetch_objcore->exp_flags & OC_EF_EXP) {
 		/* Already unbusied - expire it */
 		AN(bo->fetch_objcore);
-		EXP_Rearm(bo->fetch_objcore,
-		    bo->fetch_objcore->t_origin, 0, 0, 0);
+		HSH_Kill(bo->fetch_objcore);
 	}
 	wrk->stats->fetch_failed++;
 	ObjSetState(bo->fetch_objcore, BOS_FAILED);
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 0e88dab..008d296 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -654,9 +654,8 @@ HSH_Unbusy(struct worker *wrk, struct objcore *oc)
 
 	if (!(oc->flags & OC_F_PRIVATE)) {
 		BAN_NewObjCore(oc);
-		EXP_Insert(wrk, oc);
-		AN(oc->exp_flags & OC_EF_EXP);
 		AN(oc->ban);
+		EXP_Insert(wrk, oc);
 	}
 
 	/* XXX: pretouch neighbors on oh->objcs to prevent page-on under mtx */
@@ -687,6 +686,7 @@ HSH_Kill(struct objcore *oc)
 	Lck_Lock(&oc->objhead->mtx);
 	oc->flags |= OC_F_DYING;
 	Lck_Unlock(&oc->objhead->mtx);
+	EXP_Poke(oc);
 }
 
 /*====================================================================



From phk at FreeBSD.org  Tue Feb  9 09:40:56 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 10:40:56 +0100
Subject: [master] 992d525 Merge the EXP_Register_Callback() and
 ObjUpdateMeta() notification mechanisms into a single general
 ObjSubscribeEvents() mechanism.
Message-ID: <E1aT4n2-0000Dm-QY@project.varnish-software.com>


commit 992d5250bfad375fad376a20b7fdaafc834d1ee0
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 09:39:53 2016 +0000

    Merge the EXP_Register_Callback() and ObjUpdateMeta() notification
    mechanisms into a single general ObjSubscribeEvents() mechanism.
    
    For efficiency, only the events subscribed to will be delivered.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 3797f1d..eb16777 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -721,17 +721,6 @@ void EXP_Rearm(struct objcore *, double now, double ttl, double grace,
     double keep);
 void EXP_Poke(struct objcore *);
 
-enum exp_event_e {
-	EXP_INSERT,
-	EXP_INJECT,
-	EXP_REMOVE,
-};
-typedef void exp_callback_f(struct worker *, struct objcore *,
-    enum exp_event_e, void *priv);
-
-uintptr_t EXP_Register_Callback(exp_callback_f *func, void *priv);
-void EXP_Deregister_Callback(uintptr_t*);
-
 /* cache_fetch.c */
 enum vbf_fetch_mode_e {
 	VBF_NORMAL = 0,
@@ -884,7 +873,6 @@ void ObjTrimStore(struct worker *, struct objcore *);
 void ObjTouch(struct worker *, struct objcore *, double now);
 unsigned ObjGetXID(struct worker *, struct objcore *);
 uint64_t ObjGetLen(struct worker *, struct objcore *);
-void ObjUpdateMeta(struct worker *, struct objcore *);
 void ObjFreeObj(struct worker *, struct objcore *);
 void ObjSlim(struct worker *, struct objcore *);
 int ObjHasAttr(struct worker *, struct objcore *, enum obj_attr);
@@ -907,6 +895,20 @@ int ObjGetU64(struct worker *, struct objcore *, enum obj_attr, uint64_t *);
 int ObjCheckFlag(struct worker *, struct objcore *, enum obj_flags of);
 void ObjSetFlag(struct worker *, struct objcore *, enum obj_flags of, int val);
 
+#define OEV_BANCHG	(1U<<1)
+#define OEV_TTLCHG	(1U<<2)
+#define OEV_INSERT	(1U<<3)
+#define OEV_REMOVE	(1U<<4)
+
+#define OEV_MASK (OEV_BANCHG|OEV_TTLCHG|OEV_INSERT|OEV_REMOVE)
+
+typedef void obj_event_f(struct worker *, void *priv, struct objcore *,
+    unsigned);
+
+uintptr_t ObjSubscribeEvents(obj_event_f *, void *, unsigned mask);
+void ObjUnsubscribeEvents(uintptr_t *);
+void ObjSendEvent(struct worker *, struct objcore *oc, unsigned event);
+
 /* cache_panic.c */
 const char *body_status_2str(enum body_status e);
 const char *sess_close_2str(enum sess_close sc, int want_desc);
diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 0b333a9..f091abf 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -571,7 +571,7 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req)
 
 	if (b == NULL) {
 		/* not banned */
-		ObjUpdateMeta(wrk, oc);
+		ObjSendEvent(wrk, oc, OEV_BANCHG);
 		return (0);
 	} else {
 		VSLb(vsl, SLT_ExpBan, "%u banned lookup", ObjGetXID(wrk, oc));
diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
index 0d56883..3e355c1 100644
--- a/bin/varnishd/cache/cache_ban_lurker.c
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -192,7 +192,7 @@ ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt,
 				bd->refcount++;
 				VTAILQ_INSERT_TAIL(&bd->objcore, oc, ban_list);
 				Lck_Unlock(&ban_mtx);
-				ObjUpdateMeta(wrk, oc);
+				ObjSendEvent(wrk, oc, OEV_BANCHG);
 			}
 		}
 		(void)HSH_DerefObjCore(wrk, &oc);
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 9553841..6cc92dd 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -40,14 +40,6 @@
 #include "hash/hash_slinger.h"
 #include "vtim.h"
 
-struct exp_callback {
-	unsigned			magic;
-#define EXP_CALLBACK_MAGIC		0xab956eb1
-	exp_callback_f			*func;
-	void				*priv;
-	VTAILQ_ENTRY(exp_callback)	list;
-};
-
 struct exp_priv {
 	unsigned			magic;
 #define EXP_PRIV_MAGIC			0x9db22482
@@ -60,32 +52,11 @@ struct exp_priv {
 	struct binheap			*heap;
 	pthread_cond_t			condvar;
 
-	VTAILQ_HEAD(,exp_callback)	ecb_list;
 	pthread_rwlock_t		cb_rwl;
 };
 
 static struct exp_priv *exphdl;
 
-static void
-exp_event(struct worker *wrk, struct objcore *oc, enum exp_event_e e)
-{
-	struct exp_callback *cb;
-
-	/*
-	 * Strictly speaking this is not atomic, but neither is VMOD
-	 * loading in general, so this is a fair optimization
-	 */
-	if (VTAILQ_EMPTY(&exphdl->ecb_list))
-		return;
-
-	AZ(pthread_rwlock_rdlock(&exphdl->cb_rwl));
-	VTAILQ_FOREACH(cb, &exphdl->ecb_list, list) {
-		CHECK_OBJ_NOTNULL(cb, EXP_CALLBACK_MAGIC);
-		cb->func(wrk, oc, e, cb->priv);
-	}
-	AZ(pthread_rwlock_unlock(&exphdl->cb_rwl));
-}
-
 /*--------------------------------------------------------------------
  * Calculate an objects effective ttl time, taking req.ttl into account
  * if it is available.
@@ -159,7 +130,7 @@ EXP_Insert(struct worker *wrk, struct objcore *oc)
 
 	HSH_Ref(oc);
 
-	exp_event(wrk, oc, EXP_INSERT);
+	ObjSendEvent(wrk, oc, OEV_INSERT);
 	exp_mail_it(oc, OC_EF_INSERT | OC_EF_EXP | OC_EF_MOVE);
 }
 
@@ -194,45 +165,6 @@ EXP_Rearm(struct objcore *oc, double now, double ttl, double grace, double keep)
 		exp_mail_it(oc, OC_EF_MOVE);
 }
 
-/*--------------------------------------------------------------------*/
-
-uintptr_t
-EXP_Register_Callback(exp_callback_f *func, void *priv)
-{
-	struct exp_callback *ecb;
-
-	AN(func);
-
-	ALLOC_OBJ(ecb, EXP_CALLBACK_MAGIC);
-	AN(ecb);
-	ecb->func = func;
-	ecb->priv = priv;
-	AZ(pthread_rwlock_wrlock(&exphdl->cb_rwl));
-	VTAILQ_INSERT_TAIL(&exphdl->ecb_list, ecb, list);
-	AZ(pthread_rwlock_unlock(&exphdl->cb_rwl));
-	return ((uintptr_t)ecb);
-}
-
-void
-EXP_Deregister_Callback(uintptr_t *handle)
-{
-	struct exp_callback *ecb;
-
-	AN(handle);
-	AN(*handle);
-	AZ(pthread_rwlock_wrlock(&exphdl->cb_rwl));
-	VTAILQ_FOREACH(ecb, &exphdl->ecb_list, list) {
-		CHECK_OBJ_NOTNULL(ecb, EXP_CALLBACK_MAGIC);
-		if ((uintptr_t)ecb == *handle)
-			break;
-	}
-	AN(ecb);
-	VTAILQ_REMOVE(&exphdl->ecb_list, ecb, list);
-	AZ(pthread_rwlock_unlock(&exphdl->cb_rwl));
-	FREE_OBJ(ecb);
-	*handle = 0;
-}
-
 /*--------------------------------------------------------------------
  * Handle stuff in the inbox
  */
@@ -256,14 +188,14 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, unsigned flags)
 			binheap_delete(ep->heap, oc->timer_idx);
 		}
 		assert(oc->timer_idx == BINHEAP_NOIDX);
-		exp_event(ep->wrk, oc, EXP_REMOVE);
+		ObjSendEvent(ep->wrk, oc, OEV_REMOVE);
 		(void)HSH_DerefObjCore(ep->wrk, &oc);
 		return;
 	}
 
 	if (flags & OC_EF_MOVE) {
 		oc->timer_when = EXP_WHEN(oc);
-		ObjUpdateMeta(ep->wrk, oc);
+		ObjSendEvent(ep->wrk, oc, OEV_TTLCHG);
 	}
 
 	VSLb(&ep->vsl, SLT_ExpKill, "EXP_When p=%p e=%.9f f=0x%x", oc,
@@ -325,7 +257,7 @@ exp_expire(struct exp_priv *ep, double now)
 	CHECK_OBJ_NOTNULL(oc->objhead, OBJHEAD_MAGIC);
 	VSLb(&ep->vsl, SLT_ExpKill, "EXP_Expired x=%u t=%.0f",
 	    ObjGetXID(ep->wrk, oc), EXP_Ttl(NULL, oc) - now);
-	exp_event(ep->wrk, oc, EXP_REMOVE);
+	ObjSendEvent(ep->wrk, oc, OEV_REMOVE);
 	(void)HSH_DerefObjCore(ep->wrk, &oc);
 	return (0);
 }
@@ -412,7 +344,6 @@ EXP_Init(void)
 	AZ(pthread_cond_init(&ep->condvar, NULL));
 	VSTAILQ_INIT(&ep->inbox);
 	AZ(pthread_rwlock_init(&ep->cb_rwl, NULL));
-	VTAILQ_INIT(&ep->ecb_list);
 	exphdl = ep;
 	WRK_BgThread(&pt, "cache-timeout", exp_thread, ep);
 }
diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c
index 78ab1d2..15bd0ff 100644
--- a/bin/varnishd/cache/cache_main.c
+++ b/bin/varnishd/cache/cache_main.c
@@ -236,6 +236,8 @@ child_main(void)
 	PAN_Init();
 	VFP_Init();
 
+	ObjInit();
+
 	VCL_Init();
 
 	HTTP_Init();
@@ -252,7 +254,6 @@ child_main(void)
 
 	VCA_Init();
 
-	SMP_Init();
 	STV_open();
 
 	VMOD_Init();
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 1fa8d01..cb32f97 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -69,8 +69,6 @@
  *
  * 23	ObjTouch()	Signal to LRU(-like) facilities
  *
- * 23	ObjUpdateMeta()	ban/ttl/grace/keep changed
- *
  * 3->4	HSH_Snipe()	kill if not in use
  * 3->4	HSH_Kill()	make unavailable
  *
@@ -311,19 +309,6 @@ ObjSlim(struct worker *wrk, struct objcore *oc)
 }
 
 /*====================================================================
- */
-void
-ObjUpdateMeta(struct worker *wrk, struct objcore *oc)
-{
-	const struct obj_methods *m = obj_getmethods(oc);
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-
-	if (m->objupdatemeta != NULL)
-		m->objupdatemeta(wrk, oc);
-}
-
-/*====================================================================
  * Called when the boc used to populate the objcore is going away.
  * Useful for releasing any leftovers from Trim.
  */
@@ -584,3 +569,96 @@ ObjSetFlag(struct worker *wrk, struct objcore *oc, enum obj_flags of, int val)
 	else
 		(*fp) &= ~of;
 }
+
+/*====================================================================
+ * Object event subscribtion mechanism.
+ *
+ * XXX: it is extremely unclear what the locking circumstances are here.
+ */
+
+struct oev_entry {
+	unsigned			magic;
+#define OEV_MAGIC			0xb0b7c5a1
+	unsigned			mask;
+	obj_event_f			*func;
+	void				*priv;
+	VTAILQ_ENTRY(oev_entry)		list;
+};
+
+static VTAILQ_HEAD(,oev_entry)		oev_list;
+static pthread_rwlock_t			oev_rwl;
+static unsigned				oev_mask;
+
+uintptr_t
+ObjSubscribeEvents(obj_event_f *func, void *priv, unsigned mask)
+{
+	struct oev_entry *oev;
+
+	AN(func);
+	AZ(mask & ~OEV_MASK);
+
+	ALLOC_OBJ(oev, OEV_MAGIC);
+	AN(oev);
+	oev->func = func;
+	oev->priv = priv;
+	oev->mask = mask;
+	AZ(pthread_rwlock_wrlock(&oev_rwl));
+	VTAILQ_INSERT_TAIL(&oev_list, oev, list);
+	oev_mask |= mask;
+	AZ(pthread_rwlock_unlock(&oev_rwl));
+	return ((uintptr_t)oev);
+}
+
+void
+ObjUnsubscribeEvents(uintptr_t *handle)
+{
+	struct oev_entry *oev, *oev2 = NULL;
+	unsigned newmask = 0;
+
+	AN(handle);
+	AN(*handle);
+	AZ(pthread_rwlock_wrlock(&oev_rwl));
+	VTAILQ_FOREACH(oev, &oev_list, list) {
+		CHECK_OBJ_NOTNULL(oev, OEV_MAGIC);
+		if ((uintptr_t)oev == *handle)
+			oev2 = oev;
+		else
+			newmask |= oev->mask;
+	}
+	AN(oev2);
+	VTAILQ_REMOVE(&oev_list, oev2, list);
+	oev_mask = newmask;
+	AZ(newmask & ~OEV_MASK);
+	AZ(pthread_rwlock_unlock(&oev_rwl));
+	FREE_OBJ(oev2);
+	*handle = 0;
+}
+
+void
+ObjSendEvent(struct worker *wrk, struct objcore *oc, unsigned event)
+{
+	struct oev_entry *oev;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AN(event & OEV_MASK);
+	AZ(event & ~OEV_MASK);
+	if (!(event & oev_mask))
+		return;
+
+	AZ(pthread_rwlock_rdlock(&oev_rwl));
+	VTAILQ_FOREACH(oev, &oev_list, list) {
+		CHECK_OBJ_NOTNULL(oev, OEV_MAGIC);
+		if (event & oev->mask)
+			oev->func(wrk, oev->priv, oc, event);
+	}
+	AZ(pthread_rwlock_unlock(&oev_rwl));
+
+}
+
+void
+ObjInit(void)
+{
+	VTAILQ_INIT(&oev_list);
+	AZ(pthread_rwlock_init(&oev_rwl, NULL));
+}
diff --git a/bin/varnishd/cache/cache_obj.h b/bin/varnishd/cache/cache_obj.h
index 41e3ae5..c4eea09 100644
--- a/bin/varnishd/cache/cache_obj.h
+++ b/bin/varnishd/cache/cache_obj.h
@@ -30,7 +30,6 @@
 
 /* Methods on objcore ------------------------------------------------*/
 
-typedef void objupdatemeta_f(struct worker *, struct objcore *);
 typedef void objfree_f(struct worker *, struct objcore *);
 
 /* This method is only used by SML (...to get to persistent) */
@@ -51,8 +50,6 @@ typedef void objtouch_f(struct worker *, struct objcore *, double now);
 
 struct obj_methods {
 	objfree_f	*objfree;
-	objupdatemeta_f	*objupdatemeta;
-
 	objiterator_f	*objiterator;
 	objgetspace_f	*objgetspace;
 	objextend_f	*objextend;
diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h
index 7dd941e..3f64c8e 100644
--- a/bin/varnishd/cache/cache_priv.h
+++ b/bin/varnishd/cache/cache_priv.h
@@ -98,6 +98,9 @@ struct req * THR_GetRequest(void);
 /* cache_lck.c */
 void LCK_Init(void);
 
+/* cache_obj.c */
+void ObjInit(void);
+
 /* cache_panic.c */
 void PAN_Init(void);
 
@@ -134,6 +137,5 @@ int STV_BanInfoNew(const uint8_t *ban, unsigned len);
 void STV_BanExport(const uint8_t *banlist, unsigned len);
 
 /* storage_persistent.c */
-void SMP_Init(void);
 void SMP_Ready(void);
 
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index 7ebb747..ba1a450 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -59,6 +59,8 @@ static struct obj_methods smp_oc_realmethods;
 
 static struct VSC_C_lck *lck_smp;
 
+static void smp_init(void);
+
 /*--------------------------------------------------------------------*/
 
 /*
@@ -326,6 +328,9 @@ smp_open(struct stevedore *st)
 
 	ASSERT_CLI();
 
+	if (VTAILQ_EMPTY(&silos))
+		smp_init();
+
 	CAST_OBJ_NOTNULL(sc, st->priv, SMP_SC_MAGIC);
 
 	Lck_New(&sc->mtx, lck_smp);
@@ -684,17 +689,19 @@ static struct cli_proto debug_cmds[] = {
 	{ NULL }
 };
 
-/*--------------------------------------------------------------------*/
+/*--------------------------------------------------------------------
+ */
 
-void
-SMP_Init(void)
+static void
+smp_init(void)
 {
 	lck_smp = Lck_CreateClass("smp");
 	CLI_AddFuncs(debug_cmds);
 	smp_oc_realmethods = SML_methods;
 	smp_oc_realmethods.objtouch = NULL;
-	smp_oc_realmethods.objupdatemeta = smp_oc_objupdatemeta;
 	smp_oc_realmethods.objfree = smp_oc_objfree;
+	(void)ObjSubscribeEvents(smp_oc_event, NULL,
+	    OEV_BANCHG|OEV_TTLCHG|OEV_INSERT);
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishd/storage/storage_persistent.h b/bin/varnishd/storage/storage_persistent.h
index 9edb47b..f7af273 100644
--- a/bin/varnishd/storage/storage_persistent.h
+++ b/bin/varnishd/storage/storage_persistent.h
@@ -311,8 +311,8 @@ void smp_close_seg(struct smp_sc *sc, struct smp_seg *sg);
 void smp_init_oc(struct objcore *oc, struct smp_seg *sg, unsigned objidx);
 void smp_save_segs(struct smp_sc *sc);
 sml_getobj_f smp_sml_getobj;
-void smp_oc_objupdatemeta(struct worker *, struct objcore *);
 void smp_oc_objfree(struct worker *, struct objcore *);
+obj_event_f smp_oc_event;
 
 /* storage_persistent_subr.c */
 
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index e3cbe07..78b7e82 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -452,31 +452,6 @@ smp_sml_getobj(struct worker *wrk, struct objcore *oc)
 	return (o);
 }
 
-void __match_proto__(objupdatemeta_f)
-smp_oc_objupdatemeta(struct worker *wrk, struct objcore *oc)
-{
-	struct smp_seg *sg;
-	struct smp_object *so;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-
-	CAST_OBJ_NOTNULL(sg, oc->stobj->priv, SMP_SEG_MAGIC);
-	CHECK_OBJ_NOTNULL(sg->sc, SMP_SC_MAGIC);
-	so = smp_find_so(sg, oc->stobj->priv2);
-
-	if (sg == sg->sc->cur_seg) {
-		/* Lock necessary, we might race close_seg */
-		Lck_Lock(&sg->sc->mtx);
-		so->ban = BAN_Time(oc->ban);
-		EXP_COPY(so, oc);
-		Lck_Unlock(&sg->sc->mtx);
-	} else {
-		so->ban = BAN_Time(oc->ban);
-		EXP_COPY(so, oc);
-	}
-}
-
 void __match_proto__(objfree_f)
 smp_oc_objfree(struct worker *wrk, struct objcore *oc)
 {
@@ -518,3 +493,36 @@ smp_init_oc(struct objcore *oc, struct smp_seg *sg, unsigned objidx)
 	oc->stobj->priv = sg;
 	oc->stobj->priv2 = objidx;
 }
+
+/*--------------------------------------------------------------------*/
+
+void __match_proto__(obj_event_f)
+smp_oc_event(struct worker *wrk, void *priv, struct objcore *oc, unsigned ev)
+{
+	struct smp_seg *sg;
+	struct smp_object *so;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	AZ(priv);
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+
+	CAST_OBJ_NOTNULL(sg, oc->stobj->priv, SMP_SEG_MAGIC);
+	CHECK_OBJ_NOTNULL(sg->sc, SMP_SC_MAGIC);
+	so = smp_find_so(sg, oc->stobj->priv2);
+
+	if (sg == sg->sc->cur_seg) {
+		/* Lock necessary, we might race close_seg */
+		Lck_Lock(&sg->sc->mtx);
+		if (ev & (OEV_BANCHG|OEV_INSERT))
+			so->ban = BAN_Time(oc->ban);
+		if (ev & (OEV_TTLCHG|OEV_INSERT))
+			EXP_COPY(so, oc);
+		Lck_Unlock(&sg->sc->mtx);
+	} else {
+		if (ev & (OEV_BANCHG|OEV_INSERT))
+			so->ban = BAN_Time(oc->ban);
+		if (ev & (OEV_TTLCHG|OEV_INSERT))
+			EXP_COPY(so, oc);
+	}
+}
+
diff --git a/bin/varnishtest/tests/m00021.vtc b/bin/varnishtest/tests/m00021.vtc
index 5db3362..87f6834 100644
--- a/bin/varnishtest/tests/m00021.vtc
+++ b/bin/varnishtest/tests/m00021.vtc
@@ -10,17 +10,17 @@ varnish v1 -vcl+backend {} -start
 varnish v1 -cliok "param.set debug +vclrel"
 
 logexpect l1 -v v1 -g raw {
-	expect * 0 Debug "exp_cb: registered"
-	expect * 0 Debug "exp_cb: event insert 0x[0-9a-f]+"
-	expect * 0 Debug "exp_cb: event remove 0x[0-9a-f]+"
-	expect * 0 Debug "exp_cb: deregistered"
+	expect * 0 Debug "Subscribed to Object Events"
+	expect * 0 Debug "Object Event: insert 0x[0-9a-f]+"
+	expect * 0 Debug "Object Event: remove 0x[0-9a-f]+"
+	expect * 0 Debug "Unsubscribed from Object Events"
 } -start
 
 varnish v1 -vcl+backend {
 	import ${vmod_debug};
 
 	sub vcl_init {
-		debug.register_exp_callback();
+		debug.register_obj_events();
 	}
 
 	sub vcl_recv {
diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index 8bc0dd2..d0ebe1c 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -99,7 +99,7 @@ $Function STRING argtest(STRING one, REAL two=2, STRING three="3",
 
 $Function INT vre_limit()
 
-$Function VOID register_exp_callback(PRIV_VCL)
+$Function VOID register_obj_events(PRIV_VCL)
 
 Register the vmod to receive expiry callbacks
 
diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c
index 839d0ad..1e1b71f 100644
--- a/lib/libvmod_debug/vmod_debug.c
+++ b/lib/libvmod_debug/vmod_debug.c
@@ -44,7 +44,7 @@ struct priv_vcl {
 	unsigned		magic;
 #define PRIV_VCL_MAGIC		0x8E62FA9D
 	char			*foo;
-	uintptr_t		exp_cb;
+	uintptr_t		obj_cb;
 	struct vcl		*vcl;
 	struct vclref		*vclref;
 };
@@ -198,8 +198,8 @@ vmod_vre_limit(VRT_CTX)
 	return (cache_param->vre_limits.match);
 }
 
-static void __match_proto__(exp_callback_f)
-exp_cb(struct worker *wrk, struct objcore *oc, enum exp_event_e ev, void *priv)
+static void __match_proto__(obj_event_f)
+obj_cb(struct worker *wrk, void *priv, struct objcore *oc, unsigned event)
 {
 	const struct priv_vcl *priv_vcl;
 	const char *what;
@@ -207,26 +207,28 @@ exp_cb(struct worker *wrk, struct objcore *oc, enum exp_event_e ev, void *priv)
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CAST_OBJ_NOTNULL(priv_vcl, priv, PRIV_VCL_MAGIC);
-	switch (ev) {
-	case EXP_INSERT: what = "insert"; break;
-	case EXP_INJECT: what = "inject"; break;
-	case EXP_REMOVE: what = "remove"; break;
-	default: WRONG("Wrong exp_event");
+	switch (event) {
+	case OEV_INSERT: what = "insert"; break;
+	case OEV_REMOVE: what = "remove"; break;
+	default: WRONG("Wrong object event");
 	}
-	VSL(SLT_Debug, 0, "exp_cb: event %s 0x%jx", what,
+
+	/* We cannot trust %p to be 0x... format as expected by m00021.vtc */
+	VSL(SLT_Debug, 0, "Object Event: %s 0x%jx", what,
 	    (intmax_t)(uintptr_t)oc);
 }
 
 VCL_VOID __match_proto__()
-vmod_register_exp_callback(VRT_CTX, struct vmod_priv *priv)
+vmod_register_obj_events(VRT_CTX, struct vmod_priv *priv)
 {
 	struct priv_vcl *priv_vcl;
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	CAST_OBJ_NOTNULL(priv_vcl, priv->priv, PRIV_VCL_MAGIC);
-	AZ(priv_vcl->exp_cb);
-	priv_vcl->exp_cb = EXP_Register_Callback(exp_cb, priv_vcl);
-	VSL(SLT_Debug, 0, "exp_cb: registered");
+	AZ(priv_vcl->obj_cb);
+	priv_vcl->obj_cb = ObjSubscribeEvents(obj_cb, priv_vcl,
+		OEV_INSERT|OEV_REMOVE);
+	VSL(SLT_Debug, 0, "Subscribed to Object Events");
 }
 
 VCL_VOID __match_proto__()
@@ -246,9 +248,9 @@ priv_vcl_free(void *priv)
 	CAST_OBJ_NOTNULL(priv_vcl, priv, PRIV_VCL_MAGIC);
 	AN(priv_vcl->foo);
 	free(priv_vcl->foo);
-	if (priv_vcl->exp_cb != 0) {
-		EXP_Deregister_Callback(&priv_vcl->exp_cb);
-		VSL(SLT_Debug, 0, "exp_cb: deregistered");
+	if (priv_vcl->obj_cb != 0) {
+		ObjUnsubscribeEvents(&priv_vcl->obj_cb);
+		VSL(SLT_Debug, 0, "Unsubscribed from Object Events");
 	}
 	AZ(priv_vcl->vcl);
 	AZ(priv_vcl->vclref);



From phk at FreeBSD.org  Tue Feb  9 11:26:15 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 12:26:15 +0100
Subject: [master] e534b4b Collapse signal_close and close into just one
 stevedore method.
Message-ID: <E1aT6Qx-000529-Ip@project.varnish-software.com>


commit e534b4bc82468b1fef14c0e80411cc297ebc21a1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 11:18:11 2016 +0000

    Collapse signal_close and close into just one stevedore method.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index eb16777..e87eaa3 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -1078,8 +1078,6 @@ void RFC2616_Vary_AE(struct http *hp);
 /* stevedore.c */
 int STV_NewObject(struct worker *, struct objcore *,
     const char *hint, unsigned len);
-void STV_open(void);
-void STV_close(void);
 
 /*
  * A normal pointer difference is signed, but we never want a negative value
diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h
index 3f64c8e..181f739 100644
--- a/bin/varnishd/cache/cache_priv.h
+++ b/bin/varnishd/cache/cache_priv.h
@@ -132,6 +132,8 @@ void VRTPRIV_dynamic_kill(struct vrt_privs *privs, uintptr_t id);
 void VMOD_Init(void);
 
 /* stevedore.c */
+void STV_open(void);
+void STV_close(void);
 int STV_BanInfoDrop(const uint8_t *ban, unsigned len);
 int STV_BanInfoNew(const uint8_t *ban, unsigned len);
 void STV_BanExport(const uint8_t *banlist, unsigned len);
diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index a7648ec..2ce76dd 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -127,6 +127,7 @@ STV_open(void)
 {
 	struct stevedore *stv;
 
+	ASSERT_CLI();
 	VTAILQ_FOREACH(stv, &stv_stevedores, list) {
 		if (stv->open != NULL)
 			stv->open(stv);
@@ -141,22 +142,19 @@ void
 STV_close(void)
 {
 	struct stevedore *stv;
-
-	/* Signal intent to close */
-	VTAILQ_FOREACH(stv, &stv_stevedores, list)
-		if (stv->signal_close != NULL)
-			stv->signal_close(stv);
-	stv = stv_transient;
-	if (stv->signal_close != NULL)
-		stv->signal_close(stv);
-
-	/* Close each in turn */
-	VTAILQ_FOREACH(stv, &stv_stevedores, list)
+	int i;
+
+	ASSERT_CLI();
+	for (i = 1; i >= 0; i--) {
+		/* First send close warning */
+		VTAILQ_FOREACH(stv, &stv_stevedores, list)
+			if (stv->close != NULL)
+				stv->close(stv, i);
+		stv = stv_transient;
 		if (stv->close != NULL)
-			stv->close(stv);
-	stv = stv_transient;
-	if (stv->close != NULL)
-		stv->close(stv);
+			if (stv->close != NULL)
+				stv->close(stv, i);
+	}
 }
 
 /*-------------------------------------------------------------------
diff --git a/bin/varnishd/storage/storage.h b/bin/varnishd/storage/storage.h
index 462c74c..b17440f 100644
--- a/bin/varnishd/storage/storage.h
+++ b/bin/varnishd/storage/storage.h
@@ -66,8 +66,7 @@ typedef void storage_init_f(struct stevedore *, int ac, char * const *av);
 typedef void storage_open_f(struct stevedore *);
 typedef int storage_allocobj_f(struct worker *, const struct stevedore *,
     struct objcore *, unsigned ltot, int really);
-typedef void storage_close_f(const struct stevedore *);
-typedef void storage_signal_close_f(const struct stevedore *);
+typedef void storage_close_f(const struct stevedore *, int pass);
 typedef int storage_baninfo_f(const struct stevedore *, enum baninfo event,
     const uint8_t *ban, unsigned len);
 typedef void storage_banexport_f(const struct stevedore *, const uint8_t *bans,
@@ -96,7 +95,6 @@ struct stevedore {
 	storage_open_f		*open;
 	storage_close_f		*close;
 	storage_allocobj_f	*allocobj;
-	storage_signal_close_f	*signal_close;
 	storage_baninfo_f	*baninfo;
 	storage_banexport_f	*banexport;
 
diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c
index ba1a450..d52c2cc 100644
--- a/bin/varnishd/storage/storage_persistent.c
+++ b/bin/varnishd/storage/storage_persistent.c
@@ -387,24 +387,8 @@ smp_open(struct stevedore *st)
  * Close a silo
  */
 
-static void
-smp_signal_close(const struct stevedore *st)
-{
-	struct smp_sc	*sc;
-
-	ASSERT_CLI();
-
-	CAST_OBJ_NOTNULL(sc, st->priv, SMP_SC_MAGIC);
-	Lck_Lock(&sc->mtx);
-	if (sc->cur_seg != NULL)
-		smp_close_seg(sc, sc->cur_seg);
-	AZ(sc->cur_seg);
-	sc->flags |= SMP_SC_STOP;
-	Lck_Unlock(&sc->mtx);
-}
-
-static void
-smp_close(const struct stevedore *st)
+static void __match_proto__(storage_close_f)
+smp_close(const struct stevedore *st, int warn)
 {
 	struct smp_sc	*sc;
 	void *status;
@@ -412,9 +396,17 @@ smp_close(const struct stevedore *st)
 	ASSERT_CLI();
 
 	CAST_OBJ_NOTNULL(sc, st->priv, SMP_SC_MAGIC);
-
-	AZ(pthread_join(sc->bgthread, &status));
-	AZ(status);
+	if (warn) {
+		Lck_Lock(&sc->mtx);
+		if (sc->cur_seg != NULL)
+			smp_close_seg(sc, sc->cur_seg);
+		AZ(sc->cur_seg);
+		sc->flags |= SMP_SC_STOP;
+		Lck_Unlock(&sc->mtx);
+	} else {
+		AZ(pthread_join(sc->bgthread, &status));
+		AZ(status);
+	}
 }
 
 /*--------------------------------------------------------------------
@@ -603,7 +595,6 @@ const struct stevedore smp_stevedore = {
 	.open		= smp_open,
 	.close		= smp_close,
 	.allocobj	= smp_allocobj,
-	.signal_close	= smp_signal_close,
 	.baninfo	= smp_baninfo,
 	.banexport	= smp_banexport,
 	.methods	= &smp_oc_realmethods,



From phk at FreeBSD.org  Tue Feb  9 11:26:15 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 12:26:15 +0100
Subject: [master] dfa886a rename the objcore's 'list' to 'hsh_list' since it
 has so many.
Message-ID: <E1aT6Qx-00052G-KA@project.varnish-software.com>


commit dfa886a2115e3e0a6cd7bd448a213c48f94f5920
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 11:25:40 2016 +0000

    rename the objcore's 'list' to 'hsh_list' since it has so many.
    
    Pack oc 8 bytes tighter.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index e87eaa3..fb57436 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -433,9 +433,9 @@ struct objcore {
 	uint16_t		oa_present;
 
 	unsigned		timer_idx;	// XXX 4Gobj limit
-	VTAILQ_ENTRY(objcore)	list;
-	VTAILQ_ENTRY(objcore)	lru_list;
 	float			last_lru;
+	VTAILQ_ENTRY(objcore)	hsh_list;
+	VTAILQ_ENTRY(objcore)	lru_list;
 	VTAILQ_ENTRY(objcore)	ban_list;
 	VSTAILQ_ENTRY(objcore)	exp_list;
 	struct ban		*ban;
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 008d296..c70ef7b 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -120,7 +120,7 @@ HSH_Private(struct worker *wrk)
 	oc->objhead = private_oh;
 	oc->flags |= OC_F_PRIVATE | OC_F_BUSY;
 	Lck_Lock(&private_oh->mtx);
-	VTAILQ_INSERT_TAIL(&private_oh->objcs, oc, list);
+	VTAILQ_INSERT_TAIL(&private_oh->objcs, oc, hsh_list);
 	private_oh->refcnt++;
 	Lck_Unlock(&private_oh->mtx);
 	return (oc);
@@ -276,7 +276,7 @@ HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	AZ(oc->flags & OC_F_BUSY);
 
-	VTAILQ_INSERT_HEAD(&oh->objcs, oc, list);
+	VTAILQ_INSERT_HEAD(&oh->objcs, oc, hsh_list);
 	/* NB: do not deref objhead the new object inherits our reference */
 	oc->objhead = oh;
 	Lck_Unlock(&oh->mtx);
@@ -302,7 +302,7 @@ hsh_insert_busyobj(struct worker *wrk, struct objhead *oh)
 	AN(oc->flags & OC_F_BUSY);
 	oc->refcnt = 1;		/* Owned by busyobj */
 	oc->objhead = oh;
-	VTAILQ_INSERT_TAIL(&oh->objcs, oc, list);
+	VTAILQ_INSERT_TAIL(&oh->objcs, oc, hsh_list);
 	return (oc);
 }
 
@@ -367,7 +367,7 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 	busy_found = 0;
 	exp_oc = NULL;
 	exp_t_origin = 0.0;
-	VTAILQ_FOREACH(oc, &oh->objcs, list) {
+	VTAILQ_FOREACH(oc, &oh->objcs, hsh_list) {
 		/* Must be at least our own ref + the objcore we examine */
 		assert(oh->refcnt > 1);
 		CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
@@ -554,7 +554,7 @@ double keep)
 		Lck_Lock(&oh->mtx);
 		assert(oh->refcnt > 0);
 		now = VTIM_real();
-		VTAILQ_FOREACH(oc, &oh->objcs, list) {
+		VTAILQ_FOREACH(oc, &oh->objcs, hsh_list) {
 			CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 			assert(oc->objhead == oh);
 			if (oc->flags & OC_F_BUSY) {
@@ -662,8 +662,8 @@ HSH_Unbusy(struct worker *wrk, struct objcore *oc)
 	Lck_Lock(&oh->mtx);
 	assert(oh->refcnt > 0);
 	/* XXX: strictly speaking, we should sort in Date: order. */
-	VTAILQ_REMOVE(&oh->objcs, oc, list);
-	VTAILQ_INSERT_HEAD(&oh->objcs, oc, list);
+	VTAILQ_REMOVE(&oh->objcs, oc, hsh_list);
+	VTAILQ_INSERT_HEAD(&oh->objcs, oc, hsh_list);
 	oc->flags &= ~OC_F_BUSY;
 	if (!VTAILQ_EMPTY(&oh->waitinglist))
 		hsh_rush(wrk, oh);
@@ -819,7 +819,7 @@ HSH_DerefObjCore(struct worker *wrk, struct objcore **ocp)
 	assert(oh->refcnt > 0);
 	r = --oc->refcnt;
 	if (!r)
-		VTAILQ_REMOVE(&oh->objcs, oc, list);
+		VTAILQ_REMOVE(&oh->objcs, oc, hsh_list);
 	if (!VTAILQ_EMPTY(&oh->waitinglist))
 		hsh_rush(wrk, oh);
 	Lck_Unlock(&oh->mtx);



From phk at FreeBSD.org  Tue Feb  9 20:51:21 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 21:51:21 +0100
Subject: [master] 234395d This is embarrsing: I already this struct under a
 better name
Message-ID: <E1aTFFp-0007UX-Da@project.varnish-software.com>


commit 234395d977796d4c2962c436f5a7fc89ff488b34
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 19:27:58 2016 +0000

    This is embarrsing: I already this struct under a better name

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index d98065b..0eb9251 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -107,8 +107,8 @@ noinst_HEADERS = \
 	cache/cache_esi.h \
 	cache/cache_obj.h \
 	cache/cache_pool.h \
-	cache/cache_proto.h \
 	cache/cache_priv.h \
+	cache/cache_transport.h \
 	common/heritage.h \
 	hash/hash_slinger.h \
 	http1/cache_http1.h \
diff --git a/bin/varnishd/cache/cache_proto.h b/bin/varnishd/cache/cache_proto.h
deleted file mode 100644
index c432192..0000000
--- a/bin/varnishd/cache/cache_proto.h
+++ /dev/null
@@ -1,34 +0,0 @@
-/*-
- * Copyright (c) 2016 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- */
-
-struct proto {
-	unsigned		magic;
-#define PROTO_MAGIC		0x711e77c1
-	enum sess_step		first_step;
-};
diff --git a/bin/varnishd/cache/cache_transport.h b/bin/varnishd/cache/cache_transport.h
new file mode 100644
index 0000000..c432192
--- /dev/null
+++ b/bin/varnishd/cache/cache_transport.h
@@ -0,0 +1,34 @@
+/*-
+ * Copyright (c) 2016 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ */
+
+struct proto {
+	unsigned		magic;
+#define PROTO_MAGIC		0x711e77c1
+	enum sess_step		first_step;
+};



From phk at FreeBSD.org  Tue Feb  9 20:51:21 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 21:51:21 +0100
Subject: [master] 4d27683 Alas poor struct proto, I hardly knew ye...
Message-ID: <E1aTFFp-0007Ue-Fa@project.varnish-software.com>


commit 4d27683332ec93b335bd5c41c904ecac3adac5ff
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 19:41:49 2016 +0000

    Alas poor struct proto, I hardly knew ye...

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index fa9a5ff..6ba8e22 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -41,7 +41,7 @@
 #include <netinet/tcp.h>
 
 #include "cache.h"
-#include "cache_proto.h"
+#include "cache_transport.h"
 #include "cache_pool.h"
 #include "common/heritage.h"
 
@@ -330,7 +330,7 @@ vca_make_session(struct worker *wrk, void *arg)
 
 	sp->fd = wa->acceptsock;
 	wa->acceptsock = -1;
-	sp->sess_step = wa->acceptlsock->proto->first_step;
+	sp->sess_step = wa->acceptlsock->transport->first_step;
 
 	assert(wa->acceptaddrlen <= vsa_suckaddr_len);
 	SES_Reserve_remote_addr(sp, &sa);
@@ -349,7 +349,7 @@ vca_make_session(struct worker *wrk, void *arg)
 
 	VTCP_name(sa, laddr, sizeof laddr, lport, sizeof lport);
 
-	VSL(SLT_Begin, sp->vxid, "sess 0 %s", wa->acceptlsock->proto_name);
+	VSL(SLT_Begin, sp->vxid, "sess 0 %s", wa->acceptlsock->transport_name);
 	VSL(SLT_SessOpen, sp->vxid, "%s %s %s %s %s %.6f %d",
 	    raddr, rport, wa->acceptlsock->name, laddr, lport,
 	    sp->t_open, sp->fd);
@@ -495,7 +495,7 @@ vca_acct(void *arg)
 	(void)vca_tcp_opt_init();
 
 	VTAILQ_FOREACH(ls, &heritage.socks, list) {
-		CHECK_OBJ_NOTNULL(ls->proto, PROTO_MAGIC);
+		CHECK_OBJ_NOTNULL(ls->transport, TRANSPORT_MAGIC);
 		assert (ls->sock > 0);		// We know where stdin is
 		AZ(listen(ls->sock, cache_param->listen_depth));
 		vca_tcp_opt_set(ls->sock, 1);
diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index 88ab55c..444bf1e 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -34,6 +34,7 @@
 #include <stdlib.h>
 
 #include "cache.h"
+#include "cache_transport.h"
 #include "cache_filter.h"
 
 #include "vtim.h"
diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h
index 181f739..d5b4d10 100644
--- a/bin/varnishd/cache/cache_priv.h
+++ b/bin/varnishd/cache/cache_priv.h
@@ -29,21 +29,6 @@
  * Stuff that should *never* be exposed to a VMOD
  */
 
-/*--------------------------------------------------------------------
- * A transport is how we talk HTTP for a given request.
- * This is different from a protocol because ESI child requests have
- * their own "protocol" to talk to the parent ESI request, which may
- * or may not, be talking a "real" HTTP protocol itself.
- */
-
-typedef void vtr_deliver_f (struct req *, struct boc *, int sendbody);
-
-struct transport {
-	unsigned		magic;
-#define TRANSPORT_MAGIC		0xf157f32f
-	vtr_deliver_f		*deliver;
-};
-
 /* Prototypes etc ----------------------------------------------------*/
 
 /* cache_acceptor.c */
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 04c4e7f..2d65fbd 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -41,6 +41,7 @@
 #include "cache.h"
 #include "cache_director.h"
 #include "cache_filter.h"
+#include "cache_transport.h"
 
 #include "hash/hash_slinger.h"
 #include "vcl.h"
diff --git a/bin/varnishd/cache/cache_transport.h b/bin/varnishd/cache/cache_transport.h
index c432192..888bd3a 100644
--- a/bin/varnishd/cache/cache_transport.h
+++ b/bin/varnishd/cache/cache_transport.h
@@ -25,10 +25,19 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
+ * A transport is how we talk HTTP for a given request.
+ *
+ * This is different from a protocol because ESI child requests have
+ * their own "protocol" to talk to the parent ESI request, which may
+ * or may not, be talking a "real" HTTP protocol itself.
+ *
  */
 
-struct proto {
+typedef void vtr_deliver_f (struct req *, struct boc *, int sendbody);
+
+struct transport {
 	unsigned		magic;
-#define PROTO_MAGIC		0x711e77c1
+#define TRANSPORT_MAGIC		0xf157f32f
+	vtr_deliver_f		*deliver;
 	enum sess_step		first_step;
 };
diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h
index cbb8f9c..192d899 100644
--- a/bin/varnishd/common/heritage.h
+++ b/bin/varnishd/common/heritage.h
@@ -31,7 +31,6 @@
 
 struct vsm_sc;
 struct suckaddr;
-struct proto;
 
 struct listen_sock {
 	unsigned			magic;
@@ -40,12 +39,12 @@ struct listen_sock {
 	int				sock;
 	char				*name;
 	struct suckaddr			*addr;
-	const struct proto		*proto;
-	const char			*proto_name;
+	const struct transport		*transport;
+	const char			*transport_name;
 };
 
-extern const struct proto VPX_proto;
-extern const struct proto HTTP1_proto;
+extern const struct transport PROXY_transport;
+extern const struct transport HTTP1_transport;
 
 VTAILQ_HEAD(listen_sock_head, listen_sock);
 
diff --git a/bin/varnishd/http1/cache_http1.h b/bin/varnishd/http1/cache_http1.h
index f717b78..597b432 100644
--- a/bin/varnishd/http1/cache_http1.h
+++ b/bin/varnishd/http1/cache_http1.h
@@ -39,7 +39,7 @@ extern const int HTTP1_Req[3];
 extern const int HTTP1_Resp[3];
 
 /* cache_http1_deliver.c */
-vtr_deliver_f V1D_Deliver;
+void V1D_Deliver(struct req *, struct boc *, int sendbody);
 
 /* cache_http1_pipe.c */
 struct v1p_acct {
diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c
index d86ba28..513875d 100644
--- a/bin/varnishd/http1/cache_http1_fsm.c
+++ b/bin/varnishd/http1/cache_http1_fsm.c
@@ -38,15 +38,16 @@
 #include <stdlib.h>
 
 #include "cache/cache.h"
-#include "cache/cache_proto.h"
+#include "cache/cache_transport.h"
 #include "cache_http1.h"
 #include "hash/hash_slinger.h"
 
 #include "vtcp.h"
 
-static const struct transport http1_transport = {
-	.magic = TRANSPORT_MAGIC,
-	.deliver = V1D_Deliver,
+const struct transport HTTP1_transport = {
+	.magic =		TRANSPORT_MAGIC,
+	.deliver =		V1D_Deliver,
+	.first_step =		S_STP_H1NEWSESS,
 };
 
 /*----------------------------------------------------------------------
@@ -264,7 +265,7 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 			sp->sess_step = S_STP_H1PROC;
 			break;
 		case S_STP_H1PROC:
-			req->transport = &http1_transport;
+			req->transport = &HTTP1_transport;
 			if (CNT_Request(wrk, req) == REQ_FSM_DISEMBARK) {
 				sp->sess_step = S_STP_H1BUSY;
 				return;
@@ -286,8 +287,3 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 
 	}
 }
-
-const struct proto HTTP1_proto = {
-	.magic =		PROTO_MAGIC,
-	.first_step =		S_STP_H1NEWSESS,
-};
diff --git a/bin/varnishd/mgt/mgt_acceptor.c b/bin/varnishd/mgt/mgt_acceptor.c
index 63adbac..bda2063 100644
--- a/bin/varnishd/mgt/mgt_acceptor.c
+++ b/bin/varnishd/mgt/mgt_acceptor.c
@@ -116,8 +116,8 @@ struct mac_help {
 #define MAC_HELP_MAGIC		0x1e00a9d9
 	int			good;
 	const char		*name;
-	const char		*proto_name;
-	const struct proto	*proto;
+	const char		*transport_name;
+	const struct transport	*transport;
 };
 
 static int __match_proto__(vss_resolved_f)
@@ -140,8 +140,8 @@ mac_callback(void *priv, const struct suckaddr *sa)
 	AN(ls->addr);
 	ls->name = strdup(mh->name);
 	AN(ls->name);
-	ls->proto_name = mh->proto_name;
-	ls->proto = mh->proto;
+	ls->transport_name = mh->transport_name;
+	ls->transport = mh->transport;
 	VTAILQ_INSERT_TAIL(&heritage.socks, ls, list);
 	mh->good++;
 	return (0);
@@ -197,13 +197,13 @@ MAC_Arg(const char *arg)
 	mh->name = av[1];
 
 	if (av[2] == NULL || !strcmp(av[2], "HTTP/1")) {
-		mh->proto = &HTTP1_proto;
-		mh->proto_name = "HTTP/1";
+		mh->transport = &HTTP1_transport;
+		mh->transport_name = "HTTP/1";
 		if (av[2] != NULL && av[3] != NULL)
 			ARGV_ERR("Too many sub-arguments to -a(HTTP/1)\n");
 	} else if (!strcmp(av[2], "PROXY")) {
-		mh->proto = &VPX_proto;
-		mh->proto_name = "PROXY";
+		mh->transport = &PROXY_transport;
+		mh->transport_name = "PROXY";
 		if (av[3] != NULL)
 			ARGV_ERR("Too many sub-arguments to -a(PROXY)\n");
 	} else {
diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index a73f21a..8df3d84 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -39,7 +39,7 @@
 #include <string.h>
 
 #include "../cache/cache.h"
-#include "../cache/cache_proto.h"
+#include "../cache/cache_transport.h"
 
 #include "vend.h"
 #include "vsa.h"
@@ -381,7 +381,7 @@ VPX_Proto_Sess(struct worker *wrk, void *priv)
 	wrk->task.priv = req;
 }
 
-const struct proto VPX_proto = {
-	.magic =		PROTO_MAGIC,
+const struct transport PROXY_transport = {
+	.magic =		TRANSPORT_MAGIC,
 	.first_step =		S_STP_PROXYNEWSESS,
 };



From phk at FreeBSD.org  Tue Feb  9 20:51:21 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 21:51:21 +0100
Subject: [master] b28385e Use struct transport to untangle how we get from
 acceptor to transport
Message-ID: <E1aTFFp-0007Uo-HO@project.varnish-software.com>


commit b28385eca0ae9a65aebf4fd387fe322bf4107788
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 20:23:44 2016 +0000

    Use struct transport to untangle how we get from acceptor to transport

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index fb57436..ed69cf6 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -948,7 +948,7 @@ void SES_Wait(struct sess *sp);
 void SES_Delete(struct sess *sp, enum sess_close reason, double now);
 void SES_NewPool(struct pool *pp, unsigned pool_no);
 int SES_Reschedule_Req(struct req *);
-task_func_t SES_Proto_Sess;
+task_func_t SES_New_Session;
 task_func_t SES_Proto_Req;
 
 enum htc_status_e {
diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 6ba8e22..78ff8cc 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -330,7 +330,6 @@ vca_make_session(struct worker *wrk, void *arg)
 
 	sp->fd = wa->acceptsock;
 	wa->acceptsock = -1;
-	sp->sess_step = wa->acceptlsock->transport->first_step;
 
 	assert(wa->acceptaddrlen <= vsa_suckaddr_len);
 	SES_Reserve_remote_addr(sp, &sa);
@@ -365,9 +364,7 @@ vca_make_session(struct worker *wrk, void *arg)
 	}
 	vca_tcp_opt_set(sp->fd, 0);
 
-	/* SES_Proto_Sess() must be sceduled with reserved WS */
-	assert(8 <= WS_Reserve(sp->ws, 8));
-	wrk->task.func = SES_Proto_Sess;
+	wrk->task.func = wa->acceptlsock->transport->new_session;
 	wrk->task.priv = sp;
 }
 
diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index 34c4374..c3802c6 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -314,12 +314,8 @@ SES_Proto_Req(struct worker *wrk, void *arg)
 
 	THR_SetRequest(req);
 	AZ(wrk->aws->r);
-	if (req->sp->sess_step < S_STP_H1_LAST) {
-		HTTP1_Session(wrk, req);
-		AZ(wrk->v1l);
-	} else {
-		WRONG("Wrong session step");
-	}
+	HTTP1_Session(wrk, req);
+	AZ(wrk->v1l);
 	WS_Assert(wrk->aws);
 	THR_SetRequest(NULL);
 }
@@ -335,35 +331,44 @@ SES_Proto_Req(struct worker *wrk, void *arg)
  */
 
 void __match_proto__(task_func_t)
-SES_Proto_Sess(struct worker *wrk, void *arg)
+SES_New_Session(struct worker *wrk, void *arg)
 {
-	struct req *req;
 	struct sess *sp;
+	struct req *req;
 
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CAST_OBJ_NOTNULL(sp, arg, SESS_MAGIC);
-	WS_Release(sp->ws, 0);
 
 	/*
 	 * Assume we're going to receive something that will likely
 	 * involve a request...
 	 */
-	(void)VTCP_blocking(sp->fd);
+	if (VTCP_blocking(sp->fd)) {
+		if (errno == ECONNRESET)
+			SES_Close(sp, SC_REM_CLOSE);
+		else
+			SES_Close(sp, SC_TX_ERROR);
+		return;
+	}
 	req = Req_New(wrk, sp);
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
 	req->htc->fd = sp->fd;
 	SES_RxInit(req->htc, req->ws,
 	    cache_param->http_req_size, cache_param->http_req_hdr_len);
 
-	if (sp->sess_step < S_STP_H1_LAST) {
-		wrk->task.func = SES_Proto_Req;
-		wrk->task.priv = req;
-	} else if (sp->sess_step < S_STP_PROXY_LAST) {
-		wrk->task.func = VPX_Proto_Sess;
-		wrk->task.priv = req;
-	} else {
-		WRONG("Wrong session step");
-	}
+	sp->sess_step = S_STP_H1NEWREQ;
+	wrk->task.func = SES_Proto_Req;
+	wrk->task.priv = req;
+}
+
+static void __match_proto__(task_func_t)
+SES_Proto_Sess(struct worker *wrk, void *arg)
+{
+	struct sess *sp;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CAST_OBJ_NOTNULL(sp, arg, SESS_MAGIC);
+	WS_Release(sp->ws, 0);
+	SES_New_Session(wrk, arg);
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishd/cache/cache_transport.h b/bin/varnishd/cache/cache_transport.h
index 888bd3a..04440fc 100644
--- a/bin/varnishd/cache/cache_transport.h
+++ b/bin/varnishd/cache/cache_transport.h
@@ -38,6 +38,8 @@ typedef void vtr_deliver_f (struct req *, struct boc *, int sendbody);
 struct transport {
 	unsigned		magic;
 #define TRANSPORT_MAGIC		0xf157f32f
+
+	task_func_t		*new_session;
+
 	vtr_deliver_f		*deliver;
-	enum sess_step		first_step;
 };
diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c
index 513875d..71516a0 100644
--- a/bin/varnishd/http1/cache_http1_fsm.c
+++ b/bin/varnishd/http1/cache_http1_fsm.c
@@ -47,7 +47,7 @@
 const struct transport HTTP1_transport = {
 	.magic =		TRANSPORT_MAGIC,
 	.deliver =		V1D_Deliver,
-	.first_step =		S_STP_H1NEWSESS,
+	.new_session =		SES_New_Session,
 };
 
 /*----------------------------------------------------------------------
@@ -185,17 +185,6 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 
 	while (1) {
 		switch (sp->sess_step) {
-		case S_STP_H1NEWSESS:
-			if (VTCP_blocking(sp->fd)) {
-				if (errno == ECONNRESET)
-					SES_Close(sp, SC_REM_CLOSE);
-				else
-					SES_Close(sp, SC_TX_ERROR);
-				AN(Req_Cleanup(sp, wrk, req));
-				return;
-			}
-			sp->sess_step = S_STP_H1NEWREQ;
-			break;
 		case S_STP_H1NEWREQ:
 			assert(isnan(req->t_prev));
 			assert(isnan(req->t_req));
diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index 8df3d84..c8faa64 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -332,8 +332,8 @@ vpx_complete(struct http_conn *htc)
 	return (HTC_S_MORE);
 }
 
-void __match_proto__(task_func_t)
-VPX_Proto_Sess(struct worker *wrk, void *priv)
+static void __match_proto__(task_func_t)
+vpx_new_session(struct worker *wrk, void *arg)
 {
 	struct req *req;
 	struct sess *sp;
@@ -342,8 +342,13 @@ VPX_Proto_Sess(struct worker *wrk, void *priv)
 	int i;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CAST_OBJ_NOTNULL(req, priv, REQ_MAGIC);
-	sp = req->sp;
+	CAST_OBJ_NOTNULL(sp, arg, SESS_MAGIC);
+	(void)VTCP_blocking(sp->fd);		/* XXX redundant ? */
+	req = Req_New(wrk, sp);
+	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	req->htc->fd = sp->fd;
+	SES_RxInit(req->htc, req->ws,
+	    cache_param->http_req_size, cache_param->http_req_hdr_len);
 
 	/* Per specification */
 	assert(sizeof vpx1_sig == 5);
@@ -383,5 +388,5 @@ VPX_Proto_Sess(struct worker *wrk, void *priv)
 
 const struct transport PROXY_transport = {
 	.magic =		TRANSPORT_MAGIC,
-	.first_step =		S_STP_PROXYNEWSESS,
+	.new_session =		vpx_new_session,
 };
diff --git a/include/tbl/steps.h b/include/tbl/steps.h
index 5407615..a2fd654 100644
--- a/include/tbl/steps.h
+++ b/include/tbl/steps.h
@@ -31,15 +31,10 @@
 /*lint -save -e525 -e539 */
 
 #ifdef SESS_STEP
-SESS_STEP(h1newsess,	H1NEWSESS)
 SESS_STEP(h1newreq,	H1NEWREQ)
 SESS_STEP(h1proc,	H1PROC)
 SESS_STEP(h1busy,	H1BUSY)
 SESS_STEP(h1cleanup,	H1CLEANUP)
-SESS_STEP(h1_last,	H1_LAST)
-
-SESS_STEP(proxynewsess,	PROXYNEWSESS)
-SESS_STEP(proxy_last,	PROXY_LAST)
 #endif
 
 #ifdef REQ_STEP



From phk at FreeBSD.org  Tue Feb  9 20:51:21 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 21:51:21 +0100
Subject: [master] a721685 Use struct transport to get back from session-waiter.
Message-ID: <E1aTFFp-0007Uy-J5@project.varnish-software.com>


commit a721685093d5eeb70ff38fae9fe92f900d34a388
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 20:50:44 2016 +0000

    Use struct transport to get back from session-waiter.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index ed69cf6..0582f5b 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -635,6 +635,7 @@ struct sess {
 #define SESS_MAGIC		0x2c2f9c5a
 
 	enum sess_step		sess_step;
+	const struct transport	*transport;
 	struct lock		mtx;
 	int			fd;
 	uint32_t		vxid;
@@ -948,7 +949,6 @@ void SES_Wait(struct sess *sp);
 void SES_Delete(struct sess *sp, enum sess_close reason, double now);
 void SES_NewPool(struct pool *pp, unsigned pool_no);
 int SES_Reschedule_Req(struct req *);
-task_func_t SES_New_Session;
 task_func_t SES_Proto_Req;
 
 enum htc_status_e {
diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 78ff8cc..2fecb7b 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -364,7 +364,8 @@ vca_make_session(struct worker *wrk, void *arg)
 	}
 	vca_tcp_opt_set(sp->fd, 0);
 
-	wrk->task.func = wa->acceptlsock->transport->new_session;
+	sp->transport = wa->acceptlsock->transport;
+	wrk->task.func = sp->transport->new_session;
 	wrk->task.priv = sp;
 }
 
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 0a85637..97a697e 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -40,6 +40,7 @@
 #include <signal.h>
 
 #include "cache.h"
+#include "cache_transport.h"
 
 #include "cache_filter.h"
 #include "common/heritage.h"
@@ -448,7 +449,8 @@ pan_sess(struct vsb *vsb, const struct sess *sp)
 	if (pan_already(vsb, sp))
 		return;
 	VSB_indent(vsb, 2);
-	VSB_printf(vsb, "fd = %d, vxid = %u,\n", sp->fd, VXID(sp->vxid));
+	VSB_printf(vsb, "fd = %d, vxid = %u, xport = %s\n",
+	    sp->fd, VXID(sp->vxid), sp->transport->name);
 	AZ(SES_Get_client_ip(sp, &ci));
 	AZ(SES_Get_client_port(sp, &cp));
 	VSB_printf(vsb, "client = %s %s,\n", ci, cp);
diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index c3802c6..a944d4b 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -46,7 +46,7 @@
 
 #include "cache.h"
 #include "cache_pool.h"
-#include "http1/cache_http1.h"
+#include "cache_transport.h"
 
 #include "vsa.h"
 #include "vtcp.h"
@@ -301,77 +301,6 @@ SES_New(struct pool *pp)
 }
 
 /*--------------------------------------------------------------------
- * Call protocol for this request
- */
-
-void __match_proto__(task_func_t)
-SES_Proto_Req(struct worker *wrk, void *arg)
-{
-	struct req *req;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CAST_OBJ_NOTNULL(req, arg, REQ_MAGIC);
-
-	THR_SetRequest(req);
-	AZ(wrk->aws->r);
-	HTTP1_Session(wrk, req);
-	AZ(wrk->v1l);
-	WS_Assert(wrk->aws);
-	THR_SetRequest(NULL);
-}
-
-/*--------------------------------------------------------------------
- * Call protocol for this session (new or from waiter)
- *
- * When sessions are rescheduled from the waiter, a struct pool_task
- * is put on the reserved session workspace (for reasons of memory
- * conservation).  This reservation is released as the first thing.
- * The acceptor and any other code which schedules this function
- * must obey this calling convention with a dummy reservation.
- */
-
-void __match_proto__(task_func_t)
-SES_New_Session(struct worker *wrk, void *arg)
-{
-	struct sess *sp;
-	struct req *req;
-
-	CAST_OBJ_NOTNULL(sp, arg, SESS_MAGIC);
-
-	/*
-	 * Assume we're going to receive something that will likely
-	 * involve a request...
-	 */
-	if (VTCP_blocking(sp->fd)) {
-		if (errno == ECONNRESET)
-			SES_Close(sp, SC_REM_CLOSE);
-		else
-			SES_Close(sp, SC_TX_ERROR);
-		return;
-	}
-	req = Req_New(wrk, sp);
-	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
-	req->htc->fd = sp->fd;
-	SES_RxInit(req->htc, req->ws,
-	    cache_param->http_req_size, cache_param->http_req_hdr_len);
-
-	sp->sess_step = S_STP_H1NEWREQ;
-	wrk->task.func = SES_Proto_Req;
-	wrk->task.priv = req;
-}
-
-static void __match_proto__(task_func_t)
-SES_Proto_Sess(struct worker *wrk, void *arg)
-{
-	struct sess *sp;
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CAST_OBJ_NOTNULL(sp, arg, SESS_MAGIC);
-	WS_Release(sp->ws, 0);
-	SES_New_Session(wrk, arg);
-}
-
-/*--------------------------------------------------------------------
  * Reschedule a request on a work-thread from its sessions pool
  *
  * This is used to reschedule requests waiting on busy objects
@@ -426,7 +355,7 @@ ses_handle(struct waited *wp, enum wait_event ev, double now)
 		CHECK_OBJ_NOTNULL(pp, POOL_MAGIC);
 		assert(sizeof *tp <= WS_Reserve(sp->ws, sizeof *tp));
 		tp = (void*)sp->ws->f;
-		tp->func = SES_Proto_Sess;
+		tp->func = sp->transport->unwait;
 		tp->priv = sp;
 		if (Pool_Task(pp, tp, TASK_QUEUE_REQ))
 			SES_Delete(sp, SC_OVERLOAD, now);
diff --git a/bin/varnishd/cache/cache_transport.h b/bin/varnishd/cache/cache_transport.h
index 04440fc..6b8d7e9 100644
--- a/bin/varnishd/cache/cache_transport.h
+++ b/bin/varnishd/cache/cache_transport.h
@@ -39,7 +39,10 @@ struct transport {
 	unsigned		magic;
 #define TRANSPORT_MAGIC		0xf157f32f
 
+	const char		*name;
+
 	task_func_t		*new_session;
+	task_func_t		*unwait;
 
 	vtr_deliver_f		*deliver;
 };
diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c
index 71516a0..7313101 100644
--- a/bin/varnishd/http1/cache_http1_fsm.c
+++ b/bin/varnishd/http1/cache_http1_fsm.c
@@ -44,10 +44,83 @@
 
 #include "vtcp.h"
 
+/*--------------------------------------------------------------------
+ * Call protocol for this request
+ */
+
+void __match_proto__(task_func_t)
+SES_Proto_Req(struct worker *wrk, void *arg)
+{
+	struct req *req;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CAST_OBJ_NOTNULL(req, arg, REQ_MAGIC);
+
+	THR_SetRequest(req);
+	AZ(wrk->aws->r);
+	HTTP1_Session(wrk, req);
+	AZ(wrk->v1l);
+	WS_Assert(wrk->aws);
+	THR_SetRequest(NULL);
+}
+
+/*--------------------------------------------------------------------
+ * Call protocol for this session (new or from waiter)
+ *
+ * When sessions are rescheduled from the waiter, a struct pool_task
+ * is put on the reserved session workspace (for reasons of memory
+ * conservation).  This reservation is released as the first thing.
+ * The acceptor and any other code which schedules this function
+ * must obey this calling convention with a dummy reservation.
+ */
+
+static void __match_proto__(task_func_t)
+http1_new_session(struct worker *wrk, void *arg)
+{
+	struct sess *sp;
+	struct req *req;
+
+	CAST_OBJ_NOTNULL(sp, arg, SESS_MAGIC);
+
+	/*
+	 * Assume we're going to receive something that will likely
+	 * involve a request...
+	 */
+	if (VTCP_blocking(sp->fd)) {
+		if (errno == ECONNRESET)
+			SES_Close(sp, SC_REM_CLOSE);
+		else
+			SES_Close(sp, SC_TX_ERROR);
+		return;
+	}
+	req = Req_New(wrk, sp);
+	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	req->htc->fd = sp->fd;
+	SES_RxInit(req->htc, req->ws,
+	    cache_param->http_req_size, cache_param->http_req_hdr_len);
+
+	sp->sess_step = S_STP_H1NEWREQ;
+	wrk->task.func = SES_Proto_Req;
+	wrk->task.priv = req;
+}
+
+static void __match_proto__(task_func_t)
+http1_unwait(struct worker *wrk, void *arg)
+{
+	struct sess *sp;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CAST_OBJ_NOTNULL(sp, arg, SESS_MAGIC);
+	WS_Release(sp->ws, 0);
+	http1_new_session(wrk, arg);
+}
+
 const struct transport HTTP1_transport = {
+	.name =			"HTTP/1",
 	.magic =		TRANSPORT_MAGIC,
 	.deliver =		V1D_Deliver,
-	.new_session =		SES_New_Session,
+	.unwait =		http1_unwait,
+	.new_session =		http1_new_session,
 };
 
 /*----------------------------------------------------------------------
diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index c8faa64..9d1db5b 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -41,6 +41,8 @@
 #include "../cache/cache.h"
 #include "../cache/cache_transport.h"
 
+#include "../common/heritage.h"
+
 #include "vend.h"
 #include "vsa.h"
 #include "vtcp.h"
@@ -381,12 +383,14 @@ vpx_new_session(struct worker *wrk, void *arg)
 	else
 		req->htc->pipeline_e = req->htc->rxbuf_e;
 	SES_RxReInit(req->htc);
+	sp->transport = &HTTP1_transport;
 	req->sp->sess_step = S_STP_H1NEWREQ;
 	wrk->task.func = SES_Proto_Req;
 	wrk->task.priv = req;
 }
 
 const struct transport PROXY_transport = {
+	.name =			"PROXY",
 	.magic =		TRANSPORT_MAGIC,
 	.new_session =		vpx_new_session,
 };



From phk at FreeBSD.org  Tue Feb  9 22:51:29 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 09 Feb 2016 23:51:29 +0100
Subject: [master] 644bf08 Fix timeout calculation to give session-herding a
 chance.
Message-ID: <E1aTH85-0002mx-Vl@project.varnish-software.com>


commit 644bf0801c94fd85278d8612f8b4d6a6b23a8f51
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 22:50:57 2016 +0000

    Fix timeout calculation to give session-herding a chance.
    
    Fixes: #1857

diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index a944d4b..a12f723 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -230,10 +230,6 @@ SES_RxStuff(struct http_conn *htc, htc_complete_f *func,
 				*t2 = now;
 			return (HTC_S_COMPLETE);
 		}
-		if (tn < now) {
-			WS_ReleaseP(htc->ws, htc->rxbuf_b);
-			return (HTC_S_TIMEOUT);
-		}
 		if (hs == HTC_S_MORE) {
 			/* Working on it */
 			if (t1 != NULL && isnan(*t1))
@@ -249,6 +245,8 @@ SES_RxStuff(struct http_conn *htc, htc_complete_f *func,
 			WS_ReleaseP(htc->ws, htc->rxbuf_b);
 			return (HTC_S_OVERFLOW);
 		}
+		if (tmo <= 0.0)
+			tmo = 1e-3;
 		i = VTCP_read(htc->fd, htc->rxbuf_e, i, tmo);
 		if (i == 0 || i == -1) {
 			WS_ReleaseP(htc->ws, htc->rxbuf_b);
@@ -257,10 +255,14 @@ SES_RxStuff(struct http_conn *htc, htc_complete_f *func,
 			htc->rxbuf_e += i;
 			*htc->rxbuf_e = '\0';
 		} else if (i == -2) {
-			if (hs == HTC_S_EMPTY && ti < now) {
+			if (hs == HTC_S_EMPTY && ti <= now) {
 				WS_ReleaseP(htc->ws, htc->rxbuf_b);
 				return (HTC_S_IDLE);
 			}
+			if (tn <= now) {
+				WS_ReleaseP(htc->ws, htc->rxbuf_b);
+				return (HTC_S_TIMEOUT);
+			}
 		}
 	}
 }



From phk at FreeBSD.org  Tue Feb  9 23:08:23 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 10 Feb 2016 00:08:23 +0100
Subject: [master] 8f449e3 Add testcase for #1857
Message-ID: <E1aTHOR-0004NF-M9@project.varnish-software.com>


commit 8f449e3e238f1a39526f97d6736f9d90987ff11d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 23:08:13 2016 +0000

    Add testcase for #1857

diff --git a/bin/varnishtest/tests/r01857.vtc b/bin/varnishtest/tests/r01857.vtc
new file mode 100644
index 0000000..8bb1aa4
--- /dev/null
+++ b/bin/varnishtest/tests/r01857.vtc
@@ -0,0 +1,32 @@
+varnishtest "Check session herding"
+
+server s1 {
+	rxreq
+	txresp -hdr "Connection: close" -body "012345\n"
+} -start
+
+varnish v1 -vcl+backend { } -start
+
+client c1 {
+	txreq -url "/"
+	rxresp
+	expect resp.status == 200
+	expect resp.http.X-Varnish == "1001"
+	delay 1
+	txreq -url "/"
+	rxresp
+	expect resp.status == 200
+	expect resp.http.X-Varnish == "1003 1002"
+} -run
+
+# Give varnish a chance to update stats
+delay .1
+
+varnish v1 -expect sess_herd == 1
+varnish v1 -expect sess_conn == 1
+varnish v1 -expect cache_hit == 1
+varnish v1 -expect cache_miss == 1
+varnish v1 -expect client_req == 2
+varnish v1 -expect s_sess == 1
+varnish v1 -expect s_req == 2
+varnish v1 -expect s_fetch == 1



From phk at FreeBSD.org  Wed Feb 10 09:44:26 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 10 Feb 2016 10:44:26 +0100
Subject: [master] 87f852e Hand transport->new_session a req instead of sp,
 they'll all need it.
Message-ID: <E1aTRJy-0002ns-JC@project.varnish-software.com>


commit 87f852e175083308e7cecf85fea5e7f9558ced6f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 08:23:49 2016 +0000

    Hand transport->new_session a req instead of sp, they'll all need it.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 2fecb7b..f6d6f82 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -293,6 +293,7 @@ static void __match_proto__(task_func_t)
 vca_make_session(struct worker *wrk, void *arg)
 {
 	struct sess *sp;
+	struct req *req;
 	struct wrk_accept *wa;
 	struct sockaddr_storage ss;
 	struct suckaddr *sa;
@@ -305,6 +306,13 @@ vca_make_session(struct worker *wrk, void *arg)
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CAST_OBJ_NOTNULL(wa, arg, WRK_ACCEPT_MAGIC);
 
+	if (VTCP_blocking(wa->acceptsock)) {
+		AZ(close(wa->acceptsock));
+		wrk->stats->sess_drop++;	// XXX Better counter ?
+		WS_Release(wrk->aws, 0);
+		return;
+	}
+
 	/* Turn accepted socket into a session */
 	AN(wrk->aws->r);
 	sp = SES_New(wrk->pool);
@@ -364,9 +372,15 @@ vca_make_session(struct worker *wrk, void *arg)
 	}
 	vca_tcp_opt_set(sp->fd, 0);
 
+	req = Req_New(wrk, sp);
+	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	req->htc->fd = sp->fd;
+	SES_RxInit(req->htc, req->ws,
+	    cache_param->http_req_size, cache_param->http_req_hdr_len);
+	
 	sp->transport = wa->acceptlsock->transport;
 	wrk->task.func = sp->transport->new_session;
-	wrk->task.priv = sp;
+	wrk->task.priv = req;
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c
index 7313101..e201707 100644
--- a/bin/varnishd/http1/cache_http1_fsm.c
+++ b/bin/varnishd/http1/cache_http1_fsm.c
@@ -80,24 +80,10 @@ http1_new_session(struct worker *wrk, void *arg)
 	struct sess *sp;
 	struct req *req;
 
-	CAST_OBJ_NOTNULL(sp, arg, SESS_MAGIC);
-
-	/*
-	 * Assume we're going to receive something that will likely
-	 * involve a request...
-	 */
-	if (VTCP_blocking(sp->fd)) {
-		if (errno == ECONNRESET)
-			SES_Close(sp, SC_REM_CLOSE);
-		else
-			SES_Close(sp, SC_TX_ERROR);
-		return;
-	}
-	req = Req_New(wrk, sp);
-	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
-	req->htc->fd = sp->fd;
-	SES_RxInit(req->htc, req->ws,
-	    cache_param->http_req_size, cache_param->http_req_hdr_len);
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CAST_OBJ_NOTNULL(req, arg, REQ_MAGIC);
+	sp = req->sp;
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 
 	sp->sess_step = S_STP_H1NEWREQ;
 	wrk->task.func = SES_Proto_Req;
@@ -108,11 +94,19 @@ static void __match_proto__(task_func_t)
 http1_unwait(struct worker *wrk, void *arg)
 {
 	struct sess *sp;
+	struct req *req;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CAST_OBJ_NOTNULL(sp, arg, SESS_MAGIC);
 	WS_Release(sp->ws, 0);
-	http1_new_session(wrk, arg);
+	req = Req_New(wrk, sp);
+	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	req->htc->fd = sp->fd;
+	SES_RxInit(req->htc, req->ws,
+	    cache_param->http_req_size, cache_param->http_req_hdr_len);
+	sp->sess_step = S_STP_H1NEWREQ;
+	wrk->task.func = SES_Proto_Req;
+	wrk->task.priv = req;
 }
 
 const struct transport HTTP1_transport = {
diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index 9d1db5b..061d1be 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -344,13 +344,9 @@ vpx_new_session(struct worker *wrk, void *arg)
 	int i;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CAST_OBJ_NOTNULL(sp, arg, SESS_MAGIC);
-	(void)VTCP_blocking(sp->fd);		/* XXX redundant ? */
-	req = Req_New(wrk, sp);
-	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
-	req->htc->fd = sp->fd;
-	SES_RxInit(req->htc, req->ws,
-	    cache_param->http_req_size, cache_param->http_req_hdr_len);
+	CAST_OBJ_NOTNULL(req, arg, REQ_MAGIC);
+	sp = req->sp;
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 
 	/* Per specification */
 	assert(sizeof vpx1_sig == 5);
@@ -384,8 +380,7 @@ vpx_new_session(struct worker *wrk, void *arg)
 		req->htc->pipeline_e = req->htc->rxbuf_e;
 	SES_RxReInit(req->htc);
 	sp->transport = &HTTP1_transport;
-	req->sp->sess_step = S_STP_H1NEWREQ;
-	wrk->task.func = SES_Proto_Req;
+	wrk->task.func = sp->transport->new_session;
 	wrk->task.priv = req;
 }
 



From phk at FreeBSD.org  Wed Feb 10 09:44:26 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 10 Feb 2016 10:44:26 +0100
Subject: [master] befd5d0 Don't build VED_transport dynamically
Message-ID: <E1aTRJy-0002nz-Kk@project.varnish-software.com>


commit befd5d04513705ad7bf1675db96a6af1fa774610
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 08:46:18 2016 +0000

    Don't build VED_transport dynamically

diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index 444bf1e..ba10948 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -65,6 +65,12 @@ struct ecx {
 	uint32_t	crc;
 };
 
+static struct transport VED_transport = {
+	.magic =	TRANSPORT_MAGIC,
+	.name =		"ESI_INCLUDE",
+	.deliver =	VED_Deliver,
+};
+
 /*--------------------------------------------------------------------*/
 
 static void
@@ -74,7 +80,6 @@ ved_include(struct req *preq, const char *src, const char *host,
 	struct worker *wrk;
 	struct req *req;
 	enum req_fsm_nxt s;
-	struct transport xp;
 
 	CHECK_OBJ_NOTNULL(preq, REQ_MAGIC);
 	CHECK_OBJ_NOTNULL(ecx, ECX_MAGIC);
@@ -146,9 +151,7 @@ ved_include(struct req *preq, const char *src, const char *host,
 	assert(isnan(req->t_first));
 	assert(isnan(req->t_prev));
 
-	INIT_OBJ(&xp, TRANSPORT_MAGIC);
-	xp.deliver = VED_Deliver;
-	req->transport = &xp;
+	req->transport = &VED_transport;
 	req->transport_priv = ecx;
 
 	THR_SetRequest(req);



From phk at FreeBSD.org  Wed Feb 10 09:44:26 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 10 Feb 2016 10:44:26 +0100
Subject: [master] 6308c2e Register where we want to be resumed before
 disembarking a req.
Message-ID: <E1aTRJy-0002o8-MN@project.varnish-software.com>


commit 6308c2e1dd4698a8b65f64381b7ff8549e349c57
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 08:58:44 2016 +0000

    Register where we want to be resumed before disembarking a req.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 0582f5b..27c32d3 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -949,7 +949,6 @@ void SES_Wait(struct sess *sp);
 void SES_Delete(struct sess *sp, enum sess_close reason, double now);
 void SES_NewPool(struct pool *pp, unsigned pool_no);
 int SES_Reschedule_Req(struct req *);
-task_func_t SES_Proto_Req;
 
 enum htc_status_e {
 	HTC_S_JUNK =		-5,
diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index ba10948..e6eb949 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -65,7 +65,7 @@ struct ecx {
 	uint32_t	crc;
 };
 
-static struct transport VED_transport = {
+static const struct transport VED_transport = {
 	.magic =	TRANSPORT_MAGIC,
 	.name =		"ESI_INCLUDE",
 	.deliver =	VED_Deliver,
diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index a12f723..770ce73 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -320,8 +320,7 @@ SES_Reschedule_Req(struct req *req)
 	pp = sp->pool;
 	CHECK_OBJ_NOTNULL(pp, POOL_MAGIC);
 
-	req->task.func = SES_Proto_Req;
-	req->task.priv = req;
+	AN(req->task.func);
 
 	return (Pool_Task(pp, &req->task, TASK_QUEUE_REQ));
 }
diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c
index e201707..67274c1 100644
--- a/bin/varnishd/http1/cache_http1_fsm.c
+++ b/bin/varnishd/http1/cache_http1_fsm.c
@@ -48,8 +48,8 @@
  * Call protocol for this request
  */
 
-void __match_proto__(task_func_t)
-SES_Proto_Req(struct worker *wrk, void *arg)
+static void __match_proto__(task_func_t)
+http1_req(struct worker *wrk, void *arg)
 {
 	struct req *req;
 
@@ -57,6 +57,7 @@ SES_Proto_Req(struct worker *wrk, void *arg)
 	CAST_OBJ_NOTNULL(req, arg, REQ_MAGIC);
 
 	THR_SetRequest(req);
+	req->transport = req->sp->transport;
 	AZ(wrk->aws->r);
 	HTTP1_Session(wrk, req);
 	AZ(wrk->v1l);
@@ -86,7 +87,7 @@ http1_new_session(struct worker *wrk, void *arg)
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 
 	sp->sess_step = S_STP_H1NEWREQ;
-	wrk->task.func = SES_Proto_Req;
+	wrk->task.func = http1_req;
 	wrk->task.priv = req;
 }
 
@@ -105,7 +106,7 @@ http1_unwait(struct worker *wrk, void *arg)
 	SES_RxInit(req->htc, req->ws,
 	    cache_param->http_req_size, cache_param->http_req_hdr_len);
 	sp->sess_step = S_STP_H1NEWREQ;
-	wrk->task.func = SES_Proto_Req;
+	wrk->task.func = http1_req;
 	wrk->task.priv = req;
 }
 
@@ -323,6 +324,8 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 		case S_STP_H1PROC:
 			req->transport = &HTTP1_transport;
 			if (CNT_Request(wrk, req) == REQ_FSM_DISEMBARK) {
+				req->task.func = http1_req;
+				req->task.priv = req;
 				sp->sess_step = S_STP_H1BUSY;
 				return;
 			}



From phk at FreeBSD.org  Wed Feb 10 09:44:26 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 10 Feb 2016 10:44:26 +0100
Subject: [master] e920df6 Get struct sess back down to 128 bytes.
Message-ID: <E1aTRJy-0002oH-O3@project.varnish-software.com>


commit e920df6115ff9745635c26b88cbaf7884677fb5e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 09:43:54 2016 +0000

    Get struct sess back down to 128 bytes.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 27c32d3..36ecde1 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -635,7 +635,6 @@ struct sess {
 #define SESS_MAGIC		0x2c2f9c5a
 
 	enum sess_step		sess_step;
-	const struct transport	*transport;
 	struct lock		mtx;
 	int			fd;
 	uint32_t		vxid;
@@ -944,10 +943,10 @@ void CNT_AcctLogCharge(struct dstat *, struct req *);
 
 /* cache_session.c [SES] */
 struct sess *SES_New(struct pool *);
-void SES_Close(struct sess *sp, enum sess_close reason);
-void SES_Wait(struct sess *sp);
-void SES_Delete(struct sess *sp, enum sess_close reason, double now);
-void SES_NewPool(struct pool *pp, unsigned pool_no);
+void SES_Close(struct sess *, enum sess_close reason);
+void SES_Wait(struct sess *, const struct transport *);
+void SES_Delete(struct sess *, enum sess_close reason, double now);
+void SES_NewPool(struct pool *, unsigned pool_no);
 int SES_Reschedule_Req(struct req *);
 
 enum htc_status_e {
diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index f6d6f82..6a701ea 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -377,9 +377,8 @@ vca_make_session(struct worker *wrk, void *arg)
 	req->htc->fd = sp->fd;
 	SES_RxInit(req->htc, req->ws,
 	    cache_param->http_req_size, cache_param->http_req_hdr_len);
-	
-	sp->transport = wa->acceptlsock->transport;
-	wrk->task.func = sp->transport->new_session;
+
+	wrk->task.func = wa->acceptlsock->transport->new_session;
 	wrk->task.priv = req;
 }
 
diff --git a/bin/varnishd/cache/cache_backend_tcp.c b/bin/varnishd/cache/cache_backend_tcp.c
index 13a026c..8ad54e3 100644
--- a/bin/varnishd/cache/cache_backend_tcp.c
+++ b/bin/varnishd/cache/cache_backend_tcp.c
@@ -83,7 +83,7 @@ tcp_handle(struct waited *w, enum wait_event ev, double now)
 	struct vbc *vbc;
 	struct tcp_pool *tp;
 
-	CAST_OBJ_NOTNULL(vbc, w->ptr, VBC_MAGIC);
+	CAST_OBJ_NOTNULL(vbc, w->priv1, VBC_MAGIC);
 	(void)ev;
 	(void)now;
 	CHECK_OBJ_NOTNULL(vbc->tcp_pool, TCP_POOL_MAGIC);
@@ -264,7 +264,7 @@ VBT_Recycle(const struct worker *wrk, struct tcp_pool *tp, struct vbc **vbcp)
 	Lck_Lock(&tp->mtx);
 	tp->n_used--;
 
-	vbc->waited->ptr = vbc;
+	vbc->waited->priv1 = vbc;
 	vbc->waited->fd = vbc->fd;
 	vbc->waited->idle = VTIM_real();
 	vbc->state = VBC_STATE_AVAIL;
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 97a697e..0a2cedf 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -380,7 +380,8 @@ pan_req(struct vsb *vsb, const struct req *req)
 		return;
 	VSB_indent(vsb, 2);
 
-	VSB_printf(vsb, "vxid = %u, ", VXID(req->vsl->wid));
+	VSB_printf(vsb, "vxid = %u, transport = %s", VXID(req->vsl->wid),
+	    req->transport == NULL ? "NULL" : req->transport->name);
 
 	switch (req->req_step) {
 #define REQ_STEP(l, u, arg) case R_STP_##u: stp = "R_STP_" #u; break;
@@ -449,8 +450,8 @@ pan_sess(struct vsb *vsb, const struct sess *sp)
 	if (pan_already(vsb, sp))
 		return;
 	VSB_indent(vsb, 2);
-	VSB_printf(vsb, "fd = %d, vxid = %u, xport = %s\n",
-	    sp->fd, VXID(sp->vxid), sp->transport->name);
+	VSB_printf(vsb, "fd = %d, vxid = %u\n",
+	    sp->fd, VXID(sp->vxid));
 	AZ(SES_Get_client_ip(sp, &ci));
 	AZ(SES_Get_client_port(sp, &cp));
 	VSB_printf(vsb, "client = %s %s,\n", ci, cp);
diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index 770ce73..9775332 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -335,9 +335,12 @@ ses_handle(struct waited *wp, enum wait_event ev, double now)
 	struct sess *sp;
 	struct pool *pp;
 	struct pool_task *tp;
+	const struct transport *xp;
 
 	CHECK_OBJ_NOTNULL(wp, WAITED_MAGIC);
-	CAST_OBJ_NOTNULL(sp, wp->ptr, SESS_MAGIC);
+	CAST_OBJ_NOTNULL(sp, wp->priv1, SESS_MAGIC);
+	CAST_OBJ_NOTNULL(xp, (const void*)wp->priv2, TRANSPORT_MAGIC);
+	AN(wp->priv2);
 	assert((void *)sp->ws->f == wp);
 	wp->magic = 0;
 	wp = NULL;
@@ -356,7 +359,7 @@ ses_handle(struct waited *wp, enum wait_event ev, double now)
 		CHECK_OBJ_NOTNULL(pp, POOL_MAGIC);
 		assert(sizeof *tp <= WS_Reserve(sp->ws, sizeof *tp));
 		tp = (void*)sp->ws->f;
-		tp->func = sp->transport->unwait;
+		tp->func = xp->unwait;
 		tp->priv = sp;
 		if (Pool_Task(pp, tp, TASK_QUEUE_REQ))
 			SES_Delete(sp, SC_OVERLOAD, now);
@@ -373,12 +376,13 @@ ses_handle(struct waited *wp, enum wait_event ev, double now)
  */
 
 void
-SES_Wait(struct sess *sp)
+SES_Wait(struct sess *sp, const struct transport *xp)
 {
 	struct pool *pp;
 	struct waited *wp;
 
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(xp, TRANSPORT_MAGIC);
 	pp = sp->pool;
 	CHECK_OBJ_NOTNULL(pp, POOL_MAGIC);
 	/*
@@ -400,7 +404,8 @@ SES_Wait(struct sess *sp)
 	wp = (void*)sp->ws->f;
 	INIT_OBJ(wp, WAITED_MAGIC);
 	wp->fd = sp->fd;
-	wp->ptr = sp;
+	wp->priv1 = sp;
+	wp->priv2 = (uintptr_t)xp;
 	wp->idle = sp->t_idle;
 	wp->waitfor = &pp->wf;
 	if (Wait_Enter(pp->waiter, wp))
diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c
index 67274c1..dff2f0a 100644
--- a/bin/varnishd/http1/cache_http1_fsm.c
+++ b/bin/varnishd/http1/cache_http1_fsm.c
@@ -38,6 +38,7 @@
 #include <stdlib.h>
 
 #include "cache/cache.h"
+#include "common/heritage.h"
 #include "cache/cache_transport.h"
 #include "cache_http1.h"
 #include "hash/hash_slinger.h"
@@ -57,7 +58,7 @@ http1_req(struct worker *wrk, void *arg)
 	CAST_OBJ_NOTNULL(req, arg, REQ_MAGIC);
 
 	THR_SetRequest(req);
-	req->transport = req->sp->transport;
+	req->transport = &HTTP1_transport;
 	AZ(wrk->aws->r);
 	HTTP1_Session(wrk, req);
 	AZ(wrk->v1l);
@@ -289,7 +290,7 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 			if (hs == HTC_S_IDLE) {
 				wrk->stats->sess_herd++;
 				Req_Release(req);
-				SES_Wait(sp);
+				SES_Wait(sp, &HTTP1_transport);
 				return;
 			}
 			if (hs != HTC_S_COMPLETE)
diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index 061d1be..8290c7d 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -379,8 +379,7 @@ vpx_new_session(struct worker *wrk, void *arg)
 	else
 		req->htc->pipeline_e = req->htc->rxbuf_e;
 	SES_RxReInit(req->htc);
-	sp->transport = &HTTP1_transport;
-	wrk->task.func = sp->transport->new_session;
+	wrk->task.func = HTTP1_transport.new_session;
 	wrk->task.priv = req;
 }
 
diff --git a/bin/varnishd/waiter/waiter.h b/bin/varnishd/waiter/waiter.h
index 488c8e3..4cbabb6 100644
--- a/bin/varnishd/waiter/waiter.h
+++ b/bin/varnishd/waiter/waiter.h
@@ -66,7 +66,8 @@ struct waited {
 #define WAITED_MAGIC		0x1743992d
 	int			fd;
 	unsigned		idx;
-	void			*ptr;
+	void			*priv1;
+	uintptr_t		priv2;
 	const struct waitfor	*waitfor;
 	double			idle;
 };



From phk at FreeBSD.org  Wed Feb 10 11:19:41 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 10 Feb 2016 12:19:41 +0100
Subject: [master] fed96df Add a missing return
Message-ID: <E1aTSo9-00070Y-Hq@project.varnish-software.com>


commit fed96df6375d4bc5dfed0b17d189cbe7e2f71a98
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 10:27:41 2016 +0000

    Add a missing return

diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index 9775332..281a633 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -400,6 +400,7 @@ SES_Wait(struct sess *sp, const struct transport *xp)
 	if (WS_Reserve(sp->ws, sizeof(struct waited))
 	    < sizeof(struct waited)) {
 		SES_Delete(sp, SC_OVERLOAD, NAN);
+		return;
 	}
 	wp = (void*)sp->ws->f;
 	INIT_OBJ(wp, WAITED_MAGIC);



From phk at FreeBSD.org  Wed Feb 10 11:19:41 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 10 Feb 2016 12:19:41 +0100
Subject: [master] 7c64816 Enforce that req.body caching can only be done on
 the first tour through vcl_recv{}
Message-ID: <E1aTSo9-00070e-Js@project.varnish-software.com>


commit 7c648165657258ba667d59e16fff2ac711ee0bab
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 11:19:13 2016 +0000

    Enforce that req.body caching can only be done on the first
    tour through vcl_recv{}

diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c
index 06a43fa..51492d3 100644
--- a/bin/varnishd/cache/cache_req_body.c
+++ b/bin/varnishd/cache/cache_req_body.c
@@ -190,6 +190,14 @@ VRB_Cache(struct req *req, ssize_t maxsize)
 
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
 
+	/*
+	 * We only allow caching to happen the first time through vcl_recv{}
+	 * where we know we will have no competition or conflicts for the
+	 * updates to req.http.* etc.
+	 */
+	if (req->restarts > 0 && req->req_body_status != REQ_BODY_CACHED)
+		return (-1);
+
 	assert (req->req_step == R_STP_RECV);
 	switch(req->req_body_status) {
 	case REQ_BODY_CACHED:



From nils.goroll at uplex.de  Wed Feb 10 13:31:05 2016
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 10 Feb 2016 14:31:05 +0100
Subject: [master] 125946c improve documentation for ban.list
Message-ID: <E1aTUrJ-00053l-K8@project.varnish-software.com>


commit 125946c2311f1ed021e7574de474ee430e3d478e
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Feb 10 14:31:01 2016 +0100

    improve documentation for ban.list

diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst
index 55a2528..7dcc9a4 100644
--- a/doc/sphinx/reference/varnish-cli.rst
+++ b/doc/sphinx/reference/varnish-cli.rst
@@ -149,7 +149,23 @@ ban <field> <operator> <arg> [&& <field> <oper> <arg> ...]
   Mark obsolete all objects where all the conditions match.
 
 ban.list
-  List the active bans.
+  List the active bans. The output format is:
+
+  * time the ban was issued
+
+  * reference count
+
+  * ``C`` for completed bans (replaced by a newer ban) or ``-``
+
+  * if ``lurker`` debugging is enabled
+
+    * ``R`` for bans on request properties or ``-``
+
+    * ``O`` for bans on object properties or ``-``
+
+    * pointer to the ban object
+
+  * ban specification
 
 Backend Expression
 ------------------



From phk at FreeBSD.org  Wed Feb 10 21:27:17 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 10 Feb 2016 22:27:17 +0100
Subject: [master] e8cae03 Move VPF setup before vcl_recv{}
Message-ID: <E1aTcI9-00033Y-6i@project.varnish-software.com>


commit e8cae033eab21cce6ca4ca8f94cf4cd693d639a3
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 21:21:39 2016 +0000

    Move VPF setup before vcl_recv{}

diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 0a2cedf..2363b25 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -380,7 +380,7 @@ pan_req(struct vsb *vsb, const struct req *req)
 		return;
 	VSB_indent(vsb, 2);
 
-	VSB_printf(vsb, "vxid = %u, transport = %s", VXID(req->vsl->wid),
+	VSB_printf(vsb, "vxid = %u, transport = %s\n", VXID(req->vsl->wid),
 	    req->transport == NULL ? "NULL" : req->transport->name);
 
 	switch (req->req_step) {
diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c
index 51492d3..fa0f98f 100644
--- a/bin/varnishd/cache/cache_req_body.c
+++ b/bin/varnishd/cache/cache_req_body.c
@@ -33,7 +33,6 @@
 #include <stdlib.h>
 
 #include "cache.h"
-#include "http1/cache_http1.h"
 #include "cache_filter.h"
 #include "vtim.h"
 #include "hash/hash_slinger.h"
@@ -97,11 +96,8 @@ VRB_Iterate(struct req *req, objiterate_f *func, void *priv)
 	}
 
 	CHECK_OBJ_NOTNULL(req->htc, HTTP_CONN_MAGIC);
+	CHECK_OBJ_NOTNULL(req->htc->vfc, VFP_CTX_MAGIC);
 	vfc = req->htc->vfc;
-	VFP_Setup(vfc);
-	vfc->http = req->http;
-	vfc->wrk = req->wrk;
-	V1F_Setup_Fetch(vfc, req->htc);
 	if (VFP_Open(vfc) < 0) {
 		VSLb(req->vsl, SLT_FetchError, "Could not open Fetch Pipeline");
 		return (-1);
@@ -214,9 +210,8 @@ VRB_Cache(struct req *req, ssize_t maxsize)
 	}
 
 	CHECK_OBJ_NOTNULL(req->htc, HTTP_CONN_MAGIC);
+	CHECK_OBJ_NOTNULL(req->htc->vfc, VFP_CTX_MAGIC);
 	vfc = req->htc->vfc;
-	VFP_Setup(vfc);
-	vfc->wrk = req->wrk;
 
 	if (req->htc->content_length > maxsize) {
 		req->req_body_status = REQ_BODY_FAIL;
@@ -224,14 +219,11 @@ VRB_Cache(struct req *req, ssize_t maxsize)
 		return (-1);
 	}
 
-	vfc->http = req->http;
-
 	req->body_oc = HSH_Private(req->wrk);
 	AN(req->body_oc);
 	XXXAN(STV_NewObject(req->wrk, req->body_oc, TRANSIENT_STORAGE, 8));
 
 	vfc->oc = req->body_oc;
-	V1F_Setup_Fetch(vfc, req->htc);
 
 	if (VFP_Open(vfc) < 0) {
 		req->req_body_status = REQ_BODY_FAIL;
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 2d65fbd..92404b9 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -689,6 +689,12 @@ cnt_recv(struct worker *wrk, struct req *req)
 
 	http_CollectHdr(req->http, H_Cache_Control);
 
+	VFP_Setup(req->htc->vfc);
+	req->htc->vfc->http = req->http;
+	req->htc->vfc->wrk = wrk;
+	if (req->transport->req_body != NULL)
+		req->transport->req_body(req);
+
 	VCL_recv_method(req->vcl, wrk, req, NULL, NULL);
 
 	/* Attempts to cache req.body may fail */
diff --git a/bin/varnishd/cache/cache_transport.h b/bin/varnishd/cache/cache_transport.h
index 6b8d7e9..268eac2 100644
--- a/bin/varnishd/cache/cache_transport.h
+++ b/bin/varnishd/cache/cache_transport.h
@@ -34,6 +34,7 @@
  */
 
 typedef void vtr_deliver_f (struct req *, struct boc *, int sendbody);
+typedef void vtr_req_body_f (struct req *);
 
 struct transport {
 	unsigned		magic;
@@ -44,5 +45,6 @@ struct transport {
 	task_func_t		*new_session;
 	task_func_t		*unwait;
 
+	vtr_req_body_f		*req_body;
 	vtr_deliver_f		*deliver;
 };
diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c
index dff2f0a..592ba8f 100644
--- a/bin/varnishd/http1/cache_http1_fsm.c
+++ b/bin/varnishd/http1/cache_http1_fsm.c
@@ -38,6 +38,7 @@
 #include <stdlib.h>
 
 #include "cache/cache.h"
+#include "cache/cache_filter.h"
 #include "common/heritage.h"
 #include "cache/cache_transport.h"
 #include "cache_http1.h"
@@ -111,11 +112,28 @@ http1_unwait(struct worker *wrk, void *arg)
 	wrk->task.priv = req;
 }
 
+static void __match_proto__(vtr_req_body_t)
+http1_req_body(struct req *req)
+{
+
+	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	switch(req->htc->body_status) {
+	case BS_EOF:
+	case BS_LENGTH:
+	case BS_CHUNKED:
+		V1F_Setup_Fetch(req->htc->vfc, req->htc);
+		break;
+	default:
+		break;
+	}
+}
+
 const struct transport HTTP1_transport = {
 	.name =			"HTTP/1",
 	.magic =		TRANSPORT_MAGIC,
 	.deliver =		V1D_Deliver,
 	.unwait =		http1_unwait,
+	.req_body =		http1_req_body,
 	.new_session =		http1_new_session,
 };
 



From phk at FreeBSD.org  Wed Feb 10 22:27:12 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 10 Feb 2016 23:27:12 +0100
Subject: [master] 7df406f Unify the code which pulls the req.body through VFP.
Message-ID: <E1aTdE8-00029w-Ue@project.varnish-software.com>


commit 7df406fec67052705a19bc441a134ab1ca12030f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 22:26:37 2016 +0000

    Unify the code which pulls the req.body through VFP.
    
    Always use an oc+TRANSIENT for buffering, just don't commit the
    storage if not caching.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 36ecde1..f8722e6 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -932,10 +932,10 @@ int Req_Cleanup(struct sess *sp, struct worker *wrk, struct req *req);
 void Req_Fail(struct req *req, enum sess_close reason);
 
 /* cache_req_body.c */
-int VRB_Ignore(struct req *req);
-ssize_t VRB_Cache(struct req *req, ssize_t maxsize);
-int VRB_Iterate(struct req *req, objiterate_f *func, void *priv);
-void VRB_Free(struct req *req);
+int VRB_Ignore(struct req *);
+ssize_t VRB_Cache(struct req *, ssize_t maxsize);
+ssize_t VRB_Iterate(struct req *, objiterate_f *func, void *priv);
+void VRB_Free(struct req *);
 
 /* cache_req_fsm.c [CNT] */
 enum req_fsm_nxt CNT_Request(struct worker *, struct req *);
diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c
index fa0f98f..e8f7bf9 100644
--- a/bin/varnishd/cache/cache_req_body.c
+++ b/bin/varnishd/cache/cache_req_body.c
@@ -38,6 +38,113 @@
 #include "hash/hash_slinger.h"
 
 /*----------------------------------------------------------------------
+ * Pull the req.body in via/into a objcore
+ *
+ * This can be called only once per request
+ *
+ */
+
+static ssize_t
+vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv)
+{
+	ssize_t l, r = 0, yet;
+	struct vfp_ctx *vfc;
+	uint8_t *ptr;
+	enum vfp_status vfps = VFP_ERROR;
+
+	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+
+	CHECK_OBJ_NOTNULL(req->htc, HTTP_CONN_MAGIC);
+	CHECK_OBJ_NOTNULL(req->htc->vfc, VFP_CTX_MAGIC);
+	vfc = req->htc->vfc;
+
+	req->body_oc = HSH_Private(req->wrk);
+	AN(req->body_oc);
+	XXXAN(STV_NewObject(req->wrk, req->body_oc, TRANSIENT_STORAGE, 8));
+
+	vfc->oc = req->body_oc;
+
+	if (VFP_Open(vfc) < 0) {
+		req->req_body_status = REQ_BODY_FAIL;
+		HSH_DerefBoc(req->wrk, req->body_oc);
+		AZ(HSH_DerefObjCore(req->wrk, &req->body_oc));
+		return (-1);
+	}
+
+	AZ(req->req_bodybytes);
+	AN(req->htc);
+	yet = req->htc->content_length;
+	if (yet < 0)
+		yet = 0;
+	do {
+		AZ(vfc->failed);
+		if (maxsize >= 0 && req->req_bodybytes > maxsize) {
+			(void)VFP_Error(vfc, "Request body too big to cache");
+			break;
+		}
+		l = yet;
+		if (VFP_GetStorage(vfc, &l, &ptr) != VFP_OK)
+			break;
+		AZ(vfc->failed);
+		AN(ptr);
+		AN(l);
+		vfps = VFP_Suck(vfc, ptr, &l);
+		if (l > 0 && vfps != VFP_ERROR) {
+			req->req_bodybytes += l;
+			req->acct.req_bodybytes += l;
+			if (yet >= l)
+				yet -= l;
+			if (func != NULL) {
+				r = func(priv, 1, ptr, l);
+				if (r)
+					break;
+			} else {
+				ObjExtend(req->wrk, req->body_oc, l);
+			}
+		}
+
+	} while (vfps == VFP_OK);
+	VFP_Close(vfc);
+	VSLb_ts_req(req, "ReqBody", VTIM_real());
+	if (func != NULL) {
+		HSH_DerefBoc(req->wrk, req->body_oc);
+		AZ(HSH_DerefObjCore(req->wrk, &req->body_oc));
+		if (vfps != VFP_END) {
+			req->req_body_status = REQ_BODY_FAIL;
+			if (r == 0)
+				r = -1;
+		}
+		return (r);
+	}
+
+	ObjTrimStore(req->wrk, req->body_oc);
+	HSH_DerefBoc(req->wrk, req->body_oc);
+
+	if (vfps != VFP_END) {
+		req->req_body_status = REQ_BODY_FAIL;
+		AZ(HSH_DerefObjCore(req->wrk, &req->body_oc));
+		return (-1);
+	}
+
+	assert(req->req_bodybytes >= 0);
+	if (req->req_bodybytes != req->htc->content_length) {
+		/* We must update also the "pristine" req.* copy */
+		http_Unset(req->http0, H_Content_Length);
+		http_Unset(req->http0, H_Transfer_Encoding);
+		http_PrintfHeader(req->http0, "Content-Length: %ju",
+		    (uintmax_t)req->req_bodybytes);
+
+		http_Unset(req->http, H_Content_Length);
+		http_Unset(req->http, H_Transfer_Encoding);
+		http_PrintfHeader(req->http, "Content-Length: %ju",
+		    (uintmax_t)req->req_bodybytes);
+	}
+
+	req->req_body_status = REQ_BODY_CACHED;
+	return (req->req_bodybytes);
+}
+
+/*----------------------------------------------------------------------
  * Iterate over the req.body.
  *
  * This can be done exactly once if uncached, and multiple times if the
@@ -46,15 +153,10 @@
  * return length or -1 on error
  */
 
-int
+ssize_t
 VRB_Iterate(struct req *req, objiterate_f *func, void *priv)
 {
-	char buf[8192];
-	ssize_t l;
 	int i;
-	struct vfp_ctx *vfc;
-	enum vfp_status vfps = VFP_ERROR;
-	int ret = 0;
 
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
 	AN(func);
@@ -94,37 +196,7 @@ VRB_Iterate(struct req *req, objiterate_f *func, void *priv)
 		    "Multiple attempts to access non-cached req.body");
 		return (i);
 	}
-
-	CHECK_OBJ_NOTNULL(req->htc, HTTP_CONN_MAGIC);
-	CHECK_OBJ_NOTNULL(req->htc->vfc, VFP_CTX_MAGIC);
-	vfc = req->htc->vfc;
-	if (VFP_Open(vfc) < 0) {
-		VSLb(req->vsl, SLT_FetchError, "Could not open Fetch Pipeline");
-		return (-1);
-	}
-
-	do {
-		l = sizeof buf;
-		vfps = VFP_Suck(vfc, buf, &l);
-		if (vfps == VFP_ERROR) {
-			req->req_body_status = REQ_BODY_FAIL;
-			ret = -1;
-			break;
-		} else if (l > 0) {
-			req->req_bodybytes += l;
-			req->acct.req_bodybytes += l;
-			l = func(priv, 1, buf, l);
-			if (l) {
-				req->req_body_status = REQ_BODY_FAIL;
-				ret = -1;
-				break;
-			}
-		}
-	} while (vfps == VFP_OK);
-	VFP_Close(vfc);
-	VSLb_ts_req(req, "ReqBody", VTIM_real());
-
-	return (ret);
+	return (vrb_pull(req, -1, func, priv));
 }
 
 /*----------------------------------------------------------------------
@@ -179,12 +251,9 @@ VRB_Free(struct req *req)
 ssize_t
 VRB_Cache(struct req *req, ssize_t maxsize)
 {
-	ssize_t l, yet;
-	struct vfp_ctx *vfc;
-	uint8_t *ptr;
-	enum vfp_status vfps = VFP_ERROR;
 
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	assert(maxsize >= 0);
 
 	/*
 	 * We only allow caching to happen the first time through vcl_recv{}
@@ -209,88 +278,12 @@ VRB_Cache(struct req *req, ssize_t maxsize)
 		WRONG("Wrong req_body_status in VRB_Cache()");
 	}
 
-	CHECK_OBJ_NOTNULL(req->htc, HTTP_CONN_MAGIC);
-	CHECK_OBJ_NOTNULL(req->htc->vfc, VFP_CTX_MAGIC);
-	vfc = req->htc->vfc;
-
 	if (req->htc->content_length > maxsize) {
 		req->req_body_status = REQ_BODY_FAIL;
-		(void)VFP_Error(vfc, "Request body too big to cache");
-		return (-1);
-	}
-
-	req->body_oc = HSH_Private(req->wrk);
-	AN(req->body_oc);
-	XXXAN(STV_NewObject(req->wrk, req->body_oc, TRANSIENT_STORAGE, 8));
-
-	vfc->oc = req->body_oc;
-
-	if (VFP_Open(vfc) < 0) {
-		req->req_body_status = REQ_BODY_FAIL;
-		HSH_DerefBoc(req->wrk, req->body_oc);
-		AZ(HSH_DerefObjCore(req->wrk, &req->body_oc));
+		(void)VFP_Error(req->htc->vfc,
+		    "Request body too big to cache");
 		return (-1);
 	}
 
-	AZ(req->req_bodybytes);
-	AN(req->htc);
-	yet = req->htc->content_length;
-	if (yet < 0)
-		yet = 0;
-	do {
-		AZ(vfc->failed);
-		if (req->req_bodybytes > maxsize) {
-			req->req_body_status = REQ_BODY_FAIL;
-			(void)VFP_Error(vfc, "Request body too big to cache");
-			HSH_DerefBoc(req->wrk, req->body_oc);
-			AZ(HSH_DerefObjCore(req->wrk, &req->body_oc));
-			VFP_Close(vfc);
-			return(-1);
-		}
-		l = yet;
-		if (VFP_GetStorage(vfc, &l, &ptr) != VFP_OK)
-			break;
-		AZ(vfc->failed);
-		AN(ptr);
-		AN(l);
-		vfps = VFP_Suck(vfc, ptr, &l);
-		if (l > 0 && vfps != VFP_ERROR) {
-			req->req_bodybytes += l;
-			req->acct.req_bodybytes += l;
-			if (yet >= l)
-				yet -= l;
-			ObjExtend(req->wrk, req->body_oc, l);
-		}
-
-	} while (vfps == VFP_OK);
-	VFP_Close(vfc);
-	ObjTrimStore(req->wrk, req->body_oc);
-
-	/* XXX: check missing:
-	    if (req->htc->content_length >= 0)
-		MUSTBE (req->req_bodybytes == req->htc->content_length);
-	*/
-
-	if (vfps == VFP_END) {
-		assert(req->req_bodybytes >= 0);
-		if (req->req_bodybytes != req->htc->content_length) {
-			/* We must update also the "pristine" req.* copy */
-			http_Unset(req->http0, H_Content_Length);
-			http_Unset(req->http0, H_Transfer_Encoding);
-			http_PrintfHeader(req->http0, "Content-Length: %ju",
-			    (uintmax_t)req->req_bodybytes);
-
-			http_Unset(req->http, H_Content_Length);
-			http_Unset(req->http, H_Transfer_Encoding);
-			http_PrintfHeader(req->http, "Content-Length: %ju",
-			    (uintmax_t)req->req_bodybytes);
-		}
-
-		req->req_body_status = REQ_BODY_CACHED;
-	} else {
-		req->req_body_status = REQ_BODY_FAIL;
-	}
-	VSLb_ts_req(req, "ReqBody", VTIM_real());
-	HSH_DerefBoc(req->wrk, req->body_oc);
-	return (vfps == VFP_END ? req->req_bodybytes : -1);
+	return (vrb_pull(req, maxsize, NULL, NULL));
 }



From phk at FreeBSD.org  Thu Feb 11 09:15:39 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 11 Feb 2016 10:15:39 +0100
Subject: [master] 2eb7e01 Take another Init function private
Message-ID: <E1aTnLf-0000Tz-FA@project.varnish-software.com>


commit 2eb7e0142d31642a0ed267432a218ee39cc75d20
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 23:06:10 2016 +0000

    Take another Init function private

diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c
index 15bd0ff..04b1ee4 100644
--- a/bin/varnishd/cache/cache_main.c
+++ b/bin/varnishd/cache/cache_main.c
@@ -33,7 +33,6 @@
 #include <stdlib.h>
 
 #include "cache.h"
-#include "http1/cache_http1.h"
 #include "common/heritage.h"
 
 #include "vcli_priv.h"
diff --git a/bin/varnishd/cache/cache_priv.h b/bin/varnishd/cache/cache_priv.h
index d5b4d10..2676680 100644
--- a/bin/varnishd/cache/cache_priv.h
+++ b/bin/varnishd/cache/cache_priv.h
@@ -116,6 +116,9 @@ void VRTPRIV_dynamic_kill(struct vrt_privs *privs, uintptr_t id);
 /* cache_vrt_vmod.c */
 void VMOD_Init(void);
 
+/* http1/cache_http1_pipe.c */
+void V1P_Init(void);
+
 /* stevedore.c */
 void STV_open(void);
 void STV_close(void);
diff --git a/bin/varnishd/http1/cache_http1.h b/bin/varnishd/http1/cache_http1.h
index 597b432..07f2d63 100644
--- a/bin/varnishd/http1/cache_http1.h
+++ b/bin/varnishd/http1/cache_http1.h
@@ -49,7 +49,6 @@ struct v1p_acct {
 	uint64_t        out;
 };
 
-void V1P_Init(void);
 void V1P_Process(struct req *, int fd, struct v1p_acct *);
 void V1P_Charge(struct req *, const struct v1p_acct *, struct VSC_C_vbe *);
 



From phk at FreeBSD.org  Thu Feb 11 09:15:39 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 11 Feb 2016 10:15:39 +0100
Subject: [master] e5b6657 Comment cleanup
Message-ID: <E1aTnLf-0000U5-H4@project.varnish-software.com>


commit e5b66570d4e0f7831900565ccae704b3e5d85cf0
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 23:53:35 2016 +0000

    Comment cleanup

diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index 281a633..92ddc07 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -28,13 +28,9 @@
  *
  * Session management
  *
- * This is a little bit of a mixed bag, containing both memory management
- * and various state-change functions.
- *
- * The overall picture is complicated by the fact that requests can
- * disembark their worker-threads if they hit a busy object, then come
- * back later in a different worker thread to continue.
- * XXX: I wonder if that complexity pays of any more ?
+ * The overall goal here is to hold as little state as possible for an
+ * idle session.  This leads to various nasty-ish overloads of struct
+ * sess fields, for instance ->fd being negative ->reason.
  *
  */
 



From phk at FreeBSD.org  Thu Feb 11 09:15:39 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 11 Feb 2016 10:15:39 +0100
Subject: [master] d1eb311 More comprehensive test for illegal characters in
 HTTP/1 headers.
Message-ID: <E1aTnLf-0000UE-Im@project.varnish-software.com>


commit d1eb31109f614976f06dd506a63e0fa21185a89b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 11 09:15:07 2016 +0000

    More comprehensive test for illegal characters in HTTP/1 headers.

diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c
index 5734fab..0ae476e 100644
--- a/bin/varnishd/http1/cache_http1_proto.c
+++ b/bin/varnishd/http1/cache_http1_proto.c
@@ -122,10 +122,15 @@ http1_dissect_hdrs(struct http *hp, char *p, struct http_conn *htc)
 		if (vct_iscrlf(p))
 			break;
 		while (r < htc->rxbuf_e) {
-			if (!vct_iscrlf(r)) {
+			if (!vct_isctl(*r) || vct_issp(*r)) {
 				r++;
 				continue;
 			}
+			if (!vct_iscrlf(r)) {
+				VSLb(hp->vsl, SLT_BogoHeader,
+				    "Header has ctrl char 0x%02x", *r);
+				return (400);
+			}
 			q = r;
 			assert(r < htc->rxbuf_e);
 			r += vct_skipcrlf(r);
diff --git a/bin/varnishtest/tests/b00040.vtc b/bin/varnishtest/tests/b00040.vtc
index 7ce7615..123c3a1 100644
--- a/bin/varnishtest/tests/b00040.vtc
+++ b/bin/varnishtest/tests/b00040.vtc
@@ -8,6 +8,15 @@ server s1 {
 
 varnish v1 -vcl+backend { } -start
 
+logexpect l1 -v v1 -g raw {
+	expect * 1001 BogoHeader {1st header has white space:.*}
+	expect * 1003 BogoHeader {1st header has white space:.*}
+	expect * 1005 BogoHeader {Header has ctrl char 0x0d}
+	expect * 1010 BogoHeader {Header has ctrl char 0x01}
+	expect * 1012 BogoHeader {Header has ctrl char 0x0d}
+	expect * 1014 BogoHeader {Header has ctrl char 0x0d}
+} -start
+
 client c1 {
 	send "GET /1 HTTP/1.1\r\n"
 	send " Host: foo\r\n"
@@ -41,3 +50,27 @@ client c1 {
 	rxresp
 	expect resp.status == 200
 } -run
+delay .1
+
+client c1 {
+	send "GET /5 HTTP/1.1\r\nHost: localhost\r\nBogo: Header\001More\r\n\r\n"
+	rxresp
+	expect resp.status == 400
+} -run
+delay .1
+
+client c1 {
+	send "GET /6 HTTP/1.1\r\nHost: localhost\r\nBogo: Header\r\r\n\r\n"
+	rxresp
+	expect resp.status == 400
+} -run
+delay .1
+
+client c1 {
+	send "GET /7 HTTP/1.1\r\nHost: localhost\r\nBogo: Header\rMore\r\n\r\n"
+	rxresp
+	expect resp.status == 400
+} -run
+
+logexpect l1 -wait
+



From phk at FreeBSD.org  Thu Feb 11 09:57:27 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 11 Feb 2016 10:57:27 +0100
Subject: [master] b0e874a Add two session attributes for private storage for
 transport protocols.
Message-ID: <E1aTo07-0006AK-13@project.varnish-software.com>


commit b0e874aa3d963928d36462c05e9e37f83558ef0b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 11 09:56:51 2016 +0000

    Add two session attributes for private storage for transport protocols.

diff --git a/include/tbl/sess_attr.h b/include/tbl/sess_attr.h
index fcdfec0..69b5890 100644
--- a/include/tbl/sess_attr.h
+++ b/include/tbl/sess_attr.h
@@ -39,5 +39,7 @@ SESS_ATTR(CLIENT_ADDR,	  client_addr,	struct suckaddr *, vsa_suckaddr_len)
 SESS_ATTR(SERVER_ADDR,	  server_addr,	struct suckaddr *, vsa_suckaddr_len)
 SESS_ATTR(CLIENT_IP,	  client_ip,	char *,		   -1)
 SESS_ATTR(CLIENT_PORT,	  client_port,	char *,		   -1)
+SESS_ATTR(XPORT_PRIV1,	  xport_priv1,	void *,		   -1)
+SESS_ATTR(XPORT_PRIV2,	  xport_priv2,	uintptr_t,	   -1)
 
 /*lint -restore */



From fgsch at lodoss.net  Thu Feb 11 10:06:19 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Thu, 11 Feb 2016 11:06:19 +0100
Subject: [master] f8a3c2b Reestablish values lost when switching to PARAM
Message-ID: <E1aTo8h-0007GZ-Lw@project.varnish-software.com>


commit f8a3c2b34503630c2fc9ba34e6d685c0ac645813
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Feb 11 09:36:14 2016 +0000

    Reestablish values lost when switching to PARAM
    
    I've verified the rest and only these 2 were affected.

diff --git a/include/tbl/params.h b/include/tbl/params.h
index 611640c..ffa5db9 100644
--- a/include/tbl/params.h
+++ b/include/tbl/params.h
@@ -915,7 +915,7 @@ PARAM(
 	/* name */	shm_reclen,
 	/* typ */	vsl_reclen,
 	/* min */	"16b",
-	/* max */	"4084",
+	/* max */	"65535",
 	/* default */	"255b",
 	/* units */	"bytes",
 	/* flags */	0,
@@ -1375,7 +1375,7 @@ PARAM(
 	/* name */	vsl_reclen,
 	/* typ */	vsl_reclen,
 	/* min */	"16b",
-	/* max */	"4084b",
+	/* max */	"65535",
 	/* default */	"255b",
 	/* units */	"bytes",
 	/* flags */	0,



From lkarsten at varnish-software.com  Thu Feb 11 10:48:28 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 11:48:28 +0100
Subject: [master] 6c548b5 Use the more pythonistic os.path.join().
Message-ID: <E1aTonU-0003yG-3a@project.varnish-software.com>


commit 6c548b52334b02a35fb51bcd955eb66f4df92258
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Thu Feb 11 11:47:22 2016 +0100

    Use the more pythonistic os.path.join().
    
    Using os.path.join() instead of string concatination is easier
    to read.

diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index a54eaa2..2f9dd10 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -39,6 +39,7 @@
 # XXX: does it actually do that ?
 
 import sys
+from os.path import join
 
 srcroot = "../.."
 buildroot = "../.."
@@ -752,7 +753,7 @@ vcltypes = {
 	'STRING_LIST':	"void*",
 }
 
-fi = open(srcroot + "/include/vrt.h")
+fi = open(join(srcroot, "include/vrt.h"))
 
 for i in fi:
 	j = i.split();
@@ -858,7 +859,7 @@ def emit_vcl_tnames(fo, tokens):
 # Read a C-source file and spit out code that outputs it with VSB_cat()
 
 def emit_file(fo, fd, bn):
-	fn = fd + "/" + bn
+	fn = join(fd, bn)
 
 	fi = open(fn)
 	fc = fi.read()
@@ -936,7 +937,7 @@ def file_header(fo):
 
 polish_tokens(tokens)
 
-fo = open(buildroot + "/lib/libvcc/vcc_token_defs.h", "w")
+fo = open(join(buildroot, "lib/libvcc/vcc_token_defs.h"), "w")
 
 file_header(fo)
 
@@ -970,7 +971,7 @@ for i in returns:
 
 #######################################################################
 
-fo = open(buildroot + "/include/tbl/vcl_returns.h", "w")
+fo = open(join(buildroot, "include/tbl/vcl_returns.h"), "w")
 
 file_header(fo)
 
@@ -1008,7 +1009,7 @@ fo.close()
 
 #######################################################################
 
-fo = open(buildroot + "/include/vcl.h", "w")
+fo = open(join(buildroot, "include/vcl.h"), "w")
 
 file_header(fo)
 
@@ -1131,10 +1132,10 @@ def restrict(fo, spec):
 
 #######################################################################
 
-fh = open(buildroot + "/include/vrt_obj.h", "w")
+fh = open(join(buildroot, "include/vrt_obj.h"), "w")
 file_header(fh)
 
-fo = open(buildroot + "/lib/libvcc/vcc_obj.c", "w")
+fo = open(join(buildroot, "lib/libvcc/vcc_obj.c"), "w")
 file_header(fo)
 
 fo.write("""
@@ -1207,7 +1208,7 @@ fh.close()
 
 #######################################################################
 
-fo = open(buildroot + "/lib/libvcc/vcc_fixed_token.c", "w")
+fo = open(join(buildroot, "lib/libvcc/vcc_fixed_token.c"), "w")
 
 file_header(fo)
 fo.write("""
@@ -1241,7 +1242,7 @@ fo.write("""
 fo.close()
 
 #######################################################################
-ft = open(buildroot + "/include/tbl/vcc_types.h", "w")
+ft = open(join(buildroot, "include/tbl/vcc_types.h"), "w")
 file_header(ft)
 
 ft.write("/*lint -save -e525 -e539 */\n")
@@ -1255,7 +1256,7 @@ ft.close()
 
 #######################################################################
 
-fo = open(buildroot + "/include/tbl/vrt_stv_var.h", "w")
+fo = open(join(buildroot, "include/tbl/vrt_stv_var.h"), "w")
 
 file_header(fo)
 
@@ -1295,7 +1296,7 @@ fo.close
 
 #######################################################################
 
-fp_vclvar = open(buildroot + "/doc/sphinx/include/vcl_var.rst", "w")
+fp_vclvar = open(join(buildroot, "doc/sphinx/include/vcl_var.rst"), "w")
 
 l = list()
 for i in sp_variables:



From fgsch at lodoss.net  Thu Feb 11 11:11:35 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Thu, 11 Feb 2016 12:11:35 +0100
Subject: [master] f8dfdb4 Actually, the maximum depends on vsl_buffer
Message-ID: <E1aTp9r-0006Ri-A6@project.varnish-software.com>


commit f8dfdb469a11981d4cef7f04e2de7168d39f46cb
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Feb 11 10:32:57 2016 +0000

    Actually, the maximum depends on vsl_buffer
    
    Pointed out by phk at .

diff --git a/include/tbl/params.h b/include/tbl/params.h
index ffa5db9..402497f 100644
--- a/include/tbl/params.h
+++ b/include/tbl/params.h
@@ -915,7 +915,7 @@ PARAM(
 	/* name */	shm_reclen,
 	/* typ */	vsl_reclen,
 	/* min */	"16b",
-	/* max */	"65535",
+	/* max */	NULL,
 	/* default */	"255b",
 	/* units */	"bytes",
 	/* flags */	0,
@@ -1375,7 +1375,7 @@ PARAM(
 	/* name */	vsl_reclen,
 	/* typ */	vsl_reclen,
 	/* min */	"16b",
-	/* max */	"65535",
+	/* max */	NULL,
 	/* default */	"255b",
 	/* units */	"bytes",
 	/* flags */	0,



From lkarsten at varnish-software.com  Thu Feb 11 12:01:09 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 13:01:09 +0100
Subject: [master] ecbdb32 Make output file prefix configurable.
Message-ID: <E1aTpvp-0004Bs-2Y@project.varnish-software.com>


commit ecbdb32a61ca0fc1ad54f5d329d937bc636cd5a1
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Thu Feb 11 12:56:40 2016 +0100

    Make output file prefix configurable.
    
    When building multiple vmods from the same source directory,
    vmodtool would trample on other concurrent jobs' vcc_if.[ch].

diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index e8f890f..7662557 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -840,7 +840,7 @@ class SimpleTestCase(unittest.TestCase):
 
 
 #######################################################################
-def runmain(inputvcc, outputname="vcc_if"):
+def runmain(inputvcc, outputprefix="vcc_if"):
 	# Read the file in
 	lines = []
 	with open(inputvcc, "r") as fp:
@@ -904,8 +904,8 @@ def runmain(inputvcc, outputname="vcc_if"):
 	# Parsing done, now process
 	#
 
-	fc = open("%s.c" % outputname, "w")
-	fh = open("%s.h" % outputname, "w")
+	fc = open("%s.c" % outputprefix, "w")
+	fh = open("%s.h" % outputprefix, "w")
 
 	write_c_file_warning(fc)
 	write_c_file_warning(fh)
@@ -951,6 +951,8 @@ if __name__ == "__main__":
 
 	oparser.add_option('-N', '--strict', action='store_true', default=False,
 	    help="Be strict when parsing input file. (vmod.vcc)")
+	oparser.add_option('-o', metavar="prefix", default='vcc_if',
+	    help='Output file prefix. (default: "vcc_if")')
 	oparser.add_option('', '--runtests', action='store_true', default=False,
 	    dest="runtests", help=optparse.SUPPRESS_HELP)
 	(opts, args) = oparser.parse_args()
@@ -973,4 +975,4 @@ if __name__ == "__main__":
 		oparser.print_help()
 		exit(-1)
 
-	runmain(i_vcc)
+	runmain(i_vcc, outputprefix=opts.o)



From lkarsten at varnish-software.com  Thu Feb 11 12:01:09 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 13:01:09 +0100
Subject: [master] b47add8 Use os.path.join().
Message-ID: <E1aTpvp-0004Bm-0g@project.varnish-software.com>


commit b47add8506b62d208de76661cb236988d0022fa6
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Thu Feb 11 12:56:33 2016 +0100

    Use os.path.join().

diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index 0d55316..e8f890f 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -42,7 +42,7 @@ import optparse
 import unittest
 import random
 from os import unlink
-from os.path import dirname, realpath, exists
+from os.path import dirname, exists, join, realpath
 from pprint import pprint, pformat
 
 ctypes = {
@@ -833,10 +833,10 @@ class SimpleTestCase(unittest.TestCase):
 		from glob import glob
 		tmpfile = mktemp()
 		bdir = dirname(realpath(__file__))
-		for inputfile in glob(bdir + "/../libvmod_*/vmod.vcc"):
-			runmain(inputfile, outputname=tmpfile)
-			for suffix in [".c", ".h"]:
-				unlink(tmpfile + suffix)
+		for inputfile in glob(join(bdir, "../libvmod_*/vmod.vcc")):
+			runmain(inputfile, outputprefix=tmpfile)
+			unlink(tmpfile + ".c")
+			unlink(tmpfile + ".h")
 
 
 #######################################################################



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] aac1c61 More comprehensive test for illegal characters in
 HTTP/1 headers.
Message-ID: <E1aTtt2-0002FB-4v@project.varnish-software.com>


commit aac1c6148c71837c9f198198129cd0e76c5a27b2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 11 09:15:07 2016 +0000

    More comprehensive test for illegal characters in HTTP/1 headers.

diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c
index deb191d..2bca363 100644
--- a/bin/varnishd/http1/cache_http1_proto.c
+++ b/bin/varnishd/http1/cache_http1_proto.c
@@ -122,10 +122,15 @@ http1_dissect_hdrs(struct http *hp, char *p, struct http_conn *htc)
 		if (vct_iscrlf(p))
 			break;
 		while (r < htc->rxbuf_e) {
-			if (!vct_iscrlf(r)) {
+			if (!vct_isctl(*r) || vct_issp(*r)) {
 				r++;
 				continue;
 			}
+			if (!vct_iscrlf(r)) {
+				VSLb(hp->vsl, SLT_BogoHeader,
+				    "Header has ctrl char 0x%02x", *r);
+				return (400);
+			}
 			q = r;
 			assert(r < htc->rxbuf_e);
 			r += vct_skipcrlf(r);
diff --git a/bin/varnishtest/tests/b00040.vtc b/bin/varnishtest/tests/b00040.vtc
index 7ce7615..123c3a1 100644
--- a/bin/varnishtest/tests/b00040.vtc
+++ b/bin/varnishtest/tests/b00040.vtc
@@ -8,6 +8,15 @@ server s1 {
 
 varnish v1 -vcl+backend { } -start
 
+logexpect l1 -v v1 -g raw {
+	expect * 1001 BogoHeader {1st header has white space:.*}
+	expect * 1003 BogoHeader {1st header has white space:.*}
+	expect * 1005 BogoHeader {Header has ctrl char 0x0d}
+	expect * 1010 BogoHeader {Header has ctrl char 0x01}
+	expect * 1012 BogoHeader {Header has ctrl char 0x0d}
+	expect * 1014 BogoHeader {Header has ctrl char 0x0d}
+} -start
+
 client c1 {
 	send "GET /1 HTTP/1.1\r\n"
 	send " Host: foo\r\n"
@@ -41,3 +50,27 @@ client c1 {
 	rxresp
 	expect resp.status == 200
 } -run
+delay .1
+
+client c1 {
+	send "GET /5 HTTP/1.1\r\nHost: localhost\r\nBogo: Header\001More\r\n\r\n"
+	rxresp
+	expect resp.status == 400
+} -run
+delay .1
+
+client c1 {
+	send "GET /6 HTTP/1.1\r\nHost: localhost\r\nBogo: Header\r\r\n\r\n"
+	rxresp
+	expect resp.status == 400
+} -run
+delay .1
+
+client c1 {
+	send "GET /7 HTTP/1.1\r\nHost: localhost\r\nBogo: Header\rMore\r\n\r\n"
+	rxresp
+	expect resp.status == 400
+} -run
+
+logexpect l1 -wait
+



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] efddff4 Fix timeout calculation to give session-herding a
 chance.
Message-ID: <E1aTtt2-0002FH-6Z@project.varnish-software.com>


commit efddff421c030db3e21f949947626a259ce97c36
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 22:50:57 2016 +0000

    Fix timeout calculation to give session-herding a chance.
    
    Fixes: #1857

diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index 34c4374..bd1ba9a 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -230,10 +230,6 @@ SES_RxStuff(struct http_conn *htc, htc_complete_f *func,
 				*t2 = now;
 			return (HTC_S_COMPLETE);
 		}
-		if (tn < now) {
-			WS_ReleaseP(htc->ws, htc->rxbuf_b);
-			return (HTC_S_TIMEOUT);
-		}
 		if (hs == HTC_S_MORE) {
 			/* Working on it */
 			if (t1 != NULL && isnan(*t1))
@@ -249,6 +245,8 @@ SES_RxStuff(struct http_conn *htc, htc_complete_f *func,
 			WS_ReleaseP(htc->ws, htc->rxbuf_b);
 			return (HTC_S_OVERFLOW);
 		}
+		if (tmo <= 0.0)
+			tmo = 1e-3;
 		i = VTCP_read(htc->fd, htc->rxbuf_e, i, tmo);
 		if (i == 0 || i == -1) {
 			WS_ReleaseP(htc->ws, htc->rxbuf_b);
@@ -257,10 +255,14 @@ SES_RxStuff(struct http_conn *htc, htc_complete_f *func,
 			htc->rxbuf_e += i;
 			*htc->rxbuf_e = '\0';
 		} else if (i == -2) {
-			if (hs == HTC_S_EMPTY && ti < now) {
+			if (hs == HTC_S_EMPTY && ti <= now) {
 				WS_ReleaseP(htc->ws, htc->rxbuf_b);
 				return (HTC_S_IDLE);
 			}
+			if (tn <= now) {
+				WS_ReleaseP(htc->ws, htc->rxbuf_b);
+				return (HTC_S_TIMEOUT);
+			}
 		}
 	}
 }



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 8d02f9c Add testcase for #1857
Message-ID: <E1aTtt2-0002FQ-82@project.varnish-software.com>


commit 8d02f9c1ef5c322c76246de4341fa1b2f3a15179
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 9 23:08:13 2016 +0000

    Add testcase for #1857

diff --git a/bin/varnishtest/tests/r01857.vtc b/bin/varnishtest/tests/r01857.vtc
new file mode 100644
index 0000000..8bb1aa4
--- /dev/null
+++ b/bin/varnishtest/tests/r01857.vtc
@@ -0,0 +1,32 @@
+varnishtest "Check session herding"
+
+server s1 {
+	rxreq
+	txresp -hdr "Connection: close" -body "012345\n"
+} -start
+
+varnish v1 -vcl+backend { } -start
+
+client c1 {
+	txreq -url "/"
+	rxresp
+	expect resp.status == 200
+	expect resp.http.X-Varnish == "1001"
+	delay 1
+	txreq -url "/"
+	rxresp
+	expect resp.status == 200
+	expect resp.http.X-Varnish == "1003 1002"
+} -run
+
+# Give varnish a chance to update stats
+delay .1
+
+varnish v1 -expect sess_herd == 1
+varnish v1 -expect sess_conn == 1
+varnish v1 -expect cache_hit == 1
+varnish v1 -expect cache_miss == 1
+varnish v1 -expect client_req == 2
+varnish v1 -expect s_sess == 1
+varnish v1 -expect s_req == 2
+varnish v1 -expect s_fetch == 1



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 231e949 Minimal initial changeset for 4.1.2.
Message-ID: <E1aTtt2-0002FZ-A1@project.varnish-software.com>


commit 231e94973fab26426734e046958a3ece3e0e0ab2
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Thu Feb 11 10:56:02 2016 +0100

    Minimal initial changeset for 4.1.2.

diff --git a/configure.ac b/configure.ac
index 7b71eaa..dcba307 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2,7 +2,7 @@ AC_PREREQ(2.59)
 AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS
 Copyright (c) 2006-2016 Varnish Software AS])
 AC_REVISION([$Id$])
-AC_INIT([Varnish], [4.1.1], [varnish-dev at varnish-cache.org])
+AC_INIT([Varnish], [4.1.2-dev], [varnish-dev at varnish-cache.org])
 AC_CONFIG_SRCDIR(include/miniobj.h)
 AC_CONFIG_HEADERS([config.h])
 AC_CONFIG_MACRO_DIR([m4])
diff --git a/doc/changes.rst b/doc/changes.rst
index f06c776..73e93b5 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -1,3 +1,19 @@
+
+================================
+Varnish Cache 4.1.2 (unreleased)
+================================
+
+* Be stricter when parsing a HTTP request to avoid potential
+  HTTP smuggling attacks against vulnerable backends.
+
+Bugs fixed
+----------
+
+* 1857_ - Fix timeout calculation for session herding.
+
+.. _1857: https://www.varnish-cache.org/trac/ticket/1857
+
+
 ================================
 Varnish Cache 4.1.1 (2016-01-28)
 ================================



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 1f3d727 Fix esi:include syntax
Message-ID: <E1aTtt2-0002Fi-Be@project.varnish-software.com>


commit 1f3d727587ba8256786147fdea6c5f036f68fcd5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Jan 19 12:50:45 2016 +0000

    Fix esi:include syntax

diff --git a/bin/varnishtest/tests/r00427.vtc b/bin/varnishtest/tests/r00427.vtc
index a75247b..ebeb2ba 100644
--- a/bin/varnishtest/tests/r00427.vtc
+++ b/bin/varnishtest/tests/r00427.vtc
@@ -3,9 +3,9 @@ varnishtest "client close in ESI delivery"
 server s1 {
 	rxreq
 	txresp -body {
-		<esi:include src="/foo">
-		<esi:include src="/bar">
-		<esi:include src="/barf">
+		<esi:include src="/foo"/>
+		<esi:include src="/bar"/>
+		<esi:include src="/barf"/>
 	}
 
 	rxreq



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 4b0167c Reword
Message-ID: <E1aTtt2-0002Fr-D5@project.varnish-software.com>


commit 4b0167ccf7efa198fa8c9d204f8b822f63cf6eab
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Fri Jan 29 14:57:19 2016 +0000

    Reword

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index a5cd8ce..40b7bf7 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -356,12 +356,12 @@ The per-call vmod_privs are freed before the per-vcl vmod_priv.
 Event functions
 ===============
 
-VMODs can have an "event" function which is called when a VCL which imports
-the VMOD is loaded, made active, or discarded.  This corresponds to the
-``VCL_EVENT_LOAD``, and ``VCL_EVENT_DISCARD`` events, respectively.  In
-addition, this function will be called when the VCL temperature is changed to
-cold or warm, corresponding to the ``VCL_EVENT_COLD`` and ``VCL_EVENT_WARM``
-events.
+VMODs can have an "event" function which is called when a VCL which
+imports the VMOD is loaded or discarded.  This corresponds to the
+``VCL_EVENT_LOAD`` and ``VCL_EVENT_DISCARD`` events, respectively.
+In addition, this function will be called when the VCL temperature is
+changed to cold or warm, corresponding to the ``VCL_EVENT_COLD`` and
+``VCL_EVENT_WARM`` events.
 
 The first argument to the event function is a VRT context.
 



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] c1ed9e1 Cast time_t to work with 32 bit time_t
Message-ID: <E1aTtt2-0002G0-Ec@project.varnish-software.com>


commit c1ed9e1f7187f6a6c85f388a861247774f32e243
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Feb 1 18:16:22 2016 +0000

    Cast time_t to work with 32 bit time_t
    
    Fixes #1851.

diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c
index 174df57..484d369 100644
--- a/bin/varnishstat/varnishstat_curses.c
+++ b/bin/varnishstat/varnishstat_curses.c
@@ -540,8 +540,9 @@ static void
 print_duration(WINDOW *w, time_t t)
 {
 
-	wprintw(w, "%4lu+%02lu:%02lu:%02lu",
-	    t / 86400, (t % 86400) / 3600, (t % 3600) / 60, t % 60);
+	wprintw(w, "%4jd+%02jd:%02jd:%02jd",
+	    (intmax_t)t / 86400, (intmax_t)(t % 86400) / 3600,
+	    (intmax_t)(t % 3600) / 60, (intmax_t)t % 60);
 }
 
 static void



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] f57d8ea We had two vtc_dur variables.  Should be harmless.
Message-ID: <E1aTtt2-0002G9-G3@project.varnish-software.com>


commit f57d8ea884027ee180e9a2116bb81780cc8fead7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 1 23:46:28 2016 +0000

    We had two vtc_dur variables.  Should be harmless.

diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index 33b56e1..97ade30 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -56,7 +56,6 @@ volatile sig_atomic_t	vtc_error;	/* Error encountered */
 int			vtc_stop;	/* Stops current test without error */
 pthread_t		vtc_thread;
 static struct vtclog	*vltop;
-unsigned		vtc_maxdur;	/* Max duration of any test */
 
 /**********************************************************************
  * Macro facility



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 1a92e35 Add a missing VDP flush operation after ESI:includes.
Message-ID: <E1aTtt2-0002GI-HZ@project.varnish-software.com>


commit 1a92e35518210636e09cfb485a692de7570a218b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 00:22:04 2016 +0000

    Add a missing VDP flush operation after ESI:includes.
    
    In incredibly seldom circumstances this bug would cause ESI:include
    data to be bad.
    
    Fixes:	#1852

diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index abc705b..1bfd365 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -761,6 +761,7 @@ VED_Deliver(struct req *req, struct busyobj *bo, int wantbody)
 		else
 			VDP_push(req, ved_vdp_bytes, ecx->preq, 1);
 		(void)VDP_DeliverObj(req);
+		(void)VDP_bytes(req, VDP_FLUSH, NULL, 0);
 	}
 	VDP_close(req);
 }



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 6616996 Neuter another case where AddressSanitizer chokes on
 FreeBSD's hyperwide memcmp()
Message-ID: <E1aTtt2-0002GT-JF@project.varnish-software.com>


commit 6616996ace7f99ae42c976e1fabc346dc5396039
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 00:26:39 2016 +0000

    Neuter another case where AddressSanitizer chokes on FreeBSD's
    hyperwide memcmp()

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 2b71d94..2838914 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -143,10 +143,10 @@ ban_equal(const uint8_t *bs1, const uint8_t *bs2)
 
 	/*
 	 * Compare two ban-strings.
-	 * The memcmp() is safe because the first field we compare is the
-	 * length and that is part of the fixed header structure.
 	 */
 	u = vbe32dec(bs1 + BANS_LENGTH);
+	if (u != vbe32dec(bs2 + BANS_LENGTH))
+		return (0);
 	return (!memcmp(bs1 + BANS_LENGTH, bs2 + BANS_LENGTH, u - BANS_LENGTH));
 }
 



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 7f9dfc0 Don't use WS_Overflowed() from panic dumps,
 it has asserts which can recurse.
Message-ID: <E1aTtt2-0002Gc-Kd@project.varnish-software.com>


commit 7f9dfc027382bd28b55e85e59c1c46d92666f3ec
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 08:17:06 2016 +0000

    Don't use WS_Overflowed() from panic dumps, it has asserts which
    can recurse.

diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 0613604..8e7fd1e 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -115,7 +115,7 @@ pan_ws(struct vsb *vsb, const struct ws *ws)
 
 	VSB_printf(vsb, "ws = %p {\n", ws);
 	VSB_indent(vsb, 2);
-	if (WS_Overflowed(ws))
+	if (ws->id[0] & 0x20)
 		VSB_printf(vsb, "OVERFLOW ");
 	VSB_printf(vsb, "id = \"%s\",\n",  ws->id);
 	VSB_printf(vsb, "{s,f,r,e} = {%p",  ws->s);
diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c
index 7070f1a..615c9bb 100644
--- a/bin/varnishd/cache/cache_ws.c
+++ b/bin/varnishd/cache/cache_ws.c
@@ -251,6 +251,7 @@ WS_ReleaseP(struct ws *ws, char *ptr)
 	ws->r = NULL;
 	WS_Assert(ws);
 }
+
 int
 WS_Overflowed(const struct ws *ws)
 {



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 96b2e2d Get the sign right on the overflow test
Message-ID: <E1aTtt2-0002Go-M0@project.varnish-software.com>


commit 96b2e2de336e569df82f8fdb64e909691b281e9f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 09:14:27 2016 +0000

    Get the sign right on the overflow test
    
    Spotted by:	fgs

diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 8e7fd1e..eb4aa2c 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -115,8 +115,8 @@ pan_ws(struct vsb *vsb, const struct ws *ws)
 
 	VSB_printf(vsb, "ws = %p {\n", ws);
 	VSB_indent(vsb, 2);
-	if (ws->id[0] & 0x20)
-		VSB_printf(vsb, "OVERFLOW ");
+	if (!(ws->id[0] & 0x20))
+		VSB_printf(vsb, "OVERFLOWED ");
 	VSB_printf(vsb, "id = \"%s\",\n",  ws->id);
 	VSB_printf(vsb, "{s,f,r,e} = {%p",  ws->s);
 	if (ws->f > ws->s)



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 3885d2c Error earlier if we cannot find the probe
Message-ID: <E1aTtt2-0002Gx-Nq@project.varnish-software.com>


commit 3885d2c8fe3d5edca729b607cafc224349208c86
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 2 08:04:20 2016 +0000

    Error earlier if we cannot find the probe
    
    We don't have forward declarations in VCL.  OK'd by phk at .
    Fixes #1837.

diff --git a/bin/varnishtest/tests/r01837.vtc b/bin/varnishtest/tests/r01837.vtc
new file mode 100644
index 0000000..4dc3d68
--- /dev/null
+++ b/bin/varnishtest/tests/r01837.vtc
@@ -0,0 +1,10 @@
+varnishtest "Test VCC errors out if probe is used before it is defined"
+
+varnish v1 -errvcl "Probe p not found" {
+	backend b {
+		.host = "127.0.0.1";
+		.probe = p;
+	}
+	probe p {
+	}
+}
diff --git a/bin/varnishtest/tests/v00034.vtc b/bin/varnishtest/tests/v00034.vtc
index f7d4ed0..41626c1 100644
--- a/bin/varnishtest/tests/v00034.vtc
+++ b/bin/varnishtest/tests/v00034.vtc
@@ -20,7 +20,7 @@ varnish v1 -errvcl {Backend s1 redefined} {
 }
 
 varnish v1 -errvcl {Probe p1 redefined} {
-	backend s1 { .host = "127.0.0.1"; .probe = p1;}
 	probe p1 { }
 	probe p1 { }
+	backend s1 { .host = "127.0.0.1"; .probe = p1;}
 }
diff --git a/lib/libvcc/vcc_backend.c b/lib/libvcc/vcc_backend.c
index 8a413c7..3260f65 100644
--- a/lib/libvcc/vcc_backend.c
+++ b/lib/libvcc/vcc_backend.c
@@ -384,6 +384,12 @@ vcc_ParseHostDef(struct vcc *tl, const struct token *t_be, const char *vgcname)
 			Fb(tl, 0, "\t.probe = &%s,\n", p);
 			ERRCHK(tl);
 		} else if (vcc_IdIs(t_field, "probe") && tl->t->tok == ID) {
+			if (VCC_FindSymbol(tl, tl->t, SYM_PROBE) == NULL) {
+				VSB_printf(tl->sb, "Probe %.*s not found\n",
+				    PF(tl->t));
+				vcc_ErrWhere(tl, tl->t);
+				return;
+			}
 			Fb(tl, 0, "\t.probe = &vgc_probe_%.*s,\n", PF(tl->t));
 			vcc_AddRef(tl, tl->t, SYM_PROBE);
 			vcc_NextToken(tl);



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] b9946b4 Fail HTTP/1.0 POST and PUT requests without C-L
Message-ID: <E1aTtt2-0002H6-PL@project.varnish-software.com>


commit b9946b4b0bc65b935580fe7db999b1b6c5558e19
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 2 08:17:13 2016 +0000

    Fail HTTP/1.0 POST and PUT requests without C-L
    
    It is not allowed by the spec and we incorrectly assumed chunked
    and eventually timed out.  OK'd by phk at .
    Fixes #1843.

diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c
index 2bca363..0ae476e 100644
--- a/bin/varnishd/http1/cache_http1_proto.c
+++ b/bin/varnishd/http1/cache_http1_proto.c
@@ -288,6 +288,8 @@ http1_body_status(const struct http *hp, struct http_conn *htc)
 	htc->content_length = -1;
 
 	cl = http_GetContentLength(hp);
+	if (cl == -2)
+		return (BS_ERROR);
 	if (http_GetHdr(hp, H_Transfer_Encoding, &b)) {
 		if (strcasecmp(b, "chunked"))
 			return (BS_ERROR);
@@ -300,8 +302,6 @@ http1_body_status(const struct http *hp, struct http_conn *htc)
 		}
 		return (BS_CHUNKED);
 	}
-	if (cl == -2)
-		return (BS_ERROR);
 	if (cl >= 0) {
 		htc->content_length = cl;
 		return (cl == 0 ? BS_NONE : BS_LENGTH);
@@ -381,10 +381,13 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp)
 	p = http_GetMethod(hp);
 	AN(p);
 
-	/* We handle EOF bodies only for PUT and POST */
-	if (htc->body_status == BS_EOF &&
-	    strcasecmp(p, "put") && strcasecmp(p, "post"))
+	if (htc->body_status == BS_EOF) {
+		assert(hp->protover == 10);
+		/* RFC1945 8.3 p32 and D.1.1 p58 */
+		if (!strcasecmp(p, "post") || !strcasecmp(p, "put"))
+			return (400);
 		htc->body_status = BS_NONE;
+	}
 
 	/* HEAD with a body is a hard error */
 	if (htc->body_status != BS_NONE && !strcasecmp(p, "head"))
diff --git a/bin/varnishtest/tests/r01843.vtc b/bin/varnishtest/tests/r01843.vtc
new file mode 100644
index 0000000..20d37c9
--- /dev/null
+++ b/bin/varnishtest/tests/r01843.vtc
@@ -0,0 +1,17 @@
+varnishtest "HTTP/1.0 POST and PUT need a valid Content-Length"
+
+server s1 { } -start
+
+varnish v1 -vcl+backend {} -start
+
+client c1 {
+	txreq -proto HTTP/1.0 -req "POST" -nolen
+	rxresp
+	expect resp.status == 400
+} -run
+
+client c1 {
+	txreq -proto HTTP/1.0 -req "PUT" -nolen
+	rxresp
+	expect resp.status == 400
+} -run



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] daf5059 Add REAL add/sub/mul/div REAL support
Message-ID: <E1aTtt2-0002HF-Qv@project.varnish-software.com>


commit daf5059ba69a31020d84cfa6a4dd403c96d40366
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 2 08:43:29 2016 +0000

    Add REAL add/sub/mul/div REAL support
    
    Prompted by someone on #varnish.  OK'd by phk.

diff --git a/bin/varnishtest/tests/v00020.vtc b/bin/varnishtest/tests/v00020.vtc
index a9e9d3b..263b883 100644
--- a/bin/varnishtest/tests/v00020.vtc
+++ b/bin/varnishtest/tests/v00020.vtc
@@ -102,10 +102,18 @@ varnish v1 -vcl {
 
 		set req.http.foo = now + 1s;
 		set req.http.foo = now - 1s;
+
 		set req.http.foo = now - now;
 
 		set req.http.foo = 1 + 1;
 		set req.http.foo = 1 - 1;
+		set req.http.foo = 3 * 2;
+		set req.http.foo = 3 / 2;
+
+		set req.http.foo = 3.6 + 1.4;
+		set req.http.foo = 3.6 - 1.4;
+		set req.http.foo = 3.6 * 1.4;
+		set req.http.foo = 3.6 / 1.4;
 
 		set req.ttl = 1s;
 	}
diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c
index 5989f07..af3b3e7 100644
--- a/lib/libvcc/vcc_expr.c
+++ b/lib/libvcc/vcc_expr.c
@@ -942,6 +942,7 @@ vcc_expr_mul(struct vcc *tl, struct expr **e, enum var_type fmt)
 	case INT:	f2 = INT; break;
 	case DURATION:	f2 = REAL; break;
 	case BYTES:	f2 = REAL; break;
+	case REAL:	f2 = REAL; break;
 	default:
 		if (tl->t->tok != '*' && tl->t->tok != '/')
 			return;
@@ -1049,6 +1050,8 @@ vcc_expr_add(struct vcc *tl, struct expr **e, enum var_type fmt)
 			/* OK */
 		} else if ((*e)->fmt == INT && e2->fmt == INT) {
 			/* OK */
+		} else if ((*e)->fmt == REAL && e2->fmt == REAL) {
+			/* OK */
 		} else if ((*e)->fmt == DURATION && e2->fmt == DURATION) {
 			/* OK */
 		} else if (tk->tok == '+' &&



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 79b099d Don't leak a filename in error exits,
 it upsets Coverity.
Message-ID: <E1aTtt2-0002HO-SP@project.varnish-software.com>


commit 79b099d8f191be15f9577627c767dbb52ce6639b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 10:06:50 2016 +0000

    Don't leak a filename in error exits, it upsets Coverity.

diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c
index 87331ff..bd524c5 100644
--- a/lib/libvcc/vcc_compile.c
+++ b/lib/libvcc/vcc_compile.c
@@ -534,6 +534,7 @@ vcc_file_source(const struct vcp * const vcp, struct vsb *sb, const char *fn)
 	if (VFIL_searchpath(vcp->vcl_path, NULL, &f, fn, &fnp) || f == NULL) {
 		VSB_printf(sb, "Cannot read file '%s' (%s)\n",
 		    fnp != NULL ? fnp : fn, strerror(errno));
+		REPLACE(fnp, NULL);
 		return (NULL);
 	}
 	sp = vcc_new_source(f, NULL, fnp);
diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c
index 3a522a5..62f39fc 100644
--- a/lib/libvcc/vcc_vmod.c
+++ b/lib/libvcc/vcc_vmod.c
@@ -61,7 +61,7 @@ void
 vcc_ParseImport(struct vcc *tl)
 {
 	void *hdl;
-	char fn[1024], *fnp;
+	char fn[1024], *fnp, *fnpx;
 	char buf[256];
 	struct token *mod, *t1;
 	struct inifin *ifp;
@@ -130,14 +130,19 @@ vcc_ParseImport(struct vcc *tl)
 	SkipToken(tl, ';');
 
 	if (VFIL_searchpath(tl->param->vmod_path,
-	    vcc_path_dlopen, &hdl, fn, &fnp)) {
+	    vcc_path_dlopen, &hdl, fn, &fnpx)) {
 		VSB_printf(tl->sb, "Could not load VMOD %.*s\n", PF(mod));
-		VSB_printf(tl->sb, "\tFile name: %s\n", fnp != NULL ? fnp : fn);
+		VSB_printf(tl->sb, "\tFile name: %s\n",
+		    fnpx != NULL ? fnpx : fn);
 		VSB_printf(tl->sb, "\tdlerror: %s\n", dlerror());
 		vcc_ErrWhere(tl, mod);
+		REPLACE(fnpx, NULL);
 		return;
 	}
 
+	fnp = TlDup(tl, fnpx);
+	REPLACE(fnpx, NULL);
+
 	bprintf(buf, "Vmod_%.*s_Data", PF(mod));
 	vmd = dlsym(hdl, buf);
 	if (vmd == NULL) {
@@ -195,7 +200,6 @@ vcc_ParseImport(struct vcc *tl)
 	VSB_printf(ifp->ini, "\t    \"%.*s\",\n", PF(mod));
 	VSB_printf(ifp->ini, "\t    ");
 	EncString(ifp->ini, fnp, NULL, 0);
-	free(fnp);
 	VSB_printf(ifp->ini, ",\n");
 	AN(vmd);
 	AN(vmd->file_id);



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 2039d8a Cave into to gcc49's paranoia...
Message-ID: <E1aTtt2-0002HX-Tw@project.varnish-software.com>


commit 2039d8a574e33386591ab2bdc6f4eaa9ae569127
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 10:24:50 2016 +0000

    Cave into to gcc49's paranoia...

diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c
index bd524c5..5db516f 100644
--- a/lib/libvcc/vcc_compile.c
+++ b/lib/libvcc/vcc_compile.c
@@ -534,7 +534,7 @@ vcc_file_source(const struct vcp * const vcp, struct vsb *sb, const char *fn)
 	if (VFIL_searchpath(vcp->vcl_path, NULL, &f, fn, &fnp) || f == NULL) {
 		VSB_printf(sb, "Cannot read file '%s' (%s)\n",
 		    fnp != NULL ? fnp : fn, strerror(errno));
-		REPLACE(fnp, NULL);
+		free(fnp);
 		return (NULL);
 	}
 	sp = vcc_new_source(f, NULL, fnp);
diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c
index 62f39fc..38515b2 100644
--- a/lib/libvcc/vcc_vmod.c
+++ b/lib/libvcc/vcc_vmod.c
@@ -136,12 +136,13 @@ vcc_ParseImport(struct vcc *tl)
 		    fnpx != NULL ? fnpx : fn);
 		VSB_printf(tl->sb, "\tdlerror: %s\n", dlerror());
 		vcc_ErrWhere(tl, mod);
-		REPLACE(fnpx, NULL);
+		free(fnpx);
 		return;
 	}
 
+	AN(fnpx);
 	fnp = TlDup(tl, fnpx);
-	REPLACE(fnpx, NULL);
+	free(fnpx);
 
 	bprintf(buf, "Vmod_%.*s_Data", PF(mod));
 	vmd = dlsym(hdl, buf);



From lkarsten at varnish-software.com  Thu Feb 11 16:14:32 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:32 +0100
Subject: [4.1] 87f081e Replace alien FD's with /dev/null rather than just
 closing them
Message-ID: <E1aTtt2-0002Hg-VN@project.varnish-software.com>


commit 87f081e6e58d0e51f288f131e209aa31891db38f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 11:51:14 2016 +0000

    Replace alien FD's with /dev/null rather than just closing them
    
    When we fork the worker process, we close all filedescriptors we
    have not explictly marked for it to inherit, for security reasons.
    
    Operating system libraries may have open filedescriptors (see
    end*ent(3)) and there is no way to chase these down.
    
    At least on OSX something related to DNS lookups leaves such
    a FD around, and when that code later discovers the FD doesn't
    work, it closes it, even though it no longer owns it.
    
    In ticket 1841, that happens to be FD7 which is one of our kqueue FDs.
    
    Normally such library code should set 'close-on-exec' status with
    fcntl(2) but that doesn't seem to be the case here, and this bit
    of wisdom seems neglegted about 50/50, so it probably wouldn't
    help us to examine this.
    
    The fix here is to close the FDs, and replace them with a FD open
    to /dev/null, so that there is no risk of information leak, but we
    don't reuse the FD for something else until the library has properly
    closed it.
    
    Fixes #1841

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 4c5ffc3..793a795 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -295,7 +295,7 @@ mgt_launch_child(struct cli *cli)
 	unsigned u;
 	char *p;
 	struct vev *e;
-	int i, cp[2];
+	int i, j, k, cp[2];
 	struct sigaction sa;
 
 	if (child_state != CH_STOPPED && child_state != CH_DIED)
@@ -351,13 +351,29 @@ mgt_launch_child(struct cli *cli)
 		assert(dup2(heritage.std_fd, STDOUT_FILENO) == STDOUT_FILENO);
 		assert(dup2(heritage.std_fd, STDERR_FILENO) == STDERR_FILENO);
 
-		/* Close anything we shouldn't know about */
+		/*
+		 * Close all FDs the child shouldn't know about
+		 *
+		 * We cannot just close these filedescriptors, some random
+		 * library routine might miss it later on and wantonly close
+		 * a FD we use at that point in time. (See bug #1841).
+		 * We close the FD and replace it with /dev/null instead,
+		 * That prevents security leakage, and gives the library
+		 * code a valid FD to close when it discovers the changed
+		 * circumstances.
+		 */
 		closelog();
 
 		for (i = STDERR_FILENO + 1; i < CLOSE_FD_UP_TO; i++) {
 			if (vbit_test(fd_map, i))
 				continue;
-			(void)(close(i) == 0);
+			if (close(i) == 0) {
+				k = open("/dev/null", O_RDONLY);
+				assert(k >= 0);
+				j = dup2(k, i);
+				assert(j == i);
+				AZ(close(k));
+			}
 		}
 #ifdef HAVE_SETPROCTITLE
 		setproctitle("Varnish-Chld %s", heritage.name);



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] c656231 Missing include prompted by -std=c99
Message-ID: <E1aTtt3-0002Hp-0h@project.varnish-software.com>


commit c656231c7c4355783f803f6825c0e515dc58acef
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Feb 8 09:19:29 2016 +0000

    Missing include prompted by -std=c99

diff --git a/lib/libvarnishapi/vsl_query.c b/lib/libvarnishapi/vsl_query.c
index 1741c13..0159429 100644
--- a/lib/libvarnishapi/vsl_query.c
+++ b/lib/libvarnishapi/vsl_query.c
@@ -34,6 +34,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <strings.h>
 #include <unistd.h>
 
 #include "vdef.h"



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] 13586c0 Dont panic because a housekeeping thread races against
 orderly shutdown.
Message-ID: <E1aTtt3-0002Hy-25@project.varnish-software.com>


commit 13586c02912ea8eea777d0e20ada6e57f1cef52e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 10:32:56 2016 +0000

    Dont panic because a housekeeping thread races against orderly
    shutdown.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index a17351d..e983dca 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -506,8 +506,12 @@ vca_acct(void *arg)
 	while (1) {
 		(void)sleep(1);
 		if (vca_tcp_opt_init()) {
-			VTAILQ_FOREACH(ls, &heritage.socks, list)
+			VTAILQ_FOREACH(ls, &heritage.socks, list) {
+				if (ls->sock == -2)
+					continue;	// raced VCA_Shutdown
+				assert (ls->sock > 0);
 				vca_tcp_opt_set(ls->sock, 1);
+			}
 		}
 		now = VTIM_real();
 		VSC_C_main->uptime = (uint64_t)(now - t0);
@@ -581,7 +585,7 @@ VCA_Shutdown(void)
 
 	VTAILQ_FOREACH(ls, &heritage.socks, list) {
 		i = ls->sock;
-		ls->sock = -1;
+		ls->sock = -2;
 		(void)close(i);
 	}
 }



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] 57cb059 Another VCA shutdown race.
Message-ID: <E1aTtt3-0002I7-3S@project.varnish-software.com>


commit 57cb0598f1b7d620db99fe45e759142b84072077
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 10:40:44 2016 +0000

    Another VCA shutdown race.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index e983dca..4fa62eb 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -405,6 +405,12 @@ vca_accept_task(struct worker *wrk, void *arg)
 				   &wa.acceptaddrlen);
 		} while (i < 0 && errno == EAGAIN);
 
+		if (i < 0 && ls->sock == -2) {
+			/* Shut down in progress */
+			sleep(2);
+			continue;
+		}
+
 		if (i < 0) {
 			switch (errno) {
 			case ECONNABORTED:
@@ -416,7 +422,8 @@ vca_accept_task(struct worker *wrk, void *arg)
 			case EBADF:
 				VSL(SLT_Debug, ls->sock, "Accept failed: %s",
 				    strerror(errno));
-				return;
+				vca_pace_bad();
+				break;
 			default:
 				VSL(SLT_Debug, ls->sock, "Accept failed: %s",
 				    strerror(errno));



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] dc7689e Move VCA_Shutdown() call to the main code
Message-ID: <E1aTtt3-0002IG-4s@project.varnish-software.com>


commit dc7689ea068c2ce4003af900bd37a83fe4fe66df
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 11:40:02 2016 +0000

    Move VCA_Shutdown() call to the main code

diff --git a/bin/varnishd/cache/cache_cli.c b/bin/varnishd/cache/cache_cli.c
index 3ba8cdd..1855567 100644
--- a/bin/varnishd/cache/cache_cli.c
+++ b/bin/varnishd/cache/cache_cli.c
@@ -105,7 +105,6 @@ CLI_Run(void)
 		i = VCLS_Poll(cls, -1);
 	} while(i > 0);
 	VSL(SLT_CLI, 0, "EOF on CLI connection, worker stops");
-	VCA_Shutdown();
 }
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c
index 83de0cd..2609d7b 100644
--- a/bin/varnishd/cache/cache_main.c
+++ b/bin/varnishd/cache/cache_main.c
@@ -261,6 +261,7 @@ child_main(void)
 
 	CLI_Run();
 
+	VCA_Shutdown();
 	BAN_Shutdown();
 	STV_close();
 



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] 6a38516 Use global -t timeout for CLI timeout
Message-ID: <E1aTtt3-0002IS-6t@project.varnish-software.com>


commit 6a38516a4fdaf51a6be3b6b8196b81566de766a6
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 11:44:34 2016 +0000

    Use global -t timeout for CLI timeout

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 91c4072..1ff780f 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -109,7 +109,7 @@ varnish_ask_cli(const struct varnish *v, const char *cmd, char **repl)
 			vtc_log(v->vl, 0, "CLI write failed (%s) = %u %s",
 			    cmd, errno, strerror(errno));
 	}
-	i = VCLI_ReadResult(v->cli_fd, &retval, &r, 30.0);
+	i = VCLI_ReadResult(v->cli_fd, &retval, &r, vtc_maxdur);
 	if (i != 0) {
 		vtc_log(v->vl, 0, "CLI failed (%s) = %d %u %s",
 		    cmd, i, retval, r);



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] ca72c40 Under circumstances I don't understand,
 test p00006 takes 52 seconds on osx.
Message-ID: <E1aTtt3-0002IY-8W@project.varnish-software.com>


commit ca72c402db80f36f0b9c79f6756038450592c6a8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 11:49:14 2016 +0000

    Under circumstances I don't understand, test p00006 takes 52 seconds
    on osx.
    
    Use the cli_timeout to control how long the MGR waits before killing
    the worker process, and give this test 60 seconds.

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 793a795..a5e432e 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -483,14 +483,20 @@ mgt_reap_child(void)
 		ev_listen = NULL;
 	}
 
-	/* Wait 10 seconds for child to die */
-	for (i = 0; i < 10; i++) {
+	/* Compose obituary */
+	vsb = VSB_new_auto();
+	XXXAN(vsb);
+
+	/* Wait for child to die */
+	for (i = 0; i < mgt_param.cli_timeout; i++) {
 		r = waitpid(child_pid, &status, WNOHANG);
 		if (r == child_pid)
 			break;
 		(void)sleep(1);
 	}
 	if (r == 0) {
+		VSB_printf(vsb, "Child (%jd) not dying, killing", (intmax_t)r);
+
 		/* Kick it Jim... */
 		if (MGT_FEATURE(FEATURE_NO_COREDUMP))
 			(void)kill(child_pid, SIGKILL);
@@ -504,9 +510,6 @@ mgt_reap_child(void)
 
 	MAC_reopen_sockets(NULL);
 
-	/* Compose obituary */
-	vsb = VSB_new_auto();
-	XXXAN(vsb);
 	VSB_printf(vsb, "Child (%jd) %s", (intmax_t)r,
 	    status ? "died" : "ended");
 	if (WIFEXITED(status) && WEXITSTATUS(status)) {
diff --git a/bin/varnishtest/tests/p00006.vtc b/bin/varnishtest/tests/p00006.vtc
index fb5fcca..9303fdc 100644
--- a/bin/varnishtest/tests/p00006.vtc
+++ b/bin/varnishtest/tests/p00006.vtc
@@ -12,6 +12,7 @@ server s1 {
 
 varnish v1 \
 	-arg "-sdeprecated_persistent,${tmpdir}/_.per,10m" \
+	-arg "-pcli_timeout=60"
 	-arg "-pban_lurker_sleep=0" \
 	-vcl+backend { } -start
 



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] bcc3dc2 forgot a back-slash
Message-ID: <E1aTtt3-0002Ih-A0@project.varnish-software.com>


commit bcc3dc2254d90ac29663f4380e16a764c1f1bc9d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 12:02:50 2016 +0000

    forgot a back-slash

diff --git a/bin/varnishtest/tests/p00006.vtc b/bin/varnishtest/tests/p00006.vtc
index 9303fdc..605a88d 100644
--- a/bin/varnishtest/tests/p00006.vtc
+++ b/bin/varnishtest/tests/p00006.vtc
@@ -12,7 +12,7 @@ server s1 {
 
 varnish v1 \
 	-arg "-sdeprecated_persistent,${tmpdir}/_.per,10m" \
-	-arg "-pcli_timeout=60"
+	-arg "-pcli_timeout=60" \
 	-arg "-pban_lurker_sleep=0" \
 	-vcl+backend { } -start
 



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] 0b30502 Silence warning
Message-ID: <E1aTtt3-0002Iq-Ba@project.varnish-software.com>


commit 0b3050264a075a7d24395e60ae9c7227b4452cf5
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Feb 8 16:36:56 2016 +0000

    Silence warning

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index a5e432e..f7b3732 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -455,7 +455,7 @@ mgt_reap_child(void)
 	int i;
 	int status;
 	struct vsb *vsb;
-	pid_t r;
+	pid_t r = 0;
 
 	assert(child_pid != -1);
 



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] 6e386ef Set socket non-blocking before closing,
 if we are load-shedding it.
Message-ID: <E1aTtt3-0002J9-FW@project.varnish-software.com>


commit 6e386ef64ce7f3e985ddaaaa962ef732bc0855cd
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 20:29:09 2016 +0000

    Set socket non-blocking before closing, if we are load-shedding it.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 4fa62eb..5376203 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -313,9 +313,10 @@ vca_make_session(struct worker *wrk, void *arg)
 		 * connection with minimum effort and fuzz, rather than try
 		 * to send an intelligent message back.
 		 */
+		vca_pace_bad();
+		(void)VTCP_nonblocking(wa->acceptsock);
 		AZ(close(wa->acceptsock));
 		wrk->stats->sess_drop++;
-		vca_pace_bad();
 		WS_Release(wrk->aws, 0);
 		return;
 	}



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] ddc8718 Test esi:included uncompressed synthetic responses
Message-ID: <E1aTtt3-0002Iz-Db@project.varnish-software.com>


commit ddc8718876018a7e0748053e66f0d2d56ba464aa
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Feb 8 09:47:53 2016 +0000

    Test esi:included uncompressed synthetic responses

diff --git a/bin/varnishtest/tests/r01838.vtc b/bin/varnishtest/tests/r01838.vtc
new file mode 100644
index 0000000..ce1f20b
--- /dev/null
+++ b/bin/varnishtest/tests/r01838.vtc
@@ -0,0 +1,32 @@
+varnishtest "Uncompressed synthetic responses as esi includes"
+
+server s1 {
+	rxreq
+	txresp -body {<FOO><esi:include src="/foo"/></FOO>}
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_recv {
+		if (req.url == "/foo") {
+			return (synth(998));
+		}
+	}
+
+	sub vcl_synth {
+		if (resp.status == 998) {
+			synthetic("synthetic body");
+			return (deliver);
+		}
+	}
+
+	sub vcl_backend_response {
+		set beresp.do_esi = true;
+	}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "<FOO>synthetic body</FOO>"
+} -run



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] e5ba8e7 Initialize a variable to quieten down compilers and
 FlexeLint
Message-ID: <E1aTtt3-0002JI-Hs@project.varnish-software.com>


commit e5ba8e767c2a787e97cb7ee7f8c74a86d1110a73
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 8 20:45:18 2016 +0000

    Initialize a variable to quieten down compilers and FlexeLint

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index f7b3732..e602607 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -453,7 +453,7 @@ static void
 mgt_reap_child(void)
 {
 	int i;
-	int status;
+	int status = 0xffff;
 	struct vsb *vsb;
 	pid_t r = 0;
 



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] edf2a34 Enforce that req.body caching can only be done on the
 first tour through vcl_recv{}
Message-ID: <E1aTtt3-0002JR-KV@project.varnish-software.com>


commit edf2a34932c28da6810659ffd567da3b985e6b41
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 11:19:13 2016 +0000

    Enforce that req.body caching can only be done on the first
    tour through vcl_recv{}

diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c
index f50621e..63f0ba2 100644
--- a/bin/varnishd/cache/cache_req_body.c
+++ b/bin/varnishd/cache/cache_req_body.c
@@ -199,6 +199,14 @@ VRB_Cache(struct req *req, ssize_t maxsize)
 
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
 
+	/*
+	 * We only allow caching to happen the first time through vcl_recv{}
+	 * where we know we will have no competition or conflicts for the
+	 * updates to req.http.* etc.
+	 */
+	if (req->restarts > 0 && req->req_body_status != REQ_BODY_CACHED)
+		return (-1);
+
 	assert (req->req_step == R_STP_RECV);
 	switch(req->req_body_status) {
 	case REQ_BODY_CACHED:



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] 2f2e50e improve documentation for ban.list
Message-ID: <E1aTtt3-0002Ja-M1@project.varnish-software.com>


commit 2f2e50e49272815076c4e94aa1a17fcfed17dd02
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Feb 10 14:31:01 2016 +0100

    improve documentation for ban.list

diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst
index 3149b83..ac671ce 100644
--- a/doc/sphinx/reference/varnish-cli.rst
+++ b/doc/sphinx/reference/varnish-cli.rst
@@ -149,7 +149,23 @@ ban <field> <operator> <arg> [&& <field> <oper> <arg> ...]
   Mark obsolete all objects where all the conditions match.
 
 ban.list
-  List the active bans.
+  List the active bans. The output format is:
+
+  * time the ban was issued
+
+  * reference count
+
+  * ``C`` for completed bans (replaced by a newer ban) or ``-``
+
+  * if ``lurker`` debugging is enabled
+
+    * ``R`` for bans on request properties or ``-``
+
+    * ``O`` for bans on object properties or ``-``
+
+    * pointer to the ban object
+
+  * ban specification
 
 Backend Expression
 ------------------



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] 06c0fb6 Comment cleanup
Message-ID: <E1aTtt3-0002Jj-NW@project.varnish-software.com>


commit 06c0fb63fabef6d619870f08d2c8016d30879018
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 10 23:53:35 2016 +0000

    Comment cleanup

diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index bd1ba9a..b5be7ea 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -28,13 +28,9 @@
  *
  * Session management
  *
- * This is a little bit of a mixed bag, containing both memory management
- * and various state-change functions.
- *
- * The overall picture is complicated by the fact that requests can
- * disembark their worker-threads if they hit a busy object, then come
- * back later in a different worker thread to continue.
- * XXX: I wonder if that complexity pays of any more ?
+ * The overall goal here is to hold as little state as possible for an
+ * idle session.  This leads to various nasty-ish overloads of struct
+ * sess fields, for instance ->fd being negative ->reason.
  *
  */
 



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] c09b18b Reestablish values lost when switching to PARAM
Message-ID: <E1aTtt3-0002Js-PH@project.varnish-software.com>


commit c09b18b3789cab996f23565c609adc22063aa3ae
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Feb 11 09:36:14 2016 +0000

    Reestablish values lost when switching to PARAM
    
    I've verified the rest and only these 2 were affected.

diff --git a/include/tbl/params.h b/include/tbl/params.h
index 611640c..ffa5db9 100644
--- a/include/tbl/params.h
+++ b/include/tbl/params.h
@@ -915,7 +915,7 @@ PARAM(
 	/* name */	shm_reclen,
 	/* typ */	vsl_reclen,
 	/* min */	"16b",
-	/* max */	"4084",
+	/* max */	"65535",
 	/* default */	"255b",
 	/* units */	"bytes",
 	/* flags */	0,
@@ -1375,7 +1375,7 @@ PARAM(
 	/* name */	vsl_reclen,
 	/* typ */	vsl_reclen,
 	/* min */	"16b",
-	/* max */	"4084b",
+	/* max */	"65535",
 	/* default */	"255b",
 	/* units */	"bytes",
 	/* flags */	0,



From lkarsten at varnish-software.com  Thu Feb 11 16:14:33 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 11 Feb 2016 17:14:33 +0100
Subject: [4.1] b4359db Actually, the maximum depends on vsl_buffer
Message-ID: <E1aTtt3-0002K1-RE@project.varnish-software.com>


commit b4359dbb82604cfce4ee56bbf901bc9ff5e4272e
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Feb 11 10:32:57 2016 +0000

    Actually, the maximum depends on vsl_buffer
    
    Pointed out by phk at .

diff --git a/include/tbl/params.h b/include/tbl/params.h
index ffa5db9..402497f 100644
--- a/include/tbl/params.h
+++ b/include/tbl/params.h
@@ -915,7 +915,7 @@ PARAM(
 	/* name */	shm_reclen,
 	/* typ */	vsl_reclen,
 	/* min */	"16b",
-	/* max */	"65535",
+	/* max */	NULL,
 	/* default */	"255b",
 	/* units */	"bytes",
 	/* flags */	0,
@@ -1375,7 +1375,7 @@ PARAM(
 	/* name */	vsl_reclen,
 	/* typ */	vsl_reclen,
 	/* min */	"16b",
-	/* max */	"65535",
+	/* max */	NULL,
 	/* default */	"255b",
 	/* units */	"bytes",
 	/* flags */	0,



From phk at FreeBSD.org  Fri Feb 12 10:33:13 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 12 Feb 2016 11:33:13 +0100
Subject: [master] a963289 Shave a layer of indirection of the points for
 session attributes
Message-ID: <E1aUB2H-0003Rt-Oa@project.varnish-software.com>


commit a9632894a87bedb7b5fff051d9ca23dfb26b8c0f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 11 11:02:38 2016 +0000

    Shave a layer of indirection of the points for session attributes

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index f8722e6..cffc68d 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -967,11 +967,13 @@ void SES_RxReInit(struct http_conn *htc);
 enum htc_status_e SES_RxStuff(struct http_conn *, htc_complete_f *,
     double *t1, double *t2, double ti, double tn);
 
-#define SESS_ATTR(UP, low, typ, len)				\
-	int SES_Get_##low(const struct sess *sp, typ *dst);	\
-	void SES_Reserve_##low(struct sess *sp, typ *dst);
+#define SESS_ATTR(UP, low, typ, len)					\
+	int SES_Set_##low(const struct sess *sp, const typ *src);	\
+	int SES_Get_##low(const struct sess *sp, typ **dst);		\
+	void SES_Reserve_##low(struct sess *sp, typ **dst);
 #include "tbl/sess_attr.h"
 #undef SESS_ATTR
+
 void SES_Set_String_Attr(struct sess *sp, enum sess_attr a, const char *src);
 const char *SES_Get_String_Attr(const struct sess *sp, enum sess_attr a);
 
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 2363b25..5f960b4 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -443,8 +443,8 @@ static void
 pan_sess(struct vsb *vsb, const struct sess *sp)
 {
 	const char *stp;
-	char *ci;
-	char *cp;
+	const char *ci;
+	const char *cp;
 
 	VSB_printf(vsb, "sp = %p {\n", sp);
 	if (pan_already(vsb, sp))
@@ -452,8 +452,8 @@ pan_sess(struct vsb *vsb, const struct sess *sp)
 	VSB_indent(vsb, 2);
 	VSB_printf(vsb, "fd = %d, vxid = %u\n",
 	    sp->fd, VXID(sp->vxid));
-	AZ(SES_Get_client_ip(sp, &ci));
-	AZ(SES_Get_client_port(sp, &cp));
+	ci = SES_Get_String_Attr(sp, SA_CLIENT_IP);
+	cp = SES_Get_String_Attr(sp, SA_CLIENT_PORT);
 	VSB_printf(vsb, "client = %s %s,\n", ci, cp);
 	switch (sp->sess_step) {
 #define SESS_STEP(l, u) case S_STP_##u: stp = "S_STP_" #u; break;
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 92404b9..a0c0efd 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -643,7 +643,7 @@ cnt_recv(struct worker *wrk, struct req *req)
 	unsigned recv_handling;
 	struct SHA256Context sha256ctx;
 	const char *xff;
-	char *ci, *cp;
+	const char *ci, *cp;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
@@ -655,8 +655,8 @@ cnt_recv(struct worker *wrk, struct req *req)
 	AZ(isnan(req->t_prev));
 	AZ(isnan(req->t_req));
 
-	AZ(SES_Get_client_ip(req->sp, &ci));
-	AZ(SES_Get_client_port(req->sp, &cp));
+	ci = SES_Get_String_Attr(req->sp, SA_CLIENT_IP);
+	cp = SES_Get_String_Attr(req->sp, SA_CLIENT_PORT);
 	VSLb(req->vsl, SLT_ReqStart, "%s %s", ci, cp);
 
 	http_VSL_log(req->http);
diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index 92ddc07..7999e06 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -66,6 +66,23 @@ ses_get_attr(const struct sess *sp, enum sess_attr a, void **dst)
 	}
 }
 
+static int
+ses_set_attr(const struct sess *sp, enum sess_attr a, const void *src, int sz)
+{
+	void *dst;
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	assert(a < SA_LAST);
+	AN(src);
+	assert(sz > 0);
+
+	if (sp->sattr[a] == 0xffff)
+		return (-1);
+	dst = sp->ws->s + sp->sattr[a];
+	AN(dst);
+	memcpy(dst, src, sz);
+	return (0);
+}
+
 static void
 ses_reserve_attr(struct sess *sp, enum sess_attr a, void **dst, int sz)
 {
@@ -86,16 +103,22 @@ ses_reserve_attr(struct sess *sp, enum sess_attr a, void **dst, int sz)
 
 #define SESS_ATTR(UP, low, typ, len)					\
 	int								\
-	SES_Get_##low(const struct sess *sp, typ *dst)			\
+	SES_Set_##low(const struct sess *sp, const typ *src)		\
+	{								\
+		return (ses_set_attr(sp, SA_##UP, src, len));		\
+	}								\
+									\
+	int								\
+	SES_Get_##low(const struct sess *sp, typ **dst)			\
 	{								\
 		return (ses_get_attr(sp, SA_##UP, (void**)dst));	\
 	}								\
 									\
 	void								\
-	SES_Reserve_##low(struct sess *sp, typ *dst)			\
+	SES_Reserve_##low(struct sess *sp, typ **dst)			\
 	{								\
 		assert(len >= 0);					\
-		ses_reserve_attr(sp, SA_##UP, (void*)dst, len);		\
+		ses_reserve_attr(sp, SA_##UP, (void**)dst, len);	\
 	}
 
 #include "tbl/sess_attr.h"
diff --git a/include/tbl/sess_attr.h b/include/tbl/sess_attr.h
index 69b5890..7355b28 100644
--- a/include/tbl/sess_attr.h
+++ b/include/tbl/sess_attr.h
@@ -32,14 +32,14 @@
 
 /*lint -save -e525 -e539 */
 
-//        upper           lower         type		   len
-SESS_ATTR(REMOTE_ADDR,	  remote_addr,	struct suckaddr *, vsa_suckaddr_len)
-SESS_ATTR(LOCAL_ADDR,	  local_addr,	struct suckaddr *, vsa_suckaddr_len)
-SESS_ATTR(CLIENT_ADDR,	  client_addr,	struct suckaddr *, vsa_suckaddr_len)
-SESS_ATTR(SERVER_ADDR,	  server_addr,	struct suckaddr *, vsa_suckaddr_len)
-SESS_ATTR(CLIENT_IP,	  client_ip,	char *,		   -1)
-SESS_ATTR(CLIENT_PORT,	  client_port,	char *,		   -1)
-SESS_ATTR(XPORT_PRIV1,	  xport_priv1,	void *,		   -1)
-SESS_ATTR(XPORT_PRIV2,	  xport_priv2,	uintptr_t,	   -1)
+//        upper           lower         type			len
+SESS_ATTR(REMOTE_ADDR,	  remote_addr,	struct suckaddr,	vsa_suckaddr_len)
+SESS_ATTR(LOCAL_ADDR,	  local_addr,	struct suckaddr,	vsa_suckaddr_len)
+SESS_ATTR(CLIENT_ADDR,	  client_addr,	struct suckaddr,	vsa_suckaddr_len)
+SESS_ATTR(SERVER_ADDR,	  server_addr,	struct suckaddr,	vsa_suckaddr_len)
+SESS_ATTR(CLIENT_IP,	  client_ip,	char,			-1)
+SESS_ATTR(CLIENT_PORT,	  client_port,	char,			-1)
+SESS_ATTR(XPORT_PRIV1,	  xport_priv1,	void *,			sizeof(void*))
+SESS_ATTR(XPORT_PRIV2,	  xport_priv2,	uintptr_t,		sizeof(uintptr_t))
 
 /*lint -restore */



From phk at FreeBSD.org  Fri Feb 12 10:33:13 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 12 Feb 2016 11:33:13 +0100
Subject: [master] 2a7f495 Isolate mgr/worker better with respect to transport
 protocols.
Message-ID: <E1aUB2H-0003S0-Pr@project.varnish-software.com>


commit 2a7f495ecbdc243f80dbdd3aacde2405afe94572
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 12 10:17:30 2016 +0000

    Isolate mgr/worker better with respect to transport protocols.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 6a701ea..5acbb48 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -356,7 +356,8 @@ vca_make_session(struct worker *wrk, void *arg)
 
 	VTCP_name(sa, laddr, sizeof laddr, lport, sizeof lport);
 
-	VSL(SLT_Begin, sp->vxid, "sess 0 %s", wa->acceptlsock->transport_name);
+	VSL(SLT_Begin, sp->vxid, "sess 0 %s",
+	    wa->acceptlsock->transport->name);
 	VSL(SLT_SessOpen, sp->vxid, "%s %s %s %s %s %.6f %d",
 	    raddr, rport, wa->acceptlsock->name, laddr, lport,
 	    sp->t_open, sp->fd);
@@ -610,3 +611,40 @@ VCA_Shutdown(void)
 		(void)close(i);
 	}
 }
+
+/*--------------------------------------------------------------------
+ * Transport protocol registration
+ *
+ */
+
+static VTAILQ_HEAD(,transport)	transports =
+    VTAILQ_HEAD_INITIALIZER(transports);
+
+void
+XPORT_Init(void)
+{
+	uint16_t n;
+	struct transport *xp;
+
+	ASSERT_MGT();
+
+	VTAILQ_INSERT_TAIL(&transports, &PROXY_transport, list);
+	VTAILQ_INSERT_TAIL(&transports, &HTTP1_transport, list);
+
+	n = 0;
+	VTAILQ_FOREACH(xp, &transports, list)
+		xp->number = ++n;
+}
+
+const struct transport *
+XPORT_Find(const char *name)
+{
+	struct transport *xp;
+
+	ASSERT_MGT();
+
+	VTAILQ_FOREACH(xp, &transports, list)
+		if (!strcasecmp(xp->name, name))
+			return (xp);
+	return (NULL);
+}
diff --git a/bin/varnishd/cache/cache_transport.h b/bin/varnishd/cache/cache_transport.h
index 268eac2..c756f07 100644
--- a/bin/varnishd/cache/cache_transport.h
+++ b/bin/varnishd/cache/cache_transport.h
@@ -33,18 +33,29 @@
  *
  */
 
+struct req;
+struct boc;
+
 typedef void vtr_deliver_f (struct req *, struct boc *, int sendbody);
 typedef void vtr_req_body_f (struct req *);
 
 struct transport {
-	unsigned		magic;
-#define TRANSPORT_MAGIC		0xf157f32f
+	unsigned			magic;
+#define TRANSPORT_MAGIC			0xf157f32f
+
+	uint16_t			number;
+
+	const char			*name;
 
-	const char		*name;
+	task_func_t			*new_session;
+	task_func_t			*unwait;
 
-	task_func_t		*new_session;
-	task_func_t		*unwait;
+	vtr_req_body_f			*req_body;
+	vtr_deliver_f			*deliver;
 
-	vtr_req_body_f		*req_body;
-	vtr_deliver_f		*deliver;
+	VTAILQ_ENTRY(transport)		list;
 };
+
+extern struct transport PROXY_transport;
+extern struct transport HTTP1_transport;
+
diff --git a/bin/varnishd/common/common.h b/bin/varnishd/common/common.h
index a5674a7..eb20956 100644
--- a/bin/varnishd/common/common.h
+++ b/bin/varnishd/common/common.h
@@ -62,6 +62,11 @@ void mgt_child_inherit(int fd, const char *what);
 		exit(2);					\
 	} while (0)
 
+/* cache/cache_acceptor.c */
+struct transport;
+void XPORT_Init(void);
+const struct transport *XPORT_Find(const char *name);
+
 /* cache/cache_vcl.c */
 int VCL_TestLoad(const char *);
 
diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h
index 192d899..6a3d5a0 100644
--- a/bin/varnishd/common/heritage.h
+++ b/bin/varnishd/common/heritage.h
@@ -40,12 +40,8 @@ struct listen_sock {
 	char				*name;
 	struct suckaddr			*addr;
 	const struct transport		*transport;
-	const char			*transport_name;
 };
 
-extern const struct transport PROXY_transport;
-extern const struct transport HTTP1_transport;
-
 VTAILQ_HEAD(listen_sock_head, listen_sock);
 
 struct heritage {
diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c
index 592ba8f..faa1926 100644
--- a/bin/varnishd/http1/cache_http1_fsm.c
+++ b/bin/varnishd/http1/cache_http1_fsm.c
@@ -39,7 +39,6 @@
 
 #include "cache/cache.h"
 #include "cache/cache_filter.h"
-#include "common/heritage.h"
 #include "cache/cache_transport.h"
 #include "cache_http1.h"
 #include "hash/hash_slinger.h"
@@ -128,7 +127,7 @@ http1_req_body(struct req *req)
 	}
 }
 
-const struct transport HTTP1_transport = {
+struct transport HTTP1_transport = {
 	.name =			"HTTP/1",
 	.magic =		TRANSPORT_MAGIC,
 	.deliver =		V1D_Deliver,
diff --git a/bin/varnishd/mgt/mgt_acceptor.c b/bin/varnishd/mgt/mgt_acceptor.c
index bda2063..aa3ce5a 100644
--- a/bin/varnishd/mgt/mgt_acceptor.c
+++ b/bin/varnishd/mgt/mgt_acceptor.c
@@ -49,7 +49,6 @@
 #include "vss.h"
 #include "vtcp.h"
 
-
 static int
 mac_opensocket(struct listen_sock *ls, struct cli *cli)
 {
@@ -116,7 +115,6 @@ struct mac_help {
 #define MAC_HELP_MAGIC		0x1e00a9d9
 	int			good;
 	const char		*name;
-	const char		*transport_name;
 	const struct transport	*transport;
 };
 
@@ -140,7 +138,6 @@ mac_callback(void *priv, const struct suckaddr *sa)
 	AN(ls->addr);
 	ls->name = strdup(mh->name);
 	AN(ls->name);
-	ls->transport_name = mh->transport_name;
 	ls->transport = mh->transport;
 	VTAILQ_INSERT_TAIL(&heritage.socks, ls, list);
 	mh->good++;
@@ -185,6 +182,7 @@ MAC_Arg(const char *arg)
 	struct mac_help *mh;
 	const char *err;
 	int error;
+	const struct transport *xp;
 
 	av = VAV_Parse(arg, NULL, ARGV_COMMA);
 	if (av == NULL)
@@ -196,19 +194,17 @@ MAC_Arg(const char *arg)
 	AN(mh);
 	mh->name = av[1];
 
-	if (av[2] == NULL || !strcmp(av[2], "HTTP/1")) {
-		mh->transport = &HTTP1_transport;
-		mh->transport_name = "HTTP/1";
-		if (av[2] != NULL && av[3] != NULL)
-			ARGV_ERR("Too many sub-arguments to -a(HTTP/1)\n");
-	} else if (!strcmp(av[2], "PROXY")) {
-		mh->transport = &PROXY_transport;
-		mh->transport_name = "PROXY";
-		if (av[3] != NULL)
-			ARGV_ERR("Too many sub-arguments to -a(PROXY)\n");
+	if (av[2] == NULL) {
+		xp = XPORT_Find("http/1");
 	} else {
-		ARGV_ERR("Unknown protocol '%s'\n", av[2]);
+		xp = XPORT_Find(av[2]);
+		if (xp == NULL)
+			ARGV_ERR("Unknown protocol '%s'\n", av[2]);
+		if (av[3] != NULL)
+			ARGV_ERR("Too many sub-arguments to -a(%s)\n", av[2]);
 	}
+	AN(xp);
+	mh->transport = xp;
 
 	error = VSS_resolver(av[1], "80", mac_callback, mh, &err);
 	if (mh->good == 0 || error)
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index dab70fb..ec4ff5a 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -568,6 +568,8 @@ main(int argc, char * const *argv)
 	mgt_evb = vev_new_base();
 	AN(mgt_evb);
 
+	XPORT_Init();
+
 	init_params(cli);
 	cli_check(cli);
 
diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index 8290c7d..cd5bfcf 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -41,8 +41,6 @@
 #include "../cache/cache.h"
 #include "../cache/cache_transport.h"
 
-#include "../common/heritage.h"
-
 #include "vend.h"
 #include "vsa.h"
 #include "vtcp.h"
@@ -383,7 +381,7 @@ vpx_new_session(struct worker *wrk, void *arg)
 	wrk->task.priv = req;
 }
 
-const struct transport PROXY_transport = {
+struct transport PROXY_transport = {
 	.name =			"PROXY",
 	.magic =		TRANSPORT_MAGIC,
 	.new_session =		vpx_new_session,



From phk at FreeBSD.org  Fri Feb 12 10:33:13 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 12 Feb 2016 11:33:13 +0100
Subject: [master] 604d0fb Use one session attribute to hold the number of the
 transport protocol.
Message-ID: <E1aUB2H-0003SA-Qw@project.varnish-software.com>


commit 604d0fbac5ae9c672fe2fa9e6ea26608738ef14f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 12 10:26:29 2016 +0000

    Use one session attribute to hold the number of the transport protocol.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index cffc68d..c1e29fa 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -619,6 +619,7 @@ struct req {
 
 enum sess_attr {
 #define SESS_ATTR(UP, low, typ, len)	SA_##UP,
+	SA_TRANSPORT,
 #include "tbl/sess_attr.h"
 #undef SESS_ATTR
 	SA_LAST
@@ -948,6 +949,8 @@ void SES_Wait(struct sess *, const struct transport *);
 void SES_Delete(struct sess *, enum sess_close reason, double now);
 void SES_NewPool(struct pool *, unsigned pool_no);
 int SES_Reschedule_Req(struct req *);
+void SES_SetTransport(struct worker *, struct sess *, struct req *,
+    const struct transport *);
 
 enum htc_status_e {
 	HTC_S_JUNK =		-5,
diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 5acbb48..ee51e53 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -379,8 +379,7 @@ vca_make_session(struct worker *wrk, void *arg)
 	SES_RxInit(req->htc, req->ws,
 	    cache_param->http_req_size, cache_param->http_req_hdr_len);
 
-	wrk->task.func = wa->acceptlsock->transport->new_session;
-	wrk->task.priv = req;
+	SES_SetTransport(wrk, sp, req, wa->acceptlsock->transport);
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index 7999e06..4208fc5 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -50,6 +50,24 @@
 
 /*--------------------------------------------------------------------*/
 
+void
+SES_SetTransport(struct worker *wrk, struct sess *sp, struct req *req,
+    const struct transport *xp)
+{
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
+	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	CHECK_OBJ_NOTNULL(xp, TRANSPORT_MAGIC);
+	assert(xp->number > 0);
+
+	sp->sattr[SA_TRANSPORT] = xp->number;
+	wrk->task.func = xp->new_session;
+	wrk->task.priv = req;
+}
+
+/*--------------------------------------------------------------------*/
+
 static int
 ses_get_attr(const struct sess *sp, enum sess_attr a, void **dst)
 {
diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index cd5bfcf..ea0d676 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -377,8 +377,7 @@ vpx_new_session(struct worker *wrk, void *arg)
 	else
 		req->htc->pipeline_e = req->htc->rxbuf_e;
 	SES_RxReInit(req->htc);
-	wrk->task.func = HTTP1_transport.new_session;
-	wrk->task.priv = req;
+	SES_SetTransport(wrk, sp, req, &HTTP1_transport);
 }
 
 struct transport PROXY_transport = {
diff --git a/include/tbl/sess_attr.h b/include/tbl/sess_attr.h
index 7355b28..4ab9631 100644
--- a/include/tbl/sess_attr.h
+++ b/include/tbl/sess_attr.h
@@ -32,14 +32,13 @@
 
 /*lint -save -e525 -e539 */
 
-//        upper           lower         type			len
-SESS_ATTR(REMOTE_ADDR,	  remote_addr,	struct suckaddr,	vsa_suckaddr_len)
-SESS_ATTR(LOCAL_ADDR,	  local_addr,	struct suckaddr,	vsa_suckaddr_len)
-SESS_ATTR(CLIENT_ADDR,	  client_addr,	struct suckaddr,	vsa_suckaddr_len)
-SESS_ATTR(SERVER_ADDR,	  server_addr,	struct suckaddr,	vsa_suckaddr_len)
-SESS_ATTR(CLIENT_IP,	  client_ip,	char,			-1)
-SESS_ATTR(CLIENT_PORT,	  client_port,	char,			-1)
-SESS_ATTR(XPORT_PRIV1,	  xport_priv1,	void *,			sizeof(void*))
-SESS_ATTR(XPORT_PRIV2,	  xport_priv2,	uintptr_t,		sizeof(uintptr_t))
+//        upper           lower         type		    len
+SESS_ATTR(REMOTE_ADDR,	  remote_addr,	struct suckaddr,    vsa_suckaddr_len)
+SESS_ATTR(LOCAL_ADDR,	  local_addr,	struct suckaddr,    vsa_suckaddr_len)
+SESS_ATTR(CLIENT_ADDR,	  client_addr,	struct suckaddr,    vsa_suckaddr_len)
+SESS_ATTR(SERVER_ADDR,	  server_addr,	struct suckaddr,    vsa_suckaddr_len)
+SESS_ATTR(CLIENT_IP,	  client_ip,	char,		    -1)
+SESS_ATTR(CLIENT_PORT,	  client_port,	char,		    -1)
+SESS_ATTR(XPORT_PRIV,	  xport_priv,	uintptr_t,	    sizeof(uintptr_t))
 
 /*lint -restore */



From phk at FreeBSD.org  Fri Feb 12 10:33:13 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 12 Feb 2016 11:33:13 +0100
Subject: [master] 3d5911a Give panic access to the sessions transport protocol
 via the session number.
Message-ID: <E1aUB2H-0003SK-SM@project.varnish-software.com>


commit 3d5911a0b2d76b5f5acf728ddca36465a8aa700a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 12 10:32:42 2016 +0000

    Give panic access to the sessions transport protocol via the
    session number.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index ee51e53..42fd3f0 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -638,7 +638,7 @@ XPORT_Init(void)
 const struct transport *
 XPORT_Find(const char *name)
 {
-	struct transport *xp;
+	const struct transport *xp;
 
 	ASSERT_MGT();
 
@@ -647,3 +647,14 @@ XPORT_Find(const char *name)
 			return (xp);
 	return (NULL);
 }
+
+const struct transport *
+XPORT_ByNumber(uint16_t no)
+{
+	const struct transport *xp;
+
+	VTAILQ_FOREACH(xp, &transports, list)
+		if (xp->number == no)
+			return (xp);
+	return (NULL);
+}
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index 5f960b4..c2f0db7 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -445,13 +445,16 @@ pan_sess(struct vsb *vsb, const struct sess *sp)
 	const char *stp;
 	const char *ci;
 	const char *cp;
+	const struct transport *xp;
 
 	VSB_printf(vsb, "sp = %p {\n", sp);
 	if (pan_already(vsb, sp))
 		return;
 	VSB_indent(vsb, 2);
-	VSB_printf(vsb, "fd = %d, vxid = %u\n",
-	    sp->fd, VXID(sp->vxid));
+	xp = XPORT_ByNumber(sp->sattr[SA_TRANSPORT]);
+	VSB_printf(vsb, "fd = %d, vxid = %u, transport = %s\n",
+	    sp->fd, VXID(sp->vxid),
+	    xp == NULL ? "<none>" : xp->name);
 	ci = SES_Get_String_Attr(sp, SA_CLIENT_IP);
 	cp = SES_Get_String_Attr(sp, SA_CLIENT_PORT);
 	VSB_printf(vsb, "client = %s %s,\n", ci, cp);
diff --git a/bin/varnishd/cache/cache_transport.h b/bin/varnishd/cache/cache_transport.h
index c756f07..7048823 100644
--- a/bin/varnishd/cache/cache_transport.h
+++ b/bin/varnishd/cache/cache_transport.h
@@ -59,3 +59,5 @@ struct transport {
 extern struct transport PROXY_transport;
 extern struct transport HTTP1_transport;
 
+const struct transport *XPORT_ByNumber(uint16_t no);
+



From fgsch at lodoss.net  Fri Feb 12 11:02:03 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Fri, 12 Feb 2016 12:02:03 +0100
Subject: [master] dec0da3 Don't abort on unknown keywords by default
Message-ID: <E1aUBUB-0006w4-CG@project.varnish-software.com>


commit dec0da32e4415f095b4f1754571af8f7f31dc8fc
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Fri Feb 12 10:48:18 2016 +0000

    Don't abort on unknown keywords by default
    
    Just warn unless --strict is used.  This allows to use the same .vcc in
    4.0 and 4.1.

diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index 7662557..bb29bd4 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -791,7 +791,15 @@ class FileSection(object):
 			o = parse_func(self, pobj=vx[1].nam)
 			vx[1].add_method(o)
 		else:
-			raise FormatError("Unknown keyword: %s" % t.str, "")
+			if opts.strict:
+				raise FormatError("Unknown keyword: %s" %
+				    t.str, "")
+			else:
+				print("WARNING: Unknown keyword: %s:" %
+				    t.str, file=sys.stderr)
+				o = None
+				while len(self.tl) > 0:
+				    self.get_token()
 
 		assert len(self.tl) == 0
 		if o is None and len(self.l) > 0:



From phk at FreeBSD.org  Fri Feb 12 22:30:33 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 12 Feb 2016 23:30:33 +0100
Subject: [master] 63fcff3 Move the HTTP session steps into the HTTP1 session
 code, and have it contribute panic dumping methods
Message-ID: <E1aUMET-0008Ia-Fl@project.varnish-software.com>


commit 63fcff379e2658d44e16bb12dd66104eb9e689aa
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 12 20:15:37 2016 +0000

    Move the HTTP session steps into the HTTP1 session code, and have
    it contribute panic dumping methods

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index c1e29fa..2b8691a 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -625,31 +625,20 @@ enum sess_attr {
 	SA_LAST
 };
 
-enum sess_step {
-#define SESS_STEP(l, u)		S_STP_##u,
-#include "tbl/steps.h"
-#undef SESS_STEP
-};
-
 struct sess {
 	unsigned		magic;
 #define SESS_MAGIC		0x2c2f9c5a
 
-	enum sess_step		sess_step;
-	struct lock		mtx;
+	uint16_t		sattr[SA_LAST];
 	int			fd;
 	uint32_t		vxid;
 
-	/* Cross references ------------------------------------------*/
+	struct lock		mtx;
 
 	struct pool		*pool;
 
-	/* Session related fields ------------------------------------*/
-
 	struct ws		ws[1];
 
-	uint16_t		sattr[SA_LAST];
-
 	/* Timestamps, all on TIM_real() timescale */
 	double			t_open;		/* fd accepted */
 	double			t_idle;		/* fd accepted or resp sent */
diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c
index c2f0db7..a890d6d 100644
--- a/bin/varnishd/cache/cache_panic.c
+++ b/bin/varnishd/cache/cache_panic.c
@@ -374,15 +374,25 @@ static void
 pan_req(struct vsb *vsb, const struct req *req)
 {
 	const char *stp;
+	const struct transport *xp;
 
 	VSB_printf(vsb, "req = %p {\n", req);
 	if (pan_already(vsb, req))
 		return;
 	VSB_indent(vsb, 2);
 
-	VSB_printf(vsb, "vxid = %u, transport = %s\n", VXID(req->vsl->wid),
-	    req->transport == NULL ? "NULL" : req->transport->name);
+	xp = req->transport;
+	VSB_printf(vsb, "vxid = %u, transport = %s", VXID(req->vsl->wid),
+	    xp == NULL ? "NULL" : xp->name);
 
+	if (xp != NULL && xp->req_panic != NULL) {
+		VSB_printf(vsb, " {\n");
+		VSB_indent(vsb, 2);
+		xp->req_panic(vsb, req);
+		VSB_indent(vsb, -2);
+		VSB_printf(vsb, "}");
+	}
+	VSB_printf(vsb, "\n");
 	switch (req->req_step) {
 #define REQ_STEP(l, u, arg) case R_STP_##u: stp = "R_STP_" #u; break;
 #include "tbl/steps.h"
@@ -442,7 +452,6 @@ pan_req(struct vsb *vsb, const struct req *req)
 static void
 pan_sess(struct vsb *vsb, const struct sess *sp)
 {
-	const char *stp;
 	const char *ci;
 	const char *cp;
 	const struct transport *xp;
@@ -452,22 +461,19 @@ pan_sess(struct vsb *vsb, const struct sess *sp)
 		return;
 	VSB_indent(vsb, 2);
 	xp = XPORT_ByNumber(sp->sattr[SA_TRANSPORT]);
-	VSB_printf(vsb, "fd = %d, vxid = %u, transport = %s\n",
+	VSB_printf(vsb, "fd = %d, vxid = %u, transport = %s",
 	    sp->fd, VXID(sp->vxid),
 	    xp == NULL ? "<none>" : xp->name);
+	if (xp != NULL && xp->sess_panic != NULL) {
+		VSB_printf(vsb, " {\n");
+		VSB_indent(vsb, 2);
+		xp->sess_panic(vsb, sp);
+		VSB_indent(vsb, -2);
+		VSB_printf(vsb, "}");
+	}
 	ci = SES_Get_String_Attr(sp, SA_CLIENT_IP);
 	cp = SES_Get_String_Attr(sp, SA_CLIENT_PORT);
-	VSB_printf(vsb, "client = %s %s,\n", ci, cp);
-	switch (sp->sess_step) {
-#define SESS_STEP(l, u) case S_STP_##u: stp = "S_STP_" #u; break;
-#include "tbl/steps.h"
-#undef SESS_STEP
-		default: stp = NULL;
-	}
-	if (stp != NULL)
-		VSB_printf(vsb, "step = %s,\n", stp);
-	else
-		VSB_printf(vsb, "step = 0x%x,\n", sp->sess_step);
+	VSB_printf(vsb, "\nclient = %s %s,\n", ci, cp);
 
 	VSB_indent(vsb, -2);
 	VSB_printf(vsb, "},\n");
diff --git a/bin/varnishd/cache/cache_transport.h b/bin/varnishd/cache/cache_transport.h
index 7048823..0af1a11 100644
--- a/bin/varnishd/cache/cache_transport.h
+++ b/bin/varnishd/cache/cache_transport.h
@@ -38,6 +38,8 @@ struct boc;
 
 typedef void vtr_deliver_f (struct req *, struct boc *, int sendbody);
 typedef void vtr_req_body_f (struct req *);
+typedef void vtr_sess_panic_f (struct vsb *, const struct sess *);
+typedef void vtr_req_panic_f (struct vsb *, const struct req *);
 
 struct transport {
 	unsigned			magic;
@@ -52,6 +54,8 @@ struct transport {
 
 	vtr_req_body_f			*req_body;
 	vtr_deliver_f			*deliver;
+	vtr_sess_panic_f		*sess_panic;
+	vtr_req_panic_f			*req_panic;
 
 	VTAILQ_ENTRY(transport)		list;
 };
diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c
index faa1926..f30c3b0 100644
--- a/bin/varnishd/http1/cache_http1_fsm.c
+++ b/bin/varnishd/http1/cache_http1_fsm.c
@@ -43,8 +43,32 @@
 #include "cache_http1.h"
 #include "hash/hash_slinger.h"
 
+#include "vsb.h"
 #include "vtcp.h"
 
+static const char H1NEWREQ[] = "HTTP1::NewReq";
+static const char H1PROC[] = "HTTP1::Proc";
+static const char H1BUSY[] = "HTTP1::Busy";
+static const char H1CLEANUP[] = "HTTP1::Cleanup";
+
+static void
+http1_setstate(const struct sess *sp, const char *s)
+{
+	uintptr_t p;
+
+	p = (uintptr_t)s;
+	AZ(SES_Set_xport_priv(sp, &p));
+}
+
+static const char *
+http1_getstate(const struct sess *sp)
+{
+	uintptr_t *p;
+
+	AZ(SES_Get_xport_priv(sp, &p));
+	return (const char *)*p;
+}
+
 /*--------------------------------------------------------------------
  * Call protocol for this request
  */
@@ -81,13 +105,15 @@ http1_new_session(struct worker *wrk, void *arg)
 {
 	struct sess *sp;
 	struct req *req;
+	uintptr_t *u;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CAST_OBJ_NOTNULL(req, arg, REQ_MAGIC);
 	sp = req->sp;
 	CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
 
-	sp->sess_step = S_STP_H1NEWREQ;
+	SES_Reserve_xport_priv(sp, &u);
+	http1_setstate(sp, H1NEWREQ);
 	wrk->task.func = http1_req;
 	wrk->task.priv = req;
 }
@@ -106,7 +132,7 @@ http1_unwait(struct worker *wrk, void *arg)
 	req->htc->fd = sp->fd;
 	SES_RxInit(req->htc, req->ws,
 	    cache_param->http_req_size, cache_param->http_req_hdr_len);
-	sp->sess_step = S_STP_H1NEWREQ;
+	http1_setstate(sp, H1NEWREQ);
 	wrk->task.func = http1_req;
 	wrk->task.priv = req;
 }
@@ -127,6 +153,20 @@ http1_req_body(struct req *req)
 	}
 }
 
+static void
+http1_sess_panic(struct vsb *vsb, const struct sess *sp)
+{
+
+	VSB_printf(vsb, "state = %s\n", http1_getstate(sp));
+}
+
+static void
+http1_req_panic(struct vsb *vsb, const struct req *req)
+{
+
+	VSB_printf(vsb, "state = %s\n", http1_getstate(req->sp));
+}
+
 struct transport HTTP1_transport = {
 	.name =			"HTTP/1",
 	.magic =		TRANSPORT_MAGIC,
@@ -134,6 +174,8 @@ struct transport HTTP1_transport = {
 	.unwait =		http1_unwait,
 	.req_body =		http1_req_body,
 	.new_session =		http1_new_session,
+	.sess_panic =		http1_sess_panic,
+	.req_panic =		http1_req_panic,
 };
 
 /*----------------------------------------------------------------------
@@ -247,6 +289,7 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 {
 	enum htc_status_e hs;
 	struct sess *sp;
+	const char *st;
 	int i;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
@@ -260,7 +303,7 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 	 * or waiter, but we'd rather do the syscall in the worker thread.
 	 * On systems which return errors for ioctl, we close early
 	 */
-	if (sp->sess_step == S_STP_H1NEWREQ && VTCP_blocking(sp->fd)) {
+	if (http1_getstate(sp) == H1NEWREQ && VTCP_blocking(sp->fd)) {
 		if (errno == ECONNRESET)
 			SES_Close(sp, SC_REM_CLOSE);
 		else
@@ -270,8 +313,8 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 	}
 
 	while (1) {
-		switch (sp->sess_step) {
-		case S_STP_H1NEWREQ:
+		st = http1_getstate(sp);
+		if (st == H1NEWREQ) {
 			assert(isnan(req->t_prev));
 			assert(isnan(req->t_req));
 			AZ(req->vcl);
@@ -318,13 +361,12 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 			    req->htc->rxbuf_e - req->htc->rxbuf_b;
 			if (i) {
 				SES_Close(req->sp, req->doclose);
-				sp->sess_step = S_STP_H1CLEANUP;
-				break;
+				http1_setstate(sp, H1CLEANUP);
+			} else {
+				req->req_step = R_STP_RECV;
+				http1_setstate(sp, H1PROC);
 			}
-			req->req_step = R_STP_RECV;
-			sp->sess_step = S_STP_H1PROC;
-			break;
-		case S_STP_H1BUSY:
+		} else if (st == H1BUSY) {
 			/*
 			 * Return from waitinglist.
 			 * Check to see if the remote has left.
@@ -337,30 +379,26 @@ HTTP1_Session(struct worker *wrk, struct req *req)
 				AN(Req_Cleanup(sp, wrk, req));
 				return;
 			}
-			sp->sess_step = S_STP_H1PROC;
-			break;
-		case S_STP_H1PROC:
+			http1_setstate(sp, H1PROC);
+		} else if (st == H1PROC) {
 			req->transport = &HTTP1_transport;
 			if (CNT_Request(wrk, req) == REQ_FSM_DISEMBARK) {
 				req->task.func = http1_req;
 				req->task.priv = req;
-				sp->sess_step = S_STP_H1BUSY;
+				http1_setstate(sp, H1BUSY);
 				return;
 			}
 			req->transport = NULL;
-			sp->sess_step = S_STP_H1CLEANUP;
-			break;
-		case S_STP_H1CLEANUP:
+			http1_setstate(sp, H1CLEANUP);
+		} else if (st == H1CLEANUP) {
 			if (Req_Cleanup(sp, wrk, req))
 				return;
 			SES_RxReInit(req->htc);
 			if (req->htc->rxbuf_e != req->htc->rxbuf_b)
 				wrk->stats->sess_readahead++;
-			sp->sess_step = S_STP_H1NEWREQ;
-			break;
-		default:
+			http1_setstate(sp, H1NEWREQ);
+		} else {
 			WRONG("Wrong H1 session state");
 		}
-
 	}
 }
diff --git a/include/tbl/steps.h b/include/tbl/steps.h
index a2fd654..2d2bf66 100644
--- a/include/tbl/steps.h
+++ b/include/tbl/steps.h
@@ -30,13 +30,6 @@
 
 /*lint -save -e525 -e539 */
 
-#ifdef SESS_STEP
-SESS_STEP(h1newreq,	H1NEWREQ)
-SESS_STEP(h1proc,	H1PROC)
-SESS_STEP(h1busy,	H1BUSY)
-SESS_STEP(h1cleanup,	H1CLEANUP)
-#endif
-
 #ifdef REQ_STEP
 REQ_STEP(restart,	RESTART,	(wrk, req))
 REQ_STEP(recv,		RECV,		(wrk, req))



From phk at FreeBSD.org  Fri Feb 12 22:30:33 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 12 Feb 2016 23:30:33 +0100
Subject: [master] f0e30c5 Get a couple of SES_Close() moved back to the
 central code
Message-ID: <E1aUMET-0008Ih-Hx@project.varnish-software.com>


commit f0e30c574b2478fc5e44fc61441169646b314628
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 12 21:06:58 2016 +0000

    Get a couple of SES_Close() moved back to the central code

diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c
index 0718757..5879518 100644
--- a/bin/varnishd/cache/cache_backend.c
+++ b/bin/varnishd/cache/cache_backend.c
@@ -268,10 +268,11 @@ vbe_dir_getip(const struct director *d, struct worker *wrk,
 
 /*--------------------------------------------------------------------*/
 
-static void
+static enum sess_close
 vbe_dir_http1pipe(const struct director *d, struct req *req, struct busyobj *bo)
 {
 	int i;
+	enum sess_close retval;
 	struct backend *bp;
 	struct v1p_acct v1a;
 	struct vbc *vbc;
@@ -293,7 +294,7 @@ vbe_dir_http1pipe(const struct director *d, struct req *req, struct busyobj *bo)
 
 	if (vbc == NULL) {
 		VSLb(bo->vsl, SLT_FetchError, "no backend connection");
-		SES_Close(req->sp, SC_RX_TIMEOUT);
+		retval = SC_TX_ERROR;
 	} else {
 		i = V1F_SendReq(req->wrk, bo, &v1a.bereq, 1);
 		VSLb_ts_req(req, "Pipe", W_TIM_real(req->wrk));
@@ -302,11 +303,12 @@ vbe_dir_http1pipe(const struct director *d, struct req *req, struct busyobj *bo)
 		if (i == 0)
 			V1P_Process(req, vbc->fd, &v1a);
 		VSLb_ts_req(req, "PipeSess", W_TIM_real(req->wrk));
-		SES_Close(req->sp, SC_TX_PIPE);
 		bo->htc->doclose = SC_TX_PIPE;
 		vbe_dir_finish(d, req->wrk, bo);
+		retval = SC_TX_PIPE;
 	}
 	V1P_Charge(req, &v1a, bp->vsc);
+	return (retval);
 }
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/cache/cache_director.c b/bin/varnishd/cache/cache_director.c
index 9dac151..2fc5b5a 100644
--- a/bin/varnishd/cache/cache_director.c
+++ b/bin/varnishd/cache/cache_director.c
@@ -149,7 +149,7 @@ VDI_Finish(struct worker *wrk, struct busyobj *bo)
 
 /* Get a connection --------------------------------------------------*/
 
-int
+enum sess_close
 VDI_Http1Pipe(struct req *req, struct busyobj *bo)
 {
 	const struct director *d;
@@ -158,10 +158,11 @@ VDI_Http1Pipe(struct req *req, struct busyobj *bo)
 	CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC);
 
 	d = vdi_resolve(req->wrk, bo);
-	if (d == NULL || d->http1pipe == NULL)
-		return (-1);
-	d->http1pipe(d, req, bo);
-	return (0);
+	if (d == NULL || d->http1pipe == NULL) {
+		VSLb(bo->vsl, SLT_VCL_Error, "Backend does not support pipe");
+		return (SC_TX_ERROR);
+	}
+	return (d->http1pipe(d, req, bo));
 }
 
 /* Check health --------------------------------------------------------
diff --git a/bin/varnishd/cache/cache_director.h b/bin/varnishd/cache/cache_director.h
index cf20dc1..a02e086 100644
--- a/bin/varnishd/cache/cache_director.h
+++ b/bin/varnishd/cache/cache_director.h
@@ -58,7 +58,7 @@ typedef const struct suckaddr *vdi_getip_f(const struct director *,
 typedef void vdi_finish_f(const struct director *, struct worker *,
     struct busyobj *);
 
-typedef void vdi_http1pipe_f(const struct director *, struct req *,
+typedef enum sess_close vdi_http1pipe_f(const struct director *, struct req *,
     struct busyobj *);
 
 typedef void vdi_panic_f(const struct director *, struct vsb *);
@@ -88,7 +88,7 @@ const struct suckaddr *VDI_GetIP(struct worker *, struct busyobj *);
 
 void VDI_Finish(struct worker *wrk, struct busyobj *bo);
 
-int VDI_Http1Pipe(struct req *, struct busyobj *);
+enum sess_close VDI_Http1Pipe(struct req *, struct busyobj *);
 
 int VDI_Healthy(const struct director *, const struct busyobj *);
 void VDI_Panic(const struct director *, struct vsb *, const char *nm);
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index a0c0efd..22150d1 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -594,8 +594,7 @@ cnt_pipe(struct worker *wrk, struct req *req)
 		INCOMPL();
 	assert(wrk->handling == VCL_RET_PIPE);
 
-	if (VDI_Http1Pipe(req, bo) < 0)
-		VSLb(bo->vsl, SLT_VCL_Error, "Backend does not support pipe");
+	SES_Close(req->sp, VDI_Http1Pipe(req, bo));
 	http_Teardown(bo->bereq);
 	VBO_ReleaseBusyObj(wrk, &bo);
 	THR_SetBusyobj(NULL);
diff --git a/bin/varnishtest/tests/r01730.vtc b/bin/varnishtest/tests/r01730.vtc
index e54a410..baef368 100644
--- a/bin/varnishtest/tests/r01730.vtc
+++ b/bin/varnishtest/tests/r01730.vtc
@@ -12,4 +12,4 @@ client c1 {
        expect_close
 } -run
 
-varnish v1 -expect sc_rx_timeout == 1
+varnish v1 -expect sc_tx_error == 1



From phk at FreeBSD.org  Fri Feb 12 22:30:33 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 12 Feb 2016 23:30:33 +0100
Subject: [master] 06c9beb Concentrate the "Connection: close" logic in http1
Message-ID: <E1aUMET-0008Ir-Jx@project.varnish-software.com>


commit 06c9beb0ef36fdebbea99f94388aa8da6610b2f2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 12 22:29:54 2016 +0000

    Concentrate the "Connection: close" logic in http1

diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 22150d1..25f4e33 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -299,9 +299,6 @@ cnt_transmit(struct worker *wrk, struct req *req)
 
 	VSLb_ts_req(req, "Resp", W_TIM_real(wrk));
 
-	if (http_HdrIs(req->resp, H_Connection, "close"))
-		req->doclose = SC_RESP_CLOSE;
-
 	if (req->objcore->flags & (OC_F_PRIVATE | OC_F_PASS)) {
 		if (boc != NULL) {
 			HSH_Abandon(req->objcore);
diff --git a/bin/varnishd/http1/cache_http1_deliver.c b/bin/varnishd/http1/cache_http1_deliver.c
index 7885ad2..3af9f48 100644
--- a/bin/varnishd/http1/cache_http1_deliver.c
+++ b/bin/varnishd/http1/cache_http1_deliver.c
@@ -101,7 +101,9 @@ V1D_Deliver(struct req *req, struct boc *boc, int sendbody)
 
 	VSLb(req->vsl, SLT_Debug, "RES_MODE %x", req->res_mode);
 
-	if (req->doclose) {
+	if (!req->doclose && http_HdrIs(req->resp, H_Connection, "close")) {
+		req->doclose = SC_RESP_CLOSE;
+	} else if (req->doclose) {
 		if (!http_HdrIs(req->resp, H_Connection, "close")) {
 			http_Unset(req->resp, H_Connection);
 			http_SetHeader(req->resp, "Connection: close");



From guillaume at varnish-software.com  Mon Feb 15 12:34:34 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Mon, 15 Feb 2016 13:34:34 +0100
Subject: [master] e96478c Add https_scheme feature bit
Message-ID: <E1aVIMM-0008Dm-RD@project.varnish-software.com>


commit e96478c2aea2a5c6f2e3efaff94508c3f0d43d87
Author: Guillaume Quintard <guillaume at varnish-software.com>
Date:   Tue Jan 26 16:56:40 2016 +0100

    Add https_scheme feature bit
    
    Allow to split https URI in the request line

diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c
index 0ae476e..49478da 100644
--- a/bin/varnishd/http1/cache_http1_proto.c
+++ b/bin/varnishd/http1/cache_http1_proto.c
@@ -344,7 +344,7 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp)
 {
 	uint16_t retval;
 	const char *p;
-	const char *b, *e;
+	const char *b = NULL, *e;
 
 	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
 	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
@@ -363,8 +363,12 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp)
 		return (400);
 
 	/* RFC2616, section 5.2, point 1 */
-	if (!strncasecmp(hp->hd[HTTP_HDR_URL].b, "http://", 7)) {
+	if (!strncasecmp(hp->hd[HTTP_HDR_URL].b, "http://", 7))
 		b = e = hp->hd[HTTP_HDR_URL].b + 7;
+	else if (FEATURE(FEATURE_HTTPS_SCHEME) &&
+			!strncasecmp(hp->hd[HTTP_HDR_URL].b, "https://", 8))
+		b = e = hp->hd[HTTP_HDR_URL].b + 8;
+	if (b) {
 		while (*e != '/' && *e != '\0')
 			e++;
 		if (*e == '/') {
diff --git a/bin/varnishtest/tests/r01847.vtc b/bin/varnishtest/tests/r01847.vtc
new file mode 100644
index 0000000..a163fe3
--- /dev/null
+++ b/bin/varnishtest/tests/r01847.vtc
@@ -0,0 +1,45 @@
+varnishtest "Test https_scheme parameter"
+
+server s1 {
+	rxreq
+	expect req.url == /bar
+	txresp
+
+	rxreq
+	expect req.url == https://www.example.com/bar
+	txresp
+} -start
+
+varnish v1 -vcl+backend {
+
+	sub vcl_deliver {
+		set resp.http.rxhost = req.http.host;
+		set resp.http.rxurl = req.url;
+	}
+} -start
+
+client c1 {
+	txreq -url http://www.example.com/bar
+	rxresp
+	expect resp.http.rxhost == www.example.com
+	expect resp.http.rxurl == /bar
+
+	txreq -url https://www.example.com/bar
+	rxresp
+	expect resp.http.rxhost == ""
+	expect resp.http.rxurl == https://www.example.com/bar
+} -run
+
+varnish v1 -cliok "param.set feature +https_scheme"
+
+client c1 {
+	txreq -url http://www.example.com/bar
+	rxresp
+	expect resp.http.rxhost == www.example.com
+	expect resp.http.rxurl == /bar
+
+	txreq -url https://www.example.com/bar
+	rxresp
+	expect resp.http.rxhost == www.example.com
+	expect resp.http.rxurl == /bar
+} -run
diff --git a/include/tbl/feature_bits.h b/include/tbl/feature_bits.h
index 6fcf279..8d8870b 100644
--- a/include/tbl/feature_bits.h
+++ b/include/tbl/feature_bits.h
@@ -59,4 +59,9 @@ FEATURE_BIT(ESI_REMOVE_BOM,		esi_remove_bom,
     "Remove UTF-8 BOM from front of object."
     "Ignore and remove the UTF-8 BOM (0xeb 0xbb 0xbf) from front of object."
 )
+FEATURE_BIT(HTTPS_SCHEME,		https_scheme,
+	"Also split https URIs",
+	"Extract host from full URI in the request line if the scheme is "
+	"https."
+)
 /*lint -restore */



From daghf at varnish-software.com  Mon Feb 15 16:17:39 2016
From: daghf at varnish-software.com (Dag Haavi Finstad)
Date: Mon, 15 Feb 2016 17:17:39 +0100
Subject: [master] 733dc0f fixed minor typo
Message-ID: <E1aVLqF-0007BP-No@project.varnish-software.com>


commit 733dc0f69ae7a4ed3f64768af20efbf2dc1a16db
Author: Valentin Matei <matheuzzy at gmail.com>
Date:   Mon Feb 15 18:01:50 2016 +0200

    fixed minor typo

diff --git a/bin/varnishtest/tests/README b/bin/varnishtest/tests/README
index 85b309b..cf112f8 100644
--- a/bin/varnishtest/tests/README
+++ b/bin/varnishtest/tests/README
@@ -4,7 +4,7 @@ Test-scripts for varnishtest
 Naming scheme
 -------------
 
-	The intent is to be able to run all script sin lexicographic
+	The intent is to be able to run all script in lexicographic
 	order and get a sensible failure mode.
 
 	This requires more basic tests to be earlier and more complex



From dridi.boukelmoune at gmail.com  Tue Feb 16 09:35:26 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 16 Feb 2016 10:35:26 +0100
Subject: [master] 56e52e7 Fix the typo fix
Message-ID: <E1aVc2Y-00056w-6J@project.varnish-software.com>


commit 56e52e70e5d118d281bf6bbf9c4da1cb949075bb
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 16 10:34:43 2016 +0100

    Fix the typo fix

diff --git a/bin/varnishtest/tests/README b/bin/varnishtest/tests/README
index cf112f8..80fd68a 100644
--- a/bin/varnishtest/tests/README
+++ b/bin/varnishtest/tests/README
@@ -4,7 +4,7 @@ Test-scripts for varnishtest
 Naming scheme
 -------------
 
-	The intent is to be able to run all script in lexicographic
+	The intent is to be able to run all scripts in lexicographic
 	order and get a sensible failure mode.
 
 	This requires more basic tests to be earlier and more complex



From fgsch at lodoss.net  Tue Feb 16 17:30:59 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Tue, 16 Feb 2016 18:30:59 +0100
Subject: [master] 13c7a37 Whitespace
Message-ID: <E1aVjSl-0001Xm-RR@project.varnish-software.com>


commit 13c7a372a62a51eeccce870a5531e703110d1de3
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 16 17:28:32 2016 +0000

    Whitespace

diff --git a/Makefile.phk b/Makefile.phk
index 01d3e8c..afeeeb6 100644
--- a/Makefile.phk
+++ b/Makefile.phk
@@ -8,7 +8,7 @@ TOPDIR = $(CURDIR)
 include $(TOPDIR)/Makefile.inc.phk
 
 .PHONY: top_install
-top_install:	
+top_install:
 	@for d in man doc etc bin lib libexec ; do \
 		mkdir -p $(INSTALL_BASE)/$$d ; \
 	done
diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c
index 49478da..a13642f 100644
--- a/bin/varnishd/http1/cache_http1_proto.c
+++ b/bin/varnishd/http1/cache_http1_proto.c
@@ -366,7 +366,7 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp)
 	if (!strncasecmp(hp->hd[HTTP_HDR_URL].b, "http://", 7))
 		b = e = hp->hd[HTTP_HDR_URL].b + 7;
 	else if (FEATURE(FEATURE_HTTPS_SCHEME) &&
-			!strncasecmp(hp->hd[HTTP_HDR_URL].b, "https://", 8))
+	    !strncasecmp(hp->hd[HTTP_HDR_URL].b, "https://", 8))
 		b = e = hp->hd[HTTP_HDR_URL].b + 8;
 	if (b) {
 		while (*e != '/' && *e != '\0')
diff --git a/bin/varnishtest/witness.py b/bin/varnishtest/witness.py
index 6723ef0..f57fdf2 100644
--- a/bin/varnishtest/witness.py
+++ b/bin/varnishtest/witness.py
@@ -49,4 +49,4 @@ for i in fi:
 		l = f[0]
 
 fo.write("}\n")
-	
+
diff --git a/include/tbl/feature_bits.h b/include/tbl/feature_bits.h
index 8d8870b..7bd675f 100644
--- a/include/tbl/feature_bits.h
+++ b/include/tbl/feature_bits.h
@@ -60,8 +60,7 @@ FEATURE_BIT(ESI_REMOVE_BOM,		esi_remove_bom,
     "Ignore and remove the UTF-8 BOM (0xeb 0xbb 0xbf) from front of object."
 )
 FEATURE_BIT(HTTPS_SCHEME,		https_scheme,
-	"Also split https URIs",
-	"Extract host from full URI in the request line if the scheme is "
-	"https."
+    "Also split https URIs",
+    "Extract host from full URI in the request line if the scheme is https."
 )
 /*lint -restore */
diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map
index e5aa91d..cd3940f 100644
--- a/lib/libvarnishapi/libvarnishapi.map
+++ b/lib/libvarnishapi/libvarnishapi.map
@@ -42,7 +42,7 @@ LIBVARNISHAPI_1.0 {
 	VSM_Close;
 	VSM_iter0;
 	VSM_intern;
-	
+
 	VSC_Setup;
 	VSC_Arg;
 	VSC_Open;



From fgsch at lodoss.net  Tue Feb 16 20:28:18 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Tue, 16 Feb 2016 21:28:18 +0100
Subject: [master] 0cdcd31 Make req.body available in varnishtest
Message-ID: <E1aVmEM-0006Yd-Eh@project.varnish-software.com>


commit 0cdcd31108ede87f13e9ff6a10fda60d78ca5b91
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 16 19:59:33 2016 +0000

    Make req.body available in varnishtest

diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index 7822ebc..9160a66 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -219,6 +219,8 @@ cmd_var_resolve(struct http *hp, char *spec)
 		return(hp->chunklen);
 	if (!strcmp(spec, "req.bodylen"))
 		return(hp->bodylen);
+	if (!strcmp(spec, "req.body"))
+		return(hp->body != NULL ? hp->body : spec);
 	if (!strcmp(spec, "resp.bodylen"))
 		return(hp->bodylen);
 	if (!strcmp(spec, "resp.body"))



From fgsch at lodoss.net  Tue Feb 16 20:28:18 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Tue, 16 Feb 2016 21:28:18 +0100
Subject: [master] 9871dd7 Change VRT_CacheReqBody() signature
Message-ID: <E1aVmEM-0006Yk-GP@project.varnish-software.com>


commit 9871dd7ebd64b82d929097683c0cab381687bc81
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 16 20:07:27 2016 +0000

    Change VRT_CacheReqBody() signature
    
    OK'd by phk at .

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 09441e0..50a7232 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -460,14 +460,8 @@ VRT_ban_string(VRT_CTX, const char *str)
 	VAV_Free(av);
 }
 
-/*--------------------------------------------------------------------
- *
- * XXX this really should be ssize_t VRT_CacheReqBody(VRT_CTX, size_t)
- * - change with next VRT major bump
- */
-
-int
-VRT_CacheReqBody(VRT_CTX, long long maxsize)
+ssize_t
+VRT_CacheReqBody(VRT_CTX, size_t maxsize)
 {
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
diff --git a/include/vrt.h b/include/vrt.h
index 1f19cd4..c38fc59 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -31,6 +31,8 @@
  * NB: When this file is changed, lib/libvcc/generate.py *MUST* be rerun.
  */
 
+#include <sys/types.h>
+
 /***********************************************************************
  * Major and minor VRT API versions.
  *
@@ -38,8 +40,8 @@
  * Whenever something is deleted or changed in a way which is not
  * binary/load-time compatible, increment MAJOR version
  *
- * changes to consider with next VRT_MAJOR_VERSION bump:
- * - cache_vrt.c: -> ssize_t VRT_CacheReqBody(VRT_CTX, size_t)
+ * Bump VRT_MAJOR_VERSION due to:
+ * - VRT_CacheReqBody changed signature
  *
  */
 
@@ -242,7 +244,7 @@ void VRT_acl_log(VRT_CTX, const char *msg);
 
 /* req related */
 
-int VRT_CacheReqBody(VRT_CTX, long long maxsize);
+ssize_t VRT_CacheReqBody(VRT_CTX, size_t maxsize);
 
 /* Regexp related */
 void VRT_re_init(void **, const char *);
diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
index c4690b4..3afc937 100644
--- a/lib/libvmod_std/vmod_std.c
+++ b/lib/libvmod_std/vmod_std.c
@@ -228,15 +228,13 @@ vmod_timestamp(VRT_CTX, VCL_STRING label)
 VCL_VOID __match_proto__(td_std_cache_req_body)
 vmod_cache_req_body(VRT_CTX, VCL_BYTES size)
 {
-	int result;
-	ssize_t ss;
+	ssize_t result;
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	if (size < 0)
 		size = 0;
-	ss = (ssize_t)size;
-	result = VRT_CacheReqBody(ctx, ss);
-	VSLb(ctx->vsl, SLT_Debug, "VRT_CacheReqBody(%zd): %d", ss, result);
+	result = VRT_CacheReqBody(ctx, (size_t)size);
+	VSLb(ctx->vsl, SLT_Debug, "VRT_CacheReqBody(%zd): %zd", (size_t)size, result);
 }
 
 VCL_STRING __match_proto__(td_std_strstr)



From fgsch at lodoss.net  Tue Feb 16 20:28:18 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Tue, 16 Feb 2016 21:28:18 +0100
Subject: [master] f5a8f3b Change std.cache_req_body() to return BOOL
Message-ID: <E1aVmEM-0006Yt-Ht@project.varnish-software.com>


commit f5a8f3bf0fcf391e12a6574d06f87f7e4e54723a
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 16 20:16:58 2016 +0000

    Change std.cache_req_body() to return BOOL
    
    As discussed on irc.  OK'd by phk at .

diff --git a/bin/varnishtest/tests/c00055.vtc b/bin/varnishtest/tests/c00055.vtc
index 458718e..ccfb53c 100644
--- a/bin/varnishtest/tests/c00055.vtc
+++ b/bin/varnishtest/tests/c00055.vtc
@@ -10,25 +10,22 @@ server s1 {
 	txresp -hdr "Foo: Foo" -body "56"
 } -start
 
-varnish v1 -cliok "param.set vcc_allow_inline_c true" -vcl+backend {
+varnish v1 -vcl+backend {
 	import ${vmod_std};
 
 	sub vcl_recv {
-		std.cache_req_body(1KB);
-		C{
-			const struct gethdr_s HDR_REQ_X_BodyBytes =
-			    { HDR_REQ, "\014X-BodyBytes:"};
-			VRT_SetHdr(ctx, &HDR_REQ_X_BodyBytes,
-			    VRT_INT_string(ctx, VRT_CacheReqBody(ctx, 1024)),
-			    vrt_magic_string_end);
-		}C
+		if (std.cache_req_body(1KB)) {
+			set req.http.stored = true;
+		} else {
+			set req.http.stored = false;
+		}
 		return (pass);
 	}
 	sub vcl_deliver {
 		if (resp.http.foo == "BAR") {
 			return (restart);
 		}
-		set resp.http.X-BodyBytes = req.http.X-BodyBytes;
+		set resp.http.stored = req.http.stored;
 	}
 } -start
 
@@ -36,7 +33,6 @@ varnish v1 -cliok "param.set vcc_allow_inline_c true" -vcl+backend {
 logexpect l1 -v v1 {
 	expect * 1006	Begin
 	expect * =	FetchError	"^straight insufficient bytes"
-	expect * =	ReqHeader	"^X-BodyBytes: -1"
 } -start
 
 varnish v1 -cliok "param.set debug +syncvsl"
@@ -46,7 +42,7 @@ client c1 {
 	rxresp
 	expect resp.http.Foo == "Foo"
 	expect resp.bodylen == 2
-	expect resp.http.X-BodyBytes == 3
+	expect resp.http.stored == true
 } -run
 
 delay .1
@@ -54,6 +50,7 @@ delay .1
 client c1 {
 	txreq -req POST -nolen -hdr "Content-Length: 52"
 	delay .3
+	expect resp.http.stored == <undef>
 } -run
 
 delay .1
@@ -67,12 +64,13 @@ client c1 {
 	txreq -url "/is_varnish_still_running"
 	rxresp
 	expect resp.status == 200
-	expect resp.http.X-BodyBytes == 0
+	expect resp.http.stored == true
 } -run
 
 client c2 {
        txreq -req POST -nolen -hdr "Content-Length: 1025"
        expect_close
+	expect resp.http.stored == <undef>
 } -run
 varnish v1 -stop
 logexpect l1 -wait
diff --git a/bin/varnishtest/tests/c00067.vtc b/bin/varnishtest/tests/c00067.vtc
index 744cbcd..65303af 100644
--- a/bin/varnishtest/tests/c00067.vtc
+++ b/bin/varnishtest/tests/c00067.vtc
@@ -34,7 +34,8 @@ varnish v1 -vcl+backend {
 	import ${vmod_std};
 
 	sub vcl_recv {
-		std.cache_req_body(110B);
+		if (std.cache_req_body(110B)) {
+		}
 	}
 }
 
diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc
index cc2506e..33929af 100644
--- a/lib/libvmod_std/vmod.vcc
+++ b/lib/libvmod_std/vmod.vcc
@@ -222,17 +222,19 @@ Description
 Example
 	set req.url = std.querysort(req.url);
 
-$Function VOID cache_req_body(BYTES size)
+$Function BOOL cache_req_body(BYTES size)
 
 Description
-	Cache the req.body if it is smaller than *size*.
+	Caches the request body if it is smaller than *size*.  Returns
+	`true` if the body was cached, `false` otherwise.
 
-	Caching the req.body makes it possible to retry pass
-	operations (POST, PUT).
+	Normally the request body is not available after sending it to
+	the backend.  By caching it is possible to retry pass operations,
+	e.g. POST and PUT.
 Example
-	std.cache_req_body(1KB);
-
-	This will cache the req.body if its size is smaller than 1KB.
+	| if (std.cache_req_body(1KB)) {
+	|	...
+	| }
 
 $Function STRING strstr(STRING s1, STRING s2)
 
diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
index 3afc937..1a8f56b 100644
--- a/lib/libvmod_std/vmod_std.c
+++ b/lib/libvmod_std/vmod_std.c
@@ -225,16 +225,15 @@ vmod_timestamp(VRT_CTX, VCL_STRING label)
 	}
 }
 
-VCL_VOID __match_proto__(td_std_cache_req_body)
+VCL_BOOL __match_proto__(td_std_cache_req_body)
 vmod_cache_req_body(VRT_CTX, VCL_BYTES size)
 {
-	ssize_t result;
-
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	if (size < 0)
 		size = 0;
-	result = VRT_CacheReqBody(ctx, (size_t)size);
-	VSLb(ctx->vsl, SLT_Debug, "VRT_CacheReqBody(%zd): %zd", (size_t)size, result);
+	if (VRT_CacheReqBody(ctx, (size_t)size) < 0)
+		return (0);
+	return (1);
 }
 
 VCL_STRING __match_proto__(td_std_strstr)



From nils.goroll at uplex.de  Wed Feb 17 10:23:09 2016
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 17 Feb 2016 11:23:09 +0100
Subject: [master] 356854c Fix workspace overflow handling in VFP_Push() and
 test for it
Message-ID: <E1aVzGH-0006Ja-L9@project.varnish-software.com>


commit 356854cd86b1866b492bbe64417d1ace6e19a9b3
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Feb 17 10:28:38 2016 +0100

    Fix workspace overflow handling in VFP_Push() and test for it
    
    This fix avoids a WS_Alloc panic when pushing fetch processors
    and should thus avoid panics by overflowing the backend workspace in vcl.
    
    This overflow is logged as FetchError "Bo workspace overflowed".
    
    Other panic points due to workspace_backend being set too low still exist.
    
    Sizing estimate for workspace_backend:
    
    	sizeof(struct busyobj)
    	+ 3 * HTTP_estimate(cache_param->http_max_hdr)
    	+ cache_param->vsl_buffer
    	+ cache_param->http_resp_size
    	+ fetch processor memory
    	+ space required in VCL
    
    Fixes #1739

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index c168108..c210bd3 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -537,6 +537,15 @@ vbf_fetch_body_helper(struct busyobj *bo)
 /*--------------------------------------------------------------------
  */
 
+#define vbf_vfp_push(bo, vfp, top)					\
+	if (VFP_Push((bo)->vfc, (vfp), (top)) == NULL) {		\
+		assert (WS_Overflowed((bo)->vfc->http->ws));		\
+		(void)VFP_Error((bo)->vfc, "Bo workspace overflowed");	\
+		(bo)->htc->doclose = SC_OVERLOAD;			\
+		VDI_Finish((bo)->wrk, bo);				\
+		return (F_STP_ERROR);					\
+	}
+
 static enum fetch_step
 vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 {
@@ -592,19 +601,19 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 	assert(bo->do_gzip == 0 || bo->do_gunzip == 0);
 
 	if (bo->do_gunzip || (bo->is_gzip && bo->do_esi))
-		(void)VFP_Push(bo->vfc, &vfp_gunzip, 1);
+		vbf_vfp_push(bo, &vfp_gunzip, 1);
 
 	if (bo->htc->content_length != 0) {
 		if (bo->do_esi && bo->do_gzip) {
-			(void)VFP_Push(bo->vfc, &vfp_esi_gzip, 1);
+			vbf_vfp_push(bo, &vfp_esi_gzip, 1);
 		} else if (bo->do_esi && bo->is_gzip && !bo->do_gunzip) {
-			(void)VFP_Push(bo->vfc, &vfp_esi_gzip, 1);
+			vbf_vfp_push(bo, &vfp_esi_gzip, 1);
 		} else if (bo->do_esi) {
-			(void)VFP_Push(bo->vfc, &vfp_esi, 1);
+			vbf_vfp_push(bo, &vfp_esi, 1);
 		} else if (bo->do_gzip) {
-			(void)VFP_Push(bo->vfc, &vfp_gzip, 1);
+			vbf_vfp_push(bo, &vfp_gzip, 1);
 		} else if (bo->is_gzip && !bo->do_gunzip) {
-			(void)VFP_Push(bo->vfc, &vfp_testgunzip, 1);
+			vbf_vfp_push(bo, &vfp_testgunzip, 1);
 		}
 	}
 
diff --git a/bin/varnishd/cache/cache_fetch_proc.c b/bin/varnishd/cache/cache_fetch_proc.c
index cb0e1d1..ce9d138 100644
--- a/bin/varnishd/cache/cache_fetch_proc.c
+++ b/bin/varnishd/cache/cache_fetch_proc.c
@@ -203,8 +203,11 @@ VFP_Push(struct vfp_ctx *vc, const struct vfp *vfp, int top)
 
 	CHECK_OBJ_NOTNULL(vc, VFP_CTX_MAGIC);
 	CHECK_OBJ_NOTNULL(vc->http, HTTP_MAGIC);
+
 	vfe = WS_Alloc(vc->http->ws, sizeof *vfe);
-	AN(vfe);
+	if (vfe == NULL)
+		return NULL;
+
 	INIT_OBJ(vfe, VFP_ENTRY_MAGIC);
 	vfe->vfp = vfp;
 	vfe->closed = VFP_OK;
diff --git a/bin/varnishtest/tests/r01739.vtc b/bin/varnishtest/tests/r01739.vtc
new file mode 100644
index 0000000..5b80f1e
--- /dev/null
+++ b/bin/varnishtest/tests/r01739.vtc
@@ -0,0 +1,29 @@
+varnishtest "Check workspace overflow in fetch processor"
+
+server s1 {
+	rxreq
+	txresp -bodylen 1024
+} -start
+
+
+varnish v1 \
+ -vcl+backend {
+	import ${vmod_debug};
+	sub vcl_backend_response {
+		set beresp.do_gzip = true;
+		debug.workspace_allocate(backend, debug.workspace_free(backend) - 16);
+	}
+} -start
+
+logexpect l1 -v v1 -g raw {
+	expect * 1002 FetchError {^Bo workspace overflowed}
+	expect * =    Error      {^out of workspace [(]Bo[)]}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 503
+} -run
+
+logexpect l1 -wait



From lkarsten at varnish-software.com  Wed Feb 17 10:57:08 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 17 Feb 2016 11:57:08 +0100
Subject: [4.1] 6d7edfc Correct ENUM handling in object constructors
Message-ID: <E1aVznA-000211-V5@project.varnish-software.com>


commit 6d7edfce44476a0f078476570b689d3d335b0aed
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sat Jan 23 09:09:29 2016 +0000

    Correct ENUM handling in object constructors
    
    Discussed with phk at .  Fix confirmed by geoff at .
    
    Fixes #1844.

diff --git a/lib/libvcc/vcc_action.c b/lib/libvcc/vcc_action.c
index 41eac6b..790efea 100644
--- a/lib/libvcc/vcc_action.c
+++ b/lib/libvcc/vcc_action.c
@@ -203,9 +203,14 @@ parse_new(struct vcc *tl)
 	s_obj = p;
 	p += strlen(p) + 1;
 	s_init = p;
-	while (p[0] != '\0' || p[1] != '\0')
+	/*
+	 * Check for the end marked (\0\0) followed by s(truct) to avoid
+	 * matching an ENUM half-way through and generating illegal C code.
+	 */
+	while (p[0] != '\0' || p[1] != '\0' || p[2] != 's')
 		p++;
 	p += 2;
+	AZ(strncmp(p, "struct vmod_", 12));
 	s_struct = p;
 	p += strlen(p) + 1;
 	s_fini = p + strlen(p) + 1;
diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index 74c8c12..9218b5e 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -71,7 +71,7 @@ $Function BACKEND no_backend()
 
 Fails at backend selection
 
-$Object obj(STRING)
+$Object obj(STRING, ENUM { one, two, three } number="one")
 
 Test object
 
diff --git a/lib/libvmod_debug/vmod_debug_obj.c b/lib/libvmod_debug/vmod_debug_obj.c
index 0b9c248..2511f9a 100644
--- a/lib/libvmod_debug/vmod_debug_obj.c
+++ b/lib/libvmod_debug/vmod_debug_obj.c
@@ -43,13 +43,14 @@ struct vmod_debug_obj {
 
 VCL_VOID
 vmod_obj__init(VRT_CTX, struct vmod_debug_obj **op,
-    const char *vcl_name, VCL_STRING s)
+    const char *vcl_name, VCL_STRING s, VCL_ENUM e)
 {
 	struct vmod_debug_obj *o;
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	(void)vcl_name;
 	(void)s;
+	(void)e;
 	AN(op);
 	AZ(*op);
 	ALLOC_OBJ(o, VMOD_DEBUG_OBJ_MAGIC);



From lkarsten at varnish-software.com  Wed Feb 17 10:57:09 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 17 Feb 2016 11:57:09 +0100
Subject: [4.1] 05c44d3 Make output file prefix configurable.
Message-ID: <E1aVznB-000217-0Q@project.varnish-software.com>


commit 05c44d387eec6db96eb168385be6d6a68252edfc
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Thu Feb 11 12:56:40 2016 +0100

    Make output file prefix configurable.
    
    When building multiple vmods from the same source directory,
    vmodtool would trample on other concurrent jobs' vcc_if.[ch].

diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index 0d55316..c0e3a9b 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -840,7 +840,7 @@ class SimpleTestCase(unittest.TestCase):
 
 
 #######################################################################
-def runmain(inputvcc, outputname="vcc_if"):
+def runmain(inputvcc, outputprefix="vcc_if"):
 	# Read the file in
 	lines = []
 	with open(inputvcc, "r") as fp:
@@ -904,8 +904,8 @@ def runmain(inputvcc, outputname="vcc_if"):
 	# Parsing done, now process
 	#
 
-	fc = open("%s.c" % outputname, "w")
-	fh = open("%s.h" % outputname, "w")
+	fc = open("%s.c" % outputprefix, "w")
+	fh = open("%s.h" % outputprefix, "w")
 
 	write_c_file_warning(fc)
 	write_c_file_warning(fh)
@@ -951,6 +951,8 @@ if __name__ == "__main__":
 
 	oparser.add_option('-N', '--strict', action='store_true', default=False,
 	    help="Be strict when parsing input file. (vmod.vcc)")
+	oparser.add_option('-o', metavar="prefix", default='vcc_if',
+	    help='Output file prefix. (default: "vcc_if")')
 	oparser.add_option('', '--runtests', action='store_true', default=False,
 	    dest="runtests", help=optparse.SUPPRESS_HELP)
 	(opts, args) = oparser.parse_args()
@@ -973,4 +975,4 @@ if __name__ == "__main__":
 		oparser.print_help()
 		exit(-1)
 
-	runmain(i_vcc)
+	runmain(i_vcc, outputprefix=opts.o)



From lkarsten at varnish-software.com  Wed Feb 17 10:57:09 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 17 Feb 2016 11:57:09 +0100
Subject: [4.1] 826cee3 Don't abort on unknown keywords by default
Message-ID: <E1aVznB-00021G-1o@project.varnish-software.com>


commit 826cee31753354c58b3b705f6810dce42c1654ac
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Fri Feb 12 10:48:18 2016 +0000

    Don't abort on unknown keywords by default
    
    Just warn unless --strict is used.  This allows to use the same .vcc in
    4.0 and 4.1.

diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index c0e3a9b..c018505 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -791,7 +791,15 @@ class FileSection(object):
 			o = parse_func(self, pobj=vx[1].nam)
 			vx[1].add_method(o)
 		else:
-			raise FormatError("Unknown keyword: %s" % t.str, "")
+			if opts.strict:
+				raise FormatError("Unknown keyword: %s" %
+				    t.str, "")
+			else:
+				print("WARNING: Unknown keyword: %s:" %
+				    t.str, file=sys.stderr)
+				o = None
+				while len(self.tl) > 0:
+				    self.get_token()
 
 		assert len(self.tl) == 0
 		if o is None and len(self.l) > 0:



From lkarsten at varnish-software.com  Wed Feb 17 10:57:09 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 17 Feb 2016 11:57:09 +0100
Subject: [4.1] 840aa8e WS: Fix off-by-one in WS_Reset
Message-ID: <E1aVznB-00021P-3X@project.varnish-software.com>


commit 840aa8e5ce518c2b58c3710648a65b657d1d7dcc
Author: Devon H. O'Dell <dho at fastly.com>
Date:   Thu Jan 14 15:20:22 2016 -0800

    WS: Fix off-by-one in WS_Reset
    
    WS_Assert allows the workspace to be full in its check that `ws->f <= ws->e`.
    WS_Reset does not honor this; if a snapshot was taken of a full workspace, and
    that snapshot is reverted with WS_Reset, we panic.
    
    This issue was identified by Jozef Hatala <jozef at fastly.com>

diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c
index 615c9bb..d18b5fc 100644
--- a/bin/varnishd/cache/cache_ws.c
+++ b/bin/varnishd/cache/cache_ws.c
@@ -116,7 +116,7 @@ WS_Reset(struct ws *ws, char *p)
 		ws->f = ws->s;
 	else {
 		assert(p >= ws->s);
-		assert(p < ws->e);
+		assert(p <= ws->e);
 		ws->f = p;
 	}
 	ws_ClearOverflow(ws);
diff --git a/bin/varnishtest/tests/c00074.vtc b/bin/varnishtest/tests/c00074.vtc
new file mode 100644
index 0000000..f20f71c
--- /dev/null
+++ b/bin/varnishtest/tests/c00074.vtc
@@ -0,0 +1,25 @@
+varnishtest "Test WS_Reset off-by-one when workspace is full"
+
+server s1 {
+	rxreq
+	txresp
+} -start
+
+varnish v1 -vcl+backend {
+	import ${vmod_debug};
+	import ${vmod_std};
+
+	sub vcl_recv {
+		set req.http.ws-free = debug.workspace_free(session);
+		debug.workspace_allocate(session, std.integer(req.http.ws-free, 0));
+		debug.workspace_snap(session);
+		debug.workspace_reset(session);
+	}
+} -start
+
+client c1 {
+	txreq -url /foo
+	rxresp
+	expect resp.status == 200
+} -run
+
diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index 9218b5e..8bc0dd2 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -139,6 +139,14 @@ $Function INT workspace_free(ENUM { client, backend, session, thread })
 
 Find how much unallocated space there is left in a workspace.
 
+$Function VOID workspace_snap(ENUM { client, backend, session, thread})
+
+Snapshot the named workspace. Only one snapshot may be active at a time.
+
+$Function VOID workspace_reset(ENUM { client, backend, session, thread })
+
+Reset to the previous snapshot of a workspace, taken from debug.workspace_snap.
+
 $Function VOID vcl_release_delay(DURATION)
 
 Hold a reference to the VCL when it goes cold for the given delay.
diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c
index b87e673..839d0ad 100644
--- a/lib/libvmod_debug/vmod_debug.c
+++ b/lib/libvmod_debug/vmod_debug.c
@@ -419,6 +419,31 @@ vmod_workspace_overflowed(VRT_CTX, VCL_ENUM which)
 	return (WS_Overflowed(ws));
 }
 
+static char *debug_ws_snap;
+void
+vmod_workspace_snap(VRT_CTX, VCL_ENUM which)
+{
+	struct ws *ws;
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+
+	ws = wsfind(ctx, which);
+	WS_Assert(ws);
+
+	debug_ws_snap = WS_Snapshot(ws);
+}
+
+void
+vmod_workspace_reset(VRT_CTX, VCL_ENUM which)
+{
+	struct ws *ws;
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+
+	ws = wsfind(ctx, which);
+	WS_Assert(ws);
+
+	WS_Reset(ws, debug_ws_snap);
+}
+
 void
 vmod_workspace_overflow(VRT_CTX, VCL_ENUM which)
 {



From lkarsten at varnish-software.com  Wed Feb 17 10:57:09 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 17 Feb 2016 11:57:09 +0100
Subject: [4.1] 2a0c328 Polish things found by address-sanitizer.
Message-ID: <E1aVznB-00021Y-52@project.varnish-software.com>


commit 2a0c328554e385d412d2d56dd62ed628d8185445
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 2 00:09:22 2016 +0000

    Polish things found by address-sanitizer.
    
    None of it was important or troubling.
    
    MERGE NOTE: Changes for gcc 4.9 in the spirit of 2039d8 added.

diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index 97ade30..aea1081 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -62,6 +62,8 @@ static struct vtclog	*vltop;
  */
 
 struct macro {
+	unsigned		magic;
+#define MACRO_MAGIC		0x803423e3
 	VTAILQ_ENTRY(macro)	list;
 	char			*name;
 	char			*val;
@@ -98,18 +100,16 @@ macro_def(struct vtclog *vl, const char *instance, const char *name,
 		if (!strcmp(name, m->name))
 			break;
 	if (m == NULL) {
-		m = calloc(sizeof *m, 1);
+		ALLOC_OBJ(m, MACRO_MAGIC);
 		AN(m);
 		REPLACE(m->name, name);
 		VTAILQ_INSERT_TAIL(&macro_list, m, list);
 	}
 	AN(m);
 	va_start(ap, fmt);
-	free(m->val);
-	m->val = NULL;
 	vbprintf(buf2, fmt, ap);
 	va_end(ap);
-	m->val = strdup(buf2);
+	REPLACE(m->val, buf2);
 	AN(m->val);
 	vtc_log(vl, 4, "macro def %s=%s", name, m->val);
 	AZ(pthread_mutex_unlock(&macro_mtx));
@@ -147,6 +147,7 @@ macro_get(const char *b, const char *e)
 	int l;
 	char *retval = NULL;
 
+
 	l = e - b;
 
 	if (l == 4 && !memcmp(b, "date", l)) {
@@ -158,9 +159,11 @@ macro_get(const char *b, const char *e)
 	}
 
 	AZ(pthread_mutex_lock(&macro_mtx));
-	VTAILQ_FOREACH(m, &macro_list, list)
-		if (!memcmp(b, m->name, l) && m->name[l] == '\0')
+	VTAILQ_FOREACH(m, &macro_list, list) {
+		CHECK_OBJ_NOTNULL(m, MACRO_MAGIC);
+		if (!strncmp(b, m->name, l) && m->name[l] == '\0')
 			break;
+}
 	if (m != NULL)
 		retval = strdup(m->val);
 	AZ(pthread_mutex_unlock(&macro_mtx));
@@ -207,13 +210,16 @@ macro_expand(struct vtclog *vl, const char *text)
 	return (vsb);
 }
 
-/* extmacro is a list of macro's that are defined from the
-   command line and are applied to the macro list of each test
-   instance. No locking is required as they are set before any tests
-   are started.
-*/
+/*
+ * extmacro is a list of macro's that are defined from the
+ * command line and are applied to the macro list of each test
+ * instance. No locking is required as they are set before any tests
+ *  are started.
+ */
 
 struct extmacro {
+	unsigned		magic;
+#define EXTMACRO_MAGIC		0x51019ded
 	VTAILQ_ENTRY(extmacro)	list;
 	char			*name;
 	char			*val;
@@ -233,24 +239,24 @@ extmacro_def(const char *name, const char *fmt, ...)
 		if (!strcmp(name, m->name))
 			break;
 	if (m == NULL && fmt != NULL) {
-		m = calloc(sizeof *m, 1);
+		ALLOC_OBJ(m, EXTMACRO_MAGIC);
 		AN(m);
 		REPLACE(m->name, name);
+		AN(m->name);
 		VTAILQ_INSERT_TAIL(&extmacro_list, m, list);
 	}
 	if (fmt != NULL) {
 		AN(m);
 		va_start(ap, fmt);
-		free(m->val);
 		vbprintf(buf, fmt, ap);
 		va_end(ap);
-		m->val = strdup(buf);
+		REPLACE(m->val, buf);
 		AN(m->val);
 	} else if (m != NULL) {
 		VTAILQ_REMOVE(&extmacro_list, m, list);
 		free(m->name);
 		free(m->val);
-		free(m);
+		FREE_OBJ(m);
 	}
 }
 
@@ -652,8 +658,10 @@ exec_file(const char *fn, const char *script, const char *tmpdir,
 	init_server();
 
 	/* Apply extmacro definitions */
-	VTAILQ_FOREACH(m, &extmacro_list, list)
+	VTAILQ_FOREACH(m, &extmacro_list, list) {
+		CHECK_OBJ_NOTNULL(m, EXTMACRO_MAGIC);
 		macro_def(vltop, NULL, m->name, "%s", m->val);
+	}
 
 	/*
 	 * We need an IP number which will not repond, ever, and that is a
diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index 0440542..7822ebc 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -223,10 +223,10 @@ cmd_var_resolve(struct http *hp, char *spec)
 		return(hp->bodylen);
 	if (!strcmp(spec, "resp.body"))
 		return(hp->body != NULL ? hp->body : spec);
-	if (!memcmp(spec, "req.http.", 9)) {
+	if (!strncmp(spec, "req.http.", 9)) {
 		hh = hp->req;
 		hdr = spec + 9;
-	} else if (!memcmp(spec, "resp.http.", 10)) {
+	} else if (!strncmp(spec, "resp.http.", 10)) {
 		hh = hp->resp;
 		hdr = spec + 10;
 	} else



From fgsch at lodoss.net  Wed Feb 17 12:19:37 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 17 Feb 2016 13:19:37 +0100
Subject: [4.1] 734dd88 Spelling fix
Message-ID: <E1aW14z-00040r-Uy@project.varnish-software.com>


commit 734dd882e50d7035b8c60e85eea9537c4d56d408
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 1 22:09:39 2016 +0000

    Spelling fix

diff --git a/doc/sphinx/phk/10goingon50.rst b/doc/sphinx/phk/10goingon50.rst
index 03e6c8e..6960e1f 100644
--- a/doc/sphinx/phk/10goingon50.rst
+++ b/doc/sphinx/phk/10goingon50.rst
@@ -104,7 +104,7 @@ So what's the problem and why is this draconian solution better ?
 
 Usually, when FOSS projects start, they are started by "devops",
 Varnish certainly did:  Dag-Erling ran a couple of sites
-with Varnish, as did Kristian, and obviously Anders and Aydun of
+with Varnish, as did Kristian, and obviously Anders and Audun of
 VG did as well, so finding out if you improved or broke things
 during development didn't take long.
 



From fgsch at lodoss.net  Wed Feb 17 12:19:37 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 17 Feb 2016 13:19:37 +0100
Subject: [4.1] ab637a1 Birthdays are happening
Message-ID: <E1aW14z-00040k-Tg@project.varnish-software.com>


commit ab637a19dba91dc7f833a0fd17a7a36b1ab310b0
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Feb 1 21:48:05 2016 +0000

    Birthdays are happening

diff --git a/doc/sphinx/phk/10goingon50.rst b/doc/sphinx/phk/10goingon50.rst
new file mode 100644
index 0000000..03e6c8e
--- /dev/null
+++ b/doc/sphinx/phk/10goingon50.rst
@@ -0,0 +1,281 @@
+.. _phk_10goingon50:
+
+========================
+Varnish - 10 going on 50
+========================
+
+Ten years ago, Dag-Erling and I were busy hashing out the big lines
+of Varnish.
+
+Hashing out had started on a blackboard at University of Basel
+during the `EuroBSDcon 2005 <http://2005.eurobsdcon.org/>`_ conference,
+and had continued in email and IRC ever since.
+
+At some point in February 2006 Dag-Erling laid down the foundations of
+our Subversion and source tree.
+
+The earliest fragment which have survived the conversion to Git is
+subversion commit number 9::
+
+    commit 523166ad2dd3a65e3987f13bc54f571f98453976
+    Author: Dag Erling Sm?rgrav <des at des.no>
+    Date:   Wed Feb 22 14:31:39 2006 +0000
+    
+        Additional subdivisions.
+
+We consider this the official birth-certificate of the Varnish Cache
+FOSS project, and therefore we will celebrate the 10 year birthday
+of Varnish in a couple of weeks.
+
+We're not sure exactly how and where we will celebrate this, but
+follow Twitter user `@varnishcache <https://twitter.com/varnishcache>`_
+if you want don't want to miss the partying.
+
+--------
+VCLOCC1
+--------
+
+One part of the celebration, somehow, sometime, will be the "VCL
+Obfuscated Code Contest #1" in the same spirit as the `International
+Obfuscated C Code Contest <http://www.ioccc.org/>`_.
+
+True afficionados of Obfuscated Code will also appreciate this
+amazing `Obfuscated PL/1 <http://www.multicians.org/proc-proc.html>`_.
+
+The official VCLOCC1 contest rules are simple:
+
+* VCL code must work with Varnish 4.1.1
+* As many Varnishd instances as you'd like. 
+* No inline-C allowed
+* Any VMOD you want is OK
+* You get to choose the request(s) to send to Varnishd
+* If you need backends, they must be simulated by varnishd (4.1.1) instances.
+* *We* get to publish the winning entry on the Varnish project home-page.
+
+The *only* thing which counts is the amazing/funny/brilliant
+VCL code *you* write and what it does.  VMODs and backends are just
+scaffolding which the judges will ignore.
+
+We will announce the submission deadline one month ahead of time, but
+you are more than welcome to start already now.
+
+--------
+Releases
+--------
+
+Our 10 year aniversary was a good excuse to take stock and look at
+the way we work, and changes are and will be happening.
+
+Like any respectable FOSS project, the Varnish project has never been
+accused, or guilty, of releasing on the promised date.
+
+Not even close.
+
+With 4.1 not even close to close.
+
+Having been around that block a couple of times, (*cough* FreeBSD 5.0 *cough*)
+I think I know why and I have decided to put a stop to it.
+
+Come hell or high water [#f1]_, Varnish 5.0 will be released September
+15th 2016.
+
+And the next big release, whatever we call it, will be middle of
+march 2017, and until we change our mind, you can trust a major
+release of Varnish to happen every six months.
+
+Minor releases, typically bugfixes, will be released as need arise,
+and these should just be installable with no configuration changes.
+
+Sounds wonderful, doesn't it ?  Now you can plan your upgrades.
+
+But nothing comes free:  Until we are near September, we won't be able
+to tell you what Varnish 5 contains.
+
+We have plans and ideas for what *should* be there, and we will work
+to reach those milestones, but we will not hold the release for "just this
+one more feature" if they are not ready.
+
+If it is in on september 15th, it will be in the release, if not, it wont.
+
+And since the next release is guaranteed to come six months later,
+it's not a catastrophe to miss the deadline.
+
+So what's the problem and why is this draconian solution better ?
+
+Usually, when FOSS projects start, they are started by "devops",
+Varnish certainly did:  Dag-Erling ran a couple of sites
+with Varnish, as did Kristian, and obviously Anders and Aydun of
+VG did as well, so finding out if you improved or broke things
+during development didn't take long.
+
+But as a project grows, people gravitate from "devops" to "dev",
+and suddenly we have to ask somebody else to "please test -trunk"
+and these people have their own calendars, and are not sure why
+they should test, or even if they should test, much less what they
+should be looking for while they test, because they have not been
+part of the development process.
+
+In all honesty, going from Varnish1 to Varnish4 the amount of
+real-life testing our releases have received *before* being released
+has consistently dropped [#f2]_.
+
+So we're moving the testing on the other side of the release date,
+because the people who *can* live-test Varnish prefer to have a
+release to test.
+
+We'll run all the tests we can in our development environments and
+we'll beg and cajole people with real sites into testing also, but
+we won't wait for weeks and months for it to happen, like we did
+with the 4.1 release.
+
+All this obviously changes the dynamics of the project, and it we
+find out it is a disaster, we'll change our mind.
+
+But until then:  Two major releases a year, as clock-work, mid-september
+and mid-march.
+
+----------------
+Moving to github
+----------------
+
+We're also moving the project to github.  We're trying to find out
+a good way to preserve the old Trac contents, and once we've
+figured that out, we'll pull the handle on the transition.
+
+Trac is starting to creak in the joints and in particular we're
+sick and tired of defending it against spammers.  Moving to github
+makes that Somebody Elses Problem.
+
+We also want to overhaul the project home-page and try to get
+a/the wiki working better.
+
+We'll keep you posted about all this when and as it happens.
+
+--------------------------------------------
+We were hip before it was hip to be hipsters
+--------------------------------------------
+
+Moving to github also means moving into a different culture.
+
+Githubs statistics are neat, but whenever you start to measure
+something, it becomes a parameter for optimization and competition,
+and there are people out there who compete on github statistics.
+
+In one instance the "game" is simply to submit changes, no matter
+how trivial, to as many different projects as you can manage in
+order to claim that you "contribute to a lot of FOSS projects".
+
+There is a similar culture of "trophy hunting" amongst so-called
+"security-researchers" - who has most CVE's to their name?  It
+doesn't seem to matter to them how vacuous the charge or how
+theoretical the "vulnerability" is, a CVE is a CVE to them.
+
+I don't want to play that game.
+
+If you are a contributor to Varnish, you should already have the
+nice blue T-shirt and the mug to prove it.  (Thanks Varnish-Software!)
+
+If you merely stumble over a spelling mistake, you merely
+stumbled over a spelling mistake, and we will happily 
+correct it, and put your name in the commit message.
+
+But it takes a lot more that fixing a spelling mistake to
+become recognized as "a Varnish contributor".
+
+Yeah, we're old and boring.
+
+Speaking of which...
+
+----------------------------
+Where does 50 come into it ?
+----------------------------
+
+On january 20th I celebrated my 50 year birthday, and this was a
+much more serious affair than I had anticipated:  For the first
+time in my life I have received a basket with wine and flowers on
+my birthday.
+
+I also received books and music from certain Varnish users,
+much appreciated guys!
+
+Despite numerically growing older I will insist, until the day I
+die, that I'm a man of my best age.
+
+That doesn't mean I'm not changing.
+
+To be honest, being middle-aged sucks.
+
+Your body starts creaking and you get frustrated seeing people make
+mistakes you warned them against.
+
+But growing older also absolutely rulez, because your age allows
+you to appreciate that you live in a fantastic future with a lot
+of amazing changes - even if it will take a long time before
+progress goes too far.
+
+There does seem to be increasing tendency to want the kids of your
+lawn, but I think I can control that.
+
+But if not I hereby give them permission to steal my apples and
+yell back at me, because I've seen a lot of men, in particular in
+the technical world, grow into bitter old men who preface every
+utterance with "As *I* already said *MANY* years ago...", totally
+oblivious to how different the world has become, how wrong their
+diagnosis is and how utterly useless their advice is.
+
+I don't want to end up like that.
+
+From now on my basic assumption is that I'm an old ass who is part
+of the problem, and that being part of the solution is something I
+have to work hard for, rather than the other way around.
+
+In my case, the two primary physiological symptoms of middle age is
+that after 5-6 hours my eyes tire from focusing on the monitor and
+that my mental context-switching for big contexts is slower than
+it used to be.
+
+A couple of years ago I started taking "eye-breaks" after lunch.
+Get away from the screen, preferably outside where I could rest my
+eyes on stuff further away than 40cm, then later in the day
+come back and continue hacking.
+
+Going forward, this pattern will become more pronounced.  The amount
+of hours I work will be the same, but I will be splitting the workday
+into two halves.
+
+You can expect me to be at my keyboard morning (08-12-ish EU time)
+and evening (20-24-ish EU time) but I may be doing other stuff,
+away from the keyboard and screen, during the afternoon.
+
+Starting this year I have also changed my calendar.
+
+Rather than working on various projects and for various customers
+in increments of half days, I'm lumping things together in bigger
+units of days and weeks.
+
+Anybody who knows anything about process scheduling can see that
+this will increase throughput at the cost of latency.
+
+The major latency impact is that one of the middle weeks of each
+month I will not be doing Varnish.  On the other hand, all
+the weeks I do work on Varnish will now be full weeks.
+
+And with those small adjustments, the Varnish project and I are
+ready to tackle the next ten years.
+
+Let me conclude with a big THANK YOU! to all Contributors and Users
+of Varnish, for making the first 10 years more amazing than I ever
+thought FOSS development could be.
+
+Much Appreciated!
+
+*phk*
+
+.. rubric:: Footnotes
+
+.. [#f1] I've always wondered about that expression.  Is the assumption that
+   if *both* hell *and* high water arrives at the same time they will cancel
+   out ?
+
+.. [#f2] I've seriously considered if I should start a porn-site, just to
+   test Varnish, but the WAF of that idea was well below zero.
diff --git a/doc/sphinx/phk/index.rst b/doc/sphinx/phk/index.rst
index 7dae4e4..60a3375 100644
--- a/doc/sphinx/phk/index.rst
+++ b/doc/sphinx/phk/index.rst
@@ -8,6 +8,7 @@ You may or may not want to know what Poul-Henning thinks.
 .. toctree::
 	:maxdepth: 1
 
+	10goingon50.rst
 	brinch-hansens-arrows.rst
 	ssl_again.rst
 	persistent.rst



From fgsch at lodoss.net  Wed Feb 17 12:19:38 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 17 Feb 2016 13:19:38 +0100
Subject: [4.1] 0b51e4c Random typos and grammar fixes
Message-ID: <E1aW150-000410-0U@project.varnish-software.com>


commit 0b51e4c1772d0242a17f6696e0dedb2da27d8f1b
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Feb 3 08:16:54 2016 +0000

    Random typos and grammar fixes
    
    Some submitted by: github::sublimino

diff --git a/doc/changes.rst b/doc/changes.rst
index 73e93b5..44ad299 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -589,7 +589,7 @@ Changes from 3.0.6 to 3.0.7-rc1 (2015-03-18)
 - Requests with multiple Content-Length headers will now fail.
 
 - Stop recognizing a single CR (\r) as a HTTP line separator.
-  This opened up a possible cache poisioning attack in stacked installations
+  This opened up a possible cache poisoning attack in stacked installations
   where sslterminator/varnish/backend had different CR handling.
 
 - Improved error detection on master-child process communication, leading to
diff --git a/doc/sphinx/phk/10goingon50.rst b/doc/sphinx/phk/10goingon50.rst
index 6960e1f..ffe4349 100644
--- a/doc/sphinx/phk/10goingon50.rst
+++ b/doc/sphinx/phk/10goingon50.rst
@@ -20,7 +20,7 @@ subversion commit number 9::
     commit 523166ad2dd3a65e3987f13bc54f571f98453976
     Author: Dag Erling Sm?rgrav <des at des.no>
     Date:   Wed Feb 22 14:31:39 2006 +0000
-    
+
         Additional subdivisions.
 
 We consider this the official birth-certificate of the Varnish Cache
@@ -39,13 +39,13 @@ One part of the celebration, somehow, sometime, will be the "VCL
 Obfuscated Code Contest #1" in the same spirit as the `International
 Obfuscated C Code Contest <http://www.ioccc.org/>`_.
 
-True afficionados of Obfuscated Code will also appreciate this
+True aficionados of Obfuscated Code will also appreciate this
 amazing `Obfuscated PL/1 <http://www.multicians.org/proc-proc.html>`_.
 
 The official VCLOCC1 contest rules are simple:
 
 * VCL code must work with Varnish 4.1.1
-* As many Varnishd instances as you'd like. 
+* As many Varnishd instances as you'd like.
 * No inline-C allowed
 * Any VMOD you want is OK
 * You get to choose the request(s) to send to Varnishd
@@ -63,7 +63,7 @@ you are more than welcome to start already now.
 Releases
 --------
 
-Our 10 year aniversary was a good excuse to take stock and look at
+Our 10 year anniversary was a good excuse to take stock and look at
 the way we work, and changes are and will be happening.
 
 Like any respectable FOSS project, the Varnish project has never been
@@ -80,7 +80,7 @@ Come hell or high water [#f1]_, Varnish 5.0 will be released September
 15th 2016.
 
 And the next big release, whatever we call it, will be middle of
-march 2017, and until we change our mind, you can trust a major
+March 2017, and until we change our mind, you can trust a major
 release of Varnish to happen every six months.
 
 Minor releases, typically bugfixes, will be released as need arise,
@@ -95,7 +95,7 @@ We have plans and ideas for what *should* be there, and we will work
 to reach those milestones, but we will not hold the release for "just this
 one more feature" if they are not ready.
 
-If it is in on september 15th, it will be in the release, if not, it wont.
+If it is in on September 15th, it will be in the release, if not, it wont.
 
 And since the next release is guaranteed to come six months later,
 it's not a catastrophe to miss the deadline.
@@ -131,8 +131,8 @@ with the 4.1 release.
 All this obviously changes the dynamics of the project, and it we
 find out it is a disaster, we'll change our mind.
 
-But until then:  Two major releases a year, as clock-work, mid-september
-and mid-march.
+But until then:  Two major releases a year, as clock-work, mid-September
+and mid-March.
 
 ----------------
 Moving to github
@@ -176,7 +176,7 @@ If you are a contributor to Varnish, you should already have the
 nice blue T-shirt and the mug to prove it.  (Thanks Varnish-Software!)
 
 If you merely stumble over a spelling mistake, you merely
-stumbled over a spelling mistake, and we will happily 
+stumbled over a spelling mistake, and we will happily
 correct it, and put your name in the commit message.
 
 But it takes a lot more that fixing a spelling mistake to
@@ -190,7 +190,7 @@ Speaking of which...
 Where does 50 come into it ?
 ----------------------------
 
-On january 20th I celebrated my 50 year birthday, and this was a
+On January 20th I celebrated my 50 year birthday, and this was a
 much more serious affair than I had anticipated:  For the first
 time in my life I have received a basket with wine and flowers on
 my birthday.
diff --git a/doc/sphinx/phk/dough.rst b/doc/sphinx/phk/dough.rst
index aef598b..ea38a2e 100644
--- a/doc/sphinx/phk/dough.rst
+++ b/doc/sphinx/phk/dough.rst
@@ -86,7 +86,7 @@ at least at that time, there were no way to prevent people from
 dropping money into it, no matter how much you wanted to stop them.
 
 In the end I managed to yell loud enough and only got overfunded
-a few percent, and I belive that my attempt to deflect the surplus
+a few percent, and I believe that my attempt to deflect the surplus
 to the FreeBSD Foundation gave them a little boost that year.
 
 So about PayPal:  The first thing they did was to shut my account,
@@ -184,7 +184,7 @@ the message to other communities, that Free and Open Source Software
 does not materialize out of empty space, it is written by people.
 
 People who love what we do, which is why I'm sitting here,
-way past midnight on a friday evening, writing this pamphlet.
+way past midnight on a Friday evening, writing this pamphlet.
 
 But software *is* written by people, real people with kids, cars,
 mortgages, leaky roofs, sick pets, infirm parents and all other
@@ -225,25 +225,25 @@ collect money and hire developers.  This is a relatively complex
 thing to do, and it will only be available for larger projects.
 
 The Apache Foundation "adopts" smaller projects inside their field
-of interest, and I belive that works OK, but I'm not sure if it
+of interest, and I believe that works OK, but I'm not sure if it
 can easily be transplanted to different topics.
 
 The final way is to simply throw money a the developers, the
 way the FreeBSD and Varnish communities have done with me.
 
 It is a far more flexible solution with respect to level of
-engangement, national boundaries etc. etc, but in many ways it
+engagement, national boundaries etc. etc, but in many ways it
 demands more from both sides of the deal, in particular
 with respect to paperwork, taxes and so on.
 
 Conclusion
 ==========
 
-I am obiously biased, I derive a large fraction of my relatively
+I am obviously biased, I derive a large fraction of my relatively
 modest income from community funding, for which I am the Varnish
 community deeply grateful.
 
-But biased as I may be, I belive that the Varnish community and I
+But biased as I may be, I believe that the Varnish community and I
 has shown that a tiny investment goes a long way in Free and Open
 Source Software.
 
diff --git a/doc/sphinx/phk/persistent.rst b/doc/sphinx/phk/persistent.rst
index 536e406..6587e67 100644
--- a/doc/sphinx/phk/persistent.rst
+++ b/doc/sphinx/phk/persistent.rst
@@ -52,7 +52,7 @@ this.
 And in a sense I did.
 
 Varnish has the advantage over databases and filesystems that we
-can actually loose objects without it being a catastrophy.  It would
+can actually loose objects without it being a catastrophe.  It would
 be better if we didn't, but we can simply ditch stuff which doesn't
 look consistent and we'll be safe.
 
diff --git a/doc/sphinx/phk/platforms.rst b/doc/sphinx/phk/platforms.rst
index e85f925..047196b 100644
--- a/doc/sphinx/phk/platforms.rst
+++ b/doc/sphinx/phk/platforms.rst
@@ -108,4 +108,4 @@ are right about Oracles intentions, Solaris may get demoted.
 Until next time,
 
 Poul-Henning, 2010-08-03
-Edited Nils, 2014-03-18 with Poul-Hennings concent
+Edited Nils, 2014-03-18 with Poul-Hennings consent
diff --git a/doc/sphinx/phk/ssl_again.rst b/doc/sphinx/phk/ssl_again.rst
index 31b6efe..d0f8323 100644
--- a/doc/sphinx/phk/ssl_again.rst
+++ b/doc/sphinx/phk/ssl_again.rst
@@ -11,7 +11,7 @@ revisit that issue.
 A SSL/TLS library
 ~~~~~~~~~~~~~~~~~
 
-In 2011 I critized OpenSSL's source-code as being a nightmare,
+In 2011 I criticized OpenSSL's source-code as being a nightmare,
 and as much as I Hate To Say I Told You So, I Told You So:  See also
 "HeartBleed".
 
@@ -19,15 +19,15 @@ The good news is that HeartBleed made people realize that FOSS
 maintainers also have mortgages and hungry kids.
 
 Various initiatives have been launched to make prevent critical
-infrastructure software from being maintained sunday evening between
+infrastructure software from being maintained Sunday evening between
 11 and 12PM by a sleep-deprived and overworked parent, worried about
 about being able to pay the bills come the next month.
 
 We're not there yet, but it's certainly getting better.
 
-However, implementing TLS and SSL is stil insanely complex, and
+However, implementing TLS and SSL is still insanely complex, and
 thanks to Edward Snowdens whistle-blowing, we have very good reasons
-to belive that didn't happen by accident.
+to believe that didn't happen by accident.
 
 The issue of finding a good TLS/SSL implementation is still the
 same and I still don't see one I would want my name associated with.
@@ -40,7 +40,7 @@ Handling Certificates
 ~~~~~~~~~~~~~~~~~~~~~
 
 I still don't see a way to do that.  The Varnish worker-process is not
-built to compartementalize bits at a cryptographic level and making it
+built to compartmentalize bits at a cryptographic level and making it
 do that would be a non-trivial undertaking.
 
 But there is new loop-hole here.
@@ -148,7 +148,7 @@ the privacy of every human being with an internet connection, because
 it takes a lot more skill to look into a SSL connection than a
 plaintext HTTP connection.
 
-"Sunshine is said to be the best of disinfectantants" wrote supreme
+"Sunshine is said to be the best of disinfectants" wrote supreme
 court justice Brandeis, SSL Everywhere puts all traffic in the shade.
 
 Poul-Henning, 2015-04-28
diff --git a/doc/sphinx/phk/thetoolsweworkwith.rst b/doc/sphinx/phk/thetoolsweworkwith.rst
index 4ee56db..1ab4fc5 100644
--- a/doc/sphinx/phk/thetoolsweworkwith.rst
+++ b/doc/sphinx/phk/thetoolsweworkwith.rst
@@ -148,7 +148,7 @@ Because it's not like the call is actually guaranteed to return at
 later than that, so you have to wrap the call in a loop.
 
 Whoever defined the select(2) and poll(2) systemcalls knew better
-than the POSIX and ISO-C group-think:  They specifed a maximum
+than the POSIX and ISO-C group-think:  They specified a maximum
 duration for the call, because then it doesn't matter what time
 it is, only how long time has transpired.
 
diff --git a/doc/sphinx/phk/vcl_expr.rst b/doc/sphinx/phk/vcl_expr.rst
index 6ef02dc..254384d 100644
--- a/doc/sphinx/phk/vcl_expr.rst
+++ b/doc/sphinx/phk/vcl_expr.rst
@@ -32,7 +32,7 @@ precedence operators have.
 
 The C programming language is famous for having a couple of gottchas
 in its precedence rules and given our limited and narrow type
-repetoire, blindly importing a set of precedence rules may confuse
+repertoire, blindly importing a set of precedence rules may confuse
 a lot more than it may help.
 
 Here are the precedence rules I have settled on, from highest to
diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst
index ac671ce..7dcc9a4 100644
--- a/doc/sphinx/reference/varnish-cli.rst
+++ b/doc/sphinx/reference/varnish-cli.rst
@@ -15,7 +15,7 @@ Varnish Command Line Interface
 DESCRIPTION
 ===========
 
-Varnish as a command line interface (CLI) which can control and change
+Varnish has a command line interface (CLI) which can control and change
 most of the operational parameters and the configuration of Varnish,
 without interrupting the running service.
 
diff --git a/doc/sphinx/users-guide/performance.rst b/doc/sphinx/users-guide/performance.rst
index cd00ff5..68c20a2 100644
--- a/doc/sphinx/users-guide/performance.rst
+++ b/doc/sphinx/users-guide/performance.rst
@@ -11,7 +11,7 @@ functions of Varnish that you should be aware of. The next subsection focuses
 on the how to purge content out of your cache. Purging of content is
 essential in a performance context because it allows you to extend the
 *time-to-live* (TTL) of your cached objects. Having a long TTL allows
-Varnish to keep the content in cache longer, meaning Varnish will make fewer requests to your relativly slower backend.
+Varnish to keep the content in cache longer, meaning Varnish will make fewer requests to your relatively slower backend.
 
 The final subsection deals with compression of web content. Varnish can
 gzip content when fetching it from the backend and then deliver it
diff --git a/doc/sphinx/users-guide/vcl-actions.rst b/doc/sphinx/users-guide/vcl-actions.rst
index e66250f..57b2ea8 100644
--- a/doc/sphinx/users-guide/vcl-actions.rst
+++ b/doc/sphinx/users-guide/vcl-actions.rst
@@ -14,7 +14,7 @@ The most common actions to return are these:
 
 *hash*
   When you return hash from `vcl_recv` you tell Varnish to deliver content
-  from cache even if the request othervise indicates that the request
+  from cache even if the request otherwise indicates that the request
   should be passed.
 
 *pipe*
diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst
index a1eba28..21e5873 100644
--- a/doc/sphinx/users-guide/vcl-built-in-subs.rst
+++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst
@@ -230,7 +230,7 @@ vcl_synth
 ~~~~~~~~~
 
 Called to deliver a synthetic object. A synthetic object is generated
-in VCL, not fetched from the backend. Its body may be contructed using
+in VCL, not fetched from the backend. Its body may be constructed using
 the ``synthetic()`` function.
 
 A `vcl_synth` defined object never enters the cache, contrary to a
@@ -325,7 +325,7 @@ vcl_backend_error
 This subroutine is called if we fail the backend fetch or if
 *max_retries* has been exceeded.
 
-A synthetic object is generated in VCL, whose body may be contructed
+A synthetic object is generated in VCL, whose body may be constructed
 using the ``synthetic()`` function.
 
 The `vcl_backend_error` subroutine may terminate with calling ``return()``
diff --git a/doc/sphinx/users-guide/vcl-variables.rst b/doc/sphinx/users-guide/vcl-variables.rst
index 4531537..1e0c832 100644
--- a/doc/sphinx/users-guide/vcl-variables.rst
+++ b/doc/sphinx/users-guide/vcl-variables.rst
@@ -14,7 +14,7 @@ objects can be accessed and manipulated using VCL.
  do on or with the `req` object.
 
 *bereq*
- The backend request object. Varnish contructs this before sending it to the
+ The backend request object. Varnish constructs this before sending it to the
  backend. It is based on the `req` object.
 
 .. XXX:in what way? benc



From fgsch at lodoss.net  Wed Feb 17 12:19:38 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 17 Feb 2016 13:19:38 +0100
Subject: [4.1] 38aac8d upd
Message-ID: <E1aW150-00041C-1r@project.varnish-software.com>


commit 38aac8d0acbe82ed8a119ec287098729b5512ed2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 3 16:09:00 2016 +0000

    upd

diff --git a/doc/sphinx/phk/10goingon50.rst b/doc/sphinx/phk/10goingon50.rst
index ffe4349..2199227 100644
--- a/doc/sphinx/phk/10goingon50.rst
+++ b/doc/sphinx/phk/10goingon50.rst
@@ -213,7 +213,7 @@ you to appreciate that you live in a fantastic future with a lot
 of amazing changes - even if it will take a long time before
 progress goes too far.
 
-There does seem to be increasing tendency to want the kids of your
+There does seem to be increasing tendency to want the kids off your
 lawn, but I think I can control that.
 
 But if not I hereby give them permission to steal my apples and



From fgsch at lodoss.net  Wed Feb 17 12:19:38 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 17 Feb 2016 13:19:38 +0100
Subject: [4.1] dd758ea fixed minor typo
Message-ID: <E1aW150-00041L-3B@project.varnish-software.com>


commit dd758ea9ed4ec7d095ea79f4fb388d78c90c505e
Author: Valentin Matei <matheuzzy at gmail.com>
Date:   Mon Feb 15 18:01:50 2016 +0200

    fixed minor typo

diff --git a/bin/varnishtest/tests/README b/bin/varnishtest/tests/README
index 85b309b..cf112f8 100644
--- a/bin/varnishtest/tests/README
+++ b/bin/varnishtest/tests/README
@@ -4,7 +4,7 @@ Test-scripts for varnishtest
 Naming scheme
 -------------
 
-	The intent is to be able to run all script sin lexicographic
+	The intent is to be able to run all script in lexicographic
 	order and get a sensible failure mode.
 
 	This requires more basic tests to be earlier and more complex



From nils.goroll at uplex.de  Wed Feb 17 13:21:17 2016
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 17 Feb 2016 14:21:17 +0100
Subject: [master] 9f156af VFP_Push: document the place which still need
 attention for out-of-workspace handling
Message-ID: <E1aW22f-0008Oi-Gv@project.varnish-software.com>


commit 9f156af54b90167ade015651292eff95feb0a4b0
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Feb 17 14:17:42 2016 +0100

    VFP_Push: document the place which still need attention for out-of-workspace handling
    
    We'll need to change a couple of signatures in the call paths getting here
    to propagate up the failure.

diff --git a/bin/varnishd/http1/cache_http1_vfp.c b/bin/varnishd/http1/cache_http1_vfp.c
index fe99ebd..1c11384 100644
--- a/bin/varnishd/http1/cache_http1_vfp.c
+++ b/bin/varnishd/http1/cache_http1_vfp.c
@@ -272,16 +272,19 @@ V1F_Setup_Fetch(struct vfp_ctx *vfc, struct http_conn *htc)
 	case BS_EOF:
 		assert(htc->content_length == -1);
 		vfe = VFP_Push(vfc, &v1f_eof, 0);
+		XXXAN(vfe);
 		vfe->priv2 = 0;
 		break;
 	case BS_LENGTH:
 		assert(htc->content_length > 0);
 		vfe = VFP_Push(vfc, &v1f_straight, 0);
+		XXXAN(vfe);
 		vfe->priv2 = htc->content_length;
 		break;
 	case BS_CHUNKED:
 		assert(htc->content_length == -1);
 		vfe = VFP_Push(vfc, &v1f_chunked, 0);
+		XXXAN(vfe);
 		vfe->priv2 = -1;
 		break;
 	default:



From lkarsten at varnish-software.com  Wed Feb 17 13:30:54 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 17 Feb 2016 14:30:54 +0100
Subject: [4.1] a94e8d4 Add https_scheme feature bit
Message-ID: <E1aW2By-0001vZ-4q@project.varnish-software.com>


commit a94e8d4763c0f5f2d3874c87268485553ffc7bef
Author: Guillaume Quintard <guillaume at varnish-software.com>
Date:   Tue Jan 26 16:56:40 2016 +0100

    Add https_scheme feature bit
    
    Allow to split https URI in the request line

diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c
index 0ae476e..49478da 100644
--- a/bin/varnishd/http1/cache_http1_proto.c
+++ b/bin/varnishd/http1/cache_http1_proto.c
@@ -344,7 +344,7 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp)
 {
 	uint16_t retval;
 	const char *p;
-	const char *b, *e;
+	const char *b = NULL, *e;
 
 	CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC);
 	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
@@ -363,8 +363,12 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp)
 		return (400);
 
 	/* RFC2616, section 5.2, point 1 */
-	if (!strncasecmp(hp->hd[HTTP_HDR_URL].b, "http://", 7)) {
+	if (!strncasecmp(hp->hd[HTTP_HDR_URL].b, "http://", 7))
 		b = e = hp->hd[HTTP_HDR_URL].b + 7;
+	else if (FEATURE(FEATURE_HTTPS_SCHEME) &&
+			!strncasecmp(hp->hd[HTTP_HDR_URL].b, "https://", 8))
+		b = e = hp->hd[HTTP_HDR_URL].b + 8;
+	if (b) {
 		while (*e != '/' && *e != '\0')
 			e++;
 		if (*e == '/') {
diff --git a/bin/varnishtest/tests/r01847.vtc b/bin/varnishtest/tests/r01847.vtc
new file mode 100644
index 0000000..a163fe3
--- /dev/null
+++ b/bin/varnishtest/tests/r01847.vtc
@@ -0,0 +1,45 @@
+varnishtest "Test https_scheme parameter"
+
+server s1 {
+	rxreq
+	expect req.url == /bar
+	txresp
+
+	rxreq
+	expect req.url == https://www.example.com/bar
+	txresp
+} -start
+
+varnish v1 -vcl+backend {
+
+	sub vcl_deliver {
+		set resp.http.rxhost = req.http.host;
+		set resp.http.rxurl = req.url;
+	}
+} -start
+
+client c1 {
+	txreq -url http://www.example.com/bar
+	rxresp
+	expect resp.http.rxhost == www.example.com
+	expect resp.http.rxurl == /bar
+
+	txreq -url https://www.example.com/bar
+	rxresp
+	expect resp.http.rxhost == ""
+	expect resp.http.rxurl == https://www.example.com/bar
+} -run
+
+varnish v1 -cliok "param.set feature +https_scheme"
+
+client c1 {
+	txreq -url http://www.example.com/bar
+	rxresp
+	expect resp.http.rxhost == www.example.com
+	expect resp.http.rxurl == /bar
+
+	txreq -url https://www.example.com/bar
+	rxresp
+	expect resp.http.rxhost == www.example.com
+	expect resp.http.rxurl == /bar
+} -run
diff --git a/include/tbl/feature_bits.h b/include/tbl/feature_bits.h
index 6fcf279..8d8870b 100644
--- a/include/tbl/feature_bits.h
+++ b/include/tbl/feature_bits.h
@@ -59,4 +59,9 @@ FEATURE_BIT(ESI_REMOVE_BOM,		esi_remove_bom,
     "Remove UTF-8 BOM from front of object."
     "Ignore and remove the UTF-8 BOM (0xeb 0xbb 0xbf) from front of object."
 )
+FEATURE_BIT(HTTPS_SCHEME,		https_scheme,
+	"Also split https URIs",
+	"Extract host from full URI in the request line if the scheme is "
+	"https."
+)
 /*lint -restore */



From lkarsten at varnish-software.com  Wed Feb 17 13:30:54 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 17 Feb 2016 14:30:54 +0100
Subject: [4.1] b06e54c Fix workspace overflow handling in VFP_Push() and test
 for it
Message-ID: <E1aW2By-0001vf-6D@project.varnish-software.com>


commit b06e54c6d188f3b96cbe934e38e0a3918548e871
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Feb 17 10:28:38 2016 +0100

    Fix workspace overflow handling in VFP_Push() and test for it
    
    This fix avoids a WS_Alloc panic when pushing fetch processors
    and should thus avoid panics by overflowing the backend workspace in vcl.
    
    This overflow is logged as FetchError "Bo workspace overflowed".
    
    Other panic points due to workspace_backend being set too low still exist.
    
    Sizing estimate for workspace_backend:
    
    	sizeof(struct busyobj)
    	+ 3 * HTTP_estimate(cache_param->http_max_hdr)
    	+ cache_param->vsl_buffer
    	+ cache_param->http_resp_size
    	+ fetch processor memory
    	+ space required in VCL
    
    Fixes #1739

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 3d0517c..c41fce3 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -534,6 +534,15 @@ vbf_fetch_body_helper(struct busyobj *bo)
 /*--------------------------------------------------------------------
  */
 
+#define vbf_vfp_push(bo, vfp, top)					\
+	if (VFP_Push((bo)->vfc, (vfp), (top)) == NULL) {		\
+		assert (WS_Overflowed((bo)->vfc->http->ws));		\
+		(void)VFP_Error((bo)->vfc, "Bo workspace overflowed");	\
+		(bo)->htc->doclose = SC_OVERLOAD;			\
+		VDI_Finish((bo)->wrk, bo);				\
+		return (F_STP_ERROR);					\
+	}
+
 static enum fetch_step
 vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 {
@@ -589,19 +598,19 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 	assert(bo->do_gzip == 0 || bo->do_gunzip == 0);
 
 	if (bo->do_gunzip || (bo->is_gzip && bo->do_esi))
-		(void)VFP_Push(bo->vfc, &vfp_gunzip, 1);
+		vbf_vfp_push(bo, &vfp_gunzip, 1);
 
 	if (bo->htc->content_length != 0) {
 		if (bo->do_esi && bo->do_gzip) {
-			(void)VFP_Push(bo->vfc, &vfp_esi_gzip, 1);
+			vbf_vfp_push(bo, &vfp_esi_gzip, 1);
 		} else if (bo->do_esi && bo->is_gzip && !bo->do_gunzip) {
-			(void)VFP_Push(bo->vfc, &vfp_esi_gzip, 1);
+			vbf_vfp_push(bo, &vfp_esi_gzip, 1);
 		} else if (bo->do_esi) {
-			(void)VFP_Push(bo->vfc, &vfp_esi, 1);
+			vbf_vfp_push(bo, &vfp_esi, 1);
 		} else if (bo->do_gzip) {
-			(void)VFP_Push(bo->vfc, &vfp_gzip, 1);
+			vbf_vfp_push(bo, &vfp_gzip, 1);
 		} else if (bo->is_gzip && !bo->do_gunzip) {
-			(void)VFP_Push(bo->vfc, &vfp_testgunzip, 1);
+			vbf_vfp_push(bo, &vfp_testgunzip, 1);
 		}
 	}
 
diff --git a/bin/varnishd/cache/cache_fetch_proc.c b/bin/varnishd/cache/cache_fetch_proc.c
index 5333c62..6c38122 100644
--- a/bin/varnishd/cache/cache_fetch_proc.c
+++ b/bin/varnishd/cache/cache_fetch_proc.c
@@ -196,8 +196,11 @@ VFP_Push(struct vfp_ctx *vc, const struct vfp *vfp, int top)
 
 	CHECK_OBJ_NOTNULL(vc, VFP_CTX_MAGIC);
 	CHECK_OBJ_NOTNULL(vc->http, HTTP_MAGIC);
+
 	vfe = WS_Alloc(vc->http->ws, sizeof *vfe);
-	AN(vfe);
+	if (vfe == NULL)
+		return NULL;
+
 	INIT_OBJ(vfe, VFP_ENTRY_MAGIC);
 	vfe->vfp = vfp;
 	vfe->closed = VFP_OK;
diff --git a/bin/varnishtest/tests/r01739.vtc b/bin/varnishtest/tests/r01739.vtc
new file mode 100644
index 0000000..5b80f1e
--- /dev/null
+++ b/bin/varnishtest/tests/r01739.vtc
@@ -0,0 +1,29 @@
+varnishtest "Check workspace overflow in fetch processor"
+
+server s1 {
+	rxreq
+	txresp -bodylen 1024
+} -start
+
+
+varnish v1 \
+ -vcl+backend {
+	import ${vmod_debug};
+	sub vcl_backend_response {
+		set beresp.do_gzip = true;
+		debug.workspace_allocate(backend, debug.workspace_free(backend) - 16);
+	}
+} -start
+
+logexpect l1 -v v1 -g raw {
+	expect * 1002 FetchError {^Bo workspace overflowed}
+	expect * =    Error      {^out of workspace [(]Bo[)]}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 503
+} -run
+
+logexpect l1 -wait



From lkarsten at varnish-software.com  Wed Feb 17 13:30:54 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 17 Feb 2016 14:30:54 +0100
Subject: [4.1] afd2df2 Prepare for 4.1.2-beta1.
Message-ID: <E1aW2By-0001vo-7I@project.varnish-software.com>


commit afd2df2e4d860a185219a8047805abfe2455dc6f
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Wed Feb 17 14:00:27 2016 +0100

    Prepare for 4.1.2-beta1.

diff --git a/configure.ac b/configure.ac
index dcba307..30f6a33 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2,7 +2,7 @@ AC_PREREQ(2.59)
 AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS
 Copyright (c) 2006-2016 Varnish Software AS])
 AC_REVISION([$Id$])
-AC_INIT([Varnish], [4.1.2-dev], [varnish-dev at varnish-cache.org])
+AC_INIT([Varnish], [4.1.2-beta1], [varnish-dev at varnish-cache.org])
 AC_CONFIG_SRCDIR(include/miniobj.h)
 AC_CONFIG_HEADERS([config.h])
 AC_CONFIG_MACRO_DIR([m4])



From lkarsten at varnish-software.com  Wed Feb 17 13:30:54 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 17 Feb 2016 14:30:54 +0100
Subject: [4.1] 1f6e7cd Add changes for 4.1.2-beta1.
Message-ID: <E1aW2By-0001vx-8f@project.varnish-software.com>


commit 1f6e7cd467ccbfb2fb9e4817e92cdbf9462e274c
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Wed Feb 17 14:00:34 2016 +0100

    Add changes for 4.1.2-beta1.

diff --git a/doc/changes.rst b/doc/changes.rst
index 44ad299..7c8b511 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -1,16 +1,41 @@
 
-================================
-Varnish Cache 4.1.2 (unreleased)
-================================
+======================================
+Varnish Cache 4.1.2-beta1 (2016-02-17)
+======================================
 
 * Be stricter when parsing a HTTP request to avoid potential
   HTTP smuggling attacks against vulnerable backends.
 
+* Some fixes to minor/trivial issues found with clang AddressSanitizer.
+
+* Arithmetric on REAL data type in VCL is now possible.
+
+* vmodtool.py improvements to allow vmods for 4.0 and 4.1 to share a source tree.
+
+* Off-by-one in WS_Reset() fixed.
+
+* "https_scheme" parameter added. Enables graceful handling of compound
+  request URLs with HTTPS scheme.
+
 Bugs fixed
 ----------
 
+* 1739_ - Workspace overflow handling in VFP_Push()
+* 1837_ - Error compiling VCL if probe is referenced before it is defined
+* 1841_ - Replace alien FD's with /dev/null rather than just closing them
+* 1843_ - Fail HTTP/1.0 POST and PUT requests without Content-Length
+* 1844_ - Correct ENUM handling in object constructors
+* 1851_ - Varnish 4.1.1 fails to build on i386
+* 1852_ - Add a missing VDP flush operation after ESI:includes.
 * 1857_ - Fix timeout calculation for session herding.
 
+.. _1739: https://www.varnish-cache.org/trac/ticket/1739
+.. _1837: https://www.varnish-cache.org/trac/ticket/1837
+.. _1841: https://www.varnish-cache.org/trac/ticket/1841
+.. _1843: https://www.varnish-cache.org/trac/ticket/1843
+.. _1844: https://www.varnish-cache.org/trac/ticket/1844
+.. _1851: https://www.varnish-cache.org/trac/ticket/1851
+.. _1852: https://www.varnish-cache.org/trac/ticket/1852
 .. _1857: https://www.varnish-cache.org/trac/ticket/1857
 
 



From nils.goroll at uplex.de  Wed Feb 17 13:34:05 2016
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 17 Feb 2016 14:34:05 +0100
Subject: [master] 426183e improve FetchError message
Message-ID: <E1aW2F3-0004hm-Pt@project.varnish-software.com>


commit 426183e3003db5819a891ae0324e5dc2d5f015ed
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Feb 17 14:34:01 2016 +0100

    improve FetchError message

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index c210bd3..2f44dcd 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -540,7 +540,7 @@ vbf_fetch_body_helper(struct busyobj *bo)
 #define vbf_vfp_push(bo, vfp, top)					\
 	if (VFP_Push((bo)->vfc, (vfp), (top)) == NULL) {		\
 		assert (WS_Overflowed((bo)->vfc->http->ws));		\
-		(void)VFP_Error((bo)->vfc, "Bo workspace overflowed");	\
+		(void)VFP_Error((bo)->vfc, "workspace_backend overflow"); \
 		(bo)->htc->doclose = SC_OVERLOAD;			\
 		VDI_Finish((bo)->wrk, bo);				\
 		return (F_STP_ERROR);					\
diff --git a/bin/varnishtest/tests/r01739.vtc b/bin/varnishtest/tests/r01739.vtc
index 5b80f1e..21398cc 100644
--- a/bin/varnishtest/tests/r01739.vtc
+++ b/bin/varnishtest/tests/r01739.vtc
@@ -16,7 +16,7 @@ varnish v1 \
 } -start
 
 logexpect l1 -v v1 -g raw {
-	expect * 1002 FetchError {^Bo workspace overflowed}
+	expect * 1002 FetchError {^workspace_backend overflow}
 	expect * =    Error      {^out of workspace [(]Bo[)]}
 } -start
 



From dridi.boukelmoune at gmail.com  Wed Feb 17 15:50:19 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 17 Feb 2016 16:50:19 +0100
Subject: [master] f3b3d59 Polish
Message-ID: <E1aW4Mt-0000EC-Vz@project.varnish-software.com>


commit f3b3d592608e891e2f65bfab1fb3baf4a5510390
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Feb 17 16:50:03 2016 +0100

    Polish

diff --git a/bin/varnishd/cache/cache_fetch_proc.c b/bin/varnishd/cache/cache_fetch_proc.c
index ce9d138..9840b3d 100644
--- a/bin/varnishd/cache/cache_fetch_proc.c
+++ b/bin/varnishd/cache/cache_fetch_proc.c
@@ -206,7 +206,7 @@ VFP_Push(struct vfp_ctx *vc, const struct vfp *vfp, int top)
 
 	vfe = WS_Alloc(vc->http->ws, sizeof *vfe);
 	if (vfe == NULL)
-		return NULL;
+		return (NULL);
 
 	INIT_OBJ(vfe, VFP_ENTRY_MAGIC);
 	vfe->vfp = vfp;



From guillaume at varnish-software.com  Wed Feb 17 17:42:49 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 17 Feb 2016 18:42:49 +0100
Subject: [master] 75ceb18 build: support for asan/tsan/msan/ubsan sanitizers
Message-ID: <E1aW67l-0000yq-N4@project.varnish-software.com>


commit 75ceb1893529080014bf45d54562f9f5f249f636
Author: Devon H. O'Dell <devon.odell at gmail.com>
Date:   Thu Feb 4 23:39:26 2016 +0000

    build: support for asan/tsan/msan/ubsan sanitizers
    
    Modern versions of gcc and clang support various runtime sanitizers that
    can be used to find nasty bugs at runtime. Note that not all of these
    sanitizers may be active at the same time. However, these tools are aimed
    at developers and likely require at least some level of reference during
    testing while enabled -- which discouraged me from doing a matrix of
    conditionals in autohell.

diff --git a/bin/varnishadm/Makefile.am b/bin/varnishadm/Makefile.am
index 9033e08..f0bd8ad 100644
--- a/bin/varnishadm/Makefile.am
+++ b/bin/varnishadm/Makefile.am
@@ -13,9 +13,17 @@ varnishadm_SOURCES = \
 	$(top_srcdir)/lib/libvarnish/vtcp.c \
 	$(top_srcdir)/lib/libvarnish/vss.c
 
-varnishadm_CFLAGS = @LIBEDIT_CFLAGS@
+varnishadm_CFLAGS = @LIBEDIT_CFLAGS@ \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
 
 varnishadm_LDADD = \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
-	${PTHREAD_LIBS} ${RT_LIBS} ${NET_LIBS} @LIBEDIT_LIBS@ ${LIBM}
+	${PTHREAD_LIBS} ${RT_LIBS} ${NET_LIBS} @LIBEDIT_LIBS@ ${LIBM} \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@
diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 0eb9251..2fb09b8 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -133,8 +133,12 @@ nobase_pkginclude_HEADERS = \
 
 varnishd_CFLAGS = \
 	@PCRE_CFLAGS@ \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@ \
 	-DVARNISHD_IS_NOT_A_VMOD \
-        -DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' \
+	-DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' \
 	-DVARNISH_VMOD_DIR='"${pkglibdir}/vmods"' \
 	-DVARNISH_VCL_DIR='"${varnishconfdir}"'
 
@@ -145,6 +149,10 @@ varnishd_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvcc/libvcc.la \
 	$(top_builddir)/lib/libvgz/libvgz.la \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@ \
 	@JEMALLOC_LDADD@ \
 	@PCRE_LIBS@ \
 	${DL_LIBS} ${PTHREAD_LIBS} ${NET_LIBS} ${LIBM} ${LIBUMEM}
diff --git a/bin/varnishhist/Makefile.am b/bin/varnishhist/Makefile.am
index 5904430..b1e8e7b 100644
--- a/bin/varnishhist/Makefile.am
+++ b/bin/varnishhist/Makefile.am
@@ -16,10 +16,20 @@ varnishhist_SOURCES = varnishhist.c \
 	$(top_srcdir)/lib/libvarnish/flopen.c \
 	$(top_srcdir)/lib/libvarnishtools/vut.c
 
+varnishhist_CFLAGS = \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+
 varnishhist_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
 	-lm \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@ \
 	@CURSES_LIB@ ${RT_LIBS} ${PTHREAD_LIBS}
 
 noinst_PROGRAMS = varnishhist_opt2rst
diff --git a/bin/varnishlog/Makefile.am b/bin/varnishlog/Makefile.am
index 2df9415..1416889 100644
--- a/bin/varnishlog/Makefile.am
+++ b/bin/varnishlog/Makefile.am
@@ -18,9 +18,19 @@ varnishlog_SOURCES = \
 	$(top_srcdir)/lib/libvarnish/vpf.c \
 	$(top_srcdir)/lib/libvarnish/vtim.c
 
+varnishlog_CFLAGS = \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+
 varnishlog_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@ \
 	${RT_LIBS} ${LIBM} ${PTHREAD_LIBS}
 
 noinst_PROGRAMS = varnishlog_opt2rst
diff --git a/bin/varnishncsa/Makefile.am b/bin/varnishncsa/Makefile.am
index d63df61..b5d1bff 100644
--- a/bin/varnishncsa/Makefile.am
+++ b/bin/varnishncsa/Makefile.am
@@ -20,9 +20,19 @@ varnishncsa_SOURCES = \
 	$(top_srcdir)/lib/libvarnish/vtim.c \
 	$(top_srcdir)/lib/libvarnish/vsb.c
 
+varnishncsa_CFLAGS = \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+
 varnishncsa_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@ \
 	${RT_LIBS} ${LIBM}
 
 noinst_PROGRAMS = varnishncsa_opt2rst
diff --git a/bin/varnishstat/Makefile.am b/bin/varnishstat/Makefile.am
index a276444..50f8ad0 100644
--- a/bin/varnishstat/Makefile.am
+++ b/bin/varnishstat/Makefile.am
@@ -15,9 +15,19 @@ varnishstat_SOURCES = \
 	$(top_srcdir)/lib/libvarnish/version.c \
 	$(top_srcdir)/lib/libvarnish/vtim.c
 
+varnishstat_CFLAGS = \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+
 varnishstat_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@ \
 	@CURSES_LIB@ ${RT_LIBS} ${LIBM} ${PTHREAD_LIBS}
 
 noinst_PROGRAMS = vsc2rst
diff --git a/bin/varnishtest/Makefile.am b/bin/varnishtest/Makefile.am
index 2506a8a..c333968 100644
--- a/bin/varnishtest/Makefile.am
+++ b/bin/varnishtest/Makefile.am
@@ -42,9 +42,17 @@ varnishtest_LDADD = \
 		$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 		$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
 		$(top_builddir)/lib/libvgz/libvgz.la \
+		@ASAN_LDFLAGS@ \
+		@MSAN_LDFLAGS@ \
+		@TSAN_LDFLAGS@ \
+		@UBSAN_LDFLAGS@ \
 		${LIBM} ${PTHREAD_LIBS}
 
 varnishtest_CFLAGS = \
+		@ASAN_CFLAGS@ \
+		@MSAN_CFLAGS@ \
+		@TSAN_CFLAGS@ \
+		@UBSAN_CFLAGS@ \
 		-DTOP_BUILDDIR='"${top_builddir}"'
 
 EXTRA_DIST = $(top_srcdir)/bin/varnishtest/tests/*.vtc \
diff --git a/bin/varnishtop/Makefile.am b/bin/varnishtop/Makefile.am
index f48e972..37919e6 100644
--- a/bin/varnishtop/Makefile.am
+++ b/bin/varnishtop/Makefile.am
@@ -18,9 +18,19 @@ varnishtop_SOURCES = varnishtop.c \
 	$(top_srcdir)/lib/libvarnish/vsb.c
 
 
+varnishtop_CFLAGS = \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+
 varnishtop_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@ \
 	@CURSES_LIB@ ${RT_LIBS} ${LIBM} ${PTHREAD_LIBS}
 
 noinst_PROGRAMS = varnishtop_opt2rst
diff --git a/configure.ac b/configure.ac
index da9f7ca..f211ac7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -279,6 +279,46 @@ if test "$ac_cv_have_viz" = no; then
 fi
 CFLAGS="${save_CFLAGS}"
 
+UBSAN_CFLAGS=
+UBSAN_LDFLAGS=
+AC_ARG_ENABLE(ubsan,
+	AS_HELP_STRING([--enable-ubsan],[enable undefined behavior sanitizer (default is NO)]),
+	CFLAGS="${CFLAGS} -fPIC"
+	UBSAN_CFLAGS="-fsanitize=undefined -fPIE -fno-omit-frame-pointer"
+	UBSAN_LDFLAGS="-fsanitize=undefined -pie")
+AC_SUBST(UBSAN_CFLAGS)
+AC_SUBST(UBSAN_LDFLAGS)
+
+TSAN_CFLAGS=
+TSAN_LDFLAGS=
+AC_ARG_ENABLE(tsan,
+	AS_HELP_STRING([--enable-tsan],[enable thread sanitizer (default is NO)]),
+	CFLAGS="${CFLAGS} -fPIC"
+	TSAN_CFLAGS="-fsanitize=thread -fPIE -fno-omit-frame-pointer"
+	TSAN_LDFLAGS="-fsanitize=thread -pie")
+AC_SUBST(TSAN_CFLAGS)
+AC_SUBST(TSAN_LDFLAGS)
+
+ASAN_CFLAGS=
+ASAN_LDFLAGS=
+AC_ARG_ENABLE(asan,
+	AS_HELP_STRING([--enable-asan],[enable address sanitizer (default is NO)]),
+	CFLAGS="${CFLAGS} -fPIC"
+	ASAN_CFLAGS="-fsanitize=address -fPIE -fno-omit-frame-pointer"
+	ASAN_LDFLAGS="-fsanitize=address -pie")
+AC_SUBST(ASAN_CFLAGS)
+AC_SUBST(ASAN_LDFLAGS)
+
+MSAN_CFLAGS=
+MSAN_LDFLAGS=
+AC_ARG_ENABLE(msan,
+	AS_HELP_STRING([--enable-msan],[enable memory sanitizer (default is NO)]),
+	CFLAGS="${CFLAGS} -fPIC"
+	MSAN_CFLAGS="-fsanitize=memory -fPIE -fno-omit-frame-pointer"
+	MSAN_LDFLAGS="-fsanitize=memory -pie")
+AC_SUBST(MSAN_CFLAGS)
+AC_SUBST(MSAN_LDFLAGS)
+
 # Use jemalloc on Linux
 JEMALLOC_LDADD=
 AC_ARG_WITH([jemalloc],
diff --git a/lib/libvarnish/Makefile.am b/lib/libvarnish/Makefile.am
index b592cff..b8ed85e 100644
--- a/lib/libvarnish/Makefile.am
+++ b/lib/libvarnish/Makefile.am
@@ -36,8 +36,17 @@ libvarnish_la_SOURCES = \
 	vsha256.c \
 	vss.c
 
-libvarnish_la_CFLAGS = -DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"'
-libvarnish_la_LIBADD = ${RT_LIBS} ${NET_LIBS} ${LIBM} @PCRE_LIBS@
+libvarnish_la_CFLAGS = \
+	-DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+libvarnish_la_LIBADD = ${RT_LIBS} ${NET_LIBS} ${LIBM} @PCRE_LIBS@ \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@
 
 if ENABLE_TESTS
 TESTS = vnum_c_test
diff --git a/lib/libvarnishapi/Makefile.am b/lib/libvarnishapi/Makefile.am
index c7fb0e4..dbaece1 100644
--- a/lib/libvarnishapi/Makefile.am
+++ b/lib/libvarnishapi/Makefile.am
@@ -44,9 +44,17 @@ libvarnishapi_la_SOURCES = \
 	libvarnishapi.map
 
 libvarnishapi_la_CFLAGS = \
-	-DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"'
-
-libvarnishapi_la_LIBADD = @PCRE_LIBS@ @RT_LIBS@
+	-DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+
+libvarnishapi_la_LIBADD = @PCRE_LIBS@ @RT_LIBS@ \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@
 
 if HAVE_LD_VERSION_SCRIPT
 libvarnishapi_la_LDFLAGS += -Wl,--version-script=$(srcdir)/libvarnishapi.map
diff --git a/lib/libvarnishcompat/Makefile.am b/lib/libvarnishcompat/Makefile.am
index 78683c0..2b4c469 100644
--- a/lib/libvarnishcompat/Makefile.am
+++ b/lib/libvarnishcompat/Makefile.am
@@ -8,7 +8,17 @@ AM_LDFLAGS  = $(AM_LT_LDFLAGS)
 
 pkglib_LTLIBRARIES = libvarnishcompat.la
 
-libvarnishcompat_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version
+libvarnishcompat_la_CFLAGS = \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+
+libvarnishcompat_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@
 
 libvarnishcompat_la_SOURCES = \
 	daemon.c \
diff --git a/lib/libvcc/Makefile.am b/lib/libvcc/Makefile.am
index b5a6574..6f55208 100644
--- a/lib/libvcc/Makefile.am
+++ b/lib/libvcc/Makefile.am
@@ -8,7 +8,17 @@ AM_CPPFLAGS = \
 
 pkglib_LTLIBRARIES = libvcc.la
 
-libvcc_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version
+libvcc_la_CFLAGS = \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+
+libvcc_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@
 
 libvcc_la_SOURCES = \
 	vcc_compile.h \
diff --git a/lib/libvgz/Makefile.am b/lib/libvgz/Makefile.am
index d9c0dae..7bbc9fd 100644
--- a/lib/libvgz/Makefile.am
+++ b/lib/libvgz/Makefile.am
@@ -3,8 +3,16 @@ AM_LDFLAGS  = $(AM_LT_LDFLAGS)
 
 pkglib_LTLIBRARIES = libvgz.la
 
-libvgz_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version
-libvgz_la_CFLAGS = -D_LARGEFILE64_SOURCE=1 -DZLIB_CONST $(libvgz_extra_cflags)
+libvgz_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@
+libvgz_la_CFLAGS = -D_LARGEFILE64_SOURCE=1 -DZLIB_CONST $(libvgz_extra_cflags) \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
 
 libvgz_la_SOURCES = \
 	adler32.c \
diff --git a/lib/libvmod_debug/Makefile.am b/lib/libvmod_debug/Makefile.am
index 9115713..e64a95a 100644
--- a/lib/libvmod_debug/Makefile.am
+++ b/lib/libvmod_debug/Makefile.am
@@ -13,7 +13,17 @@ vmodtoolargs = --strict
 
 noinst_LTLIBRARIES = libvmod_debug.la
 
-libvmod_debug_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared -rpath /nowhere
+libvmod_debug_la_CFLAGS = \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+
+libvmod_debug_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared -rpath /nowhere \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@
 
 libvmod_debug_la_SOURCES = \
 	vmod_debug.c \
diff --git a/lib/libvmod_directors/Makefile.am b/lib/libvmod_directors/Makefile.am
index 64b72a8..d58b108 100644
--- a/lib/libvmod_directors/Makefile.am
+++ b/lib/libvmod_directors/Makefile.am
@@ -12,7 +12,17 @@ vmodtool = $(top_srcdir)/lib/libvcc/vmodtool.py
 vmodtoolargs = --strict
 vmod_LTLIBRARIES = libvmod_directors.la
 
-libvmod_directors_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared
+libvmod_directors_la_CFLAGS = \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+
+libvmod_directors_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@
 
 libvmod_directors_la_SOURCES = \
 	vdir.c \
diff --git a/lib/libvmod_std/Makefile.am b/lib/libvmod_std/Makefile.am
index 47dd5dc..f895478 100644
--- a/lib/libvmod_std/Makefile.am
+++ b/lib/libvmod_std/Makefile.am
@@ -13,7 +13,17 @@ vmodtool = $(top_srcdir)/lib/libvcc/vmodtool.py
 vmodtoolargs = --strict
 vmod_LTLIBRARIES = libvmod_std.la
 
-libvmod_std_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared
+libvmod_std_la_CFLAGS = \
+	@ASAN_CFLAGS@ \
+	@MSAN_CFLAGS@ \
+	@TSAN_CFLAGS@ \
+	@UBSAN_CFLAGS@
+
+libvmod_std_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared \
+	@ASAN_LDFLAGS@ \
+	@MSAN_LDFLAGS@ \
+	@TSAN_LDFLAGS@ \
+	@UBSAN_LDFLAGS@
 
 libvmod_std_la_SOURCES = \
 	vmod_std.c \



From guillaume at varnish-software.com  Wed Feb 17 17:42:49 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 17 Feb 2016 18:42:49 +0100
Subject: [master] a37579b Use variables to declare deps on objects
Message-ID: <E1aW67l-0000yx-OZ@project.varnish-software.com>


commit a37579b2813791062dc3f8f12f4bc8032b604b6d
Author: Guillaume Quintard <guillaume at varnish-software.com>
Date:   Fri Feb 12 13:46:09 2016 +0100

    Use variables to declare deps on objects

diff --git a/lib/libvmod_debug/Makefile.am b/lib/libvmod_debug/Makefile.am
index e64a95a..653e2b4 100644
--- a/lib/libvmod_debug/Makefile.am
+++ b/lib/libvmod_debug/Makefile.am
@@ -35,7 +35,7 @@ nodist_libvmod_debug_la_SOURCES = \
 	vcc_if.h
 
 # BUILT_SOURCES is only a hack and dependency tracking does not help for the first build
-vmod_debug.lo vmod_debug_obj.lo vmod_debug_dyn.lo: vcc_if.h
+$(libvmod_debug_la_OBJECTS):vcc_if.h
 
 vcc_if.c vcc_if.h vmod_debug.rst vmod_debug.man.rst: $(vmodtool) $(vmod_srcdir)/vmod.vcc
 	@PYTHON@ $(vmodtool) $(vmodtoolargs) $(vmod_srcdir)/vmod.vcc
diff --git a/lib/libvmod_directors/Makefile.am b/lib/libvmod_directors/Makefile.am
index d58b108..4d0f8e1 100644
--- a/lib/libvmod_directors/Makefile.am
+++ b/lib/libvmod_directors/Makefile.am
@@ -37,7 +37,7 @@ nodist_libvmod_directors_la_SOURCES = \
 	vcc_if.h
 
 # BUILT_SOURCES is only a hack and dependency tracking does not help for the first build
-vdir.lo fall_back.lo hash.lo random.lo round_robin.lo: vcc_if.h
+$(libvmod_directors_la_OBJECTS):vcc_if.h
 
 vcc_if.c vcc_if.h vmod_directors.rst vmod_directors.man.rst: $(vmodtool) $(vmod_srcdir)/vmod.vcc
 	@PYTHON@ $(vmodtool) $(vmodtoolargs) $(vmod_srcdir)/vmod.vcc



From guillaume at varnish-software.com  Wed Feb 17 17:42:49 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 17 Feb 2016 18:42:49 +0100
Subject: [master] 26f2e6f Refactor SAN_FLAGS a bit
Message-ID: <E1aW67l-0000z6-Q6@project.varnish-software.com>


commit 26f2e6f34cd975cbb910386142e041351e2f7a5a
Author: Guillaume Quintard <guillaume at varnish-software.com>
Date:   Mon Feb 15 11:04:30 2016 +0100

    Refactor SAN_FLAGS a bit

diff --git a/bin/varnishadm/Makefile.am b/bin/varnishadm/Makefile.am
index f0bd8ad..d218bc9 100644
--- a/bin/varnishadm/Makefile.am
+++ b/bin/varnishadm/Makefile.am
@@ -14,16 +14,10 @@ varnishadm_SOURCES = \
 	$(top_srcdir)/lib/libvarnish/vss.c
 
 varnishadm_CFLAGS = @LIBEDIT_CFLAGS@ \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 varnishadm_LDADD = \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	${PTHREAD_LIBS} ${RT_LIBS} ${NET_LIBS} @LIBEDIT_LIBS@ ${LIBM} \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@
+	@SAN_LDFLAGS@
diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 2fb09b8..792e7fa 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -133,10 +133,7 @@ nobase_pkginclude_HEADERS = \
 
 varnishd_CFLAGS = \
 	@PCRE_CFLAGS@ \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@ \
+	@SAN_CFLAGS@ \
 	-DVARNISHD_IS_NOT_A_VMOD \
 	-DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' \
 	-DVARNISH_VMOD_DIR='"${pkglibdir}/vmods"' \
@@ -149,10 +146,7 @@ varnishd_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvcc/libvcc.la \
 	$(top_builddir)/lib/libvgz/libvgz.la \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@ \
+	@SAN_LDFLAGS@ \
 	@JEMALLOC_LDADD@ \
 	@PCRE_LIBS@ \
 	${DL_LIBS} ${PTHREAD_LIBS} ${NET_LIBS} ${LIBM} ${LIBUMEM}
diff --git a/bin/varnishhist/Makefile.am b/bin/varnishhist/Makefile.am
index b1e8e7b..21d4285 100644
--- a/bin/varnishhist/Makefile.am
+++ b/bin/varnishhist/Makefile.am
@@ -17,19 +17,13 @@ varnishhist_SOURCES = varnishhist.c \
 	$(top_srcdir)/lib/libvarnishtools/vut.c
 
 varnishhist_CFLAGS = \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 varnishhist_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
 	-lm \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@ \
+	@SAN_LDFLAGS@ \
 	@CURSES_LIB@ ${RT_LIBS} ${PTHREAD_LIBS}
 
 noinst_PROGRAMS = varnishhist_opt2rst
diff --git a/bin/varnishlog/Makefile.am b/bin/varnishlog/Makefile.am
index 1416889..9747e49 100644
--- a/bin/varnishlog/Makefile.am
+++ b/bin/varnishlog/Makefile.am
@@ -19,18 +19,12 @@ varnishlog_SOURCES = \
 	$(top_srcdir)/lib/libvarnish/vtim.c
 
 varnishlog_CFLAGS = \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 varnishlog_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@ \
+	@SAN_LDFLAGS@ \
 	${RT_LIBS} ${LIBM} ${PTHREAD_LIBS}
 
 noinst_PROGRAMS = varnishlog_opt2rst
diff --git a/bin/varnishncsa/Makefile.am b/bin/varnishncsa/Makefile.am
index b5d1bff..b51df5f 100644
--- a/bin/varnishncsa/Makefile.am
+++ b/bin/varnishncsa/Makefile.am
@@ -21,18 +21,12 @@ varnishncsa_SOURCES = \
 	$(top_srcdir)/lib/libvarnish/vsb.c
 
 varnishncsa_CFLAGS = \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 varnishncsa_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@ \
+	@SAN_LDFLAGS@ \
 	${RT_LIBS} ${LIBM}
 
 noinst_PROGRAMS = varnishncsa_opt2rst
diff --git a/bin/varnishstat/Makefile.am b/bin/varnishstat/Makefile.am
index 50f8ad0..52fb5d8 100644
--- a/bin/varnishstat/Makefile.am
+++ b/bin/varnishstat/Makefile.am
@@ -16,18 +16,12 @@ varnishstat_SOURCES = \
 	$(top_srcdir)/lib/libvarnish/vtim.c
 
 varnishstat_CFLAGS = \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 varnishstat_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@ \
+	@SAN_LDFLAGS@ \
 	@CURSES_LIB@ ${RT_LIBS} ${LIBM} ${PTHREAD_LIBS}
 
 noinst_PROGRAMS = vsc2rst
diff --git a/bin/varnishtest/Makefile.am b/bin/varnishtest/Makefile.am
index c333968..924f21c 100644
--- a/bin/varnishtest/Makefile.am
+++ b/bin/varnishtest/Makefile.am
@@ -42,17 +42,11 @@ varnishtest_LDADD = \
 		$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 		$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
 		$(top_builddir)/lib/libvgz/libvgz.la \
-		@ASAN_LDFLAGS@ \
-		@MSAN_LDFLAGS@ \
-		@TSAN_LDFLAGS@ \
-		@UBSAN_LDFLAGS@ \
+		@SAN_LDFLAGS@ \
 		${LIBM} ${PTHREAD_LIBS}
 
 varnishtest_CFLAGS = \
-		@ASAN_CFLAGS@ \
-		@MSAN_CFLAGS@ \
-		@TSAN_CFLAGS@ \
-		@UBSAN_CFLAGS@ \
+		@SAN_CFLAGS@ \
 		-DTOP_BUILDDIR='"${top_builddir}"'
 
 EXTRA_DIST = $(top_srcdir)/bin/varnishtest/tests/*.vtc \
diff --git a/bin/varnishtop/Makefile.am b/bin/varnishtop/Makefile.am
index 37919e6..f176728 100644
--- a/bin/varnishtop/Makefile.am
+++ b/bin/varnishtop/Makefile.am
@@ -19,18 +19,12 @@ varnishtop_SOURCES = varnishtop.c \
 
 
 varnishtop_CFLAGS = \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 varnishtop_LDADD = \
 	$(top_builddir)/lib/libvarnishcompat/libvarnishcompat.la \
 	$(top_builddir)/lib/libvarnishapi/libvarnishapi.la \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@ \
+	@SAN_LDFLAGS@ \
 	@CURSES_LIB@ ${RT_LIBS} ${LIBM} ${PTHREAD_LIBS}
 
 noinst_PROGRAMS = varnishtop_opt2rst
diff --git a/configure.ac b/configure.ac
index f211ac7..edec1ec 100644
--- a/configure.ac
+++ b/configure.ac
@@ -283,41 +283,30 @@ UBSAN_CFLAGS=
 UBSAN_LDFLAGS=
 AC_ARG_ENABLE(ubsan,
 	AS_HELP_STRING([--enable-ubsan],[enable undefined behavior sanitizer (default is NO)]),
-	CFLAGS="${CFLAGS} -fPIC"
-	UBSAN_CFLAGS="-fsanitize=undefined -fPIE -fno-omit-frame-pointer"
-	UBSAN_LDFLAGS="-fsanitize=undefined -pie")
-AC_SUBST(UBSAN_CFLAGS)
-AC_SUBST(UBSAN_LDFLAGS)
+	UBSAN_FLAGS="-fsanitize=undefined")
 
 TSAN_CFLAGS=
 TSAN_LDFLAGS=
 AC_ARG_ENABLE(tsan,
 	AS_HELP_STRING([--enable-tsan],[enable thread sanitizer (default is NO)]),
-	CFLAGS="${CFLAGS} -fPIC"
-	TSAN_CFLAGS="-fsanitize=thread -fPIE -fno-omit-frame-pointer"
-	TSAN_LDFLAGS="-fsanitize=thread -pie")
-AC_SUBST(TSAN_CFLAGS)
-AC_SUBST(TSAN_LDFLAGS)
+	TSAN_FLAGS="-fsanitize=thread")
 
 ASAN_CFLAGS=
 ASAN_LDFLAGS=
 AC_ARG_ENABLE(asan,
 	AS_HELP_STRING([--enable-asan],[enable address sanitizer (default is NO)]),
-	CFLAGS="${CFLAGS} -fPIC"
-	ASAN_CFLAGS="-fsanitize=address -fPIE -fno-omit-frame-pointer"
-	ASAN_LDFLAGS="-fsanitize=address -pie")
-AC_SUBST(ASAN_CFLAGS)
-AC_SUBST(ASAN_LDFLAGS)
+	ASAN_FLAGS="-fsanitize=address")
 
 MSAN_CFLAGS=
 MSAN_LDFLAGS=
 AC_ARG_ENABLE(msan,
 	AS_HELP_STRING([--enable-msan],[enable memory sanitizer (default is NO)]),
-	CFLAGS="${CFLAGS} -fPIC"
-	MSAN_CFLAGS="-fsanitize=memory -fPIE -fno-omit-frame-pointer"
-	MSAN_LDFLAGS="-fsanitize=memory -pie")
-AC_SUBST(MSAN_CFLAGS)
-AC_SUBST(MSAN_LDFLAGS)
+	MSAN_FLAGS="-fsanitize=memory")
+
+SAN_CFLAGS="-fPIC ${UBSAN_FLAGS} ${TSAN_FLAGS} ${ASAN_FLAGS} ${MSAN_FLAGS} -fPIE -fno-omit-frame-pointer"
+AC_SUBST(SAN_CFLAGS)
+SAN_LDFLAGS="${UBSAN_FLAGS} ${TSAN_FLAGS} ${ASAN_FLAGS} ${MSAN_FLAGS} -pie"
+AC_SUBST(SAN_LDFLAGS)
 
 # Use jemalloc on Linux
 JEMALLOC_LDADD=
diff --git a/lib/libvarnish/Makefile.am b/lib/libvarnish/Makefile.am
index b8ed85e..967ec9c 100644
--- a/lib/libvarnish/Makefile.am
+++ b/lib/libvarnish/Makefile.am
@@ -38,15 +38,9 @@ libvarnish_la_SOURCES = \
 
 libvarnish_la_CFLAGS = \
 	-DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 libvarnish_la_LIBADD = ${RT_LIBS} ${NET_LIBS} ${LIBM} @PCRE_LIBS@ \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@
+	@SAN_LDFLAGS@
 
 if ENABLE_TESTS
 TESTS = vnum_c_test
diff --git a/lib/libvarnishapi/Makefile.am b/lib/libvarnishapi/Makefile.am
index dbaece1..b5eb2b0 100644
--- a/lib/libvarnishapi/Makefile.am
+++ b/lib/libvarnishapi/Makefile.am
@@ -45,16 +45,10 @@ libvarnishapi_la_SOURCES = \
 
 libvarnishapi_la_CFLAGS = \
 	-DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 libvarnishapi_la_LIBADD = @PCRE_LIBS@ @RT_LIBS@ \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@
+	@SAN_LDFLAGS@
 
 if HAVE_LD_VERSION_SCRIPT
 libvarnishapi_la_LDFLAGS += -Wl,--version-script=$(srcdir)/libvarnishapi.map
diff --git a/lib/libvarnishcompat/Makefile.am b/lib/libvarnishcompat/Makefile.am
index 2b4c469..f2c878b 100644
--- a/lib/libvarnishcompat/Makefile.am
+++ b/lib/libvarnishcompat/Makefile.am
@@ -9,16 +9,10 @@ AM_LDFLAGS  = $(AM_LT_LDFLAGS)
 pkglib_LTLIBRARIES = libvarnishcompat.la
 
 libvarnishcompat_la_CFLAGS = \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 libvarnishcompat_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@
+	@SAN_LDFLAGS@
 
 libvarnishcompat_la_SOURCES = \
 	daemon.c \
diff --git a/lib/libvcc/Makefile.am b/lib/libvcc/Makefile.am
index 6f55208..6cb09fb 100644
--- a/lib/libvcc/Makefile.am
+++ b/lib/libvcc/Makefile.am
@@ -9,16 +9,10 @@ AM_CPPFLAGS = \
 pkglib_LTLIBRARIES = libvcc.la
 
 libvcc_la_CFLAGS = \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 libvcc_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@
+	@SAN_LDFLAGS@
 
 libvcc_la_SOURCES = \
 	vcc_compile.h \
diff --git a/lib/libvgz/Makefile.am b/lib/libvgz/Makefile.am
index 7bbc9fd..f4aa8e9 100644
--- a/lib/libvgz/Makefile.am
+++ b/lib/libvgz/Makefile.am
@@ -4,15 +4,9 @@ AM_LDFLAGS  = $(AM_LT_LDFLAGS)
 pkglib_LTLIBRARIES = libvgz.la
 
 libvgz_la_LDFLAGS = $(AM_LDFLAGS) -avoid-version \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@
+	@SAN_LDFLAGS@
 libvgz_la_CFLAGS = -D_LARGEFILE64_SOURCE=1 -DZLIB_CONST $(libvgz_extra_cflags) \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 libvgz_la_SOURCES = \
 	adler32.c \
diff --git a/lib/libvmod_debug/Makefile.am b/lib/libvmod_debug/Makefile.am
index 653e2b4..aeb3e42 100644
--- a/lib/libvmod_debug/Makefile.am
+++ b/lib/libvmod_debug/Makefile.am
@@ -14,16 +14,10 @@ vmodtoolargs = --strict
 noinst_LTLIBRARIES = libvmod_debug.la
 
 libvmod_debug_la_CFLAGS = \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 libvmod_debug_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared -rpath /nowhere \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@
+	@SAN_LDFLAGS@
 
 libvmod_debug_la_SOURCES = \
 	vmod_debug.c \
diff --git a/lib/libvmod_directors/Makefile.am b/lib/libvmod_directors/Makefile.am
index 4d0f8e1..65306a8 100644
--- a/lib/libvmod_directors/Makefile.am
+++ b/lib/libvmod_directors/Makefile.am
@@ -13,16 +13,10 @@ vmodtoolargs = --strict
 vmod_LTLIBRARIES = libvmod_directors.la
 
 libvmod_directors_la_CFLAGS = \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 libvmod_directors_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@
+	@SAN_LDFLAGS@
 
 libvmod_directors_la_SOURCES = \
 	vdir.c \
diff --git a/lib/libvmod_std/Makefile.am b/lib/libvmod_std/Makefile.am
index f895478..650db7d 100644
--- a/lib/libvmod_std/Makefile.am
+++ b/lib/libvmod_std/Makefile.am
@@ -14,16 +14,10 @@ vmodtoolargs = --strict
 vmod_LTLIBRARIES = libvmod_std.la
 
 libvmod_std_la_CFLAGS = \
-	@ASAN_CFLAGS@ \
-	@MSAN_CFLAGS@ \
-	@TSAN_CFLAGS@ \
-	@UBSAN_CFLAGS@
+	@SAN_CFLAGS@
 
 libvmod_std_la_LDFLAGS = $(AM_LDFLAGS) -module -export-dynamic -avoid-version -shared \
-	@ASAN_LDFLAGS@ \
-	@MSAN_LDFLAGS@ \
-	@TSAN_LDFLAGS@ \
-	@UBSAN_LDFLAGS@
+	@SAN_LDFLAGS@
 
 libvmod_std_la_SOURCES = \
 	vmod_std.c \



From dridi.boukelmoune at gmail.com  Thu Feb 18 17:41:48 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Thu, 18 Feb 2016 18:41:48 +0100
Subject: [master] a8381a2 Polish
Message-ID: <E1aWSaK-00032j-Dw@project.varnish-software.com>


commit a8381a2c0ac53f8f8509f401fe2445f5086331c2
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Thu Feb 18 18:41:20 2016 +0100

    Polish

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 2f44dcd..4fc5359 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -538,13 +538,16 @@ vbf_fetch_body_helper(struct busyobj *bo)
  */
 
 #define vbf_vfp_push(bo, vfp, top)					\
-	if (VFP_Push((bo)->vfc, (vfp), (top)) == NULL) {		\
-		assert (WS_Overflowed((bo)->vfc->http->ws));		\
-		(void)VFP_Error((bo)->vfc, "workspace_backend overflow"); \
-		(bo)->htc->doclose = SC_OVERLOAD;			\
-		VDI_Finish((bo)->wrk, bo);				\
-		return (F_STP_ERROR);					\
-	}
+	do {								\
+		if (VFP_Push((bo)->vfc, (vfp), (top)) == NULL) {	\
+			assert (WS_Overflowed((bo)->vfc->http->ws));	\
+			(void)VFP_Error((bo)->vfc,			\
+			    "workspace_backend overflow");		\
+			(bo)->htc->doclose = SC_OVERLOAD;		\
+			VDI_Finish((bo)->wrk, bo);			\
+			return (F_STP_ERROR);				\
+		}							\
+	} while (0)
 
 static enum fetch_step
 vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)



From guillaume at varnish-software.com  Thu Feb 18 19:27:29 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Thu, 18 Feb 2016 20:27:29 +0100
Subject: [master] daafa94 Don't add extra flags if no sanitizer is enabled
Message-ID: <E1aWUEb-0006Dh-SX@project.varnish-software.com>


commit daafa949f0ebaaff70b7279b7d161c3606b26ac9
Author: Guillaume Quintard <guillaume at varnish-software.com>
Date:   Thu Feb 18 19:23:53 2016 +0000

    Don't add extra flags if no sanitizer is enabled
    
    Thanks fgs for catching it

diff --git a/configure.ac b/configure.ac
index edec1ec..9526ce1 100644
--- a/configure.ac
+++ b/configure.ac
@@ -303,9 +303,14 @@ AC_ARG_ENABLE(msan,
 	AS_HELP_STRING([--enable-msan],[enable memory sanitizer (default is NO)]),
 	MSAN_FLAGS="-fsanitize=memory")
 
-SAN_CFLAGS="-fPIC ${UBSAN_FLAGS} ${TSAN_FLAGS} ${ASAN_FLAGS} ${MSAN_FLAGS} -fPIE -fno-omit-frame-pointer"
+if test "x$UBSAN_FLAGS$TSAN_FLAGS$ASAN_FLAGS$MSAN_FLAGS" = x; then 
+	SAN_CFLAGS=
+	SAN_LDFLAGS=
+else
+	SAN_CFLAGS="-fPIC ${UBSAN_FLAGS} ${TSAN_FLAGS} ${ASAN_FLAGS} ${MSAN_FLAGS} -fPIE -fno-omit-frame-pointer"
+	SAN_LDFLAGS="${UBSAN_FLAGS} ${TSAN_FLAGS} ${ASAN_FLAGS} ${MSAN_FLAGS} -pie"
+fi
 AC_SUBST(SAN_CFLAGS)
-SAN_LDFLAGS="${UBSAN_FLAGS} ${TSAN_FLAGS} ${ASAN_FLAGS} ${MSAN_FLAGS} -pie"
 AC_SUBST(SAN_LDFLAGS)
 
 # Use jemalloc on Linux



From fgsch at lodoss.net  Fri Feb 19 19:06:41 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Fri, 19 Feb 2016 20:06:41 +0100
Subject: [master] 4985dea Polish
Message-ID: <E1aWqO1-0007Kp-Mi@project.varnish-software.com>


commit 4985dea5861d990aebf1336a7fc2810715fcd233
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Fri Feb 19 10:38:54 2016 +0000

    Polish

diff --git a/configure.ac b/configure.ac
index 9526ce1..29afdfa 100644
--- a/configure.ac
+++ b/configure.ac
@@ -78,7 +78,7 @@ AC_ARG_WITH([dot],
   [AC_CHECK_PROGS(DOT, [dot], [no])
    if test "x$DOT" = "xno"; then
      AC_MSG_WARN(
-       [dot not found - can't generate graphviz output for documentation.])
+       [dot not found - cannot generate graphviz output for documentation.])
    fi])
 AM_CONDITIONAL(HAVE_DOT,[test "x$DOT" != "xno"])
 
@@ -279,6 +279,8 @@ if test "$ac_cv_have_viz" = no; then
 fi
 CFLAGS="${save_CFLAGS}"
 
+SAN_CFLAGS=
+SAN_LDFLAGS=
 UBSAN_CFLAGS=
 UBSAN_LDFLAGS=
 AC_ARG_ENABLE(ubsan,
@@ -303,11 +305,8 @@ AC_ARG_ENABLE(msan,
 	AS_HELP_STRING([--enable-msan],[enable memory sanitizer (default is NO)]),
 	MSAN_FLAGS="-fsanitize=memory")
 
-if test "x$UBSAN_FLAGS$TSAN_FLAGS$ASAN_FLAGS$MSAN_FLAGS" = x; then 
-	SAN_CFLAGS=
-	SAN_LDFLAGS=
-else
-	SAN_CFLAGS="-fPIC ${UBSAN_FLAGS} ${TSAN_FLAGS} ${ASAN_FLAGS} ${MSAN_FLAGS} -fPIE -fno-omit-frame-pointer"
+if test "x$UBSAN_FLAGS$TSAN_FLAGS$ASAN_FLAGS$MSAN_FLAGS" != "x"; then
+	SAN_CFLAGS="${UBSAN_FLAGS} ${TSAN_FLAGS} ${ASAN_FLAGS} ${MSAN_FLAGS} -fPIC -fPIE -fno-omit-frame-pointer"
 	SAN_LDFLAGS="${UBSAN_FLAGS} ${TSAN_FLAGS} ${ASAN_FLAGS} ${MSAN_FLAGS} -pie"
 fi
 AC_SUBST(SAN_CFLAGS)



From fgsch at lodoss.net  Fri Feb 19 19:06:41 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Fri, 19 Feb 2016 20:06:41 +0100
Subject: [master] 33eac64 Rename {vcl,vmod}_dir to {vcl,vmod}_path
Message-ID: <E1aWqO1-0007Kv-O1@project.varnish-software.com>


commit 33eac64fc32f7b884057ee13ec4470ebe432cc6a
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Fri Feb 19 10:39:27 2016 +0000

    Rename {vcl,vmod}_dir to {vcl,vmod}_path
    
    The former are kept for backward compatibility as aliases for now.
    While here update some related doc and remove dead code.

diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index fe96360..3cd8ddd 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -170,8 +170,8 @@ void mgt_vcc_startup(struct cli *, const char *b_arg, const char *f_arg,
 int mgt_push_vcls_and_start(struct cli *, unsigned *status, char **p);
 int mgt_has_vcl(void);
 extern char *mgt_cc_cmd;
-extern const char *mgt_vcl_dir;
-extern const char *mgt_vmod_dir;
+extern const char *mgt_vcl_path;
+extern const char *mgt_vmod_path;
 extern unsigned mgt_vcc_err_unref;
 extern unsigned mgt_vcc_allow_inline_c;
 extern unsigned mgt_vcc_unsafe_path;
diff --git a/bin/varnishd/mgt/mgt_param_tbl.c b/bin/varnishd/mgt/mgt_param_tbl.c
index 59fcfed..6ca1aeb 100644
--- a/bin/varnishd/mgt/mgt_param_tbl.c
+++ b/bin/varnishd/mgt/mgt_param_tbl.c
@@ -56,7 +56,13 @@ struct parspec mgt_parspec[] = {
 		"and %o will be replaced with the output file name.",
 		MUST_RELOAD,
 		VCC_CC , NULL },
-	{ "vcl_dir", tweak_string, &mgt_vcl_dir,
+	{ "vcl_dir", tweak_string, &mgt_vcl_path,
+		NULL, NULL,
+		"Old name for vcl_path, use that instead.",
+		0,
+		VARNISH_VCL_DIR,
+		NULL },
+	{ "vcl_path", tweak_string, &mgt_vcl_path,
 		NULL, NULL,
 		"Directory (or colon separated list of directories) "
 		"from which relative VCL filenames (vcl.load and "
@@ -64,7 +70,13 @@ struct parspec mgt_parspec[] = {
 		0,
 		VARNISH_VCL_DIR,
 		NULL },
-	{ "vmod_dir", tweak_string, &mgt_vmod_dir,
+	{ "vmod_dir", tweak_string, &mgt_vmod_path,
+		NULL, NULL,
+		"Old name for vmod_path, use that instead.",
+		0,
+		VARNISH_VMOD_DIR,
+		NULL },
+	{ "vmod_path", tweak_string, &mgt_vmod_path,
 		NULL, NULL,
 		"Directory (or colon separated list of directories) "
 		"where VMODs are to be found.",
diff --git a/bin/varnishd/mgt/mgt_vcc.c b/bin/varnishd/mgt/mgt_vcc.c
index 77f67c0..ac2c5f6 100644
--- a/bin/varnishd/mgt/mgt_vcc.c
+++ b/bin/varnishd/mgt/mgt_vcc.c
@@ -59,8 +59,8 @@ struct vcc_priv {
 };
 
 char *mgt_cc_cmd;
-const char *mgt_vcl_dir;
-const char *mgt_vmod_dir;
+const char *mgt_vcl_path;
+const char *mgt_vmod_path;
 unsigned mgt_vcc_err_unref;
 unsigned mgt_vcc_allow_inline_c;
 unsigned mgt_vcc_unsafe_path;
@@ -95,8 +95,8 @@ run_vcc(void *priv)
 
 	sb = VSB_new_auto();
 	XXXAN(sb);
-	VCP_VCL_dir(vcp, mgt_vcl_dir);
-	VCP_VMOD_dir(vcp, mgt_vmod_dir);
+	VCP_VCL_path(vcp, mgt_vcl_path);
+	VCP_VMOD_path(vcp, mgt_vmod_path);
 	VCP_Err_Unref(vcp, mgt_vcc_err_unref);
 	VCP_Allow_InlineC(vcp, mgt_vcc_allow_inline_c);
 	VCP_Unsafe_Path(vcp, mgt_vcc_unsafe_path);
diff --git a/bin/varnishtest/tests/c00053.vtc b/bin/varnishtest/tests/c00053.vtc
index 6a34045..6a8e50b 100644
--- a/bin/varnishtest/tests/c00053.vtc
+++ b/bin/varnishtest/tests/c00053.vtc
@@ -2,7 +2,7 @@ varnishtest "Test include vs. unsafe_path"
 
 server s1 {
 	rxreq
-	txresp -hdr "foo: bAr" -hdr "bar: fOo" -bodylen 4
+	txresp
 } -start
 
 shell "echo > ${tmpdir}/_.c00053"
@@ -14,13 +14,11 @@ varnish v1 -vcl+backend {
 varnish v1 -cliok "param.set vcc_unsafe_path off"
 
 varnish v1 -errvcl {' is unsafe} {
-	backend default {
-		.host = "${s1_sock}";
-	}
+	backend default { .host = "${s1_sock}"; }
 	include "${tmpdir}/_.c00053";
 }
 
-varnish v1 -cliok "param.set vcl_dir ${tmpdir}"
+varnish v1 -cliok "param.set vcl_path ${tmpdir}"
 
 varnish v1 -vcl+backend {
 	include "_.c00053";
diff --git a/bin/varnishtest/tests/m00003.vtc b/bin/varnishtest/tests/m00003.vtc
index 85970dd..f0b3120 100644
--- a/bin/varnishtest/tests/m00003.vtc
+++ b/bin/varnishtest/tests/m00003.vtc
@@ -1,4 +1,4 @@
-varnishtest "Test vmod_dir param"
+varnishtest "Test vmod_path param"
 
 feature topbuild
 
@@ -7,17 +7,19 @@ server s1 {
 	txresp
 } -start
 
-varnish v1 -arg "-pvmod_dir=${topbuild}/lib/libvmod_std/.libs/" \
+varnish v1 -arg "-pvmod_path=${topbuild}/lib/libvmod_std/.libs/" \
     -vcl+backend {
 	import std;
 } -start
 
-varnish v1 -cliok "param.set vmod_dir /nonexistent"
+varnish v1 -cliok "param.set vmod_path /nonexistent"
 
 varnish v1 -errvcl {Could not load VMOD std} {
 	import std;
 }
 
-varnish v1 -errvcl {Expected 'from path ...'} {
-	import std to;
+varnish v1 -cliok "param.set vmod_dir ${topbuild}/lib/libvmod_std/.libs/"
+
+varnish v1 -vcl+backend {
+	import std;
 }
diff --git a/bin/varnishtest/tests/m00008.vtc b/bin/varnishtest/tests/m00008.vtc
index 10cb23d..9386ae9 100644
--- a/bin/varnishtest/tests/m00008.vtc
+++ b/bin/varnishtest/tests/m00008.vtc
@@ -1,10 +1,10 @@
-varnishtest "Test std vmod vs. unsafe_path"
+varnishtest "Test import vs. unsafe_path"
 
 feature topbuild
 
 server s1 {
 	rxreq
-	txresp -hdr "foo: bAr" -hdr "bar: fOo" -bodylen 4
+	txresp
 } -start
 
 varnish v1 -vcl+backend {
@@ -18,7 +18,8 @@ varnish v1 -errvcl {'import ... from path ...' is unsafe.} {
 	import ${vmod_std};
 }
 
-varnish v1 -cliok "param.set vmod_dir /nowhere:${topbuild}/lib/libvmod_std/.libs/:/else"
+varnish v1 \
+    -cliok "param.set vmod_path /nowhere:${topbuild}/lib/libvmod_std/.libs/"
 
 varnish v1 -vcl+backend {
 	import std;
@@ -26,7 +27,7 @@ varnish v1 -vcl+backend {
 
 varnish v1 -cliok "param.set vcc_unsafe_path on"
 
-varnish v1 -cliok "param.set vmod_dir /nowhere:/else"
+varnish v1 -cliok "param.set vmod_path /nowhere:/else"
 
 varnish v1 -vcl+backend {
 	import std from "${topbuild}/lib/libvmod_std/.libs/";
diff --git a/bin/varnishtest/tests/v00020.vtc b/bin/varnishtest/tests/v00020.vtc
index 263b883..ff178f6 100644
--- a/bin/varnishtest/tests/v00020.vtc
+++ b/bin/varnishtest/tests/v00020.vtc
@@ -284,3 +284,7 @@ varnish v1 -errvcl {Names of VCL acl's cannot contain '-'} {
 		}
 	}
 }
+
+varnish v1 -errvcl {Expected 'from path ...'} {
+	import std to;
+}
diff --git a/bin/varnishtest/tests/v00046.vtc b/bin/varnishtest/tests/v00046.vtc
index f87cd21..2bbe9d4 100644
--- a/bin/varnishtest/tests/v00046.vtc
+++ b/bin/varnishtest/tests/v00046.vtc
@@ -1,8 +1,8 @@
-varnishtest "Test relative to vcl_dir, dot-include and absolute includes"
+varnishtest "Test relative to vcl_path, dot-include and absolute includes"
 
 # relative plain
 shell "true > ${tmpdir}/_start.vcl"
-varnish v1 -arg "-p vcl_dir=${tmpdir}" -vcl {
+varnish v1 -arg "-p vcl_path=${tmpdir}" -vcl {
 	backend b { .host = "127.0.0.1"; }
 	include "_start.vcl" ;
 }
@@ -22,7 +22,7 @@ varnish v1 -vcl {
 	include "${tmpdir}/1/a.vcl" ;
 }
 
-# same but relative to vcl_dir
+# same but relative to vcl_path
 shell "echo 'include \"1/2/b.vcl\";' > ${tmpdir}/1/ab.vcl"
 varnish v1 -vcl {
 	backend b { .host = "127.0.0.1"; }
@@ -70,4 +70,3 @@ shell "rm -f ${tmpdir}/_start.vcl"
 varnish v1 -errvcl {needs absolute filename of including file.} {
 	include "./foobar";
 }
-
diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst
index 3372939..535b403 100644
--- a/doc/sphinx/reference/varnishd.rst
+++ b/doc/sphinx/reference/varnishd.rst
@@ -120,7 +120,7 @@ OPTIONS
   Make the listed parameters read only. This gives the system
   administrator a way to limit what the Varnish CLI can do.  Consider
   making parameters such as *cc_command*, *vcc_allow_inline_c* and
-  *vmod_dir* read only as these can potentially be used to escalate
+  *vmod_path* read only as these can potentially be used to escalate
   privileges from the CLI.
 
 -S file
diff --git a/doc/sphinx/users-guide/run_security.rst b/doc/sphinx/users-guide/run_security.rst
index 2beb4aa..5340e6c 100644
--- a/doc/sphinx/users-guide/run_security.rst
+++ b/doc/sphinx/users-guide/run_security.rst
@@ -158,12 +158,12 @@ Furthermore you may want to look at and lock down:
 	Log all CLI commands to `syslog(8)`, so you know what goes on.
 
 :ref:`ref_param_vcc_unsafe_path`
-	Restrict VCL/VMODS to :ref:`ref_param_vcl_dir` and :ref:`ref_param_vmod_dir`
+	Restrict VCL/VMODs to :ref:`ref_param_vcl_path` and :ref:`ref_param_vmod_path`
 
-:ref:`ref_param_vmod_dir`
-        The directory where Varnish will will look
-        for modules. This could potentially be used to load rouge
-        modules into Varnish.
+:ref:`ref_param_vmod_path`
+        The directory (or colon separated list of directories) where
+        Varnish will will look for modules. This could potentially be
+        used to load rogue modules into Varnish.
 
 The CLI interface
 -----------------
diff --git a/include/libvcc.h b/include/libvcc.h
index fbd7bd1..0d3bb53 100644
--- a/include/libvcc.h
+++ b/include/libvcc.h
@@ -32,8 +32,8 @@ struct vcc;
 
 struct vcp *VCP_New(void);
 void VCP_Builtin_VCL(struct vcp *, const char *str);
-void VCP_VCL_dir(struct vcp *, const char *str);
-void VCP_VMOD_dir(struct vcp *, const char *str);
+void VCP_VCL_path(struct vcp *, const char *str);
+void VCP_VMOD_path(struct vcp *, const char *str);
 void VCP_Err_Unref(struct vcp *tl, unsigned u);
 void VCP_Allow_InlineC(struct vcp *tl, unsigned u);
 void VCP_Unsafe_Path(struct vcp *tl, unsigned u);
diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c
index 5db516f..0f94b21 100644
--- a/lib/libvcc/vcc_compile.c
+++ b/lib/libvcc/vcc_compile.c
@@ -898,28 +898,26 @@ VCP_Builtin_VCL(struct vcp *vcp, const char *str)
 }
 
 /*--------------------------------------------------------------------
- * Configure default VCL source directory
+ * Configure default VCL source path
  */
 
 void
-VCP_VCL_dir(struct vcp *vcp, const char *str)
+VCP_VCL_path(struct vcp *vcp, const char *str)
 {
 
 	CHECK_OBJ_NOTNULL(vcp, VCP_MAGIC);
-	REPLACE(vcp->vcl_dir, str);
 	VFIL_setpath(&vcp->vcl_path, str);
 }
 
 /*--------------------------------------------------------------------
- * Configure default VMOD directory
+ * Configure default VMOD path
  */
 
 void
-VCP_VMOD_dir(struct vcp *vcp, const char *str)
+VCP_VMOD_path(struct vcp *vcp, const char *str)
 {
 
 	CHECK_OBJ_NOTNULL(vcp, VCP_MAGIC);
-	REPLACE(vcp->vmod_dir, str);
 	VFIL_setpath(&vcp->vmod_path, str);
 }
 
diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h
index 3feecd3..7816a3e 100644
--- a/lib/libvcc/vcc_compile.h
+++ b/lib/libvcc/vcc_compile.h
@@ -160,9 +160,7 @@ struct vcp {
 #define VCP_MAGIC		0xd90acfbc
 
 	char			*builtin_vcl;
-	char			*vcl_dir;
 	struct vfil_path	*vcl_path;
-	char			*vmod_dir;
 	struct vfil_path	*vmod_path;
 	unsigned		err_unref;
 	unsigned		allow_inline_c;



From fgsch at lodoss.net  Fri Feb 19 19:06:41 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Fri, 19 Feb 2016 20:06:41 +0100
Subject: [master] 11f7b74 Further tweaking for smartOS-amd64
Message-ID: <E1aWqO1-0007L5-P4@project.varnish-software.com>


commit 11f7b74f3fab2e1151f84efa424a45dfb6d6535f
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Fri Feb 19 19:03:30 2016 +0000

    Further tweaking for smartOS-amd64

diff --git a/bin/varnishtest/tests/c00070.vtc b/bin/varnishtest/tests/c00070.vtc
index 2f8e992..b2df5b4 100644
--- a/bin/varnishtest/tests/c00070.vtc
+++ b/bin/varnishtest/tests/c00070.vtc
@@ -39,7 +39,7 @@ client c1 {
 	expect resp.http.overflowed == "false"
 
 	expect resp.http.free_backend > 9000
-	expect resp.http.free_session >= 240
+	expect resp.http.free_session >= 232
 	expect resp.http.free_thread > 2000
 } -run
 



From phk at FreeBSD.org  Sat Feb 20 21:08:20 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sat, 20 Feb 2016 22:08:20 +0100
Subject: [master] d3a647c A handful of design documents from the very start of
 Varnish
Message-ID: <E1aXElI-0003BM-CA@project.varnish-software.com>


commit d3a647c5280bfe4fda30a7b94f7057fc4d3754e8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sat Feb 20 21:07:48 2016 +0000

    A handful of design documents from the very start of Varnish

diff --git a/doc/sphinx/phk/firstdesign.rst b/doc/sphinx/phk/firstdesign.rst
new file mode 100644
index 0000000..216101a
--- /dev/null
+++ b/doc/sphinx/phk/firstdesign.rst
@@ -0,0 +1,1635 @@
+.. _phk_firstdesign:
+
+===========================
+The first design of Varnish
+===========================
+
+I have been working on a "bit-storage" facility for datamuseum.dk,
+and as part of my "eat your own dog-food" policy, I converting my
+own personal archive (41 DVD's worth) as a test.
+
+Along the way I passed through 2006 and found some files from
+the birth of Varnish 10 years ago.
+
+The first Varnish Design notes
+------------------------------
+
+This file are notes taken during a meeting in Oslo on 2nd feb 2006,
+which in essence consisted of Anders Berg cursing Squid for a couple
+of hours.
+
+(Originally the meeting was scheduled for jan 24th but a SAS pilot
+strike put an end to that.)
+
+To be honest I knew very little about web-traffic, my own homepage
+was written in HTML in vi(1), so I had a bit of catching up to do
+on that front, but the overall job was pretty simple:  A program
+to move bytes ... fast.
+
+It is quite interesting to see how many things we got right and
+where we kept thinking in the old frame of reference (ie: Squid)::
+
+	Notes on Varnish
+	----------------
+
+	Collected 2006-02-02 to 2006-02-..
+
+	Poul-Henning Kamp
+
+
+	Philosophy
+	----------
+
+	It is not enough to deliver a technically superior piece of software,
+	if it is not possible for people to deploy it usefully in a sensible
+	way and timely fashion.
+
+
+	Deployment scenarios
+	--------------------
+
+	There are two fundamental usage scenarios for Varnish: when the
+	first machine is brought up to offload a struggling backend and
+	when a subsequent machine is brought online to help handle the load.
+
+
+	The first (layer of) Varnish
+	----------------------------
+
+	Somebodys webserver is struggling and they decide to try Varnish.
+
+	Often this will be a skunkworks operation with some random PC
+	purloined from wherever it wasn't being used and the Varnish "HOWTO"
+	in one hand.
+
+	If they do it in an orderly fashion before things reach panic proportions,
+	a sensible model is to setup the Varnish box, test it out from your
+	own browser, see that it answers correctly.  Test it some more and
+	then add the IP# to the DNS records so that it takes 50% of the load
+	off the backend.
+
+	If it happens as firefighting at 3AM the backend will be moved to another
+	IP, the Varnish box given the main IP and things had better work real
+	well, really fast.
+
+	In both cases, it would be ideal if all that is necessary to tell
+	Varnish are two pieces of information:
+
+		Storage location
+			Alternatively we can offer an "auto" setting that makes
+			Varnish discover what is available and use what it find.
+
+		DNS or IP# of backend.
+
+			IP# is useful when the DNS settings are not quite certain
+			or when split DNS horizon setups are used.
+
+	Ideally this can be done on the commandline so that there is no
+	configuration file to edit to get going, just
+
+		varnish -d /home/varnish -s backend.example.dom
+
+	and you're off running.
+
+	A text, curses or HTML based based facility to give some instant
+	feedback and stats is necessary.
+
+	If circumstances are not conductive to strucured approach, it should
+	be possible to repeat this process and set up N independent Varnish
+	boxes and get some sort of relief without having to read any further
+	documentation.
+
+
+	The subsequent (layers of) Varnish
+	----------------------------------
+
+	This is what happens once everybody has caught their breath,
+	and where we start to talk about Varnish clusters.
+
+	We can assume that at this point, the already installed Varnish
+	machines have been configured more precisely and that people
+	have studied Varnish configuration to some level of detail.
+
+	When Varnish machines are put in a cluster, the administrator should
+	be able to consider the cluster as a unit and not have to think and
+	interact with the individual nodes.
+
+	Some sort of central management node or facility must exist and
+	it would be preferable if this was not a physical but a logical
+	entity so that it can follow the admin to the beach.  Ideally it
+	would give basic functionality in any browser, even mobile phones.
+
+	The focus here is scaleability, we want to avoid per-machine
+	configuration if at all possible.  Ideally, preconfigured hardware
+	can be plugged into power and net, find an address with DHCP, contact
+	preconfigured management node, get a configuration and start working.
+
+	But we also need to think about how we avoid a site of Varnish
+	machines from acting like a stampeeding horde when the power or
+	connectivity is brought back after a disruption.  Some sort of
+	slow starting ("warm-up" ?) must be implemented to prevent them
+	from hitting all the backend with the full force.
+
+	An important aspect of cluster operations is giving a statistically
+	meaninful judgement of the cluster size, in particular answering
+	the question "would adding another machine help ?" precisely.
+
+	We should have a facility that allows the administrator to type
+	in a REGEXP/URL and have all the nodes answer with a checksum, age
+	and expiry timer for any documents they have which match.  The
+	results should be grouped by URL and checksum.
+
+
+	Technical concepts
+	------------------
+
+	We want the central Varnish process to be that, just one process, and
+	we want to keep it small and efficient at all cost.
+
+	Code that will not be used for the central functionality should not
+	be part of the central process.  For instance code to parse, validate
+	and interpret the (possibly) complex configuration file should be a
+	separate program.
+
+	Depending on the situation, the Varnish process can either invoke
+	this program via a pipe or receive the ready to use data structures
+	via a network connection.
+
+	Exported data from the Varnish process should be made as cheap as
+	possible, likely shared memory.  That will allow us to deploy separate
+	processes for log-grabbing, statistics monitoring and similar
+	"off-duty" tasks and let the central process get on with the
+	important job.
+
+
+	Backend interaction
+	-------------------
+
+	We need a way to tune the backend interaction further than what the
+	HTTP protocol offers out of the box.
+
+	We can assume that all documents we get from the backend has an
+	expiry timer, if not we will set a default timer (configurable of
+	course).
+
+	But we need further policy than that.  Amongst the questions we have
+	to ask are:
+
+		How long time after the expiry can we serve a cached copy
+		of this document while we have reason to belive the backend
+		can supply us with an update ?
+		
+		How long time after the expiry can we serve a cached copy
+		of this document if the backend does not reply or is
+		unreachable.
+
+		If we cannot serve this document out of cache and the backend
+		cannot inform us, what do we serve instead (404 ?  A default
+		document of some sort ?)
+
+		Should we just not serve this page at all if we are in a
+		bandwidth crush (DoS/stampede) situation ?
+
+	It may also make sense to have a "emergency detector" which triggers
+	when the backend is overloaded and offer a scaling factor for all
+	timeouts for when in such an emergency state.  Something like "If
+	the average response time of the backend rises above 10 seconds,
+	multiply all expiry timers by two".
+
+	It probably also makes sense to have a bandwidth/request traffic
+	shaper for backend traffic to prevent any one Varnish machine from
+	pummeling the backend in case of attacks or misconfigured
+	expiry headers.
+
+
+	Startup/consistency
+	-------------------
+
+	We need to decide what to do about the cache when the Varnish
+	process starts.  There may be a difference between it starting
+	first time after the machine booted and when it is subsequently
+	(re)started.
+
+	By far the easiest thing to do is to disregard the cache, that saves
+	a lot of code for locating and validating the contents, but this
+	carries a penalty in backend or cluster fetches whenever a node
+	comes up.  Lets call this the "transient cache model"
+
+	The alternative is to allow persistently cached contents to be used
+	according to configured criteria:
+
+		Can expired contents be served if we can't contact the
+		backend ?  (dangerous...)
+
+		Can unexpired contents be served if we can't contact the
+		backend ?  If so, how much past the expiry ?
+
+	It is a very good question how big a fraction of the persistent
+	cache would be usable after typical downtimes:
+
+		After a Varnish process restart:  Nearly all.
+
+		After a power-failure ?  Probably at least half, but probably
+		not the half that contains the most busy pages.
+
+	And we need to take into consideration if validating the format and
+	contents of the cache might take more resources and time than getting
+	the content from the backend.
+
+	Off the top of my head, I would prefer the transient model any day
+	because of the simplicity and lack of potential consistency problems,
+	but if the load on the back end is intolerable this may not be
+	practically feasible.
+
+	The best way to decide is to carefully analyze a number of cold
+	starts and cache content replacement traces.
+
+	The choice we make does affect the storage management part of Varnish,
+	but I see that is being modular in any instance, so it may merely be
+	that some storage modules come up clean on any start while other
+	will come up with existing objects cached.
+
+
+	Clustering
+	----------
+
+	I'm somewhat torn on clustering for traffic purposes.  For admin
+	and management: Yes, certainly, but starting to pass objects from
+	one machine in a cluster to another is likely to be just be a waste
+	of time and code.
+
+	Today one can trivially fit 1TB into a 1U machine so the partitioning
+	argument for cache clusters doesn't sound particularly urgent to me.
+
+	If all machines in the cluster have sufficient cache capacity, the
+	other remaining argument is backend offloading, that would likely
+	be better mitigated by implementing a 1:10 style two-layer cluster
+	with the second level node possibly having twice the storage of
+	the front row nodes.
+
+	The coordination necessary for keeping track of, or discovering in
+	real-time, who has a given object can easily turn into a traffic
+	and cpu load nightmare.
+
+	And from a performance point of view, it only reduces quality:
+	First we send out a discovery multicast, then we wait some amount
+	of time to see if a response arrives only then should we start
+	to ask the backend for the object.  With a two-level cluster
+	we can ask the layer-two node right away and if it doesn't have
+	the object it can ask the back-end right away, no timeout is
+	involved in that.
+
+	Finally Consider the impact on a cluster of a "must get" object
+	like an IMG tag with a misspelled URL.  Every hit on the front page
+	results in one get of the wrong URL.  One machine in the cluster
+	ask everybody else in the cluster "do you have this URL" every
+	time somebody gets the frontpage.
+
+	If we implement a negative feedback protocol ("No I don't"), then
+	each hit on the wrong URL will result in N+1 packets (assuming multicast).
+
+	If we use a silent negative protocol the result is less severe for
+	the machine that got the request, but still everybody wakes up to
+	to find out that no, we didn't have that URL.
+
+	Negative caching can mitigate this to some extent.
+
+
+	Privacy
+	-------
+
+	Configuration data and instructions passed forth and back should
+	be encrypted and signed if so configured.  Using PGP keys is
+	a very tempting and simple solution which would pave the way for
+	administrators typing a short ascii encoded pgp signed message
+	into a SMS from their Bahamas beach vacation...
+
+
+	Implementation ideas
+	--------------------
+
+	The simplest storage method mmap(2)'s a disk or file and puts
+	objects into the virtual memory on page aligned boundaries,
+	using a small struct for metadata.  Data is not persistant
+	across reboots.  Object free is incredibly cheap.  Object
+	allocation should reuse recently freed space if at all possible.
+	"First free hole" is probably a good allocation strategy.
+	Sendfile can be used if filebacked.  If nothing else disks
+	can be used by making a 1-file filesystem on them.
+
+	More complex storage methods are object per file and object
+	in database models.  They are relatively trival and well
+	understood.  May offer persistence.
+
+	Read-Only storage methods may make sense for getting hold
+	of static emergency contents from CD-ROM etc.
+
+	Treat each disk arm as a separate storage unit and keep track of
+	service time (if possible) to decide storage scheduling.
+
+	Avoid regular expressions at runtime.  If config file contains
+	regexps, compile them into executable code and dlopen() it
+	into the Varnish process.  Use versioning and refcounts to
+	do memory management on such segments.
+
+	Avoid committing transmit buffer space until we have bandwidth
+	estimate for client.  One possible way:  Send HTTP header
+	and time ACKs getting back, then calculate transmit buffer size
+	and send object.  This makes DoS attacks more harmless and
+	mitigates traffic stampedes.
+
+	Kill all TCP connections after N seconds, nobody waits an hour
+	for a web-page to load.
+
+	Abuse mitigation interface to firewall/traffic shaping:  Allow
+	the central node to put an IP/Net into traffic shaping or take
+	it out of traffic shaping firewall rules.  Monitor/interface
+	process (not main Varnish process) calls script to config
+	firewalling.
+
+	"Warm-up" instructions can take a number of forms and we don't know
+	what is the most efficient or most usable.  Here are some ideas:
+
+	    Start at these URL's then...
+
+		... follow all links down to N levels.
+
+		... follow all links that match REGEXP no deeper than N levels down.
+
+		... follow N random links no deeper than M levels down.
+
+		... load N objects by following random links no deeper than
+		    M levels down.
+
+	    But...
+
+		... never follow any links that match REGEXP
+
+		... never pick up objects larger than N bytes
+
+		... never pick up objects older than T seconds
+
+
+	It makes a lot of sense to not actually implement this in the main
+	Varnish process, but rather supply a template perl or python script
+	that primes the cache by requesting the objects through Varnish.
+	(That would require us to listen separately on 127.0.0.1
+	so the perlscript can get in touch with Varnish while in warm-up.)
+
+	One interesting but quite likely overengineered option in the
+	cluster case is if the central monitor tracks a fraction of the
+	requests through the logs of the running machines in the cluster,
+	spots the hot objects and tell the warming up varnish what objects
+	to get and from where.
+
+
+	In the cluster configuration, it is probably best to run the cluster
+	interaction in a separate process rather than the main Varnish
+	process.  From Varnish to cluster info would go through the shared
+	memory, but we don't want to implement locking in the shmem so
+	some sort of back-channel (UNIX domain or UDP socket ?) is necessary.
+
+	If we have such an "supervisor" process, it could also be tasked
+	with restarting the varnish process if vitals signs fail:  A time
+	stamp in the shmem or kill -0 $pid.
+
+	It may even make sense to run the "supervisor" process in stand
+	alone mode as well, there it can offer a HTML based interface
+	to the Varnish process (via shmem).
+
+	For cluster use the user would probably just pass an extra argument
+	when he starts up Varnish:
+
+		varnish -c $cluster_args $other_args
+	vs
+
+		varnish $other_args
+
+	and a "varnish" shell script will Do The Right Thing.
+
+
+	Shared memory
+	-------------
+
+	The shared memory layout needs to be thought about somewhat.  On one
+	hand we want it to be stable enough to allow people to write programs
+	or scripts that inspect it, on the other hand doing it entirely in
+	ascii is both slow and prone to race conditions.
+
+	The various different data types in the shared memory can either be
+	put into one single segment(= 1 file) or into individual segments
+	(= multiple files).  I don't think the number of small data types to
+	be big enough to make the latter impractical.
+
+	Storing the "big overview" data in shmem in ASCII or HTML would
+	allow one to point cat(1) or a browser directly at the mmaped file
+	with no interpretation necessary, a big plus in my book.
+
+	Similarly, if we don't update them too often, statistics could be stored
+	in shared memory in perl/awk friendly ascii format.
+
+	But the logfile will have to be (one or more) FIFO logs, probably at least
+	three in fact:  Good requests, Bad requests, and exception messages.
+
+	If we decide to make logentries fixed length, we could make them ascii
+	so that a simple "sort -n /tmp/shmem.log" would put them in order after
+	a leading numeric timestamp, but it is probably better to provide a
+	utility to cat/tail-f the log and keep the log in a bytestring FIFO
+	format.  Overruns should be marked in the output.
+
+
+	*END*
+
+The second Varnish Design notes
+-------------------------------
+
+You will notice above that there is no mention of VCL, it took a
+couple of weeks for that particular lightning to strike.
+
+Interestingly I know exactly where the lightning came from, and
+what it hit.
+
+The timeframe was around GCC 4.0.0 which was not their best release,
+and I had for some time been pondering a pre-processor for the C
+language to make up for the ISO-C stagnation and braindamage.
+
+I've read most of the "classic" compiler books, and probably read
+more compilers many people (Still to go: `GIER Algol 4 <http://datamuseum.dk/wiki/GIER/GA4GuideToDocumentationAndCode>`_) but to be honest I found
+them far too theoretical and not very helpful from a *practical* compiler
+construction point of view.
+
+But there is one compiler-book which takes an entirely different
+take:  `Hanson and Fraser's LCC book. <http://www.amazon.com/gp/search/?field-isbn=0805316701>`_ which throws LEX and YACC under the truck and
+concentrates on compiling.
+
+Taking their low-down approach to parsing, and emitting C code,
+there really isn't much compiler left to write, and I had done
+several interesting hacks towards my 'K' language.
+
+The lightning rod was all the ideas Anders had for how Varnish
+should be able to manipulate the traffic passing through, how
+to decide what to cache, how long time to cache it, where to
+cache it and ... it sounded like a lot of very detailed code
+which had to be incredibly configurable.
+
+Soon those two inspiratons collided::
+
+
+	Notes on Varnish
+	----------------
+
+	Collected 2006-02-24 to 2006-02-..
+
+	Poul-Henning Kamp
+
+	-----------------------------------------------------------------------
+	Policy Configuration
+
+	Policy is configured in a simple unidirectional (no loops, no goto)
+	programming language which is compiled into 'C' and from there binary
+	modules which are dlopen'ed by the main Varnish process.
+
+	The dl object contains one exported symbol, a pointer to a structure
+	which contains a reference count, a number of function pointers,
+	a couple of string variables with identifying information.
+
+	All access into the config is protected by the reference counts.
+
+	Multiple policy configurations can be loaded at the same time
+	but only one is the "active configuration".  Loading, switching and
+	unloading of policy configurations happen via the managment
+	process.
+
+	A global config sequence number is incremented on each switch and
+	policy modified object attributes (ttl, cache/nocache) are all
+	qualified by the config-sequence under which they were calculated
+	and invalid if a different policy is now in effect.
+
+	-----------------------------------------------------------------------
+	Configuration Language
+
+	XXX: include lines.
+
+	BNF:
+		program:	function
+				| program function
+
+		function:	"sub" function_name compound_statement
+
+		compound_statement:	"{" statements "}"
+
+		statements:	/* empty */
+				| statement
+				| statements statement
+				
+
+		statement:	if_statement
+				| call_statement
+				| "finish"
+				| assignment_statement
+				| action_statement
+
+		if_statement:	"if" condition compound_statement elif_parts else_part
+
+		elif_parts:	/* empty */
+				| elif_part
+				| elif_parts elif_part
+
+		elif_part:	"elseif" condition compound_statement
+				| "elsif" condition compound_statement
+				| "else if" condition compound_statement
+
+		else_part:	/* empty */
+				| "else" compound_statement
+
+		call_statement:	"call" function_name
+
+		assign_statement:	field "=" value
+
+		field:		object
+				field "." variable
+
+		action_statement:	action arguments
+
+		arguments:	/* empty */
+				arguments | argument
+
+	-----------------------------------------------------------------------
+	Sample request policy program
+
+		sub request_policy {
+
+			if (client.ip in 10.0.0.0/8) {
+				no-cache
+				finish
+			}
+
+			if (req.url.host ~ "cnn.no$") {
+				rewrite	s/cnn.no$/vg.no/
+			}
+
+			if (req.url.path ~ "cgi-bin") {
+				no-cache
+			}
+
+			if (req.useragent ~ "spider") {
+				no-new-cache
+			}
+
+			if (backend.response_time > 0.8s) {
+				set req.ttlfactor = 1.5
+			} elseif (backend.response_time > 1.5s) {
+				set req.ttlfactor = 2.0
+			} elseif (backend.response_time > 2.5s) {
+				set req.ttlfactor = 5.0
+			}
+
+			/*
+			 * the program contains no references to
+			 * maxage, s-maxage and expires, so the
+			 * default handling (RFC2616) applies
+			 */
+		}
+
+	-----------------------------------------------------------------------
+	Sample fetch policy program
+
+		sub backends {
+			set backend.vg.ip = {...}
+			set backend.ads.ip = {...}
+			set backend.chat.ip = {...}
+			set backend.chat.timeout = 10s
+			set backend.chat.bandwidth = 2000 MB/s
+			set backend.other.ip = {...}
+		}
+
+		sub vg_backend {
+			set backend.ip = {10.0.0.1-5}
+			set backend.timeout = 4s
+			set backend.bandwidth = 2000Mb/s
+		}
+
+		sub fetch_policy {
+
+			if (req.url.host ~ "/vg.no$/") {
+				set req.backend = vg
+				call vg_backend
+			} else {
+				/* XXX: specify 404 page url ? */
+				error 404
+			}
+
+			if (backend.response_time > 2.0s) {
+				if (req.url.path ~ "/landbrugspriser/") {
+					error 504
+				}
+			}
+			fetch
+			if (backend.down) {
+				if (obj.exist) {
+					set obj.ttl += 10m
+					finish
+				}
+				switch_config ohhshit
+			}
+			if (obj.result == 404) {
+				error 300 "http://www.vg.no"
+			}
+			if (obj.result != 200) {
+				finish
+			}
+			if (obj.size > 256k) {
+				no-cache
+			} else if (obj.size > 32k && obj.ttl < 2m) {
+				obj.tll = 5m				
+			}
+			if (backend.response_time > 2.0s) {
+				set ttl *= 2.0
+			}
+		}
+
+		sub prefetch_policy {
+
+			if (obj.usage < 10 && obj.ttl < 5m) {
+				fetch
+			}
+		}
+
+	-----------------------------------------------------------------------
+	Purging
+
+	When a purge request comes in, the regexp is tagged with the next
+	generation number and added to the tail of the list of purge regexps.
+
+	Before a sender transmits an object, it is checked against any
+	purge-regexps which have higher generation number than the object
+	and if it matches the request is sent to a fetcher and the object
+	purged.
+
+	If there were purge regexps with higher generation to match, but
+	they didn't match, the object is tagged with the current generation
+	number and moved to the tail of the list.
+
+	Otherwise, the object does not change generation number and is
+	not moved on the generation list.
+
+	New Objects are tagged with the current generation number and put
+	at the tail of the list.
+
+	Objects are removed from the generation list when deleted.
+
+	When a purge object has a lower generation number than the first
+	object on the generation list, the purge object has been completed
+	and will be removed.  A log entry is written with number of compares
+	and number of hits.
+		
+	-----------------------------------------------------------------------
+	Random notes
+
+		swap backed storage
+
+		slowstart by config-flipping
+			start-config has peer servers as backend
+			once hitrate goes above limit, management process
+			flips config to 'real' config.
+
+		stat-object
+			always URL, not regexp
+
+		management + varnish process in one binary, comms via pipe
+
+		Change from config with long expiry to short expiry, how
+		does the ttl drop ?  (config sequence number invalidates
+		all calculated/modified attributes.)
+
+		Mgt process holds copy of acceptor socket ->  Restart without
+		lost client requests.
+
+		BW limit per client IP: create shortlived object (<4sec)
+		to hold status.  Enforce limits by delaying responses.
+
+
+	-----------------------------------------------------------------------
+	Source structure
+
+
+		libvarnish
+			library with interface facilities, for instance
+			functions to open&read shmem log
+
+		varnish
+			varnish sources in three classes
+
+	-----------------------------------------------------------------------
+	protocol cluster/mgt/varnish
+
+	object_query url -> TTL, size, checksum
+	{purge,invalidate} regexp
+	object_status url -> object metadata
+
+	load_config filename
+	switch_config configname
+	list_configs
+	unload_config
+
+	freeze 	# stop the clock, freezes the object store
+	thaw
+
+	suspend	# stop acceptor accepting new requests
+	resume
+
+	stop	# forced stop (exits) varnish process
+	start
+	restart = "stop;start"
+
+	ping $utc_time -> pong $utc_time
+
+	# cluster only
+	config_contents filename $inline -> compilation messages
+
+	stats [-mr] -> $data
+
+	zero stats
+
+	help
+
+	-----------------------------------------------------------------------
+	CLI (local)
+		import protocol from above
+
+		telnet localhost someport
+		authentication:
+			password $secret
+		secret stored in {/usr/local}/etc/varnish.secret (400 root:wheel)
+
+
+	-----------------------------------------------------------------------
+	HTML (local)
+
+		php/cgi-bin thttpd ?
+		(alternatively direct from C-code.)
+		Everything the CLI can do +
+		stats
+			popen("rrdtool");
+		log view
+
+	-----------------------------------------------------------------------
+	CLI (cluster)
+		import protocol from above, prefix machine/all
+		compound stats
+		accept / deny machine (?)
+		curses if you set termtype
+
+	-----------------------------------------------------------------------
+	HTML (cluster)
+		ditto
+		ditto
+
+		http://clustercontrol/purge?regexp=fslkdjfslkfdj
+			POST with list of regexp
+			authentication ? (IP access list)
+
+	-----------------------------------------------------------------------
+	Mail (cluster)
+
+		pgp signed emails with CLI commands
+
+	-----------------------------------------------------------------------
+	connection varnish -> cluster controller
+
+		Encryption
+			SSL
+		Authentication (?)
+			IP number checks.
+
+		varnish -c clusterid -C mycluster_ctrl.vg.no
+
+	-----------------------------------------------------------------------
+	Filer
+		/usr/local/sbin/varnish
+			contains mgt + varnish process.
+			if -C argument, open SSL to cluster controller.
+			Arguments:
+				-p portnumber
+				-c clusterid at cluster_controller
+				-f config_file
+				-m memory_limit
+				-s kind[,storage-options]
+				-l logfile,logsize
+				-b backend ip...
+				-d debug
+				-u uid
+				-a CLI_port
+
+			KILL SIGTERM	-> suspend, stop
+
+		/usr/local/sbin/varnish_cluster
+			Cluster controller.
+			Use syslog
+
+			Arguments:
+				-f config file
+				-d debug
+				-u uid (?)
+
+		/usr/local/sbin/varnish_logger
+			Logfile processor
+			-i shmemfile
+			-e regexp
+			-o "/var/log/varnish.%Y%m%d.traffic" 
+			-e regexp2
+			-n "/var/log/varnish.%Y%m%d.exception"  (NCSA format)
+			-e regexp3
+			-s syslog_level,syslogfacility
+			-r host:port	send via TCP, prefix hostname
+
+			SIGHUP: reopen all files.
+
+		/usr/local/bin/varnish_cli
+			Command line tool.
+
+		/usr/local/share/varnish/etc/varnish.conf
+			default request + fetch + backend scripts
+
+		/usr/local/share/varnish/etc/rfc2616.conf
+			RFC2616 compliant handling function
+
+		/usr/local/etc/varnish.conf (optional)
+			request + fetch + backend scripts
+
+		/usr/local/share/varnish/etc/varnish.startup
+			default startup sequence
+
+		/usr/local/etc/varnish.startup (optional)
+			startup sequence
+
+		/usr/local/etc/varnish_cluster.conf
+			XXX
+
+		{/usr/local}/etc/varnish.secret
+			CLI password file.
+
+	-----------------------------------------------------------------------
+	varnish.startup
+
+		load config /foo/bar startup_conf
+		switch config startup_conf
+		!mypreloadscript
+		load config /foo/real real_conf
+		switch config real_conf
+		resume
+
+
+The third Varnish Design notes
+-------------------------------
+
+A couple of days later the ideas had gel'ed::
+
+
+	Notes on Varnish
+	----------------
+
+	Collected 2006-02-26 to 2006-03-..
+
+	Poul-Henning Kamp
+
+	-----------------------------------------------------------------------
+
+	Objects available to functions in VCL
+
+		client	# The client
+
+		req	# The request
+
+		obj	# The object from which we satisfy it
+
+		backend	# The chosen supplier
+
+	-----------------------------------------------------------------------
+	Configuration Language
+
+	XXX: declare IP lists ?
+
+	BNF:
+		program:	part
+				| program part
+
+		part:		"sub" function_name compound
+				| "backend" backend_name compound
+
+		compound:	"{" statements "}"
+
+		statements:	/* empty */
+				| statement
+				| statements statement
+
+		statement:	conditional
+				| functioncall
+				| "set" field value
+				| field "=" value
+				| "no_cache"
+				| "finish"
+				| "no_new_cache"
+				| call function_name
+				| fetch
+				| error status_code
+				| error status_code string(message)
+				| switch_config config_id
+				| rewrite field string(match) string(replace)
+
+		conditional:	"if" condition compound elif_parts else_part
+
+		elif_parts:	/* empty */
+				| elif_part
+				| elif_parts elif_part
+
+		elif_part:	"elseif" condition compound
+				| "elsif" condition compound
+				| "else if" condition compound
+
+		else_part:	/* empty */
+				| "else" compound
+
+		functioncal:	"call" function_name
+
+		field:		object
+				field "." variable
+
+		condition:	'(' cond_or ')'
+
+		cond_or:	cond_and
+				| cond_or '||' cond_and
+
+		cond_and:	cond_part
+				| cond_and '&&' cond_part
+
+		cond_part:	'!' cond_part2
+				| cond_part2
+
+		cond_part2:	condition
+				| field(int) '<' number
+				| field(int) '<=' number
+				| field(int) '>' number
+				| field(int) '>=' number
+				| field(int) '=' number
+				| field(int) '!=' number
+				| field(IP)  ~ ip_list
+				| field(string) ~ string(regexp)
+
+	-----------------------------------------------------------------------
+	Sample request policy program
+
+		sub request_policy {
+
+			if (client.ip in 10.0.0.0/8) {
+				no-cache
+				finish
+			}
+
+			if (req.url.host ~ "cnn.no$") {
+				rewrite	s/cnn.no$/vg.no/
+			}
+
+			if (req.url.path ~ "cgi-bin") {
+				no-cache
+			}
+
+			if (req.useragent ~ "spider") {
+				no-new-cache
+			}
+
+			if (backend.response_time > 0.8s) {
+				set req.ttlfactor = 1.5
+			} elseif (backend.response_time > 1.5s) {
+				set req.ttlfactor = 2.0
+			} elseif (backend.response_time > 2.5s) {
+				set req.ttlfactor = 5.0
+			}
+
+			/*
+			 * the program contains no references to
+			 * maxage, s-maxage and expires, so the
+			 * default handling (RFC2616) applies
+			 */
+		}
+
+	-----------------------------------------------------------------------
+	Sample fetch policy program
+
+		sub backends {
+			set backend.vg.ip = {...}
+			set backend.ads.ip = {...}
+			set backend.chat.ip = {...}
+			set backend.chat.timeout = 10s
+			set backend.chat.bandwidth = 2000 MB/s
+			set backend.other.ip = {...}
+		}
+
+		sub vg_backend {
+			set backend.ip = {10.0.0.1-5}
+			set backend.timeout = 4s
+			set backend.bandwidth = 2000Mb/s
+		}
+
+		sub fetch_policy {
+
+			if (req.url.host ~ "/vg.no$/") {
+				set req.backend = vg
+				call vg_backend
+			} else {
+				/* XXX: specify 404 page url ? */
+				error 404
+			}
+
+			if (backend.response_time > 2.0s) {
+				if (req.url.path ~ "/landbrugspriser/") {
+					error 504
+				}
+			}
+			fetch
+			if (backend.down) {
+				if (obj.exist) {
+					set obj.ttl += 10m
+					finish
+				}
+				switch_config ohhshit
+			}
+			if (obj.result == 404) {
+				error 300 "http://www.vg.no"
+			}
+			if (obj.result != 200) {
+				finish
+			}
+			if (obj.size > 256k) {
+				no-cache
+			} else if (obj.size > 32k && obj.ttl < 2m) {
+				obj.tll = 5m				
+			}
+			if (backend.response_time > 2.0s) {
+				set ttl *= 2.0
+			}
+		}
+
+		sub prefetch_policy {
+
+			if (obj.usage < 10 && obj.ttl < 5m) {
+				fetch
+			}
+		}
+
+	-----------------------------------------------------------------------
+	Purging
+
+	When a purge request comes in, the regexp is tagged with the next
+	generation number and added to the tail of the list of purge regexps.
+
+	Before a sender transmits an object, it is checked against any
+	purge-regexps which have higher generation number than the object
+	and if it matches the request is sent to a fetcher and the object
+	purged.
+
+	If there were purge regexps with higher generation to match, but
+	they didn't match, the object is tagged with the current generation
+	number and moved to the tail of the list.
+
+	Otherwise, the object does not change generation number and is
+	not moved on the generation list.
+
+	New Objects are tagged with the current generation number and put
+	at the tail of the list.
+
+	Objects are removed from the generation list when deleted.
+
+	When a purge object has a lower generation number than the first
+	object on the generation list, the purge object has been completed
+	and will be removed.  A log entry is written with number of compares
+	and number of hits.
+		
+	-----------------------------------------------------------------------
+	Random notes
+
+		swap backed storage
+
+		slowstart by config-flipping
+			start-config has peer servers as backend
+			once hitrate goes above limit, management process
+			flips config to 'real' config.
+
+		stat-object
+			always URL, not regexp
+
+		management + varnish process in one binary, comms via pipe
+
+		Change from config with long expiry to short expiry, how
+		does the ttl drop ?  (config sequence number invalidates
+		all calculated/modified attributes.)
+
+		Mgt process holds copy of acceptor socket ->  Restart without
+		lost client requests.
+
+		BW limit per client IP: create shortlived object (<4sec)
+		to hold status.  Enforce limits by delaying responses.
+
+
+	-----------------------------------------------------------------------
+	Source structure
+
+
+		libvarnish
+			library with interface facilities, for instance
+			functions to open&read shmem log
+
+		varnish
+			varnish sources in three classes
+
+	-----------------------------------------------------------------------
+	protocol cluster/mgt/varnish
+
+	object_query url -> TTL, size, checksum
+	{purge,invalidate} regexp
+	object_status url -> object metadata
+
+	load_config filename
+	switch_config configname
+	list_configs
+	unload_config
+
+	freeze 	# stop the clock, freezes the object store
+	thaw
+
+	suspend	# stop acceptor accepting new requests
+	resume
+
+	stop	# forced stop (exits) varnish process
+	start
+	restart = "stop;start"
+
+	ping $utc_time -> pong $utc_time
+
+	# cluster only
+	config_contents filename $inline -> compilation messages
+
+	stats [-mr] -> $data
+
+	zero stats
+
+	help
+
+	-----------------------------------------------------------------------
+	CLI (local)
+		import protocol from above
+
+		telnet localhost someport
+		authentication:
+			password $secret
+		secret stored in {/usr/local}/etc/varnish.secret (400 root:wheel)
+
+
+	-----------------------------------------------------------------------
+	HTML (local)
+
+		php/cgi-bin thttpd ?
+		(alternatively direct from C-code.)
+		Everything the CLI can do +
+		stats
+			popen("rrdtool");
+		log view
+
+	-----------------------------------------------------------------------
+	CLI (cluster)
+		import protocol from above, prefix machine/all
+		compound stats
+		accept / deny machine (?)
+		curses if you set termtype
+
+	-----------------------------------------------------------------------
+	HTML (cluster)
+		ditto
+		ditto
+
+		http://clustercontrol/purge?regexp=fslkdjfslkfdj
+			POST with list of regexp
+			authentication ? (IP access list)
+
+	-----------------------------------------------------------------------
+	Mail (cluster)
+
+		pgp signed emails with CLI commands
+
+	-----------------------------------------------------------------------
+	connection varnish -> cluster controller
+
+		Encryption
+			SSL
+		Authentication (?)
+			IP number checks.
+
+		varnish -c clusterid -C mycluster_ctrl.vg.no
+
+	-----------------------------------------------------------------------
+	Filer
+		/usr/local/sbin/varnish
+			contains mgt + varnish process.
+			if -C argument, open SSL to cluster controller.
+			Arguments:
+				-p portnumber
+				-c clusterid at cluster_controller
+				-f config_file
+				-m memory_limit
+				-s kind[,storage-options]
+				-l logfile,logsize
+				-b backend ip...
+				-d debug
+				-u uid
+				-a CLI_port
+
+			KILL SIGTERM	-> suspend, stop
+
+		/usr/local/sbin/varnish_cluster
+			Cluster controller.
+			Use syslog
+
+			Arguments:
+				-f config file
+				-d debug
+				-u uid (?)
+
+		/usr/local/sbin/varnish_logger
+			Logfile processor
+			-i shmemfile
+			-e regexp
+			-o "/var/log/varnish.%Y%m%d.traffic" 
+			-e regexp2
+			-n "/var/log/varnish.%Y%m%d.exception"  (NCSA format)
+			-e regexp3
+			-s syslog_level,syslogfacility
+			-r host:port	send via TCP, prefix hostname
+
+			SIGHUP: reopen all files.
+
+		/usr/local/bin/varnish_cli
+			Command line tool.
+
+		/usr/local/share/varnish/etc/varnish.conf
+			default request + fetch + backend scripts
+
+		/usr/local/share/varnish/etc/rfc2616.conf
+			RFC2616 compliant handling function
+
+		/usr/local/etc/varnish.conf (optional)
+			request + fetch + backend scripts
+
+		/usr/local/share/varnish/etc/varnish.startup
+			default startup sequence
+
+		/usr/local/etc/varnish.startup (optional)
+			startup sequence
+
+		/usr/local/etc/varnish_cluster.conf
+			XXX
+
+		{/usr/local}/etc/varnish.secret
+			CLI password file.
+
+	-----------------------------------------------------------------------
+	varnish.startup
+
+		load config /foo/bar startup_conf
+		switch config startup_conf
+		!mypreloadscript
+		load config /foo/real real_conf
+		switch config real_conf
+		resume
+
+Fourth Varnish Design Note
+--------------------------
+
+You'd think we'd be cookin' with gas now, and indeed we were, but now
+all the difficult details started to raise ugly questions, and it
+has never stopped since::
+
+	Questions:
+
+	*  Which "Host:" do we put in the request to the backend ?
+
+	      The one we got from the client ?
+
+	      The ip/dns-name of the backend ?
+
+	      Configurable in VCL backend declaration ?
+
+	      (test with www.ing.dk)
+
+	*  Construction of headers for queries to backend ?
+
+	      How much do we take from client headers, how much do we make up ?
+
+	      Some sites discriminate contents based on User-Agent header.
+		 (test with www.krak.dk/www.rs-components.dk)
+
+	      Cookies
+
+	*  Mapping of headers from backend reply to the reply to client
+
+	      Which fields come from the backend ?
+
+	      Which fields are made up on the spot ? (expiry time ?)
+
+	      (Static header fields can be prepended to contents in storage)
+
+
+	*  3xx replies from the backend
+
+	      Does varnish follow a redirection or do we pass it to the client ?
+
+	      Do we cache 3xx replies ?
+
+
+The first live traffic
+----------------------
+
+The final bit of history I want to share is the IRC log from the
+first time tried to put real live traffic through Varnish.
+
+The language is interscandinavian, but I think non-vikings can get
+still get the drift::
+
+	**** BEGIN LOGGING AT Thu Jul  6 12:36:48 2006
+
+	Jul 06 12:36:48 *	Now talking on #varnish
+	Jul 06 12:36:48 *	EvilDES gives channel operator status to andersb
+	Jul 06 12:36:53 *	EvilDES gives channel operator status to phk
+	Jul 06 12:36:53 <andersb>	hehe
+	Jul 06 12:36:56 <EvilDES>	s?nn
+	Jul 06 12:37:00 <andersb>	Jepps, er dere klare?
+	Jul 06 12:37:08 <phk>	Jeg har varnish oppe og k?re med leonora som backend.
+	Jul 06 12:37:12 *	EvilDES has changed the topic to: Live testing in progress!
+	Jul 06 12:37:16 *	EvilDES sets mode +t #varnish
+	Jul 06 12:37:19 <andersb>	Da setter jeg p? trafikk
+	Jul 06 12:37:36 <phk>	andersb: kan du starte med bare at give us trafiik i 10 sekunder eller s? ?
+	Jul 06 12:37:49 *	edward (edward at f95.linpro.no) has joined #varnish
+	Jul 06 12:38:32 <andersb>	hmm, f?rst m? jeg f? trafikk dit.
+	Jul 06 12:38:55 <andersb>	Har noe kommet? Eller har det blitt suprt etter /systemmeldinger/h.html som er helsefilen?
+	Jul 06 12:39:10 <andersb>	s/suprt/spurt/
+	Jul 06 12:39:41 <EvilDES>	ser ingenting
+	Jul 06 12:39:45 <phk>	jeg har ikke set noget endnu...
+	Jul 06 12:40:35 <phk>	den pr?ver p? port 80
+	Jul 06 12:41:24 <andersb>	okay..
+	Jul 06 12:41:31 <EvilDES>	kan vi ikke bare kj?re varnishd p? port 80?
+	Jul 06 12:41:46 <phk>	ok, jeg ville bare helst ikke k?re som root.
+	Jul 06 12:41:47 <andersb>	Pr?ver den noe annet n??
+	Jul 06 12:41:59 <phk>	nej stadig 80.
+	Jul 06 12:42:03 <phk>	Jeg starter varnishd som root
+	Jul 06 12:42:08 <EvilDES>	nei, vent
+	Jul 06 12:42:08 <andersb>	Topp
+	Jul 06 12:42:11 <andersb>	okay
+	Jul 06 12:42:15 <andersb>	kom det 8080 n??
+	Jul 06 12:42:18 <EvilDES>	sysctl reserved_port
+	Jul 06 12:43:04 <andersb>	okay? F?r dere 8080 trafikk n??
+	Jul 06 12:43:08 <EvilDES>	sysctl net.inet.ip.portrange.reservedhigh=79
+	Jul 06 12:44:41 <andersb>	Okay, avventer om vi skal kj?re 8080 eller 80.
+	Jul 06 12:45:56 <EvilDES>	starter den p? port 80 som root
+	Jul 06 12:46:01 <phk>	den k?rer nu
+	Jul 06 12:46:01 <andersb>	Okay, vi har funnet ut at m?ten jeg satte 8080 p? i lastbalanserern var feil.
+	Jul 06 12:46:07 <andersb>	okay p? 80?
+	Jul 06 12:46:12 <phk>	vi k?rer
+	Jul 06 12:46:14 <EvilDES>	ja, masse trafikk
+	Jul 06 12:46:29 <phk>	omtrent 100 req/sec
+	Jul 06 12:46:37 <phk>	and we're dead...
+	Jul 06 12:46:40 <EvilDES>	stopp!
+	Jul 06 12:46:58 <andersb>	den stopper automatisk.
+	Jul 06 12:47:04 <andersb>	Vi kan bare kj?re det slik.
+	Jul 06 12:47:06 <EvilDES>	tok noen sekunder
+	Jul 06 12:47:20 <andersb>	Npr den begynner svar p? 80 s? vil lastbalanserern finne den fort og sende trafikk.
+	Jul 06 12:47:41 <EvilDES>	ca 1500 connection requests kom inn f?r den sluttet ? sende oss trafikk
+	Jul 06 12:47:49 <EvilDES>	alts?, 1500 etter at varnishd d?de
+	Jul 06 12:48:02 <andersb>	tror det er en god nok m?te ? gj?re det p?. S? slipper vi ? configge hele tiden.
+	Jul 06 12:48:07 <EvilDES>	greit
+	Jul 06 12:48:11 <EvilDES>	det er dine lesere :)
+	Jul 06 12:48:19 <andersb>	ja :)
+	Jul 06 12:48:35 <andersb>	kan sette ned retry raten litt.
+	Jul 06 12:49:15 <andersb>	>> AS3408-2 VG Nett - Real server 21 # retry
+	Jul 06 12:49:16 <andersb>	Current number of failure retries: 4
+	Jul 06 12:49:16 <andersb>	Enter new number of failure retries [1-63]: 1
+	Jul 06 12:49:33 <andersb>	^^ before de decalres dead
+	Jul 06 12:49:41 <andersb>	he declairs :)
+	Jul 06 12:51:45 <phk>	I've saved the core, lets try again for another shot.
+	Jul 06 12:52:09 <andersb>	sure :)
+	Jul 06 12:52:34 <andersb>	When you start port 80 loadbalancer will send 8 req's for h.html then start gicing traficc
+	Jul 06 12:53:00 <andersb>	^^ Microsoft keyboard
+	Jul 06 12:53:09 <phk>	ok, jeg starter
+	Jul 06 12:53:10 <EvilDES>	you need to get a Linux keyboard
+	Jul 06 12:53:16 <andersb>	Yeah :)
+	Jul 06 12:53:18 <EvilDES>	woo!
+	Jul 06 12:53:21 <phk>	boom.
+	Jul 06 12:53:25 <EvilDES>	oops
+	Jul 06 12:53:35 <EvilDES>	18 connections, 77 requests
+	Jul 06 12:53:40 <EvilDES>	that didn't last long...
+	Jul 06 12:54:41 <andersb>	longer than me :) *rude joke
+	Jul 06 12:55:04 <phk>	bewm
+	Jul 06 12:55:22 <andersb>	can I follow a log?
+	Jul 06 12:55:39 <andersb>	with: lt-varnishlog ?
+	Jul 06 12:56:27 <phk>	samme fejl
+	Jul 06 12:56:38 <phk>	andersb: jeg gemmer logfilerne
+	Jul 06 12:57:00 <phk>	bewm
+	Jul 06 12:57:13 <andersb>	phk: Jepp, men for min egen del for ? se n?r dere skrur p? etc. Da l?rer jeg loadbalancer ting.
+	Jul 06 12:57:51 <phk>	ok, samme fejl igen.
+	Jul 06 12:58:02 <phk>	jeg foresl?r vi holder en lille pause mens jeg debugger.
+	Jul 06 12:58:09 <andersb>	sure.
+	Jul 06 12:58:16 <EvilDES>	andersb: cd ~varnish/varnish/trunk/varnish-cache/bin/varnishlog
+	Jul 06 12:58:21 <EvilDES>	andersb: ./varnishlog -o
+	Jul 06 12:58:37 <EvilDES>	andersb: cd ~varnish/varnish/trunk/varnish-cache/bin/varnishstat
+	Jul 06 12:58:43 <EvilDES>	andersb: ./varnishstat -c
+	Jul 06 12:58:44 <phk>	eller ./varnislog -r _vlog3 -o | less
+	Jul 06 13:00:02 <andersb>	Jeg g?r meg en kort tur. Straks tilbake.
+	Jul 06 13:01:27 <phk>	vi k?rer igen
+	Jul 06 13:02:31 <phk>	2k requests
+	Jul 06 13:02:57 <phk>	3k
+	Jul 06 13:03:39 <phk>	5k
+	Jul 06 13:03:55 <EvilDES>	ser veldig bra ut
+	Jul 06 13:04:06 <EvilDES>	hit rate > 93%
+	Jul 06 13:04:13 <EvilDES>	95%
+	Jul 06 13:05:14 <phk>	800 objects
+	Jul 06 13:05:32 <EvilDES>	load 0.28
+	Jul 06 13:05:37 <EvilDES>	0.22
+	Jul 06 13:05:52 <EvilDES>	CPU 98.9% idle :)
+	Jul 06 13:06:12 <phk>	4-5 Mbit/sec
+	Jul 06 13:06:42 <andersb>	nice :)
+	Jul 06 13:06:49 <andersb>	vi kj?rer til det krasjer?
+	Jul 06 13:06:58 <phk>	jep
+	Jul 06 13:07:05 <phk>	du m? gerne ?bne lidt mere
+	Jul 06 13:07:20 <andersb>	okay
+	Jul 06 13:07:41 <andersb>	3 ganger mer...
+	Jul 06 13:08:04 <andersb>	si fra n?r dere vil ha mer.
+	Jul 06 13:08:24 <phk>	vi gir den lige et par minutter p? det her niveau
+	Jul 06 13:09:17 <phk>	bewm
+	Jul 06 13:09:31 <EvilDES>	        3351        0.00 Client connections accepted
+	Jul 06 13:09:31 <EvilDES>	       23159        0.00 Client requests received
+	Jul 06 13:09:31 <EvilDES>	       21505        0.00 Cache hits
+	Jul 06 13:09:31 <EvilDES>	        1652        0.00 Cache misses
+	Jul 06 13:10:17 <phk>	k?rer igen
+	Jul 06 13:10:19 <EvilDES>	here we go again
+	Jul 06 13:11:06 <phk>	20mbit/sec
+	Jul 06 13:11:09 <phk>	100 req/sec
+	Jul 06 13:12:30 <andersb>	nice :)
+	Jul 06 13:12:46 <andersb>	det er gode tall, og jeg skal fortelle dere hvorfor senere
+	Jul 06 13:12:49 <phk>	steady 6-8 mbit/sec
+	Jul 06 13:12:52 <andersb>	okay.
+	Jul 06 13:13:00 <phk>	ca 50 req/sec
+	Jul 06 13:13:04 <EvilDES>	skal vi ?ke?
+	Jul 06 13:13:14 <phk>	ja, giv den det dobbelte hvis du kan
+	Jul 06 13:13:19 <andersb>	vi startet med 1 -> 3 -> ?
+	Jul 06 13:13:22 <phk>	6
+	Jul 06 13:13:23 <andersb>	6
+	Jul 06 13:13:34 <andersb>	done
+	Jul 06 13:13:42 <andersb>	den hopper opp graceful.
+	Jul 06 13:13:54 <EvilDES>	boom
+	Jul 06 13:14:06 <andersb>	:)
+	Jul 06 13:14:11 <EvilDES>	men ingen ytelsesproblemer
+	Jul 06 13:14:19 <EvilDES>	bare bugs i requestparsering
+	Jul 06 13:14:20 <phk>	k?rer igen
+	Jul 06 13:14:26 <phk>	bewm
+	Jul 06 13:14:31 <phk>	ok, vi pauser lige...
+	Jul 06 13:17:40 <phk>	jeg har et problem med "pass" requests, det skal jeg lige have fundet inden vi g?r videre.
+	Jul 06 13:18:51 <andersb>	Sure.
+	Jul 06 13:28:50 <phk>	ok, vi pr?ver igen
+	Jul 06 13:29:09 <phk>	bewm
+	Jul 06 13:29:35 <phk>	more debugging
+	Jul 06 13:33:56 <phk>	OK, found the/one pass-mode bug
+	Jul 06 13:33:58 <phk>	trying again
+	Jul 06 13:35:23 <phk>	150 req/s 24mbit/s, still alive
+	Jul 06 13:37:02 <EvilDES>	andersb: tror du du klarer ? komme deg hit til foredraget, eller er du helt ?delagt?
+	Jul 06 13:37:06 <phk>	andersb: giv den 50% mere trafik
+	Jul 06 13:39:46 <andersb>	mer trafikk
+	Jul 06 13:39:56 <andersb>	EvilDES: Nei :(( Men Stein fra VG Nett kommer.
+	Jul 06 13:41:25 <EvilDES>	btw, har du noen data om hva load balanceren synes om varnish?
+	Jul 06 13:41:50 <EvilDES>	jeg regner med at den f?lger med litt p? hvor god jobb vi gj?r
+	Jul 06 13:43:10 <phk>	Jeg genstarter lige med flere workerthreads...
+	Jul 06 13:43:43 <phk>	jeg tror 20 workerthreads var for lidt nu...
+	Jul 06 13:43:47 <phk>	nu har den 220
+	Jul 06 13:44:40 <EvilDES>	        2976      107.89 Client connections accepted
+	Jul 06 13:44:41 <EvilDES>	       10748      409.57 Client requests received
+	Jul 06 13:44:41 <EvilDES>	        9915      389.59 Cache hits
+	Jul 06 13:45:13 <EvilDES>	det var alts? 400 i sekundet :)
+	Jul 06 13:45:45 <phk>	og ingen indlysende fejl p? www.vg.no siden :-)
+	Jul 06 13:45:54 <phk>	bewm
+	Jul 06 13:47:16 <EvilDES>	andersb: hvor stor andel av trafikken hadde vi n??
+	Jul 06 13:48:06 <EvilDES>	alts?, vekt i load balanceren i forhold til totalen
+	Jul 06 13:49:20 <phk>	ok, kun 120 threads s?...
+	Jul 06 13:50:48 <andersb>	9
+	Jul 06 13:52:45 <phk>	andersb: 9 -> 12 ?
+	Jul 06 13:52:48 <EvilDES>	andersb: 9 til varnish, men hvor mye er den totale vekten?
+	Jul 06 13:52:58 <EvilDES>	har vi 1%? 5%? 10%?
+	Jul 06 13:54:37 <EvilDES>	n? passerte vi nettopp 50000 requests uten kr?sj
+	Jul 06 13:55:36 <phk>	maskinen laver ingenting...  98.5% idle
+	Jul 06 13:56:21 <andersb>	12 maskiner med weight 20
+	Jul 06 13:56:26 <andersb>	1 med weight 40
+	Jul 06 13:56:29 <andersb>	varnish med 9
+	Jul 06 13:57:01 <andersb>	si fra n?r dere vil ha mer trafikk.
+	Jul 06 13:57:02 <phk>	9/289 = 3.1%
+	Jul 06 13:57:12 <phk>	andersb: giv den 15
+	Jul 06 13:57:44 <andersb>	gjort
+	Jul 06 13:59:43 <andersb>	dette er morro. Jeg m? si det.
+	Jul 06 14:00:27 <phk>	20-23 Mbit/sec steady, 200 req/sec, 92.9% idle
+	Jul 06 14:00:30 <phk>	bewm
+	Jul 06 14:00:46 <EvilDES>	OK
+	Jul 06 14:00:57 <EvilDES>	jeg tror vi kan sl? fast at ytelsen er som den skal v?re
+	Jul 06 14:01:33 <EvilDES>	det er en del bugs, men de b?r det g? an ? fikse.
+	Jul 06 14:01:34 <andersb>	Jepp :) Det s? pent ut...
+	Jul 06 14:01:53 <phk>	jeg tror ikke vi har set skyggen af hvad Varnish kan yde endnu...
+	Jul 06 14:01:53 <EvilDES>	andersb: hvordan ligger vi an i forhold til Squid?
+	Jul 06 14:01:58 <andersb>	pent :)
+	Jul 06 14:02:13 <andersb>	Jeg har ikke f?tt SNMP opp p? dene boksen, jeg burde grafe det...
+	Jul 06 14:02:23 <EvilDES>	snmp kj?rer p? c21
+	Jul 06 14:02:33 <EvilDES>	tror agero satte det opp
+	Jul 06 14:02:36 <EvilDES>	aagero
+	Jul 06 14:02:38 <andersb>	Ja, men jeg har ikke mal i cacti for bsnmpd
+	Jul 06 14:02:43 <EvilDES>	ah, ok
+	Jul 06 14:03:03 <EvilDES>	men den burde st?tte standard v2 mib?
+	Jul 06 14:03:26 <andersb>	det er ikke protocoll feil :)
+	Jul 06 14:03:42 <andersb>	Hva er byte hitratio forresetn?
+	Jul 06 14:03:52 <EvilDES>	det tror jeg ikke vi m?ler
+	Jul 06 14:03:55 <EvilDES>	enda
+	Jul 06 14:03:59 <phk>	andersb: den har jeg ikke stats p? endnu.
+	Jul 06 14:04:22 <phk>	ok, forrige crash ligner en 4k+ HTTP header...
+	Jul 06 14:04:27 <phk>	(eller en kodefejl)
+	Jul 06 14:06:03 <phk>	andersb: pr?v at ?ge vores andel til 20
+	Jul 06 14:06:26 <EvilDES>	hvilken vekt har hver av de andre cachene?
+	Jul 06 14:06:49 <phk>	20 og en med 40
+	Jul 06 14:07:50 <andersb>	gjort
+	Jul 06 14:08:59 <phk>	440 req/s 43mbit/s
+	Jul 06 14:09:17 <phk>	bewm
+	Jul 06 14:09:18 <EvilDES>	bewm
+	Jul 06 14:10:30 <EvilDES>	oj
+	Jul 06 14:10:39 <EvilDES>	vi var oppe over 800 req/s et ?yeblikk
+	Jul 06 14:10:46 <phk>	60mbit/sec
+	Jul 06 14:10:52 <phk>	og 90% idle :-)
+	Jul 06 14:10:59 <EvilDES>	ingen swapping
+	Jul 06 14:11:58 <EvilDES>	og vi bruker nesten ikke noe minne - 3 GB ledig fysisk RAM
+	Jul 06 14:13:02 <phk>	ca 60 syscall / req
+	Jul 06 14:14:31 <andersb>	nice :)
+	Jul 06 14:14:58 <phk>	andersb: pr?v at give os 40
+	Jul 06 14:17:26 <andersb>	gjort
+	Jul 06 14:18:17 <phk>	det ligner at trafikken falder her sidst p? eftermiddagen...
+	Jul 06 14:19:07 <andersb>	ja :)
+	Jul 06 14:19:43 <phk>	andersb: s? skal vi nok ikke ?ge mere, nu n?rmer vi os hvad 100Mbit ethernet kan klare.
+	Jul 06 14:19:58 <andersb>	bra :)
+	Jul 06 14:20:36 <phk>	42mbit/s steady
+	Jul 06 14:20:59 <EvilDES>	40 av 320?
+	Jul 06 14:21:06 <EvilDES>	12,5%
+	Jul 06 14:21:43 *	nicholas (nicholas at nfsd.linpro.no) has joined #varnish
+	Jul 06 14:22:00 <phk>	det der cluster-noget bliver der da ikke brug for n?r vi har 87% idle
+	Jul 06 14:23:05 <andersb>	hehe :)
+	Jul 06 14:24:38 <andersb>	skal stille de andre ned litt for 48 er max
+	Jul 06 14:24:57 <phk>	jeg tror ikke vi skal g? h?jere f?r vi har gigE
+	Jul 06 14:25:14 <andersb>	4-5MB/s
+	Jul 06 14:25:32 <andersb>	lastbalanserer backer off p? 100 Mbit
+	Jul 06 14:25:35 <andersb>	:)
+	Jul 06 14:25:42 <andersb>	S? vi kan kj?re nesten til taket.
+	Jul 06 14:26:01 <andersb>	hvis det har noe poeng.
+	Jul 06 14:26:09 <andersb>	crash :)
+	Jul 06 14:27:33 <phk>	bewm
+	Jul 06 14:29:08 <andersb>	Stilt inn alle p? weight 5
+	Jul 06 14:29:17 <andersb>	bortsett fra 1 som er 10
+	Jul 06 14:29:20 <andersb>	varnish er 5
+	Jul 06 14:29:24 <phk>	s? giv os 20
+	Jul 06 14:29:51 <andersb>	gjort
+	Jul 06 14:30:58 <phk>	vi f?r kun 300 req/s
+	Jul 06 14:31:04 <phk>	Ahh der skete noget.
+	Jul 06 14:32:41 <phk>	ok, ved denne last bliver backend connections et problem, jeg har set dns fejl og connection refused
+	Jul 06 14:33:10 <phk>	dns fejl
+	Jul 06 14:33:21 <andersb>	okay, pek den mot 10.0.2.5
+	Jul 06 14:33:28 <andersb>	det er layer 2 squid cache
+	Jul 06 14:33:35 <andersb>	morro ? teste det og.
+	Jul 06 14:33:54 <phk>	det g?r jeg n?ste gang den falder
+	Jul 06 14:34:48 <phk>	jeg kunne jo ogs? bare give leonors IP# istedet... men nu k?rer vi imod squid
+	Jul 06 14:36:05 <andersb>	ja, gi leonora IP det er sikkert bedre. Eller det kan jo v?re fint ? teste mot squid og :)
+	Jul 06 14:39:04 <phk>	nu k?rer vi med leonora's IP#
+	Jul 06 14:39:33 <phk>	nu k?rer vi med leonora's *rigtige* IP#
+	Jul 06 14:41:20 <phk>	Nu er vi f?rdige med det her 100Mbit/s ethernet, kan vi f? et til ?  :-)
+	Jul 06 14:41:42 <andersb>	lol :)
+	Jul 06 14:42:00 <andersb>	For ? si det slik. Det tar ikke mange dagene f?r Gig switch er bestilt :)
+	Jul 06 14:43:05 <phk>	bewm
+	Jul 06 14:43:13 <phk>	ok, jeg synes vi skal stoppe her.
+	Jul 06 14:43:41 <EvilDES>	jepp, foredrag om 15 min
+	Jul 06 14:43:57 <andersb>	jepp
+	Jul 06 14:44:23 <andersb>	disabled server
+	Jul 06 14:45:29 <EvilDES>	dette har v?rt en veldig bra dag.
+	Jul 06 14:45:49 <EvilDES>	hva skal vi finne p? i morgen? skifte ut hele Squid-riggen med en enkelt Varnish-boks? ;)
+	Jul 06 14:45:53 <andersb>	lol
+	Jul 06 14:46:15 *	EvilDES m? begynne ? sette i stand til foredraget
+	Jul 06 14:46:17 <andersb>	da m? jeg har Gig switch. Eller s? kan v?re b?re med en HP maskin ? koble rett p? lastbal :)
+	Jul 06 14:46:22 <phk>	kan vi ikke n?jes med en halv varnish box ?
+	Jul 06 14:46:41 <EvilDES>	vi m? ha begge halvdeler for failover
+	Jul 06 14:47:01 <andersb>	:)
+	Jul 06 14:47:14 <andersb>	kan faile tilbake til de andre.
+	Jul 06 14:47:25 <andersb>	Jeg klarer ikke holde meg hjemme.
+	Jul 06 14:47:33 <andersb>	 Jeg kommer oppover om litt :)
+	Jul 06 14:47:39 <andersb>	Ringer p?.
+	Jul 06 14:48:19 <andersb>	m? g? en tur n? :)
+	Jul 06 14:48:29 *	andersb has quit (BitchX: no additives or preservatives)
+	Jul 06 14:49:44 <EvilDES>	http://www.des.no/varnish/
+	**** ENDING LOGGING AT Thu Jul  6 14:52:04 2006
+
+*phk*
+
diff --git a/doc/sphinx/phk/index.rst b/doc/sphinx/phk/index.rst
index 60a3375..65108ed 100644
--- a/doc/sphinx/phk/index.rst
+++ b/doc/sphinx/phk/index.rst
@@ -8,6 +8,7 @@ You may or may not want to know what Poul-Henning thinks.
 .. toctree::
 	:maxdepth: 1
 
+	firstdesign.rst
 	10goingon50.rst
 	brinch-hansens-arrows.rst
 	ssl_again.rst



From nils.goroll at uplex.de  Mon Feb 22 13:29:11 2016
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 22 Feb 2016 14:29:11 +0100
Subject: [master] b6be5d0 Error handling for V1F_Setup_Fetch
Message-ID: <E1aXqY3-0004h2-2R@project.varnish-software.com>


commit b6be5d0fefb788e665365987ef57b8072a61bd98
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Feb 17 15:06:02 2016 +0100

    Error handling for V1F_Setup_Fetch

diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c
index 5879518..8b11386 100644
--- a/bin/varnishd/cache/cache_backend.c
+++ b/bin/varnishd/cache/cache_backend.c
@@ -247,8 +247,7 @@ vbe_dir_getbody(const struct director *d, struct worker *wrk,
 	CHECK_OBJ_NOTNULL(bo->vfc, VFP_CTX_MAGIC);
 	CHECK_OBJ_NOTNULL(bo->htc, HTTP_CONN_MAGIC);
 
-	V1F_Setup_Fetch(bo->vfc, bo->htc);
-	return (0);
+	return (V1F_Setup_Fetch(bo->vfc, bo->htc));
 }
 
 static const struct suckaddr * __match_proto__(vdi_getip_f)
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 4fc5359..5b15581 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -655,8 +655,14 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 	    http_GetHdr(bo->beresp, H_ETag, &p)))
 		ObjSetFlag(bo->wrk, bo->fetch_objcore, OF_IMSCAND, 1);
 
-	if (bo->htc->body_status != BS_NONE)
-		AZ(VDI_GetBody(bo->wrk, bo));
+	if (bo->htc->body_status != BS_NONE &&
+	    VDI_GetBody(bo->wrk, bo) != 0) {
+		(void)VFP_Error(bo->vfc,
+		    "GetBody failed - workspace_backend overflow?");
+		bo->htc->doclose = SC_OVERLOAD;
+		VDI_Finish(bo->wrk, bo);
+		return (F_STP_ERROR);
+	}
 
 	assert(bo->fetch_objcore->boc->refcount >= 1);
 
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 25f4e33..733b21d 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -688,9 +688,15 @@ cnt_recv(struct worker *wrk, struct req *req)
 	VFP_Setup(req->htc->vfc);
 	req->htc->vfc->http = req->http;
 	req->htc->vfc->wrk = wrk;
-	if (req->transport->req_body != NULL)
+	if (req->transport->req_body != NULL) {
 		req->transport->req_body(req);
 
+		if (req->req_body_status == REQ_BODY_FAIL) {
+			req->doclose = SC_OVERLOAD;
+			return (REQ_FSM_DONE);
+		}
+	}
+
 	VCL_recv_method(req->vcl, wrk, req, NULL, NULL);
 
 	/* Attempts to cache req.body may fail */
diff --git a/bin/varnishd/http1/cache_http1.h b/bin/varnishd/http1/cache_http1.h
index 07f2d63..bd74ed4 100644
--- a/bin/varnishd/http1/cache_http1.h
+++ b/bin/varnishd/http1/cache_http1.h
@@ -31,7 +31,7 @@
 int V1F_SendReq(struct worker *, struct busyobj *, uint64_t *ctr,
     int onlycached);
 int V1F_FetchRespHdr(struct busyobj *);
-void V1F_Setup_Fetch(struct vfp_ctx *vfc, struct http_conn *htc);
+int V1F_Setup_Fetch(struct vfp_ctx *vfc, struct http_conn *htc);
 
 /* cache_http1_fsm.c [HTTP1] */
 void HTTP1_Session(struct worker *, struct req *);
diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c
index f30c3b0..073684e 100644
--- a/bin/varnishd/http1/cache_http1_fsm.c
+++ b/bin/varnishd/http1/cache_http1_fsm.c
@@ -146,7 +146,8 @@ http1_req_body(struct req *req)
 	case BS_EOF:
 	case BS_LENGTH:
 	case BS_CHUNKED:
-		V1F_Setup_Fetch(req->htc->vfc, req->htc);
+		if (V1F_Setup_Fetch(req->htc->vfc, req->htc) != 0)
+			req->req_body_status = REQ_BODY_FAIL;
 		break;
 	default:
 		break;
diff --git a/bin/varnishd/http1/cache_http1_vfp.c b/bin/varnishd/http1/cache_http1_vfp.c
index 1c11384..b836cd3 100644
--- a/bin/varnishd/http1/cache_http1_vfp.c
+++ b/bin/varnishd/http1/cache_http1_vfp.c
@@ -260,7 +260,7 @@ static const struct vfp v1f_eof = {
 /*--------------------------------------------------------------------
  */
 
-void
+int
 V1F_Setup_Fetch(struct vfp_ctx *vfc, struct http_conn *htc)
 {
 	struct vfp_entry *vfe;
@@ -272,19 +272,22 @@ V1F_Setup_Fetch(struct vfp_ctx *vfc, struct http_conn *htc)
 	case BS_EOF:
 		assert(htc->content_length == -1);
 		vfe = VFP_Push(vfc, &v1f_eof, 0);
-		XXXAN(vfe);
+		if (vfe == NULL)
+			return (ENOSPC);
 		vfe->priv2 = 0;
 		break;
 	case BS_LENGTH:
 		assert(htc->content_length > 0);
 		vfe = VFP_Push(vfc, &v1f_straight, 0);
-		XXXAN(vfe);
+		if (vfe == NULL)
+			return (ENOSPC);
 		vfe->priv2 = htc->content_length;
 		break;
 	case BS_CHUNKED:
 		assert(htc->content_length == -1);
 		vfe = VFP_Push(vfc, &v1f_chunked, 0);
-		XXXAN(vfe);
+		if (vfe == NULL)
+			return (ENOSPC);
 		vfe->priv2 = -1;
 		break;
 	default:
@@ -292,4 +295,5 @@ V1F_Setup_Fetch(struct vfp_ctx *vfc, struct http_conn *htc)
 		break;
 	}
 	vfe->priv1 = htc;
+	return 0;
 }



From dridi.boukelmoune at gmail.com  Mon Feb 22 15:26:52 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 22 Feb 2016 16:26:52 +0100
Subject: [master] 0a05e52 Retire ${varnish*} macros in varnishtest
Message-ID: <E1aXsNw-0000sW-2D@project.varnish-software.com>


commit 0a05e52c9c84ac4eca5d5049bf02954d1ec1dba3
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Sun Feb 21 19:09:37 2016 +0100

    Retire ${varnish*} macros in varnishtest
    
    Instead, varnishtest users should set up a proper PATH, just like
    `varnishtest -i` does to use binaries from the build tree.

diff --git a/bin/varnishtest/tests/a00009.vtc b/bin/varnishtest/tests/a00009.vtc
index b2b63dc..b62483d 100644
--- a/bin/varnishtest/tests/a00009.vtc
+++ b/bin/varnishtest/tests/a00009.vtc
@@ -1,11 +1,11 @@
 varnishtest "Code coverage of VCL compiler and RSTdump etc"
 
-shell "${varnishd} -b 127.0.0.1:80 -C -n ${tmpdir} > /dev/null 2>&1"
-shell "${varnishd} -x dumprstparam > /dev/null 2>&1"
-shell "${varnishd} -x dumprstvsl > /dev/null 2>&1"
-err_shell {VCL version declaration missing} "echo 'bad vcl' >${tmpdir}/t.vcl ; ${varnishd} -f ${tmpdir}/t.vcl -n ${tmpdir} 2>&1"
-err_shell {VCL version declaration missing} "echo 'bad vcl' >${tmpdir}/t.vcl ; ${varnishd} -C -f ${tmpdir}/t.vcl -n ${tmpdir} 2>&1"
-err_shell {-spersistent has been deprecated} "${varnishd} -spersistent 2>&1"
-err_shell {Unknown jail method "xyz"} "${varnishd} -jxyz 2>&1"
-err_shell {-j must be the first argument} "${varnishd} -jnone -jxyz 2>&1"
-err_shell {-j must be the first argument} "${varnishd} -d -jxyz 2>&1"
+shell "varnishd -b 127.0.0.1:80 -C -n ${tmpdir} > /dev/null 2>&1"
+shell "varnishd -x dumprstparam > /dev/null 2>&1"
+shell "varnishd -x dumprstvsl > /dev/null 2>&1"
+err_shell {VCL version declaration missing} "echo 'bad vcl' >${tmpdir}/t.vcl ; varnishd -f ${tmpdir}/t.vcl -n ${tmpdir} 2>&1"
+err_shell {VCL version declaration missing} "echo 'bad vcl' >${tmpdir}/t.vcl ; varnishd -C -f ${tmpdir}/t.vcl -n ${tmpdir} 2>&1"
+err_shell {-spersistent has been deprecated} "varnishd -spersistent 2>&1"
+err_shell {Unknown jail method "xyz"} "varnishd -jxyz 2>&1"
+err_shell {-j must be the first argument} "varnishd -jnone -jxyz 2>&1"
+err_shell {-j must be the first argument} "varnishd -d -jxyz 2>&1"
diff --git a/bin/varnishtest/tests/b00041.vtc b/bin/varnishtest/tests/b00041.vtc
index f299c1b..7c004e7 100644
--- a/bin/varnishtest/tests/b00041.vtc
+++ b/bin/varnishtest/tests/b00041.vtc
@@ -2,4 +2,4 @@ varnishtest "Test varnishadm and the Telnet CLI"
 
 varnish v1 -vcl {backend foo { .host = "127.0.0.1"; } } -start
 
-shell "${varnishadm} -n ${tmpdir}/v1 help > /dev/null"
+shell "varnishadm -n ${tmpdir}/v1 help > /dev/null"
diff --git a/bin/varnishtest/tests/b00045.vtc b/bin/varnishtest/tests/b00045.vtc
index e1e524d..8b67483 100644
--- a/bin/varnishtest/tests/b00045.vtc
+++ b/bin/varnishtest/tests/b00045.vtc
@@ -15,4 +15,4 @@ client c1 {
 delay .2
 
 err_shell {Could not open pid/lock} \
-	"${varnishd} -P ${tmpdir}/v1/varnishd.pid -b 127.0.0.1:80 -a :0 -n ${tmpdir} 2>&1"
+	"varnishd -P ${tmpdir}/v1/varnishd.pid -b 127.0.0.1:80 -a :0 -n ${tmpdir} 2>&1"
diff --git a/bin/varnishtest/tests/c00003.vtc b/bin/varnishtest/tests/c00003.vtc
index 1013b86..75db736 100644
--- a/bin/varnishtest/tests/c00003.vtc
+++ b/bin/varnishtest/tests/c00003.vtc
@@ -5,10 +5,10 @@ varnishtest "Check that we fail to start if any listen address does not work"
 
 # All bad listen addresses
 err_shell "could not be resolved to an IP address" {
-	${varnishd} -F -a "${bad_ip}:0" -b '***' -n ${tmpdir} 2>&1
+	varnishd -F -a "${bad_ip}:0" -b '***' -n ${tmpdir} 2>&1
 }
 
 # old style address list
 err_shell "Unknown protocol" {
-	${varnishd} -F -a "127.0.0.1:0,${bad_ip}:0" -b '***' -n ${tmpdir} 2>&1
+	varnishd -F -a "127.0.0.1:0,${bad_ip}:0" -b '***' -n ${tmpdir} 2>&1
 }
diff --git a/bin/varnishtest/tests/j00002.vtc b/bin/varnishtest/tests/j00002.vtc
index 1ab0ade..96231d1 100644
--- a/bin/varnishtest/tests/j00002.vtc
+++ b/bin/varnishtest/tests/j00002.vtc
@@ -2,11 +2,11 @@ varnishtest "-junix bad subarg handling"
 
 feature root
 
-err_shell "unknown sub-argument" "${varnishd} -junix,bla=foo 2>&1"
-err_shell "user not found" "${varnishd} -junix,user=/// 2>&1"
-err_shell "user not found" "${varnishd} -junix,workuser=/// 2>&1"
-err_shell "group not found" "${varnishd} -junix,ccgroup=/// 2>&1"
+err_shell "unknown sub-argument" "varnishd -junix,bla=foo 2>&1"
+err_shell "user not found" "varnishd -junix,user=/// 2>&1"
+err_shell "user not found" "varnishd -junix,workuser=/// 2>&1"
+err_shell "group not found" "varnishd -junix,ccgroup=/// 2>&1"
 
 feature user_varnish
 
-err_shell "have different login groups" "${varnishd} -junix,workuser=root 2>&1"
+err_shell "have different login groups" "varnishd -junix,workuser=root 2>&1"
diff --git a/bin/varnishtest/tests/r01770.vtc b/bin/varnishtest/tests/r01770.vtc
index 838e44b..5243b4a 100644
--- a/bin/varnishtest/tests/r01770.vtc
+++ b/bin/varnishtest/tests/r01770.vtc
@@ -1,4 +1,4 @@
 varnishtest "bad protocol specs for -a"
 
-err_shell "Too many sub-arguments" "${varnishd} -a 127.0.0.1:80000,PROXY,FOO -d 2>&1"
-err_shell "Too many sub-arguments" "${varnishd} -a 127.0.0.1:80000,HTTP/1,FOO -d 2>&1"
+err_shell "Too many sub-arguments" "varnishd -a 127.0.0.1:80000,PROXY,FOO -d 2>&1"
+err_shell "Too many sub-arguments" "varnishd -a 127.0.0.1:80000,HTTP/1,FOO -d 2>&1"
diff --git a/bin/varnishtest/tests/r01813.vtc b/bin/varnishtest/tests/r01813.vtc
index f37aa54..88bb5ae 100644
--- a/bin/varnishtest/tests/r01813.vtc
+++ b/bin/varnishtest/tests/r01813.vtc
@@ -1,3 +1,3 @@
 varnishtest "Duplicate -a arguments"
 
-err_shell {have same address} "${varnishd} -d -a 127.0.0.1:38484 -a 127.0.0.1:38484 -b localhost:80 2>&1"
+err_shell {have same address} "varnishd -d -a 127.0.0.1:38484 -a 127.0.0.1:38484 -b localhost:80 2>&1"
diff --git a/bin/varnishtest/tests/u00001.vtc b/bin/varnishtest/tests/u00001.vtc
index 1a1c039..b06a315 100644
--- a/bin/varnishtest/tests/u00001.vtc
+++ b/bin/varnishtest/tests/u00001.vtc
@@ -12,7 +12,7 @@ server s1 {
 
 varnish v1 -vcl+backend "" -start
 
-shell "${varnishncsa} -D -P ${tmpdir}/ncsa.pid -n ${v1_name} -w ${tmpdir}/ncsa.log -F %s"
+shell "varnishncsa -D -P ${tmpdir}/ncsa.pid -n ${v1_name} -w ${tmpdir}/ncsa.log -F %s"
 
 # give varnishncsa enough time to open the VSM
 delay 2
diff --git a/bin/varnishtest/tests/v00045.vtc b/bin/varnishtest/tests/v00045.vtc
index 936dec9..dc57c74 100644
--- a/bin/varnishtest/tests/v00045.vtc
+++ b/bin/varnishtest/tests/v00045.vtc
@@ -17,21 +17,21 @@ varnish v1 -cliok "vcl.state vcl1 cold"
 # We should now see it as cooling
 delay 1
 shell {
-	${varnishadm} -n ${v1_name} vcl.list |
+	varnishadm -n ${v1_name} vcl.list |
 	grep "auto/cooling.*vcl1" >/dev/null
 }
 
 # It can't be warmed up yet
 delay 1
 shell {
-	${varnishadm} -n ${v1_name} vcl.state vcl1 warm 2>/dev/null |
+	varnishadm -n ${v1_name} vcl.state vcl1 warm 2>/dev/null |
 	grep "vmod-debug ref on vcl1" >/dev/null
 }
 
 # It will eventually cool down
 delay 2
 shell {
-	${varnishadm} -n ${v1_name} vcl.list |
+	varnishadm -n ${v1_name} vcl.list |
 	grep "auto/cold.*vcl1" >/dev/null
 }
 
diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index 7272e22..69b9c0a 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -474,11 +474,6 @@ main(int argc, char * const *argv)
 	char *p;
 	uintmax_t bufsiz;
 
-	/* Default names of programs */
-#define VTC_PROG(l)	extmacro_def(#l, #l);
-#include "programs.h"
-#undef VTC_PROG
-
 	/* Default import spec of vmods */
 #define VTC_VMOD(l)	extmacro_def("vmod_" #l, #l);
 #include "vmods.h"
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 1ff780f..e81fe17 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -402,7 +402,7 @@ varnish_launch(struct varnish *v)
 	vsb = VSB_new_auto();
 	AN(vsb);
 	VSB_printf(vsb, "cd ${pwd} &&");
-	VSB_printf(vsb, " exec ${varnishd} %s -d -n %s",
+	VSB_printf(vsb, " exec varnishd %s -d -n %s",
 	    v->jail, v->workdir);
 	if (vtc_witness)
 		VSB_cat(vsb, " -p debug=+witness");
diff --git a/doc/sphinx/reference/varnishtest.rst b/doc/sphinx/reference/varnishtest.rst
index 5bfca7b..057542e 100644
--- a/doc/sphinx/reference/varnishtest.rst
+++ b/doc/sphinx/reference/varnishtest.rst
@@ -36,7 +36,7 @@ The following options are available:
 
 -h               Show help
 
--i               Find varnishd in build tree
+-i               Set PATH to find varnish binaries in build tree
 
 -j jobs          Run this many tests in parallel
 
@@ -59,10 +59,6 @@ The following options are available:
 file             File to use as a script
 
 
-Macro definitions that can be overridden.
-
-varnishd         Path to varnishd to use [varnishd]
-
 If `TMPDIR` is set in the environment, varnishtest creates temporary
 `vtc.*` directories for each test in `$TMPDIR`, otherwise in `/tmp`.
 



From dridi.boukelmoune at gmail.com  Mon Feb 22 15:26:52 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 22 Feb 2016 16:26:52 +0100
Subject: [master] 9813d08 varnishtest -i builds vmod_path instead of macros
Message-ID: <E1aXsNw-0000sd-3e@project.varnish-software.com>


commit 9813d08597b72df14a9f0a2242326ec748b49dd3
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Sun Feb 21 19:31:55 2016 +0100

    varnishtest -i builds vmod_path instead of macros
    
    The macros are still here but ${vmod_std} is now expanded to std and so
    on.

diff --git a/bin/varnishtest/tests/m00008.vtc b/bin/varnishtest/tests/m00008.vtc
index 9386ae9..8e93d58 100644
--- a/bin/varnishtest/tests/m00008.vtc
+++ b/bin/varnishtest/tests/m00008.vtc
@@ -15,7 +15,7 @@ varnish v1 -cliok "param.set vcc_unsafe_path off"
 
 varnish v1 -errvcl {'import ... from path ...' is unsafe.} {
 	backend default { .host = "${s1_sock}"; }
-	import ${vmod_std};
+	import std from "${topbuild}/lib/libvmod_std/.libs/";
 }
 
 varnish v1 \
diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index 69b9c0a..03ced80 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -94,6 +94,7 @@ static int vtc_good;
 static int vtc_fail;
 static char *tmppath;
 static char *cwd = NULL;
+char *vmod_path = NULL;
 int leave_temp;
 int vtc_witness = 0;
 int feature_dns;
@@ -130,7 +131,7 @@ usage(void)
 	fprintf(stderr, FMT, "-b size",
 	    "Set internal buffer size (default: 512K)");
 	fprintf(stderr, FMT, "-D name=val", "Define macro");
-	fprintf(stderr, FMT, "-i", "Find varnishd in build tree");
+	fprintf(stderr, FMT, "-i", "Find varnish binaries in build tree");
 	fprintf(stderr, FMT, "-j jobs", "Run this many tests in parallel");
 	fprintf(stderr, FMT, "-k", "Continue on test failure");
 	fprintf(stderr, FMT, "-L", "Always leave temporary vtc.*");
@@ -376,19 +377,22 @@ i_mode(void)
 	AZ(putenv(strdup(VSB_data(vsb))));
 
 	/*
-	 * Redefine VMOD macros
+	 * Build vmod_path which can find all VMODs in the build tree
 	 */
+	VSB_clear(vsb);
+	sep = "";
 #define VTC_VMOD(l)							\
 	do {								\
-		VSB_clear(vsb);						\
-		VSB_printf(vsb,						\
-		   "%s from \"%s/lib/libvmod_%s/.libs/libvmod_%s.so\"",	\
-		    #l, topbuild, #l, #l);				\
-		AZ(VSB_finish(vsb));					\
-	    extmacro_def("vmod_" #l, "%s", VSB_data(vsb));		\
+		VSB_printf(vsb, "%s%s/lib/libvmod_" #l "/.libs",	\
+		    sep, topbuild);					\
+		sep = ":";						\
+		extmacro_def("vmod_" #l, #l);				\
 	} while (0);
 #include "vmods.h"
 #undef VTC_VMOD
+	AZ(VSB_finish(vsb));
+	vmod_path = strdup(VSB_data(vsb));
+	AN(vmod_path);
 	free(topbuild);
 	VSB_delete(vsb);
 }
@@ -487,6 +491,8 @@ main(int argc, char * const *argv)
 	cwd = getcwd(NULL, PATH_MAX);
 	extmacro_def("pwd", "%s", cwd);
 
+	vmod_path = NULL;
+
 	setbuf(stdout, NULL);
 	setbuf(stderr, NULL);
 	while ((ch = getopt(argc, argv, "b:D:hij:kLln:qt:vW")) != -1) {
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index e81fe17..1757722 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -54,6 +54,7 @@
 #include "vtim.h"
 
 extern int leave_temp;
+extern char *vmod_path;
 
 struct varnish {
 	unsigned		magic;
@@ -419,6 +420,8 @@ varnish_launch(struct varnish *v)
 		VSB_printf(vsb, ",%s", v->proto);
 	VSB_printf(vsb, " -M '%s %s'", abuf, pbuf);
 	VSB_printf(vsb, " -P %s/varnishd.pid", v->workdir);
+	if (vmod_path != NULL)
+		VSB_printf(vsb, " -p vmod_path=%s", vmod_path);
 	VSB_printf(vsb, " %s", VSB_data(v->args));
 	AZ(VSB_finish(vsb));
 	vtc_log(v->vl, 3, "CMD: %s", VSB_data(vsb));
diff --git a/doc/sphinx/reference/varnishtest.rst b/doc/sphinx/reference/varnishtest.rst
index 057542e..d689929 100644
--- a/doc/sphinx/reference/varnishtest.rst
+++ b/doc/sphinx/reference/varnishtest.rst
@@ -36,7 +36,7 @@ The following options are available:
 
 -h               Show help
 
--i               Set PATH to find varnish binaries in build tree
+-i               Set PATH and vmod_path to find varnish binaries in build tree
 
 -j jobs          Run this many tests in parallel
 



From dridi.boukelmoune at gmail.com  Mon Feb 22 15:26:52 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 22 Feb 2016 16:26:52 +0100
Subject: [master] d2ad24e Retire ${vmod_*} macros for varnishtest -i
Message-ID: <E1aXsNw-0000sm-5U@project.varnish-software.com>


commit d2ad24e49e8b8179b11c92a6cc440002ccbc2101
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Sun Feb 21 20:15:08 2016 +0100

    Retire ${vmod_*} macros for varnishtest -i

diff --git a/bin/varnishtest/tests/b00016.vtc b/bin/varnishtest/tests/b00016.vtc
index f4763ee..abf25b3 100644
--- a/bin/varnishtest/tests/b00016.vtc
+++ b/bin/varnishtest/tests/b00016.vtc
@@ -19,7 +19,7 @@ client c1 {
 } -run
 
 varnish v1 -vcl+backend {
-	import ${vmod_directors};
+	import directors;
 
 	sub vcl_init {
 		new bar = directors.random();
diff --git a/bin/varnishtest/tests/c00055.vtc b/bin/varnishtest/tests/c00055.vtc
index ccfb53c..10c87f2 100644
--- a/bin/varnishtest/tests/c00055.vtc
+++ b/bin/varnishtest/tests/c00055.vtc
@@ -11,7 +11,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_recv {
 		if (std.cache_req_body(1KB)) {
diff --git a/bin/varnishtest/tests/c00067.vtc b/bin/varnishtest/tests/c00067.vtc
index 65303af..fff734c 100644
--- a/bin/varnishtest/tests/c00067.vtc
+++ b/bin/varnishtest/tests/c00067.vtc
@@ -31,7 +31,7 @@ client c1 {
 delay .2
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_recv {
 		if (std.cache_req_body(110B)) {
diff --git a/bin/varnishtest/tests/c00070.vtc b/bin/varnishtest/tests/c00070.vtc
index b2df5b4..a3064d8 100644
--- a/bin/varnishtest/tests/c00070.vtc
+++ b/bin/varnishtest/tests/c00070.vtc
@@ -9,7 +9,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 	sub vcl_backend_response {
 		set beresp.http.free_backend = debug.workspace_free(backend);
 	}
diff --git a/bin/varnishtest/tests/c00071.vtc b/bin/varnishtest/tests/c00071.vtc
index 7572613..e589386 100644
--- a/bin/varnishtest/tests/c00071.vtc
+++ b/bin/varnishtest/tests/c00071.vtc
@@ -12,7 +12,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 	sub vcl_deliver {
 		debug.workspace_allocate(client, debug.workspace_free(client) - 150);
 
diff --git a/bin/varnishtest/tests/c00074.vtc b/bin/varnishtest/tests/c00074.vtc
index f20f71c..61f2dec 100644
--- a/bin/varnishtest/tests/c00074.vtc
+++ b/bin/varnishtest/tests/c00074.vtc
@@ -6,8 +6,8 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
-	import ${vmod_std};
+	import debug;
+	import std;
 
 	sub vcl_recv {
 		set req.http.ws-free = debug.workspace_free(session);
diff --git a/bin/varnishtest/tests/d00000.vtc b/bin/varnishtest/tests/d00000.vtc
index 2ec92ff..507e9c7 100644
--- a/bin/varnishtest/tests/d00000.vtc
+++ b/bin/varnishtest/tests/d00000.vtc
@@ -21,7 +21,7 @@ server s4 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_directors};
+	import directors;
 
 	sub vcl_init {
 		new rr = directors.round_robin();
diff --git a/bin/varnishtest/tests/d00001.vtc b/bin/varnishtest/tests/d00001.vtc
index 2311b0a..4051476 100644
--- a/bin/varnishtest/tests/d00001.vtc
+++ b/bin/varnishtest/tests/d00001.vtc
@@ -16,7 +16,7 @@ server s3 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_directors};
+	import directors;
 
 	sub vcl_init {
 		new fb1 = directors.fallback();
diff --git a/bin/varnishtest/tests/d00002.vtc b/bin/varnishtest/tests/d00002.vtc
index 0a0ba98..b13518c 100644
--- a/bin/varnishtest/tests/d00002.vtc
+++ b/bin/varnishtest/tests/d00002.vtc
@@ -8,7 +8,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_directors};
+	import directors;
 
 	sub vcl_init {
 		new foo = directors.random();
@@ -43,7 +43,7 @@ server s2 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_directors};
+	import directors;
 
 	sub vcl_init {
 		new foo = directors.random();
diff --git a/bin/varnishtest/tests/d00003.vtc b/bin/varnishtest/tests/d00003.vtc
index 970ed2d..7f12096 100644
--- a/bin/varnishtest/tests/d00003.vtc
+++ b/bin/varnishtest/tests/d00003.vtc
@@ -21,8 +21,8 @@ server s2 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
-	import ${vmod_directors};
+	import std;
+	import directors;
 
 	sub vcl_init {
 		new h1 = directors.hash();
diff --git a/bin/varnishtest/tests/d00004.vtc b/bin/varnishtest/tests/d00004.vtc
index cef05f0..82b35b5 100644
--- a/bin/varnishtest/tests/d00004.vtc
+++ b/bin/varnishtest/tests/d00004.vtc
@@ -35,7 +35,7 @@ server s4 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_directors};
+	import directors;
 
 	sub vcl_init {
 		new foo = directors.random();
diff --git a/bin/varnishtest/tests/d00005.vtc b/bin/varnishtest/tests/d00005.vtc
index d00e9c0..4d3b857 100644
--- a/bin/varnishtest/tests/d00005.vtc
+++ b/bin/varnishtest/tests/d00005.vtc
@@ -21,7 +21,7 @@ server s4 -repeat 2 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_directors};
+	import directors;
 
 	sub vcl_init {
 		new rr1 = directors.round_robin();
diff --git a/bin/varnishtest/tests/d00006.vtc b/bin/varnishtest/tests/d00006.vtc
index 9d8ba1d..e98d66a 100644
--- a/bin/varnishtest/tests/d00006.vtc
+++ b/bin/varnishtest/tests/d00006.vtc
@@ -23,7 +23,7 @@ server s4 {
 } -start
 
 varnish v1 -errvcl {Names of VCL objects cannot contain '-'} {
-	import ${vmod_directors};
+	import directors;
 	backend b1 { .host = "127.0.0.1"; .port = "8080";}
 	sub vcl_init {
 		new rr1-xx = directors.round_robin();
@@ -32,7 +32,7 @@ varnish v1 -errvcl {Names of VCL objects cannot contain '-'} {
 
 varnish v1 -vcl+backend {
 
-	import ${vmod_directors};
+	import directors;
 	sub vcl_init {
 		new rr1 = directors.round_robin();
 		rr1.add_backend(s1);
diff --git a/bin/varnishtest/tests/d00007.vtc b/bin/varnishtest/tests/d00007.vtc
index 26112e6..cdc889c 100644
--- a/bin/varnishtest/tests/d00007.vtc
+++ b/bin/varnishtest/tests/d00007.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl {
-	import ${vmod_debug};
+	import debug;
 
 	backend dummy { .host = "${bad_ip}"; .port = "9080"; }
 
diff --git a/bin/varnishtest/tests/d00008.vtc b/bin/varnishtest/tests/d00008.vtc
index 79096b9..465fcda 100644
--- a/bin/varnishtest/tests/d00008.vtc
+++ b/bin/varnishtest/tests/d00008.vtc
@@ -13,7 +13,7 @@ server s2 {
 } -start
 
 varnish v1 -vcl {
-	import ${vmod_debug};
+	import debug;
 
 	backend dummy { .host = "${bad_ip}"; .port = "9080"; }
 
diff --git a/bin/varnishtest/tests/d00009.vtc b/bin/varnishtest/tests/d00009.vtc
index 46d05c4..6b20030 100644
--- a/bin/varnishtest/tests/d00009.vtc
+++ b/bin/varnishtest/tests/d00009.vtc
@@ -16,7 +16,7 @@ server s2 {
 } -start
 
 varnish v1 -vcl {
-	import ${vmod_debug};
+	import debug;
 
 	backend dummy { .host = "${bad_ip}"; .port = "9080"; }
 
diff --git a/bin/varnishtest/tests/d00010.vtc b/bin/varnishtest/tests/d00010.vtc
index 145112a..e67fd54 100644
--- a/bin/varnishtest/tests/d00010.vtc
+++ b/bin/varnishtest/tests/d00010.vtc
@@ -16,7 +16,7 @@ server s2 {
 } -start
 
 varnish v1 -vcl {
-	import ${vmod_debug};
+	import debug;
 
 	backend dummy { .host = "${bad_ip}"; .port = "9080"; }
 
diff --git a/bin/varnishtest/tests/d00011.vtc b/bin/varnishtest/tests/d00011.vtc
index 639161e..ab1c8e0 100644
--- a/bin/varnishtest/tests/d00011.vtc
+++ b/bin/varnishtest/tests/d00011.vtc
@@ -9,8 +9,8 @@ server s2 {
 } -start
 
 varnish v1 -vcl {
-	import ${vmod_std};
-	import ${vmod_debug};
+	import std;
+	import debug;
 
 	backend dummy { .host = "${bad_ip}"; .port = "9080"; }
 
diff --git a/bin/varnishtest/tests/d00012.vtc b/bin/varnishtest/tests/d00012.vtc
index edc028b..aba0aec 100644
--- a/bin/varnishtest/tests/d00012.vtc
+++ b/bin/varnishtest/tests/d00012.vtc
@@ -10,7 +10,7 @@ server s1 {
 } -start
 
 varnish v1 -arg "-p thread_pools=1" -vcl {
-	import ${vmod_debug};
+	import debug;
 
 	backend dummy { .host = "${bad_ip}"; .port = "9080"; }
 
@@ -39,7 +39,7 @@ server s2 {
 } -start
 
 varnish v1 -vcl {
-	import ${vmod_debug};
+	import debug;
 
 	backend dummy { .host = "${bad_ip}"; .port = "9080"; }
 
diff --git a/bin/varnishtest/tests/d00013.vtc b/bin/varnishtest/tests/d00013.vtc
index 6762f41..4a397ee 100644
--- a/bin/varnishtest/tests/d00013.vtc
+++ b/bin/varnishtest/tests/d00013.vtc
@@ -9,8 +9,8 @@ server s2 {
 } -start
 
 varnish v1 -vcl {
-	import ${vmod_std};
-	import ${vmod_debug};
+	import std;
+	import debug;
 
 	backend dummy { .host = "${bad_ip}"; .port = "9080"; }
 
diff --git a/bin/varnishtest/tests/m00000.vtc b/bin/varnishtest/tests/m00000.vtc
index b3ac748..b4d29c8 100644
--- a/bin/varnishtest/tests/m00000.vtc
+++ b/bin/varnishtest/tests/m00000.vtc
@@ -7,8 +7,8 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
-	import ${vmod_debug};
+	import std;
+	import debug;
 
 	sub vcl_recv {
 		debug.rot52(req);
@@ -37,14 +37,14 @@ client c1 {
 } -run
 
 varnish v1 -errvcl {Wrong enum value.  Expected one of:} {
-	import ${vmod_debug};
+	import debug;
 	sub vcl_deliver {
 		set resp.http.who = debug.author(jfk);
 	}
 }
 
 varnish v1 -errvcl {Wrong argument type.  Expected REAL.  Got STRING.} {
-	import ${vmod_std};
+	import std;
 	sub vcl_deliver {
 		set resp.http.who = std.random("foo", "bar");
 	}
diff --git a/bin/varnishtest/tests/m00001.vtc b/bin/varnishtest/tests/m00001.vtc
index 968345a..5f7979f 100644
--- a/bin/varnishtest/tests/m00001.vtc
+++ b/bin/varnishtest/tests/m00001.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.foo = std.toupper(resp.http.foo);
@@ -48,12 +48,12 @@ varnish v1 -cliok "vcl.list"
 varnish v1 -cliok "debug.vmod"
 
 varnish v1 -errvcl {Module std already imported.} {
-	import ${vmod_std};
-	import ${vmod_std};
+	import std;
+	import std;
 }
 
 varnish v1 -errvcl {Symbol not found: 'std' (expected type BOOL):} {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_recv {
 		if (std == 2) {
diff --git a/bin/varnishtest/tests/m00002.vtc b/bin/varnishtest/tests/m00002.vtc
index bb5626d..9223f0d 100644
--- a/bin/varnishtest/tests/m00002.vtc
+++ b/bin/varnishtest/tests/m00002.vtc
@@ -9,7 +9,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.rnd1 = std.random(0, 1);
diff --git a/bin/varnishtest/tests/m00004.vtc b/bin/varnishtest/tests/m00004.vtc
index 9e1d047..bb3ba56 100644
--- a/bin/varnishtest/tests/m00004.vtc
+++ b/bin/varnishtest/tests/m00004.vtc
@@ -14,7 +14,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		if (req.url == "/one") {
@@ -51,7 +51,7 @@ client c1 {
 } -run
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		if (req.url == "/one") {
diff --git a/bin/varnishtest/tests/m00005.vtc b/bin/varnishtest/tests/m00005.vtc
index d89396c..fecbe3f 100644
--- a/bin/varnishtest/tests/m00005.vtc
+++ b/bin/varnishtest/tests/m00005.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.ttl = std.duration(req.http.ttl, 1s) + 1000000s;
diff --git a/bin/varnishtest/tests/m00006.vtc b/bin/varnishtest/tests/m00006.vtc
index d002112..2afad24 100644
--- a/bin/varnishtest/tests/m00006.vtc
+++ b/bin/varnishtest/tests/m00006.vtc
@@ -8,7 +8,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_recv {
 		std.collect(req.http.foo);
diff --git a/bin/varnishtest/tests/m00007.vtc b/bin/varnishtest/tests/m00007.vtc
index 9a1fe39..b66fc0b 100644
--- a/bin/varnishtest/tests/m00007.vtc
+++ b/bin/varnishtest/tests/m00007.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.gtzero = (std.integer(req.http.foo, 0) > 0);
diff --git a/bin/varnishtest/tests/m00008.vtc b/bin/varnishtest/tests/m00008.vtc
index 8e93d58..5912dc5 100644
--- a/bin/varnishtest/tests/m00008.vtc
+++ b/bin/varnishtest/tests/m00008.vtc
@@ -8,7 +8,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 }
 
 varnish v1 -cliok "param.set vcc_unsafe_path off"
diff --git a/bin/varnishtest/tests/m00011.vtc b/bin/varnishtest/tests/m00011.vtc
index 45528c9..1c51a07 100644
--- a/bin/varnishtest/tests/m00011.vtc
+++ b/bin/varnishtest/tests/m00011.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.foo0 = std.ip("8.8.8.*", client.ip);
diff --git a/bin/varnishtest/tests/m00012.vtc b/bin/varnishtest/tests/m00012.vtc
index ef07f90..de3e835 100644
--- a/bin/varnishtest/tests/m00012.vtc
+++ b/bin/varnishtest/tests/m00012.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 
 	sub vcl_deliver {
 		set resp.http.foo = debug.blob2hex(debug.str2blob("gunk"));
@@ -25,7 +25,7 @@ varnish v1 -errvcl {BLOBs can only be used as arguments to VMOD functions.} {
 
 	backend b1 {.host = "127.0.0.1";}
 
-	import ${vmod_debug};
+	import debug;
 
 	sub vcl_deliver {
 		set resp.http.foo = debug.str2blob("gunk");
@@ -36,7 +36,7 @@ varnish v1 -errvcl {Wrong argument type.  Expected BLOB.  Got STRING.} {
 
 	backend b1 {.host = "127.0.0.1";}
 
-	import ${vmod_debug};
+	import debug;
 
 	sub vcl_deliver {
 		set resp.http.foo = debug.blob2hex("gunk");
diff --git a/bin/varnishtest/tests/m00013.vtc b/bin/varnishtest/tests/m00013.vtc
index 6f27c57..e657fb3 100644
--- a/bin/varnishtest/tests/m00013.vtc
+++ b/bin/varnishtest/tests/m00013.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_init {
 		# Make sure that this doesn't cause ill effects
diff --git a/bin/varnishtest/tests/m00014.vtc b/bin/varnishtest/tests/m00014.vtc
index 28c4c79..6d60def 100644
--- a/bin/varnishtest/tests/m00014.vtc
+++ b/bin/varnishtest/tests/m00014.vtc
@@ -6,7 +6,7 @@ server s1 -repeat 6 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.url = std.querysort(req.url);
diff --git a/bin/varnishtest/tests/m00015.vtc b/bin/varnishtest/tests/m00015.vtc
index a5ec8ba..1845fa7 100644
--- a/bin/varnishtest/tests/m00015.vtc
+++ b/bin/varnishtest/tests/m00015.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.t0 = std.real(req.http.foo, 0.0);
diff --git a/bin/varnishtest/tests/m00016.vtc b/bin/varnishtest/tests/m00016.vtc
index 261f840..a216085 100644
--- a/bin/varnishtest/tests/m00016.vtc
+++ b/bin/varnishtest/tests/m00016.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.x-foo = std.integer(req.http.foo, 0);
diff --git a/bin/varnishtest/tests/m00017.vtc b/bin/varnishtest/tests/m00017.vtc
index 5622ec0..8fa38f3 100644
--- a/bin/varnishtest/tests/m00017.vtc
+++ b/bin/varnishtest/tests/m00017.vtc
@@ -23,7 +23,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_recv {
 		if (req.url == "/foo") {
@@ -46,7 +46,7 @@ client c1 {
 } -run
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_backend_fetch {
 		if (bereq.url == "/baz") {
diff --git a/bin/varnishtest/tests/m00018.vtc b/bin/varnishtest/tests/m00018.vtc
index 35000ce..33d68f3 100644
--- a/bin/varnishtest/tests/m00018.vtc
+++ b/bin/varnishtest/tests/m00018.vtc
@@ -6,7 +6,7 @@ server s1 -repeat 2 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.sub = std.strstr(req.url, "b");
diff --git a/bin/varnishtest/tests/m00019.vtc b/bin/varnishtest/tests/m00019.vtc
index 89ad9b0..b640a44 100644
--- a/bin/varnishtest/tests/m00019.vtc
+++ b/bin/varnishtest/tests/m00019.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 
 	sub vcl_deliver {
 		set resp.http.foo1 = debug.argtest("1", 2.0, "3");
@@ -33,7 +33,7 @@ client c1 {
 delay .1
 
 varnish v1 -errvcl {Argument 'one' already used} {
-	import ${vmod_debug};
+	import debug;
 	backend b1 {.host = "127.0.0.1";}
 	sub vcl_deliver {
 		set resp.http.foo5 = debug.argtest("1", one="1");
@@ -41,7 +41,7 @@ varnish v1 -errvcl {Argument 'one' already used} {
 }
 
 varnish v1 -errvcl {Argument 'one' missing} {
-	import ${vmod_debug};
+	import debug;
 	backend b1 {.host = "127.0.0.1";}
 	sub vcl_deliver {
 		set resp.http.foo5 = debug.argtest(two=2.0, three="3");
@@ -49,7 +49,7 @@ varnish v1 -errvcl {Argument 'one' missing} {
 }
 
 varnish v1 -errvcl {Unknown argument 'four'} {
-	import ${vmod_debug};
+	import debug;
 	backend b1 {.host = "127.0.0.1";}
 	sub vcl_deliver {
 		set resp.http.foo5 = debug.argtest("1", two=2.0, four="3");
diff --git a/bin/varnishtest/tests/m00020.vtc b/bin/varnishtest/tests/m00020.vtc
index 4d76b40..8fe5c83 100644
--- a/bin/varnishtest/tests/m00020.vtc
+++ b/bin/varnishtest/tests/m00020.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.x-date = std.time(
diff --git a/bin/varnishtest/tests/m00021.vtc b/bin/varnishtest/tests/m00021.vtc
index 87f6834..007728a 100644
--- a/bin/varnishtest/tests/m00021.vtc
+++ b/bin/varnishtest/tests/m00021.vtc
@@ -17,7 +17,7 @@ logexpect l1 -v v1 -g raw {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 
 	sub vcl_init {
 		debug.register_obj_events();
diff --git a/bin/varnishtest/tests/m00022.vtc b/bin/varnishtest/tests/m00022.vtc
index 29da34e..8b3a249 100644
--- a/bin/varnishtest/tests/m00022.vtc
+++ b/bin/varnishtest/tests/m00022.vtc
@@ -9,7 +9,7 @@ varnish v1 -vcl+backend { } -start
 
 varnish v1 -errvcl "Planned failure in vcl_init" {
 
-	import ${vmod_debug};
+	import debug;
 
 	backend default {
 		.host = "${s1_addr}";
@@ -24,7 +24,7 @@ varnish v1 -cliok "param.set nuke_limit 42"
 
 varnish v1 -errvcl "nuke_limit is not the answer." {
 
-	import ${vmod_debug};
+	import debug;
 
 	backend default {
 		.host = "${s1_addr}";
diff --git a/bin/varnishtest/tests/o00000.vtc b/bin/varnishtest/tests/o00000.vtc
index ccfaa65..1bf1e1d 100644
--- a/bin/varnishtest/tests/o00000.vtc
+++ b/bin/varnishtest/tests/o00000.vtc
@@ -15,7 +15,7 @@ server s1 {
 } -start
 
 varnish v1 -proto "PROXY" -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.url = req.url;
diff --git a/bin/varnishtest/tests/o00001.vtc b/bin/varnishtest/tests/o00001.vtc
index ecf025a..c906963 100644
--- a/bin/varnishtest/tests/o00001.vtc
+++ b/bin/varnishtest/tests/o00001.vtc
@@ -15,7 +15,7 @@ server s1 {
 } -start
 
 varnish v1 -proto "PROXY" -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	acl fwd_client {
 		"1.2.3.4";
diff --git a/bin/varnishtest/tests/r00263.vtc b/bin/varnishtest/tests/r00263.vtc
index c9036d5..5b75b6f 100644
--- a/bin/varnishtest/tests/r00263.vtc
+++ b/bin/varnishtest/tests/r00263.vtc
@@ -3,7 +3,7 @@ varnishtest "Test refcounting backends from director"
 # XXX: This test is probably not relevant with VMOD.directors
 
 varnish v1  -vcl {
-	import ${vmod_directors};
+	import directors;
 
 	backend node1 { .host = "10.0.0.1"; .port = "80"; }
 	backend node2 { .host = "10.0.0.2"; .port = "80"; }
diff --git a/bin/varnishtest/tests/r00306.vtc b/bin/varnishtest/tests/r00306.vtc
index 0d4c465..bbed997 100644
--- a/bin/varnishtest/tests/r00306.vtc
+++ b/bin/varnishtest/tests/r00306.vtc
@@ -19,7 +19,7 @@ server s2 {
 
 varnish v1 -vcl {
 
-	import ${vmod_directors};
+	import directors;
 
 	backend s1 {
 		.host = "${s1_addr}"; .port = "${s1_port}";
diff --git a/bin/varnishtest/tests/r00722.vtc b/bin/varnishtest/tests/r00722.vtc
index 739b8f0..b6a6ff7 100644
--- a/bin/varnishtest/tests/r00722.vtc
+++ b/bin/varnishtest/tests/r00722.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_directors};
+	import directors;
 
 	backend b2 {
 		.host = "${s1_addr}"; .port = "${s1_port}";
diff --git a/bin/varnishtest/tests/r00878.vtc b/bin/varnishtest/tests/r00878.vtc
index 5725393..c6bc015 100644
--- a/bin/varnishtest/tests/r00878.vtc
+++ b/bin/varnishtest/tests/r00878.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 	sub vcl_deliver {
 		set resp.http.who = debug.author(phk);
 	}
@@ -18,7 +18,7 @@ client c1 {
 	rxresp
 } -run
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 	sub vcl_deliver {
 		set resp.http.who = debug.author(des);
 	}
@@ -30,7 +30,7 @@ client c1 {
 } -run
 
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 	sub vcl_deliver {
 		set resp.http.who = debug.author(kristian);
 	}
diff --git a/bin/varnishtest/tests/r01002.vtc b/bin/varnishtest/tests/r01002.vtc
index 6e9e3fd..d9bfb52 100644
--- a/bin/varnishtest/tests/r01002.vtc
+++ b/bin/varnishtest/tests/r01002.vtc
@@ -1,7 +1,7 @@
 varnishtest "Real relational comparisons"
 
 varnish v1 -vcl {
-	import ${vmod_std};
+	import std;
 
 	backend foo { .host = "${bad_ip}"; .port = "9080"; }
 	sub vcl_recv {
diff --git a/bin/varnishtest/tests/r01145.vtc b/bin/varnishtest/tests/r01145.vtc
index cb26a2f..ee33bb0 100644
--- a/bin/varnishtest/tests/r01145.vtc
+++ b/bin/varnishtest/tests/r01145.vtc
@@ -15,7 +15,7 @@ server s1 {
 
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 	    set resp.http.foo = std.fileread("${tmpdir}" + req.url);
diff --git a/bin/varnishtest/tests/r01212.vtc b/bin/varnishtest/tests/r01212.vtc
index a1e91ee..0fc2634 100644
--- a/bin/varnishtest/tests/r01212.vtc
+++ b/bin/varnishtest/tests/r01212.vtc
@@ -1,7 +1,7 @@
 varnishtest "#1212 - Vmod with HEADER argument given a STRING asserts the VCL compiler"
 
 varnish v1 -errvcl {Wrong argument type.  Expected HEADER.  Got STRING.} {
-	import ${vmod_std};
+	import std;
 	backend b { .host = "127.0.0.1"; }
 	sub vcl_recv {
 		std.collect("foo");
diff --git a/bin/varnishtest/tests/r01312.vtc b/bin/varnishtest/tests/r01312.vtc
index c39dadc..762da43 100644
--- a/bin/varnishtest/tests/r01312.vtc
+++ b/bin/varnishtest/tests/r01312.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	acl foo {
 		"127.0.0.2";
diff --git a/bin/varnishtest/tests/r01332.vtc b/bin/varnishtest/tests/r01332.vtc
index 518c558..43ca406 100644
--- a/bin/varnishtest/tests/r01332.vtc
+++ b/bin/varnishtest/tests/r01332.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 
 	sub vcl_init {
 		new obj = debug.obj("don't care");
diff --git a/bin/varnishtest/tests/r01406.vtc b/bin/varnishtest/tests/r01406.vtc
index 491f9e9..ee4e5e4 100644
--- a/bin/varnishtest/tests/r01406.vtc
+++ b/bin/varnishtest/tests/r01406.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -arg "-p vcc_allow_inline_c=true" -vcl+backend {
-	import ${vmod_std} ;
+	import std ;
 
 	C{
 		static const struct gethdr_s VGC_HDR_REQ_foo =
diff --git a/bin/varnishtest/tests/r01498.vtc b/bin/varnishtest/tests/r01498.vtc
index d321448..0c81af5 100644
--- a/bin/varnishtest/tests/r01498.vtc
+++ b/bin/varnishtest/tests/r01498.vtc
@@ -2,7 +2,7 @@ varnishtest "backend name VCC crash"
 
 varnish v1 -vcl {
 	vcl 4.0;
-	import ${vmod_directors};
+	import directors;
 	backend s1 {
 	     .host = "127.0.0.1";
 	     .port = "80";
diff --git a/bin/varnishtest/tests/r01501.vtc b/bin/varnishtest/tests/r01501.vtc
index c7ce3b8..38f220c 100644
--- a/bin/varnishtest/tests/r01501.vtc
+++ b/bin/varnishtest/tests/r01501.vtc
@@ -7,7 +7,7 @@ server s1 {
 
 varnish v1 -vcl+backend {
 
-	import ${vmod_debug};
+	import debug;
 
 	sub vcl_recv {
 		set req.backend_hint = debug.no_backend();
diff --git a/bin/varnishtest/tests/r01510.vtc b/bin/varnishtest/tests/r01510.vtc
index 6b74f54..9597240 100644
--- a/bin/varnishtest/tests/r01510.vtc
+++ b/bin/varnishtest/tests/r01510.vtc
@@ -1,7 +1,7 @@
 varnishtest "Duplicate object names"
 
 varnish v1 -errvcl {Object name 'first' already used.} {
-	import ${vmod_debug};
+	import debug;
 	sub vcl_init {
 		new first = debug.obj("FOO");
 		new first = debug.obj("BAH");
@@ -9,7 +9,7 @@ varnish v1 -errvcl {Object name 'first' already used.} {
 }
 
 varnish v1 -errvcl {Object name 'first' already used.} {
-	import ${vmod_debug};
+	import debug;
 
 	backend first { .host = "${bad_ip}"; .port = "9080"; }
 
diff --git a/bin/varnishtest/tests/r01532.vtc b/bin/varnishtest/tests/r01532.vtc
index b1ff007..271e51e 100644
--- a/bin/varnishtest/tests/r01532.vtc
+++ b/bin/varnishtest/tests/r01532.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.x-foo = std.real2time(1140618699.00, now);
diff --git a/bin/varnishtest/tests/r01569.vtc b/bin/varnishtest/tests/r01569.vtc
index 7a05938..af3c489 100644
--- a/bin/varnishtest/tests/r01569.vtc
+++ b/bin/varnishtest/tests/r01569.vtc
@@ -2,7 +2,7 @@ varnishtest "symbol lookup order issue"
 
 varnish v1 -vcl {
 	vcl 4.0;
-	import ${vmod_debug};
+	import debug;
 
 	backend debug {
 		.host = "127.0.0.1";
diff --git a/bin/varnishtest/tests/r01575.vtc b/bin/varnishtest/tests/r01575.vtc
index 11c82c1..c74fad5 100644
--- a/bin/varnishtest/tests/r01575.vtc
+++ b/bin/varnishtest/tests/r01575.vtc
@@ -29,7 +29,7 @@ server s5 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_directors};
+	import directors;
 	sub vcl_init {
 		new rd = directors.random();
 		rd.add_backend(s1, 10000);
diff --git a/bin/varnishtest/tests/r01613.vtc b/bin/varnishtest/tests/r01613.vtc
index 62c7213..4fca7c4 100644
--- a/bin/varnishtest/tests/r01613.vtc
+++ b/bin/varnishtest/tests/r01613.vtc
@@ -12,7 +12,7 @@ varnish v2 -vcl+backend {
 } -start
 
 varnish v1 -vcl {
-	import ${vmod_std};
+	import std;
 
 	backend b {
 		.host = "${v2_addr}";
diff --git a/bin/varnishtest/tests/r01644.vtc b/bin/varnishtest/tests/r01644.vtc
index 90728a0..a6e4fbf 100644
--- a/bin/varnishtest/tests/r01644.vtc
+++ b/bin/varnishtest/tests/r01644.vtc
@@ -8,7 +8,7 @@ server s1 {
 
 varnish v1 -vcl+backend {
 
-	import ${vmod_debug};
+	import debug;
 
 	sub vcl_deliver {
 		set resp.http.foo = debug.vre_limit();
diff --git a/bin/varnishtest/tests/r01739.vtc b/bin/varnishtest/tests/r01739.vtc
index 21398cc..dbbd04d 100644
--- a/bin/varnishtest/tests/r01739.vtc
+++ b/bin/varnishtest/tests/r01739.vtc
@@ -8,7 +8,7 @@ server s1 {
 
 varnish v1 \
  -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 	sub vcl_backend_response {
 		set beresp.do_gzip = true;
 		debug.workspace_allocate(backend, debug.workspace_free(backend) - 16);
diff --git a/bin/varnishtest/tests/r01801.vtc b/bin/varnishtest/tests/r01801.vtc
index 7052ead..b0ef516 100644
--- a/bin/varnishtest/tests/r01801.vtc
+++ b/bin/varnishtest/tests/r01801.vtc
@@ -6,7 +6,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.foo1 = std.ip("..", "1.2.3.4");
@@ -42,7 +42,7 @@ client c1 {
 } -run
 
 varnish v1 -errvcl "could not be resolved to an IP address" {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.foo = std.ip("..", "::1::2");
@@ -50,7 +50,7 @@ varnish v1 -errvcl "could not be resolved to an IP address" {
 }
 
 varnish v1 -errvcl "could not be resolved to an IP address" {
-	import ${vmod_std};
+	import std;
 
 	sub vcl_deliver {
 		set resp.http.foo = std.ip("..", "1.2.3.4::80");
diff --git a/bin/varnishtest/tests/v00010.vtc b/bin/varnishtest/tests/v00010.vtc
index 7b0d57e..7dd839c 100644
--- a/bin/varnishtest/tests/v00010.vtc
+++ b/bin/varnishtest/tests/v00010.vtc
@@ -23,7 +23,7 @@ server s1 {
 
 varnish v1 -arg "-smalloc,1m" -vcl+backend {
 
-	import ${vmod_debug};
+	import debug;
 
 	sub vcl_backend_response {
 		if (beresp.http.panic == "fetch") {
diff --git a/bin/varnishtest/tests/v00014.vtc b/bin/varnishtest/tests/v00014.vtc
index 4a290b0..78b0288 100644
--- a/bin/varnishtest/tests/v00014.vtc
+++ b/bin/varnishtest/tests/v00014.vtc
@@ -17,7 +17,7 @@ server s1 {
 
 varnish v1 -vcl {
 
-	import ${vmod_std};
+	import std;
 
 	probe foo {
 		.url = "/";
diff --git a/bin/varnishtest/tests/v00025.vtc b/bin/varnishtest/tests/v00025.vtc
index 02788d4..44426ad 100644
--- a/bin/varnishtest/tests/v00025.vtc
+++ b/bin/varnishtest/tests/v00025.vtc
@@ -7,7 +7,7 @@ server s1 {
 
 varnish v1 -arg "-i J.F.Nobody" -vcl+backend {
 
-	import ${vmod_std};
+	import std;
 
 	sub vcl_recv {
 		set client.identity = "Samuel B. Nobody";
diff --git a/bin/varnishtest/tests/v00041.vtc b/bin/varnishtest/tests/v00041.vtc
index 9604eb2..b5c053c 100644
--- a/bin/varnishtest/tests/v00041.vtc
+++ b/bin/varnishtest/tests/v00041.vtc
@@ -8,8 +8,8 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
-	import ${vmod_std};
+	import debug;
+	import std;
 
 	sub vcl_recv {
 		set req.http.x0 = debug.test_priv_task(req.url);
diff --git a/bin/varnishtest/tests/v00042.vtc b/bin/varnishtest/tests/v00042.vtc
index 1640f56..6b7ed9c 100644
--- a/bin/varnishtest/tests/v00042.vtc
+++ b/bin/varnishtest/tests/v00042.vtc
@@ -34,7 +34,7 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 
 	sub vcl_recv {
 		set req.http.x0 = debug.test_priv_task(req.url + req.esi_level);
diff --git a/bin/varnishtest/tests/v00043.vtc b/bin/varnishtest/tests/v00043.vtc
index ae562a7..9bf7b47 100644
--- a/bin/varnishtest/tests/v00043.vtc
+++ b/bin/varnishtest/tests/v00043.vtc
@@ -36,7 +36,7 @@ server s1 {
 } -start
 
 varnish v1 -cliok "param.set debug +syncvsl" -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 
 	sub vcl_recv {
 		set req.http.x0 = debug.test_priv_top(req.url + req.esi_level);
diff --git a/bin/varnishtest/tests/v00044.vtc b/bin/varnishtest/tests/v00044.vtc
index c056d6c..d09501b 100644
--- a/bin/varnishtest/tests/v00044.vtc
+++ b/bin/varnishtest/tests/v00044.vtc
@@ -9,7 +9,7 @@ server s1 -repeat 20 {
 
 # The debug vmod logs temperature vcl events
 varnish v1 -arg "-p vcl_cooldown=1" -vcl {
-	import ${vmod_debug};
+	import debug;
 	backend default {
 		.host = "${s1_addr}";
 		.probe = { .interval = 1s; .initial = 1;}
diff --git a/bin/varnishtest/tests/v00045.vtc b/bin/varnishtest/tests/v00045.vtc
index dc57c74..fe23158 100644
--- a/bin/varnishtest/tests/v00045.vtc
+++ b/bin/varnishtest/tests/v00045.vtc
@@ -4,7 +4,7 @@ server s1 -start
 
 # Load and use a VCL that will hold a reference
 varnish v1 -vcl+backend {
-	import ${vmod_debug};
+	import debug;
 	sub vcl_init {
 		debug.vcl_release_delay(3s);
 	}
diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index 03ced80..ecca005 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -386,7 +386,6 @@ i_mode(void)
 		VSB_printf(vsb, "%s%s/lib/libvmod_" #l "/.libs",	\
 		    sep, topbuild);					\
 		sep = ":";						\
-		extmacro_def("vmod_" #l, #l);				\
 	} while (0);
 #include "vmods.h"
 #undef VTC_VMOD
@@ -478,11 +477,6 @@ main(int argc, char * const *argv)
 	char *p;
 	uintmax_t bufsiz;
 
-	/* Default import spec of vmods */
-#define VTC_VMOD(l)	extmacro_def("vmod_" #l, #l);
-#include "vmods.h"
-#undef VTC_VMOD
-
 	if (getenv("TMPDIR") != NULL)
 		tmppath = strdup(getenv("TMPDIR"));
 	else



From dridi.boukelmoune at gmail.com  Mon Feb 22 15:26:52 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 22 Feb 2016 16:26:52 +0100
Subject: [master] da1d336 Polish
Message-ID: <E1aXsNw-0000sw-6s@project.varnish-software.com>


commit da1d3365efcb635fdfce371aa710c7fc637d8d82
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Sun Feb 21 20:28:53 2016 +0100

    Polish

diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index ecca005..0f00a80 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -324,6 +324,7 @@ i_mode(void)
 	 */
 
 	vsb = VSB_new_auto();
+	AN(vsb);
 
 	q = p = VFIL_readfile(NULL, "Makefile", NULL);
 	if (p == NULL) {
@@ -361,12 +362,11 @@ i_mode(void)
 	/*
 	 * Build $PATH which can find all programs in the build tree
 	 */
-	AN(vsb);
 	VSB_printf(vsb, "PATH=");
 	sep = "";
 #define VTC_PROG(l)							\
 	do {								\
-		VSB_printf(vsb, "%s%s/bin/%s/", sep, topbuild, #l);	\
+		VSB_printf(vsb, "%s%s/bin/" #l, sep, topbuild);		\
 		sep = ":";						\
 	} while (0);
 #include "programs.h"



From dridi.boukelmoune at gmail.com  Mon Feb 22 15:26:52 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 22 Feb 2016 16:26:52 +0100
Subject: [master] dd5aa53 Pass varnishd parameters through varnishtest
Message-ID: <E1aXsNw-0000t5-8G@project.varnish-software.com>


commit dd5aa5322d1a3b4934672d562084cbad2cd8bce9
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Sun Feb 21 21:22:53 2016 +0100

    Pass varnishd parameters through varnishtest
    
    Parameters are not checked and passed as is to varnishd command lines.
    The parameters are prepended at the beginning of the command line to
    preserve parameters needed by varnishtest or used in test cases.

diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index 0f00a80..cd3fb1d 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -95,6 +95,7 @@ static int vtc_fail;
 static char *tmppath;
 static char *cwd = NULL;
 char *vmod_path = NULL;
+struct vsb *params_vsb = NULL;
 int leave_temp;
 int vtc_witness = 0;
 int feature_dns;
@@ -137,6 +138,7 @@ usage(void)
 	fprintf(stderr, FMT, "-L", "Always leave temporary vtc.*");
 	fprintf(stderr, FMT, "-l", "Leave temporary vtc.* if test fails");
 	fprintf(stderr, FMT, "-n iterations", "Run tests this many times");
+	fprintf(stderr, FMT, "-p name=val", "Pass a varnishd parameter");
 	fprintf(stderr, FMT, "-q", "Quiet mode: report only failures");
 	fprintf(stderr, FMT, "-t duration", "Time tests out after this long");
 	fprintf(stderr, FMT, "-v", "Verbose mode: always report test log");
@@ -487,9 +489,12 @@ main(int argc, char * const *argv)
 
 	vmod_path = NULL;
 
+	params_vsb = VSB_new_auto();
+	AN(params_vsb);
+
 	setbuf(stdout, NULL);
 	setbuf(stderr, NULL);
-	while ((ch = getopt(argc, argv, "b:D:hij:kLln:qt:vW")) != -1) {
+	while ((ch = getopt(argc, argv, "b:D:hij:kLln:p:qt:vW")) != -1) {
 		switch (ch) {
 		case 'b':
 			if (VNUM_2bytes(optarg, &bufsiz, 0)) {
@@ -529,6 +534,10 @@ main(int argc, char * const *argv)
 		case 'n':
 			ntest = strtoul(optarg, NULL, 0);
 			break;
+		case 'p':
+			VSB_printf(params_vsb, " -p ");
+			VSB_quote(params_vsb, optarg, -1, VSB_QUOTE_NONL);
+			break;
 		case 'q':
 			if (vtc_verbosity > 0)
 				vtc_verbosity--;
@@ -567,6 +576,8 @@ main(int argc, char * const *argv)
 		VTAILQ_INSERT_TAIL(&tst_head, tp, list);
 	}
 
+	VSB_finish(params_vsb);
+
 	feature_dns = dns_works();
 	ip_magic();
 
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 1757722..8df63ea 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -55,6 +55,7 @@
 
 extern int leave_temp;
 extern char *vmod_path;
+extern struct vsb *params_vsb;
 
 struct varnish {
 	unsigned		magic;
@@ -405,6 +406,7 @@ varnish_launch(struct varnish *v)
 	VSB_printf(vsb, "cd ${pwd} &&");
 	VSB_printf(vsb, " exec varnishd %s -d -n %s",
 	    v->jail, v->workdir);
+	VSB_printf(vsb, VSB_data(params_vsb));
 	if (vtc_witness)
 		VSB_cat(vsb, " -p debug=+witness");
 	if (leave_temp)
diff --git a/doc/sphinx/reference/varnishtest.rst b/doc/sphinx/reference/varnishtest.rst
index d689929..ba8708d 100644
--- a/doc/sphinx/reference/varnishtest.rst
+++ b/doc/sphinx/reference/varnishtest.rst
@@ -48,6 +48,8 @@ The following options are available:
 
 -n iterations    Run tests this many times
 
+-p name=val      Pass parameters to all varnishd command lines
+
 -q               Quiet mode: report only failures
 
 -t duration      Time tests out after this long



From dridi.boukelmoune at gmail.com  Mon Feb 22 15:26:52 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 22 Feb 2016 16:26:52 +0100
Subject: [master] 238f56a Document how to test a build tree with varnishtest
Message-ID: <E1aXsNw-0000tE-9c@project.varnish-software.com>


commit 238f56a9461f25d23b7f197b1cb6b4df53d689d8
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Feb 22 16:19:16 2016 +0100

    Document how to test a build tree with varnishtest
    
    Autotools stuff is kept out of the picture, except pkg-config(1) which
    is essential for testing foreign code against a fresh Varnish build.

diff --git a/doc/sphinx/reference/varnishtest.rst b/doc/sphinx/reference/varnishtest.rst
index ba8708d..86990be 100644
--- a/doc/sphinx/reference/varnishtest.rst
+++ b/doc/sphinx/reference/varnishtest.rst
@@ -114,6 +114,62 @@ different requests. It will start a Varnish server (v1) and add the backend
 definition to the VCL specified (-vcl+backend). Finally it starts the
 c1-client, which is a single client sending two requests.
 
+TESTING A BUILD TREE
+====================
+
+Whether you are building a VMOD or trying to use one that you freshly built,
+you can tell ``varnishtest`` to pass a *vmod_path* to ``varnishd`` instances
+started using the ``varnish -start`` command in your test case::
+
+    varnishtest -p vmod_path=... /path/to/*.vtc
+
+This way you can use the same test cases on both installed and built VMODs::
+
+    server s1 {...} -start
+
+    varnish v1 -vcl+backend {
+        import wossname;
+
+        ...
+    } -start
+
+    ...
+
+You are not limited to the *vmod_path* and can pass any parameter, allowing
+you to run a build matrix without changing the test suite. You can achieve the
+same with macros, but then they need to be defined on each run.
+
+You can see the actual ``varnishd`` command lines in test outputs, they look
+roughly like this::
+
+    exec varnishd [varnishtest -p params] [testing params] [vtc -arg params]
+
+Parameters you define with ``varnishtest -p`` may be overriden by parameters
+needed by ``varnishtest`` to run properly, and they may in turn be overriden
+by parameters set in test scripts.
+
+There's also a special mode in which ``varnishtest`` builds itself a PATH and
+a *vmod_path* in order to find Varnish binaries (programs and VMODs) in the
+build tree surrounding the ``varnishtest`` binary. This is meant for testing
+of Varnish under development and will disregard your *vmod_path* if you set
+one.
+
+If you need to test your VMOD against a Varnish build tree, you must install
+it first, in a temp directory for instance. With information provided by the
+installation's *pkg-config(1)* you can build a proper PATH in order to access
+Varnish programs, and a *vmod_path* to access both your VMOD and the built-in
+VMODs::
+
+    export PKG_CONFIG_PATH=/path/to/install/lib/pkgconfig
+
+    BINDIR="$(pkg-config --variable=bindir varnishapi)"
+    SBINDIR="$(pkg-config --variable=sbindir varnishapi)"
+    PATH="SBINDIR:BINDIR:$PATH"
+
+    VMOD_PATH="<your-build-path>:$(pkg-config --variable=vmoddir varnishapi)"
+
+    varnishtest -p vmod_path="$VMOD_PATH" ...
+
 SEE ALSO
 ========
 



From dridi.boukelmoune at gmail.com  Mon Feb 22 15:36:54 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 22 Feb 2016 16:36:54 +0100
Subject: [master] d9e27bd s/VSB_printf/VSB_cat/
Message-ID: <E1aXsXe-0004GW-SM@project.varnish-software.com>


commit d9e27bda157e57fff60fbe622a4e354cf3974155
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Feb 22 16:36:22 2016 +0100

    s/VSB_printf/VSB_cat/

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 8df63ea..0afe475 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -406,7 +406,7 @@ varnish_launch(struct varnish *v)
 	VSB_printf(vsb, "cd ${pwd} &&");
 	VSB_printf(vsb, " exec varnishd %s -d -n %s",
 	    v->jail, v->workdir);
-	VSB_printf(vsb, VSB_data(params_vsb));
+	VSB_cat(vsb, VSB_data(params_vsb));
 	if (vtc_witness)
 		VSB_cat(vsb, " -p debug=+witness");
 	if (leave_temp)



From fgsch at lodoss.net  Tue Feb 23 09:42:38 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Tue, 23 Feb 2016 10:42:38 +0100
Subject: [master] b211dd2 Retire rollback
Message-ID: <E1aY9UM-0001XI-Mn@project.varnish-software.com>


commit b211dd2e2b8a0e2c469aa20a7665f5b2c9c18b6b
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Feb 22 13:15:41 2016 +0000

    Retire rollback
    
    This has been available via the std module for some time now.
    OK'd by phk at .

diff --git a/bin/varnishtest/tests/c00032.vtc b/bin/varnishtest/tests/c00032.vtc
deleted file mode 100644
index d5d06e3..0000000
--- a/bin/varnishtest/tests/c00032.vtc
+++ /dev/null
@@ -1,35 +0,0 @@
-varnishtest "Test Rollback"
-
-
-server s1 {
-	rxreq
-	expect req.url == "/foo"
-	expect req.http.foobar == "harck-coff"
-	txresp -status 400
-	accept
-	rxreq
-	expect req.url == "/bar"
-	expect req.http.foobar == "snark"
-	txresp -bodylen 5
-} -start
-
-varnish v1 -vcl+backend {
-	sub vcl_recv {
-		if (req.url == "/foo") {
-			set req.http.foobar = "harck-coff";
-		}
-	}
-
-	sub vcl_deliver {
-		if (resp.status == 400) {
-			rollback;
-			set req.url = "/bar";
-			return (restart);
-		}
-	}
-} -start
-
-client c1 {
-	txreq -url "/foo" -hdr "foobar: snark"
-	rxresp
-} -run
diff --git a/bin/varnishtest/tests/r01168.vtc b/bin/varnishtest/tests/r01168.vtc
index e26b2ce..ece1c9d 100644
--- a/bin/varnishtest/tests/r01168.vtc
+++ b/bin/varnishtest/tests/r01168.vtc
@@ -11,8 +11,9 @@ server s1 {
 } -start
 
 varnish v1 -vcl+backend {
+	import std;
 	sub vcl_recv {
-		rollback;
+		std.rollback(req);
 	}
 	sub vcl_backend_response {
 		set beresp.do_esi = true;
diff --git a/bin/varnishtest/tests/v00018.vtc b/bin/varnishtest/tests/v00018.vtc
index aae3723..9c653ba 100644
--- a/bin/varnishtest/tests/v00018.vtc
+++ b/bin/varnishtest/tests/v00018.vtc
@@ -2,12 +2,6 @@ varnishtest "VCL compiler coverage test: vcc_action.c"
 
 varnish v1 -vcl {
 	backend b { .host = "127.0.0.1"; }
-	sub vcl_hit { return (restart) ; }
-	sub vcl_miss { rollback; return (restart); }
-}
-
-varnish v1 -vcl {
-	backend b { .host = "127.0.0.1"; }
 	sub vcl_miss { return(synth(100,req.url)); }
 	sub vcl_hit { return(synth(100,"the butter please")); }
 }
diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst
index c982cb9..b8650f6 100644
--- a/doc/sphinx/reference/vcl.rst
+++ b/doc/sphinx/reference/vcl.rst
@@ -378,10 +378,6 @@ hash_data(input)
   Adds an input to the hash input. In the built-in VCL hash_data()
   is called on the host and URL of the *request*. Available in vcl_hash.
 
-rollback()
-  Restore *req* HTTP headers to their original state. This function is
-  deprecated.  Use std.rollback() instead.
-
 synthetic(STRING)
   Prepare a synthetic response body containing the STRING. Available in
   vcl_synth and vcl_backend_error.
diff --git a/lib/libvcc/vcc_action.c b/lib/libvcc/vcc_action.c
index 790efea..516f628 100644
--- a/lib/libvcc/vcc_action.c
+++ b/lib/libvcc/vcc_action.c
@@ -376,16 +376,6 @@ parse_return(struct vcc *tl)
 /*--------------------------------------------------------------------*/
 
 static void
-parse_rollback(struct vcc *tl)
-{
-
-	vcc_NextToken(tl);
-	Fb(tl, 1, "VRT_Rollback(ctx, VRT_r_req(ctx));\n");
-}
-
-/*--------------------------------------------------------------------*/
-
-static void
 parse_synthetic(struct vcc *tl)
 {
 	vcc_NextToken(tl);
@@ -419,7 +409,6 @@ static struct action_table {
 	{ "hash_data",		parse_hash_data, VCL_MET_HASH },
 	{ "new",		parse_new, VCL_MET_INIT},
 	{ "return",		parse_return },
-	{ "rollback",		parse_rollback },
 	{ "set",		parse_set },
 	{ "synthetic",		parse_synthetic,
 		VCL_MET_SYNTH | VCL_MET_BACKEND_ERROR },



From phk at FreeBSD.org  Tue Feb 23 09:51:13 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Feb 2016 10:51:13 +0100
Subject: [master] 0be53bc Make this slightly more understandable for FlexeLint
Message-ID: <E1aY9cf-0005F2-TJ@project.varnish-software.com>


commit 0be53bceafd8d57b803ef464e63690006a23633c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 23 09:50:56 2016 +0000

    Make this slightly more understandable for FlexeLint

diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c
index a13642f..7d1e9b1 100644
--- a/bin/varnishd/http1/cache_http1_proto.c
+++ b/bin/varnishd/http1/cache_http1_proto.c
@@ -364,13 +364,13 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp)
 
 	/* RFC2616, section 5.2, point 1 */
 	if (!strncasecmp(hp->hd[HTTP_HDR_URL].b, "http://", 7))
-		b = e = hp->hd[HTTP_HDR_URL].b + 7;
+		b = hp->hd[HTTP_HDR_URL].b + 7;
 	else if (FEATURE(FEATURE_HTTPS_SCHEME) &&
 	    !strncasecmp(hp->hd[HTTP_HDR_URL].b, "https://", 8))
-		b = e = hp->hd[HTTP_HDR_URL].b + 8;
+		b = hp->hd[HTTP_HDR_URL].b + 8;
 	if (b) {
-		while (*e != '/' && *e != '\0')
-			e++;
+		for (e = b; *e != '/' && *e != '\0'; e++)
+			continue;
 		if (*e == '/') {
 			http_Unset(hp, H_Host);
 			http_PrintfHeader(hp, "Host: %.*s", (int)(e - b), b);



From phk at FreeBSD.org  Tue Feb 23 10:07:44 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Feb 2016 11:07:44 +0100
Subject: [master] 543e45e Add an assert to ensure we never go into a simple
 stevedore with a request for zero bytes.
Message-ID: <E1aY9se-00078M-NP@project.varnish-software.com>


commit 543e45e8a7dc69701e79837a71a3acb8526f68dc
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 23 10:07:02 2016 +0000

    Add an assert to ensure we never go into a simple stevedore with
    a request for zero bytes.
    
    Suggested by:	github::onovy

diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 40e3b31..f985719 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -55,6 +55,7 @@ sml_stv_alloc(const struct stevedore *stv, size_t size)
 
 	for (;;) {
 		/* try to allocate from it */
+		assert(size > 0);
 		AN(stv->sml_alloc);
 		st = stv->sml_alloc(stv, size);
 		if (st != NULL)
@@ -132,6 +133,7 @@ SML_allocobj(struct worker *wrk, const struct stevedore *stv,
 				return (0);
 			really--;
 		}
+		AN(stv->sml_alloc);
 		st = stv->sml_alloc(stv, ltot);
 		if (st != NULL && st->space < ltot) {
 			stv->sml_free(st);
@@ -324,7 +326,6 @@ objallocwithnuke(struct worker *wrk, const struct stevedore *stv, size_t size)
 
 	for (fail = 0; fail <= cache_param->nuke_limit; fail++) {
 		/* try to allocate from it */
-		AN(stv->sml_alloc);
 		st = sml_stv_alloc(stv, size);
 		if (st != NULL)
 			break;



From dridi.boukelmoune at gmail.com  Tue Feb 23 10:59:39 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 23 Feb 2016 11:59:39 +0100
Subject: [master] e964656 Polish
Message-ID: <E1aYAgt-0006QG-VG@project.varnish-software.com>


commit e964656b545ef5cf9a89f98016340e47436cd955
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 11:58:48 2016 +0100

    Polish

diff --git a/doc/sphinx/reference/varnishtest.rst b/doc/sphinx/reference/varnishtest.rst
index 86990be..040b109 100644
--- a/doc/sphinx/reference/varnishtest.rst
+++ b/doc/sphinx/reference/varnishtest.rst
@@ -166,7 +166,8 @@ VMODs::
     SBINDIR="$(pkg-config --variable=sbindir varnishapi)"
     PATH="SBINDIR:BINDIR:$PATH"
 
-    VMOD_PATH="<your-build-path>:$(pkg-config --variable=vmoddir varnishapi)"
+    VMODDIR"$(pkg-config --variable=vmoddir varnishapi)"
+    VMOD_PATH="/path/to/your/vmod/build/dir:$VMODDIR"
 
     varnishtest -p vmod_path="$VMOD_PATH" ...
 



From phk at FreeBSD.org  Tue Feb 23 13:40:42 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Feb 2016 14:40:42 +0100
Subject: [master] 85b8a22 Tell stevedores about oc->state changes.
Message-ID: <E1aYDCk-0007cm-U0@project.varnish-software.com>


commit 85b8a22dbefa95397f8c7cffb033378c9f0646bf
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 23 13:39:58 2016 +0000

    Tell stevedores about oc->state changes.
    
    This allows Martin to stash attributes on going into BOS_STREAM.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 2b8691a..54e7ead 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -857,7 +857,8 @@ int ObjIterate(struct worker *, struct objcore *,
 int ObjGetSpace(struct worker *, struct objcore *, ssize_t *sz, uint8_t **ptr);
 void ObjExtend(struct worker *, struct objcore *, ssize_t l);
 uint64_t ObjWaitExtend(struct worker *, struct objcore *, uint64_t l);
-void ObjSetState(const struct objcore *, enum boc_state_e next);
+void ObjSetState(struct worker *, const struct objcore *,
+    enum boc_state_e next);
 void ObjWaitState(const struct objcore *, enum boc_state_e want);
 void ObjTrimStore(struct worker *, struct objcore *);
 void ObjTouch(struct worker *, struct objcore *, double now);
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 5b15581..ae6621c 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -202,7 +202,7 @@ vbf_stp_mkbereq(const struct worker *wrk, struct busyobj *bo)
 	bo->ws_bo = WS_Snapshot(bo->ws);
 	HTTP_Copy(bo->bereq, bo->bereq0);
 
-	ObjSetState(bo->fetch_objcore, BOS_REQ_DONE);
+	ObjSetState(bo->wrk, bo->fetch_objcore, BOS_REQ_DONE);
 	return (F_STP_STARTFETCH);
 }
 
@@ -670,7 +670,7 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 
 	if (bo->do_stream) {
 		HSH_Unbusy(wrk, bo->fetch_objcore);
-		ObjSetState(bo->fetch_objcore, BOS_STREAM);
+		ObjSetState(wrk, bo->fetch_objcore, BOS_STREAM);
 	}
 
 	VSLb(bo->vsl, SLT_Fetch_Body, "%u %s %s",
@@ -704,7 +704,7 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 	   give predictable backend reuse behavior for varnishtest */
 	VDI_Finish(bo->wrk, bo);
 
-	ObjSetState(bo->fetch_objcore, BOS_FINISHED);
+	ObjSetState(wrk, bo->fetch_objcore, BOS_FINISHED);
 	VSLb_ts_busyobj(bo, "BerespBody", W_TIM_real(wrk));
 	if (bo->stale_oc != NULL)
 		HSH_Kill(bo->stale_oc);
@@ -758,7 +758,7 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 
 	if (bo->do_stream) {
 		HSH_Unbusy(wrk, bo->fetch_objcore);
-		ObjSetState(bo->fetch_objcore, BOS_STREAM);
+		ObjSetState(wrk, bo->fetch_objcore, BOS_STREAM);
 	}
 
 	if (ObjIterate(wrk, bo->stale_oc, bo, vbf_objiterator))
@@ -780,7 +780,7 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 	   give predictable backend reuse behavior for varnishtest */
 	VDI_Finish(bo->wrk, bo);
 
-	ObjSetState(bo->fetch_objcore, BOS_FINISHED);
+	ObjSetState(wrk, bo->fetch_objcore, BOS_FINISHED);
 	VSLb_ts_busyobj(bo, "BerespBody", W_TIM_real(wrk));
 	return (F_STP_DONE);
 }
@@ -881,7 +881,7 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
 	VSB_delete(synth_body);
 
 	HSH_Unbusy(wrk, bo->fetch_objcore);
-	ObjSetState(bo->fetch_objcore, BOS_FINISHED);
+	ObjSetState(wrk, bo->fetch_objcore, BOS_FINISHED);
 	return (F_STP_DONE);
 }
 
@@ -903,7 +903,7 @@ vbf_stp_fail(struct worker *wrk, const struct busyobj *bo)
 		HSH_Kill(bo->fetch_objcore);
 	}
 	wrk->stats->fetch_failed++;
-	ObjSetState(bo->fetch_objcore, BOS_FAILED);
+	ObjSetState(wrk, bo->fetch_objcore, BOS_FAILED);
 	return (F_STP_DONE);
 }
 
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index cb32f97..a0d82b9 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -220,13 +220,21 @@ ObjWaitExtend(struct worker *wrk, struct objcore *oc, uint64_t l)
  */
 
 void
-ObjSetState(const struct objcore *oc, enum boc_state_e next)
+ObjSetState(struct worker *wrk, const struct objcore *oc, enum boc_state_e next)
 {
+	const struct obj_methods *om;
 
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
-
 	assert(next > oc->boc->state);
+
+	CHECK_OBJ_ORNULL(oc->stobj->stevedore, STEVEDORE_MAGIC);
+	if (oc->stobj->stevedore != NULL) {
+		om = oc->stobj->stevedore->methods;
+		if (om->objsetstate != NULL)
+			om->objsetstate(wrk, oc, next);
+	}
+
 	Lck_Lock(&oc->boc->mtx);
 	oc->boc->state = next;
 	AZ(pthread_cond_broadcast(&oc->boc->cond));
diff --git a/bin/varnishd/cache/cache_obj.h b/bin/varnishd/cache/cache_obj.h
index c4eea09..0b0e2ca 100644
--- a/bin/varnishd/cache/cache_obj.h
+++ b/bin/varnishd/cache/cache_obj.h
@@ -32,7 +32,8 @@
 
 typedef void objfree_f(struct worker *, struct objcore *);
 
-/* This method is only used by SML (...to get to persistent) */
+typedef void objsetstate_f(struct worker *, const struct objcore *,
+    enum boc_state_e);
 
 typedef int objiterator_f(struct worker *, struct objcore *,
     void *priv, objiterate_f *func);
@@ -59,5 +60,6 @@ struct obj_methods {
 	objgetattr_f	*objgetattr;
 	objsetattr_f	*objsetattr;
 	objtouch_f	*objtouch;
+	objsetstate_f	*objsetstate;
 };
 



From phk at FreeBSD.org  Tue Feb 23 15:59:58 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Feb 2016 16:59:58 +0100
Subject: [master] b1f0b29 Don't abuse OA_LEN to assemble the object,
 but set it once, and only once, once we know it.
Message-ID: <E1aYFNW-0007ZF-CQ@project.varnish-software.com>


commit b1f0b29e660f9d45e4309b21c663708087109776
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 23 15:59:19 2016 +0000

    Don't abuse OA_LEN to assemble the object, but set it once, and
    only once, once we know it.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 54e7ead..42bd867 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -379,6 +379,7 @@ struct boc {
 	void			*stevedore_priv;
 	enum boc_state_e	state;
 	uint8_t			*vary;
+	uint64_t		len_so_far;
 };
 
 /* Object core structure ---------------------------------------------
@@ -856,7 +857,8 @@ int ObjIterate(struct worker *, struct objcore *,
     void *priv, objiterate_f *func);
 int ObjGetSpace(struct worker *, struct objcore *, ssize_t *sz, uint8_t **ptr);
 void ObjExtend(struct worker *, struct objcore *, ssize_t l);
-uint64_t ObjWaitExtend(struct worker *, struct objcore *, uint64_t l);
+uint64_t ObjWaitExtend(const struct worker *, const struct objcore *,
+    uint64_t l);
 void ObjSetState(struct worker *, const struct objcore *,
     enum boc_state_e next);
 void ObjWaitState(const struct objcore *, enum boc_state_e want);
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index ae6621c..83945b3 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -627,6 +627,8 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 	if (bo->htc->body_status == BS_NONE)
 		bo->do_stream = 0;
 
+	bo->fetch_objcore->boc->len_so_far = 0;
+
 	if (VFP_Open(bo->vfc)) {
 		(void)VFP_Error(bo->vfc, "Fetch pipeline failed to open");
 		bo->htc->doclose = SC_RX_BODY;
@@ -693,6 +695,9 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 		}
 	}
 
+	AZ(ObjSetU64(wrk, bo->fetch_objcore, OA_LEN,
+	    bo->fetch_objcore->boc->len_so_far));
+
 	if (bo->do_stream)
 		assert(bo->fetch_objcore->boc->state == BOS_STREAM);
 	else {
@@ -726,8 +731,7 @@ vbf_objiterator(void *priv, int flush, const void *ptr, ssize_t len)
 	CAST_OBJ_NOTNULL(bo, priv, BUSYOBJ_MAGIC);
 
 	while (len > 0) {
-		l = ObjGetLen(bo->wrk, bo->stale_oc);
-		assert(l > 0);
+		l = len;
 		if (VFP_GetStorage(bo->vfc, &l, &pd) != VFP_OK)
 			return (1);
 		if (len < l)
@@ -771,6 +775,9 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 		return (F_STP_FAIL);
 	}
 
+	AZ(ObjSetU64(wrk, bo->fetch_objcore, OA_LEN,
+	    bo->fetch_objcore->boc->len_so_far));
+
 	if (!bo->do_stream)
 		HSH_Unbusy(wrk, bo->fetch_objcore);
 
@@ -878,8 +885,8 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
 		ll -= l;
 		o += l;
 	}
+	AZ(ObjSetU64(wrk, bo->fetch_objcore, OA_LEN, VSB_len(synth_body) - ll));
 	VSB_delete(synth_body);
-
 	HSH_Unbusy(wrk, bo->fetch_objcore);
 	ObjSetState(wrk, bo->fetch_objcore, BOS_FINISHED);
 	return (F_STP_DONE);
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index a0d82b9..a930fc9 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -175,19 +175,15 @@ void
 ObjExtend(struct worker *wrk, struct objcore *oc, ssize_t l)
 {
 	const struct obj_methods *om = obj_getmethods(oc);
-	uint64_t len;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 	assert(l > 0);
 
-	AZ(ObjGetU64(wrk, oc, OA_LEN, &len));
-	len += l;
-
 	Lck_Lock(&oc->boc->mtx);
 	AN(om->objextend);
 	om->objextend(wrk, oc, l);
-	AZ(ObjSetU64(wrk, oc, OA_LEN, len));
+	oc->boc->len_so_far += l;
 	Lck_Unlock(&oc->boc->mtx);
 	AZ(pthread_cond_broadcast(&oc->boc->cond));
 }
@@ -196,7 +192,7 @@ ObjExtend(struct worker *wrk, struct objcore *oc, ssize_t l)
  */
 
 uint64_t
-ObjWaitExtend(struct worker *wrk, struct objcore *oc, uint64_t l)
+ObjWaitExtend(const struct worker *wrk, const struct objcore *oc, uint64_t l)
 {
 	uint64_t rv;
 
@@ -204,14 +200,14 @@ ObjWaitExtend(struct worker *wrk, struct objcore *oc, uint64_t l)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 	Lck_Lock(&oc->boc->mtx);
-	AZ(ObjGetU64(wrk, oc, OA_LEN, &rv));
 	while (1) {
+		rv = oc->boc->len_so_far;
 		assert(l <= rv || oc->boc->state == BOS_FAILED);
 		if (rv > l || oc->boc->state >= BOS_FINISHED)
 			break;
 		(void)Lck_CondWait(&oc->boc->cond, &oc->boc->mtx, 0);
-		AZ(ObjGetU64(wrk, oc, OA_LEN, &rv));
 	}
+	rv = oc->boc->len_so_far;
 	Lck_Unlock(&oc->boc->mtx);
 	return (rv);
 }
@@ -220,7 +216,8 @@ ObjWaitExtend(struct worker *wrk, struct objcore *oc, uint64_t l)
  */
 
 void
-ObjSetState(struct worker *wrk, const struct objcore *oc, enum boc_state_e next)
+ObjSetState(struct worker *wrk, const struct objcore *oc,
+    enum boc_state_e next)
 {
 	const struct obj_methods *om;
 
@@ -229,6 +226,8 @@ ObjSetState(struct worker *wrk, const struct objcore *oc, enum boc_state_e next)
 	assert(next > oc->boc->state);
 
 	CHECK_OBJ_ORNULL(oc->stobj->stevedore, STEVEDORE_MAGIC);
+	AN(next != BOS_FINISHED || (oc->oa_present & (1 << OA_LEN)));
+
 	if (oc->stobj->stevedore != NULL) {
 		om = oc->stobj->stevedore->methods;
 		if (om->objsetstate != NULL)
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 733b21d..991883f 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -162,7 +162,6 @@ cnt_synth(struct worker *wrk, struct req *req)
 	if (req->err_code < 100 || req->err_code > 999)
 		req->err_code = 501;
 
-
 	HTTP_Setup(req->resp, req->ws, req->vsl, SLT_RespMethod);
 	h = req->resp;
 	http_TimeHeader(h, "Date: ", now);
@@ -208,6 +207,8 @@ cnt_synth(struct worker *wrk, struct req *req)
 		}
 	}
 
+	if (szl >= 0)
+		AZ(ObjSetU64(wrk, req->objcore, OA_LEN, szl));
 	HSH_DerefBoc(wrk, req->objcore);
 	VSB_delete(synth_body);
 



From martin at varnish-software.com  Tue Feb 23 16:44:46 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 23 Feb 2016 17:44:46 +0100
Subject: [master] 1732352 If the OC doesn't have a boc,
 don't lock unnecessarily lock the OH
Message-ID: <E1aYG4s-0005cx-Nv@project.varnish-software.com>


commit 17323520af6afc72261fc43bdd7ded803497f8d6
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Feb 23 14:39:41 2016 +0100

    If the OC doesn't have a boc, don't lock unnecessarily lock the OH
    
    The objcore won't gain a boc in a way that would call for locking the
    OH when checking if it wasn't there in the first place. So if the
    oc->boc is NULL, return early. This saves a mutex lock/unlock in the
    common case.

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index c70ef7b..797677b 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -749,6 +749,8 @@ HSH_RefBoc(const struct objcore *oc)
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	oh = oc->objhead;
 	CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC);
+	if (oc->boc == NULL)
+		return (NULL);
 	Lck_Lock(&oh->mtx);
 	assert(oc->refcnt > 0);
 	boc = oc->boc;



From martin at varnish-software.com  Tue Feb 23 16:44:46 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 23 Feb 2016 17:44:46 +0100
Subject: [master] 87921c1 Constify return from ObjGetattr
Message-ID: <E1aYG4s-0005d3-PP@project.varnish-software.com>


commit 87921c1bf31e35da0eff19309a421d6fd794f916
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Wed Feb 17 20:34:24 2016 +0100

    Constify return from ObjGetattr
    
    This is to make it clear that one can not change an object attribute
    through the returned pointer from ObjGetattr()

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 42bd867..d288bab 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -869,7 +869,7 @@ uint64_t ObjGetLen(struct worker *, struct objcore *);
 void ObjFreeObj(struct worker *, struct objcore *);
 void ObjSlim(struct worker *, struct objcore *);
 int ObjHasAttr(struct worker *, struct objcore *, enum obj_attr);
-void *ObjGetAttr(struct worker *, struct objcore *, enum obj_attr,
+const void *ObjGetAttr(struct worker *, struct objcore *, enum obj_attr,
     ssize_t *len);
 void *ObjSetAttr(struct worker *, struct objcore *, enum obj_attr,
     ssize_t len, const void *);
diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index e6eb949..bbed50d 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -54,8 +54,8 @@ static const uint8_t gzip_hdr[] = {
 struct ecx {
 	unsigned	magic;
 #define ECX_MAGIC	0x0b0f9163
-	uint8_t		*p;
-	uint8_t		*e;
+	const uint8_t	*p;
+	const uint8_t	*e;
 	int		state;
 	ssize_t		l;
 	int		isgzip;
@@ -191,9 +191,9 @@ ved_include(struct req *preq, const char *src, const char *host,
 #define Debug(fmt, ...) /**/
 
 static ssize_t
-ved_decode_len(struct req *req, uint8_t **pp)
+ved_decode_len(struct req *req, const uint8_t **pp)
 {
-	uint8_t *p;
+	const uint8_t *p;
 	ssize_t l;
 
 	p = *pp;
@@ -675,7 +675,7 @@ static void
 ved_stripgzip(struct req *req, const struct boc *boc)
 {
 	ssize_t l;
-	char *p;
+	const char *p;
 	uint32_t icrc;
 	uint32_t ilen;
 	uint8_t *dbits;
diff --git a/bin/varnishd/cache/cache_gzip.c b/bin/varnishd/cache/cache_gzip.c
index 98523a0..97082db 100644
--- a/bin/varnishd/cache/cache_gzip.c
+++ b/bin/varnishd/cache/cache_gzip.c
@@ -285,7 +285,7 @@ VDP_gunzip(struct req *req, enum vdp_action act, void **priv,
 	const void *dp;
 	struct worker *wrk;
 	struct vgz *vg;
-	char *p;
+	const char *p;
 	uint64_t u;
 
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 797677b..117204b 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -320,7 +320,7 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 	double exp_t_origin;
 	int busy_found;
 	enum lookup_e retval;
-	uint8_t *vary;
+	const uint8_t *vary;
 
 	AN(ocp);
 	*ocp = NULL;
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index a930fc9..13f814a 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -371,7 +371,7 @@ ObjHasAttr(struct worker *wrk, struct objcore *oc, enum obj_attr attr)
  * Get an attribute of the object.
  */
 
-void *
+const void *
 ObjGetAttr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
    ssize_t *len)
 {
@@ -427,7 +427,8 @@ int
 ObjCopyAttr(struct worker *wrk, struct objcore *oc, struct objcore *ocs,
     enum obj_attr attr)
 {
-	void *vps, *vpd;
+	const void *vps;
+	void *vpd;
 	ssize_t l;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
@@ -480,7 +481,7 @@ ObjSetDouble(struct worker *wrk, struct objcore *oc, enum obj_attr a, double t)
 int
 ObjGetDouble(struct worker *wrk, struct objcore *oc, enum obj_attr a, double *d)
 {
-	void *vp;
+	const void *vp;
 	uint64_t u;
 	ssize_t l;
 
@@ -514,7 +515,7 @@ ObjSetU64(struct worker *wrk, struct objcore *oc, enum obj_attr a, uint64_t t)
 int
 ObjGetU64(struct worker *wrk, struct objcore *oc, enum obj_attr a, uint64_t *d)
 {
-	void *vp;
+	const void *vp;
 	ssize_t l;
 
 	vp = ObjGetAttr(wrk, oc, a, &l);
@@ -540,7 +541,7 @@ ObjSetU32(struct worker *wrk, struct objcore *oc, enum obj_attr a, uint32_t t)
 int
 ObjGetU32(struct worker *wrk, struct objcore *oc, enum obj_attr a, uint32_t *d)
 {
-	void *vp;
+	const void *vp;
 	ssize_t l;
 
 	vp = ObjGetAttr(wrk, oc, a, &l);
@@ -557,7 +558,7 @@ ObjGetU32(struct worker *wrk, struct objcore *oc, enum obj_attr a, uint32_t *d)
 int
 ObjCheckFlag(struct worker *wrk, struct objcore *oc, enum obj_flags of)
 {
-	uint8_t *fp;
+	const uint8_t *fp;
 
 	fp = ObjGetAttr(wrk, oc, OA_FLAGS, NULL);
 	AN(fp);
diff --git a/bin/varnishd/cache/cache_obj.h b/bin/varnishd/cache/cache_obj.h
index 0b0e2ca..45f7e61 100644
--- a/bin/varnishd/cache/cache_obj.h
+++ b/bin/varnishd/cache/cache_obj.h
@@ -43,7 +43,7 @@ typedef void objextend_f(struct worker *, struct objcore *, ssize_t l);
 typedef void objtrimstore_f(struct worker *, struct objcore *);
 typedef void objstable_f(struct worker *, struct objcore *, struct boc *);
 typedef void objslim_f(struct worker *, struct objcore *);
-typedef void *objgetattr_f(struct worker *, struct objcore *,
+typedef const void *objgetattr_f(struct worker *, struct objcore *,
     enum obj_attr attr, ssize_t *len);
 typedef void *objsetattr_f(struct worker *, struct objcore *,
     enum obj_attr attr, ssize_t len, const void *ptr);
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index f985719..f2f2111 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -476,7 +476,7 @@ sml_stable(struct worker *wrk, struct objcore *oc, struct boc *boc)
 		LRU_Add(oc, wrk->lastused);	// approx timestamp is OK
 }
 
-static void * __match_proto__(objgetattr_f)
+static const void * __match_proto__(objgetattr_f)
 sml_getattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
    ssize_t *len)
 {



From martin at varnish-software.com  Tue Feb 23 16:44:46 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 23 Feb 2016 17:44:46 +0100
Subject: [master] 79826a6 Assert that Vary setting succeeds
Message-ID: <E1aYG4s-0005dD-Qe@project.varnish-software.com>


commit 79826a6ae5e25bdcd731dea1b3eb205133ed80aa
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Oct 30 10:50:22 2015 +0100

    Assert that Vary setting succeeds

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 83945b3..8febc5e 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -129,8 +129,8 @@ vbf_beresp2obj(struct busyobj *bo)
 		return (-1);
 
 	if (vary != NULL) {
-		b = ObjSetAttr(bo->wrk, bo->fetch_objcore, OA_VARY, varyl,
-		    VSB_data(vary));
+		AN(ObjSetAttr(bo->wrk, bo->fetch_objcore, OA_VARY, varyl,
+			VSB_data(vary)));
 		VSB_delete(vary);
 	}
 



From martin at varnish-software.com  Tue Feb 23 16:44:46 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 23 Feb 2016 17:44:46 +0100
Subject: [master] ef95a08 Implement three different types of attributes and
 generate get/set code from table
Message-ID: <E1aYG4s-0005dM-S3@project.varnish-software.com>


commit ef95a084055f1c80754cbf0eac38ec79556a8eae
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Oct 30 10:36:42 2015 +0100

    Implement three different types of attributes and generate get/set code from table
    
    Fixed size attributes has it's space always reserved.
    
    Variable size attributes change in size, and their combined total
    length needs to be given to allocobj.
    
    Auxiliary attributes do not need to be predeclared.
    
    The code for setting and getting the attributes is generated from a
    table that lists the different attributes according to their type.
    
    Add documentation about the behaviour of the get/set functions.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index d288bab..d735df2 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -391,9 +391,14 @@ struct boc {
  */
 
 enum obj_attr {
-#define OBJ_ATTR(U, l)  OA_##U,
+#define OBJ_FIXATTR(U, l, s)	OA_##U,
+#define OBJ_VARATTR(U, l)	OA_##U,
+#define OBJ_AUXATTR(U, l)	OA_##U,
 #include "tbl/obj_attr.h"
-#undef OBJ_ATTR
+#undef OBJ_AUXATTR
+#undef OBJ_VARATTR
+#undef OBJ_FIXATTR
+				OA__MAX,
 };
 
 enum obj_flags {
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 13f814a..57961cf 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -369,6 +369,10 @@ ObjHasAttr(struct worker *wrk, struct objcore *oc, enum obj_attr attr)
  * ObjGetAttr()
  *
  * Get an attribute of the object.
+ *
+ * Returns NULL on unset or zero length attributes and len set to
+ * zero. Returns Non-NULL otherwise and len is updated with the attributes
+ * length.
  */
 
 const void *
@@ -386,8 +390,23 @@ ObjGetAttr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
 /*====================================================================
  * ObjSetAttr()
  *
+ * Setting fixed size attributes always succeeds.
+ *
+ * Setting a variable size attribute asserts if the combined size of the
+ * variable attributes exceeds the total variable attribute space set at
+ * object creation. If there is space it always succeeds.
+ *
+ * Setting an auxiliary attribute can fail.
+ *
+ * Resetting any variable asserts if the new length does not match the
+ * previous length exactly.
+ *
  * If ptr is Non-NULL, it points to the new content which is copied into
  * the attribute.  Otherwise the caller will have to do the copying.
+ *
+ * Return value is non-NULL on success and NULL on failure. If ptr was
+ * non-NULL, it is an error to use the returned pointer to set the
+ * attribute data, it is only a success indicator in that case.
  */
 
 void *
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 78b7e82..9478702 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -435,7 +435,7 @@ smp_sml_getobj(struct worker *wrk, struct objcore *oc)
 				break;
 			l += st->len;
 		}
-		if (l != vbe64dec(o->oa_len))
+		if (l != vbe64dec(o->fa_len))
 			bad |= 0x100;
 
 		if(bad) {
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index f2f2111..8b862db 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -187,10 +187,14 @@ sml_slim(struct worker *wrk, struct objcore *oc)
 	o = sml_getobj(wrk, oc);
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 
-	if (o->esidata != NULL) {
-		sml_stv_free(stv, o->esidata);
-		o->esidata = NULL;
+#define OBJ_AUXATTR(U, l)						\
+	if (o->aa_##l != NULL) {					\
+		sml_stv_free(stv, o->aa_##l);				\
+		o->aa_##l = NULL;					\
 	}
+#include "tbl/obj_attr.h"
+#undef OBJ_AUXATTR
+
 	VTAILQ_FOREACH_SAFE(st, &o->list, list, stn) {
 		CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
 		VTAILQ_REMOVE(&o->list, st, list);
@@ -489,33 +493,37 @@ sml_getattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
 		len = &dummy;
 	o = sml_getobj(wrk, oc);
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
+
 	switch (attr) {
-	case OA_ESIDATA:
-		if (o->esidata == NULL)
-			return (NULL);
-		*len = o->esidata->len;
-		return (o->esidata->ptr);
-	case OA_FLAGS:
-		*len = sizeof o->oa_flags;
-		return (o->oa_flags);
-	case OA_GZIPBITS:
-		*len = sizeof o->oa_gzipbits;
-		return (o->oa_gzipbits);
-	case OA_HEADERS:
-		*len = 0;			// XXX: hack
-		return (o->oa_http);
-	case OA_LASTMODIFIED:
-		*len = sizeof o->oa_lastmodified;
-		return (o->oa_lastmodified);
-	case OA_VARY:
-		*len = 4;			// XXX: hack
-		return (o->oa_vary);
-	case OA_LEN:
-		*len = sizeof o->oa_len;
-		return (o->oa_len);
-	case OA_VXID:
-		*len = sizeof o->oa_vxid;
-		return (o->oa_vxid);
+		/* Fixed size attributes */
+#define OBJ_FIXATTR(U, l, s)						\
+	case OA_##U:							\
+		*len = sizeof o->fa_##l;				\
+		return (o->fa_##l);
+#include "tbl/obj_attr.h"
+#undef OBJ_FIXATTR
+
+		/* Variable size attributes */
+#define OBJ_VARATTR(U, l)						\
+	case OA_##U:							\
+		if (o->va_##l == NULL)					\
+			return (NULL);					\
+		*len = o->va_##l##_len;					\
+		return (o->va_##l);
+#include "tbl/obj_attr.h"
+#undef OBJ_VARATTR
+
+		/* Auxiliary attributes */
+#define OBJ_AUXATTR(U, l)						\
+	case OA_##U:							\
+		if (o->aa_##l == NULL)					\
+			return (NULL);					\
+		CHECK_OBJ_NOTNULL(o->aa_##l, STORAGE_MAGIC);		\
+		*len = o->aa_##l->len;					\
+		return (o->aa_##l->ptr);
+#include "tbl/obj_attr.h"
+#undef OBJ_AUXATTR
+
 	default:
 		break;
 	}
@@ -535,53 +543,63 @@ sml_setattr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
 	o = sml_getobj(wrk, oc);
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 	st = o->objstore;
+
 	switch (attr) {
-	case OA_ESIDATA:
-		o->esidata = objallocwithnuke(wrk, oc->stobj->stevedore, len);
-		if (o->esidata == NULL)
-			return (NULL);
-		o->esidata->len = len;
-		retval = o->esidata->ptr;
-		break;
-	case OA_FLAGS:
-		assert(len == sizeof o->oa_flags);
-		retval = o->oa_flags;
-		break;
-	case OA_GZIPBITS:
-		assert(len == sizeof o->oa_gzipbits);
-		retval = o->oa_gzipbits;
+		/* Fixed size attributes */
+#define OBJ_FIXATTR(U, l, s)						\
+	case OA_##U:							\
+		assert(len == sizeof o->fa_##l);			\
+		retval = o->fa_##l;					\
 		break;
-	case OA_HEADERS:
-		len = PRNDUP(len);
-		assert(st->len + len <= st->space);
-		o->oa_http = (void*)(st->ptr + st->len);
-		st->len += len;
-		retval = o->oa_http;
+#include "tbl/obj_attr.h"
+#undef OBJ_FIXATTR
+
+		/* Variable size attributes */
+#define OBJ_VARATTR(U, l)						\
+	case OA_##U:							\
+		if (o->va_##l##_len > 0) {				\
+			AN(o->va_##l);					\
+			assert(len == o->va_##l##_len);			\
+			retval = o->va_##l;				\
+		} else if (len > 0) {					\
+			assert(len <= UINT_MAX);			\
+			assert(st->len + len <= st->space);		\
+			o->va_##l = st->ptr + st->len;			\
+			st->len += len;					\
+			retval = o->va_##l;				\
+		}							\
 		break;
-	case OA_LASTMODIFIED:
-		assert(len == sizeof o->oa_lastmodified);
-		retval = o->oa_lastmodified;
-		break;
-	case OA_VARY:
-		len = PRNDUP(len);
-		assert(st->len + len <= st->space);
-		o->oa_vary = (void*)(st->ptr + st->len);
-		st->len += len;
-		retval = o->oa_vary;
-		break;
-	case OA_LEN:
-		assert(len == sizeof o->oa_len);
-		retval = o->oa_len;
-		break;
-	case OA_VXID:
-		assert(len == sizeof o->oa_vxid);
-		retval = o->oa_vxid;
+#include "tbl/obj_attr.h"
+#undef OBJ_VARATTR
+
+		/* Auxiliary attributes */
+#define OBJ_AUXATTR(U, l)						\
+	case OA_##U:							\
+		if (o->aa_##l != NULL) {				\
+			CHECK_OBJ_NOTNULL(o->aa_##l, STORAGE_MAGIC);	\
+			assert(len == o->aa_##l->len);			\
+			retval = o->aa_##l->ptr;			\
+			break;						\
+		}							\
+		if (len == 0)						\
+			break;						\
+		o->aa_##l = objallocwithnuke(wrk, oc->stobj->stevedore,	len); \
+		if (o->aa_##l == NULL)					\
+			break;						\
+		CHECK_OBJ_NOTNULL(o->aa_##l, STORAGE_MAGIC);		\
+		assert(len <= o->aa_##l->space);			\
+		o->aa_##l->len = len;					\
+		retval = o->aa_##l->ptr;				\
 		break;
+#include "tbl/obj_attr.h"
+#undef OBJ_AUXATTR
+
 	default:
 		WRONG("Unsupported OBJ_ATTR");
 		break;
 	}
-	if (ptr != NULL)
+
+	if (retval != NULL && ptr != NULL)
 		memcpy(retval, ptr, len);
 	return (retval);
 }
diff --git a/bin/varnishd/storage/storage_simple.h b/bin/varnishd/storage/storage_simple.h
index 84a3277..53d2b94 100644
--- a/bin/varnishd/storage/storage_simple.h
+++ b/bin/varnishd/storage/storage_simple.h
@@ -40,17 +40,29 @@ struct object {
 #define OBJECT_MAGIC		0x32851d42
 	struct storage		*objstore;
 
-	uint8_t			oa_len[8];
-	uint8_t			oa_vxid[4];
-	uint8_t			*oa_vary;
-	uint8_t			*oa_http;
-	uint8_t			oa_flags[1];
-	char			oa_gzipbits[32];
-	char			oa_lastmodified[8];
+	/* Fixed size attributes */
+#define OBJ_FIXATTR(U, l, s)			\
+	uint8_t			fa_##l[s];
+#include "tbl/obj_attr.h"
+#undef OBJ_FIXATTR
 
-	struct storagehead	list;
+	/* Variable size attributes */
+#define OBJ_VARATTR(U, l)			\
+	uint8_t			*va_##l;
+#include "tbl/obj_attr.h"
+#undef OBJ_VARATTR
+#define OBJ_VARATTR(U, l)			\
+	unsigned		va_##l##_len;
+#include "tbl/obj_attr.h"
+#undef OBJ_VARATTR
+
+	/* Auxiliary attributes */
+#define OBJ_AUXATTR(U, l)			\
+	struct storage		*aa_##l;
+#include "tbl/obj_attr.h"
+#undef OBJ_AUXATTR
 
-	struct storage		*esidata;
+	struct storagehead	list;
 };
 
 extern const struct obj_methods SML_methods;
@@ -59,4 +71,3 @@ struct object *SML_MkObject(const struct stevedore *, struct objcore *,
     void *ptr);
 
 storage_allocobj_f SML_allocobj;
-
diff --git a/include/tbl/obj_attr.h b/include/tbl/obj_attr.h
index 310ffab..d05b598 100644
--- a/include/tbl/obj_attr.h
+++ b/include/tbl/obj_attr.h
@@ -29,17 +29,24 @@
 
 /*lint -save -e525 -e539 */
 
+/* upper, lower, size */
+#ifdef OBJ_FIXATTR
+OBJ_FIXATTR(LEN, len, 8)
+OBJ_FIXATTR(VXID, vxid, 4)
+OBJ_FIXATTR(FLAGS, flags, 1)
+OBJ_FIXATTR(GZIPBITS, gzipbits, 32)
+OBJ_FIXATTR(LASTMODIFIED, lastmodified, 8)
+#endif
+
+/* upper, lower */
+#ifdef OBJ_VARATTR
+OBJ_VARATTR(VARY, vary)
+OBJ_VARATTR(HEADERS, headers)
+#endif
+
 /* upper, lower */
-#ifdef OBJ_ATTR
-OBJ_ATTR(LEN,		len)
-OBJ_ATTR(VXID,		vxid)
-OBJ_ATTR(EXP,		exp)
-OBJ_ATTR(VARY,		vary)
-OBJ_ATTR(HEADERS,	headers)
-OBJ_ATTR(FLAGS,		flags)
-OBJ_ATTR(GZIPBITS,	gzipbits)
-OBJ_ATTR(ESIDATA,	esidata)
-OBJ_ATTR(LASTMODIFIED,	lastmodified)
+#ifdef OBJ_AUXATTR
+OBJ_AUXATTR(ESIDATA, esidata)
 #endif
 
 #ifdef OBJ_FLAG



From martin at varnish-software.com  Tue Feb 23 16:44:46 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 23 Feb 2016 17:44:46 +0100
Subject: [master] d12625f Do not set the oa_present bit when ObjSetAttr
 returns failure
Message-ID: <E1aYG4s-0005dW-TR@project.varnish-software.com>


commit d12625f3c5cc4851b13519503f8e51269e7b0407
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Feb 23 15:43:30 2016 +0100

    Do not set the oa_present bit when ObjSetAttr returns failure

diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 6cc92dd..9bdd92e 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -180,7 +180,7 @@ exp_inbox(struct exp_priv *ep, struct objcore *oc, unsigned flags)
 	    oc->timer_when, oc->flags);
 
 	if (oc->flags & OC_F_DYING) {
-		oc->exp_flags &= OC_EF_EXP;
+		oc->exp_flags &= ~OC_EF_EXP;
 		VSLb(&ep->vsl, SLT_ExpKill, "EXP_Kill p=%p e=%.9f f=0x%x", oc,
 		    oc->timer_when, oc->flags);
 		if (!(flags & OC_EF_INSERT)) {
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 57961cf..0248ee7 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -414,14 +414,17 @@ ObjSetAttr(struct worker *wrk, struct objcore *oc, enum obj_attr attr,
     ssize_t len, const void *ptr)
 {
 	const struct obj_methods *om = obj_getmethods(oc);
+	void *r;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 
 	AN(om->objsetattr);
 	assert((int)attr < 16);
-	oc->oa_present |= (1 << attr);
-	return (om->objsetattr(wrk, oc, attr, len, ptr));
+	r = om->objsetattr(wrk, oc, attr, len, ptr);
+	if (r)
+		oc->oa_present |= (1 << attr);
+	return (r);
 }
 
 /*====================================================================



From martin at varnish-software.com  Tue Feb 23 16:44:46 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 23 Feb 2016 17:44:46 +0100
Subject: [master] eacc469 Merge with "Add missing not..."
Message-ID: <E1aYG4s-0005df-UV@project.varnish-software.com>


commit eacc46946260cdc2a620edde97f6a3c3e8a572f9
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Feb 23 17:40:10 2016 +0100

    Merge with "Add missing not..."

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 117204b..4ff82be 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -828,6 +828,8 @@ HSH_DerefObjCore(struct worker *wrk, struct objcore **ocp)
 	if (r != 0)
 		return (r);
 
+	AZ(oc->exp_flags);
+
 	BAN_DestroyObj(oc);
 	AZ(oc->ban);
 



From martin at varnish-software.com  Tue Feb 23 16:44:46 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 23 Feb 2016 17:44:46 +0100
Subject: [master] bfb5813 Split BAN_RefBan into BAN_FindBan and BAN_RefBan
Message-ID: <E1aYG4s-0005do-Ve@project.varnish-software.com>


commit bfb5813ef441c1da6d89603571d34a2edf9bc29c
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Feb 23 16:23:58 2016 +0100

    Split BAN_RefBan into BAN_FindBan and BAN_RefBan
    
    BAN_FindBan looks up a ban based on a ban timestamp and returns a
    pointer to the ban. Returns NULL if no ban with that timestamp is
    present.
    
    BAN_RefBan takes an objcore and a ban. It grabs a reference to the ban
    and associates the objcore with that ban.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index d735df2..cbeccbf 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -672,7 +672,8 @@ void BAN_Abandon(struct ban_proto *b);
 void BAN_Hold(void);
 void BAN_Release(void);
 void BAN_Reload(const uint8_t *ban, unsigned len);
-struct ban *BAN_RefBan(struct objcore *oc, double t0);
+struct ban *BAN_FindBan(double t0);
+void BAN_RefBan(struct objcore *oc, struct ban *);
 double BAN_Time(const struct ban *ban);
 
 /* cache_busyobj.c */
diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index f091abf..aa4c68f 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -249,29 +249,45 @@ BAN_DestroyObj(struct objcore *oc)
 }
 
 /*--------------------------------------------------------------------
- * Find and/or Grab a reference to an objects ban based on timestamp
+ * Find a ban based on timestamp a ban timestamp.
  * Assume we have a BAN_Hold, so list traversal is safe.
  */
 
 struct ban *
-BAN_RefBan(struct objcore *oc, double t0)
+BAN_FindBan(double t0)
 {
 	struct ban *b;
-	double t1 = 0;
+	double t1;
 
+	assert(ban_holds > 0);
 	VTAILQ_FOREACH(b, &ban_head, list) {
 		t1 = ban_time(b->spec);
+		if (t1 == t0)
+			return (b);
 		if (t1 <= t0)
 			break;
 	}
-	AN(b);
-	assert(t1 == t0);
+	return (NULL);
+}
+
+/*--------------------------------------------------------------------
+ * Grab a reference to a ban and associate the objcore with that ban.
+ * Assume we have a BAN_Hold, so list traversal is safe.
+ */
+
+void
+BAN_RefBan(struct objcore *oc, struct ban *b)
+{
+
 	Lck_Lock(&ban_mtx);
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AZ(oc->ban);
+	CHECK_OBJ_NOTNULL(b, BAN_MAGIC);
 	assert(ban_holds > 0);
 	b->refcount++;
 	VTAILQ_INSERT_TAIL(&b->objcore, oc, ban_list);
+	oc->ban = b;
 	Lck_Unlock(&ban_mtx);
-	return (b);
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 9478702..0e1637f 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -133,6 +133,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 {
 	struct smp_object *so;
 	struct objcore *oc;
+	struct ban *ban;
 	uint32_t no;
 	double t_now = VTIM_real();
 	struct smp_signctx ctx[1];
@@ -159,13 +160,16 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 	for (;no > 0; so++,no--) {
 		if (EXP_WHEN(so) < t_now)
 			continue;
+		ban = BAN_FindBan(so->ban);
+		AN(ban);
 		oc = ObjNew(wrk);
 		oc->flags &= ~OC_F_BUSY;
 		oc->stobj->stevedore = sc->parent;
 		smp_init_oc(oc, sg, no);
 		VTAILQ_INSERT_TAIL(&sg->objcores, oc, lru_list);
 		oc->stobj->priv2 |= NEED_FIXUP;
-		oc->ban = BAN_RefBan(oc, so->ban);
+		BAN_RefBan(oc, ban);
+		AN(oc->ban);
 		EXP_COPY(oc, so);
 		sg->nobj++;
 		oc->refcnt++;



From martin at varnish-software.com  Tue Feb 23 16:44:47 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 23 Feb 2016 17:44:47 +0100
Subject: [master] 56c38b6 Make HSH_Insert() more like HSH_Unbusy()
Message-ID: <E1aYG4t-0005dx-0j@project.varnish-software.com>


commit 56c38b6456e185409bd3c82173a3fda97933a98a
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Feb 23 16:39:43 2016 +0100

    Make HSH_Insert() more like HSH_Unbusy()
    
    HSH_Insert() needs to take the same precautions as HSH_Unbusy() in
    order to not race against e.g. the ban lurker. In order to do this, we
    pass it the ban that the objcore should be associated with, and it
    will take care of calling BAN_RefBan() correctly.

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 4ff82be..90fbb25 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -65,6 +65,8 @@
 static const struct hash_slinger *hash;
 static struct objhead *private_oh;
 
+static void hsh_rush(struct worker *wrk, struct objhead *oh);
+
 /*---------------------------------------------------------------------*/
 
 static struct objhead *
@@ -256,13 +258,16 @@ hsh_testmagic(void *result)
  */
 
 void
-HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc)
+HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc,
+    struct ban *ban)
 {
 	struct objhead *oh;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	AN(digest);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	AN(ban);
+	AZ(oc->flags & OC_F_PRIVATE);
 
 	hsh_prealloc(wrk);
 
@@ -272,13 +277,25 @@ HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc)
 	Lck_AssertHeld(&oh->mtx);
 	assert(oh->refcnt > 0);
 
-	/* Insert (precreated) objcore in objecthead */
-	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
-	AZ(oc->flags & OC_F_BUSY);
+	/* Mark object busy and insert (precreated) objcore in
+	   objecthead. The new object inherits our objhead reference. */
+	oc->flags |= OC_F_BUSY;
+	oc->objhead = oh;
+	VTAILQ_INSERT_TAIL(&oh->objcs, oc, hsh_list);
+	Lck_Unlock(&oh->mtx);
+
+	BAN_RefBan(oc, ban);
+	AN(oc->ban);
+	EXP_Insert(wrk, oc);
 
+	/* Move the object first in the oh list, unbusy it and run the
+	   waitinglist if necessary */
+	Lck_Lock(&oh->mtx);
+	VTAILQ_REMOVE(&oh->objcs, oc, hsh_list);
 	VTAILQ_INSERT_HEAD(&oh->objcs, oc, hsh_list);
-	/* NB: do not deref objhead the new object inherits our reference */
-	oc->objhead = oh;
+	oc->flags &= ~OC_F_BUSY;
+	if (!VTAILQ_EMPTY(&oh->waitinglist))
+		hsh_rush(wrk, oh);
 	Lck_Unlock(&oh->mtx);
 	wrk->stats->n_objectcore++;
 	wrk->stats->n_vampireobject++;
diff --git a/bin/varnishd/hash/hash_slinger.h b/bin/varnishd/hash/hash_slinger.h
index 807c83d..b10e674 100644
--- a/bin/varnishd/hash/hash_slinger.h
+++ b/bin/varnishd/hash/hash_slinger.h
@@ -61,13 +61,15 @@ enum lookup_e {
 };
 
 /* cache_hash.c */
+struct ban;
 void HSH_Cleanup(struct worker *w);
 enum lookup_e HSH_Lookup(struct req *, struct objcore **, struct objcore **,
     int wait_for_busy, int always_insert);
 void HSH_Ref(struct objcore *o);
 void HSH_Init(const struct hash_slinger *slinger);
 void HSH_AddString(struct req *, void *ctx, const char *str);
-void HSH_Insert(struct worker *, const void *hash, struct objcore *);
+void HSH_Insert(struct worker *, const void *hash, struct objcore *,
+    struct ban *);
 void HSH_Purge(struct worker *, struct objhead *, double ttl, double grace,
     double keep);
 void HSH_config(const char *h_arg);
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 0e1637f..1a03708 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -163,18 +163,15 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		ban = BAN_FindBan(so->ban);
 		AN(ban);
 		oc = ObjNew(wrk);
-		oc->flags &= ~OC_F_BUSY;
 		oc->stobj->stevedore = sc->parent;
 		smp_init_oc(oc, sg, no);
 		VTAILQ_INSERT_TAIL(&sg->objcores, oc, lru_list);
 		oc->stobj->priv2 |= NEED_FIXUP;
-		BAN_RefBan(oc, ban);
-		AN(oc->ban);
 		EXP_COPY(oc, so);
 		sg->nobj++;
 		oc->refcnt++;
-		HSH_Insert(wrk, so->hash, oc);
-		EXP_Insert(wrk, oc);
+		HSH_Insert(wrk, so->hash, oc, ban);
+		AN(oc->ban);
 		HSH_DerefBoc(wrk, oc);	// XXX Keep it an stream resurrection?
 		(void)HSH_DerefObjCore(wrk, &oc);
 	}



From martin at varnish-software.com  Tue Feb 23 16:44:47 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 23 Feb 2016 17:44:47 +0100
Subject: [master] ec63cdb Fix double counting n_objectcore
Message-ID: <E1aYG4t-0005e7-26@project.varnish-software.com>


commit ec63cdb1a06f13121311b675decd1ea4a8b4b1d0
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Feb 23 16:44:12 2016 +0100

    Fix double counting n_objectcore
    
    n_objectcore is incremented by ObjNew(), so HSH_Insert() should not
    increase it again.

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 90fbb25..7ab868f 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -297,7 +297,6 @@ HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc,
 	if (!VTAILQ_EMPTY(&oh->waitinglist))
 		hsh_rush(wrk, oh);
 	Lck_Unlock(&oh->mtx);
-	wrk->stats->n_objectcore++;
 	wrk->stats->n_vampireobject++;
 }
 



From martin at varnish-software.com  Tue Feb 23 16:44:47 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue, 23 Feb 2016 17:44:47 +0100
Subject: [master] f67315c Move n_vampireobject increment to the persistent
 stevedore
Message-ID: <E1aYG4t-0005eG-3G@project.varnish-software.com>


commit f67315c52a86bcfe2944b71369ed4cf948412356
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Feb 23 16:45:17 2016 +0100

    Move n_vampireobject increment to the persistent stevedore
    
    This keeps the increment and decrement in the same compilation unit.

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 7ab868f..86523be 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -297,7 +297,6 @@ HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc,
 	if (!VTAILQ_EMPTY(&oh->waitinglist))
 		hsh_rush(wrk, oh);
 	Lck_Unlock(&oh->mtx);
-	wrk->stats->n_vampireobject++;
 }
 
 /*---------------------------------------------------------------------
diff --git a/bin/varnishd/storage/storage_persistent_silo.c b/bin/varnishd/storage/storage_persistent_silo.c
index 1a03708..61ae93b 100644
--- a/bin/varnishd/storage/storage_persistent_silo.c
+++ b/bin/varnishd/storage/storage_persistent_silo.c
@@ -174,6 +174,7 @@ smp_load_seg(struct worker *wrk, const struct smp_sc *sc,
 		AN(oc->ban);
 		HSH_DerefBoc(wrk, oc);	// XXX Keep it an stream resurrection?
 		(void)HSH_DerefObjCore(wrk, &oc);
+		wrk->stats->n_vampireobject++;
 	}
 	Pool_Sumstat(wrk);
 	sg->flags |= SMP_SEG_LOADED;



From phk at FreeBSD.org  Tue Feb 23 19:37:15 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Feb 2016 20:37:15 +0100
Subject: [master] 04dc5a1 Simplify
Message-ID: <E1aYIln-00022b-HM@project.varnish-software.com>


commit 04dc5a1191a6b19063562d3d8fdc3748ec4edee5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Feb 23 19:37:05 2016 +0000

    Simplify

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 8febc5e..0a4471e 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -885,7 +885,7 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
 		ll -= l;
 		o += l;
 	}
-	AZ(ObjSetU64(wrk, bo->fetch_objcore, OA_LEN, VSB_len(synth_body) - ll));
+	AZ(ObjSetU64(wrk, bo->fetch_objcore, OA_LEN, o));
 	VSB_delete(synth_body);
 	HSH_Unbusy(wrk, bo->fetch_objcore);
 	ObjSetState(wrk, bo->fetch_objcore, BOS_FINISHED);



From dridi.boukelmoune at gmail.com  Tue Feb 23 20:04:43 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 23 Feb 2016 21:04:43 +0100
Subject: [master] f471dae Reword
Message-ID: <E1aYJCN-0004ix-Qj@project.varnish-software.com>


commit f471dae7cfa561c60743a61c19b75bb5da8f30bd
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 20:57:54 2016 +0100

    Reword

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index aa4c68f..949ee93 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -249,7 +249,7 @@ BAN_DestroyObj(struct objcore *oc)
 }
 
 /*--------------------------------------------------------------------
- * Find a ban based on timestamp a ban timestamp.
+ * Find a ban based on a timestamp.
  * Assume we have a BAN_Hold, so list traversal is safe.
  */
 



From dridi.boukelmoune at gmail.com  Tue Feb 23 20:04:43 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 23 Feb 2016 21:04:43 +0100
Subject: [master] 1679bbe Simplify
Message-ID: <E1aYJCN-0004j3-SE@project.varnish-software.com>


commit 1679bbedcd16784e62055d3560e22011785cda48
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 21:02:48 2016 +0100

    Simplify

diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c
index 7d1e9b1..412370a 100644
--- a/bin/varnishd/http1/cache_http1_proto.c
+++ b/bin/varnishd/http1/cache_http1_proto.c
@@ -369,9 +369,8 @@ HTTP1_DissectRequest(struct http_conn *htc, struct http *hp)
 	    !strncasecmp(hp->hd[HTTP_HDR_URL].b, "https://", 8))
 		b = hp->hd[HTTP_HDR_URL].b + 8;
 	if (b) {
-		for (e = b; *e != '/' && *e != '\0'; e++)
-			continue;
-		if (*e == '/') {
+		e = strchr(b, '/');
+		if (e) {
 			http_Unset(hp, H_Host);
 			http_PrintfHeader(hp, "Host: %.*s", (int)(e - b), b);
 			hp->hd[HTTP_HDR_URL].b = e;



From fgsch at lodoss.net  Tue Feb 23 23:31:42 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 24 Feb 2016 00:31:42 +0100
Subject: [master] 0c11339 Typo
Message-ID: <E1aYMQg-0002o6-Fm@project.varnish-software.com>


commit 0c11339664ef4313f1c2686322a49b17e3304556
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 23 23:30:46 2016 +0000

    Typo

diff --git a/bin/varnishd/waiter/cache_waiter_ports.c b/bin/varnishd/waiter/cache_waiter_ports.c
index f9e3b52..49c6790 100644
--- a/bin/varnishd/waiter/cache_waiter_ports.c
+++ b/bin/varnishd/waiter/cache_waiter_ports.c
@@ -180,7 +180,7 @@ vws_thread(void *priv)
 		nevents = 1;
 
 		/*
-		 * see disucssion in
+		 * see discussion in
 		 * - https://issues.apache.org/bugzilla/show_bug.cgi?id=47645
 		 * - http://mail.opensolaris.org/pipermail/\
 		 *       networking-discuss/2009-August/011979.html



From fgsch at lodoss.net  Tue Feb 23 23:31:42 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 24 Feb 2016 00:31:42 +0100
Subject: [master] ad70720 Sync with reality
Message-ID: <E1aYMQg-0002oD-HM@project.varnish-software.com>


commit ad7072001e988fe311c03db458dff24cc74d9263
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 23 23:31:21 2016 +0000

    Sync with reality

diff --git a/bin/varnishd/builtin.vcl b/bin/varnishd/builtin.vcl
index 6430cd7..68216db 100644
--- a/bin/varnishd/builtin.vcl
+++ b/bin/varnishd/builtin.vcl
@@ -121,7 +121,7 @@ sub vcl_deliver {
 }
 
 /*
- * We can come here "invisibly" with the following errors: 413, 417 & 503
+ * We can come here "invisibly" with the following errors: 500 & 503
  */
 sub vcl_synth {
     set resp.http.Content-Type = "text/html; charset=utf-8";



From fgsch at lodoss.net  Tue Feb 23 23:49:35 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 24 Feb 2016 00:49:35 +0100
Subject: [master] 86cbba9 Re-add snippet needed for websockets support
Message-ID: <E1aYMhz-0004fW-45@project.varnish-software.com>


commit 86cbba9c12eb6c508b4d207797c8af2c90bab973
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 23 23:40:43 2016 +0000

    Re-add snippet needed for websockets support
    
    Pointed out by nirn on irc.

diff --git a/doc/sphinx/users-guide/vcl-example-websockets.rst b/doc/sphinx/users-guide/vcl-example-websockets.rst
index 24ea561..9401b37 100644
--- a/doc/sphinx/users-guide/vcl-example-websockets.rst
+++ b/doc/sphinx/users-guide/vcl-example-websockets.rst
@@ -5,10 +5,17 @@ Adding WebSockets support
 WebSockets is a technology for creating a bidirectional stream-based
 channel over HTTP.
 
-To run WebSockets through Varnish you need to pipe it as follows::
+To run WebSockets through Varnish you need to pipe the request and copy
+the Upgrade header as follows::
 
     sub vcl_recv {
         if (req.http.upgrade ~ "(?i)websocket") {
             return (pipe);
         }
     }
+
+    sub vcl_pipe {
+        if (req.http.upgrade) {
+            set bereq.http.upgrade = req.http.upgrade;
+        }
+    }



From martin at varnish-software.com  Wed Feb 24 12:34:13 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Wed, 24 Feb 2016 13:34:13 +0100
Subject: [master] 1f37998 Hit-for-pass objects are not IMS candidates
Message-ID: <E1aYYdx-000488-Pf@project.varnish-software.com>


commit 1f37998adbc5e68d0b20006436d9bef226ca9bdc
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Wed Feb 24 13:26:05 2016 +0100

    Hit-for-pass objects are not IMS candidates
    
    Original test case by fgsch
    
    Fixes: #1858

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 0a4471e..aa3676b 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -652,9 +652,10 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 	if (bo->do_gzip || bo->do_gunzip)
 		ObjSetFlag(bo->wrk, bo->fetch_objcore, OF_CHGGZIP, 1);
 
-	if (http_IsStatus(bo->beresp, 200) && (
-	    http_GetHdr(bo->beresp, H_Last_Modified, &p) ||
-	    http_GetHdr(bo->beresp, H_ETag, &p)))
+	if (!(bo->fetch_objcore->flags & OC_F_PASS) &&
+	    http_IsStatus(bo->beresp, 200) && (
+	      http_GetHdr(bo->beresp, H_Last_Modified, &p) ||
+	      http_GetHdr(bo->beresp, H_ETag, &p)))
 		ObjSetFlag(bo->wrk, bo->fetch_objcore, OF_IMSCAND, 1);
 
 	if (bo->htc->body_status != BS_NONE &&
diff --git a/bin/varnishtest/tests/r01858.vtc b/bin/varnishtest/tests/r01858.vtc
new file mode 100644
index 0000000..7335b6d
--- /dev/null
+++ b/bin/varnishtest/tests/r01858.vtc
@@ -0,0 +1,38 @@
+varnishtest "Test a hit-for-pass does not issue an IMS request"
+
+server s1 {
+	rxreq
+	txresp \
+	    -hdr {Etag: "foo"} \
+	    -body "foo"
+	rxreq
+	expect req.http.if-none-match == <undef>
+	txresp \
+	    -hdr {Etag: "bar"} \
+	    -body "bar"
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_backend_response {
+		if (beresp.status == 200) {
+			set beresp.ttl = 1s;
+			set beresp.uncacheable = true;
+			return (deliver);
+		}
+	}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "foo"
+
+	delay 1.5
+
+	txreq
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "bar"
+} -run
+



From martin at varnish-software.com  Wed Feb 24 12:36:24 2016
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Wed, 24 Feb 2016 13:36:24 +0100
Subject: [4.1] 4f2af50 Hit-for-pass objects are not IMS candidates
Message-ID: <E1aYYg4-0004e7-Iu@project.varnish-software.com>


commit 4f2af500ee64091e8a7c17a2a68026ae8920804c
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Wed Feb 24 13:26:05 2016 +0100

    Hit-for-pass objects are not IMS candidates
    
    Original test case by fgsch
    
    Fixes: #1858

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index c41fce3..e03d5c2 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -644,9 +644,10 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 	if (bo->do_gzip || bo->do_gunzip)
 		ObjSetFlag(bo->wrk, bo->fetch_objcore, OF_CHGGZIP, 1);
 
-	if (http_IsStatus(bo->beresp, 200) && (
-	    http_GetHdr(bo->beresp, H_Last_Modified, &p) ||
-	    http_GetHdr(bo->beresp, H_ETag, &p)))
+	if (!(bo->fetch_objcore->flags & OC_F_PASS) &&
+	    http_IsStatus(bo->beresp, 200) && (
+	      http_GetHdr(bo->beresp, H_Last_Modified, &p) ||
+	      http_GetHdr(bo->beresp, H_ETag, &p)))
 		ObjSetFlag(bo->wrk, bo->fetch_objcore, OF_IMSCAND, 1);
 
 	if (bo->htc->body_status != BS_NONE)
diff --git a/bin/varnishtest/tests/r01858.vtc b/bin/varnishtest/tests/r01858.vtc
new file mode 100644
index 0000000..7335b6d
--- /dev/null
+++ b/bin/varnishtest/tests/r01858.vtc
@@ -0,0 +1,38 @@
+varnishtest "Test a hit-for-pass does not issue an IMS request"
+
+server s1 {
+	rxreq
+	txresp \
+	    -hdr {Etag: "foo"} \
+	    -body "foo"
+	rxreq
+	expect req.http.if-none-match == <undef>
+	txresp \
+	    -hdr {Etag: "bar"} \
+	    -body "bar"
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_backend_response {
+		if (beresp.status == 200) {
+			set beresp.ttl = 1s;
+			set beresp.uncacheable = true;
+			return (deliver);
+		}
+	}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "foo"
+
+	delay 1.5
+
+	txreq
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "bar"
+} -run
+



From dridi.boukelmoune at gmail.com  Wed Feb 24 14:27:03 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 24 Feb 2016 15:27:03 +0100
Subject: [master] b2e4dc3 Don't capture function typedefs in generate.py
Message-ID: <E1aYaPA-0000eU-00@project.varnish-software.com>


commit b2e4dc36960cc881d51b6556120cb0baf92cddbf
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Feb 24 11:35:16 2016 +0100

    Don't capture function typedefs in generate.py

diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index 2f9dd10..44b8b2b 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -763,6 +763,8 @@ for i in fi:
 		continue
 	if j[-1][-1] != ";":
 		continue
+	if j[-1][-2] == ")":
+		continue
 	if j[-1][:4] != "VCL_":
 		continue
 	d = " ".join(j[1:-1])



From dridi.boukelmoune at gmail.com  Wed Feb 24 14:27:04 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 24 Feb 2016 15:27:04 +0100
Subject: [master] 8e90772 Allow ACLs to be used outside of transactions
Message-ID: <E1aYaPA-0000ea-1H@project.varnish-software.com>


commit 8e90772dfb93767aa1b76a93da636f5d2ce06f06
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 16:22:34 2016 +0100

    Allow ACLs to be used outside of transactions

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 50a7232..2bb697c 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -67,7 +67,11 @@ VRT_acl_log(VRT_CTX, const char *msg)
 {
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
-	VSLb(ctx->vsl, SLT_VCL_acl, "%s", msg);
+	AN(msg);
+	if (ctx->vsl != NULL)
+		VSLb(ctx->vsl, SLT_VCL_acl, "%s", msg);
+	else
+		VSL(SLT_VCL_acl, 0, "%s", msg);
 }
 
 /*--------------------------------------------------------------------*/



From dridi.boukelmoune at gmail.com  Wed Feb 24 14:27:04 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 24 Feb 2016 15:27:04 +0100
Subject: [master] f2bc770 Introduce a new ACL type for VMODs
Message-ID: <E1aYaPA-0000el-2g@project.varnish-software.com>


commit f2bc7702948feb3311fe38c16adcc296adad6a22
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 16:39:24 2016 +0100

    Introduce a new ACL type for VMODs

diff --git a/include/vrt.h b/include/vrt.h
index c38fc59..ccebea0 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -53,6 +53,7 @@
 /***********************************************************************/
 
 struct VCL_conf;
+struct vrt_acl;
 struct busyobj;
 struct director;
 struct http;
@@ -70,6 +71,7 @@ struct ws;
  * (alphabetic order)
  */
 
+typedef const struct vrt_acl *			VCL_ACL;
 typedef const struct director *			VCL_BACKEND;
 typedef const struct vmod_priv *		VCL_BLOB;
 typedef unsigned				VCL_BOOL;
@@ -240,6 +242,14 @@ struct vrt_ref {
 /* ACL related */
 #define VRT_ACL_MAXADDR		16	/* max(IPv4, IPv6) */
 
+typedef int acl_f (VRT_CTX, VCL_IP);
+
+struct vrt_acl {
+	unsigned	magic;
+#define VRT_ACL_MAGIC	0x78329d96
+	acl_f		*match;
+};
+
 void VRT_acl_log(VRT_CTX, const char *msg);
 
 /* req related */
diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 78d9ec5..98f24d8 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -351,7 +351,7 @@ vcc_acl_emit(struct vcc *tl, const char *acln, int anon)
 	struct token *t;
 	struct inifin *ifp;
 
-	Fh(tl, 0, "\nstatic int\n");
+	Fh(tl, 0, "\nstatic int __match_proto__(acl_f)\n");
 	Fh(tl, 0,
 	    "match_acl_%s_%s(VRT_CTX, const VCL_IP p)\n",
 	    anon ? "anon" : "named", acln);
@@ -443,6 +443,15 @@ vcc_acl_emit(struct vcc *tl, const char *acln, int anon)
 	if (!anon)
 		Fh(tl, 0, "\tVRT_acl_log(ctx, \"NO_MATCH %s\");\n", acln);
 	Fh(tl, 0, "\treturn (0);\n}\n");
+
+	if (anon)
+		return;
+
+	/* Emit the struct that will be referenced */
+	Fh(tl, 0, "\nconst struct vrt_acl vrt_acl_named_%s = {\n", acln);
+	Fh(tl, 0, "\t.magic = VRT_ACL_MAGIC,\n");
+	Fh(tl, 0, "\t.match = &match_acl_named_%s,\n", acln);
+	Fh(tl, 0, "};\n\n");
 }
 
 void



From dridi.boukelmoune at gmail.com  Wed Feb 24 14:27:04 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 24 Feb 2016 15:27:04 +0100
Subject: [master] 05d2f56 Make named ACLs available to VMODs
Message-ID: <E1aYaPA-0000ev-5d@project.varnish-software.com>


commit 05d2f561c61b90dc5d19b476fe03e9950380a0e7
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 17:55:13 2016 +0100

    Make named ACLs available to VMODs

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 40b7bf7..3020b5d 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -115,7 +115,10 @@ language representation.  Here is a description of them.
 All but the PRIV and STRING_LIST types have typedefs: VCL_INT, VCL_REAL,
 etc.
 
-.. TODO document ACL if patchwork #314 is merged
+ACL
+	C-type: ``const struct vrt_acl *``
+
+        A type for named ACLs declared in VCL.
 
 BACKEND
 	C-type: ``const struct director *``
diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 98f24d8..2fc5fa4 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -474,7 +474,7 @@ void
 vcc_ParseAcl(struct vcc *tl)
 {
 	struct token *an;
-	int i;
+	struct symbol *sym;
 	char acln[1024];
 
 	vcc_NextToken(tl);
@@ -490,13 +490,20 @@ vcc_ParseAcl(struct vcc *tl)
 	an = tl->t;
 	vcc_NextToken(tl);
 
-	i = vcc_AddDef(tl, an, SYM_ACL);
-	if (i > 1) {
+	bprintf(acln, "%.*s", PF(an));
+
+	sym = VCC_GetSymbolTok(tl, an, SYM_ACL);
+	AN(sym);
+	if (sym->ndef > 0) {
 		VSB_printf(tl->sb, "ACL %.*s redefined\n", PF(an));
 		vcc_ErrWhere(tl, an);
 		return;
 	}
-	bprintf(acln, "%.*s", PF(an));
+	sym->fmt = ACL;
+	sym->eval = vcc_Eval_Acl;
+	sym->eval_priv = TlDup(tl, acln);
+	sym->ndef++;
+	ERRCHK(tl);
 
 	SkipToken(tl, '{');
 
diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h
index 7816a3e..20bc18c 100644
--- a/lib/libvcc/vcc_compile.h
+++ b/lib/libvcc/vcc_compile.h
@@ -286,6 +286,7 @@ sym_expr_t vcc_Eval_Var;
 sym_expr_t vcc_Eval_SymFunc;
 void vcc_Eval_Func(struct vcc *tl, const char *cfunc, const char *extra,
     const char *name, const char *args);
+sym_expr_t vcc_Eval_Acl;
 sym_expr_t vcc_Eval_Backend;
 sym_expr_t vcc_Eval_Probe;
 
diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c
index af3b3e7..574ca53 100644
--- a/lib/libvcc/vcc_expr.c
+++ b/lib/libvcc/vcc_expr.c
@@ -494,10 +494,29 @@ vcc_Eval_BoolConst(struct vcc *tl, struct expr **e, const struct symbol *sym)
  */
 
 void
+vcc_Eval_Acl(struct vcc *tl, struct expr **e, const struct symbol *sym)
+{
+
+	assert(sym->kind == SYM_ACL);
+	AN(sym->eval_priv);
+
+	vcc_ExpectCid(tl);
+	vcc_AddRef(tl, tl->t, SYM_ACL);
+	*e = vcc_mk_expr(ACL, "&vrt_acl_named_%s",
+	    (const char *)sym->eval_priv);
+	(*e)->constant = EXPR_VAR;	/* XXX ? */
+	vcc_NextToken(tl);
+}
+
+/*--------------------------------------------------------------------
+ */
+
+void
 vcc_Eval_Backend(struct vcc *tl, struct expr **e, const struct symbol *sym)
 {
 
 	assert(sym->kind == SYM_BACKEND);
+	AN(sym->eval_priv);
 
 	vcc_ExpectCid(tl);
 	vcc_AddRef(tl, tl->t, SYM_BACKEND);
@@ -800,6 +819,7 @@ vcc_expr4(struct vcc *tl, struct expr **e, enum var_type fmt)
 	struct expr *e1, *e2;
 	const char *ip;
 	const struct symbol *sym;
+	enum symkind kind;
 	double d;
 	int i;
 
@@ -819,10 +839,14 @@ vcc_expr4(struct vcc *tl, struct expr **e, enum var_type fmt)
 		 * XXX: look for SYM_VAR first for consistency ?
 		 */
 		sym = NULL;
-		if (fmt == BACKEND)
-			sym = VCC_FindSymbol(tl, tl->t, SYM_BACKEND);
-		if (fmt == PROBE)
-			sym = VCC_FindSymbol(tl, tl->t, SYM_PROBE);
+		switch (fmt) {
+		case ACL:	kind = SYM_ACL; break;
+		case BACKEND:	kind = SYM_BACKEND; break;
+		case PROBE:	kind = SYM_PROBE; break;
+		default:	kind = SYM_NONE; break;
+		}
+		if (kind != SYM_NONE)
+			sym = VCC_FindSymbol(tl, tl->t, kind);
 		if (sym == NULL)
 			sym = VCC_FindSymbol(tl, tl->t, SYM_VAR);
 		if (sym == NULL)
@@ -841,6 +865,7 @@ vcc_expr4(struct vcc *tl, struct expr **e, enum var_type fmt)
 		switch(sym->kind) {
 		case SYM_VAR:
 		case SYM_FUNC:
+		case SYM_ACL:
 		case SYM_BACKEND:
 		case SYM_PROBE:
 			AN(sym->eval);
diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index bb29bd4..024d78a 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -46,6 +46,7 @@ from os.path import dirname, exists, join, realpath
 from pprint import pprint, pformat
 
 ctypes = {
+	'ACL':		"VCL_ACL",
 	'BACKEND':	"VCL_BACKEND",
 	'BLOB':		"VCL_BLOB",
 	'BOOL':		"VCL_BOOL",



From dridi.boukelmoune at gmail.com  Wed Feb 24 14:27:04 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 24 Feb 2016 15:27:04 +0100
Subject: [master] 9e07b94 Add a new VRT_acl_match function to the VMODs ABI
Message-ID: <E1aYaPA-0000f7-6y@project.varnish-software.com>


commit 9e07b94744a883dc0d0c6fc58b364cbe6f61e0ba
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 18:26:08 2016 +0100

    Add a new VRT_acl_match function to the VMODs ABI
    
    VRT_MINOR_VERSION needs to be incremented for the 4.1 branch.

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 2bb697c..a82b1e4 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -39,6 +39,7 @@
 #include "vcl.h"
 #include "vrt.h"
 #include "vrt_obj.h"
+#include "vsa.h"
 #include "vtcp.h"
 #include "vtim.h"
 
@@ -74,6 +75,16 @@ VRT_acl_log(VRT_CTX, const char *msg)
 		VSL(SLT_VCL_acl, 0, "%s", msg);
 }
 
+int
+VRT_acl_match(VRT_CTX, VCL_ACL acl, VCL_IP ip)
+{
+
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+	CHECK_OBJ_NOTNULL(acl, VRT_ACL_MAGIC);
+	assert(VSA_Sane(ip));
+	return (acl->match(ctx, ip));
+}
+
 /*--------------------------------------------------------------------*/
 
 struct http *
diff --git a/include/vrt.h b/include/vrt.h
index ccebea0..ae30ee4 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -251,6 +251,7 @@ struct vrt_acl {
 };
 
 void VRT_acl_log(VRT_CTX, const char *msg);
+int VRT_acl_match(VRT_CTX, VCL_ACL, VCL_IP);
 
 /* req related */
 



From dridi.boukelmoune at gmail.com  Wed Feb 24 14:27:04 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 24 Feb 2016 15:27:04 +0100
Subject: [master] 038557a Test VMOD ACLs using vmod-debug
Message-ID: <E1aYaPA-0000fH-9C@project.varnish-software.com>


commit 038557a7af76fe64b46e4af43b6d8ba7bc9690bf
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 18:44:14 2016 +0100

    Test VMOD ACLs using vmod-debug

diff --git a/bin/varnishtest/tests/m00023.vtc b/bin/varnishtest/tests/m00023.vtc
new file mode 100644
index 0000000..b45d5bf
--- /dev/null
+++ b/bin/varnishtest/tests/m00023.vtc
@@ -0,0 +1,32 @@
+varnishtest "Test VMOD ACLs"
+
+varnish v1 -vcl {
+	import debug;
+
+	backend dummy {
+		.host = "${bad_ip}";
+	}
+
+	acl loopback {
+		"127"/24;
+	}
+
+	sub vcl_init {
+		if (!debug.match_acl(loopback, "127.0.0.127")) {
+			debug.init_fail();
+		}
+	}
+
+	sub vcl_recv {
+		if (debug.match_acl(loopback, client.ip)) {
+			return (synth(200));
+		}
+		return (synth(500));
+	}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+} -run
diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index d0ebe1c..1725689 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -150,3 +150,7 @@ Reset to the previous snapshot of a workspace, taken from debug.workspace_snap.
 $Function VOID vcl_release_delay(DURATION)
 
 Hold a reference to the VCL when it goes cold for the given delay.
+
+$Function BOOL match_acl(ACL acl, IP ip)
+
+Perform an IP match against a named ACL.
diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c
index 1e1b71f..d1e79ed 100644
--- a/lib/libvmod_debug/vmod_debug.c
+++ b/lib/libvmod_debug/vmod_debug.c
@@ -36,6 +36,7 @@
 
 #include "vcl.h"
 #include "vrt.h"
+#include "vsa.h"
 #include "vsb.h"
 #include "vtim.h"
 #include "vcc_if.h"
@@ -466,3 +467,14 @@ vmod_vcl_release_delay(VRT_CTX, VCL_DURATION delay)
 	assert(delay > 0.0);
 	vcl_release_delay = delay;
 }
+
+VCL_BOOL
+vmod_match_acl(VRT_CTX, VCL_ACL acl, VCL_IP ip)
+{
+
+	CHECK_OBJ_ORNULL(ctx, VRT_CTX_MAGIC);
+	CHECK_OBJ_ORNULL(acl, VRT_ACL_MAGIC);
+	assert(VSA_Sane(ip));
+
+	return (VRT_acl_match(ctx, acl, ip));
+}



From fgsch at lodoss.net  Wed Feb 24 15:11:59 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 24 Feb 2016 16:11:59 +0100
Subject: [master] 9f7a44f Polish
Message-ID: <E1aYb6d-0002XN-6C@project.varnish-software.com>


commit 9f7a44f1fed328b0328455f7ebc32d710b8f937a
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Feb 24 15:04:25 2016 +0000

    Polish

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 3020b5d..d3271b5 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -118,7 +118,7 @@ etc.
 ACL
 	C-type: ``const struct vrt_acl *``
 
-        A type for named ACLs declared in VCL.
+	A type for named ACLs declared in VCL.
 
 BACKEND
 	C-type: ``const struct director *``
@@ -152,9 +152,9 @@ DURATION
 	A time interval, as in 25 seconds.
 
 ENUM
-        C-type: ``const char *``
+	C-type: ``const char *``
 
-        TODO
+	TODO
 
 HEADER
 	C-type: ``const struct gethdr_s *``
@@ -169,9 +169,9 @@ HEADER
 	the value, but not where it came from.
 
 HTTP
-        C-type: ``struct http *``
+	C-type: ``struct http *``
 
-        TODO
+	TODO
 
 INT
 	C-type: ``long``
diff --git a/include/vrt.h b/include/vrt.h
index ae30ee4..beaf148 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -242,12 +242,12 @@ struct vrt_ref {
 /* ACL related */
 #define VRT_ACL_MAXADDR		16	/* max(IPv4, IPv6) */
 
-typedef int acl_f (VRT_CTX, VCL_IP);
+typedef int acl_match_f(VRT_CTX, const VCL_IP);
 
 struct vrt_acl {
 	unsigned	magic;
 #define VRT_ACL_MAGIC	0x78329d96
-	acl_f		*match;
+	acl_match_f	*match;
 };
 
 void VRT_acl_log(VRT_CTX, const char *msg);
diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 2fc5fa4..97728a9 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -351,7 +351,7 @@ vcc_acl_emit(struct vcc *tl, const char *acln, int anon)
 	struct token *t;
 	struct inifin *ifp;
 
-	Fh(tl, 0, "\nstatic int __match_proto__(acl_f)\n");
+	Fh(tl, 0, "\nstatic int __match_proto__(acl_match_f)\n");
 	Fh(tl, 0,
 	    "match_acl_%s_%s(VRT_CTX, const VCL_IP p)\n",
 	    anon ? "anon" : "named", acln);
@@ -364,7 +364,7 @@ vcc_acl_emit(struct vcc *tl, const char *acln, int anon)
 	Fh(tl, 0, "\t\tVRT_acl_log(ctx, \"NO_FAM %s\");\n", acln);
 	Fh(tl, 0, "\t\treturn(0);\n");
 	Fh(tl, 0, "\t}\n\n");
-	if (!tl->param->err_unref && !anon ) {
+	if (!tl->param->err_unref && !anon) {
 		ifp = New_IniFin(tl);
 		VSB_printf(ifp->ini,
 			"\tif (0) match_acl_named_%s(0, 0);\n", acln);
@@ -419,9 +419,9 @@ vcc_acl_emit(struct vcc *tl, const char *acln, int anon)
 			t = ae->t_addr;
 			do {
 				if (t->tok == CSTR) {
-					Fh(tl, 0, " \"\\\"\" " );
+					Fh(tl, 0, " \"\\\"\" ");
 					EncToken(tl->fh, t);
-					Fh(tl, 0, " \"\\\"\" " );
+					Fh(tl, 0, " \"\\\"\" ");
 				} else
 					Fh(tl, 0, " \"%.*s\"", PF(t));
 				if (t == ae->t_mask)



From fgsch at lodoss.net  Wed Feb 24 15:15:47 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 24 Feb 2016 16:15:47 +0100
Subject: [master] cbe3974 Reminder
Message-ID: <E1aYbAJ-00065n-SZ@project.varnish-software.com>


commit cbe3974f67d8c8f3a575ec7f0e27b626dec9f45c
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Feb 24 15:15:34 2016 +0000

    Reminder

diff --git a/include/vrt.h b/include/vrt.h
index beaf148..2b243c3 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -40,6 +40,8 @@
  * Whenever something is deleted or changed in a way which is not
  * binary/load-time compatible, increment MAJOR version
  *
+ * Bump VRT_MINOR_VERSION due to:
+ * - VCL_ACL type added
  * Bump VRT_MAJOR_VERSION due to:
  * - VRT_CacheReqBody changed signature
  *



From phk at FreeBSD.org  Wed Feb 24 19:04:04 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 24 Feb 2016 20:04:04 +0100
Subject: [master] 90862a7 Add a BOS_PREP_STREAM state to notify stevedores
 before streaming starts.
Message-ID: <E1aYejE-0004hW-QT@project.varnish-software.com>


commit 90862a70077fc25463e776d72325baab15675709
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 24 15:31:25 2016 +0000

    Add a BOS_PREP_STREAM state to notify stevedores before streaming
    starts.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index cbeccbf..08eaa2d 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -365,6 +365,7 @@ struct storeobj {
 enum boc_state_e {
 	BOS_INVALID = 0,	/* don't touch (yet) */
 	BOS_REQ_DONE,		/* beresp.* can be examined */
+	BOS_PREP_STREAM,	/* Prepare for streaming */
 	BOS_STREAM,		/* beresp.* can be examined */
 	BOS_FINISHED,		/* object is complete */
 	BOS_FAILED,		/* something went wrong */
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index aa3676b..609e6c2 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -672,6 +672,7 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 	assert(bo->fetch_objcore->boc->state == BOS_REQ_DONE);
 
 	if (bo->do_stream) {
+		ObjSetState(wrk, bo->fetch_objcore, BOS_PREP_STREAM);
 		HSH_Unbusy(wrk, bo->fetch_objcore);
 		ObjSetState(wrk, bo->fetch_objcore, BOS_STREAM);
 	}
@@ -762,6 +763,7 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 	AZ(ObjCopyAttr(bo->wrk, bo->fetch_objcore, bo->stale_oc, OA_GZIPBITS));
 
 	if (bo->do_stream) {
+		ObjSetState(wrk, bo->fetch_objcore, BOS_PREP_STREAM);
 		HSH_Unbusy(wrk, bo->fetch_objcore);
 		ObjSetState(wrk, bo->fetch_objcore, BOS_STREAM);
 	}
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 0248ee7..1bae372 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -54,7 +54,7 @@
  * 2	  ObjSetU32()
  * 2	  ObjSetU64()
  *
- * 2->3	ObjStable()	Will no longer be modified
+ * 2->3	ObjStable()	Will no longer be modified (clean up boc)
  *
  * 23	ObjHasAttr()
  * 23	ObjGetAttr()



From phk at FreeBSD.org  Wed Feb 24 19:04:04 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 24 Feb 2016 20:04:04 +0100
Subject: [master] 5052cf6 Fix asserts
Message-ID: <E1aYejE-0004hc-St@project.varnish-software.com>


commit 5052cf6f2e7df462478663960f5a95e7328891d2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Feb 24 19:03:50 2016 +0000

    Fix asserts

diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 1bae372..09ddeed 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -226,7 +226,8 @@ ObjSetState(struct worker *wrk, const struct objcore *oc,
 	assert(next > oc->boc->state);
 
 	CHECK_OBJ_ORNULL(oc->stobj->stevedore, STEVEDORE_MAGIC);
-	AN(next != BOS_FINISHED || (oc->oa_present & (1 << OA_LEN)));
+	assert(next != BOS_STREAM || oc->boc->state == BOS_PREP_STREAM);
+	assert(next != BOS_FINISHED || (oc->oa_present & (1 << OA_LEN)));
 
 	if (oc->stobj->stevedore != NULL) {
 		om = oc->stobj->stevedore->methods;



From phk at FreeBSD.org  Thu Feb 25 07:22:46 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 25 Feb 2016 08:22:46 +0100
Subject: [master] d74cdaf Polish the ESI parser code a little bit while
 staring at it with barely concealed evil grin.
Message-ID: <E1aYqG6-0005lp-DX@project.varnish-software.com>


commit d74cdaf4ce086b5e61052881131b5ace267a2ab8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 25 07:21:29 2016 +0000

    Polish the ESI parser code a little bit while staring at it with
    barely concealed evil grin.

diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c
index 383731f..1e86b25 100644
--- a/bin/varnishd/cache/cache_esi_parse.c
+++ b/bin/varnishd/cache/cache_esi_parse.c
@@ -104,7 +104,7 @@ struct vep_state {
 	struct vep_match	*match;
 	struct vep_match	*match_hit;
 
-	char			tag[10];
+	char			tag[8];
 	int			tag_i;
 
 	dostuff_f		*dostuff;
@@ -217,28 +217,21 @@ vep_warn(const struct vep_state *vep, const char *p)
  */
 
 static struct vep_match *
-vep_match(struct vep_state *vep, const char *b, const char *e)
+vep_match(const struct vep_state *vep, const char *b, const char *e)
 {
 	struct vep_match *vm;
 	const char *q, *r;
-	ssize_t l;
 
-	for (vm = vep->match; vm->match; vm++) {
+	for (vm = vep->match; vm->match != NULL; vm++) {
+		assert(strlen(vm->match) <= sizeof (vep->tag));
 		r = b;
-		for (q = vm->match; *q && r < e; q++, r++)
+		for (q = vm->match; *q != '\0' && r < e; q++, r++)
 			if (*q != *r)
 				break;
-		if (*q != '\0' && r == e) {
-			if (b != vep->tag) {
-				l = e - b;
-				assert(l < sizeof vep->tag);
-				memmove(vep->tag, b, l);
-				vep->tag_i = l;
-			}
-			return (NULL);
-		}
 		if (*q == '\0')
-			return (vm);
+			break;
+		if (r == e)
+			return (NULL);
 	}
 	return (vm);
 }
@@ -948,6 +941,7 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l)
 				vep->match = NULL;
 				vep->tag_i = 0;
 			} else {
+				assert(e - p <= sizeof(vep->tag));
 				memcpy(vep->tag, p, e - p);
 				vep->tag_i = e - p;
 				vep->state = VEP_MATCHBUF;
@@ -959,28 +953,23 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l)
 			 * sections.
 			 */
 			AN(vep->match);
-			do {
-				if (*p == '>') {
-					for (vm = vep->match;
-					    vm->match != NULL; vm++)
-						continue;
-					AZ(vm->match);
-				} else {
-					vep->tag[vep->tag_i++] = *p++;
-					vm = vep_match(vep,
-					    vep->tag, vep->tag + vep->tag_i);
-					if (vm && vm->match == NULL) {
-						vep->tag_i--;
-						p--;
-					}
-				}
-			} while (vm == NULL && p < e);
-			vep->match_hit = vm;
+			i = sizeof(vep->tag) - vep->tag_i;
+			if (i > e - p)
+				i = e - p;
+			memcpy(vep->tag + vep->tag_i, p, i);
+			vm = vep_match(vep, vep->tag,
+			    vep->tag + vep->tag_i + i);
 			if (vm == NULL) {
+				vep->tag_i += i;
+				p += i;
 				assert(p == e);
 			} else {
+				vep->match_hit = vm;
 				vep->state = *vm->state;
+				if (vm->match != NULL)
+					p += strlen(vm->match) - vep->tag_i;
 				vep->match = NULL;
+				vep->tag_i = 0;
 			}
 		} else if (vep->state == VEP_UNTIL) {
 			/*
@@ -1019,7 +1008,7 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l)
 /*---------------------------------------------------------------------
  */
 
-static ssize_t __match_proto__()
+static ssize_t __match_proto__(vep_callback_t)
 vep_default_cb(struct vfp_ctx *vc, void *priv, ssize_t l, enum vgz_flag flg)
 {
 	ssize_t *s;



From phk at FreeBSD.org  Thu Feb 25 08:45:06 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 25 Feb 2016 09:45:06 +0100
Subject: [master] c360e97 Simplify the ESI parser a tiny bit,
 and prevent certain bogus XML constructs ("</!--" and "</!CDATA[["
 from confusing it.
Message-ID: <E1aYrXm-0005ZU-I7@project.varnish-software.com>


commit c360e97a0280dd566cd63d3dc62e982877c6e779
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Feb 25 08:44:13 2016 +0000

    Simplify the ESI parser a tiny bit, and prevent certain bogus
    XML constructs ("</!--" and "</!CDATA[[" from confusing it.

diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c
index 1e86b25..e4de056 100644
--- a/bin/varnishd/cache/cache_esi_parse.c
+++ b/bin/varnishd/cache/cache_esi_parse.c
@@ -128,9 +128,11 @@ static const char * const VEP_NEXTTAG =		"[NxtTag]";
 static const char * const VEP_NOTMYTAG =	"[NotMyTag]";
 
 static const char * const VEP_STARTTAG =	"[StartTag]";
+static const char * const VEP_COMMENTESI =	"[CommentESI]";
 static const char * const VEP_COMMENT =		"[Comment]";
 static const char * const VEP_CDATA =		"[CDATA]";
 static const char * const VEP_ESITAG =		"[ESITag]";
+static const char * const VEP_ESIENDTAG =	"[/ESITag]";
 
 static const char * const VEP_ESIREMOVE =	"[ESI:Remove]";
 static const char * const VEP_ESIINCLUDE =	"[ESI:Include]";
@@ -153,7 +155,10 @@ static const char * const VEP_MATCH =		"[Match]";
 /*---------------------------------------------------------------------*/
 
 static struct vep_match vep_match_starttag[] = {
+	{ "!--esi",	&VEP_COMMENTESI },
+	{ "!---->",	&VEP_NEXTTAG },
 	{ "!--",	&VEP_COMMENT },
+	{ "/esi:",	&VEP_ESIENDTAG },
 	{ "esi:",	&VEP_ESITAG },
 	{ "![CDATA[",	&VEP_CDATA },
 	{ NULL,		&VEP_NOTMYTAG }
@@ -413,19 +418,14 @@ vep_do_remove(struct vep_state *vep, enum dowhat what)
 	Debug("DO_REMOVE(%d, end %d empty %d remove %d)\n",
 	    what, vep->endtag, vep->emptytag, vep->remove);
 	assert(what == DO_TAG);
-	if (vep->emptytag) {
-		vep_error(vep,
-		    "ESI 1.0 <esi:remove/> not legal");
-	} else {
-		if (vep->remove && !vep->endtag)
-			vep_error(vep,
-			    "ESI 1.0 <esi:remove> already open");
-		else if (!vep->remove && vep->endtag)
-			vep_error(vep,
-			    "ESI 1.0 <esi:remove> not open");
-		else
-			vep->remove = !vep->endtag;
-	}
+	if (vep->emptytag)
+		vep_error(vep, "ESI 1.0 <esi:remove/> not legal");
+	else if (vep->remove && !vep->endtag)
+		vep_error(vep, "ESI 1.0 <esi:remove> already open");
+	else if (!vep->remove && vep->endtag)
+		vep_error(vep, "ESI 1.0 <esi:remove> not open");
+	else
+		vep->remove = !vep->endtag;
 }
 
 /*---------------------------------------------------------------------
@@ -457,11 +457,9 @@ vep_do_include(struct vep_state *vep, enum dowhat what)
 	}
 	assert(what == DO_TAG);
 	if (!vep->emptytag)
-		vep_warn(vep,
-		    "ESI 1.0 <esi:include> lacks final '/'");
+		vep_warn(vep, "ESI 1.0 <esi:include> lacks final '/'");
 	if (vep->include_src == NULL) {
-		vep_error(vep,
-		    "ESI 1.0 <esi:include> lacks src attr");
+		vep_error(vep, "ESI 1.0 <esi:include> lacks src attr");
 		return;
 	}
 
@@ -672,7 +670,6 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l)
 			 * out for end of EsiCmt if armed.
 			 */
 			vep->emptytag = 0;
-			vep->endtag = 0;
 			vep->attr = NULL;
 			vep->dostuff = NULL;
 			while (p < e && *p != '<') {
@@ -685,8 +682,7 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l)
 					vep->esicmt_p = vep->esicmt;
 					continue;
 				}
-				if (!vep->remove &&
-				    vep->esicmt_p == vep->esicmt)
+				if (!vep->remove && vep->esicmt_p == vep->esicmt)
 					vep_mark_verbatim(vep, p);
 				p++;
 				if (*++vep->esicmt_p == '\0') {
@@ -715,44 +711,23 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l)
 		 */
 
 		} else if (vep->state == VEP_STARTTAG) {
-			/*
-			 * Start of tag, set up match table
-			 */
-			if (p < e) {
-				if (*p == '/') {
-					vep->endtag = 1;
-					p++;
-				}
-				vep->match = vep_match_starttag;
-				vep->state = VEP_MATCH;
-			}
+			/* Start of tag, set up match table */
+			vep->endtag = 0;
+			vep->match = vep_match_starttag;
+			vep->state = VEP_MATCH;
 		} else if (vep->state == VEP_COMMENT) {
-			/*
-			 * We are in a comment, find out if it is an
-			 * ESI comment or a regular comment
-			 */
-			if (vep->esicmt == NULL)
-				vep->esicmt_p = vep->esicmt = "esi";
-			while (p < e) {
-				if (*p != *vep->esicmt_p) {
-					vep->esicmt_p = vep->esicmt = NULL;
-					vep->until_p = vep->until = "-->";
-					vep->until_s = VEP_NEXTTAG;
-					vep->state = VEP_UNTIL;
-					break;
-				}
-				p++;
-				if (*++vep->esicmt_p != '\0')
-					continue;
-				if (vep->remove)
-					vep_error(vep,
-					    "ESI 1.0 Nested <!--esi"
-					    " element in <esi:remove>");
-				vep->esicmt_p = vep->esicmt = "-->";
-				vep->state = VEP_NEXTTAG;
-				vep_mark_skip(vep, p);
-				break;
-			}
+			vep->esicmt_p = vep->esicmt = NULL;
+			vep->until_p = vep->until = "-->";
+			vep->until_s = VEP_NEXTTAG;
+			vep->state = VEP_UNTIL;
+		} else if (vep->state == VEP_COMMENTESI) {
+			if (vep->remove)
+				vep_error(vep,
+				    "ESI 1.0 Nested <!--esi"
+				    " element in <esi:remove>");
+			vep->esicmt_p = vep->esicmt = "-->";
+			vep->state = VEP_NEXTTAG;
+			vep_mark_skip(vep, p);
 		} else if (vep->state == VEP_CDATA) {
 			/*
 			 * Easy: just look for the end of CDATA
@@ -760,6 +735,9 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l)
 			vep->until_p = vep->until = "]]>";
 			vep->until_s = VEP_NEXTTAG;
 			vep->state = VEP_UNTIL;
+		} else if (vep->state == VEP_ESIENDTAG) {
+			vep->endtag = 1;
+			vep->state = VEP_ESITAG;
 		} else if (vep->state == VEP_ESITAG) {
 			vep->in_esi_tag = 1;
 			vep->esi_found = 1;
@@ -987,7 +965,7 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l)
 				vep_mark_verbatim(vep, p);
 		} else {
 			Debug("*** Unknown state %s\n", vep->state);
-			INCOMPL();
+			WRONG("WRONG ESI PARSER STATE");
 		}
 	}
 	/*
diff --git a/bin/varnishtest/tests/e00008.vtc b/bin/varnishtest/tests/e00008.vtc
index 0345844..51c8c88 100644
--- a/bin/varnishtest/tests/e00008.vtc
+++ b/bin/varnishtest/tests/e00008.vtc
@@ -36,8 +36,11 @@ server s1 {
 		<esi:include  src=/>			30
 		<esi:include  src= />			31
 		</esi:include>				32
-		<esi:include  src="foofof />		33
-		<esi:include  foo=bar src=/body2 />	34
+		<!---->					33
+		</!-- bogocommentt -->			34
+		</![CDATA[ bogo-cdata ]]>		35
+		<esi:include  src="foofof />		36
+		<esi:include  foo=bar src=/body2 />	37
 	}
 	rxreq
 	expect req.url == "/body"
@@ -69,7 +72,7 @@ client c1 {
 	txreq
 	rxresp
 	expect resp.status == 200
-	expect resp.bodylen == 385
+	expect resp.bodylen == 464
 }
 
 client c1 -run



From fgsch at lodoss.net  Thu Feb 25 12:32:26 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Thu, 25 Feb 2016 13:32:26 +0100
Subject: [4.1] af29bc7 Re-add snippet needed for websockets support
Message-ID: <E1aYv5m-0006xK-9K@project.varnish-software.com>


commit af29bc7d735f53b6d801286feb31795063c0a8d7
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Tue Feb 23 23:40:43 2016 +0000

    Re-add snippet needed for websockets support
    
    Pointed out by nirn on irc.

diff --git a/doc/sphinx/users-guide/vcl-example-websockets.rst b/doc/sphinx/users-guide/vcl-example-websockets.rst
index 24ea561..9401b37 100644
--- a/doc/sphinx/users-guide/vcl-example-websockets.rst
+++ b/doc/sphinx/users-guide/vcl-example-websockets.rst
@@ -5,10 +5,17 @@ Adding WebSockets support
 WebSockets is a technology for creating a bidirectional stream-based
 channel over HTTP.
 
-To run WebSockets through Varnish you need to pipe it as follows::
+To run WebSockets through Varnish you need to pipe the request and copy
+the Upgrade header as follows::
 
     sub vcl_recv {
         if (req.http.upgrade ~ "(?i)websocket") {
             return (pipe);
         }
     }
+
+    sub vcl_pipe {
+        if (req.http.upgrade) {
+            set bereq.http.upgrade = req.http.upgrade;
+        }
+    }



From fgsch at lodoss.net  Thu Feb 25 13:09:04 2016
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Thu, 25 Feb 2016 14:09:04 +0100
Subject: [master] 7bae758 Mention the join character
Message-ID: <E1aYvfE-0002jZ-Or@project.varnish-software.com>


commit 7bae7582fea8aac00b6e5bdb1d83f54f4fe2d59a
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Feb 25 13:01:27 2016 +0000

    Mention the join character

diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc
index 33929af..6b5683a 100644
--- a/lib/libvmod_std/vmod.vcc
+++ b/lib/libvmod_std/vmod.vcc
@@ -108,6 +108,7 @@ $Function VOID collect(HEADER hdr)
 
 Description
 	Collapses multiple *hdr* headers into one long header.
+	The header values are joined with a comma (",").
 
 	Care should be taken when collapsing headers. In particular
 	collapsing Set-Cookie will lead to unexpected results on the



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] 78a0ecd VFP_Push: document the place which still need attention
 for out-of-workspace handling
Message-ID: <E1aYwdO-0006Jr-3R@project.varnish-software.com>


commit 78a0ecdcb02f5201120a1b8fab5794ce4d974688
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Feb 17 14:17:42 2016 +0100

    VFP_Push: document the place which still need attention for out-of-workspace handling
    
    We'll need to change a couple of signatures in the call paths getting here
    to propagate up the failure.

diff --git a/bin/varnishd/http1/cache_http1_vfp.c b/bin/varnishd/http1/cache_http1_vfp.c
index fe99ebd..1c11384 100644
--- a/bin/varnishd/http1/cache_http1_vfp.c
+++ b/bin/varnishd/http1/cache_http1_vfp.c
@@ -272,16 +272,19 @@ V1F_Setup_Fetch(struct vfp_ctx *vfc, struct http_conn *htc)
 	case BS_EOF:
 		assert(htc->content_length == -1);
 		vfe = VFP_Push(vfc, &v1f_eof, 0);
+		XXXAN(vfe);
 		vfe->priv2 = 0;
 		break;
 	case BS_LENGTH:
 		assert(htc->content_length > 0);
 		vfe = VFP_Push(vfc, &v1f_straight, 0);
+		XXXAN(vfe);
 		vfe->priv2 = htc->content_length;
 		break;
 	case BS_CHUNKED:
 		assert(htc->content_length == -1);
 		vfe = VFP_Push(vfc, &v1f_chunked, 0);
+		XXXAN(vfe);
 		vfe->priv2 = -1;
 		break;
 	default:



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] abc099d improve FetchError message
Message-ID: <E1aYwdO-0006Jz-4d@project.varnish-software.com>


commit abc099de31c7bbd4b6e0eef05af31920faf0e627
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Feb 17 14:34:01 2016 +0100

    improve FetchError message

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index e03d5c2..52623f9 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -537,7 +537,7 @@ vbf_fetch_body_helper(struct busyobj *bo)
 #define vbf_vfp_push(bo, vfp, top)					\
 	if (VFP_Push((bo)->vfc, (vfp), (top)) == NULL) {		\
 		assert (WS_Overflowed((bo)->vfc->http->ws));		\
-		(void)VFP_Error((bo)->vfc, "Bo workspace overflowed");	\
+		(void)VFP_Error((bo)->vfc, "workspace_backend overflow"); \
 		(bo)->htc->doclose = SC_OVERLOAD;			\
 		VDI_Finish((bo)->wrk, bo);				\
 		return (F_STP_ERROR);					\
diff --git a/bin/varnishtest/tests/r01739.vtc b/bin/varnishtest/tests/r01739.vtc
index 5b80f1e..21398cc 100644
--- a/bin/varnishtest/tests/r01739.vtc
+++ b/bin/varnishtest/tests/r01739.vtc
@@ -16,7 +16,7 @@ varnish v1 \
 } -start
 
 logexpect l1 -v v1 -g raw {
-	expect * 1002 FetchError {^Bo workspace overflowed}
+	expect * 1002 FetchError {^workspace_backend overflow}
 	expect * =    Error      {^out of workspace [(]Bo[)]}
 } -start
 



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] 870b74e Polish
Message-ID: <E1aYwdO-0006K9-5i@project.varnish-software.com>


commit 870b74edef9c686cbb7478d4d65a03255f8d8cc8
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Feb 17 16:50:03 2016 +0100

    Polish

diff --git a/bin/varnishd/cache/cache_fetch_proc.c b/bin/varnishd/cache/cache_fetch_proc.c
index 6c38122..e3ee3eb 100644
--- a/bin/varnishd/cache/cache_fetch_proc.c
+++ b/bin/varnishd/cache/cache_fetch_proc.c
@@ -199,7 +199,7 @@ VFP_Push(struct vfp_ctx *vc, const struct vfp *vfp, int top)
 
 	vfe = WS_Alloc(vc->http->ws, sizeof *vfe);
 	if (vfe == NULL)
-		return NULL;
+		return (NULL);
 
 	INIT_OBJ(vfe, VFP_ENTRY_MAGIC);
 	vfe->vfp = vfp;



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] 88cf4e0 Polish
Message-ID: <E1aYwdO-0006KM-7i@project.varnish-software.com>


commit 88cf4e0d537c0f9405f3faa0ab2af7a30246e22a
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Thu Feb 18 18:41:20 2016 +0100

    Polish

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 52623f9..1800b71 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -535,13 +535,16 @@ vbf_fetch_body_helper(struct busyobj *bo)
  */
 
 #define vbf_vfp_push(bo, vfp, top)					\
-	if (VFP_Push((bo)->vfc, (vfp), (top)) == NULL) {		\
-		assert (WS_Overflowed((bo)->vfc->http->ws));		\
-		(void)VFP_Error((bo)->vfc, "workspace_backend overflow"); \
-		(bo)->htc->doclose = SC_OVERLOAD;			\
-		VDI_Finish((bo)->wrk, bo);				\
-		return (F_STP_ERROR);					\
-	}
+	do {								\
+		if (VFP_Push((bo)->vfc, (vfp), (top)) == NULL) {	\
+			assert (WS_Overflowed((bo)->vfc->http->ws));	\
+			(void)VFP_Error((bo)->vfc,			\
+			    "workspace_backend overflow");		\
+			(bo)->htc->doclose = SC_OVERLOAD;		\
+			VDI_Finish((bo)->wrk, bo);			\
+			return (F_STP_ERROR);				\
+		}							\
+	} while (0)
 
 static enum fetch_step
 vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] 5a19445 Further tweaking for smartOS-amd64
Message-ID: <E1aYwdO-0006KU-92@project.varnish-software.com>


commit 5a1944555b9e1b2b0d8fc85c909bdec29294f06e
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Fri Feb 19 19:03:30 2016 +0000

    Further tweaking for smartOS-amd64

diff --git a/bin/varnishtest/tests/c00070.vtc b/bin/varnishtest/tests/c00070.vtc
index 2f8e992..b2df5b4 100644
--- a/bin/varnishtest/tests/c00070.vtc
+++ b/bin/varnishtest/tests/c00070.vtc
@@ -39,7 +39,7 @@ client c1 {
 	expect resp.http.overflowed == "false"
 
 	expect resp.http.free_backend > 9000
-	expect resp.http.free_session >= 240
+	expect resp.http.free_session >= 232
 	expect resp.http.free_thread > 2000
 } -run
 



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] 214db16 Don't capture function typedefs in generate.py
Message-ID: <E1aYwdO-0006Kd-Ae@project.varnish-software.com>


commit 214db1648ead7148e3f7f2ff38db4141ed30a0f4
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Feb 24 11:35:16 2016 +0100

    Don't capture function typedefs in generate.py

diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index a54eaa2..7f2b50b 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -762,6 +762,8 @@ for i in fi:
 		continue
 	if j[-1][-1] != ";":
 		continue
+	if j[-1][-2] == ")":
+		continue
 	if j[-1][:4] != "VCL_":
 		continue
 	d = " ".join(j[1:-1])



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] b16b673 Allow ACLs to be used outside of transactions
Message-ID: <E1aYwdO-0006Ko-CV@project.varnish-software.com>


commit b16b6733d6a0f3c415e3412cf2b8bf5f63c43953
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 16:22:34 2016 +0100

    Allow ACLs to be used outside of transactions

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 09441e0..665dcf3 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -67,7 +67,11 @@ VRT_acl_log(VRT_CTX, const char *msg)
 {
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
-	VSLb(ctx->vsl, SLT_VCL_acl, "%s", msg);
+	AN(msg);
+	if (ctx->vsl != NULL)
+		VSLb(ctx->vsl, SLT_VCL_acl, "%s", msg);
+	else
+		VSL(SLT_VCL_acl, 0, "%s", msg);
 }
 
 /*--------------------------------------------------------------------*/



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] c4c5027 Introduce a new ACL type for VMODs
Message-ID: <E1aYwdO-0006Ky-E4@project.varnish-software.com>


commit c4c5027418232c503afd4a3cf1e96327c008c007
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 16:39:24 2016 +0100

    Introduce a new ACL type for VMODs

diff --git a/include/vrt.h b/include/vrt.h
index ad43ef6..9a27305 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -50,6 +50,7 @@
 /***********************************************************************/
 
 struct VCL_conf;
+struct vrt_acl;
 struct busyobj;
 struct director;
 struct http;
@@ -67,6 +68,7 @@ struct ws;
  * (alphabetic order)
  */
 
+typedef const struct vrt_acl *			VCL_ACL;
 typedef const struct director *			VCL_BACKEND;
 typedef const struct vmod_priv *		VCL_BLOB;
 typedef unsigned				VCL_BOOL;
@@ -237,6 +239,14 @@ struct vrt_ref {
 /* ACL related */
 #define VRT_ACL_MAXADDR		16	/* max(IPv4, IPv6) */
 
+typedef int acl_f (VRT_CTX, VCL_IP);
+
+struct vrt_acl {
+	unsigned	magic;
+#define VRT_ACL_MAGIC	0x78329d96
+	acl_f		*match;
+};
+
 void VRT_acl_log(VRT_CTX, const char *msg);
 
 /* req related */
diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 78d9ec5..98f24d8 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -351,7 +351,7 @@ vcc_acl_emit(struct vcc *tl, const char *acln, int anon)
 	struct token *t;
 	struct inifin *ifp;
 
-	Fh(tl, 0, "\nstatic int\n");
+	Fh(tl, 0, "\nstatic int __match_proto__(acl_f)\n");
 	Fh(tl, 0,
 	    "match_acl_%s_%s(VRT_CTX, const VCL_IP p)\n",
 	    anon ? "anon" : "named", acln);
@@ -443,6 +443,15 @@ vcc_acl_emit(struct vcc *tl, const char *acln, int anon)
 	if (!anon)
 		Fh(tl, 0, "\tVRT_acl_log(ctx, \"NO_MATCH %s\");\n", acln);
 	Fh(tl, 0, "\treturn (0);\n}\n");
+
+	if (anon)
+		return;
+
+	/* Emit the struct that will be referenced */
+	Fh(tl, 0, "\nconst struct vrt_acl vrt_acl_named_%s = {\n", acln);
+	Fh(tl, 0, "\t.magic = VRT_ACL_MAGIC,\n");
+	Fh(tl, 0, "\t.match = &match_acl_named_%s,\n", acln);
+	Fh(tl, 0, "};\n\n");
 }
 
 void



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] 3c11cfd Make named ACLs available to VMODs
Message-ID: <E1aYwdO-0006L8-G3@project.varnish-software.com>


commit 3c11cfd29b2530199aa619c77c8dc3cfe7c5ad91
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 17:55:13 2016 +0100

    Make named ACLs available to VMODs

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 40b7bf7..3020b5d 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -115,7 +115,10 @@ language representation.  Here is a description of them.
 All but the PRIV and STRING_LIST types have typedefs: VCL_INT, VCL_REAL,
 etc.
 
-.. TODO document ACL if patchwork #314 is merged
+ACL
+	C-type: ``const struct vrt_acl *``
+
+        A type for named ACLs declared in VCL.
 
 BACKEND
 	C-type: ``const struct director *``
diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 98f24d8..2fc5fa4 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -474,7 +474,7 @@ void
 vcc_ParseAcl(struct vcc *tl)
 {
 	struct token *an;
-	int i;
+	struct symbol *sym;
 	char acln[1024];
 
 	vcc_NextToken(tl);
@@ -490,13 +490,20 @@ vcc_ParseAcl(struct vcc *tl)
 	an = tl->t;
 	vcc_NextToken(tl);
 
-	i = vcc_AddDef(tl, an, SYM_ACL);
-	if (i > 1) {
+	bprintf(acln, "%.*s", PF(an));
+
+	sym = VCC_GetSymbolTok(tl, an, SYM_ACL);
+	AN(sym);
+	if (sym->ndef > 0) {
 		VSB_printf(tl->sb, "ACL %.*s redefined\n", PF(an));
 		vcc_ErrWhere(tl, an);
 		return;
 	}
-	bprintf(acln, "%.*s", PF(an));
+	sym->fmt = ACL;
+	sym->eval = vcc_Eval_Acl;
+	sym->eval_priv = TlDup(tl, acln);
+	sym->ndef++;
+	ERRCHK(tl);
 
 	SkipToken(tl, '{');
 
diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h
index 3feecd3..5da3d53 100644
--- a/lib/libvcc/vcc_compile.h
+++ b/lib/libvcc/vcc_compile.h
@@ -288,6 +288,7 @@ sym_expr_t vcc_Eval_Var;
 sym_expr_t vcc_Eval_SymFunc;
 void vcc_Eval_Func(struct vcc *tl, const char *cfunc, const char *extra,
     const char *name, const char *args);
+sym_expr_t vcc_Eval_Acl;
 sym_expr_t vcc_Eval_Backend;
 sym_expr_t vcc_Eval_Probe;
 
diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c
index af3b3e7..574ca53 100644
--- a/lib/libvcc/vcc_expr.c
+++ b/lib/libvcc/vcc_expr.c
@@ -494,10 +494,29 @@ vcc_Eval_BoolConst(struct vcc *tl, struct expr **e, const struct symbol *sym)
  */
 
 void
+vcc_Eval_Acl(struct vcc *tl, struct expr **e, const struct symbol *sym)
+{
+
+	assert(sym->kind == SYM_ACL);
+	AN(sym->eval_priv);
+
+	vcc_ExpectCid(tl);
+	vcc_AddRef(tl, tl->t, SYM_ACL);
+	*e = vcc_mk_expr(ACL, "&vrt_acl_named_%s",
+	    (const char *)sym->eval_priv);
+	(*e)->constant = EXPR_VAR;	/* XXX ? */
+	vcc_NextToken(tl);
+}
+
+/*--------------------------------------------------------------------
+ */
+
+void
 vcc_Eval_Backend(struct vcc *tl, struct expr **e, const struct symbol *sym)
 {
 
 	assert(sym->kind == SYM_BACKEND);
+	AN(sym->eval_priv);
 
 	vcc_ExpectCid(tl);
 	vcc_AddRef(tl, tl->t, SYM_BACKEND);
@@ -800,6 +819,7 @@ vcc_expr4(struct vcc *tl, struct expr **e, enum var_type fmt)
 	struct expr *e1, *e2;
 	const char *ip;
 	const struct symbol *sym;
+	enum symkind kind;
 	double d;
 	int i;
 
@@ -819,10 +839,14 @@ vcc_expr4(struct vcc *tl, struct expr **e, enum var_type fmt)
 		 * XXX: look for SYM_VAR first for consistency ?
 		 */
 		sym = NULL;
-		if (fmt == BACKEND)
-			sym = VCC_FindSymbol(tl, tl->t, SYM_BACKEND);
-		if (fmt == PROBE)
-			sym = VCC_FindSymbol(tl, tl->t, SYM_PROBE);
+		switch (fmt) {
+		case ACL:	kind = SYM_ACL; break;
+		case BACKEND:	kind = SYM_BACKEND; break;
+		case PROBE:	kind = SYM_PROBE; break;
+		default:	kind = SYM_NONE; break;
+		}
+		if (kind != SYM_NONE)
+			sym = VCC_FindSymbol(tl, tl->t, kind);
 		if (sym == NULL)
 			sym = VCC_FindSymbol(tl, tl->t, SYM_VAR);
 		if (sym == NULL)
@@ -841,6 +865,7 @@ vcc_expr4(struct vcc *tl, struct expr **e, enum var_type fmt)
 		switch(sym->kind) {
 		case SYM_VAR:
 		case SYM_FUNC:
+		case SYM_ACL:
 		case SYM_BACKEND:
 		case SYM_PROBE:
 			AN(sym->eval);
diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index c018505..07ff129 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -46,6 +46,7 @@ from os.path import dirname, realpath, exists
 from pprint import pprint, pformat
 
 ctypes = {
+	'ACL':		"VCL_ACL",
 	'BACKEND':	"VCL_BACKEND",
 	'BLOB':		"VCL_BLOB",
 	'BOOL':		"VCL_BOOL",



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] c713147 Add a new VRT_acl_match function to the VMODs ABI
Message-ID: <E1aYwdO-0006LM-IM@project.varnish-software.com>


commit c713147813bfedca7e36986769aa4f468ad38729
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 18:26:08 2016 +0100

    Add a new VRT_acl_match function to the VMODs ABI
    
    VRT_MINOR_VERSION needs to be incremented for the 4.1 branch.

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 665dcf3..a73489b 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -39,6 +39,7 @@
 #include "vcl.h"
 #include "vrt.h"
 #include "vrt_obj.h"
+#include "vsa.h"
 #include "vtcp.h"
 #include "vtim.h"
 
@@ -74,6 +75,16 @@ VRT_acl_log(VRT_CTX, const char *msg)
 		VSL(SLT_VCL_acl, 0, "%s", msg);
 }
 
+int
+VRT_acl_match(VRT_CTX, VCL_ACL acl, VCL_IP ip)
+{
+
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+	CHECK_OBJ_NOTNULL(acl, VRT_ACL_MAGIC);
+	assert(VSA_Sane(ip));
+	return (acl->match(ctx, ip));
+}
+
 /*--------------------------------------------------------------------*/
 
 struct http *
diff --git a/include/vrt.h b/include/vrt.h
index 9a27305..449d8b1 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -248,6 +248,7 @@ struct vrt_acl {
 };
 
 void VRT_acl_log(VRT_CTX, const char *msg);
+int VRT_acl_match(VRT_CTX, VCL_ACL, VCL_IP);
 
 /* req related */
 



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] ba2fdde Test VMOD ACLs using vmod-debug
Message-ID: <E1aYwdO-0006LY-K4@project.varnish-software.com>


commit ba2fddea071875191c29d92274a839234d8f6ab0
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Feb 23 18:44:14 2016 +0100

    Test VMOD ACLs using vmod-debug

diff --git a/bin/varnishtest/tests/m00023.vtc b/bin/varnishtest/tests/m00023.vtc
new file mode 100644
index 0000000..b45d5bf
--- /dev/null
+++ b/bin/varnishtest/tests/m00023.vtc
@@ -0,0 +1,32 @@
+varnishtest "Test VMOD ACLs"
+
+varnish v1 -vcl {
+	import debug;
+
+	backend dummy {
+		.host = "${bad_ip}";
+	}
+
+	acl loopback {
+		"127"/24;
+	}
+
+	sub vcl_init {
+		if (!debug.match_acl(loopback, "127.0.0.127")) {
+			debug.init_fail();
+		}
+	}
+
+	sub vcl_recv {
+		if (debug.match_acl(loopback, client.ip)) {
+			return (synth(200));
+		}
+		return (synth(500));
+	}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+} -run
diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index 8bc0dd2..ccf0d80 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -150,3 +150,7 @@ Reset to the previous snapshot of a workspace, taken from debug.workspace_snap.
 $Function VOID vcl_release_delay(DURATION)
 
 Hold a reference to the VCL when it goes cold for the given delay.
+
+$Function BOOL match_acl(ACL acl, IP ip)
+
+Perform an IP match against a named ACL.
diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c
index 839d0ad..971e04c 100644
--- a/lib/libvmod_debug/vmod_debug.c
+++ b/lib/libvmod_debug/vmod_debug.c
@@ -36,6 +36,7 @@
 
 #include "vcl.h"
 #include "vrt.h"
+#include "vsa.h"
 #include "vsb.h"
 #include "vtim.h"
 #include "vcc_if.h"
@@ -464,3 +465,14 @@ vmod_vcl_release_delay(VRT_CTX, VCL_DURATION delay)
 	assert(delay > 0.0);
 	vcl_release_delay = delay;
 }
+
+VCL_BOOL
+vmod_match_acl(VRT_CTX, VCL_ACL acl, VCL_IP ip)
+{
+
+	CHECK_OBJ_ORNULL(ctx, VRT_CTX_MAGIC);
+	CHECK_OBJ_ORNULL(acl, VRT_ACL_MAGIC);
+	assert(VSA_Sane(ip));
+
+	return (VRT_acl_match(ctx, acl, ip));
+}



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] 7b16449 Polish
Message-ID: <E1aYwdO-0006Lj-Lm@project.varnish-software.com>


commit 7b164496abbe6dabd89bc5abd818d2b37319e06b
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Feb 24 15:04:25 2016 +0000

    Polish

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 3020b5d..d3271b5 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -118,7 +118,7 @@ etc.
 ACL
 	C-type: ``const struct vrt_acl *``
 
-        A type for named ACLs declared in VCL.
+	A type for named ACLs declared in VCL.
 
 BACKEND
 	C-type: ``const struct director *``
@@ -152,9 +152,9 @@ DURATION
 	A time interval, as in 25 seconds.
 
 ENUM
-        C-type: ``const char *``
+	C-type: ``const char *``
 
-        TODO
+	TODO
 
 HEADER
 	C-type: ``const struct gethdr_s *``
@@ -169,9 +169,9 @@ HEADER
 	the value, but not where it came from.
 
 HTTP
-        C-type: ``struct http *``
+	C-type: ``struct http *``
 
-        TODO
+	TODO
 
 INT
 	C-type: ``long``
diff --git a/include/vrt.h b/include/vrt.h
index 449d8b1..cc7fc20 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -239,12 +239,12 @@ struct vrt_ref {
 /* ACL related */
 #define VRT_ACL_MAXADDR		16	/* max(IPv4, IPv6) */
 
-typedef int acl_f (VRT_CTX, VCL_IP);
+typedef int acl_match_f(VRT_CTX, const VCL_IP);
 
 struct vrt_acl {
 	unsigned	magic;
 #define VRT_ACL_MAGIC	0x78329d96
-	acl_f		*match;
+	acl_match_f	*match;
 };
 
 void VRT_acl_log(VRT_CTX, const char *msg);
diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 2fc5fa4..97728a9 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -351,7 +351,7 @@ vcc_acl_emit(struct vcc *tl, const char *acln, int anon)
 	struct token *t;
 	struct inifin *ifp;
 
-	Fh(tl, 0, "\nstatic int __match_proto__(acl_f)\n");
+	Fh(tl, 0, "\nstatic int __match_proto__(acl_match_f)\n");
 	Fh(tl, 0,
 	    "match_acl_%s_%s(VRT_CTX, const VCL_IP p)\n",
 	    anon ? "anon" : "named", acln);
@@ -364,7 +364,7 @@ vcc_acl_emit(struct vcc *tl, const char *acln, int anon)
 	Fh(tl, 0, "\t\tVRT_acl_log(ctx, \"NO_FAM %s\");\n", acln);
 	Fh(tl, 0, "\t\treturn(0);\n");
 	Fh(tl, 0, "\t}\n\n");
-	if (!tl->param->err_unref && !anon ) {
+	if (!tl->param->err_unref && !anon) {
 		ifp = New_IniFin(tl);
 		VSB_printf(ifp->ini,
 			"\tif (0) match_acl_named_%s(0, 0);\n", acln);
@@ -419,9 +419,9 @@ vcc_acl_emit(struct vcc *tl, const char *acln, int anon)
 			t = ae->t_addr;
 			do {
 				if (t->tok == CSTR) {
-					Fh(tl, 0, " \"\\\"\" " );
+					Fh(tl, 0, " \"\\\"\" ");
 					EncToken(tl->fh, t);
-					Fh(tl, 0, " \"\\\"\" " );
+					Fh(tl, 0, " \"\\\"\" ");
 				} else
 					Fh(tl, 0, " \"%.*s\"", PF(t));
 				if (t == ae->t_mask)



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] 7d2dbd4 Port test case to 4.1.
Message-ID: <E1aYwdO-0006Lt-Nj@project.varnish-software.com>


commit 7d2dbd438a7c21d84940c422b17d583b35e9f546
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Thu Feb 25 14:55:38 2016 +0100

    Port test case to 4.1.
    
    0a05e52 and friends are missing in 4.1.

diff --git a/bin/varnishtest/tests/m00023.vtc b/bin/varnishtest/tests/m00023.vtc
index b45d5bf..062e30b 100644
--- a/bin/varnishtest/tests/m00023.vtc
+++ b/bin/varnishtest/tests/m00023.vtc
@@ -1,7 +1,7 @@
 varnishtest "Test VMOD ACLs"
 
 varnish v1 -vcl {
-	import debug;
+	import ${vmod_debug};
 
 	backend dummy {
 		.host = "${bad_ip}";



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] 919609f Increase VRT_MINOR_VERSION.
Message-ID: <E1aYwdO-0006M3-PF@project.varnish-software.com>


commit 919609fa6f9fe1fc74106657d3f4af5772d527d2
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Thu Feb 25 14:57:21 2016 +0100

    Increase VRT_MINOR_VERSION.
    
    Cause:
    
      9e07b947 Add a new VRT_acl_match function to the VMODs ABI

diff --git a/include/vrt.h b/include/vrt.h
index cc7fc20..0697ef4 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -44,7 +44,7 @@
 
 #define VRT_MAJOR_VERSION	3U
 
-#define VRT_MINOR_VERSION	1U
+#define VRT_MINOR_VERSION	2U
 
 
 /***********************************************************************/



From lkarsten at varnish-software.com  Thu Feb 25 14:11:14 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 25 Feb 2016 15:11:14 +0100
Subject: [4.1] 8a2dfef Prepare for 4.1.2-beta2.
Message-ID: <E1aYwdO-0006ME-QP@project.varnish-software.com>


commit 8a2dfefef4ac14989666a5f42aa7e78e656fa59b
Author: Lasse Karstensen <lkarsten at varnish-software.com>
Date:   Thu Feb 25 15:07:28 2016 +0100

    Prepare for 4.1.2-beta2.

diff --git a/configure.ac b/configure.ac
index 30f6a33..ab17b65 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2,7 +2,7 @@ AC_PREREQ(2.59)
 AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS
 Copyright (c) 2006-2016 Varnish Software AS])
 AC_REVISION([$Id$])
-AC_INIT([Varnish], [4.1.2-beta1], [varnish-dev at varnish-cache.org])
+AC_INIT([Varnish], [4.1.2-beta2], [varnish-dev at varnish-cache.org])
 AC_CONFIG_SRCDIR(include/miniobj.h)
 AC_CONFIG_HEADERS([config.h])
 AC_CONFIG_MACRO_DIR([m4])
diff --git a/doc/changes.rst b/doc/changes.rst
index 7c8b511..54b877f 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -1,5 +1,24 @@
 
 ======================================
+Varnish Cache 4.1.2-beta2 (2016-02-25)
+======================================
+
+* [vmods] Passing VCL ACL to a vmod is now possible.
+
+* [vmods] VRT_MINOR_VERSION increase due to new function: VRT_acl_match()
+
+* Some test case stabilization fixes and minor documentation updates.
+
+* Improved handling of workspace exhaustion when fetching objects.
+
+Bugs fixed
+----------
+
+* 1858_ - Hit-for-pass objects are not IMS candidates
+
+.. _1858: https://www.varnish-cache.org/trac/ticket/1858
+
+======================================
 Varnish Cache 4.1.2-beta1 (2016-02-17)
 ======================================
 



From phk at FreeBSD.org  Fri Feb 26 08:32:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 26 Feb 2016 09:32:11 +0100
Subject: [master] 362289a Add VSB_destroy(). It's like VSB_delete() but it
 NULL's the pointer.
Message-ID: <E1aZDop-0000ae-3o@project.varnish-software.com>


commit 362289ae023fb02442f0106bd97d69dbbf627b59
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 26 08:19:54 2016 +0000

    Add VSB_destroy().  It's like VSB_delete() but it NULL's the pointer.

diff --git a/include/vsb.h b/include/vsb.h
index a3bdd3d..7761255 100644
--- a/include/vsb.h
+++ b/include/vsb.h
@@ -75,6 +75,7 @@ int		 VSB_finish(struct vsb *);
 char		*VSB_data(const struct vsb *);
 ssize_t		 VSB_len(const struct vsb *);
 void		 VSB_delete(struct vsb *);
+void		 VSB_destroy(struct vsb **);
 #define VSB_QUOTE_NONL	1
 #define VSB_QUOTE_JSON	2
 void		 VSB_quote(struct vsb *s, const char *p, int len, int how);
diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c
index d737ef1..f5915bd 100644
--- a/lib/libvarnish/vsb.c
+++ b/lib/libvarnish/vsb.c
@@ -489,6 +489,13 @@ VSB_delete(struct vsb *s)
 		SBFREE(s);
 }
 
+void
+VSB_destroy(struct vsb **s)
+{
+	VSB_delete(*s);
+	*s = NULL;
+}
+
 /*
  * Quote a string
  */



From phk at FreeBSD.org  Fri Feb 26 08:32:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 26 Feb 2016 09:32:11 +0100
Subject: [master] c36ef1c Use VSB_destroy() instead of VSB_delete() wherever
 possible.
Message-ID: <E1aZDop-0000ak-5v@project.varnish-software.com>


commit c36ef1c08fdc94010aed1c03d967769b8c51c4be
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 26 08:31:51 2016 +0000

    Use VSB_destroy() instead of VSB_delete() wherever possible.

diff --git a/bin/varnishd/cache/cache_backend_cfg.c b/bin/varnishd/cache/cache_backend_cfg.c
index ad96330..21f0782 100644
--- a/bin/varnishd/cache/cache_backend_cfg.c
+++ b/bin/varnishd/cache/cache_backend_cfg.c
@@ -98,7 +98,7 @@ VRT_new_backend(VRT_CTX, const struct vrt_backend *vrt)
 
 	b->display_name = strdup(VSB_data(vsb));
 	AN(b->display_name);
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 
 	b->vcl = vcl;
 
@@ -316,7 +316,7 @@ backend_find(struct cli *cli, const char *matcher, bf_func *func, void *priv)
 		}
 	}
 	Lck_Unlock(&backends_mtx);
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	VCL_Rel(&vcc);
 	return (found);
 }
diff --git a/bin/varnishd/cache/cache_backend_probe.c b/bin/varnishd/cache/cache_backend_probe.c
index e0590d8..5693408 100644
--- a/bin/varnishd/cache/cache_backend_probe.c
+++ b/bin/varnishd/cache/cache_backend_probe.c
@@ -472,7 +472,7 @@ vbp_build_req(struct vbp_target *vt, const struct vrt_backend_probe *vbp,
 	vt->req = strdup(VSB_data(vsb));
 	AN(vt->req);
 	vt->req_len = VSB_len(vsb);
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c
index 4dec68b..55c78ae 100644
--- a/bin/varnishd/cache/cache_ban_build.c
+++ b/bin/varnishd/cache/cache_ban_build.c
@@ -91,7 +91,7 @@ BAN_Abandon(struct ban_proto *bp)
 {
 
 	CHECK_OBJ_NOTNULL(bp, BAN_PROTO_MAGIC);
-	VSB_delete(bp->vsb);
+	VSB_destroy(&bp->vsb);
 	FREE_OBJ(bp);
 }
 
diff --git a/bin/varnishd/cache/cache_deliver_proc.c b/bin/varnishd/cache/cache_deliver_proc.c
index 5060f9f..4f2af67 100644
--- a/bin/varnishd/cache/cache_deliver_proc.c
+++ b/bin/varnishd/cache/cache_deliver_proc.c
@@ -71,7 +71,7 @@ VDP_push(struct req *req, vdp_bytes *func, void *priv, int bottom)
 	Symbol_Lookup(vsb, func);
 	AZ(VSB_finish(vsb));
 	VSLb(req->vsl, SLT_Debug, "VDP %s %p %d", VSB_data(vsb), priv, bottom);
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 #endif
 	vdp = WS_Alloc(req->ws, sizeof *vdp);
 	if (vdp == NULL)
diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
index e1b9a56..734ac2a 100644
--- a/bin/varnishd/cache/cache_esi_fetch.c
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -130,7 +130,7 @@ vfp_esi_end(struct vfp_ctx *vc, struct vef_priv *vef,
 				    "Could not allocate storage for esidata");
 			}
 		}
-		VSB_delete(vsb);
+		VSB_destroy(&vsb);
 	}
 
 	if (vef->vgz != NULL) {
diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c
index e4de056..aa62545 100644
--- a/bin/varnishd/cache/cache_esi_parse.c
+++ b/bin/varnishd/cache/cache_esi_parse.c
@@ -446,8 +446,8 @@ vep_do_include(struct vep_state *vep, enum dowhat what)
 			    "ESI 1.0 <esi:include> "
 			    "has multiple src= attributes");
 			vep->state = VEP_TAGERROR;
-			VSB_delete(vep->attr_vsb);
-			VSB_delete(vep->include_src);
+			VSB_destroy(&vep->attr_vsb);
+			VSB_destroy(&vep->include_src);
 			vep->attr_vsb = NULL;
 			vep->include_src = NULL;
 			return;
@@ -539,7 +539,7 @@ vep_do_include(struct vep_state *vep, enum dowhat what)
 #undef R
 	VSB_printf(vep->vsb, "%c", 0);
 
-	VSB_delete(vep->include_src);
+	VSB_destroy(&vep->include_src);
 	vep->include_src = NULL;
 }
 
@@ -887,7 +887,7 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l)
 				vep->attr_delim = 0;
 				if (vep->attr_vsb != NULL) {
 					AZ(VSB_finish(vep->attr_vsb));
-					VSB_delete(vep->attr_vsb);
+					VSB_destroy(&vep->attr_vsb);
 					vep->attr_vsb = NULL;
 				}
 			} else if (p < e) {
@@ -1061,7 +1061,7 @@ VEP_Finish(struct vep_state *vep)
 	l = VSB_len(vep->vsb);
 	if (vep->esi_found && l > 0)
 		return (vep->vsb);
-	VSB_delete(vep->vsb);
+	VSB_destroy(&vep->vsb);
 	return (NULL);
 }
 
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 609e6c2..6067324 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -131,7 +131,7 @@ vbf_beresp2obj(struct busyobj *bo)
 	if (vary != NULL) {
 		AN(ObjSetAttr(bo->wrk, bo->fetch_objcore, OA_VARY, varyl,
 			VSB_data(vary)));
-		VSB_delete(vary);
+		VSB_destroy(&vary);
 	}
 
 	AZ(ObjSetU32(bo->wrk, bo->fetch_objcore, OA_VXID, VXID(bo->vsl->wid)));
@@ -852,12 +852,12 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
 	AZ(VSB_finish(synth_body));
 
 	if (wrk->handling == VCL_RET_ABANDON) {
-		VSB_delete(synth_body);
+		VSB_destroy(&synth_body);
 		return (F_STP_FAIL);
 	}
 
 	if (wrk->handling == VCL_RET_RETRY) {
-		VSB_delete(synth_body);
+		VSB_destroy(&synth_body);
 		if (bo->retries++ < cache_param->max_retries)
 			return (F_STP_RETRY);
 		VSLb(bo->vsl, SLT_VCL_Error, "Too many retries, failing");
@@ -873,7 +873,7 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
 	bo->vfc->esi_req = bo->bereq;
 
 	if (vbf_beresp2obj(bo)) {
-		VSB_delete(synth_body);
+		VSB_destroy(&synth_body);
 		return (F_STP_FAIL);
 	}
 
@@ -889,7 +889,7 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
 		o += l;
 	}
 	AZ(ObjSetU64(wrk, bo->fetch_objcore, OA_LEN, o));
-	VSB_delete(synth_body);
+	VSB_destroy(&synth_body);
 	HSH_Unbusy(wrk, bo->fetch_objcore);
 	ObjSetState(wrk, bo->fetch_objcore, BOS_FINISHED);
 	return (F_STP_DONE);
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 991883f..94f893b 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -185,7 +185,7 @@ cnt_synth(struct worker *wrk, struct req *req)
 
 	if (wrk->handling == VCL_RET_RESTART) {
 		HTTP_Setup(h, req->ws, req->vsl, SLT_RespMethod);
-		VSB_delete(synth_body);
+		VSB_destroy(&synth_body);
 		req->req_step = R_STP_RESTART;
 		return (REQ_FSM_MORE);
 	}
@@ -210,7 +210,7 @@ cnt_synth(struct worker *wrk, struct req *req)
 	if (szl >= 0)
 		AZ(ObjSetU64(wrk, req->objcore, OA_LEN, szl));
 	HSH_DerefBoc(wrk, req->objcore);
-	VSB_delete(synth_body);
+	VSB_destroy(&synth_body);
 
 	if (szl < 0) {
 		VSLb(req->vsl, SLT_Error, "Could not get storage");
diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c
index a7bd7a3..b983b54 100644
--- a/bin/varnishd/cache/cache_vary.c
+++ b/bin/varnishd/cache/cache_vary.c
@@ -158,15 +158,15 @@ VRY_Create(struct busyobj *bo, struct vsb **psb)
 	}
 
 	if (error) {
-		VSB_delete(sbh);
-		VSB_delete(sb);
+		VSB_destroy(&sbh);
+		VSB_destroy(&sb);
 		return (-1);
 	}
 
 	/* Terminate vary matching string */
 	VSB_printf(sb, "%c%c%c", 0xff, 0xff, 0);
 
-	VSB_delete(sbh);
+	VSB_destroy(&sbh);
 	AZ(VSB_finish(sb));
 	*psb = sb;
 	return (VSB_len(sb));
diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c
index e167db3..bfcba8b 100644
--- a/bin/varnishd/cache/cache_vcl.c
+++ b/bin/varnishd/cache/cache_vcl.c
@@ -339,7 +339,7 @@ VCL_TestLoad(const char *fn)
 		retval = -1;
 	} else
 		VCL_Close(&vcl);
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	return (retval);
 }
 
@@ -600,7 +600,7 @@ VCL_Load(struct cli *cli, const char *name, const char *fn, const char *state)
 		AZ(VSB_finish(vsb));
 		VCLI_SetResult(cli, CLIS_PARAM);
 		VCLI_Out(cli, "%s", VSB_data(vsb));
-		VSB_delete(vsb);
+		VSB_destroy(&vsb);
 		return;
 	}
 
@@ -630,7 +630,7 @@ VCL_Load(struct cli *cli, const char *name, const char *fn, const char *state)
 		vcl_cancel_load(&ctx, cli, name, "warmup");
 		return;
 	}
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	bprintf(vcl->state, "%s", state + 1);
 	assert(hand == VCL_RET_OK);
 	VCLI_Out(cli, "Loaded \"%s\" as \"%s\"", fn , name);
@@ -747,7 +747,7 @@ ccf_config_state(struct cli *cli, const char * const *av, void *priv)
 	AN(ctx.vcl);			// MGT ensures this
 	if (vcl_set_state(&ctx, av[3]) == 0) {
 		bprintf(ctx.vcl->state, "%s", av[3] + 1);
-		VSB_delete(ctx.msg);
+		VSB_destroy(&ctx.msg);
 		return;
 	}
 	AZ(VSB_finish(ctx.msg));
@@ -756,7 +756,7 @@ ccf_config_state(struct cli *cli, const char * const *av, void *priv)
 	    av[3] + 1);
 	if (VSB_len(ctx.msg))
 		VCLI_Out(cli, "\nMessage:\n\t%s", VSB_data(ctx.msg));
-	VSB_delete(ctx.msg);
+	VSB_destroy(&ctx.msg);
 }
 
 static void __match_proto__(cli_func_t)
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index e484252..55c3303 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -111,7 +111,7 @@ mgt_panic_record(pid_t r)
 	char time_str[30];
 
 	if (child_panic != NULL)
-		VSB_delete(child_panic);
+		VSB_destroy(&child_panic);
 	child_panic = VSB_new_auto();
 	AN(child_panic);
 	VTIM_format(VTIM_real(), time_str);
@@ -127,7 +127,7 @@ mgt_panic_record(pid_t r)
 static void
 mgt_panic_clear(void)
 {
-	VSB_delete(child_panic);
+	VSB_destroy(&child_panic);
 	child_panic = NULL;
 }
 
@@ -534,7 +534,7 @@ mgt_reap_child(void)
 #endif
 	AZ(VSB_finish(vsb));
 	MGT_complain(status ? C_ERR : C_INFO, "%s", VSB_data(vsb));
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 
 	/* Dispose of shared memory but evacuate panic messages first */
 	if (heritage.panic_str[0] != '\0') {
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index 6df8f9e..06e1989 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -163,13 +163,13 @@ mcf_askchild(struct cli *cli, const char * const *av, void *priv)
 	AZ(VSB_finish(vsb));
 	i = write(cli_o, VSB_data(vsb), VSB_len(vsb));
 	if (i != VSB_len(vsb)) {
-		VSB_delete(vsb);
+		VSB_destroy(&vsb);
 		VCLI_SetResult(cli, CLIS_COMMS);
 		VCLI_Out(cli, "CLI communication error");
 		MGT_Child_Cli_Fail();
 		return;
 	}
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	if (VCLI_ReadResult(cli_i, &u, &q, mgt_param.cli_timeout))
 		MGT_Child_Cli_Fail();
 	VCLI_SetResult(cli, u);
@@ -485,7 +485,7 @@ telnet_accept(const struct vev *ev, int what)
 	tn = telnet_new(i);
 	vsb = sock_id("telnet", i);
 	mgt_cli_setup(i, i, 0, VSB_data(vsb), telnet_close, tn);
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	return (0);
 }
 
@@ -567,7 +567,7 @@ mgt_cli_telnet(const char *T_arg)
 		ARGV_ERR("-T %s could not be listened on.", T_arg);
 	/* Save in shmem */
 	mgt_SHM_static_alloc(VSB_data(vsb), VSB_len(vsb) + 1, "Arg", "-T", "");
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 }
 
 /* Reverse CLI ("Master") connections --------------------------------*/
@@ -617,7 +617,7 @@ Marg_connect(const struct vev *e, int what)
 	}
 	vsb = sock_id("master", M_fd);
 	mgt_cli_setup(M_fd, M_fd, 0, VSB_data(vsb), Marg_closer, NULL);
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	M_poll = 1;
 	return (1);
 }
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index ec4ff5a..2dc452a 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -170,7 +170,7 @@ MGT_complain(const char *loud, const char *fmt, ...)
 
 	if (!MGT_DO_DEBUG(DBG_VTC_MODE))
 		syslog(sf, "%s", VSB_data(vsb));
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 }
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index 0ee9397..71f9751 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -307,7 +307,7 @@ mcf_param_show(struct cli *cli, const char * const *av, void *priv)
 		VCLI_SetResult(cli, CLIS_PARAM);
 		VCLI_Out(cli, "Unknown parameter \"%s\".", av[2]);
 	}
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 }
 
 /*--------------------------------------------------------------------
@@ -466,7 +466,7 @@ MCF_InitParams(struct cli *cli)
 		AN(pp->def);
 		mcf_wash_param(cli, pp, &pp->def, "default", vsb);
 	}
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 }
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishd/mgt/mgt_vcc.c b/bin/varnishd/mgt/mgt_vcc.c
index ac2c5f6..3810c74 100644
--- a/bin/varnishd/mgt/mgt_vcc.c
+++ b/bin/varnishd/mgt/mgt_vcc.c
@@ -104,7 +104,7 @@ run_vcc(void *priv)
 	AZ(VSB_finish(sb));
 	if (VSB_len(sb))
 		printf("%s", VSB_data(sb));
-	VSB_delete(sb);
+	VSB_destroy(&sb);
 	if (csrc == NULL)
 		exit(2);
 
@@ -173,7 +173,7 @@ run_cc(void *priv)
 
 	(void)umask(027);
 	(void)execl("/bin/sh", "/bin/sh", "-c", VSB_data(sb), (char*)0);
-	VSB_delete(sb);				// For flexelint
+	VSB_destroy(&sb);				// For flexelint
 }
 
 /*--------------------------------------------------------------------
@@ -291,7 +291,7 @@ mgt_VccCompile(struct cli *cli, const char *vclname, const char *vclsrc,
 	AZ(VSB_finish(sb));
 	if (VSB_len(sb) > 0)
 		VCLI_Out(cli, "%s", VSB_data(sb));
-	VSB_delete(sb);
+	VSB_destroy(&sb);
 
 	(void)unlink(vp.csrcfile);
 	free(vp.csrcfile);
diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c
index 249efee..cd42e48 100644
--- a/bin/varnishncsa/varnishncsa.c
+++ b/bin/varnishncsa/varnishncsa.c
@@ -681,7 +681,7 @@ parse_format(const char *format)
 		VSB_clear(vsb);
 	}
 
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 }
 
 static int
diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index 8890a62..5e6770a 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -234,7 +234,7 @@ macro_expand(struct vtclog *vl, const char *text)
 		p += 2;
 		m = macro_get(p, q);
 		if (m == NULL) {
-			VSB_delete(vsb);
+			VSB_destroy(&vsb);
 			vtc_log(vl, 0, "Macro ${%.*s} not found", (int)(q - p),
 			    p);
 			NEEDLESS_RETURN (NULL);
@@ -480,7 +480,7 @@ cmd_err_shell(CMD_ARGS)
 	else
 		vtc_log(vl, 4,
 		    "Found expected string: (\"%s\")", av[1]);
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 }
 
 /**********************************************************************
diff --git a/bin/varnishtest/vtc_client.c b/bin/varnishtest/vtc_client.c
index 804ef6d..74bc8e7 100644
--- a/bin/varnishtest/vtc_client.c
+++ b/bin/varnishtest/vtc_client.c
@@ -106,7 +106,7 @@ client_thread(void *priv)
 		VTCP_close(&fd);
 	}
 	vtc_log(vl, 2, "Ending");
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	free(p);
 	return (NULL);
 }
diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index 9160a66..7a7cd24 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -1399,7 +1399,7 @@ http_process(struct vtclog *vl, const char *spec, int sock, int *sfd)
 	VTCP_hisname(sock, hp->rem_ip, VTCP_ADDRBUFSIZE, hp->rem_port, VTCP_PORTBUFSIZE);
 	parse_string(spec, http_cmds, hp, vl);
 	retval = hp->fd;
-	VSB_delete(hp->vsb);
+	VSB_destroy(&hp->vsb);
 	free(hp->rxbuf);
 	free(hp->rem_ip);
 	free(hp->rem_port);
diff --git a/bin/varnishtest/vtc_log.c b/bin/varnishtest/vtc_log.c
index 9b1ddf0..67918c3 100644
--- a/bin/varnishtest/vtc_log.c
+++ b/bin/varnishtest/vtc_log.c
@@ -77,7 +77,7 @@ vtc_logclose(struct vtclog *vl)
 	CHECK_OBJ_NOTNULL(vl, VTCLOG_MAGIC);
 	if (pthread_getspecific(log_key) == vl)
 		AZ(pthread_setspecific(log_key, NULL));
-	VSB_delete(vl->vsb);
+	VSB_destroy(&vl->vsb);
 	AZ(pthread_mutex_destroy(&vl->mtx));
 	FREE_OBJ(vl);
 }
diff --git a/bin/varnishtest/vtc_logexp.c b/bin/varnishtest/vtc_logexp.c
index 1c0ea70..c82b8e4 100644
--- a/bin/varnishtest/vtc_logexp.c
+++ b/bin/varnishtest/vtc_logexp.c
@@ -122,7 +122,7 @@ logexp_delete_tests(struct logexp *le)
 	while ((test = VTAILQ_FIRST(&le->tests))) {
 		CHECK_OBJ_NOTNULL(test, LOGEXP_TEST_MAGIC);
 		VTAILQ_REMOVE(&le->tests, test, list);
-		VSB_delete(test->str);
+		VSB_destroy(&test->str);
 		if (test->vre)
 			VRE_free(&test->vre);
 		FREE_OBJ(test);
@@ -142,7 +142,7 @@ logexp_delete(struct logexp *le)
 	free(le->query);
 	VSM_Delete(le->vsm);
 	if (le->n_arg)
-		VSB_delete(le->n_arg);
+		VSB_destroy(&le->n_arg);
 	FREE_OBJ(le);
 }
 
@@ -529,7 +529,7 @@ cmd_logexp(CMD_ARGS)
 				return;
 			}
 			if (le->n_arg != NULL) {
-				VSB_delete(le->n_arg);
+				VSB_destroy(&le->n_arg);
 				le->n_arg = NULL;
 			}
 			vsb = VSB_new_auto();
@@ -537,7 +537,7 @@ cmd_logexp(CMD_ARGS)
 			AZ(VSB_printf(vsb, "%s/%s", tmpdir, av[1]));
 			AZ(VSB_finish(vsb));
 			le->n_arg = macro_expand(le->vl, VSB_data(vsb));
-			VSB_delete(vsb);
+			VSB_destroy(&vsb);
 			if (le->n_arg == NULL)
 				return;
 			av++;
diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index cd3fb1d..56efdb9 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -395,7 +395,7 @@ i_mode(void)
 	vmod_path = strdup(VSB_data(vsb));
 	AN(vmod_path);
 	free(topbuild);
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 }
 
 /**********************************************************************
diff --git a/bin/varnishtest/vtc_process.c b/bin/varnishtest/vtc_process.c
index f63c2e5..c87f7dc 100644
--- a/bin/varnishtest/vtc_process.c
+++ b/bin/varnishtest/vtc_process.c
@@ -78,7 +78,7 @@ static VTAILQ_HEAD(, process)	processes =
 		AN(vsb);				\
 		p->field = strdup(VSB_data(vsb));	\
 		AN(p->field);				\
-		VSB_delete(vsb);			\
+		VSB_destroy(&vsb);			\
 	} while (0)
 
 static struct process *
@@ -214,7 +214,7 @@ process_start(struct process *p)
 	AZ(close(out_fd));
 	AZ(close(err_fd));
 	p->fds[0] = -1;
-	VSB_delete(cl);
+	VSB_destroy(&cl);
 	AZ(pthread_create(&p->tp, NULL, process_thread, p));
 }
 
diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index 0afe475..794f615 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -296,7 +296,7 @@ varnish_new(const char *name)
 	AN(vsb);
 	v->workdir = strdup(VSB_data(vsb));
 	AN(v->workdir);
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 
 	bprintf(buf, "rm -rf %s ; mkdir -p %s", v->workdir, v->workdir);
 	AZ(system(buf));
@@ -429,7 +429,7 @@ varnish_launch(struct varnish *v)
 	vtc_log(v->vl, 3, "CMD: %s", VSB_data(vsb));
 	vsb1 = macro_expand(v->vl, VSB_data(vsb));
 	AN(vsb1);
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	vsb = vsb1;
 	vtc_log(v->vl, 3, "CMD: %s", VSB_data(vsb));
 	AZ(pipe(&v->fds[0]));
@@ -456,7 +456,7 @@ varnish_launch(struct varnish *v)
 	AZ(close(v->fds[3]));
 	v->fds[0] = v->fds[2];
 	v->fds[2] = v->fds[3] = -1;
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	AZ(pthread_create(&v->tp, NULL, varnish_thread, v));
 	AZ(pthread_create(&v->tp_vsl, NULL, varnishlog_thread, v));
 
@@ -712,7 +712,7 @@ varnish_vcl(struct varnish *v, const char *vcl, enum VCLI_status_e expect,
 
 	u = varnish_ask_cli(v, VSB_data(vsb), resp);
 	if (u != expect) {
-		VSB_delete(vsb);
+		VSB_destroy(&vsb);
 		vtc_log(v->vl, 0,
 		    "VCL compilation got %u expected %u",
 		    u, expect);
@@ -727,7 +727,7 @@ varnish_vcl(struct varnish *v, const char *vcl, enum VCLI_status_e expect,
 	} else {
 		vtc_log(v->vl, 2, "VCL compilation failed (as expected)");
 	}
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 }
 
 /**********************************************************************
@@ -762,8 +762,8 @@ varnish_vclbackend(struct varnish *v, const char *vcl)
 
 	u = varnish_ask_cli(v, VSB_data(vsb), NULL);
 	if (u != CLIS_OK) {
-		VSB_delete(vsb);
-		VSB_delete(vsb2);
+		VSB_destroy(&vsb);
+		VSB_destroy(&vsb2);
 		vtc_log(v->vl, 0, "FAIL VCL does not compile");
 		return;
 	}
@@ -772,8 +772,8 @@ varnish_vclbackend(struct varnish *v, const char *vcl)
 	AZ(VSB_finish(vsb));
 	u = varnish_ask_cli(v, VSB_data(vsb), NULL);
 	assert(u == CLIS_OK);
-	VSB_delete(vsb);
-	VSB_delete(vsb2);
+	VSB_destroy(&vsb);
+	VSB_destroy(&vsb2);
 }
 
 /**********************************************************************
diff --git a/lib/libvarnish/cli_serve.c b/lib/libvarnish/cli_serve.c
index 9861e0c..87e8b54 100644
--- a/lib/libvarnish/cli_serve.c
+++ b/lib/libvarnish/cli_serve.c
@@ -452,7 +452,7 @@ cls_vlu(void *priv, const char *p)
 		cfd->argv = NULL;
 		free(cli->cmd);
 		cli->cmd = NULL;
-		VSB_delete(cfd->last_arg);
+		VSB_destroy(&cfd->last_arg);
 		cfd->last_arg = NULL;
 		cfd->last_idx = 0;
 		return (i);
@@ -512,7 +512,7 @@ cls_close_fd(struct VCLS *cs, struct VCLS_fd *cfd)
 	VTAILQ_REMOVE(&cs->fds, cfd, list);
 	cs->nfd--;
 	VLU_Destroy(cfd->cli->vlu);
-	VSB_delete(cfd->cli->sb);
+	VSB_destroy(&cfd->cli->sb);
 	if (cfd->closefunc == NULL) {
 		(void)close(cfd->fdi);
 		if (cfd->fdo != cfd->fdi)
diff --git a/lib/libvarnish/vfil.c b/lib/libvarnish/vfil.c
index 319c3e1..9c7af4d 100644
--- a/lib/libvarnish/vfil.c
+++ b/lib/libvarnish/vfil.c
@@ -330,11 +330,11 @@ VFIL_searchpath(const struct vfil_path *vp, vfil_path_func_f *func, void *priv,
 			e = errno;
 			*fno = strdup(VSB_data(vsb));
 			AN(*fno);
-			VSB_delete(vsb);
+			VSB_destroy(&vsb);
 			errno = e;
 			return (i);
 		}
 	}
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	return (-1);
 }
diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c
index 0197e11..24feeee 100644
--- a/lib/libvarnishapi/vsl.c
+++ b/lib/libvarnishapi/vsl.c
@@ -160,7 +160,7 @@ VSL_ResetError(struct VSL_data *vsl)
 
 	if (vsl->diag == NULL)
 		return;
-	VSB_delete(vsl->diag);
+	VSB_destroy(&vsl->diag);
 	vsl->diag = NULL;
 }
 
diff --git a/lib/libvarnishapi/vsl_query.c b/lib/libvarnishapi/vsl_query.c
index 0159429..b2e0860 100644
--- a/lib/libvarnishapi/vsl_query.c
+++ b/lib/libvarnishapi/vsl_query.c
@@ -323,7 +323,7 @@ vslq_newquery(struct VSL_data *vsl, enum VSL_grouping_e grouping,
 		XXXAN(query);
 		query->vex = vex;
 	}
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	return (query);
 }
 
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 8e1aa7f..92b636e 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -123,7 +123,7 @@ VSM_ResetError(struct VSM_data *vd)
 
 	if (vd->diag == NULL)
 		return;
-	VSB_delete(vd->diag);
+	VSB_destroy(&vd->diag);
 	vd->diag = NULL;
 }
 
diff --git a/lib/libvarnishapi/vxp_test.c b/lib/libvarnishapi/vxp_test.c
index 55e5820..cbb598a 100644
--- a/lib/libvarnishapi/vxp_test.c
+++ b/lib/libvarnishapi/vxp_test.c
@@ -72,11 +72,11 @@ main(int argc, char * const *argv)
 	if (vex == NULL) {
 		VSB_finish(vsb);
 		fprintf(stderr, "Error:\n%s", VSB_data(vsb));
-		VSB_delete(vsb);
+		VSB_destroy(&vsb);
 		free(q_arg);
 		exit(1);
 	}
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 
 	vex_Free(&vex);
 	AZ(vex);
diff --git a/lib/libvcc/vcc_backend.c b/lib/libvcc/vcc_backend.c
index 3260f65..d544ffd 100644
--- a/lib/libvcc/vcc_backend.c
+++ b/lib/libvcc/vcc_backend.c
@@ -123,7 +123,7 @@ vcc_ParseProbeSpec(struct vcc *tl, const struct token *nm, char **name)
 		VSB_printf(vsb, "vgc_probe__%d", tl->nprobe++);
 	AZ(VSB_finish(vsb));
 	retval = TlDup(tl, VSB_data(vsb));
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 	if (name != NULL)
 		*name = retval;
 
@@ -436,7 +436,7 @@ vcc_ParseHostDef(struct vcc *tl, const struct token *t_be, const char *vgcname)
 	tl->fb = NULL;
 	AZ(VSB_finish(vsb));
 	Fh(tl, 0, "%s", VSB_data(vsb));
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 
 	ifp = New_IniFin(tl);
 	VSB_printf(ifp->ini,
diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c
index 0f94b21..2161cb7 100644
--- a/lib/libvcc/vcc_compile.c
+++ b/lib/libvcc/vcc_compile.c
@@ -351,7 +351,7 @@ EmitInitFini(const struct vcc *tl)
 		if (VSB_len(p->ini))
 			Fc(tl, 0, "\t/* %u */\n%s\n", p->n, VSB_data(p->ini));
 		Fc(tl, 0, "\tvgc_inistep = %u;\n\n", p->n);
-		VSB_delete(p->ini);
+		VSB_destroy(&p->ini);
 
 		AZ(VSB_finish(p->event));
 		if (VSB_len(p->event))
@@ -376,7 +376,7 @@ EmitInitFini(const struct vcc *tl)
 			Fc(tl, 0, "%s\n", VSB_data(p->fin));
 			Fc(tl, 0, "\t}\n\n");
 		}
-		VSB_delete(p->fin);
+		VSB_destroy(&p->fin);
 	}
 
 	Fc(tl, 0, "\treturn(0);\n");
@@ -435,7 +435,7 @@ EmitInitFini(const struct vcc *tl)
 				Fc(tl, 0, "\t    %s != 0)\n", VSB_data(p->event));
 				Fc(tl, 0, "\t\tretval = 1;\n\n");
 			}
-			VSB_delete(p->event);
+			VSB_destroy(&p->event);
 		}
 
 		Fc(tl, 0, "\treturn (retval);\n");
@@ -596,7 +596,7 @@ vcc_resolve_includes(struct vcc *tl)
 			VSB_cat(vsb, t1->dec + 1);
 			AZ(VSB_finish(vsb));
 			sp = vcc_file_source(tl->param, tl->sb, VSB_data(vsb));
-			VSB_delete(vsb);
+			VSB_destroy(&vsb);
 		} else {
 			sp = vcc_file_source(tl->param, tl->sb, t1->dec);
 		}
@@ -683,10 +683,10 @@ vcc_DestroyTokenList(struct vcc *tl, char *ret)
 		FREE_OBJ(sym);
 	}
 
-	VSB_delete(tl->fh);
-	VSB_delete(tl->fc);
+	VSB_destroy(&tl->fh);
+	VSB_destroy(&tl->fc);
 	for (i = 0; i < VCL_MET_MAX; i++)
-		VSB_delete(tl->fm[i]);
+		VSB_destroy(&tl->fm[i]);
 
 	free(tl);
 	return (ret);
@@ -839,7 +839,7 @@ vcc_CompileSource(const struct vcp * const vcp, struct vsb *sb,
 	of = strdup(VSB_data(vsb));
 	AN(of);
 
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 
 	/* done */
 	return (vcc_DestroyTokenList(tl, of));
diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c
index 574ca53..b84d67f 100644
--- a/lib/libvcc/vcc_expr.c
+++ b/lib/libvcc/vcc_expr.c
@@ -271,7 +271,7 @@ vcc_delete_expr(struct expr *e)
 	if (e == NULL)
 		return;
 	CHECK_OBJ_NOTNULL(e, EXPR_MAGIC);
-	VSB_delete(e->vsb);
+	VSB_destroy(&e->vsb);
 	FREE_OBJ(e);
 }
 /*--------------------------------------------------------------------
diff --git a/lib/libvcc/vcc_utils.c b/lib/libvcc/vcc_utils.c
index 40f0f86..a95c20c 100644
--- a/lib/libvcc/vcc_utils.c
+++ b/lib/libvcc/vcc_utils.c
@@ -219,7 +219,7 @@ Resolve_Sockaddr(struct vcc *tl,
 		vcc_ErrWhere(tl, t_err);
 		free(rss->vsa4);
 		free(rss->vsa6);
-		VSB_delete(rss->vsb);
+		VSB_destroy(&rss->vsb);
 		FREE_OBJ(rss);
 		return;
 	}
@@ -250,6 +250,6 @@ Resolve_Sockaddr(struct vcc *tl,
 		    VSB_data(rss->vsb));
 		vcc_ErrWhere(tl, t_err);
 	}
-	VSB_delete(rss->vsb);
+	VSB_destroy(&rss->vsb);
 	FREE_OBJ(rss);
 }
diff --git a/lib/libvcc/vcc_var.c b/lib/libvcc/vcc_var.c
index a018f36..4b03269 100644
--- a/lib/libvcc/vcc_var.c
+++ b/lib/libvcc/vcc_var.c
@@ -79,7 +79,7 @@ vcc_Var_Wildcard(struct vcc *tl, const struct token *t, const struct symbol *wc)
 	VSB_printf(vsb, "VRT_SetHdr(ctx, %s,", v->rname);
 	AZ(VSB_finish(vsb));
 	v->lname = TlDup(tl, VSB_data(vsb));
-	VSB_delete(vsb);
+	VSB_destroy(&vsb);
 
 	sym = VCC_AddSymbolTok(tl, t, SYM_VAR);
 	AN(sym);



From phk at FreeBSD.org  Fri Feb 26 08:46:48 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 26 Feb 2016 09:46:48 +0100
Subject: [master] 6ec734a Try to close one of the leaks reported in #1860
Message-ID: <E1aZE2y-0002vO-96@project.varnish-software.com>


commit 6ec734af32e13f13e22764b056314328873a8062
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 26 08:46:25 2016 +0000

    Try to close one of the leaks reported in #1860

diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c
index aa62545..dcfa96c 100644
--- a/bin/varnishd/cache/cache_esi_parse.c
+++ b/bin/varnishd/cache/cache_esi_parse.c
@@ -448,11 +448,10 @@ vep_do_include(struct vep_state *vep, enum dowhat what)
 			vep->state = VEP_TAGERROR;
 			VSB_destroy(&vep->attr_vsb);
 			VSB_destroy(&vep->include_src);
-			vep->attr_vsb = NULL;
-			vep->include_src = NULL;
 			return;
 		}
 		vep->include_src = vep->attr_vsb;
+		vep->attr_vsb = NULL;
 		return;
 	}
 	assert(what == DO_TAG);
@@ -490,8 +489,8 @@ vep_do_include(struct vep_state *vep, enum dowhat what)
 			vep_warn(vep,
 			    "ESI 1.0 <esi:include> with https:// ignored");
 			vep->state = VEP_TAGERROR;
-			vep->attr_vsb = NULL;
-			vep->include_src = NULL;
+			AZ(vep->attr_vsb);
+			VSB_destroy(&vep->include_src);
 			return;
 		}
 		vep_warn(vep,
@@ -1048,6 +1047,8 @@ VEP_Finish(struct vep_state *vep)
 
 	CHECK_OBJ_NOTNULL(vep, VEP_MAGIC);
 
+	AZ(vep->include_src);
+	AZ(vep->attr_vsb);
 	if (vep->o_pending)
 		vep_mark_common(vep, vep->ver_p, vep->last_mark);
 	if (vep->o_wait > 0) {



From dridi.boukelmoune at gmail.com  Fri Feb 26 09:09:11 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Fri, 26 Feb 2016 10:09:11 +0100
Subject: [master] be7d170 No need to assign NULL after VSB_destroy
Message-ID: <E1aZEOd-0005Is-CO@project.varnish-software.com>


commit be7d170db031d538d07c1802453a5cfc8add13a5
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Fri Feb 26 10:08:35 2016 +0100

    No need to assign NULL after VSB_destroy

diff --git a/bin/varnishd/cache/cache_esi_parse.c b/bin/varnishd/cache/cache_esi_parse.c
index dcfa96c..6ba5eb1 100644
--- a/bin/varnishd/cache/cache_esi_parse.c
+++ b/bin/varnishd/cache/cache_esi_parse.c
@@ -537,9 +537,7 @@ vep_do_include(struct vep_state *vep, enum dowhat what)
 	}
 #undef R
 	VSB_printf(vep->vsb, "%c", 0);
-
 	VSB_destroy(&vep->include_src);
-	vep->include_src = NULL;
 }
 
 /*---------------------------------------------------------------------
@@ -887,7 +885,6 @@ VEP_Parse(struct vep_state *vep, const char *p, size_t l)
 				if (vep->attr_vsb != NULL) {
 					AZ(VSB_finish(vep->attr_vsb));
 					VSB_destroy(&vep->attr_vsb);
-					vep->attr_vsb = NULL;
 				}
 			} else if (p < e) {
 				vep->attr_delim = 0;
diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 55c3303..1673169 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -128,7 +128,6 @@ static void
 mgt_panic_clear(void)
 {
 	VSB_destroy(&child_panic);
-	child_panic = NULL;
 }
 
 void __match_proto__(cli_func_t)
diff --git a/bin/varnishtest/vtc_logexp.c b/bin/varnishtest/vtc_logexp.c
index c82b8e4..cca71c8 100644
--- a/bin/varnishtest/vtc_logexp.c
+++ b/bin/varnishtest/vtc_logexp.c
@@ -528,10 +528,8 @@ cmd_logexp(CMD_ARGS)
 				vtc_log(le->vl, 0, "Missing -v argument");
 				return;
 			}
-			if (le->n_arg != NULL) {
+			if (le->n_arg != NULL)
 				VSB_destroy(&le->n_arg);
-				le->n_arg = NULL;
-			}
 			vsb = VSB_new_auto();
 			AN(vsb);
 			AZ(VSB_printf(vsb, "%s/%s", tmpdir, av[1]));
diff --git a/lib/libvarnish/cli_serve.c b/lib/libvarnish/cli_serve.c
index 87e8b54..6e3e41f 100644
--- a/lib/libvarnish/cli_serve.c
+++ b/lib/libvarnish/cli_serve.c
@@ -453,7 +453,6 @@ cls_vlu(void *priv, const char *p)
 		free(cli->cmd);
 		cli->cmd = NULL;
 		VSB_destroy(&cfd->last_arg);
-		cfd->last_arg = NULL;
 		cfd->last_idx = 0;
 		return (i);
 	}
diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c
index 24feeee..2bd801c 100644
--- a/lib/libvarnishapi/vsl.c
+++ b/lib/libvarnishapi/vsl.c
@@ -161,7 +161,6 @@ VSL_ResetError(struct VSL_data *vsl)
 	if (vsl->diag == NULL)
 		return;
 	VSB_destroy(&vsl->diag);
-	vsl->diag = NULL;
 }
 
 static int
diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c
index 92b636e..4f614d5 100644
--- a/lib/libvarnishapi/vsm.c
+++ b/lib/libvarnishapi/vsm.c
@@ -124,7 +124,6 @@ VSM_ResetError(struct VSM_data *vd)
 	if (vd->diag == NULL)
 		return;
 	VSB_destroy(&vd->diag);
-	vd->diag = NULL;
 }
 
 /*--------------------------------------------------------------------*/



From phk at FreeBSD.org  Fri Feb 26 17:14:11 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 26 Feb 2016 18:14:11 +0100
Subject: [master] 29313f1 Always close the VFP stack, also on BS_NONE.
Message-ID: <E1aZLxz-0007YK-2U@project.varnish-software.com>


commit 29313f17d58cd36e8926103316ad4e866e195527
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 26 17:13:42 2016 +0000

    Always close the VFP stack, also on BS_NONE.
    
    Fixes:	#1860

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 6067324..a7dcc51 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -523,8 +523,6 @@ vbf_fetch_body_helper(struct busyobj *bo)
 		}
 	} while (vfps == VFP_OK);
 
-	VFP_Close(vfc);
-
 	if (vfps == VFP_ERROR) {
 		AN(vfc->failed);
 		(void)VFP_Error(vfc, "Fetch pipeline failed to process");
@@ -686,6 +684,8 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 		vbf_fetch_body_helper(bo);
 	}
 
+	VFP_Close(bo->vfc);
+
 	if (bo->vfc->failed) {
 		VDI_Finish(bo->wrk, bo);
 		if (!bo->do_stream) {



From phk at FreeBSD.org  Fri Feb 26 21:55:23 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 26 Feb 2016 22:55:23 +0100
Subject: [master] 4410846 Centralize objcore and boc create/destroy.
Message-ID: <E1aZQM7-0002No-Nq@project.varnish-software.com>


commit 44108468b65654f894a1a39634dc7f0ba2b95e31
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 26 21:54:39 2016 +0000

    Centralize objcore and boc create/destroy.
    
    Rename ObjStable() to ObjBocDone() which is more precise.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 08eaa2d..fc3ff5e 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -859,6 +859,7 @@ void MPL_Free(struct mempool *mpl, void *item);
 
 /* cache_obj.c */
 struct objcore * ObjNew(struct worker *);
+void ObjDestroy(struct worker *, struct objcore **);
 typedef int objiterate_f(void *priv, int flush, const void *ptr, ssize_t len);
 int ObjIterate(struct worker *, struct objcore *,
     void *priv, objiterate_f *func);
@@ -882,7 +883,7 @@ void *ObjSetAttr(struct worker *, struct objcore *, enum obj_attr,
     ssize_t len, const void *);
 int ObjCopyAttr(struct worker *, struct objcore *, struct objcore *,
     enum obj_attr attr);
-void ObjStable(struct worker *, struct objcore *, struct boc *);
+void ObjBocDone(struct worker *, struct objcore *, struct boc **);
 
 int ObjSetDouble(struct worker *, struct objcore *, enum obj_attr, double);
 int ObjSetU32(struct worker *, struct objcore *, enum obj_attr, uint32_t);
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 86523be..f952769 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -91,10 +91,8 @@ hsh_prealloc(struct worker *wrk)
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
-	if (wrk->nobjcore == NULL) {
+	if (wrk->nobjcore == NULL)
 		wrk->nobjcore = ObjNew(wrk);
-		wrk->nobjcore->flags |= OC_F_BUSY;
-	}
 	CHECK_OBJ_NOTNULL(wrk->nobjcore, OBJCORE_MAGIC);
 
 	if (wrk->nobjhead == NULL) {
@@ -120,7 +118,7 @@ HSH_Private(struct worker *wrk)
 	AN(oc);
 	oc->refcnt = 1;
 	oc->objhead = private_oh;
-	oc->flags |= OC_F_PRIVATE | OC_F_BUSY;
+	oc->flags |= OC_F_PRIVATE;
 	Lck_Lock(&private_oh->mtx);
 	VTAILQ_INSERT_TAIL(&private_oh->objcs, oc, hsh_list);
 	private_oh->refcnt++;
@@ -133,11 +131,9 @@ void
 HSH_Cleanup(struct worker *wrk)
 {
 
-	if (wrk->nobjcore != NULL) {
-		FREE_OBJ(wrk->nobjcore);
-		wrk->stats->n_objectcore--;
-		wrk->nobjcore = NULL;
-	}
+	if (wrk->nobjcore != NULL)
+		ObjDestroy(wrk, &wrk->nobjcore);
+
 	if (wrk->nobjhead != NULL) {
 		Lck_Delete(&wrk->nobjhead->mtx);
 		FREE_OBJ(wrk->nobjhead);
@@ -798,14 +794,8 @@ HSH_DerefBoc(struct worker *wrk, struct objcore *oc)
 	if (r == 0)
 		oc->boc = NULL;
 	Lck_Unlock(&oc->objhead->mtx);
-	if (r == 0) {
-		if (oc->stobj->stevedore != NULL)
-			ObjStable(wrk, oc, boc);
-		AZ(pthread_cond_destroy(&boc->cond));
-		Lck_Delete(&boc->mtx);
-		free(boc->vary);
-		FREE_OBJ(boc);
-	}
+	if (r == 0)
+		ObjBocDone(wrk, oc, &boc);
 }
 
 /*--------------------------------------------------------------------
@@ -850,9 +840,8 @@ HSH_DerefObjCore(struct worker *wrk, struct objcore **ocp)
 
 	if (oc->stobj->stevedore != NULL)
 		ObjFreeObj(wrk, oc);
-	FREE_OBJ(oc);
+	ObjDestroy(wrk, &oc);
 
-	wrk->stats->n_objectcore--;
 	/* Drop our ref on the objhead */
 	assert(oh->refcnt > 0);
 	(void)HSH_DerefObjHead(wrk, &oh);
diff --git a/bin/varnishd/cache/cache_obj.c b/bin/varnishd/cache/cache_obj.c
index 09ddeed..b4201cb 100644
--- a/bin/varnishd/cache/cache_obj.c
+++ b/bin/varnishd/cache/cache_obj.c
@@ -54,7 +54,7 @@
  * 2	  ObjSetU32()
  * 2	  ObjSetU64()
  *
- * 2->3	ObjStable()	Will no longer be modified (clean up boc)
+ * 2->3	ObjBocDone()	Boc removed from OC, clean it up
  *
  * 23	ObjHasAttr()
  * 23	ObjGetAttr()
@@ -99,6 +99,34 @@ obj_getmethods(const struct objcore *oc)
 	return (oc->stobj->stevedore->methods);
 }
 
+static struct boc *
+obj_newboc(void)
+{
+	struct boc *boc;
+
+	ALLOC_OBJ(boc, BOC_MAGIC);
+	AN(boc);
+	Lck_New(&boc->mtx, lck_busyobj);
+	AZ(pthread_cond_init(&boc->cond, NULL));
+	boc->refcount = 1;
+	return (boc);
+}
+
+static void
+obj_deleteboc(struct boc **p)
+{
+	struct boc *boc;
+
+	AN(p);
+	boc = *p;
+	*p = NULL;
+	Lck_Delete(&boc->mtx);
+	AZ(pthread_cond_destroy(&boc->cond));
+	if (boc->vary != NULL)
+		free(boc->vary);
+	FREE_OBJ(boc);
+}
+
 /*====================================================================
  * ObjNew()
  *
@@ -115,15 +143,35 @@ ObjNew(struct worker *wrk)
 	AN(oc);
 	wrk->stats->n_objectcore++;
 	oc->last_lru = NAN;
-	ALLOC_OBJ(oc->boc, BOC_MAGIC);
-	AN(oc->boc);
-	Lck_New(&oc->boc->mtx, lck_busyobj);
-	AZ(pthread_cond_init(&oc->boc->cond, NULL));
-	oc->boc->refcount = 1;
+	oc->flags = OC_F_BUSY;
+
+	oc->boc = obj_newboc();
+
 	return (oc);
 }
 
 /*====================================================================
+ * ObjDestroy()
+ *
+ */
+
+void
+ObjDestroy(struct worker *wrk, struct objcore **p)
+{
+	struct objcore *oc;
+
+	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
+	AN(p);
+	oc = *p;
+	*p = NULL;
+	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	if (oc->boc != NULL)
+		obj_deleteboc(&oc->boc);
+	FREE_OBJ(oc);
+	wrk->stats->n_objectcore--;
+}
+
+/*====================================================================
  * ObjIterate()
  *
  */
@@ -322,14 +370,19 @@ ObjSlim(struct worker *wrk, struct objcore *oc)
  */
 
 void
-ObjStable(struct worker *wrk, struct objcore *oc, struct boc *boc)
+ObjBocDone(struct worker *wrk, struct objcore *oc, struct boc **boc)
 {
-	const struct obj_methods *m = obj_getmethods(oc);
+	const struct obj_methods *m;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	CHECK_OBJ_NOTNULL(boc, BOC_MAGIC);
-	if (m->objstable != NULL)
-		m->objstable(wrk, oc, boc);
+	AN(boc);
+	CHECK_OBJ_NOTNULL(*boc, BOC_MAGIC);
+	if (oc->stobj->stevedore != NULL) {
+		m = obj_getmethods(oc);
+		if (m->objbocdone != NULL)
+			m->objbocdone(wrk, oc, *boc);
+	}
+	obj_deleteboc(boc);
 }
 
 /*====================================================================
diff --git a/bin/varnishd/cache/cache_obj.h b/bin/varnishd/cache/cache_obj.h
index 45f7e61..3d8e245 100644
--- a/bin/varnishd/cache/cache_obj.h
+++ b/bin/varnishd/cache/cache_obj.h
@@ -41,7 +41,7 @@ typedef int objgetspace_f(struct worker *, struct objcore *,
      ssize_t *sz, uint8_t **ptr);
 typedef void objextend_f(struct worker *, struct objcore *, ssize_t l);
 typedef void objtrimstore_f(struct worker *, struct objcore *);
-typedef void objstable_f(struct worker *, struct objcore *, struct boc *);
+typedef void objbocdone_f(struct worker *, struct objcore *, struct boc *);
 typedef void objslim_f(struct worker *, struct objcore *);
 typedef const void *objgetattr_f(struct worker *, struct objcore *,
     enum obj_attr attr, ssize_t *len);
@@ -55,7 +55,7 @@ struct obj_methods {
 	objgetspace_f	*objgetspace;
 	objextend_f	*objextend;
 	objtrimstore_f	*objtrimstore;
-	objstable_f	*objstable;
+	objbocdone_f	*objbocdone;
 	objslim_f	*objslim;
 	objgetattr_f	*objgetattr;
 	objsetattr_f	*objsetattr;
diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 8b862db..d9c858f 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -451,13 +451,13 @@ sml_trimstore(struct worker *wrk, struct objcore *oc)
 	VTAILQ_REMOVE(&o->list, st, list);
 	VTAILQ_INSERT_TAIL(&o->list, st1, list);
 	Lck_Unlock(&oc->boc->mtx);
-	/* sml_stable frees this */
+	/* sml_bocdone frees this */
 	AZ(oc->boc->stevedore_priv);
 	oc->boc->stevedore_priv = st;
 }
 
-static void __match_proto__(objstable_f)
-sml_stable(struct worker *wrk, struct objcore *oc, struct boc *boc)
+static void __match_proto__(objbocdone_f)
+sml_bocdone(struct worker *wrk, struct objcore *oc, struct boc *boc)
 {
 	const struct stevedore *stv;
 	struct storage *st;
@@ -610,7 +610,7 @@ const struct obj_methods SML_methods = {
 	.objgetspace	= sml_getspace,
 	.objextend	= sml_extend,
 	.objtrimstore	= sml_trimstore,
-	.objstable	= sml_stable,
+	.objbocdone	= sml_bocdone,
 	.objslim	= sml_slim,
 	.objgetattr	= sml_getattr,
 	.objsetattr	= sml_setattr,



From phk at FreeBSD.org  Fri Feb 26 22:53:40 2016
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Fri, 26 Feb 2016 23:53:40 +0100
Subject: [master] d232d7f Reduce cost of HSH_Insert by one oh->mtx lock/unlock
 by grabbing the refcount adjecent to the call.
Message-ID: <E1aZRGW-0002F1-SC@project.varnish-software.com>


commit d232d7f6635da66b5e7d24edd5501023e6506ff4
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Fri Feb 26 22:53:08 2016 +0000

    Reduce cost of HSH_Insert by one oh->mtx lock/unlock by grabbing
    the refcount adjecent to the call.

diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 9bdd92e..a6c2d3f 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -124,12 +124,11 @@ EXP_Insert(struct worker *wrk, struct objcore *oc)
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
+	assert(oc->refcnt >= 2);
 
 	AZ(oc->exp_flags & (OC_EF_INSERT | OC_EF_MOVE));
 	AZ(oc->flags & OC_F_DYING);
 
-	HSH_Ref(oc);
-
 	ObjSendEvent(wrk, oc, OEV_INSERT);
 	exp_mail_it(oc, OC_EF_INSERT | OC_EF_EXP | OC_EF_MOVE);
 }
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index f952769..3a48d86 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -263,7 +263,9 @@ HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc,
 	AN(digest);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
 	AN(ban);
+	AN(oc->flags & OC_F_BUSY);
 	AZ(oc->flags & OC_F_PRIVATE);
+	assert(oc->refcnt == 1);
 
 	hsh_prealloc(wrk);
 
@@ -275,9 +277,9 @@ HSH_Insert(struct worker *wrk, const void *digest, struct objcore *oc,
 
 	/* Mark object busy and insert (precreated) objcore in
 	   objecthead. The new object inherits our objhead reference. */
-	oc->flags |= OC_F_BUSY;
 	oc->objhead = oh;
 	VTAILQ_INSERT_TAIL(&oh->objcs, oc, hsh_list);
+	oc->refcnt++;				// For EXP_Insert
 	Lck_Unlock(&oh->mtx);
 
 	BAN_RefBan(oc, ban);
@@ -662,16 +664,19 @@ HSH_Unbusy(struct worker *wrk, struct objcore *oc)
 	AN(oc->stobj->stevedore);
 	AN(oc->flags & OC_F_BUSY);
 	assert(oh->refcnt > 0);
+	assert(oc->refcnt > 0);
 
 	if (!(oc->flags & OC_F_PRIVATE)) {
 		BAN_NewObjCore(oc);
 		AN(oc->ban);
-		EXP_Insert(wrk, oc);
 	}
 
 	/* XXX: pretouch neighbors on oh->objcs to prevent page-on under mtx */
 	Lck_Lock(&oh->mtx);
 	assert(oh->refcnt > 0);
+	assert(oc->refcnt > 0);
+	if (!(oc->flags & OC_F_PRIVATE)) 
+		oc->refcnt++;			// For EXP_Insert
 	/* XXX: strictly speaking, we should sort in Date: order. */
 	VTAILQ_REMOVE(&oh->objcs, oc, hsh_list);
 	VTAILQ_INSERT_HEAD(&oh->objcs, oc, hsh_list);
@@ -679,6 +684,8 @@ HSH_Unbusy(struct worker *wrk, struct objcore *oc)
 	if (!VTAILQ_EMPTY(&oh->waitinglist))
 		hsh_rush(wrk, oh);
 	Lck_Unlock(&oh->mtx);
+	if (!(oc->flags & OC_F_PRIVATE)) 
+		EXP_Insert(wrk, oc);
 }
 
 /*====================================================================



From dridi.boukelmoune at gmail.com  Mon Feb 29 22:05:45 2016
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 29 Feb 2016 23:05:45 +0100
Subject: [master] f928e49 Fix the VSL Proxy tag documentation
Message-ID: <E1aaVwn-0006Fh-JI@project.varnish-software.com>


commit f928e497283132d6102fd09dbe34782709e31577
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Feb 29 22:57:21 2016 +0100

    Fix the VSL Proxy tag documentation
    
    Both contents and formatting were off.

diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h
index 56a9604..e1c946d 100644
--- a/include/tbl/vsl_tags.h
+++ b/include/tbl/vsl_tags.h
@@ -142,14 +142,14 @@ SLTM(HttpGarbage, SLT_F_BINARY, "Unparseable HTTP request",
 SLTM(Proxy, 0, "PROXY protocol information",
 	"PROXY protocol information.\n\n"
 	"The format is::\n\n"
-	"\t%d %s %d %s %d [key value]...\n"
-	"\t|  |  |  |  |  |\n"
-	"\t|  |  |  |  |  +- optional information\n"
+	"\t%d %s %d %s %d\n"
+	"\t|  |  |  |  |\n"
 	"\t|  |  |  |  +- server port\n"
-	"\t|  |  |  +- server ip\n"
-	"\t|  |  +- client port\n"
-	"\t|  +- client ip\n"
-	"\t+---- PROXY protocol version\n"
+	"\t|  |  |  +---- server ip\n"
+	"\t|  |  +------- client port\n"
+	"\t|  +---------- client ip\n"
+	"\t+------------- PROXY protocol version\n"
+	"\n"
 )
 
 SLTM(ProxyGarbage, 0, "Unparseable PROXY request",