From dridi at varni.sh  Fri Jun  5 08:16:49 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 5 Jun 2015 10:16:49 +0200
Subject: [PATCH] Run background processes in varnishtest
Message-ID: <CABoVN9C3RWsLUTB7L=XkZXOx9wU5vPp=zmUNjN5hVNfycBEv7w@mail.gmail.com>

Hi,

I would like to add non-blocking process execution in varnishtest,
something different than the existing shell and err_shell commands.

I have found several use cases:
- testing varnish tools (there's a varnishncsa test)
- running unrelated programs (eg. redis for a redis vmod)
- attach strace to a pid (varnish or other process)

The second patch introduces a new macro for "-n" options, eg. ${v1_name}.

This was inspired by #1742.

Of course it's useless for all interactive executions, but you can
currently start a process, send signals, write to its stdin, and
stdout/err redirection to files... That's probably enough for many
use cases.

I haven't done much more (like -expect on the status code) because
it can easily be added afterwards if needed.

Best Regards,
Dridi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Run-background-processes-in-varnishtest.patch
Type: text/x-patch
Size: 16597 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150605/99fb1036/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Add-a-varnish-name-macro-in-varnishtest.patch
Type: text/x-patch
Size: 1306 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150605/99fb1036/attachment-0001.bin>

From dridi at varni.sh  Fri Jun  5 10:29:32 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 5 Jun 2015 12:29:32 +0200
Subject: [PATCH] Run background processes in varnishtest
In-Reply-To: <CABoVN9C3RWsLUTB7L=XkZXOx9wU5vPp=zmUNjN5hVNfycBEv7w@mail.gmail.com>
References: <CABoVN9C3RWsLUTB7L=XkZXOx9wU5vPp=zmUNjN5hVNfycBEv7w@mail.gmail.com>
Message-ID: <CABoVN9BQJatpFiHOiFAuQSQJktcRFtwRqMn080yUtpqaiGEgtg@mail.gmail.com>

As discussed, an update of the signal handling.

Cheers,
Dridi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Rewrite-signal-handling-in-a-varnishtest-bg-process.patch
Type: text/x-patch
Size: 5430 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150605/3de159b2/attachment.bin>

From dridi at varni.sh  Fri Jun  5 10:58:16 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 5 Jun 2015 12:58:16 +0200
Subject: [PATCH] Run background processes in varnishtest
In-Reply-To: <CABoVN9BQJatpFiHOiFAuQSQJktcRFtwRqMn080yUtpqaiGEgtg@mail.gmail.com>
References: <CABoVN9C3RWsLUTB7L=XkZXOx9wU5vPp=zmUNjN5hVNfycBEv7w@mail.gmail.com>
 <CABoVN9BQJatpFiHOiFAuQSQJktcRFtwRqMn080yUtpqaiGEgtg@mail.gmail.com>
Message-ID: <CABoVN9B7J_0ahxg+ESEQ+gzfDYVCwucRdaET-=JrSfXpsT6Yxw@mail.gmail.com>

And this time an all-in-one patch with all the changes, and the
removal of a GNU extension I let slip in in the previous patch.

Cheers,
Dridi

On Fri, Jun 5, 2015 at 12:29 PM, Dridi Boukelmoune <dridi at varni.sh> wrote:
> As discussed, an update of the signal handling.
>
> Cheers,
> Dridi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Run-background-processes-in-varnishtest.patch
Type: text/x-patch
Size: 14429 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150605/5f8310f2/attachment.bin>

From jyw at epochtimes.com  Sat Jun  6 17:59:46 2015
From: jyw at epochtimes.com (Jingyi Wei)
Date: Sat, 06 Jun 2015 13:59:46 -0400
Subject: return(refresh) from vcl_hit / Re: [PATCH] Add
 "hash_always_expire" option to treat any existing objects as expired
In-Reply-To: <5559CDEC.6020102@schokola.de>
References: <5500D746.4070707@epochtimes.com> <5559CDEC.6020102@schokola.de>
Message-ID: <55733512.8050304@epochtimes.com>

Hello Nils,

Thanks for your email. I think the difficulty of the return(refresh)
approach might be: VRY_Finish(req, DISCARD) will run instead of
VRY_Finish(req, KEEP) in cnt_lookup(), and do not fetch again if there
is already a busy object after VCL_hit_method().

Thanks,
Jingyi

On 05/18/2015 07:33 AM, Nils Goroll wrote:
> Hi Jingyi,
>
> this has been discussed on irc today:
>
> You have come up with a useful idea to allow refresh of a cache object despite a
> cache hit, which we want to implement. We'd prefer a different implementation
> though and will probably look in the direction of a return(refresh) from vcl_hit.
>
> Cheers, Nils
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150606/ae932b6d/attachment.html>

From lkarsten at varnish-software.com  Mon Jun  8 14:39:29 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Mon, 8 Jun 2015 16:39:29 +0200
Subject: Planning VDD15Q3
Message-ID: <20150608143928.GB4317@immer.varnish-software.com>

Hello.

It is time for planning a the next developer meeting.

A lack of scheduling ate up our semi-planned meetup before the summer,
so I'm trying to be early for a post-summer meeting.

I plan to have it in Oslo, Norway in September 2015.

Can anyone that wants to join please fill out (non-binding) the
following doodle, so we can find a timeslot for it?


    http://doodle.com/uxr5zxq6necwqvnb


Varnish Software is open to covering travel expenses for non-commercial
participants. Some previous project involvement is required. Contact me
directly for details.

-- 
Lasse Karstensen
Varnish Software AS



From arianna.aondio at varnish-software.com  Tue Jun  9 16:08:25 2015
From: arianna.aondio at varnish-software.com (Arianna Aondio)
Date: Tue, 9 Jun 2015 18:08:25 +0200
Subject: Request/response body access functions.
Message-ID: <CAL6a+5j=8EMUY-ULPkVx6OwPBd=xto4xGufPDzwzp09hbgmfoA@mail.gmail.com>

Hi everyone,
this email is about body transformations.
Both Apache and Nginx allow transformations, such as regular
expression with substitution, on request and response bodies, in my
personal opinion implementing such a thing in varnish would add value
to our software.

What do we have so far?
request body transformations have already been discussed at Hamburg
VDD and later on, but patches never had the chance to become part of
the varnish core.

Notes form Hamburg VDD:
------------------------------------------------------------------------------------------------------------------------------
* Request body functions, review of suggested design. (arianna)
  phk brought up the issue of response body access. this ties it to
body access in VCL.

// req.body in vcl is immutable
// you need a VMOD VFP to modify the body going to the backend (see TODO above)
in vcl_recv{}
  req.cache_body(100k); // fails request if longer req.body
  req.bodylen = N;      // actual length
  hash_data(req.body);  // takes BLOB, requires STRING -> BLOB silent conversion
                        // can be microoptimized by VCC to walk storage chunks
  regsub(std.tostring(req.body), "..", "..");

IDEA: make BLOBS iterators?  // phk needs to ponder
-----------------------------------------------------------------------------------------------------------------------------

Since no more input has been given regarding this topic, I have
implemented a vmod that allows the user to do different operations
using the request body.
The vmod build against Varnish 4, but it can be improved when Varnish
4.1 will be released; at the moment the code is a little bit redundant
due to the presence, in branch 4.0, of VRB_Cache() not able to handle
properly POST body larger than the provided size limitation( this has
been fixed in master branch).
Please feel free to use and test the
vmod(https://github.com/aondio/libvmod-bodyaccess/).

At this point it comes natural to start thinking about response bodies:
the idea is to modify or do any kind of operations on the response
body before the object is in the cache, this means if we want to be
able to make transformations on the response body those have to be
done at fetch time.
When VBF_Fetch is triggered a new object is fetched from backend, at
that point a "chain" is initialized and every link of this chain
contributes to give shape to the final response, for example:

VBF_Fetch ---> GZIP---> suck bytes

In this simple situation when a new object is fetched the GZIP "link"
will actually gziped the "sucked" bytes and then deliver them.
The same logic applies for UNGZIP and ESI( |GZIP|UNGZIP).
I'd like to have a vmod that can insert itself in the fetch chain,
modify the response body as required and return without making varnish
crash or start behaving strangely.
In order to make it possible some changes to the varnish core need to
be done, but I don't have any patches for those changes because I'd
first like to hear what you think about this idea.

Comments are much appreciated.
-- 
Arianna Aondio
Software Developer | Varnish Software AS
Mobile: +47 980 62 619

We Make Websites Fly
www.varnish-software.com



From dridi at varni.sh  Wed Jun 10 08:37:10 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 10 Jun 2015 10:37:10 +0200
Subject: [PATCH] Run background processes in varnishtest
In-Reply-To: <CABoVN9B7J_0ahxg+ESEQ+gzfDYVCwucRdaET-=JrSfXpsT6Yxw@mail.gmail.com>
References: <CABoVN9C3RWsLUTB7L=XkZXOx9wU5vPp=zmUNjN5hVNfycBEv7w@mail.gmail.com>
 <CABoVN9BQJatpFiHOiFAuQSQJktcRFtwRqMn080yUtpqaiGEgtg@mail.gmail.com>
 <CABoVN9B7J_0ahxg+ESEQ+gzfDYVCwucRdaET-=JrSfXpsT6Yxw@mail.gmail.com>
Message-ID: <CABoVN9BaLT0YRW_EtAdJ6hDj2Uu5bMVNJ-Ns7Nf-AE9BuWK9-A@mail.gmail.com>

New patch rebased against current master, trying to solve portability
issues with kill(1)

On Fri, Jun 5, 2015 at 12:58 PM, Dridi Boukelmoune <dridi at varni.sh> wrote:
> And this time an all-in-one patch with all the changes, and the
> removal of a GNU extension I let slip in in the previous patch.
>
> Cheers,
> Dridi
>
> On Fri, Jun 5, 2015 at 12:29 PM, Dridi Boukelmoune <dridi at varni.sh> wrote:
>> As discussed, an update of the signal handling.
>>
>> Cheers,
>> Dridi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Run-background-processes-in-varnishtest.patch
Type: text/x-patch
Size: 14422 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150610/a65b59a2/attachment.bin>

From martin at varnish-software.com  Wed Jun 10 09:05:58 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Wed, 10 Jun 2015 11:05:58 +0200
Subject: [PATCH 1/5] Keep track of VCL initialization
Message-ID: <1433927162-17564-1-git-send-email-martin@varnish-software.com>

Keep track of which VCL initialization steps has succeeded, and run
the fini steps in reverse only for those that were completed.
---
 lib/libvcc/vcc_compile.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c
index 06f3a3e..ac02736 100644
--- a/lib/libvcc/vcc_compile.c
+++ b/lib/libvcc/vcc_compile.c
@@ -309,14 +309,19 @@ EmitInitFini(const struct vcc *tl)
 {
 	struct inifin *p;
 
+	Fh(tl, 0, "\nstatic unsigned inistep;\n");
+
 	/*
 	 * INIT
 	 */
 	Fc(tl, 0, "\nstatic int\nVGC_Load(VRT_CTX)\n{\n\n");
+	Fc(tl, 0, "\tinistep = 0;\n\n");
 	VTAILQ_FOREACH(p, &tl->inifin, list) {
 		AZ(VSB_finish(p->ini));
+		assert(p->n > 0);
 		if (VSB_len(p->ini))
 			Fc(tl, 0, "\t/* %u */\n%s\n", p->n, VSB_data(p->ini));
+		Fc(tl, 0, "\tinistep = %u;\n\n", p->n);
 		VSB_delete(p->ini);
 	}
 
@@ -330,8 +335,12 @@ EmitInitFini(const struct vcc *tl)
 
 	VTAILQ_FOREACH_REVERSE(p, &tl->inifin, inifinhead, list) {
 		AZ(VSB_finish(p->fin));
-		if (VSB_len(p->fin))
-			Fc(tl, 0, "\t/* %u */\n%s\n", p->n, VSB_data(p->fin));
+		if (VSB_len(p->fin)) {
+			Fc(tl, 0, "\t/* %u */\n", p->n);
+			Fc(tl, 0, "\tif (inistep >= %u) {\n", p->n);
+			Fc(tl, 0, "%s\n", VSB_data(p->fin));
+			Fc(tl, 0, "\t}\n\n");
+		}
 		VSB_delete(p->fin);
 	}
 
-- 
2.1.4




From martin at varnish-software.com  Wed Jun 10 09:05:59 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Wed, 10 Jun 2015 11:05:59 +0200
Subject: [PATCH 2/5] Honor return value from vmod init functions
In-Reply-To: <1433927162-17564-1-git-send-email-martin@varnish-software.com>
References: <1433927162-17564-1-git-send-email-martin@varnish-software.com>
Message-ID: <1433927162-17564-2-git-send-email-martin@varnish-software.com>

Make use of the integer return value from vmod init functions, and
abort the vcl init on non-zero return.
---
 lib/libvcc/vcc_vmod.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c
index 8391c98..be36a44 100644
--- a/lib/libvcc/vcc_vmod.c
+++ b/lib/libvcc/vcc_vmod.c
@@ -199,8 +199,9 @@ vcc_ParseImport(struct vcc *tl)
 			if (ifp == NULL)
 				ifp = New_IniFin(tl);
 			VSB_printf(ifp->ini,
-			    "\t%s(&vmod_priv_%.*s, &VCL_conf);",
+			    "\tif (%s(&vmod_priv_%.*s, &VCL_conf))\n",
 			    p, PF(mod));
+			VSB_printf(ifp->ini, "\t\treturn (1);");
 		} else {
 			sym = VCC_AddSymbolStr(tl, p, SYM_FUNC);
 			ERRCHK(tl);
-- 
2.1.4




From martin at varnish-software.com  Wed Jun 10 09:06:01 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Wed, 10 Jun 2015 11:06:01 +0200
Subject: [PATCH 4/5] Add a VRT_cliout function
In-Reply-To: <1433927162-17564-1-git-send-email-martin@varnish-software.com>
References: <1433927162-17564-1-git-send-email-martin@varnish-software.com>
Message-ID: <1433927162-17564-4-git-send-email-martin@varnish-software.com>

The VRT_cliout function enables VRT users to output diagnostics
strings to the CLI interface.

This is mostly useful from vmod init functions, where the output would
become visible for the user during vcl.load.

If there is no CLI context available, the messages are sent to stdout.
---
 bin/varnishd/cache/cache_vrt.c | 14 ++++++++++++++
 include/vcli_priv.h            |  1 +
 include/vrt.h                  |  2 ++
 lib/libvarnish/cli_common.c    |  9 ++++++++-
 4 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 9b4aaff..e930f72 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -31,6 +31,8 @@
 
 #include "config.h"
 
+#include <stdarg.h>
+
 #include "cache.h"
 
 #include "cache_director.h"
@@ -41,6 +43,7 @@
 #include "vrt_obj.h"
 #include "vtcp.h"
 #include "vtim.h"
+#include "vcli_priv.h"
 
 const void * const vrt_magic_string_end = &vrt_magic_string_end;
 const void * const vrt_magic_string_unset = &vrt_magic_string_unset;
@@ -524,3 +527,14 @@ VRT_memmove(void *dst, const void *src, unsigned len)
 
 	(void)memmove(dst, src, len);
 }
+
+void
+VRT_cliout(VRT_CTX, const char *fmt, ...)
+{
+	va_list ap;
+
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+	va_start(ap, fmt);
+	VCLI_Out_VA(ctx->cli, fmt, ap);
+	va_end(ap);
+}
diff --git a/include/vcli_priv.h b/include/vcli_priv.h
index 2f98d7d..c71f66f 100644
--- a/include/vcli_priv.h
+++ b/include/vcli_priv.h
@@ -55,5 +55,6 @@ struct cli_proto {
 int VCLI_Overflow(struct cli *cli);
 void VCLI_Out(struct cli *cli, const char *fmt, ...)
     __v_printflike(2, 3);
+void VCLI_Out_VA(struct cli *cli, const char *fmt, va_list ap);
 void VCLI_Quote(struct cli *cli, const char *str);
 void VCLI_SetResult(struct cli *cli, unsigned r);
diff --git a/include/vrt.h b/include/vrt.h
index 499eaac..7e04062 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -239,6 +239,8 @@ void VRT_memmove(void *dst, const void *src, unsigned len);
 
 void VRT_Rollback(VRT_CTX, const struct http *);
 
+void VRT_cliout(VRT_CTX, const char *fmt, ...);
+
 /* Synthetic pages */
 void VRT_synth_page(VRT_CTX, const char *, ...);
 
diff --git a/lib/libvarnish/cli_common.c b/lib/libvarnish/cli_common.c
index 44a6869..7ccd6a0 100644
--- a/lib/libvarnish/cli_common.c
+++ b/lib/libvarnish/cli_common.c
@@ -59,6 +59,14 @@ VCLI_Out(struct cli *cli, const char *fmt, ...)
 	va_list ap;
 
 	va_start(ap, fmt);
+	VCLI_Out_VA(cli, fmt, ap);
+	va_end(ap);
+}
+
+void
+VCLI_Out_VA(struct cli *cli, const char *fmt, va_list ap)
+{
+
 	if (cli != NULL) {
 		CHECK_OBJ_NOTNULL(cli, CLI_MAGIC);
 		if (VSB_len(cli->sb) < *cli->limit)
@@ -68,7 +76,6 @@ VCLI_Out(struct cli *cli, const char *fmt, ...)
 	} else {
 		(void)vfprintf(stdout, fmt, ap);
 	}
-	va_end(ap);
 }
 
 /*lint -e{818} cli could be const */
-- 
2.1.4




From martin at varnish-software.com  Wed Jun 10 09:06:00 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Wed, 10 Jun 2015 11:06:00 +0200
Subject: [PATCH 3/5] Pass the VRT_CTX down to the vmod init function
In-Reply-To: <1433927162-17564-1-git-send-email-martin@varnish-software.com>
References: <1433927162-17564-1-git-send-email-martin@varnish-software.com>
Message-ID: <1433927162-17564-3-git-send-email-martin@varnish-software.com>

This enables the vmod's to access the cli pointer and report errors
back to the user during vcl.load().

Also drop the VCL_conf pointer, as that is already included in the ctx.

This changes the VMOD ABI without backwards compatibility, we should
bump the major before release.
---
 include/vrt.h                  | 3 ++-
 lib/libvcc/vcc_vmod.c          | 2 +-
 lib/libvcc/vmodtool.py         | 3 +--
 lib/libvmod_debug/vmod_debug.c | 4 ++--
 4 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/include/vrt.h b/include/vrt.h
index dca5902..499eaac 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -40,6 +40,7 @@
  *
  * changes to consider with next VRT_MAJOR_VERSION bump:
  * - cache_vrt.c: -> ssize_t VRT_CacheReqBody(VRT_CTX, size_t)
+ * - cache_vrt.h: vmod_init_f add VRT_CTX param
  */
 
 #define VRT_MAJOR_VERSION	2U
@@ -265,7 +266,7 @@ struct vmod_priv {
 	vmod_priv_free_f	*free;
 };
 
-typedef int vmod_init_f(struct vmod_priv *,  const struct VCL_conf *);
+typedef int vmod_init_f(VRT_CTX, struct vmod_priv *);
 
 void VRT_priv_fini(const struct vmod_priv *p);
 struct vmod_priv *VRT_priv_task(VRT_CTX, void *vmod_id);
diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c
index be36a44..2204f53 100644
--- a/lib/libvcc/vcc_vmod.c
+++ b/lib/libvcc/vcc_vmod.c
@@ -199,7 +199,7 @@ vcc_ParseImport(struct vcc *tl)
 			if (ifp == NULL)
 				ifp = New_IniFin(tl);
 			VSB_printf(ifp->ini,
-			    "\tif (%s(&vmod_priv_%.*s, &VCL_conf))\n",
+			    "\tif (%s(ctx, &vmod_priv_%.*s))\n",
 			    p, PF(mod));
 			VSB_printf(ifp->ini, "\t\treturn (1);");
 		} else {
diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index bc4c3c4..5380e01 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -188,8 +188,7 @@ class Vmod(object):
 		if self.init != None:
 			fo.write("\n")
 			fo.write("int " + self.init)
-			fo.write(
-			    "(struct vmod_priv *, const struct VCL_conf *);\n")
+			fo.write("(VRT_CTX, struct vmod_priv *);\n")
 
 	def c_typedefs_(self):
 		l = list()
diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c
index ccdcab3..38a8f68 100644
--- a/lib/libvmod_debug/vmod_debug.c
+++ b/lib/libvmod_debug/vmod_debug.c
@@ -237,11 +237,11 @@ priv_vcl_free(void *priv)
 }
 
 int __match_proto__(vmod_init_f)
-init_function(struct vmod_priv *priv, const struct VCL_conf *cfg)
+init_function(VRT_CTX, struct vmod_priv *priv)
 {
 	struct priv_vcl *priv_vcl;
 
-	(void)cfg;
+	AN(ctx);
 
 	ALLOC_OBJ(priv_vcl, PRIV_VCL_MAGIC);
 	AN(priv_vcl);
-- 
2.1.4




From martin at varnish-software.com  Wed Jun 10 09:06:02 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Wed, 10 Jun 2015 11:06:02 +0200
Subject: [PATCH 5/5] Test VCL and vmod init failures
In-Reply-To: <1433927162-17564-1-git-send-email-martin@varnish-software.com>
References: <1433927162-17564-1-git-send-email-martin@varnish-software.com>
Message-ID: <1433927162-17564-5-git-send-email-martin@varnish-software.com>

---
 bin/varnishtest/tests/m00022.vtc | 55 ++++++++++++++++++++++++++++++++++++++++
 lib/libvmod_debug/vmod.vcc       |  4 +++
 lib/libvmod_debug/vmod_debug.c   | 16 ++++++++++++
 3 files changed, 75 insertions(+)
 create mode 100644 bin/varnishtest/tests/m00022.vtc

diff --git a/bin/varnishtest/tests/m00022.vtc b/bin/varnishtest/tests/m00022.vtc
new file mode 100644
index 0000000..0fd318d
--- /dev/null
+++ b/bin/varnishtest/tests/m00022.vtc
@@ -0,0 +1,55 @@
+varnishtest "VCL initialization failures from VCL/vmod"
+
+server s1 {
+} -start
+
+# Doing debug.failinit(1) will cause the debug vmod to return failure
+# from it's init routine in subsequent initializations. Start varnish
+# with a VCL that lets us set failinit later.
+varnish v1 -vcl {
+	backend default {
+		.host = "${s1_addr}";
+		.port = "${s1_port}";
+	}
+
+	import ${vmod_debug};
+
+	sub vcl_recv {
+		if (req.url == "/failinit") {
+			debug.set_failinit(1);
+			return (synth (200, "Failing"));
+		} else {
+			return (synth (404));
+		}
+	}
+} -start
+
+# Check that we can fail the VCL load in vcl_init
+varnish v1 -errvcl "vcl_init{} failed" {
+	backend default {
+		.host = "${s1_addr}";
+		.port = "${s1_port}";
+	}
+
+	sub vcl_init {
+		return (fail);
+	}
+}
+
+# Make vmod_debug fail on next vmod init
+client c1 {
+	txreq -url /failinit
+	rxresp
+	expect resp.status == 200
+	expect resp.msg == "Failing"
+} -run
+
+# Check that init failures from vmods are caught
+varnish v1 -errvcl {failinit=1} {
+	backend default {
+		.host = "${s1_addr}";
+		.port = "${s1_port}";
+	}
+
+	import ${vmod_debug};
+}
diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index 434ee78..cbd89a8 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -39,6 +39,10 @@ $Function VOID panic(STRING_LIST)
 
 Don't.
 
+$Function VOID set_failinit(INT)
+
+If set non-zero, the vmod initialization function will return failure.
+
 $Function STRING author(ENUM { phk, des, kristian, mithrandir } person="phk")
 
 Test function for ENUM arguments
diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c
index 38a8f68..519f0cd 100644
--- a/lib/libvmod_debug/vmod_debug.c
+++ b/lib/libvmod_debug/vmod_debug.c
@@ -43,6 +43,8 @@ struct priv_vcl {
 	uintptr_t		exp_cb;
 };
 
+static int failinit = 0;
+
 VCL_VOID __match_proto__(td_debug_panic)
 vmod_panic(VRT_CTX, const char *str, ...)
 {
@@ -56,6 +58,14 @@ vmod_panic(VRT_CTX, const char *str, ...)
 	VAS_Fail("VCL", "", 0, b, VAS_VCL);
 }
 
+VCL_VOID __match_proto__(td_debug_set_initfail)
+vmod_set_failinit(VRT_CTX, VCL_INT val)
+{
+
+	(void)ctx;
+	failinit = val;
+}
+
 VCL_STRING __match_proto__(td_debug_author)
 vmod_author(VRT_CTX, VCL_ENUM id)
 {
@@ -249,5 +259,11 @@ init_function(VRT_CTX, struct vmod_priv *priv)
 	AN(priv_vcl->foo);
 	priv->priv = priv_vcl;
 	priv->free = priv_vcl_free;
+
+	if (failinit) {
+		VRT_cliout(ctx, "vmod_debug:init:failinit=%u:", failinit);
+		return (1);
+	}
+
 	return (0);
 }
-- 
2.1.4




From martin at varnish-software.com  Wed Jun 10 14:36:38 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Wed, 10 Jun 2015 16:36:38 +0200
Subject: [PATCH] Allow vmod objects to trigger initialization failures
Message-ID: <1433946998-23505-1-git-send-email-martin@varnish-software.com>

This allows vmod objects to signal failures if the vmod fails to set
it's object pointer. The VCL will then return early and a VCL
initialization failure is reported.

Also change the deinitialization to only call the fini if the object
pointer is non-NULL.
---
 bin/varnishtest/tests/m00022.vtc   | 30 ++++++++++++++++++++++++++++++
 lib/libvcc/vcc_action.c            | 11 ++++++++++-
 lib/libvmod_debug/vmod.vcc         |  4 ++++
 lib/libvmod_debug/vmod_debug_obj.c | 15 +++++++++++++++
 4 files changed, 59 insertions(+), 1 deletion(-)

diff --git a/bin/varnishtest/tests/m00022.vtc b/bin/varnishtest/tests/m00022.vtc
index c070f19..a173c08 100644
--- a/bin/varnishtest/tests/m00022.vtc
+++ b/bin/varnishtest/tests/m00022.vtc
@@ -14,9 +14,18 @@ varnish v1 -vcl {
 
 	import ${vmod_debug};
 
+	sub vcl_init {
+		new obj = debug.obj("asdf");
+	}
+
 	sub vcl_recv {
 		if (req.url == "/failinit") {
 			debug.set_failinit(1);
+			obj.set_failobj(0);
+			return (synth (200));
+		} else if (req.url == "/failobj") {
+			debug.set_failinit(0);
+			obj.set_failobj(1);
 			return (synth (200));
 		} else {
 			return (synth (404));
@@ -52,3 +61,24 @@ varnish v1 -errvcl {failinit=1} {
 
 	import ${vmod_debug};
 }
+
+# Change to make object init fail
+client c1 {
+	txreq -url /failobj
+	rxresp
+	expect resp.status == 200
+} -run
+
+# Check that object init fails are caught
+varnish v1 -errvcl {failobj=1} {
+	backend default {
+		.host = "${s1_addr}";
+		.port = "${s1_port}";
+	}
+
+	import ${vmod_debug};
+
+	sub vcl_init {
+		new obj = debug.obj("asdf");
+	}
+}
diff --git a/lib/libvcc/vcc_action.c b/lib/libvcc/vcc_action.c
index c4f984d..b70c5ad 100644
--- a/lib/libvcc/vcc_action.c
+++ b/lib/libvcc/vcc_action.c
@@ -220,8 +220,17 @@ parse_new(struct vcc *tl)
 
 	bprintf(buf1, ", &vo_%s, \"%s\"", sy1->name, sy1->name);
 	vcc_Eval_Func(tl, s_init, buf1, sy2->name, s_init + strlen(s_init) + 1);
+
+	Fb(tl, 1, "if (!vo_%s) {\n", sy1->name);
+	tl->indent += INDENT;
+	Fb(tl, 1, "VRT_handling(ctx, VCL_RET_FAIL);\n");
+	Fb(tl, 1, "return (1);\n");
+	tl->indent -= INDENT;
+	Fb(tl, 1, "}\n");
+
 	ifp = New_IniFin(tl);
-	VSB_printf(ifp->fin, "\t%s(&vo_%s);", s_fini, sy1->name);
+	VSB_printf(ifp->fin, "\tif (vo_%s)\n", sy1->name);
+	VSB_printf(ifp->fin, "\t\t%s(&vo_%s);", s_fini, sy1->name);
 	ExpectErr(tl, ';');
 
 	bprintf(buf1, ", vo_%s", sy1->name);
diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index cbd89a8..8584878 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -93,6 +93,10 @@ $Method TIME .date()
 
 You never know when you need a date.
 
+$Method VOID .set_failobj(INT)
+
+If set non-zero, the obj vmod object will fail during initialization.
+
 $Function VOID rot52(HTTP hdr)
 
 Encrypt the HTTP header with quad-ROT13 encryption,
diff --git a/lib/libvmod_debug/vmod_debug_obj.c b/lib/libvmod_debug/vmod_debug_obj.c
index 19a966c..7a20495 100644
--- a/lib/libvmod_debug/vmod_debug_obj.c
+++ b/lib/libvmod_debug/vmod_debug_obj.c
@@ -41,6 +41,8 @@ struct vmod_debug_obj {
 	int foobar;
 };
 
+static int failobj = 0;
+
 VCL_VOID
 vmod_obj__init(VRT_CTX, struct vmod_debug_obj **op,
     const char *vcl_name, VCL_STRING s)
@@ -48,6 +50,10 @@ vmod_obj__init(VRT_CTX, struct vmod_debug_obj **op,
 	struct vmod_debug_obj *o;
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+	if (failobj) {
+		VRT_cliout(ctx, "vmod_debug:obj_init:failobj=%u:", failobj);
+		return;
+	}
 	(void)vcl_name;
 	(void)s;
 	AN(op);
@@ -98,3 +104,12 @@ vmod_obj_date(VRT_CTX, struct vmod_debug_obj *o)
 	assert(o->foobar == 42);
 	return (21.4);
 }
+
+VCL_VOID __match_proto__(td_debug_obj_set_failobj)
+vmod_obj_set_failobj(VRT_CTX, struct vmod_debug_obj *o, VCL_INT val)
+{
+
+	(void)ctx;
+	(void)o;
+	failobj = val;
+}
-- 
2.1.4




From phk at phk.freebsd.dk  Thu Jun 11 20:48:53 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 11 Jun 2015 20:48:53 +0000
Subject: [PATCH 3/5] Pass the VRT_CTX down to the vmod init function
In-Reply-To: <1433927162-17564-3-git-send-email-martin@varnish-software.com>
References: <1433927162-17564-1-git-send-email-martin@varnish-software.com>
 <1433927162-17564-3-git-send-email-martin@varnish-software.com>
Message-ID: <2697.1434055733@critter.freebsd.dk>

--------
In message <1433927162-17564-3-git-send-email-martin at varnish-software.com>, Mar
tin Blix Grydeland writes:

After due consideration I have decided that I do not want to expose
the CLI to the vmods.  I belive that is also why we never really
did, despite the field technically being there.

Instead we should expose just a VSB and report its contents in the
CLI in case of error.

I'm a bit worried about quality of error messages here, so we may
want the VCC to emit code along the lines of:

	VSB_clear(ctx->vsb);
	VSB_cat(ctx->vsb, "Initializing vmod_debug:\n");
	if (VRT_Vmod_Init(&VGC_vmod_debug,
            &Vmod_debug_Func,
            sizeof(Vmod_debug_Func),
            "debug",
            "/critter/Varnish/trunk/varnish-cache/lib/libvmod_debug/.libs/libvmo
d_debug.so",
            "PQVQBUUFQXFQEDALV at NCKOZKMBDGXJRU",
            ctx))
                return(1);
	VSB_clear(ctx->vsb);

so that we are sure the message tells us which vmod emitted "File
Not Found" or such non-informative messages.

Related to this:

Should the vmod-fini function also be able to fail?

This would allow vmods to veto dlclose() for instance when threads
are not stoppable etc.

Of course this is sort of an invitation to not bother writing proper
cleanup mechanisms for fini() to begin with.

However, I very much suspect that temptation to be there no matter
what we do, so providing a way to clearly signal this fact and
thereby avoiding panics may be a good idea ?


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From phk at phk.freebsd.dk  Thu Jun 11 21:16:49 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 11 Jun 2015 21:16:49 +0000
Subject: Questions on backend allocations
In-Reply-To: <CABoVN9CdoeXaZz9RBZZk=We9UG9Z=uJ6dVje9vMcXgij3b0Lvg@mail.gmail.com>
References: <CABoVN9CsXFTJy2Q5F54MoWyNQ5qdgJKvu6R7t3wzemwTqryffA@mail.gmail.com>
 <CABoVN9BG5eOVUgAiDYTt9arFwn04pj8NBUG8E1T0X3m6efn1HQ@mail.gmail.com>
 <CABoVN9BU9OVznwq-+u-chfH7r8ywO3djqe4ZmL-M2JKtddUfHg@mail.gmail.com>
 <CABoVN9CdoeXaZz9RBZZk=We9UG9Z=uJ6dVje9vMcXgij3b0Lvg@mail.gmail.com>
Message-ID: <3296.1434057409@critter.freebsd.dk>

--------

I've been pop'ing stack and come back down to this old topic.

The current hurdle is the cold/warm VCL events and how they get
communicated to VMODS, directors and backends, and possibly other
thingmajicks as well.

It is very important to remember here, that backends is a subtype
of directors, and that what the VMOD would do is create dynamic
directors (which happen to be backends).

Broadly speaking one avenue is to register all directors on a
list of the VCL, and tell them warm/cold via this list.

Right now we do not register directors anywhere, they are free
standing objects and can come and go as they please.

Making it mandatory to register all directors, just so we can
send them events they probably don't care about seems rather
silly.

The alternative way is to make cold/warm events independent and
provide a "call this function with this argument" registration
mechanism for any interested temperature-sensitive vmod, object,
backend or doodad.

The disadvantage to this model is that there is no obvious
or natural order to call these call-back functions.

For instance we could waste time sending "COLD" to a bunch of
backends which did individual registrations, on to see the vmod
destroying them all when it gets the COLD event.

Calling in first-registered-first-called order would probably
avoid that scenario and give some kind of predictability.

This ordering insecurity will require some thinking in some VMODs
I can think of, but I can't think of any VMOD for which there
isn't a reasonable solution.

So I'm currently leaning towards this model.

Backends should obviously register to get the events (in
cache_backend_cfg), no matter if they are static or dynamic,
so VMODs won't need to propagate the events.

The third option is to make the vmod_priv the "base object"
in compiled VCL, and tell those instead.  This has a lot of
merit from many angles, but I think that would have to wait
for 5.0.



-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From fgsch at lodoss.net  Fri Jun 12 01:45:43 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Fri, 12 Jun 2015 02:45:43 +0100
Subject: Vmod constants
Message-ID: <CAJV_h0aYsidiaE006R2Z-3iEU0+8sgqwR2A3wGZ6Kq7HWvVscg@mail.gmail.com>

Hi,

I've been wanting something like this for a while so I decided to give it a
try last night after Dridi moaned about the use of somewhat magic numbers
in the utf8 vmod.
The result is that you can now define constants in .vcc like this:

$Const TYPE name value

e.g.:

$Const INT CASEFOLD 0x0400
$Const INT LUMP 0x1000
$Const INT STRIPMARK 0x2000

And access it in VCL via <vmod>.<name>.

e.g.:

set req.url = utf8.transform(req.url, utf8.CASEFOLD + utf8.LUMP +
utf8.STRIPMARK);

This doesn't generate any C code in the current incarnation, it merely
creates the symbol and replaces the value when referencing it so the
resulting C will have e.g. 0x0001 but I could change it if required.

Personally I think this is quite useful but I might be biased so before I
clean the patches and post them here is there any interest?

Comments welcome.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150612/0f23c515/attachment.html>

From slink at schokola.de  Fri Jun 12 06:21:21 2015
From: slink at schokola.de (Nils Goroll)
Date: Fri, 12 Jun 2015 08:21:21 +0200
Subject: [PATCH 3/5] Pass the VRT_CTX down to the vmod init function
In-Reply-To: <2697.1434055733@critter.freebsd.dk>
References: <1433927162-17564-1-git-send-email-martin@varnish-software.com>
 <1433927162-17564-3-git-send-email-martin@varnish-software.com>
 <2697.1434055733@critter.freebsd.dk>
Message-ID: <557A7A61.1050709@schokola.de>



On 11/06/15 22:48, Poul-Henning Kamp wrote:
> 
> However, I very much suspect that temptation to be there no matter
> what we do, so providing a way to clearly signal this fact and
> thereby avoiding panics may be a good idea ?

Sounds like a really good idea to me.



From carlos.abalde at gmail.com  Fri Jun 12 06:52:52 2015
From: carlos.abalde at gmail.com (Carlos Abalde)
Date: Fri, 12 Jun 2015 08:52:52 +0200
Subject: Vmod constants
In-Reply-To: <CAJV_h0aYsidiaE006R2Z-3iEU0+8sgqwR2A3wGZ6Kq7HWvVscg@mail.gmail.com>
References: <CAJV_h0aYsidiaE006R2Z-3iEU0+8sgqwR2A3wGZ6Kq7HWvVscg@mail.gmail.com>
Message-ID: <61DF58A6-6037-4821-8207-42E3FE1D8273@gmail.com>


> On 12 Jun 2015, at 03:45, Federico Schwindt <fgsch at lodoss.net> wrote:
> 
> ...
> 
> $Const TYPE name value
> 
> e.g.:
> 
> $Const INT CASEFOLD 0x0400
> $Const INT LUMP 0x1000
> $Const INT STRIPMARK 0x2000
> 
> ...
> 
> This doesn't generate any C code in the current incarnation, it merely creates the symbol and replaces the value when referencing it so the resulting C will have e.g. 0x0001 but I could change it if required.
> 
> Personally I think this is quite useful but I might be biased so before I clean the patches and post them here is there any interest?
> 
> Comments welcome.

Hi,

I agree this would be very useful. I suppose that at the moment these constants defined in the VCC file won't be available in the implementation of the VMOD. I think it would be great to generate some C code including these values and make them readable in the code of the VMOD. 

My two cents :)

Thanks,

--
Carlos Abalde




From dridi at varni.sh  Fri Jun 12 08:18:33 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 12 Jun 2015 10:18:33 +0200
Subject: Vmod constants
In-Reply-To: <CAJV_h0aYsidiaE006R2Z-3iEU0+8sgqwR2A3wGZ6Kq7HWvVscg@mail.gmail.com>
References: <CAJV_h0aYsidiaE006R2Z-3iEU0+8sgqwR2A3wGZ6Kq7HWvVscg@mail.gmail.com>
Message-ID: <CABoVN9BTKND5++sXLiYR-mSWb9whFRPNVbXv_y-kR81WmaD6XQ@mail.gmail.com>

On Fri, Jun 12, 2015 at 3:45 AM, Federico Schwindt <fgsch at lodoss.net> wrote:
> $Const TYPE name value
>
> e.g.:
>
> $Const INT CASEFOLD 0x0400
> $Const INT LUMP 0x1000
> $Const INT STRIPMARK 0x2000

Would you be able to use the underlying C constants instead of
hard-coded values? Do we want to reinvent cpp on top of vmodtool
for non-portable constants?

> Personally I think this is quite useful but I might be biased so before I
> clean the patches and post them here is there any interest?

Obviously there is me ;)



From phk at phk.freebsd.dk  Fri Jun 12 08:26:15 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 12 Jun 2015 08:26:15 +0000
Subject: Vmod constants
In-Reply-To: <CAJV_h0aYsidiaE006R2Z-3iEU0+8sgqwR2A3wGZ6Kq7HWvVscg@mail.gmail.com>
References: <CAJV_h0aYsidiaE006R2Z-3iEU0+8sgqwR2A3wGZ6Kq7HWvVscg@mail.gmail.com>
Message-ID: <5702.1434097575@critter.freebsd.dk>

--------
In message <CAJV_h0aYsidiaE006R2Z-3iEU0+8sgqwR2A3wGZ6Kq7HWvVscg at mail.gmail.com>
, Federico Schwindt writes:

>$Const INT CASEFOLD 0x0400
>$Const INT LUMP 0x1000
>$Const INT STRIPMARK 0x2000
>
>And access it in VCL via <vmod>.<name>.
>
>set req.url = utf8.transform(req.url, utf8.CASEFOLD + utf8.LUMP +
>utf8.STRIPMARK);

At the very least it would have to spit out some #defines in the
vcc_if.h file, so that you don't have to duplicate the magic
numbers in the C code ?

What types do you see us needing ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Fri Jun 12 09:28:07 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 12 Jun 2015 11:28:07 +0200
Subject: Questions on backend allocations
In-Reply-To: <3296.1434057409@critter.freebsd.dk>
References: <CABoVN9CsXFTJy2Q5F54MoWyNQ5qdgJKvu6R7t3wzemwTqryffA@mail.gmail.com>
 <CABoVN9BG5eOVUgAiDYTt9arFwn04pj8NBUG8E1T0X3m6efn1HQ@mail.gmail.com>
 <CABoVN9BU9OVznwq-+u-chfH7r8ywO3djqe4ZmL-M2JKtddUfHg@mail.gmail.com>
 <CABoVN9CdoeXaZz9RBZZk=We9UG9Z=uJ6dVje9vMcXgij3b0Lvg@mail.gmail.com>
 <3296.1434057409@critter.freebsd.dk>
Message-ID: <CABoVN9BF6HKmQbGPR5AV9Qk8Wu-Dvxf7os7eCdCPSCOQo8YmaA@mail.gmail.com>

Hi,

I will try to wrap up the model I was suggesting on IRC.

For starters, I refer to dynamic backends as *native* backends (ones
you can see in varnishstat) managed by a VMOD instead of VCL.

We agreed that even though a VMOD lifespan can exceed the lifespan of
a VCL, a dynamic backend cannot survive its VCL. For this reason I
find VCL objects compelling for VMOD writers, because VMODs merely
allocate/free objects, while VCLs manage/use them.

One wouldn't be required to tie dynamic backends to an object, but
they would have to keep track of the per-VCL resources themselves.

For cold/warm state, I'm all for passing the events to VMODs (if the
event specifies which VCL is sending the event) or even better to
objects (since they *are* managed by a VCL).

Allowing VMODs to follow the cold/warm state on their own resources
is interesting[1].

Putting all this together:
- VCLs own static backends
- VMODs should own dynamic backends
- Objects are the best candidates to manage dynamic backends
- VCLs should only send events to things it manages:
  static resources, objects (not sure about VMODs)
- VMODs *can* propagate events to backends

Of course the last bullet point means more work for VMOD writers, but
in most cases it's probably just walking a data structure and forwarding
the event to each item.

It also has the advantage of not introducing order issues, and allowing
VMODs to override the VCL warm/cold state policy.

The first bullet point on the other hand means only *minor* modifications
are needed in Varnish to enable that.

Best Regards,
Dridi

[1] but what happens on vmod_tour, stays on vmod_tour

On Thu, Jun 11, 2015 at 11:16 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> --------
>
> I've been pop'ing stack and come back down to this old topic.



From martin at varnish-software.com  Fri Jun 12 10:01:41 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Fri, 12 Jun 2015 12:01:41 +0200
Subject: [PATCH 3/5] Pass the VRT_CTX down to the vmod init function
In-Reply-To: <557A7A61.1050709@schokola.de>
References: <1433927162-17564-1-git-send-email-martin@varnish-software.com>
 <1433927162-17564-3-git-send-email-martin@varnish-software.com>
 <2697.1434055733@critter.freebsd.dk> <557A7A61.1050709@schokola.de>
Message-ID: <CANTn4cq9G0Wo3znp=j1KSvtafPAF6AOtZ28dBW7gxjNJ8c3+Sw@mail.gmail.com>

+1

Martin

On 12 June 2015 at 08:21, Nils Goroll <slink at schokola.de> wrote:

>
>
> On 11/06/15 22:48, Poul-Henning Kamp wrote:
> >
> > However, I very much suspect that temptation to be there no matter
> > what we do, so providing a way to clearly signal this fact and
> > thereby avoiding panics may be a good idea ?
>
> Sounds like a really good idea to me.
>



-- 
<http://varnish-software.com>*Martin Blix Grydeland*
Senior Developer | Varnish Software AS
Mobile: +47 992 74 756
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150612/cba79608/attachment.html>

From slink at schokola.de  Fri Jun 12 12:36:26 2015
From: slink at schokola.de (Nils Goroll)
Date: Fri, 12 Jun 2015 14:36:26 +0200
Subject: Vmod constants
In-Reply-To: <CAJV_h0aYsidiaE006R2Z-3iEU0+8sgqwR2A3wGZ6Kq7HWvVscg@mail.gmail.com>
References: <CAJV_h0aYsidiaE006R2Z-3iEU0+8sgqwR2A3wGZ6Kq7HWvVscg@mail.gmail.com>
Message-ID: <557AD24A.903@schokola.de>

I like vmod constants and I wish the implementation would also cover the enum
case, which, atm, triggers my COPOD (Compulsive Obsessive Premature Optimization
Disorder) ;p

We are currently passing strings to vmods which need to be parsed, rather than
real c enums resolved at vcc time.

Thanks, Nils



From lkarsten at varnish-software.com  Tue Jun 16 12:37:13 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Tue, 16 Jun 2015 14:37:13 +0200
Subject: Planning VDD15Q3
In-Reply-To: <20150608143928.GB4317@immer.varnish-software.com>
References: <20150608143928.GB4317@immer.varnish-software.com>
Message-ID: <20150616123712.GA31003@immer.varnish-software.com>

On Mon, Jun 08, 2015 at 04:39:29PM +0200, Lasse Karstensen wrote:
> Can anyone that wants to join please fill out (non-binding) the
> following doodle, so we can find a timeslot for it?
>     http://doodle.com/uxr5zxq6necwqvnb

Hi all.

Just a quick reminder; please fill this out today or tomorrow if you
want to join.

-- 
Lasse Karstensen
Varnish Software AS



From dridi at varni.sh  Tue Jun 16 13:13:08 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 16 Jun 2015 15:13:08 +0200
Subject: [PATCH] Allow probes references in VCL and VMODs
Message-ID: <CABoVN9Cj8Cadbpv7XCCDM7D2VzuScx90LkQAh-itPL3qc7T-jw@mail.gmail.com>

Hi,

Continuing on the dynamic backends track, here is a patch to allow
manipulation of probes. It is not usable yet, but will hopefully be
soon.

Best Regards,
Dridi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Allow-probes-references-in-VCL-and-VMODs.patch
Type: text/x-patch
Size: 5957 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150616/7d2fca35/attachment.bin>

From phk at phk.freebsd.dk  Tue Jun 16 13:27:07 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 16 Jun 2015 13:27:07 +0000
Subject: [PATCH] Allow probes references in VCL and VMODs
In-Reply-To: <CABoVN9Cj8Cadbpv7XCCDM7D2VzuScx90LkQAh-itPL3qc7T-jw@mail.gmail.com>
References: <CABoVN9Cj8Cadbpv7XCCDM7D2VzuScx90LkQAh-itPL3qc7T-jw@mail.gmail.com>
Message-ID: <74518.1434461227@critter.freebsd.dk>

--------
In message <CABoVN9Cj8Cadbpv7XCCDM7D2VzuScx90LkQAh-itPL3qc7T-jw at mail.gmail.com>
, Dridi Boukelmoune writes:

>Continuing on the dynamic backends track, here is a patch to allow
>manipulation of probes. It is not usable yet, but will hopefully be
>soon.

Please say a little bit more or write a mock-up VCL ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Tue Jun 16 14:27:29 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 16 Jun 2015 16:27:29 +0200
Subject: [PATCH] Allow probes references in VCL and VMODs
In-Reply-To: <74518.1434461227@critter.freebsd.dk>
References: <CABoVN9Cj8Cadbpv7XCCDM7D2VzuScx90LkQAh-itPL3qc7T-jw@mail.gmail.com>
 <74518.1434461227@critter.freebsd.dk>
Message-ID: <CABoVN9Ag5A4RNDz84KYW_=U8pMr3M6vCxrZ-Y9Gn8uGhMLk8Yg@mail.gmail.com>

On Tue, Jun 16, 2015 at 3:27 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> --------
> In message <CABoVN9Cj8Cadbpv7XCCDM7D2VzuScx90LkQAh-itPL3qc7T-jw at mail.gmail.com>
> , Dridi Boukelmoune writes:
>
>>Continuing on the dynamic backends track, here is a patch to allow
>>manipulation of probes. It is not usable yet, but will hopefully be
>>soon.
>
> Please say a little bit more or write a mock-up VCL ?

Sure, it will allow dynamic backends to have probes, and not force
vmod writers to build the probes themselves (but they could too,
probes have been miniojb'd). A vmod building a dynamic backend
could use a native VCL static probe.

--->8-------->8-------->8-------->8-------->8-------->8---
import std;
import vmod;

probe www_probe {
   ...
}

sub vcl_init {
  new dir = vmod.construct(..., www_probe);
}

sub vcl_recv {
  set req.backend_hint = dir.backend();
}

sub vcl_hit {
  if (std.healthy(req.backend_hint)) {
    ...
  }
}
--->8-------->8-------->8-------->8-------->8-------->8---



From Gauthier.Delacroix at coreye.fr  Tue Jun 16 17:08:25 2015
From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier)
Date: Tue, 16 Jun 2015 17:08:25 +0000
Subject: [PATCH] Allow varnish_reload_vcl to discard old VCL after reload
In-Reply-To: <54D8AC42.50506@schokola.de>
References: <5F530A9242E7F84F999DB40E0E268FBD45EB9CBA@mercalli.lild01.pictime.fr>
 <54D8AC42.50506@schokola.de>
Message-ID: <5F530A9242E7F84F999DB40E0E268FBD4B0E205E@mercalli.lild01.pictime.fr>

Here is an updated patch including your requests.
There is still only one parameter which disables the feature (by default) if different from zero or a positive integer, otherwise keeps N lasts available configurations.
I assumed vcl.list displays configs in the loading order, which seems to be always true.
I've removed the exit on error (just logged).

Gauthier

---
 redhat/varnish.params     |  6 ++++++
 redhat/varnish.sysconfig  |  6 ++++++
 redhat/varnish_reload_vcl | 22 +++++++++++++++++++++-
 3 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/redhat/varnish.params b/redhat/varnish.params
index 636e975..964018b 100644
--- a/redhat/varnish.params
+++ b/redhat/varnish.params
@@ -4,6 +4,12 @@
 # Set this to 1 to make systemd reload try to switch vcl without restart. 
 RELOAD_VCL=1
 
+# Set this to 0 or more to discard old VCL configurations on reload.
+# If greater than 0, these last configurations will be kept available.
+# Any other value will disable configuration discarding.
+# To make this work, you need to set RELOAD_VCL variable to 1.
+KEEP_AVAILABLE_VCL=
+
 # Main configuration file. You probably want to change it.
 VARNISH_VCL_CONF=/etc/varnish/default.vcl 
 
diff --git a/redhat/varnish.sysconfig b/redhat/varnish.sysconfig
index 6aa2354..c1dd404 100644
--- a/redhat/varnish.sysconfig
+++ b/redhat/varnish.sysconfig
@@ -24,6 +24,12 @@ NPROCS="unlimited"
 # use Alternative 3, Advanced configuration, below
 RELOAD_VCL=1
 
+# Set this to 0 or more to discard old VCL configurations on reload.
+# If greater than 0, the n last configurations will be kept available.
+# Any other value will disable configuration discarding.
+# To make this work, you need to set RELOAD_VCL variable to 1.
+KEEP_AVAILABLE_VCL=
+
 # This file contains 4 alternatives, please use only one.
 
 ## Alternative 1, Minimal configuration, no VCL
diff --git a/redhat/varnish_reload_vcl b/redhat/varnish_reload_vcl
index be3c21a..5f12c52 100755
--- a/redhat/varnish_reload_vcl
+++ b/redhat/varnish_reload_vcl
@@ -10,7 +10,7 @@
 # The following environment variables have to be set:
 # RELOAD_VCL, VARNISH_VCL_CONF, VARNISH_ADMIN_LISTEN_PORT
 # The following are optional:
-# VARNISH_SECRET_FILE, VARNISH_ADMIN_LISTEN_ADDRESS
+# VARNISH_SECRET_FILE, VARNISH_ADMIN_LISTEN_ADDRESS, KEEP_AVAILABLE_VCL
 #
 # Requires GNU bash and GNU date
 #
@@ -26,6 +26,7 @@ print_debug() {
   echo "
 Parsed configuration:
 RELOAD_VCL=\"$RELOAD_VCL\"
+KEEP_AVAILABLE_VCL=\"$KEEP_AVAILABLE_VCL\"
 VARNISH_VCL_CONF=\"$VARNISH_VCL_CONF\"
 VARNISH_ADMIN_LISTEN_ADDRESS=\"$VARNISH_ADMIN_LISTEN_ADDRESS\"
 VARNISH_ADMIN_LISTEN_PORT=\"$VARNISH_ADMIN_LISTEN_PORT\"
@@ -108,6 +109,25 @@ else
   echo "$VARNISHADM vcl.use failed"
   exit 1
 fi
+
+if [ "$KEEP_AVAILABLE_VCL" -ge 0 ] 2> /dev/null; then
+  $debug && echo "Keeping last $KEEP_AVAILABLE_VCL available configs"
+  available=`$VARNISHADM vcl.list | grep "^available"`
+  available_count=`echo "$available" | wc -l`
+  $debug && echo "Found $available_count available configs"
+  discard_count=$((available_count - KEEP_AVAILABLE_VCL))
+  $debug && echo "Discarding $discard_count configs"
+  if [ $discard_count -gt 0 ]; then
+    echo "$available" | head -n $discard_count | awk '{ print $3 }' | while read discard; do
+      if $VARNISHADM vcl.discard $discard > /dev/null; then
+        $debug && echo "$VARNISHADM vcl.discard $discard succeded"
+      else
+        echo "$VARNISHADM vcl.discard $discard failed"
+      fi
+    done
+  fi
+fi
+
 $VARNISHADM vcl.list
 echo Done
 exit 0
-- 
1.8.3.1



From slink at schokola.de  Tue Jun 16 17:21:36 2015
From: slink at schokola.de (Nils Goroll)
Date: Tue, 16 Jun 2015 19:21:36 +0200
Subject: vmod_blob / vmod_blobobj for encoding/decoding to/from blobs?
Message-ID: <55805B20.70107@schokola.de>

Hi,

ever since we have VCL BLOBs, I wanted to write a new digest module which takes
blobs as arguments and return values and leaves all the encode/decode stuff to
some encode/decode module.

So the encode/decode module is first to be done and I have been pondering how to
best design the interface, but none of the ideas I could come up with quite
matched my expectations.

So I re-interated and got to the idea of a functional-style vmod_blob to create
blobs from encoded stings and vice versa and an oo-style vmod_blobobj to do the
same oo-style.

The only disadvantage I can see with this approach is that the simple "decode
this string as a string" function looks clumsy.

As I'd be about to "claim" the quite generic name "blob" and because
encoding/decoding is so fundamental, I'd really like to hear opinions about this
suggestion.

vcc interfaces and some sample code mocks follow, see
http://etherpad.wikimedia.org/p/libvmod_code.vcc for my full WIP document

	--


/* functional */
$Module blob
$Function BLOB from(ENUM { IDENTITY, BASE64, URL, HEX, ... }, STRING_LIST)
$Function STRING as(ENUM { IDENTITY, BASE64, URL, HEXLC, ....}, BLOB)

/* oo */
$Module blobobj
$Object variable()
$Object const(ENUM { IDENTITY, BASE64, URL, HEX, ... }, STRING_LIST)

// oo getters
$Method BLOB .get()
$Method STRING .as(ENUM { IDENTITY, BASE64, URL, HEXLC, ... });

// oo setters for variable() objects - for const log warning and have no effect
$Method VOID .set(BLOB)
$Method VOID .set_from(ENUM { IDENTITY, BASE64, URL, HEX, ... }, STRING_LIST)
// functional


VCL MOCK:

// functional
set req.http.x-digest = blob.as(BASE64, digest4.hmac(MD5,
    blob.from(HEX, "2193763978af5623bc42"), # key
    blob.from(HEX, req.http.value)));

// a clumsy way of writing ..: = decode.from(URL, req.url);
set req.http.x-decoded-url = blob.as(IDENTITY, blob.from(URL, req.url));

// oo
sub vcl_init {
        new tohash = blobobj.variable();
        new secret_key = blobobj.const(HEX, "2193763978af5623bc42");
        new hashval = blobobj.variable();

}
sub vcl_recv {
        // could also write encoded.set(blob.from(....))
        tohash.set_from(HEX, req.http.value);
        hashval.set(digest4.hmac(MD5, secret_key.get, tohash.get));
        set req.http.x-digest = hashval.as(BASE64);
}



From slink at schokola.de  Wed Jun 17 12:42:17 2015
From: slink at schokola.de (Nils Goroll)
Date: Wed, 17 Jun 2015 14:42:17 +0200
Subject: vmod_blob / vmod_blobobj for encoding/decoding to/from blobs?
In-Reply-To: <55805B20.70107@schokola.de>
References: <55805B20.70107@schokola.de>
Message-ID: <55816B29.9050702@schokola.de>

Hi,

after some discussion with dridi and martin at uplex this is what I have in
http://etherpad.wikimedia.org/p/libvmod_code.vcc now:

- vmod_convert as a functional interface with nmemonic short function names for
  the common use cases

- vmod_blob for converting to/from blob objects oo-style

comments are still welcome.


= CURRENT DRAFT =

== Formats to support ==

(L) is for (L)ater

=== encode + decode formats ===
* IDENTITY / STRING
* BASE64
** for encode, defaults to BASE64 (no padding)
** for decode, accept BASE64URL tolerates missing padding at end of string
* URL
** as in vmod_urlcode
* HEX
** for encode, default to HEXLC
** for decode, accept any
* ROT13 (L)
* H2HUFF (L) (http/2 HPACK huffman table)

=== additional encode formats ===
* BASE64URL : encode: +/ replaced by -_, no padding
* BASE64URLPAD: same with padding (exotic)
* HEXLC (same as HEX)
* HEXUC (upcase)

== vmod_convert : functional interface ==

$Module convert 3 convert to/from different encodings
// common use cases
// b->s
$Function STRING encode(ENUM { IDENTITY, BASE64, HEXLC, ... }, BLOB)
// s->b
$Function BLOB decode(ENUM { IDENTITY, BASE64, HEX, ... }, STRING_LIST)
// s->s
$Function STRING transcode(ENUM { IDENTITY, BASE64, HEX, ... }, ENUM { IDENTITY,
BASE64, HEXLC, ... }, STRING_LIST)
// EXOTIC use case b->b
$Function BLOB transcode_blob(ENUM { IDENTITY, BASE64, HEX, ... }, ENUM {
IDENTITY, BASE64, HEXLC, ... }, BLOB)

# example

sub vcl_recv {
	    set req.http.x-hash = convert.encode(BASE64, digest4.hash(MD5,
convert.decode(HEX, "0x123abcdef", "ffaacc11")));
	    // inefficient, but just an example
	    if (req.http.Authentication != "Basic " + convert.transcode(IDENTITY,
BASE64, "user:passwort")) {
			return (synth(403));
	    }
}

== vmod blob : oo-interface for handling blob objects with encode/decode ==

$Module blob 3 Blob objects
// create a const object which cannot be set/apppended to - PRIV_VCL scope
$Object const(ENUM { IDENTITY, BASE64, HEXLC, ... }, STRING_LIST)
// create a blob variable object - PRIV_TASK scope
$Object var()

$Method BLOB get()
$Method VOID set(BLOB)
$Method VOID append(BLOB)
$Method BLOB splice(BLOB, INT, INT)

$Method VOID from(ENUM { IDENTITY, BASE64, HEX, ... }, STRING_LIST)
$Method STRING encode(ENUM { IDENTITY, BASE64, HEX, ... })

vcl_init {
	    new constblob = blob.const(BASE64, "V2VkIDE3IEp1biAxMTozNjoxNCBDRVNUIDIwMTUK");
	    new myhash = blob.var();
	    new myhmac = blob.var();
}
vcl_recv {
	    myhash.set(digest4.hash(SHA256, convert.decode(HEX, "0xabcdef")));
	    set req.http.x-out = myhash.encode(BASE64);
	    myhmac.set(digest4.hmac(SHA256, constblob.get(), myhash.get()));
}





From dridi at varni.sh  Wed Jun 17 13:33:25 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 17 Jun 2015 15:33:25 +0200
Subject: [PATCH] Flush varnishncsa output during rotations
Message-ID: <CABoVN9D34ORUYaSGnwh91V3qGO3rLdJ2VqbTySHi17J-GDc-uQ@mail.gmail.com>

Hi,

Looking at how to fix a test failure on an ARMv7 box [1] I looked at
the log rotation code in varnishncsa and noticed it didn't flush before
closing the file.

Comments ?

Regarding the failing test, I think it's a simpler matter of time-sensitive
testing on a slow box. I will increase the delays and see whether it still
fails.

Best Regards,
Dridi

[1] http://jenkins.varnish-cache.org/view/varnish-master/job/varnish-master-gitbuild-armv7l/202/console
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Flush-varnishncsa-output-during-rotations.patch
Type: text/x-patch
Size: 1038 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150617/6d406e77/attachment.bin>

From slink at schokola.de  Wed Jun 17 13:42:06 2015
From: slink at schokola.de (Nils Goroll)
Date: Wed, 17 Jun 2015 15:42:06 +0200
Subject: [PATCH] Flush varnishncsa output during rotations
In-Reply-To: <CABoVN9D34ORUYaSGnwh91V3qGO3rLdJ2VqbTySHi17J-GDc-uQ@mail.gmail.com>
References: <CABoVN9D34ORUYaSGnwh91V3qGO3rLdJ2VqbTySHi17J-GDc-uQ@mail.gmail.com>
Message-ID: <5581792E.2050100@schokola.de>

On 17/06/15 15:33, Dridi Boukelmoune wrote:
> 
> Regarding the failing test, I think it's a simpler matter of time-sensitive
> testing on a slow box. I will increase the delays and see whether it still
> fails.

No idea if it is related (didn't look), but the log api is racy and generates
spurious overruns. I have a patch ready which awaits long load testing by Geoff,
should this be successful, I will post it for review.

Regarding anything which could be related to spurious overruns, please hold your
horses until next week (hopefully).

Nils



From dridi at varni.sh  Wed Jun 17 13:52:42 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 17 Jun 2015 15:52:42 +0200
Subject: [PATCH] Flush varnishncsa output during rotations
In-Reply-To: <5581792E.2050100@schokola.de>
References: <CABoVN9D34ORUYaSGnwh91V3qGO3rLdJ2VqbTySHi17J-GDc-uQ@mail.gmail.com>
 <5581792E.2050100@schokola.de>
Message-ID: <CABoVN9AiPrhHtVabwU3Dbs_RJh7wYNNRHLw05yqy8EiONCGYGw@mail.gmail.com>

On Wed, Jun 17, 2015 at 3:42 PM, Nils Goroll <slink at schokola.de> wrote:
> Regarding anything which could be related to spurious overruns, please hold your
> horses until next week (hopefully).

As discussed on IRC, I'm in no hurry. And my patch should probably
call fsync instead of reusing flushout (fflush).

Dridi



From dridi at varni.sh  Wed Jun 17 14:29:52 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 17 Jun 2015 16:29:52 +0200
Subject: [PATCH] Flush varnishncsa output during rotations
In-Reply-To: <CABoVN9AiPrhHtVabwU3Dbs_RJh7wYNNRHLw05yqy8EiONCGYGw@mail.gmail.com>
References: <CABoVN9D34ORUYaSGnwh91V3qGO3rLdJ2VqbTySHi17J-GDc-uQ@mail.gmail.com>
 <5581792E.2050100@schokola.de>
 <CABoVN9AiPrhHtVabwU3Dbs_RJh7wYNNRHLw05yqy8EiONCGYGw@mail.gmail.com>
Message-ID: <CABoVN9CRtH8RHBYQ5SmrteoiytH5n4zmnyEugEr4zjQiJiWinQ@mail.gmail.com>

After more discussions on IRC I'm now convinced that nothing's wrong here.

In the mean time, giving more time to the ARMv7 box was enough to make
the test suite pass again:
http://jenkins.varnish-cache.org/view/varnish-master/job/varnish-master-gitbuild-armv7l/203/

Dridi



From dridi at varni.sh  Thu Jun 18 22:43:31 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 19 Jun 2015 00:43:31 +0200
Subject: [PATCH] Allow ACL references in VMODs
Message-ID: <CABoVN9C-614NB9KSJspjUB1kM4fZSBVwYGNTZf8Zk_PistkJFg@mail.gmail.com>

Hi,

Once again related to my work on dynamic backends, but also probably
useful in other cases, checking ACLs in VMODs. Unlike my similar patch
for probes[1], this one is already tested. I'll first give some background,
then the rationale, and finally the caveats.

Background:
The DNS director from Varnish 3 used to take IP addresses and ranges
to generate static backends, and would enable/disable the backends
after DNS lookups. The director would also ignore DNS entries that
don't match the static list of IP addresses. DNS director backends
don't have probes, probably because the backends always exist, and
that they'd be marked as sick when no DNS entry match them.

Please correct me if I'm wrong, I'm basing my DNS director for Varnish
4.1 on this understanding.

Rationale:
In Varnish 4.1, provided that we introduce dynamic backends, we don't
need that at all. We can simply create/destroy backends after lookups.
However, I see two reasons for keeping IP addresses around:
- it's familiar to Varnish 3 users
- it may mitigate DNS spoofing (I am no security expert)

I don't want to rewrite this logic, especially since we already have a
good "data structure" in Varnish for that. So I thought it could benefit
VMOD writers to have access to the named ACLs.

Caveats:
The ACL declaration ends up being a function, so that would be the
first VCL type exposed as a function to VMODs (I don't see that as a
problem). Because of the way code is generated, I had to move things
around and put an ugly comment to circumvent spurious code generation.
With this patch, its "probe" sibling[1] and the existing backend
handling, code duplication is emerging in libvcc.

This email is probably^W bigger than the patch, comments ?

Best Regards,
Dridi

[1] https://www.varnish-cache.org/lists/pipermail/varnish-dev/2015-June/008357.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Allow-ACL-references-in-VMODs.patch
Type: text/x-patch
Size: 6518 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150619/28c667da/attachment.bin>

From dridi at varni.sh  Thu Jun 18 22:51:02 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 19 Jun 2015 00:51:02 +0200
Subject: [PATCH] Allow ACL references in VMODs
In-Reply-To: <CABoVN9C-614NB9KSJspjUB1kM4fZSBVwYGNTZf8Zk_PistkJFg@mail.gmail.com>
References: <CABoVN9C-614NB9KSJspjUB1kM4fZSBVwYGNTZf8Zk_PistkJFg@mail.gmail.com>
Message-ID: <CABoVN9DXGDFP-8ZpEY0zaBVRps-FirRUavn+q4JXmCYT40eDHQ@mail.gmail.com>

Same patch, with the missing test case.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Allow-ACL-references-in-VMODs.patch
Type: text/x-patch
Size: 7209 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150619/ad0378ae/attachment.bin>

From Gauthier.Delacroix at coreye.fr  Tue Jun 23 16:15:28 2015
From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier)
Date: Tue, 23 Jun 2015 16:15:28 +0000
Subject: [PATCH] Variable for vcl_dir in startup scripts
Message-ID: <5F530A9242E7F84F999DB40E0E268FBD4B0E69A5@mercalli.lild01.pictime.fr>

This commit (https://github.com/varnish/Varnish-Cache/commit/09c93b40c1820faf564ec7e39bfaccad6ca88343) added a syntax check when starting the varnish service, but it didn't take into account the fact that the config could setup a vcl_dir where relative includes are stored.

My proposal is to add a startup variable for the vcl_dir, so that the syntax check works with a different vcl_dir.

It seems to work fine with old config files with undefined VARNISH_VCL_DIR and absolute path in VARNISH_VCL_CONF so it's not really a breaking change, even if working with an old config file would make a dirty start command.

Gauthier

---
 redhat/varnish.initrc    |  6 +++---
 redhat/varnish.params    |  5 ++++-
 redhat/varnish.service   |  7 +++++--
 redhat/varnish.sysconfig | 10 +++++++---
 4 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc
index 117e334..d1c6691 100755
--- a/redhat/varnish.initrc
+++ b/redhat/varnish.initrc
@@ -125,10 +125,10 @@ rh_status_q() {
 }
 
 configtest() {
-    if [ -f "$VARNISH_VCL_CONF" ]; then
-        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp > /dev/null && echo "Syntax ok"
+    if [ -f "${VARNISH_VCL_DIR}/${VARNISH_VCL_CONF}" ]; then
+        $exec -f "${VARNISH_VCL_DIR}/${VARNISH_VCL_CONF}" -p vcl_dir="${VARNISH_VCL_DIR}" -C -n /tmp > /dev/null && echo "Syntax ok"
     else
-	echo "VARNISH_VCL_CONF is  unset or does not point to a file"
+	echo "VARNISH_VCL_CONF is  unset or does not point to a file in VARNISH_VCL_DIR"
     fi
 }
 
diff --git a/redhat/varnish.params b/redhat/varnish.params
index 27a14dd..5f2a357 100644
--- a/redhat/varnish.params
+++ b/redhat/varnish.params
@@ -4,8 +4,11 @@
 # Set this to 1 to make systemd reload try to switch vcl without restart.
 RELOAD_VCL=1
 
+# Main configuration directory.
+VARNISH_VCL_DIR=/etc/varnish
+
 # Main configuration file. You probably want to change it.
-VARNISH_VCL_CONF=/etc/varnish/default.vcl
+VARNISH_VCL_CONF=default.vcl
 
 # Default address and port to bind to. Blank address means all IPv4
 # and IPv6 interfaces, otherwise specify a host name, an IPv4 dotted
diff --git a/redhat/varnish.service b/redhat/varnish.service
index a4f3355..3d070ed 100644
--- a/redhat/varnish.service
+++ b/redhat/varnish.service
@@ -27,10 +27,12 @@ EnvironmentFile=/etc/varnish/varnish.params
 Type=forking
 PIDFile=/var/run/varnish.pid
 PrivateTmp=true
-ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF
+ExecStartPre=/usr/sbin/varnishd -C \
+	-f ${VARNISH_VCL_DIR}/${VARNISH_VCL_CONF} \
+	-p vcl_dir=${VARNISH_VCL_DIR}
 ExecStart=/usr/sbin/varnishd \
 	-P /var/run/varnish.pid \
-	-f $VARNISH_VCL_CONF \
+	-f ${VARNISH_VCL_DIR}/${VARNISH_VCL_CONF} \
 	-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
 	-T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
 	-t $VARNISH_TTL \
@@ -38,6 +40,7 @@ ExecStart=/usr/sbin/varnishd \
 	-g $VARNISH_GROUP \
 	-S $VARNISH_SECRET_FILE \
 	-s $VARNISH_STORAGE \
+	-p vcl_dir=${VARNISH_VCL_DIR}
 	$DAEMON_OPTS
 
 ExecReload=/usr/sbin/varnish_reload_vcl
diff --git a/redhat/varnish.sysconfig b/redhat/varnish.sysconfig
index 6aa2354..1ad7f1a 100644
--- a/redhat/varnish.sysconfig
+++ b/redhat/varnish.sysconfig
@@ -56,8 +56,11 @@ RELOAD_VCL=1
 #
 # See varnishd(1) for more information.
 #
+# # Main configuration directory.
+VARNISH_VCL_DIR=/etc/varnish
+#
 # # Main configuration file. You probably want to change it :)
-VARNISH_VCL_CONF=/etc/varnish/default.vcl
+VARNISH_VCL_CONF=default.vcl
 #
 # # Default address and port to bind to
 # # Blank address means all IPv4 and IPv6 interfaces, otherwise specify
@@ -94,7 +97,7 @@ VARNISH_TTL=120
 # # DAEMON_OPTS is used by the init script.  If you add or remove options, make
 # # sure you update this section, too.
 DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
-             -f ${VARNISH_VCL_CONF} \
+             -f ${VARNISH_VCL_DIR}/${VARNISH_VCL_CONF} \
              -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
              -t ${VARNISH_TTL} \
              -p thread_pool_min=${VARNISH_MIN_THREADS} \
@@ -102,7 +105,8 @@ DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
              -p thread_pool_timeout=${VARNISH_THREAD_TIMEOUT} \
              -u varnish -g varnish \
              -S ${VARNISH_SECRET_FILE} \
-             -s ${VARNISH_STORAGE}"
+             -s ${VARNISH_STORAGE}" \
+             -p vcl_dir=${VARNISH_VCL_DIR}
 #
 
 
-- 
1.8.3.1





From fgsch at lodoss.net  Tue Jun 23 17:44:42 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Tue, 23 Jun 2015 18:44:42 +0100
Subject: [PATCH] Rename VRT_error to VCL_synth
Message-ID: <CAJV_h0YDHXVwu-bje6meaerqqs0sKLtvZbAY8Q9jaNkBVg-cww@mail.gmail.com>

Hi,

This is only called when doing return(synth(..)) so I believe it makes
sense to call it VRT_synth (before it was done on return(error(..)) thus
VRT_error).

Whether we should keep the SLT_Debug is a different story IMO but logic
above still applies.

f.-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150623/cf778c59/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Rename-VRT_error-to-VCL_synth.patch
Type: text/x-patch
Size: 1990 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150623/cf778c59/attachment.bin>

From fgsch at lodoss.net  Tue Jun 23 17:49:19 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Tue, 23 Jun 2015 18:49:19 +0100
Subject: [PATCH] Remove superflous debug
Message-ID: <CAJV_h0YVbcoCYYowO1NeRSUKEVTJFgCBBmbNDEB=nODQPb_gQw@mail.gmail.com>

Hi,

Remove "XXXX MISS" debug message - it's always followed by VCL_call MISS.

f.-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150623/d22ad982/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Remove-superflous-debug.patch
Type: text/x-patch
Size: 758 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150623/d22ad982/attachment.bin>

From dridi at varni.sh  Tue Jun 23 18:48:01 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 23 Jun 2015 20:48:01 +0200
Subject: [PATCH] Remove superflous debug
In-Reply-To: <CAJV_h0YVbcoCYYowO1NeRSUKEVTJFgCBBmbNDEB=nODQPb_gQw@mail.gmail.com>
References: <CAJV_h0YVbcoCYYowO1NeRSUKEVTJFgCBBmbNDEB=nODQPb_gQw@mail.gmail.com>
Message-ID: <CABoVN9D8bsZ9iNuFF7eV7zecQNKcYjOCNQjW4-BUMK5Kz8F9jg@mail.gmail.com>

+1

On Tue, Jun 23, 2015 at 7:49 PM, Federico Schwindt <fgsch at lodoss.net> wrote:
> Hi,
>
> Remove "XXXX MISS" debug message - it's always followed by VCL_call MISS.
>
> f.-
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev



From phk at phk.freebsd.dk  Wed Jun 24 09:47:25 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 24 Jun 2015 09:47:25 +0000
Subject: [PATCH] Rename VRT_error to VCL_synth
In-Reply-To: <CAJV_h0YDHXVwu-bje6meaerqqs0sKLtvZbAY8Q9jaNkBVg-cww@mail.gmail.com>
References: <CAJV_h0YDHXVwu-bje6meaerqqs0sKLtvZbAY8Q9jaNkBVg-cww@mail.gmail.com>
Message-ID: <55831.1435139245@critter.freebsd.dk>

--------
In message <CAJV_h0YDHXVwu-bje6meaerqqs0sKLtvZbAY8Q9jaNkBVg-cww at mail.gmail.com>,
 Federico Schwindt writes:

>This is only called when doing return(synth(..)) so I believe it makes
>sense to call it VRT_synth (before it was done on return(error(..)) thus
>VRT_error).
>
>Whether we should keep the SLT_Debug is a different story IMO but logic
>above still applies.

I don't quite have enough OCD to think we need to rename it, but I
certainly think the SLT_Debug can go.  (Also the XXXX miss from
the other email).

If you feel strongly about the rename, then go ahead.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Wed Jun 24 16:21:47 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 24 Jun 2015 18:21:47 +0200
Subject: Documentation for a DNS director
Message-ID: <CABoVN9CG1w2TAEDqLb77-QV=V2Vu=mwBFM6A8wQx6cKqbKX_hQ@mail.gmail.com>

Hi,

I have a working DNS director prototype on another branch, and I
would like to port that to the current master. Before I do that, I have
to write test cases around dynamic backends support, coming soon
hopefully.

I'd like feedback on what the director looks like to end-users, before
I start porting it.

Current generated man:
--->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8---
OBJECT DNS
       Description
              Create a DNS director.

              The director creates backends with DNS lookups and chooses  them
              in  a  round  robin  fashion. An ACL (Access Control List) can be
              used as a white-list to restrict the use of resolved addresses.

       Example

                 acl www_backends {
                      "192.168.15.0"/24;
                      !"192.168.15.1";
                 }

                 sub vcl_init {
                      new www_dir = directors.dns("www.example.com", "80");
                      www_dir.restrict_to(www_backends);
                      www_dir.set_ttl(5m);
                 }

                 sub vcl_recv {
                      set req.backend_hint = www_dir.backend();
                 }

   VOID dns.restrict_to(ACL)
       Prototype
              VOID dns.restrict_to(ACL)

       Description
              Restrict usage of resolved IP addresses to a named ACL.

   VOID dns.set_ttl(DURATION)
       Prototype
              VOID dns.set_ttl(DURATION)

       Description
              Set the DNS lookup TTL (defaults to one hour).

   BACKEND dns.backend()
       Prototype
              BACKEND dns.backend()

       Description
              Pick a backend from the director.
--->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8---

Yes, I hope this director will be part of the directors built-in, and
don't want to maintain an out-of-tree module :) However, due to
its nature, it doesn't rely on the vdir code shared by the other
directors.

Comments? Questions?

Related:
https://www.varnish-cache.org/lists/pipermail/varnish-dev/2015-June/008367.html

Related but not supported yet by my prototype:
https://www.varnish-cache.org/lists/pipermail/varnish-dev/2015-June/008359.html

Somewhat related (to the not sharing vdir code), but not for 4.1:
https://www.varnish-cache.org/lists/pipermail/varnish-dev/2015-May/008329.html

Best Regards,
Dridi

PS. Oops!... I did it again, big email :-/



From Gauthier.Delacroix at coreye.fr  Thu Jun 25 11:52:47 2015
From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier)
Date: Thu, 25 Jun 2015 11:52:47 +0000
Subject: [PATCH] Variable for vcl_dir in startup scripts
Message-ID: <5F530A9242E7F84F999DB40E0E268FBD4B0EAF3F@mercalli.lild01.pictime.fr>

The syntax check actually blocks other startup parameters, like vcc_err_unref=off.

The syntax check can ignore tuning parameters, but those required to run the VCL config should be included (are there others ?).

If you think the way I used to set vcl_dir as a startup script variable is correct, I can follow with other ones in a new patch. If not, what would be the better way ?

Gauthier




From Gauthier.Delacroix at coreye.fr  Thu Jun 25 14:26:58 2015
From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier)
Date: Thu, 25 Jun 2015 14:26:58 +0000
Subject: [PATCH] Variable for vcl_dir in startup scripts
Message-ID: <5F530A9242E7F84F999DB40E0E268FBD4B0EB01D@mercalli.lild01.pictime.fr>

Here is another patch proposal to make syntax check handle parameters required to compile the VCL (vcl_dir, etc.) without creating a startup variable for each parameter.

It just adds a COMPILE_OPTS that is merged in DAEMON_OPTS to start Varnish but is used alone in the syntax check.

Gauthier

>From d1567a956d53a489aa4ace66ce0b1c1ef745570b Mon Sep 17 00:00:00 2001
From: Gauthier Delacroix <gauthier.delacroix at coreye.fr>
Date: Thu, 25 Jun 2015 15:06:08 +0200
Subject: [PATCH] Add COMPILE_OPTS in startup scripts to make syntax check
 check handle compilation parameters

---
 redhat/varnish.initrc    |  6 ++++--
 redhat/varnish.params    |  8 ++++++++
 redhat/varnish.service   |  3 ++-
 redhat/varnish.sysconfig | 11 ++++++++++-
 4 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc
index 117e334..0bde074 100755
--- a/redhat/varnish.initrc
+++ b/redhat/varnish.initrc
@@ -126,9 +126,11 @@ rh_status_q() {
 
 configtest() {
     if [ -f "$VARNISH_VCL_CONF" ]; then
-        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp > /dev/null && echo "Syntax ok"
+        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp $COMPILE_OPTS > /dev/null \
+          && echo "Syntax ok"
     else
-	echo "VARNISH_VCL_CONF is  unset or does not point to a file"
+	echo "VARNISH_VCL_CONF is unset or does not point to a file"
+	echo "Also check that COMPILE_OPTS is set depending on the VCL config"
     fi
 }
 
diff --git a/redhat/varnish.params b/redhat/varnish.params
index 27a14dd..970d088 100644
--- a/redhat/varnish.params
+++ b/redhat/varnish.params
@@ -31,5 +31,13 @@ VARNISH_TTL=120
 VARNISH_USER=varnish
 VARNISH_GROUP=varnish
 
+# Startup options required to compile the configuration.
+# The following run-time parameters must be defined here, if needed:
+# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref, vcc_unsafe_path,
+# vcl_dir, vmod_dir
+# Defining them in DAEMON_OPTS may result in a syntax check failure.
+# See the man page varnishd(1).
+#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
+
 # Other options, see the man page varnishd(1)
 #DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300"
diff --git a/redhat/varnish.service b/redhat/varnish.service
index a4f3355..a08db58 100644
--- a/redhat/varnish.service
+++ b/redhat/varnish.service
@@ -27,7 +27,7 @@ EnvironmentFile=/etc/varnish/varnish.params
 Type=forking
 PIDFile=/var/run/varnish.pid
 PrivateTmp=true
-ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF
+ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF $COMPILE_OPTS
 ExecStart=/usr/sbin/varnishd \
 	-P /var/run/varnish.pid \
 	-f $VARNISH_VCL_CONF \
@@ -38,6 +38,7 @@ ExecStart=/usr/sbin/varnishd \
 	-g $VARNISH_GROUP \
 	-S $VARNISH_SECRET_FILE \
 	-s $VARNISH_STORAGE \
+	$COMPILE_OPTS \
 	$DAEMON_OPTS
 
 ExecReload=/usr/sbin/varnish_reload_vcl
diff --git a/redhat/varnish.sysconfig b/redhat/varnish.sysconfig
index 6aa2354..0e376ff 100644
--- a/redhat/varnish.sysconfig
+++ b/redhat/varnish.sysconfig
@@ -91,6 +91,14 @@ VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
 # # Default TTL used when the backend does not specify one
 VARNISH_TTL=120
 #
+# Startup options required to compile the configuration.
+# The following run-time parameters must be defined here, if needed:
+# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref, vcc_unsafe_path,
+# vcl_dir, vmod_dir
+# Defining them in DAEMON_OPTS may result in a syntax check failure.
+# See the man page varnishd(1).
+#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
+#
 # # DAEMON_OPTS is used by the init script.  If you add or remove options, make
 # # sure you update this section, too.
 DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
@@ -102,7 +110,8 @@ DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
              -p thread_pool_timeout=${VARNISH_THREAD_TIMEOUT} \
              -u varnish -g varnish \
              -S ${VARNISH_SECRET_FILE} \
-             -s ${VARNISH_STORAGE}"
+             -s ${VARNISH_STORAGE}" \
+             ${COMPILE_OPTS}
 #
 
 
-- 
1.8.3.msysgit.0





From fgsch at lodoss.net  Thu Jun 25 15:52:36 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Thu, 25 Jun 2015 16:52:36 +0100
Subject: [PATCH] Variable for vcl_dir in startup scripts
In-Reply-To: <5F530A9242E7F84F999DB40E0E268FBD4B0EB01D@mercalli.lild01.pictime.fr>
References: <5F530A9242E7F84F999DB40E0E268FBD4B0EB01D@mercalli.lild01.pictime.fr>
Message-ID: <CAJV_h0Zy_=Tub7x7Rm=HJgiA2v83ACoVCDo_Jc1F-xf2+gNK0A@mail.gmail.com>

Hi,

My first inclination is to ensure that the ExecStartPre line uses the same
parameters as ExecStart.

So my questions is how do you set vcl_dir? Do you edit varnish.service or
varnish.params?

Wouldn't adding DAEMON_OPTS to ExecStartPre (and configtest) do it?


On Thu, Jun 25, 2015 at 3:26 PM, Delacroix, Gauthier <
Gauthier.Delacroix at coreye.fr> wrote:

> Here is another patch proposal to make syntax check handle parameters
> required to compile the VCL (vcl_dir, etc.) without creating a startup
> variable for each parameter.
>
> It just adds a COMPILE_OPTS that is merged in DAEMON_OPTS to start Varnish
> but is used alone in the syntax check.
>
> Gauthier
>
> From d1567a956d53a489aa4ace66ce0b1c1ef745570b Mon Sep 17 00:00:00 2001
> From: Gauthier Delacroix <gauthier.delacroix at coreye.fr>
> Date: Thu, 25 Jun 2015 15:06:08 +0200
> Subject: [PATCH] Add COMPILE_OPTS in startup scripts to make syntax check
>  check handle compilation parameters
>
> ---
>  redhat/varnish.initrc    |  6 ++++--
>  redhat/varnish.params    |  8 ++++++++
>  redhat/varnish.service   |  3 ++-
>  redhat/varnish.sysconfig | 11 ++++++++++-
>  4 files changed, 24 insertions(+), 4 deletions(-)
>
> diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc
> index 117e334..0bde074 100755
> --- a/redhat/varnish.initrc
> +++ b/redhat/varnish.initrc
> @@ -126,9 +126,11 @@ rh_status_q() {
>
>  configtest() {
>      if [ -f "$VARNISH_VCL_CONF" ]; then
> -        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp > /dev/null && echo
> "Syntax ok"
> +        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp $COMPILE_OPTS > /dev/null
> \
> +          && echo "Syntax ok"
>      else
> -       echo "VARNISH_VCL_CONF is  unset or does not point to a file"
> +       echo "VARNISH_VCL_CONF is unset or does not point to a file"
> +       echo "Also check that COMPILE_OPTS is set depending on the VCL
> config"
>      fi
>  }
>
> diff --git a/redhat/varnish.params b/redhat/varnish.params
> index 27a14dd..970d088 100644
> --- a/redhat/varnish.params
> +++ b/redhat/varnish.params
> @@ -31,5 +31,13 @@ VARNISH_TTL=120
>  VARNISH_USER=varnish
>  VARNISH_GROUP=varnish
>
> +# Startup options required to compile the configuration.
> +# The following run-time parameters must be defined here, if needed:
> +# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref,
> vcc_unsafe_path,
> +# vcl_dir, vmod_dir
> +# Defining them in DAEMON_OPTS may result in a syntax check failure.
> +# See the man page varnishd(1).
> +#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
> +
>  # Other options, see the man page varnishd(1)
>  #DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p
> thread_pool_timeout=300"
> diff --git a/redhat/varnish.service b/redhat/varnish.service
> index a4f3355..a08db58 100644
> --- a/redhat/varnish.service
> +++ b/redhat/varnish.service
> @@ -27,7 +27,7 @@ EnvironmentFile=/etc/varnish/varnish.params
>  Type=forking
>  PIDFile=/var/run/varnish.pid
>  PrivateTmp=true
> -ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF
> +ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF $COMPILE_OPTS
>  ExecStart=/usr/sbin/varnishd \
>         -P /var/run/varnish.pid \
>         -f $VARNISH_VCL_CONF \
> @@ -38,6 +38,7 @@ ExecStart=/usr/sbin/varnishd \
>         -g $VARNISH_GROUP \
>         -S $VARNISH_SECRET_FILE \
>         -s $VARNISH_STORAGE \
> +       $COMPILE_OPTS \
>         $DAEMON_OPTS
>
>  ExecReload=/usr/sbin/varnish_reload_vcl
> diff --git a/redhat/varnish.sysconfig b/redhat/varnish.sysconfig
> index 6aa2354..0e376ff 100644
> --- a/redhat/varnish.sysconfig
> +++ b/redhat/varnish.sysconfig
> @@ -91,6 +91,14 @@ VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
>  # # Default TTL used when the backend does not specify one
>  VARNISH_TTL=120
>  #
> +# Startup options required to compile the configuration.
> +# The following run-time parameters must be defined here, if needed:
> +# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref,
> vcc_unsafe_path,
> +# vcl_dir, vmod_dir
> +# Defining them in DAEMON_OPTS may result in a syntax check failure.
> +# See the man page varnishd(1).
> +#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
> +#
>  # # DAEMON_OPTS is used by the init script.  If you add or remove
> options, make
>  # # sure you update this section, too.
>  DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
> @@ -102,7 +110,8 @@ DAEMON_OPTS="-a
> ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
>               -p thread_pool_timeout=${VARNISH_THREAD_TIMEOUT} \
>               -u varnish -g varnish \
>               -S ${VARNISH_SECRET_FILE} \
> -             -s ${VARNISH_STORAGE}"
> +             -s ${VARNISH_STORAGE}" \
> +             ${COMPILE_OPTS}
>  #
>
>
> --
> 1.8.3.msysgit.0
>
>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150625/e70f5015/attachment.html>

From Gauthier.Delacroix at coreye.fr  Thu Jun 25 16:12:15 2015
From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier)
Date: Thu, 25 Jun 2015 16:12:15 +0000
Subject: [PATCH] Variable for vcl_dir in startup scripts
In-Reply-To: <CAJV_h0Zy_=Tub7x7Rm=HJgiA2v83ACoVCDo_Jc1F-xf2+gNK0A@mail.gmail.com>
References: <5F530A9242E7F84F999DB40E0E268FBD4B0EB01D@mercalli.lild01.pictime.fr>
 <CAJV_h0Zy_=Tub7x7Rm=HJgiA2v83ACoVCDo_Jc1F-xf2+gNK0A@mail.gmail.com>
Message-ID: <5F530A9242E7F84F999DB40E0E268FBD4B0EB0F2@mercalli.lild01.pictime.fr>

Federico,

I've initially set vcl_dir in DAEMON_OPTS in varnish.params with 4.0.2, but it's not included in the syntax check added in 4.0.3, making my config unable to start without changing varnish.service.

I'm not aware of the reason behind this syntax check (won't a syntax error make startup fail anyway ?) but I assumed you had a good reason not to include DAEMON_OPTS.

As it's better to change varnish.params rather than varnish.service, my proposal was to add a way to include parameters needed to make the config work in the syntax check, without including the full DAEMON_OPTS.

If you finally think DAEMON_OPTS can be included in the syntax check, then I can send a really smaller patch.

Gauthier

De?: Federico Schwindt [mailto:fgsch at lodoss.net] 
Envoy??: jeudi 25 juin 2015 17:53
??: Delacroix, Gauthier
Cc?: varnish-dev at varnish-cache.org
Objet?: Re: [PATCH] Variable for vcl_dir in startup scripts

Hi,

My first inclination is to ensure that the ExecStartPre line uses the same parameters as ExecStart.

So my questions is how do you set vcl_dir? Do you edit varnish.service or varnish.params?

Wouldn't adding DAEMON_OPTS to ExecStartPre (and configtest) do it?

On Thu, Jun 25, 2015 at 3:26 PM, Delacroix, Gauthier <Gauthier.Delacroix at coreye.fr> wrote:
Here is another patch proposal to make syntax check handle parameters required to compile the VCL (vcl_dir, etc.) without creating a startup variable for each parameter.

It just adds a COMPILE_OPTS that is merged in DAEMON_OPTS to start Varnish but is used alone in the syntax check.

Gauthier

From d1567a956d53a489aa4ace66ce0b1c1ef745570b Mon Sep 17 00:00:00 2001
From: Gauthier Delacroix <gauthier.delacroix at coreye.fr>
Date: Thu, 25 Jun 2015 15:06:08 +0200
Subject: [PATCH] Add COMPILE_OPTS in startup scripts to make syntax check
?check handle compilation parameters

---
?redhat/varnish.initrc? ? |? 6 ++++--
?redhat/varnish.params? ? |? 8 ++++++++
?redhat/varnish.service? ?|? 3 ++-
?redhat/varnish.sysconfig | 11 ++++++++++-
?4 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc
index 117e334..0bde074 100755
--- a/redhat/varnish.initrc
+++ b/redhat/varnish.initrc
@@ -126,9 +126,11 @@ rh_status_q() {

?configtest() {
? ? ?if [ -f "$VARNISH_VCL_CONF" ]; then
-? ? ? ? $exec -f "$VARNISH_VCL_CONF" -C -n /tmp > /dev/null && echo "Syntax ok"
+? ? ? ? $exec -f "$VARNISH_VCL_CONF" -C -n /tmp $COMPILE_OPTS > /dev/null \
+? ? ? ? ? && echo "Syntax ok"
? ? ?else
-? ? ? ?echo "VARNISH_VCL_CONF is? unset or does not point to a file"
+? ? ? ?echo "VARNISH_VCL_CONF is unset or does not point to a file"
+? ? ? ?echo "Also check that COMPILE_OPTS is set depending on the VCL config"
? ? ?fi
?}

diff --git a/redhat/varnish.params b/redhat/varnish.params
index 27a14dd..970d088 100644
--- a/redhat/varnish.params
+++ b/redhat/varnish.params
@@ -31,5 +31,13 @@ VARNISH_TTL=120
?VARNISH_USER=varnish
?VARNISH_GROUP=varnish

+# Startup options required to compile the configuration.
+# The following run-time parameters must be defined here, if needed:
+# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref, vcc_unsafe_path,
+# vcl_dir, vmod_dir
+# Defining them in DAEMON_OPTS may result in a syntax check failure.
+# See the man page varnishd(1).
+#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
+
?# Other options, see the man page varnishd(1)
?#DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300"
diff --git a/redhat/varnish.service b/redhat/varnish.service
index a4f3355..a08db58 100644
--- a/redhat/varnish.service
+++ b/redhat/varnish.service
@@ -27,7 +27,7 @@ EnvironmentFile=/etc/varnish/varnish.params
?Type=forking
?PIDFile=/var/run/varnish.pid
?PrivateTmp=true
-ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF
+ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF $COMPILE_OPTS
?ExecStart=/usr/sbin/varnishd \
? ? ? ? -P /var/run/varnish.pid \
? ? ? ? -f $VARNISH_VCL_CONF \
@@ -38,6 +38,7 @@ ExecStart=/usr/sbin/varnishd \
? ? ? ? -g $VARNISH_GROUP \
? ? ? ? -S $VARNISH_SECRET_FILE \
? ? ? ? -s $VARNISH_STORAGE \
+? ? ? ?$COMPILE_OPTS \
? ? ? ? $DAEMON_OPTS

?ExecReload=/usr/sbin/varnish_reload_vcl
diff --git a/redhat/varnish.sysconfig b/redhat/varnish.sysconfig
index 6aa2354..0e376ff 100644
--- a/redhat/varnish.sysconfig
+++ b/redhat/varnish.sysconfig
@@ -91,6 +91,14 @@ VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
?# # Default TTL used when the backend does not specify one
?VARNISH_TTL=120
?#
+# Startup options required to compile the configuration.
+# The following run-time parameters must be defined here, if needed:
+# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref, vcc_unsafe_path,
+# vcl_dir, vmod_dir
+# Defining them in DAEMON_OPTS may result in a syntax check failure.
+# See the man page varnishd(1).
+#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
+#
?# # DAEMON_OPTS is used by the init script.? If you add or remove options, make
?# # sure you update this section, too.
?DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
@@ -102,7 +110,8 @@ DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
? ? ? ? ? ? ? -p thread_pool_timeout=${VARNISH_THREAD_TIMEOUT} \
? ? ? ? ? ? ? -u varnish -g varnish \
? ? ? ? ? ? ? -S ${VARNISH_SECRET_FILE} \
-? ? ? ? ? ? ?-s ${VARNISH_STORAGE}"
+? ? ? ? ? ? ?-s ${VARNISH_STORAGE}" \
+? ? ? ? ? ? ?${COMPILE_OPTS}
?#


--
1.8.3.msysgit.0



_______________________________________________
varnish-dev mailing list
varnish-dev at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


From dridi at varni.sh  Mon Jun 29 17:23:23 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 29 Jun 2015 19:23:23 +0200
Subject: [PATCH] Decrement backend counters on a connection close
Message-ID: <CABoVN9BtGZWDF9vNPHBrZVqCSs8=sEXiKX_+mw6znhq+NXxC4g@mail.gmail.com>

Making progress on dynamic backends testing, I found a glitch in the
max_connections handling, among other issues.

Patch discussed with fgs and phk, not yet reviewed.

Best Regards,
Dridi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Decrement-backend-counters-on-a-connection-close.patch
Type: text/x-patch
Size: 1654 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150629/621bc445/attachment.bin>

From dridi at varni.sh  Tue Jun 30 07:08:01 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 30 Jun 2015 09:08:01 +0200
Subject: [PATCH] Decrement backend counters on a connection close
In-Reply-To: <CABoVN9BtGZWDF9vNPHBrZVqCSs8=sEXiKX_+mw6znhq+NXxC4g@mail.gmail.com>
References: <CABoVN9BtGZWDF9vNPHBrZVqCSs8=sEXiKX_+mw6znhq+NXxC4g@mail.gmail.com>
Message-ID: <CABoVN9DbBr+y3L-8yxQzu9rWRcQ1nOHFWaj9F8Y38BV8c=8f2g@mail.gmail.com>

I have been too optimistic. While this solved one of my issues, it
didn't take pipe into account.

A pipe must hold a ref to a backend until the session is actually
closed, because it will increment the relevant counters. After playing
around with the backend's refcount, trying to inc/decrement it in all
code paths, I didn't come to a satisfying result. I still have 2 test
cases (1 pipe-related) where a backend hot swap (outside of the cli
thread) fails.

This patch is no good, it doesn't solve anything.

Best Regards,
Dridi

On Mon, Jun 29, 2015 at 7:23 PM, Dridi Boukelmoune <dridi at varni.sh> wrote:
> Making progress on dynamic backends testing, I found a glitch in the
> max_connections handling, among other issues.
>
> Patch discussed with fgs and phk, not yet reviewed.
>
> Best Regards,
> Dridi