From fgsch at lodoss.net  Wed Jul  1 13:05:20 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Wed, 1 Jul 2015 14:05:20 +0100
Subject: Time to retire client.identify?
Message-ID: <CAJV_h0bBJFsYnSAqhavRdRXFQ1zWs-__rzhm5imcFKRsA9H2ew@mail.gmail.com>

Hi,

The hash director needs a string now so this no longer needed.

Opinions?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150701/11aa3e6a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Retire-client.identity.patch
Type: text/x-patch
Size: 4353 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150701/11aa3e6a/attachment.bin>

From dridi at varni.sh  Wed Jul  1 13:58:51 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 1 Jul 2015 15:58:51 +0200
Subject: Time to retire client.identify?
In-Reply-To: <CAJV_h0bBJFsYnSAqhavRdRXFQ1zWs-__rzhm5imcFKRsA9H2ew@mail.gmail.com>
References: <CAJV_h0bBJFsYnSAqhavRdRXFQ1zWs-__rzhm5imcFKRsA9H2ew@mail.gmail.com>
Message-ID: <CABoVN9DB4hOnkd5wmUKtsLS7F7WmHUO6=OrJuyJZxSVMtBuf-g@mail.gmail.com>

On Wed, Jul 1, 2015 at 3:05 PM, Federico Schwindt <fgsch at lodoss.net> wrote:
> Hi,
>
> The hash director needs a string now so this no longer needed.
>
> Opinions?

Agreed



From martin at varnish-software.com  Thu Jul  2 12:07:18 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 2 Jul 2015 14:07:18 +0200
Subject: Varnish panic API
Message-ID: <CANTn4cpDVxyimMkH9mchFRAZShx-sxCOUVYFQCLDq77pBf5UEg@mail.gmail.com>

Hi phk,

Now that we expose the panic functions also to vmods, I believe we need to
define a somewhat cleaner way for different parts to produce the panic
output. It's the indentation and how that's hard coded into each panic
generator that's the major issue in my opinion, e.g. how the panic handler
in the backend vmod would have to now to insert 4 character whitespace on
each line in order to match it with where it's output would probably go in
the panic string.

The attached patch series changes this. It's a lot of patches, as it
changes all of the panic code slightly. First one shows the utility
functions I created for this purpose, and the rest are patches for
different panic string generators.

Martin

-- 
<http://varnish-software.com>*Martin Blix Grydeland*
Senior Developer | Varnish Software AS
Mobile: +47 992 74 756
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Panic-utility-functions.patch
Type: text/x-patch
Size: 3996 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Change-pan_ic-to-use-new-panic.patch
Type: text/x-patch
Size: 3120 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Change-pan_backtrace.patch
Type: text/x-patch
Size: 2087 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-Change-pan_req.patch
Type: text/x-patch
Size: 2950 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0005-Change-pan_sess.patch
Type: text/x-patch
Size: 2371 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0006-Change-pan_wrk.patch
Type: text/x-patch
Size: 2699 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0005.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0007-Change-pan_busyobj.patch
Type: text/x-patch
Size: 3482 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0006.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0008-Change-pan_ws.patch
Type: text/x-patch
Size: 3202 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0007.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0009-Change-pan_objcore.patch
Type: text/x-patch
Size: 2747 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0008.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0010-Change-pan_http.patch
Type: text/x-patch
Size: 2261 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0009.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0011-Change-pan_htc.patch
Type: text/x-patch
Size: 1467 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0010.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0012-Change-VDI_Panic.patch
Type: text/x-patch
Size: 5468 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0011.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0013-Change-VCL_Panic.patch
Type: text/x-patch
Size: 2834 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0012.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0014-Call-VCL_Panic-also-from-pan_busyobj.patch
Type: text/x-patch
Size: 769 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0013.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0015-Drop-pan_vsp-now-that-it-s-unused.patch
Type: text/x-patch
Size: 1131 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0014.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0016-Change-a-couple-of-unused-panic-generators-to-the-ne.patch
Type: text/x-patch
Size: 2963 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/0837c58c/attachment-0015.bin>

From phk at phk.freebsd.dk  Thu Jul  2 12:16:05 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 02 Jul 2015 12:16:05 +0000
Subject: Varnish panic API
In-Reply-To: <CANTn4cpDVxyimMkH9mchFRAZShx-sxCOUVYFQCLDq77pBf5UEg@mail.gmail.com>
References: <CANTn4cpDVxyimMkH9mchFRAZShx-sxCOUVYFQCLDq77pBf5UEg@mail.gmail.com>
Message-ID: <780.1435839365@critter.freebsd.dk>

--------
In message <CANTn4cpDVxyimMkH9mchFRAZShx-sxCOUVYFQCLDq77pBf5UEg at mail.gmail.com>, Martin Blix Grydeland writes:

>Now that we expose the panic functions also to vmods, I believe we need to
>define a somewhat cleaner way for different parts to produce the panic
>output.

Yes, it's getting unwieldy.

A very similar situation exists in VCC with respect to the C code output.

I've been pondering extending the VSB api so that it would automatically
indent after \n.

Something like:
	VSB_indent(struct vsb *, int more);


	VSB_printf(vsb, "{\n");
	VSB_indent(vsb, 2);
	VSB_printf(vsb, "Foo = bla;\n");
	VSB_indent(vsb, -2);
	VSB_printf(vsb, "}\n");

As you can see from that example, the indentation needs to happen on first char *after* then \n
rather than the \n itself.  I think that is the most logical and useful semantics.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From martin at varnish-software.com  Thu Jul  2 12:48:13 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 2 Jul 2015 14:48:13 +0200
Subject: Time to retire client.identify?
In-Reply-To: <CABoVN9DB4hOnkd5wmUKtsLS7F7WmHUO6=OrJuyJZxSVMtBuf-g@mail.gmail.com>
References: <CAJV_h0bBJFsYnSAqhavRdRXFQ1zWs-__rzhm5imcFKRsA9H2ew@mail.gmail.com>
 <CABoVN9DB4hOnkd5wmUKtsLS7F7WmHUO6=OrJuyJZxSVMtBuf-g@mail.gmail.com>
Message-ID: <CANTn4cq8dnj0zjo9hvdJENpQp62LtJWmy7Wo9G1QC37P3mSp2g@mail.gmail.com>

Yeah, this is OBE now.

Martin

On 1 July 2015 at 15:58, Dridi Boukelmoune <dridi at varni.sh> wrote:

> On Wed, Jul 1, 2015 at 3:05 PM, Federico Schwindt <fgsch at lodoss.net>
> wrote:
> > Hi,
> >
> > The hash director needs a string now so this no longer needed.
> >
> > Opinions?
>
> Agreed
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>



-- 
<http://varnish-software.com>*Martin Blix Grydeland*
Senior Developer | Varnish Software AS
Mobile: +47 992 74 756
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/56ee0f9d/attachment.html>

From fgsch at lodoss.net  Thu Jul  2 15:23:11 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Thu, 2 Jul 2015 16:23:11 +0100
Subject: Time to retire client.identify?
In-Reply-To: <CANTn4cq8dnj0zjo9hvdJENpQp62LtJWmy7Wo9G1QC37P3mSp2g@mail.gmail.com>
References: <CAJV_h0bBJFsYnSAqhavRdRXFQ1zWs-__rzhm5imcFKRsA9H2ew@mail.gmail.com>
 <CABoVN9DB4hOnkd5wmUKtsLS7F7WmHUO6=OrJuyJZxSVMtBuf-g@mail.gmail.com>
 <CANTn4cq8dnj0zjo9hvdJENpQp62LtJWmy7Wo9G1QC37P3mSp2g@mail.gmail.com>
Message-ID: <CAJV_h0YmBepq5Jkw7uQXWiKL8me5LqmtSZ-rbUqVE+kxq-WmRw@mail.gmail.com>

One of the things Dag pointed about this is that if you want to hash based
on the client IP address you will need to do "" + client.ip.
This is a valid concern although the only situation I can think of where
client.identity will be missed.

There are a few options:

1. Use "" + client.ip. Somewhat ugly but will do the job.

2. Do automatic conversion for arguments when the expected type is STRING
or STRING_LIST. See diff below.
We do this already in a number of places but not for arguments.

3. Stash it in the req.http space. e.g.

set req.http.session = client.ip;
set req.backend_hint = hash.backend(req.http.session);
#unset req.http.session;

4. Change the hash director to use client.identity again. e.g. drop the
STRING_LIST and use client.identity.

5. Do nothing.

As things are in master, client.identity lost it's place and it's more like
a scratch space or throw away variable and feels like a special case but
YMMV.

Any opinions?

f.-

diff --git a/lib/libvcc/vcc_expr.c
b/lib/libvcc/vcc_expr.c
index b9c6531..4019fec 100644
--- a/lib/libvcc/vcc_expr.c
+++ b/lib/libvcc/vcc_expr.c
@@ -609,6 +609,9 @@ vcc_do_arg(struct vcc *tl, struct func_arg *fa)
        } else {
                vcc_expr0(tl, &e2, fa->type);
                ERRCHK(tl);
+               if (e2->fmt != fa->type &&
+                   (fa->type == STRING || fa->type == STRING_LIST))
+                       vcc_expr_tostring(tl, &e2, fa->type);
                if (e2->fmt != fa->type) {
                        VSB_printf(tl->sb, "Wrong argument type.");
                        VSB_printf(tl->sb, "  Expected %s.",




On Thu, Jul 2, 2015 at 1:48 PM, Martin Blix Grydeland <
martin at varnish-software.com> wrote:

> Yeah, this is OBE now.
>
> Martin
>
> On 1 July 2015 at 15:58, Dridi Boukelmoune <dridi at varni.sh> wrote:
>
>> On Wed, Jul 1, 2015 at 3:05 PM, Federico Schwindt <fgsch at lodoss.net>
>> wrote:
>> > Hi,
>> >
>> > The hash director needs a string now so this no longer needed.
>> >
>> > Opinions?
>>
>> Agreed
>>
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>
>
>
>
> --
> <http://varnish-software.com>*Martin Blix Grydeland*
> Senior Developer | Varnish Software AS
> Mobile: +47 992 74 756
> We Make Websites Fly!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150702/f48d9dd7/attachment.html>

From dridi at varni.sh  Fri Jul  3 15:15:02 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 3 Jul 2015 17:15:02 +0200
Subject: Time to retire client.identify?
In-Reply-To: <CAJV_h0YmBepq5Jkw7uQXWiKL8me5LqmtSZ-rbUqVE+kxq-WmRw@mail.gmail.com>
References: <CAJV_h0bBJFsYnSAqhavRdRXFQ1zWs-__rzhm5imcFKRsA9H2ew@mail.gmail.com>
 <CABoVN9DB4hOnkd5wmUKtsLS7F7WmHUO6=OrJuyJZxSVMtBuf-g@mail.gmail.com>
 <CANTn4cq8dnj0zjo9hvdJENpQp62LtJWmy7Wo9G1QC37P3mSp2g@mail.gmail.com>
 <CAJV_h0YmBepq5Jkw7uQXWiKL8me5LqmtSZ-rbUqVE+kxq-WmRw@mail.gmail.com>
Message-ID: <CABoVN9BTTFdsLz1E4LLBFyEAN304wSQzaAqQfA4duVGFum6P-Q@mail.gmail.com>

On Thu, Jul 2, 2015 at 5:23 PM, Federico Schwindt <fgsch at lodoss.net> wrote:
>
> One of the things Dag pointed about this is that if you want to hash based on the client IP address you will need to do "" + client.ip.
> This is a valid concern although the only situation I can think of where client.identity will be missed.
>
> There are a few options:
>
> 1. Use "" + client.ip. Somewhat ugly but will do the job.

-1

> 2. Do automatic conversion for arguments when the expected type is STRING or STRING_LIST. See diff below.
> We do this already in a number of places but not for arguments.

+1

> 3. Stash it in the req.http space. e.g.
>
> set req.http.session = client.ip;
> set req.backend_hint = hash.backend(req.http.session);
> #unset req.http.session;

-1

Sounds like a useless extra step, compared to option 2

> 4. Change the hash director to use client.identity again. e.g. drop the STRING_LIST and use client.identity.

No, the 4.x hash director is a combination of 3.0 hash and client directors.

> 5. Do nothing.

-1

> As things are in master, client.identity lost it's place and it's more like a scratch space or throw away variable and feels like a special case but YMMV.
>
> Any opinions?
>
> f.-
>
> diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c
> index b9c6531..4019fec 100644
> --- a/lib/libvcc/vcc_expr.c
> +++ b/lib/libvcc/vcc_expr.c
> @@ -609,6 +609,9 @@ vcc_do_arg(struct vcc *tl, struct func_arg *fa)
>         } else {
>                 vcc_expr0(tl, &e2, fa->type);
>                 ERRCHK(tl);
> +               if (e2->fmt != fa->type &&
> +                   (fa->type == STRING || fa->type == STRING_LIST))
> +                       vcc_expr_tostring(tl, &e2, fa->type);
>                 if (e2->fmt != fa->type) {
>                         VSB_printf(tl->sb, "Wrong argument type.");
>                         VSB_printf(tl->sb, "  Expected %s.",
>
>
>
>
> On Thu, Jul 2, 2015 at 1:48 PM, Martin Blix Grydeland <martin at varnish-software.com> wrote:
>>
>> Yeah, this is OBE now.
>>
>> Martin
>>
>> On 1 July 2015 at 15:58, Dridi Boukelmoune <dridi at varni.sh> wrote:
>>>
>>> On Wed, Jul 1, 2015 at 3:05 PM, Federico Schwindt <fgsch at lodoss.net> wrote:
>>> > Hi,
>>> >
>>> > The hash director needs a string now so this no longer needed.
>>> >
>>> > Opinions?
>>>
>>> Agreed
>>>
>>> _______________________________________________
>>> varnish-dev mailing list
>>> varnish-dev at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>
>>
>>
>>
>> --
>> Martin Blix Grydeland
>> Senior Developer | Varnish Software AS
>> Mobile: +47 992 74 756
>> We Make Websites Fly!
>
>



From slink at schokola.de  Mon Jul  6 06:06:04 2015
From: slink at schokola.de (Nils Goroll)
Date: Mon, 06 Jul 2015 08:06:04 +0200
Subject: What is C in practice? (Cerberus survey): Analysis of Responses
Message-ID: <559A1ACC.8080608@schokola.de>

I just saw this on pgsql-hackers and think it may be of interest:
http://www.cl.cam.ac.uk/~pes20/cerberus/notes50-2015-05-24-survey-discussion.html



From dridi at varni.sh  Mon Jul  6 06:50:23 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 6 Jul 2015 08:50:23 +0200
Subject: Test suite for dynamic backends
Message-ID: <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg@mail.gmail.com>

Hi,

As requested, I have prepared a test suite for dynamic backends. This
patch set also contains API changes and bug fixes (see #1755).

Best Regards,
Dridi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Change-the-VRT-API-for-adding-deleting-backends.patch
Type: text/x-patch
Size: 4656 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150706/ade64be6/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Keep-track-of-backends-references-and-connections.patch
Type: text/x-patch
Size: 6968 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150706/ade64be6/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Add-a-debug.sleep-DURATION-synchronization-helper.patch
Type: text/x-patch
Size: 1207 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150706/ade64be6/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-Add-a-test-suite-for-dynamic-backends.patch
Type: text/x-patch
Size: 16289 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150706/ade64be6/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0005-Add-more-information-to-a-backend-panic-message.patch
Type: text/x-patch
Size: 1166 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150706/ade64be6/attachment-0004.bin>

From dridi at varni.sh  Mon Jul  6 08:21:27 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 6 Jul 2015 10:21:27 +0200
Subject: What is C in practice? (Cerberus survey): Analysis of Responses
In-Reply-To: <559A1ACC.8080608@schokola.de>
References: <559A1ACC.8080608@schokola.de>
Message-ID: <CABoVN9CgC0Ckap8UM1Z3GZAWprMRjG-chysRdBF9X+Ye_Z_NQQ@mail.gmail.com>

On Mon, Jul 6, 2015 at 8:06 AM, Nils Goroll <slink at schokola.de> wrote:
> I just saw this on pgsql-hackers and think it may be of interest:
> http://www.cl.cam.ac.uk/~pes20/cerberus/notes50-2015-05-24-survey-discussion.html

So, miniobj.h is undefined behaviour, huh.



From phk at phk.freebsd.dk  Mon Jul  6 09:25:56 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 06 Jul 2015 09:25:56 +0000
Subject: What is C in practice? (Cerberus survey): Analysis of Responses
In-Reply-To: <CABoVN9CgC0Ckap8UM1Z3GZAWprMRjG-chysRdBF9X+Ye_Z_NQQ@mail.gmail.com>
References: <559A1ACC.8080608@schokola.de>
 <CABoVN9CgC0Ckap8UM1Z3GZAWprMRjG-chysRdBF9X+Ye_Z_NQQ@mail.gmail.com>
Message-ID: <66185.1436174756@critter.freebsd.dk>

--------
In message <CABoVN9CgC0Ckap8UM1Z3GZAWprMRjG-chysRdBF9X+Ye_Z_NQQ at mail.gmail.com>, Dridi Boukelmoune writes:
>On Mon, Jul 6, 2015 at 8:06 AM, Nils Goroll <slink at schokola.de> wrote:
>> I just saw this on pgsql-hackers and think it may be of interest:
>> http://www.cl.cam.ac.uk/~pes20/cerberus/notes50-2015-05-24-survey-discussion.html
>
>So, miniobj.h is undefined behaviour, huh.

Uhm no ?

miniobj.h is always used on valid pointers and only cast to the
type the pointer is supposed to be.

Only if things have already gone horribly wrong would miniobj.h
risk doing something undefined.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Mon Jul  6 09:42:19 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 6 Jul 2015 11:42:19 +0200
Subject: What is C in practice? (Cerberus survey): Analysis of Responses
In-Reply-To: <66185.1436174756@critter.freebsd.dk>
References: <559A1ACC.8080608@schokola.de>
 <CABoVN9CgC0Ckap8UM1Z3GZAWprMRjG-chysRdBF9X+Ye_Z_NQQ@mail.gmail.com>
 <66185.1436174756@critter.freebsd.dk>
Message-ID: <CABoVN9AFyU0U3G5=KYZ+whtBO7SL+3yzE7UOPSYU2LvSqkPyFA@mail.gmail.com>

On Mon, Jul 6, 2015 at 11:25 AM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> --------
> In message <CABoVN9CgC0Ckap8UM1Z3GZAWprMRjG-chysRdBF9X+Ye_Z_NQQ at mail.gmail.com>, Dridi Boukelmoune writes:
>>So, miniobj.h is undefined behaviour, huh.
>
> Uhm no ?
>
> miniobj.h is always used on valid pointers and only cast to the
> type the pointer is supposed to be.
>
> Only if things have already gone horribly wrong would miniobj.h
> risk doing something undefined.

I was referring to question 13 (Null pointer representations)

One of the responses suggests that NULL does not imply (void*)0, which
miniobj.h assumes with memset.



From phk at phk.freebsd.dk  Mon Jul  6 10:12:32 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 06 Jul 2015 10:12:32 +0000
Subject: What is C in practice? (Cerberus survey): Analysis of Responses
In-Reply-To: <CABoVN9AFyU0U3G5=KYZ+whtBO7SL+3yzE7UOPSYU2LvSqkPyFA@mail.gmail.com>
References: <559A1ACC.8080608@schokola.de>
 <CABoVN9CgC0Ckap8UM1Z3GZAWprMRjG-chysRdBF9X+Ye_Z_NQQ@mail.gmail.com>
 <66185.1436174756@critter.freebsd.dk>
 <CABoVN9AFyU0U3G5=KYZ+whtBO7SL+3yzE7UOPSYU2LvSqkPyFA@mail.gmail.com>
Message-ID: <66742.1436177552@critter.freebsd.dk>

--------
In message <CABoVN9AFyU0U3G5=KYZ+whtBO7SL+3yzE7UOPSYU2LvSqkPyFA at mail.gmail.com>, Dridi Boukelmoune writes:

>I was referring to question 13 (Null pointer representations)
>
>One of the responses suggests that NULL does not imply (void*)0, which
>miniobj.h assumes with memset.

All the kernels we support have the same assumption, so I'm not worried.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From arianna.aondio at varnish-software.com  Tue Jul  7 13:40:46 2015
From: arianna.aondio at varnish-software.com (Arianna Aondio)
Date: Tue, 7 Jul 2015 15:40:46 +0200
Subject: rtstatus backend counters.
Message-ID: <CAL6a+5hUZwyCbVAG-jhF1QtAXhS=G+7CTY_rUZy=JR79jDkGVA@mail.gmail.com>

Hi everyone,
some time ago I wrote a vmod called
libvmod-rtstatus(https://github.com/varnish/libvmod-rtstatus).
this vmod has two functions:
1- rtstatus(delta): creates a json object containing different Varnish counters.
2- html(): this function is just a nice interface for the user, it
uses the json object provided by the other function and shows them in
a "nice" way (https://github.com/varnish/libvmod-rtstatus/blob/4.0/varnishstatus.png).
In function 2 some math is done on counters related to backends.

Now, I think having function 1 as part of std_vmod is a good idea,
having an overview on what is happening on backends can be very
important for understanding how Varnish works and debugging various
situations.
The json file collect all the counters we have in varnishstat, but
there are also new counters, such as:
*hitrate and load calculated on a delta time decided by the user.
*total amount of KB and KB/s sent to backends.
*total amount of KB and KB/s received from backends.
*piped KB.
*health of backends.
etc,etc...

Please note I'd like to move the math from function 2 to function 1 so
the json object will have counters and numbers ready to be used and
analyzed , if some users feel like using a nicer interface than the
raw json object they can write their own html/javascript or just use
the default one I provide on my github page, but that will be a
different vmod and a different story.

Comments much appreciated.


Arianna Aondio
Software Developer | Varnish Software AS
Mobile: +47 98062619
We Make Websites Fly
www.varnish-software.com



From phk at phk.freebsd.dk  Tue Jul  7 22:55:44 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 07 Jul 2015 22:55:44 +0000
Subject: Test suite for dynamic backends
In-Reply-To: <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg@mail.gmail.com>
References: <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg@mail.gmail.com>
Message-ID: <94731.1436309744@critter.freebsd.dk>

--------
In message <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg at mail.gmail.com>
, Dridi Boukelmoune writes:

>As requested, I have prepared a test suite for dynamic backends. This
>patch set also contains API changes and bug fixes (see #1755).

>Subject: [PATCH 1/5] Change the VRT API for adding/deleting backends

> A VRT_CTX may not be available in all situations for backends deletion.

They will have to be.  If need be, the caller must construct one.

> Adding a backend to a warm VCL needs a proper initialization of the
> backend, through an event.

Fixed slightly simpler than in your patch.

> Events can now be sent outside of the CLI thread for dynamically-created
> backends.

Also fixed.

>Subject: [PATCH 2/5] Keep track of backends references and connections

I've done this entirely differently.

Fixes #1755

> [PATCH 3/5] Add a debug.sleep(DURATION) synchronization helper

>+	sleep((unsigned int)t);

This should use VTIM_sleep().

Commit it when fixed & tested.

> Subject: [PATCH 4/5] Add a test suite for dynamic backends

I've run out of steam, will look at this tomorrow.

> Subject: [PATCH 5/5] Add more information to a backend panic message

This doesn't work with the way I did #2/5 above.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From phk at phk.freebsd.dk  Tue Jul  7 23:01:51 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 07 Jul 2015 23:01:51 +0000
Subject: rtstatus backend counters.
In-Reply-To: <CAL6a+5hUZwyCbVAG-jhF1QtAXhS=G+7CTY_rUZy=JR79jDkGVA@mail.gmail.com>
References: <CAL6a+5hUZwyCbVAG-jhF1QtAXhS=G+7CTY_rUZy=JR79jDkGVA@mail.gmail.com>
Message-ID: <8024.1436310111@critter.freebsd.dk>

--------
In message <CAL6a+5hUZwyCbVAG-jhF1QtAXhS=G+7CTY_rUZy=JR79jDkGVA at mail.gmail.com>, Arianna Aondio writes:

>Now, I think having function 1 as part of std_vmod is a good idea,

I don't think this belongs in std_vmod, it is a well defined
specialised type of functionality and that naturally belongs in
its own vmod I think.

If somebody feels adventurous it could be a candidate for
vmod implemented VDP-based synthetic object generator.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Wed Jul  8 09:44:24 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 8 Jul 2015 11:44:24 +0200
Subject: Test suite for dynamic backends
In-Reply-To: <94731.1436309744@critter.freebsd.dk>
References: <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg@mail.gmail.com>
 <94731.1436309744@critter.freebsd.dk>
Message-ID: <CABoVN9CP=YzV73D5vhu-5raCuuFSy1Ad18m-ZqAmhKjakVBetA@mail.gmail.com>

On Wed, Jul 8, 2015 at 12:55 AM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
>> A VRT_CTX may not be available in all situations for backends deletion.
>
> They will have to be.  If need be, the caller must construct one.

I didn't think it was OK to make my own context, thanks.

>>Subject: [PATCH 2/5] Keep track of backends references and connections
>
> I've done this entirely differently.
>
> Fixes #1755

Please revisit this one, my patch also fixes two panics I just got while
rebasing against the current trunk. See attached test logs.

>> [PATCH 3/5] Add a debug.sleep(DURATION) synchronization helper
> This should use VTIM_sleep().
>
> Commit it when fixed & tested.

Ack

>> Subject: [PATCH 4/5] Add a test suite for dynamic backends
>
> I've run out of steam, will look at this tomorrow.

Thanks

>> Subject: [PATCH 5/5] Add more information to a backend panic message
>
> This doesn't work with the way I did #2/5 above.

Do I send a patch or simply commit?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: d00008.log
Type: text/x-log
Size: 18887 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150708/a5f3ca8d/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: d00009.log
Type: text/x-log
Size: 13475 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150708/a5f3ca8d/attachment-0001.bin>

From phk at phk.freebsd.dk  Wed Jul  8 21:11:54 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 08 Jul 2015 21:11:54 +0000
Subject: Test suite for dynamic backends
In-Reply-To: <CABoVN9CP=YzV73D5vhu-5raCuuFSy1Ad18m-ZqAmhKjakVBetA@mail.gmail.com>
References: <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg@mail.gmail.com>
 <94731.1436309744@critter.freebsd.dk>
 <CABoVN9CP=YzV73D5vhu-5raCuuFSy1Ad18m-ZqAmhKjakVBetA@mail.gmail.com>
Message-ID: <49423.1436389914@critter.freebsd.dk>

--------
In message <CABoVN9CP=YzV73D5vhu-5raCuuFSy1Ad18m-ZqAmhKjakVBetA at mail.gmail.com>, Dridi Boukelmoune writes:

>>>Subject: [PATCH 2/5] Keep track of backends references and connections
>>
>> I've done this entirely differently.
>>
>> Fixes #1755
>
>Please revisit this one, my patch also fixes two panics I just got while
>rebasing against the current trunk. See attached test logs.

I tried releaseing the lock as we talked about on IRC and that solves
the problem.  Committed.

>>> Subject: [PATCH 5/5] Add more information to a backend panic message
>>
>> This doesn't work with the way I did #2/5 above.
>
>Do I send a patch or simply commit?

Just commit it.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Thu Jul  9 00:25:28 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 9 Jul 2015 02:25:28 +0200
Subject: Test suite for dynamic backends
In-Reply-To: <49423.1436389914@critter.freebsd.dk>
References: <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg@mail.gmail.com>
 <94731.1436309744@critter.freebsd.dk>
 <CABoVN9CP=YzV73D5vhu-5raCuuFSy1Ad18m-ZqAmhKjakVBetA@mail.gmail.com>
 <49423.1436389914@critter.freebsd.dk>
Message-ID: <CABoVN9AK928Di9T4Vy0-3vxzGikUjH+_J5JNj-JTFTac=bvFLA@mail.gmail.com>

On Wed, Jul 8, 2015 at 11:11 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> --------
> In message <CABoVN9CP=YzV73D5vhu-5raCuuFSy1Ad18m-ZqAmhKjakVBetA at mail.gmail.com>, Dridi Boukelmoune writes:
>
>>>>Subject: [PATCH 2/5] Keep track of backends references and connections
>>>
>>> I've done this entirely differently.
>>>
>>> Fixes #1755
>>
>>Please revisit this one, my patch also fixes two panics I just got while
>>rebasing against the current trunk. See attached test logs.
>
> I tried releaseing the lock as we talked about on IRC and that solves
> the problem.  Committed.

Ack.

However, I see a flaw in the VBE_Poll implementation. If you have
deleted backends with a zero n_conn, but the first one is still being
used, nothing's deleted. It makes it less predictable than my
(convoluted) delete-when-deletable approach. It doesn't hurt the test
suite, which *should be* predictable anyway. And I have updated the
test suite to trigger a VBE_Poll before assertions on the n_backend
counter.

>>>> Subject: [PATCH 5/5] Add more information to a backend panic message
>>>
>>> This doesn't work with the way I did #2/5 above.
>>
>>Do I send a patch or simply commit?
>
> Just commit it.

Done.

I have attached a new patch set which would have been ready before
my rehearsal if it weren't for a last-minute phone call (let's call the
culprit Dave). I have rearranged the patches against the current
master hoping that they'll be reviewed this week.

The first patch is my initial fix for 1755 and dynamic-backends-related
issues, reworked after the recent changes in master (probe "deleted"
and VBE_Poll). I rely on refcount to avoid dangling struct backend
pointers, which can now happen with dynamic backends.

The second patch is the actual test suite.

The last two bonus patches are a first stab at a DNS director for 4.1,
but requests fail with IPv6 backends.

Cheers,
Dridi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Keep-track-of-backends-references-and-connections.patch
Type: text/x-patch
Size: 4768 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/98a6fac5/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Add-a-test-suite-for-dynamic-backends.patch
Type: text/x-patch
Size: 16382 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/98a6fac5/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Resurrect-VSA_Len.patch
Type: text/x-patch
Size: 1489 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/98a6fac5/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-Resurrect-the-DNS-director.patch
Type: text/x-patch
Size: 12080 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/98a6fac5/attachment-0003.bin>

From arianna.aondio at varnish-software.com  Thu Jul  9 12:05:01 2015
From: arianna.aondio at varnish-software.com (Arianna Aondio)
Date: Thu, 9 Jul 2015 14:05:01 +0200
Subject: rtstatus backend counters.
Message-ID: <CAL6a+5iEu96C8s5248uKCDwJFtDvKwJSzyG7M3kM59J7CMJd4Q@mail.gmail.com>

>>Now, I think having function 1 as part of std_vmod is a good idea,

>I don't think this belongs in std_vmod, it is a well defined
>specialised type of functionality and that naturally belongs in
>its own vmod I think.

With "own vmod" do you mean a vmod that can be part of the main
distribution or a vmod that doesn't live in Varnish at all?

>If somebody feels adventurous it could be a candidate for
>vmod implemented VDP-based synthetic object generator.

I'm sorry,but I don't really understand this statement.
I don't get how involving VDP could simplify or improve such a vmod.

-- 
Arianna Aondio
Software Developer | Varnish Software AS
Mobile: +47 98062619
We Make Websites Fly
www.varnish-software.com



From phk at phk.freebsd.dk  Thu Jul  9 14:48:30 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 09 Jul 2015 14:48:30 +0000
Subject: rtstatus backend counters.
In-Reply-To: <CAL6a+5iEu96C8s5248uKCDwJFtDvKwJSzyG7M3kM59J7CMJd4Q@mail.gmail.com>
References: <CAL6a+5iEu96C8s5248uKCDwJFtDvKwJSzyG7M3kM59J7CMJd4Q@mail.gmail.com>
Message-ID: <53357.1436453310@critter.freebsd.dk>

--------
In message <CAL6a+5iEu96C8s5248uKCDwJFtDvKwJSzyG7M3kM59J7CMJd4Q at mail.gmail.com>
, Arianna Aondio writes:
>>>Now, I think having function 1 as part of std_vmod is a good idea,
>
>>I don't think this belongs in std_vmod, it is a well defined
>>specialised type of functionality and that naturally belongs in
>>its own vmod I think.
>
>With "own vmod" do you mean a vmod that can be part of the main
>distribution or a vmod that doesn't live in Varnish at all?

A vmod separate from vmod_std.

And it can be part of the varnish dist if we decide it is
sufficiently popular/useful/quality etc. etc.

>>If somebody feels adventurous it could be a candidate for
>>vmod implemented VDP-based synthetic object generator.
>
>I'm sorry,but I don't really understand this statement.
>I don't get how involving VDP could simplify or improve such a vmod.

I didn't say it would simplify or improve the vmod.

The main benefit would be that you wouldn't have to
buffer the output, but could produce it incrementally.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From martin at varnish-software.com  Thu Jul  9 15:32:18 2015
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 9 Jul 2015 17:32:18 +0200
Subject: struct http_conn usage changes
Message-ID: <CANTn4crN63L4HDECdoYN=ArVza6EpXn7LhPWZBLP=Y09L+bOLw@mail.gmail.com>

Hi PHK,

Attached is a set of patches to fix up the connection handling in Varnish,
making it more general for both client and backend connections. This is a
fairly benign cleanup set, that makes the code paths clearer. I propose
this for inclusion in 4.1 (and with very small font, almost unreadable, I
mention that it also sneaks in the necessary hooks for my Backend SSL
stuff).

Martin

-- 
<http://varnish-software.com>*Martin Blix Grydeland*
Senior Developer | Varnish Software AS
Mobile: +47 992 74 756
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/63e2af85/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Move-the-HTTP-request-receive-code-away-from-cache_s.patch
Type: text/x-patch
Size: 14866 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/63e2af85/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Remove-unused-wrk-argument-from-HTC_RxReq.patch
Type: text/x-patch
Size: 2736 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/63e2af85/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Create-a-HTC_RxResp-function-to-receive-backend-resp.patch
Type: text/x-patch
Size: 4973 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/63e2af85/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-Make-HTC_Rx-static.patch
Type: text/x-patch
Size: 3780 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/63e2af85/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0005-Make-struct-http_conn-have-an-int-pointer-to-the-fd.patch
Type: text/x-patch
Size: 6502 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/63e2af85/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0006-Make-V1L_Reserve-take-a-pointer-to-a-struct-http_con.patch
Type: text/x-patch
Size: 5123 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/63e2af85/attachment-0005.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0007-Add-pluggable-read-write-function-pointers-to-struct.patch
Type: text/x-patch
Size: 6325 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/63e2af85/attachment-0006.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0008-Update-pipe-to-use-struct-http_conn.patch
Type: text/x-patch
Size: 4933 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/63e2af85/attachment-0007.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0009-Make-pan_htc-more-useful.patch
Type: text/x-patch
Size: 2681 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/63e2af85/attachment-0008.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0010-Add-a-panic-operation-to-htc-for-custom-panic-messag.patch
Type: text/x-patch
Size: 1505 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150709/63e2af85/attachment-0009.bin>

From dridi at varni.sh  Fri Jul 10 13:42:37 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 10 Jul 2015 15:42:37 +0200
Subject: Test suite for dynamic backends
In-Reply-To: <CABoVN9AK928Di9T4Vy0-3vxzGikUjH+_J5JNj-JTFTac=bvFLA@mail.gmail.com>
References: <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg@mail.gmail.com>
 <94731.1436309744@critter.freebsd.dk>
 <CABoVN9CP=YzV73D5vhu-5raCuuFSy1Ad18m-ZqAmhKjakVBetA@mail.gmail.com>
 <49423.1436389914@critter.freebsd.dk>
 <CABoVN9AK928Di9T4Vy0-3vxzGikUjH+_J5JNj-JTFTac=bvFLA@mail.gmail.com>
Message-ID: <CABoVN9BxwdRFd7SZWZ-Yc18WYcTz5p7eUx8743dAOuoJ+3WZ2Q@mail.gmail.com>

On Thu, Jul 9, 2015 at 2:25 AM, Dridi Boukelmoune <dridi at varni.sh> wrote:
> I have attached a new patch set...

Just gave it a try, all the arrows point to the right :)



From phk at phk.freebsd.dk  Sun Jul 12 09:20:30 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 12 Jul 2015 09:20:30 +0000
Subject: Test suite for dynamic backends
In-Reply-To: <CABoVN9AK928Di9T4Vy0-3vxzGikUjH+_J5JNj-JTFTac=bvFLA@mail.gmail.com>
References: <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg@mail.gmail.com>
 <94731.1436309744@critter.freebsd.dk>
 <CABoVN9CP=YzV73D5vhu-5raCuuFSy1Ad18m-ZqAmhKjakVBetA@mail.gmail.com>
 <49423.1436389914@critter.freebsd.dk>
 <CABoVN9AK928Di9T4Vy0-3vxzGikUjH+_J5JNj-JTFTac=bvFLA@mail.gmail.com>
Message-ID: <3471.1436692830@critter.freebsd.dk>

--------

I have finally given up on adding "interrim" refcounting of VCL backends.

The .resolve() trick you had implemented to narrow down the race
condition did only that, it didn't completely close it, and I could
provoke it in heavy-load synthetic tests.

Instead I have simply given dynamic backends a 60 second cool-off
before we nuke them.

This is only a stop-gap measure, and in Varnish 5 we will need to
bite the bullet and do proper refcounting on all non-atomic VCL
types.

But for now, dynamic backends should work for 4.1, but VMODS should
take care to not needlessly replace dynamic backends with identical
new ones (as vmod_debug does intentionally).

Thanks for your heroic effort Dridi,  Good work!

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dridi at varni.sh  Sun Jul 12 10:00:51 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Sun, 12 Jul 2015 12:00:51 +0200
Subject: Test suite for dynamic backends
In-Reply-To: <3471.1436692830@critter.freebsd.dk>
References: <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg@mail.gmail.com>
 <94731.1436309744@critter.freebsd.dk>
 <CABoVN9CP=YzV73D5vhu-5raCuuFSy1Ad18m-ZqAmhKjakVBetA@mail.gmail.com>
 <49423.1436389914@critter.freebsd.dk>
 <CABoVN9AK928Di9T4Vy0-3vxzGikUjH+_J5JNj-JTFTac=bvFLA@mail.gmail.com>
 <3471.1436692830@critter.freebsd.dk>
Message-ID: <CABoVN9D7u3FEztctjvtJmkY_J5h4tyfg_-J9ON+qXSEG+5pqQg@mail.gmail.com>

On Sun, Jul 12, 2015 at 11:20 AM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> --------
>
> I have finally given up on adding "interrim" refcounting of VCL backends.
>
> The .resolve() trick you had implemented to narrow down the race
> condition did only that, it didn't completely close it, and I could
> provoke it in heavy-load synthetic tests.

Damn, I fixed all the races I could see/prove, but it wasn't enough.

> Instead I have simply given dynamic backends a 60 second cool-off
> before we nuke them.
>
> This is only a stop-gap measure, and in Varnish 5 we will need to
> bite the bullet and do proper refcounting on all non-atomic VCL
> types.

Speaking of Varnish 5, I wish to expose VCL probes and ACLs to VMODs
in 4.1, the missing pieces for dynamic backends. The code is in
patchwork, but I won't be around to support it before the release. So
hopefully in 4.1 or 4.1.1 (same for the DNS director).

> But for now, dynamic backends should work for 4.1, but VMODS should
> take care to not needlessly replace dynamic backends with identical
> new ones (as vmod_debug does intentionally).

If you look at the DNS director patch, you'll see extra care in not
removing backends needlessly.

> Thanks for your heroic effort Dridi,  Good work!

Thanks for the review on a weekend, and for your patience.

I will be back in August, and I intend to write proper documentation
on directors (with static, dynamic, or vmod backends) if that can wait
until then. I also intend to write a test suite for vmod backends, I'm
thinking about a "stupid" director doing TCP connections without any
kind of pooling.



From phk at phk.freebsd.dk  Sun Jul 12 10:09:20 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 12 Jul 2015 10:09:20 +0000
Subject: Test suite for dynamic backends
In-Reply-To: <CABoVN9D7u3FEztctjvtJmkY_J5h4tyfg_-J9ON+qXSEG+5pqQg@mail.gmail.com>
References: <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg@mail.gmail.com>
 <94731.1436309744@critter.freebsd.dk>
 <CABoVN9CP=YzV73D5vhu-5raCuuFSy1Ad18m-ZqAmhKjakVBetA@mail.gmail.com>
 <49423.1436389914@critter.freebsd.dk>
 <CABoVN9AK928Di9T4Vy0-3vxzGikUjH+_J5JNj-JTFTac=bvFLA@mail.gmail.com>
 <3471.1436692830@critter.freebsd.dk>
 <CABoVN9D7u3FEztctjvtJmkY_J5h4tyfg_-J9ON+qXSEG+5pqQg@mail.gmail.com>
Message-ID: <14720.1436695760@critter.freebsd.dk>

--------
In message <CABoVN9D7u3FEztctjvtJmkY_J5h4tyfg_-J9ON+qXSEG+5pqQg at mail.gmail.com>
, Dridi Boukelmoune writes:

>Speaking of Varnish 5, I wish to expose VCL probes and ACLs to VMODs
>in 4.1, the missing pieces for dynamic backends. The code is in
>patchwork, but I won't be around to support it before the release. So
>hopefully in 4.1 or 4.1.1 (same for the DNS director).

As long as we only talk static ACLs and probes, refcounting is
a non-issue, and I'll get to those patches after I look at martins
panic patches.

>> But for now, dynamic backends should work for 4.1, but VMODS should
>> take care to not needlessly replace dynamic backends with identical
>> new ones (as vmod_debug does intentionally).
>
>If you look at the DNS director patch, you'll see extra care in not
>removing backends needlessly.

I havn't looked at that patch yet.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From slink at schokola.de  Sun Jul 12 21:12:21 2015
From: slink at schokola.de (Nils Goroll)
Date: Sun, 12 Jul 2015 23:12:21 +0200
Subject: [PATCH 1/2] Add a feature to always keep the vsm file
Message-ID: <55A2D835.3080905@schokola.de>

A non-text attachment was scrubbed...
Name: 0001-Add-a-feature-to-always-keep-the-vsm-file.patch
Type: text/x-patch
Size: 1288 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150712/66eeeab3/attachment.bin>

From slink at schokola.de  Sun Jul 12 21:12:43 2015
From: slink at schokola.de (Nils Goroll)
Date: Sun, 12 Jul 2015 23:12:43 +0200
Subject: [PATCH 2/2] use feature=+vsm_keep with leave_temp (varnishtest -L
 /, -l options)
Message-ID: <55A2D84B.6040406@schokola.de>

A non-text attachment was scrubbed...
Name: 0002-use-feature-vsm_keep-with-leave_temp-varnishtest-L-l.patch
Type: text/x-patch
Size: 1562 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150712/614e8664/attachment.bin>

From fgsch at lodoss.net  Mon Jul 13 18:35:15 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Mon, 13 Jul 2015 19:35:15 +0100
Subject: Counters in master
Message-ID: <CAJV_h0aUvxrL5=0zH=RUS_hBdGtJjr=FNTU9qsBtN2V2-KJNcA@mail.gmail.com>

Hi,

Below is a summary of counters removed and renamed in master wrt 4.0 and
IMO what we need to bring back for 4.1. Unless specifically mentioned they
are relative to MAIN.

Removed:

backend_toolate - IIUC cannot happen anymore
client_req_413 - unused in 4.0
fetch_close
fetch_oldhttp - used to indicate backends < 1.1
VBE.<backend exp>.vcls - no longer needed

Renamed:

fetch_none (was fetch_zero)
sc_pipe_overflow (was sess_pipe_overflow)
rx_body (was client_req_411)

Broken:

backend_conn is never incremented (#1725)

To be in parity with 4.0 we should address #1725.
Not sure if there is any value in fetch_oldhttp or fetch_close.

As for counters we might want to add, several people have mentioned the
need for per-backend errors. There are 3 places in cache_backend.c that
have the "XXX: Per backend stats ?" comment that we can start with and what
the attached patch covers.
That said, this might be post 4.1 material.

Haven't looked at #1725 yet.

f.-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150713/03ed4b7c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Add-some-error-counters-to-VBE.patch
Type: text/x-patch
Size: 4983 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150713/03ed4b7c/attachment.bin>

From fgsch at lodoss.net  Mon Jul 13 18:42:00 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Mon, 13 Jul 2015 19:42:00 +0100
Subject: [PATCH] Rearrange so it's lock-protected
Message-ID: <CAJV_h0bUBTYeguFpiA8GmVKApTN27cJy-6Q2tYEb0v8ep=r-jg@mail.gmail.com>

Based on the discussion on irc.

f.-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150713/ca6cdddd/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Rearrange-so-it-s-lock-protected.patch
Type: text/x-patch
Size: 1084 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150713/ca6cdddd/attachment.bin>

From dridi at varni.sh  Mon Jul 13 20:59:11 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 13 Jul 2015 22:59:11 +0200
Subject: [PATCH] Rearrange so it's lock-protected
In-Reply-To: <CAJV_h0bUBTYeguFpiA8GmVKApTN27cJy-6Q2tYEb0v8ep=r-jg@mail.gmail.com>
References: <CAJV_h0bUBTYeguFpiA8GmVKApTN27cJy-6Q2tYEb0v8ep=r-jg@mail.gmail.com>
Message-ID: <CABoVN9AjLJp+XEA5RMjxbJGt3PY_beA+epN6+OPbQ49mdY5ToQ@mail.gmail.com>

I have actually seen problems with this counter under modest load
today, but the patch doesn't solve it.

Now that backends share tcp pools, the req and conn counters are
basically the same (conn could be turned into a gauge or be removed).

On Mon, Jul 13, 2015 at 8:42 PM, Federico Schwindt <fgsch at lodoss.net> wrote:
> Based on the discussion on irc.
>
> f.-
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev



From dridi at varni.sh  Tue Jul 14 07:34:15 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 14 Jul 2015 09:34:15 +0200
Subject: Test suite for dynamic backends
In-Reply-To: <14720.1436695760@critter.freebsd.dk>
References: <CABoVN9C3U444oYsO2yHfFm-Kr9d3OhTBexSO=bkmPUFcZX7smg@mail.gmail.com>
 <94731.1436309744@critter.freebsd.dk>
 <CABoVN9CP=YzV73D5vhu-5raCuuFSy1Ad18m-ZqAmhKjakVBetA@mail.gmail.com>
 <49423.1436389914@critter.freebsd.dk>
 <CABoVN9AK928Di9T4Vy0-3vxzGikUjH+_J5JNj-JTFTac=bvFLA@mail.gmail.com>
 <3471.1436692830@critter.freebsd.dk>
 <CABoVN9D7u3FEztctjvtJmkY_J5h4tyfg_-J9ON+qXSEG+5pqQg@mail.gmail.com>
 <14720.1436695760@critter.freebsd.dk>
Message-ID: <CABoVN9AaUWoNcL4E1Jw=0NvS-P0-=Ds61d52ZXomYuZ+Xrv5eA@mail.gmail.com>

On Sun, Jul 12, 2015 at 12:09 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
>>If you look at the DNS director patch, you'll see extra care in not
>>removing backends needlessly.
>
> I havn't looked at that patch yet.

Forgot to update my DNS director draft. I also realized thanks to
telnet that my ISP doesn't seem to support IPv6 (Network is
unreachable), but loopback tests went well.

Cheers,
Dridi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Resurrect-VSA_Len.patch
Type: text/x-patch
Size: 1489 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150714/cf7ca3ba/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Resurrect-the-DNS-director.patch
Type: text/x-patch
Size: 12156 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150714/cf7ca3ba/attachment-0001.bin>

From fgsch at lodoss.net  Wed Jul 15 18:07:23 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Wed, 15 Jul 2015 19:07:23 +0100
Subject: [PATCH] Variable for vcl_dir in startup scripts
In-Reply-To: <5F530A9242E7F84F999DB40E0E268FBD4B0EB0F2@mercalli.lild01.pictime.fr>
References: <5F530A9242E7F84F999DB40E0E268FBD4B0EB01D@mercalli.lild01.pictime.fr>
 <CAJV_h0Zy_=Tub7x7Rm=HJgiA2v83ACoVCDo_Jc1F-xf2+gNK0A@mail.gmail.com>
 <5F530A9242E7F84F999DB40E0E268FBD4B0EB0F2@mercalli.lild01.pictime.fr>
Message-ID: <CAJV_h0afV1wJXtt2-TzVtfCHBG3te1XKc2BEMTC15_y2Vmu49Q@mail.gmail.com>

Hi Gauthier,

Apologies for the delay.

In all fairness I copied this from the Debian systemd files [1] in an
attempt to unify the service files.
I don't see any particular reason for having this, specially if it's the
source of issues.

Stig, is there any reason for the ExecStartPre? The commit message says
"debian/varnish.service: Test configuration before starting, add reload".
I don't see any difference in the output for an invalid config with
ExecStartPre when I run either `systemctl start' or `systemctl status'.

Thanks.

1. http://anonscm.debian.org/cgit/pkg-varnish/pkg-varnish.git/tree/debian


On Thu, Jun 25, 2015 at 5:12 PM, Delacroix, Gauthier <
Gauthier.Delacroix at coreye.fr> wrote:

> Federico,
>
> I've initially set vcl_dir in DAEMON_OPTS in varnish.params with 4.0.2,
> but it's not included in the syntax check added in 4.0.3, making my config
> unable to start without changing varnish.service.
>
> I'm not aware of the reason behind this syntax check (won't a syntax error
> make startup fail anyway ?) but I assumed you had a good reason not to
> include DAEMON_OPTS.
>
> As it's better to change varnish.params rather than varnish.service, my
> proposal was to add a way to include parameters needed to make the config
> work in the syntax check, without including the full DAEMON_OPTS.
>
> If you finally think DAEMON_OPTS can be included in the syntax check, then
> I can send a really smaller patch.
>
> Gauthier
>
> De : Federico Schwindt [mailto:fgsch at lodoss.net]
> Envoy? : jeudi 25 juin 2015 17:53
> ? : Delacroix, Gauthier
> Cc : varnish-dev at varnish-cache.org
> Objet : Re: [PATCH] Variable for vcl_dir in startup scripts
>
> Hi,
>
> My first inclination is to ensure that the ExecStartPre line uses the same
> parameters as ExecStart.
>
> So my questions is how do you set vcl_dir? Do you edit varnish.service or
> varnish.params?
>
> Wouldn't adding DAEMON_OPTS to ExecStartPre (and configtest) do it?
>
> On Thu, Jun 25, 2015 at 3:26 PM, Delacroix, Gauthier <
> Gauthier.Delacroix at coreye.fr> wrote:
> Here is another patch proposal to make syntax check handle parameters
> required to compile the VCL (vcl_dir, etc.) without creating a startup
> variable for each parameter.
>
> It just adds a COMPILE_OPTS that is merged in DAEMON_OPTS to start Varnish
> but is used alone in the syntax check.
>
> Gauthier
>
> From d1567a956d53a489aa4ace66ce0b1c1ef745570b Mon Sep 17 00:00:00 2001
> From: Gauthier Delacroix <gauthier.delacroix at coreye.fr>
> Date: Thu, 25 Jun 2015 15:06:08 +0200
> Subject: [PATCH] Add COMPILE_OPTS in startup scripts to make syntax check
>  check handle compilation parameters
>
> ---
>  redhat/varnish.initrc    |  6 ++++--
>  redhat/varnish.params    |  8 ++++++++
>  redhat/varnish.service   |  3 ++-
>  redhat/varnish.sysconfig | 11 ++++++++++-
>  4 files changed, 24 insertions(+), 4 deletions(-)
>
> diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc
> index 117e334..0bde074 100755
> --- a/redhat/varnish.initrc
> +++ b/redhat/varnish.initrc
> @@ -126,9 +126,11 @@ rh_status_q() {
>
>  configtest() {
>      if [ -f "$VARNISH_VCL_CONF" ]; then
> -        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp > /dev/null && echo
> "Syntax ok"
> +        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp $COMPILE_OPTS > /dev/null
> \
> +          && echo "Syntax ok"
>      else
> -       echo "VARNISH_VCL_CONF is  unset or does not point to a file"
> +       echo "VARNISH_VCL_CONF is unset or does not point to a file"
> +       echo "Also check that COMPILE_OPTS is set depending on the VCL
> config"
>      fi
>  }
>
> diff --git a/redhat/varnish.params b/redhat/varnish.params
> index 27a14dd..970d088 100644
> --- a/redhat/varnish.params
> +++ b/redhat/varnish.params
> @@ -31,5 +31,13 @@ VARNISH_TTL=120
>  VARNISH_USER=varnish
>  VARNISH_GROUP=varnish
>
> +# Startup options required to compile the configuration.
> +# The following run-time parameters must be defined here, if needed:
> +# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref,
> vcc_unsafe_path,
> +# vcl_dir, vmod_dir
> +# Defining them in DAEMON_OPTS may result in a syntax check failure.
> +# See the man page varnishd(1).
> +#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
> +
>  # Other options, see the man page varnishd(1)
>  #DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p
> thread_pool_timeout=300"
> diff --git a/redhat/varnish.service b/redhat/varnish.service
> index a4f3355..a08db58 100644
> --- a/redhat/varnish.service
> +++ b/redhat/varnish.service
> @@ -27,7 +27,7 @@ EnvironmentFile=/etc/varnish/varnish.params
>  Type=forking
>  PIDFile=/var/run/varnish.pid
>  PrivateTmp=true
> -ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF
> +ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF $COMPILE_OPTS
>  ExecStart=/usr/sbin/varnishd \
>         -P /var/run/varnish.pid \
>         -f $VARNISH_VCL_CONF \
> @@ -38,6 +38,7 @@ ExecStart=/usr/sbin/varnishd \
>         -g $VARNISH_GROUP \
>         -S $VARNISH_SECRET_FILE \
>         -s $VARNISH_STORAGE \
> +       $COMPILE_OPTS \
>         $DAEMON_OPTS
>
>  ExecReload=/usr/sbin/varnish_reload_vcl
> diff --git a/redhat/varnish.sysconfig b/redhat/varnish.sysconfig
> index 6aa2354..0e376ff 100644
> --- a/redhat/varnish.sysconfig
> +++ b/redhat/varnish.sysconfig
> @@ -91,6 +91,14 @@ VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
>  # # Default TTL used when the backend does not specify one
>  VARNISH_TTL=120
>  #
> +# Startup options required to compile the configuration.
> +# The following run-time parameters must be defined here, if needed:
> +# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref,
> vcc_unsafe_path,
> +# vcl_dir, vmod_dir
> +# Defining them in DAEMON_OPTS may result in a syntax check failure.
> +# See the man page varnishd(1).
> +#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
> +#
>  # # DAEMON_OPTS is used by the init script.  If you add or remove
> options, make
>  # # sure you update this section, too.
>  DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
> @@ -102,7 +110,8 @@ DAEMON_OPTS="-a
> ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
>               -p thread_pool_timeout=${VARNISH_THREAD_TIMEOUT} \
>               -u varnish -g varnish \
>               -S ${VARNISH_SECRET_FILE} \
> -             -s ${VARNISH_STORAGE}"
> +             -s ${VARNISH_STORAGE}" \
> +             ${COMPILE_OPTS}
>  #
>
>
> --
> 1.8.3.msysgit.0
>
>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150715/cd2bcfb1/attachment.html>

From ksorensen at nordija.com  Wed Jul 15 21:08:30 2015
From: ksorensen at nordija.com (Kristian =?ISO-8859-1?Q?Gr=F8nfeldt_S=F8rensen?=)
Date: Wed, 15 Jul 2015 23:08:30 +0200
Subject: [PATCH] Allow varnishncsa logformat to be read from a file
In-Reply-To: <5492D5D7.3030503@schokola.de>
References: <1418682312.5583.9.camel@nordija.com>
 <5492D5D7.3030503@schokola.de>
Message-ID: <1436994510.13789.0.camel@nordija.com>

On Thu, 2014-12-18 at 14:25 +0100, Nils Goroll wrote:
> looks ok to me, I will double check  during the next bugwash I 
> participate in.

Any chance of getting this in to 4.1?

/Kristian



From ruben at varnish-software.com  Fri Jul 24 23:44:14 2015
From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=)
Date: Sat, 25 Jul 2015 01:44:14 +0200
Subject: Planning VDD15Q3
In-Reply-To: <20150616123712.GA31003@immer.varnish-software.com>
References: <20150608143928.GB4317@immer.varnish-software.com>
 <20150616123712.GA31003@immer.varnish-software.com>
Message-ID: <CAEfnkDbUJEWZwkS=rVET_mVix+qx6f8EXCBQ+SQwjPdmQWsmhQ@mail.gmail.com>

Hello there,

A quick update:

Date is now set for September 1st, 2015. I'm happy to let you know that
Opera Software will be hosting us at their HQ at Nydalen in Oslo, Norway.
All details are, as usual, in our wiki:
https://www.varnish-cache.org/trac/wiki/VDD15Q3

If you should be on the list of attendees and are not currently, please let
Lasse or myself know and we'll get things rolling off-thread.

Have a lovely weekend!


On Tue, Jun 16, 2015 at 2:37 PM, Lasse Karstensen <
lkarsten at varnish-software.com> wrote:

> On Mon, Jun 08, 2015 at 04:39:29PM +0200, Lasse Karstensen wrote:
> > Can anyone that wants to join please fill out (non-binding) the
> > following doodle, so we can find a timeslot for it?
> >     http://doodle.com/uxr5zxq6necwqvnb
>
> Hi all.
>
> Just a quick reminder; please fill this out today or tomorrow if you
> want to join.
>
> --
> Lasse Karstensen
> Varnish Software AS
>


Best regards,
-- 
<http://www.varnish-software.com/>*Rub?n Romero*
Varnish Software AS
Cell: +47 95964088 / Office: +47 21989260
Skype, Twitter & IRC: ruben_varnish
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150725/549be768/attachment.html>

From phk at phk.freebsd.dk  Sun Jul 26 08:36:15 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 26 Jul 2015 08:36:15 +0000
Subject: bugwash july 27
Message-ID: <49407.1437899775@critter.freebsd.dk>

I'll be at the HTTP2 workshop in M?nster, Germany and I have
no idea if will be able to participate in the bugwash.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.



From Gauthier.Delacroix at coreye.fr  Mon Jul 27 10:04:40 2015
From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier)
Date: Mon, 27 Jul 2015 10:04:40 +0000
Subject: [PATCH] Variable for vcl_dir in startup scripts
In-Reply-To: <CAJV_h0afV1wJXtt2-TzVtfCHBG3te1XKc2BEMTC15_y2Vmu49Q@mail.gmail.com>
References: <5F530A9242E7F84F999DB40E0E268FBD4B0EB01D@mercalli.lild01.pictime.fr>
 <CAJV_h0Zy_=Tub7x7Rm=HJgiA2v83ACoVCDo_Jc1F-xf2+gNK0A@mail.gmail.com>
 <5F530A9242E7F84F999DB40E0E268FBD4B0EB0F2@mercalli.lild01.pictime.fr>
 <CAJV_h0afV1wJXtt2-TzVtfCHBG3te1XKc2BEMTC15_y2Vmu49Q@mail.gmail.com>
Message-ID: <5F530A9242E7F84F999DB40E0E268FBD4B1796F7@mercalli.lild01.pictime.fr>

Hi Federico,

Seems the prestart is required for reloads.

Here is a new patch, just including DAEMON_OPTS in both tests.

Gauthier

From 4c11e58d6270591f93be58551496412e3118aeda Mon Sep 17 00:00:00 2001
From: Gauthier Delacroix <gauthier.delacroix at coreye.fr>
Date: Mon, 27 Jul 2015 11:55:13 +0200
Subject: [PATCH] Include DAEMON_OPTS in service prestart tests

---
 redhat/varnish.initrc  | 2 +-
 redhat/varnish.service | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc
index 117e334..03f47fb 100755
--- a/redhat/varnish.initrc
+++ b/redhat/varnish.initrc
@@ -126,7 +126,7 @@ rh_status_q() {
 
 configtest() {
     if [ -f "$VARNISH_VCL_CONF" ]; then
-        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp > /dev/null && echo "Syntax ok"
+        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp $DAEMON_OPTS > /dev/null && echo "Syntax ok"
     else
 	echo "VARNISH_VCL_CONF is  unset or does not point to a file"
     fi
diff --git a/redhat/varnish.service b/redhat/varnish.service
index a4f3355..ca93b3d 100644
--- a/redhat/varnish.service
+++ b/redhat/varnish.service
@@ -27,7 +27,7 @@ EnvironmentFile=/etc/varnish/varnish.params
 Type=forking
 PIDFile=/var/run/varnish.pid
 PrivateTmp=true
-ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF
+ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF $DAEMON_OPTS
 ExecStart=/usr/sbin/varnishd \
 	-P /var/run/varnish.pid \
 	-f $VARNISH_VCL_CONF \
-- 
1.8.3.msysgit.0

De?: Federico Schwindt [mailto:fgsch at lodoss.net] 
Envoy??: mercredi 15 juillet 2015 20:07
??: Delacroix, Gauthier <Gauthier.Delacroix at coreye.fr>
Cc?: varnish-dev at varnish-cache.org; ssm at redpill-linpro.com
Objet?: Re: [PATCH] Variable for vcl_dir in startup scripts

Hi Gauthier,

Apologies for the delay. 

In all fairness I copied this from the Debian systemd files [1] in an attempt to unify the service files. 
I don't see any particular reason for having this, specially if it's the source of issues. 

Stig, is there any reason for the ExecStartPre? The commit message says "debian/varnish.service: Test configuration before starting, add reload".
I don't see any difference in the output for an invalid config with ExecStartPre when I run either `systemctl start' or `systemctl status'.

Thanks.

1. http://anonscm.debian.org/cgit/pkg-varnish/pkg-varnish.git/tree/debian

On Thu, Jun 25, 2015 at 5:12 PM, Delacroix, Gauthier <Gauthier.Delacroix at coreye.fr> wrote:
Federico,

I've initially set vcl_dir in DAEMON_OPTS in varnish.params with 4.0.2, but it's not included in the syntax check added in 4.0.3, making my config unable to start without changing varnish.service.

I'm not aware of the reason behind this syntax check (won't a syntax error make startup fail anyway ?) but I assumed you had a good reason not to include DAEMON_OPTS.

As it's better to change varnish.params rather than varnish.service, my proposal was to add a way to include parameters needed to make the config work in the syntax check, without including the full DAEMON_OPTS.

If you finally think DAEMON_OPTS can be included in the syntax check, then I can send a really smaller patch.

Gauthier

De?: Federico Schwindt [mailto:fgsch at lodoss.net]
Envoy??: jeudi 25 juin 2015 17:53
??: Delacroix, Gauthier
Cc?: varnish-dev at varnish-cache.org
Objet?: Re: [PATCH] Variable for vcl_dir in startup scripts

Hi,

My first inclination is to ensure that the ExecStartPre line uses the same parameters as ExecStart.

So my questions is how do you set vcl_dir? Do you edit varnish.service or varnish.params?

Wouldn't adding DAEMON_OPTS to ExecStartPre (and configtest) do it?

On Thu, Jun 25, 2015 at 3:26 PM, Delacroix, Gauthier <Gauthier.Delacroix at coreye.fr> wrote:
Here is another patch proposal to make syntax check handle parameters required to compile the VCL (vcl_dir, etc.) without creating a startup variable for each parameter.

It just adds a COMPILE_OPTS that is merged in DAEMON_OPTS to start Varnish but is used alone in the syntax check.

Gauthier

From d1567a956d53a489aa4ace66ce0b1c1ef745570b Mon Sep 17 00:00:00 2001
From: Gauthier Delacroix <gauthier.delacroix at coreye.fr>
Date: Thu, 25 Jun 2015 15:06:08 +0200
Subject: [PATCH] Add COMPILE_OPTS in startup scripts to make syntax check
?check handle compilation parameters

---
?redhat/varnish.initrc? ? |? 6 ++++--
?redhat/varnish.params? ? |? 8 ++++++++
?redhat/varnish.service? ?|? 3 ++-
?redhat/varnish.sysconfig | 11 ++++++++++-
?4 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc
index 117e334..0bde074 100755
--- a/redhat/varnish.initrc
+++ b/redhat/varnish.initrc
@@ -126,9 +126,11 @@ rh_status_q() {

?configtest() {
? ? ?if [ -f "$VARNISH_VCL_CONF" ]; then
-? ? ? ? $exec -f "$VARNISH_VCL_CONF" -C -n /tmp > /dev/null && echo "Syntax ok"
+? ? ? ? $exec -f "$VARNISH_VCL_CONF" -C -n /tmp $COMPILE_OPTS > /dev/null \
+? ? ? ? ? && echo "Syntax ok"
? ? ?else
-? ? ? ?echo "VARNISH_VCL_CONF is? unset or does not point to a file"
+? ? ? ?echo "VARNISH_VCL_CONF is unset or does not point to a file"
+? ? ? ?echo "Also check that COMPILE_OPTS is set depending on the VCL config"
? ? ?fi
?}

diff --git a/redhat/varnish.params b/redhat/varnish.params
index 27a14dd..970d088 100644
--- a/redhat/varnish.params
+++ b/redhat/varnish.params
@@ -31,5 +31,13 @@ VARNISH_TTL=120
?VARNISH_USER=varnish
?VARNISH_GROUP=varnish

+# Startup options required to compile the configuration.
+# The following run-time parameters must be defined here, if needed:
+# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref, vcc_unsafe_path,
+# vcl_dir, vmod_dir
+# Defining them in DAEMON_OPTS may result in a syntax check failure.
+# See the man page varnishd(1).
+#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
+
?# Other options, see the man page varnishd(1)
?#DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300"
diff --git a/redhat/varnish.service b/redhat/varnish.service
index a4f3355..a08db58 100644
--- a/redhat/varnish.service
+++ b/redhat/varnish.service
@@ -27,7 +27,7 @@ EnvironmentFile=/etc/varnish/varnish.params
?Type=forking
?PIDFile=/var/run/varnish.pid
?PrivateTmp=true
-ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF
+ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF $COMPILE_OPTS
?ExecStart=/usr/sbin/varnishd \
? ? ? ? -P /var/run/varnish.pid \
? ? ? ? -f $VARNISH_VCL_CONF \
@@ -38,6 +38,7 @@ ExecStart=/usr/sbin/varnishd \
? ? ? ? -g $VARNISH_GROUP \
? ? ? ? -S $VARNISH_SECRET_FILE \
? ? ? ? -s $VARNISH_STORAGE \
+? ? ? ?$COMPILE_OPTS \
? ? ? ? $DAEMON_OPTS

?ExecReload=/usr/sbin/varnish_reload_vcl
diff --git a/redhat/varnish.sysconfig b/redhat/varnish.sysconfig
index 6aa2354..0e376ff 100644
--- a/redhat/varnish.sysconfig
+++ b/redhat/varnish.sysconfig
@@ -91,6 +91,14 @@ VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
?# # Default TTL used when the backend does not specify one
?VARNISH_TTL=120
?#
+# Startup options required to compile the configuration.
+# The following run-time parameters must be defined here, if needed:
+# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref, vcc_unsafe_path,
+# vcl_dir, vmod_dir
+# Defining them in DAEMON_OPTS may result in a syntax check failure.
+# See the man page varnishd(1).
+#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
+#
?# # DAEMON_OPTS is used by the init script.? If you add or remove options, make
?# # sure you update this section, too.
?DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
@@ -102,7 +110,8 @@ DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
? ? ? ? ? ? ? -p thread_pool_timeout=${VARNISH_THREAD_TIMEOUT} \
? ? ? ? ? ? ? -u varnish -g varnish \
? ? ? ? ? ? ? -S ${VARNISH_SECRET_FILE} \
-? ? ? ? ? ? ?-s ${VARNISH_STORAGE}"
+? ? ? ? ? ? ?-s ${VARNISH_STORAGE}" \
+? ? ? ? ? ? ?${COMPILE_OPTS}
?#


--
1.8.3.msysgit.0



_______________________________________________
varnish-dev mailing list
varnish-dev at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


From fgsch at lodoss.net  Mon Jul 27 15:22:57 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Mon, 27 Jul 2015 16:22:57 +0100
Subject: [PATCH] Kill unused session_max
Message-ID: <CAJV_h0ZGw1DLQkSCUMix6PsTrY_Qp3MLh+5TCLx7rpnMfwFR_Q@mail.gmail.com>

This looks like a relic from V3.

OK?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150727/349b9e15/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Kill-unused-session_max.patch
Type: text/x-patch
Size: 1545 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150727/349b9e15/attachment.bin>

From fgsch at lodoss.net  Mon Jul 27 15:23:40 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Mon, 27 Jul 2015 16:23:40 +0100
Subject: [PATCH] Only load the default config if the env is not set
Message-ID: <CAJV_h0aLZK76Yo6dnMyuz2jh59AEDL543TSGsj96gQ_-Eazi_g@mail.gmail.com>

This makes it possible to use the same reload script with different systemd
instances.

OK?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150727/8570f049/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Only-load-the-default-config-if-the-env-is-not-set.patch
Type: text/x-patch
Size: 837 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150727/8570f049/attachment.bin>

From fgsch at lodoss.net  Mon Jul 27 15:24:21 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Mon, 27 Jul 2015 16:24:21 +0100
Subject: [PATCH] Remove unreachable sess_drop
Message-ID: <CAJV_h0beG7u613+Qy3THYS7cGtE6iawO50J0rBubAjJpT5K-oA@mail.gmail.com>

SES_New() will never return NULL. Replace noop with AN().

OK?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150727/07f45014/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Remove-unreachable-sess_drop.patch
Type: text/x-patch
Size: 1737 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150727/07f45014/attachment.bin>

From fgsch at lodoss.net  Mon Jul 27 15:44:37 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Mon, 27 Jul 2015 16:44:37 +0100
Subject: [PATCH] Remove unreachable sess_drop
In-Reply-To: <CAJV_h0beG7u613+Qy3THYS7cGtE6iawO50J0rBubAjJpT5K-oA@mail.gmail.com>
References: <CAJV_h0beG7u613+Qy3THYS7cGtE6iawO50J0rBubAjJpT5K-oA@mail.gmail.com>
Message-ID: <CAJV_h0amYdppEx6NDz=gtt0pP-DW-VUtvW+6pUB-PYAJoz1=Rg@mail.gmail.com>

Actually the AN() is not needed since the check below will trigger the
assert anyway.

On Mon, Jul 27, 2015 at 4:24 PM, Federico Schwindt <fgsch at lodoss.net> wrote:

> SES_New() will never return NULL. Replace noop with AN().
>
> OK?
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150727/325f927b/attachment.html>

From fgsch at lodoss.net  Mon Jul 27 20:38:02 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Mon, 27 Jul 2015 21:38:02 +0100
Subject: [PATCH] Variable for vcl_dir in startup scripts
In-Reply-To: <5F530A9242E7F84F999DB40E0E268FBD4B1796F7@mercalli.lild01.pictime.fr>
References: <5F530A9242E7F84F999DB40E0E268FBD4B0EB01D@mercalli.lild01.pictime.fr>
 <CAJV_h0Zy_=Tub7x7Rm=HJgiA2v83ACoVCDo_Jc1F-xf2+gNK0A@mail.gmail.com>
 <5F530A9242E7F84F999DB40E0E268FBD4B0EB0F2@mercalli.lild01.pictime.fr>
 <CAJV_h0afV1wJXtt2-TzVtfCHBG3te1XKc2BEMTC15_y2Vmu49Q@mail.gmail.com>
 <5F530A9242E7F84F999DB40E0E268FBD4B1796F7@mercalli.lild01.pictime.fr>
Message-ID: <CAJV_h0aB5rLtD9jq0rNDek-eVJvr3JvuADGEi6f=V2c2sKjLkA@mail.gmail.com>

> Seems the prestart is required for reloads.

I haven't tested RH but this doesn't seem to be the case in Debian -
ExecStartPre is only called before (re)start.

The sysv init script change might be needed though. Setting vmod_dir will
also affects the test btw.

On Mon, Jul 27, 2015 at 11:04 AM, Delacroix, Gauthier <
Gauthier.Delacroix at coreye.fr> wrote:

> Hi Federico,
>
> Seems the prestart is required for reloads.
>
> Here is a new patch, just including DAEMON_OPTS in both tests.
>
> Gauthier
>
> From 4c11e58d6270591f93be58551496412e3118aeda Mon Sep 17 00:00:00 2001
> From: Gauthier Delacroix <gauthier.delacroix at coreye.fr>
> Date: Mon, 27 Jul 2015 11:55:13 +0200
> Subject: [PATCH] Include DAEMON_OPTS in service prestart tests
>
> ---
>  redhat/varnish.initrc  | 2 +-
>  redhat/varnish.service | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc
> index 117e334..03f47fb 100755
> --- a/redhat/varnish.initrc
> +++ b/redhat/varnish.initrc
> @@ -126,7 +126,7 @@ rh_status_q() {
>
>  configtest() {
>      if [ -f "$VARNISH_VCL_CONF" ]; then
> -        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp > /dev/null && echo
> "Syntax ok"
> +        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp $DAEMON_OPTS > /dev/null
> && echo "Syntax ok"
>      else
>         echo "VARNISH_VCL_CONF is  unset or does not point to a file"
>      fi
> diff --git a/redhat/varnish.service b/redhat/varnish.service
> index a4f3355..ca93b3d 100644
> --- a/redhat/varnish.service
> +++ b/redhat/varnish.service
> @@ -27,7 +27,7 @@ EnvironmentFile=/etc/varnish/varnish.params
>  Type=forking
>  PIDFile=/var/run/varnish.pid
>  PrivateTmp=true
> -ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF
> +ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF $DAEMON_OPTS
>  ExecStart=/usr/sbin/varnishd \
>         -P /var/run/varnish.pid \
>         -f $VARNISH_VCL_CONF \
> --
> 1.8.3.msysgit.0
>
> De : Federico Schwindt [mailto:fgsch at lodoss.net]
> Envoy? : mercredi 15 juillet 2015 20:07
> ? : Delacroix, Gauthier <Gauthier.Delacroix at coreye.fr>
> Cc : varnish-dev at varnish-cache.org; ssm at redpill-linpro.com
> Objet : Re: [PATCH] Variable for vcl_dir in startup scripts
>
> Hi Gauthier,
>
> Apologies for the delay.
>
> In all fairness I copied this from the Debian systemd files [1] in an
> attempt to unify the service files.
> I don't see any particular reason for having this, specially if it's the
> source of issues.
>
> Stig, is there any reason for the ExecStartPre? The commit message says
> "debian/varnish.service: Test configuration before starting, add reload".
> I don't see any difference in the output for an invalid config with
> ExecStartPre when I run either `systemctl start' or `systemctl status'.
>
> Thanks.
>
> 1. http://anonscm.debian.org/cgit/pkg-varnish/pkg-varnish.git/tree/debian
>
> On Thu, Jun 25, 2015 at 5:12 PM, Delacroix, Gauthier <
> Gauthier.Delacroix at coreye.fr> wrote:
> Federico,
>
> I've initially set vcl_dir in DAEMON_OPTS in varnish.params with 4.0.2,
> but it's not included in the syntax check added in 4.0.3, making my config
> unable to start without changing varnish.service.
>
> I'm not aware of the reason behind this syntax check (won't a syntax error
> make startup fail anyway ?) but I assumed you had a good reason not to
> include DAEMON_OPTS.
>
> As it's better to change varnish.params rather than varnish.service, my
> proposal was to add a way to include parameters needed to make the config
> work in the syntax check, without including the full DAEMON_OPTS.
>
> If you finally think DAEMON_OPTS can be included in the syntax check, then
> I can send a really smaller patch.
>
> Gauthier
>
> De : Federico Schwindt [mailto:fgsch at lodoss.net]
> Envoy? : jeudi 25 juin 2015 17:53
> ? : Delacroix, Gauthier
> Cc : varnish-dev at varnish-cache.org
> Objet : Re: [PATCH] Variable for vcl_dir in startup scripts
>
> Hi,
>
> My first inclination is to ensure that the ExecStartPre line uses the same
> parameters as ExecStart.
>
> So my questions is how do you set vcl_dir? Do you edit varnish.service or
> varnish.params?
>
> Wouldn't adding DAEMON_OPTS to ExecStartPre (and configtest) do it?
>
> On Thu, Jun 25, 2015 at 3:26 PM, Delacroix, Gauthier <
> Gauthier.Delacroix at coreye.fr> wrote:
> Here is another patch proposal to make syntax check handle parameters
> required to compile the VCL (vcl_dir, etc.) without creating a startup
> variable for each parameter.
>
> It just adds a COMPILE_OPTS that is merged in DAEMON_OPTS to start Varnish
> but is used alone in the syntax check.
>
> Gauthier
>
> From d1567a956d53a489aa4ace66ce0b1c1ef745570b Mon Sep 17 00:00:00 2001
> From: Gauthier Delacroix <gauthier.delacroix at coreye.fr>
> Date: Thu, 25 Jun 2015 15:06:08 +0200
> Subject: [PATCH] Add COMPILE_OPTS in startup scripts to make syntax check
>  check handle compilation parameters
>
> ---
>  redhat/varnish.initrc    |  6 ++++--
>  redhat/varnish.params    |  8 ++++++++
>  redhat/varnish.service   |  3 ++-
>  redhat/varnish.sysconfig | 11 ++++++++++-
>  4 files changed, 24 insertions(+), 4 deletions(-)
>
> diff --git a/redhat/varnish.initrc b/redhat/varnish.initrc
> index 117e334..0bde074 100755
> --- a/redhat/varnish.initrc
> +++ b/redhat/varnish.initrc
> @@ -126,9 +126,11 @@ rh_status_q() {
>
>  configtest() {
>      if [ -f "$VARNISH_VCL_CONF" ]; then
> -        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp > /dev/null && echo
> "Syntax ok"
> +        $exec -f "$VARNISH_VCL_CONF" -C -n /tmp $COMPILE_OPTS > /dev/null
> \
> +          && echo "Syntax ok"
>      else
> -       echo "VARNISH_VCL_CONF is  unset or does not point to a file"
> +       echo "VARNISH_VCL_CONF is unset or does not point to a file"
> +       echo "Also check that COMPILE_OPTS is set depending on the VCL
> config"
>      fi
>  }
>
> diff --git a/redhat/varnish.params b/redhat/varnish.params
> index 27a14dd..970d088 100644
> --- a/redhat/varnish.params
> +++ b/redhat/varnish.params
> @@ -31,5 +31,13 @@ VARNISH_TTL=120
>  VARNISH_USER=varnish
>  VARNISH_GROUP=varnish
>
> +# Startup options required to compile the configuration.
> +# The following run-time parameters must be defined here, if needed:
> +# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref,
> vcc_unsafe_path,
> +# vcl_dir, vmod_dir
> +# Defining them in DAEMON_OPTS may result in a syntax check failure.
> +# See the man page varnishd(1).
> +#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
> +
>  # Other options, see the man page varnishd(1)
>  #DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p
> thread_pool_timeout=300"
> diff --git a/redhat/varnish.service b/redhat/varnish.service
> index a4f3355..a08db58 100644
> --- a/redhat/varnish.service
> +++ b/redhat/varnish.service
> @@ -27,7 +27,7 @@ EnvironmentFile=/etc/varnish/varnish.params
>  Type=forking
>  PIDFile=/var/run/varnish.pid
>  PrivateTmp=true
> -ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF
> +ExecStartPre=/usr/sbin/varnishd -C -f $VARNISH_VCL_CONF $COMPILE_OPTS
>  ExecStart=/usr/sbin/varnishd \
>         -P /var/run/varnish.pid \
>         -f $VARNISH_VCL_CONF \
> @@ -38,6 +38,7 @@ ExecStart=/usr/sbin/varnishd \
>         -g $VARNISH_GROUP \
>         -S $VARNISH_SECRET_FILE \
>         -s $VARNISH_STORAGE \
> +       $COMPILE_OPTS \
>         $DAEMON_OPTS
>
>  ExecReload=/usr/sbin/varnish_reload_vcl
> diff --git a/redhat/varnish.sysconfig b/redhat/varnish.sysconfig
> index 6aa2354..0e376ff 100644
> --- a/redhat/varnish.sysconfig
> +++ b/redhat/varnish.sysconfig
> @@ -91,6 +91,14 @@ VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
>  # # Default TTL used when the backend does not specify one
>  VARNISH_TTL=120
>  #
> +# Startup options required to compile the configuration.
> +# The following run-time parameters must be defined here, if needed:
> +# cc_command, group_cc, vcc_allow_inline_c, vcc_err_unref,
> vcc_unsafe_path,
> +# vcl_dir, vmod_dir
> +# Defining them in DAEMON_OPTS may result in a syntax check failure.
> +# See the man page varnishd(1).
> +#COMPILE_OPTS="-p vcl_dir=/etc/varnish -p vcc_err_unref=on"
> +#
>  # # DAEMON_OPTS is used by the init script.  If you add or remove
> options, make
>  # # sure you update this section, too.
>  DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
> @@ -102,7 +110,8 @@ DAEMON_OPTS="-a
> ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
>               -p thread_pool_timeout=${VARNISH_THREAD_TIMEOUT} \
>               -u varnish -g varnish \
>               -S ${VARNISH_SECRET_FILE} \
> -             -s ${VARNISH_STORAGE}"
> +             -s ${VARNISH_STORAGE}" \
> +             ${COMPILE_OPTS}
>  #
>
>
> --
> 1.8.3.msysgit.0
>
>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150727/bc935c87/attachment.html>

From alanj at sportsmole.com  Tue Jul 28 20:52:06 2015
From: alanj at sportsmole.com (ALan Jay)
Date: Tue, 28 Jul 2015 13:52:06 -0700
Subject: test-suite log output as requested in the output
Message-ID: <55B7EB76.6040704@sportsmole.com>


-- 
Alan Jay
+44 7973 183 001
+1 424 442 0197
Skype:      alanjay
Twitter:  @alanjay

[+] Add me to your address book 
<https://ws.writethat.name/kwaga-bin/titan/WEB/me.pl/10445863511369763413/i>


www.SportsMole.co.uk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150728/a1ce0c8a/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test-suite.log
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150728/a1ce0c8a/attachment.ksh>

From lkarsten at varnish-software.com  Wed Jul 29 11:38:31 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 29 Jul 2015 13:38:31 +0200
Subject: test-suite log output as requested in the output
In-Reply-To: <55B7EB76.6040704@sportsmole.com>
References: <55B7EB76.6040704@sportsmole.com>
Message-ID: <20150729113830.GC21270@immer.varnish-software.com>

On Tue, Jul 28, 2015 at 01:52:06PM -0700, ALan Jay wrote:
[..]
> **** v1    0.4 CLI RX| Could not load VMOD directors\n
> **** v1    0.4 CLI RX| \tFile name: /home/ec2-user/varnish-4.0.3/lib/libvmod_directors/.libs/libvmod_directors.so\n
> **** v1    0.4 CLI RX| \tdlerror:: /home/ec2-user/varnish-4.0.3/lib/libvmod_directors/.libs/libvmod_directors.so: cannot open shared object file: Permission denied\n
> **** v1    0.4 CLI RX| ('input' Line 5 Pos 16)\n
> **** v1    0.4 CLI RX|         import directors from "/home/ec2-user/varnish-4.0.3/lib/libvmod_directors/.libs/libvmod_directors.so";\n
> **** v1    0.4 CLI RX| ---------------#########-------------------------------------------------------------------...
> ---- v1    0.4 FAIL VCL does not compile


You have a permission problem in your source tree.

Most likely you've built Varnish as root at some point, and now your
ordinary user can't access the built files.


-- 
Lasse Karstensen
Varnish Software AS



From eric.andrew.lewis at gmail.com  Wed Jul 29 15:48:54 2015
From: eric.andrew.lewis at gmail.com (Eric Andrew Lewis)
Date: Wed, 29 Jul 2015 11:48:54 -0400
Subject: Couldn't install from source
Message-ID: <CAMi+fEmskEybSYKEi0st2y4-HrGrut78=c+dE+F48dqxnh=ngg@mail.gmail.com>

I tried installing Varnish 4 from source following these directions
<https://www.varnish-cache.org/docs/4.0/installation/install.html> in
Ubuntu 14.04.2 running in a Docker linux container. Here's a list the
commands I run
<https://gist.github.com/ericandrewlewis/c973a4213dc809241689>, all as root.

After `make install`, the file "/usr/local/etc/varnish/default.vcl" did not
exist, and the system didn't recognize the service "varnish". The varnishd
binary did install to the expected location.

Eric Andrew Lewis
ericandrewlewis.com
610.715.8560
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150729/c0ef2f6e/attachment.html>

From alanj at sportsmole.com  Wed Jul 29 21:35:03 2015
From: alanj at sportsmole.com (ALan Jay)
Date: Wed, 29 Jul 2015 14:35:03 -0700
Subject: test-suite log output as requested in the output
In-Reply-To: <20150729113830.GC21270@immer.varnish-software.com>
References: <55B7EB76.6040704@sportsmole.com>
 <20150729113830.GC21270@immer.varnish-software.com>
Message-ID: <55B94707.4020006@sportsmole.com>

Thanks for the feedback - I did build Varnish while logged in as root, 
but I thought I also ran the test as root but will check, out of 
curiosity should the compilation and install be done as root?  And if 
not which user should I be using?

Alan

Lasse Karstensen wrote:
> On Tue, Jul 28, 2015 at 01:52:06PM -0700, ALan Jay wrote:
> [..]
>> **** v1    0.4 CLI RX| Could not load VMOD directors\n
>> **** v1    0.4 CLI RX| \tFile name: /home/ec2-user/varnish-4.0.3/lib/libvmod_directors/.libs/libvmod_directors.so\n
>> **** v1    0.4 CLI RX| \tdlerror:: /home/ec2-user/varnish-4.0.3/lib/libvmod_directors/.libs/libvmod_directors.so: cannot open shared object file: Permission denied\n
>> **** v1    0.4 CLI RX| ('input' Line 5 Pos 16)\n
>> **** v1    0.4 CLI RX|         import directors from "/home/ec2-user/varnish-4.0.3/lib/libvmod_directors/.libs/libvmod_directors.so";\n
>> **** v1    0.4 CLI RX| ---------------#########-------------------------------------------------------------------...
>> ---- v1    0.4 FAIL VCL does not compile
>
>
> You have a permission problem in your source tree.
>
> Most likely you've built Varnish as root at some point, and now your
> ordinary user can't access the built files.
>
>

-- 
Alan Jay
+44 7973 183 001
+1 424 442 0197
Skype:      alanjay
Twitter:  @alanjay

[+] Add me to your address book 
<https://ws.writethat.name/kwaga-bin/titan/WEB/me.pl/10445863511369763413/i>


www.SportsMole.co.uk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150729/64363d22/attachment.html>

From alanj at sportsmole.com  Wed Jul 29 23:07:06 2015
From: alanj at sportsmole.com (ALan Jay)
Date: Wed, 29 Jul 2015 16:07:06 -0700
Subject: test-suite log output as requested in the output
In-Reply-To: <20150729113830.GC21270@immer.varnish-software.com>
References: <55B7EB76.6040704@sportsmole.com>
 <20150729113830.GC21270@immer.varnish-software.com>
Message-ID: <55B95C9A.3040301@sportsmole.com>

Further to my query about this I ran the check again as "root" and the 
attached is what the output looked like, I'd be keen to understand how 
the permissions need to be set or as what user I need to run this to 
make this work.

Alan

Lasse Karstensen wrote:
> On Tue, Jul 28, 2015 at 01:52:06PM -0700, ALan Jay wrote:
> [..]
>> **** v1    0.4 CLI RX| Could not load VMOD directors\n
>> **** v1    0.4 CLI RX| \tFile name: /home/ec2-user/varnish-4.0.3/lib/libvmod_directors/.libs/libvmod_directors.so\n
>> **** v1    0.4 CLI RX| \tdlerror:: /home/ec2-user/varnish-4.0.3/lib/libvmod_directors/.libs/libvmod_directors.so: cannot open shared object file: Permission denied\n
>> **** v1    0.4 CLI RX| ('input' Line 5 Pos 16)\n
>> **** v1    0.4 CLI RX|         import directors from "/home/ec2-user/varnish-4.0.3/lib/libvmod_directors/.libs/libvmod_directors.so";\n
>> **** v1    0.4 CLI RX| ---------------#########-------------------------------------------------------------------...
>> ---- v1    0.4 FAIL VCL does not compile
>
>
> You have a permission problem in your source tree.
>
> Most likely you've built Varnish as root at some point, and now your
> ordinary user can't access the built files.
>
>

-- 
Alan Jay
+44 7973 183 001
+1 424 442 0197
Skype:      alanjay
Twitter:  @alanjay

[+] Add me to your address book 
<https://ws.writethat.name/kwaga-bin/titan/WEB/me.pl/10445863511369763413/i>


www.SportsMole.co.uk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150729/07b6f2d6/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test-suite.log
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150729/07b6f2d6/attachment.ksh>

From lkarsten at varnish-software.com  Thu Jul 30 08:50:20 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 30 Jul 2015 10:50:20 +0200
Subject: Couldn't install from source
In-Reply-To: <CAMi+fEmskEybSYKEi0st2y4-HrGrut78=c+dE+F48dqxnh=ngg@mail.gmail.com>
References: <CAMi+fEmskEybSYKEi0st2y4-HrGrut78=c+dE+F48dqxnh=ngg@mail.gmail.com>
Message-ID: <20150730085019.GA16958@immer.varnish-software.com>

On Wed, Jul 29, 2015 at 11:48:54AM -0400, Eric Andrew Lewis wrote:
> I tried installing Varnish 4 from source following these directions
> <https://www.varnish-cache.org/docs/4.0/installation/install.html> in
> Ubuntu 14.04.2 running in a Docker linux container. Here's a list the
> commands I run
> <https://gist.github.com/ericandrewlewis/c973a4213dc809241689>, all as root.
> 
> After `make install`, the file "/usr/local/etc/varnish/default.vcl" did not
> exist, and the system didn't recognize the service "varnish". The varnishd
> binary did install to the expected location.

Hi.

varnish-dev@ is a list for discussion on Varnish development.

It is not a help forum. Please use varnish-misc@ for such requests.

You should also consider adding more details, as there is no way to
know what went wrong from what you wrote here.

-- 
Lasse Karstensen
Varnish Software AS



From lkarsten at varnish-software.com  Thu Jul 30 08:56:14 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 30 Jul 2015 10:56:14 +0200
Subject: test-suite log output as requested in the output
In-Reply-To: <55B95C9A.3040301@sportsmole.com>
References: <55B7EB76.6040704@sportsmole.com>
 <20150729113830.GC21270@immer.varnish-software.com>
 <55B95C9A.3040301@sportsmole.com>
Message-ID: <20150730085613.GB16958@immer.varnish-software.com>

On Wed, Jul 29, 2015 at 04:07:06PM -0700, ALan Jay wrote:
> Further to my query about this I ran the check again as "root" and the
> attached is what the output looked like, I'd be keen to understand how the
> permissions need to be set or as what user I need to run this to make this
> work.


Hi Alan.

First of all, please direct this to varnish-misc@ in the future.
varnish-dev@ is for development discussion.

Second, as far as I can see the log file shows the same errors as
before. Check basic file rights, and verify with strace. If nothing
else works, consider starting from scratch/tarball in a different
direction, and avoid typing "sudo" at any point.

You can run a single test case with: "cd bin/varnishtest/; varnishtest
-i tests/nnnnnn.vtc".

-- 
Lasse Karstensen
Varnish Software AS



From lkarsten at varnish-software.com  Fri Jul 31 07:48:26 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Fri, 31 Jul 2015 09:48:26 +0200
Subject: [PATCH 1/4] Add ws overflow functions to vmod-debug
Message-ID: <20150731074823.GA17848@immer.varnish-software.com>

Add support for deterministic overflow of the different
workspaces in vtc.
---
 lib/libvmod_debug/vmod.vcc     | 16 ++++++++++
 lib/libvmod_debug/vmod_debug.c | 70 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+)

diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index 8b32e92..dad36ec 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -122,3 +122,19 @@ Return the dynamic backend.
 $Method VOID .refresh(STRING addr, STRING port)
 
 Dynamically refresh & (always!) replace the backend by a new one.
+
+$Function VOID workspace_allocate(ENUM { client, backend, session, thread }, INT SIZE)
+
+Allocate and zero out SIZE bytes from a workspace.
+
+$Function BOOL workspace_overflowed(ENUM { client, backend, session, thread })
+
+Return if the workspace overflow mark is set or not.
+
+$Function VOID workspace_overflow(ENUM { client, backend, session, thread })
+
+Mark a workspace as overflowed.
+
+$Function INT workspace_free(ENUM { client, backend, session, thread })
+
+Find how much unallocated space there is left in a workspace.
diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c
index 4d86baa..47b1240 100644
--- a/lib/libvmod_debug/vmod_debug.c
+++ b/lib/libvmod_debug/vmod_debug.c
@@ -279,3 +279,73 @@ vmod_sleep(VRT_CTX, VCL_DURATION t)
 	CHECK_OBJ_ORNULL(ctx, VRT_CTX_MAGIC);
 	VTIM_sleep(t);
 }
+
+static struct ws *
+wsfind(VRT_CTX, VCL_ENUM which) {
+	if (!strcmp(which, "client")) {
+		return ctx->ws;
+	} else if (!strcmp(which, "backend")) {
+		return ctx->bo->ws;
+	} else if (!strcmp(which, "session")) {
+		return ctx->req->sp->ws;
+	} else if (!strcmp(which, "thread")) {
+		return ctx->req->wrk->aws;
+	} else
+		WRONG("No such workspace.");
+}
+
+void
+vmod_workspace_allocate(VRT_CTX, VCL_ENUM which, VCL_INT size)
+{
+	struct ws *ws;
+	char *s;
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+
+	ws = wsfind(ctx, which);
+
+	WS_Assert(ws);
+	AZ(ws->r);
+
+	s = WS_Alloc(ws, size);
+	if (!s)
+		return;
+	memset(s, '\0', size);
+}
+
+VCL_INT
+vmod_workspace_free(VRT_CTX, VCL_ENUM which)
+{
+	struct ws *ws;
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+
+	ws = wsfind(ctx, which);
+
+	WS_Assert(ws);
+	AZ(ws->r);
+
+	return pdiff(ws->f, ws->e);
+}
+
+VCL_BOOL
+vmod_workspace_overflowed(VRT_CTX, VCL_ENUM which)
+{
+	struct ws *ws;
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+
+	ws = wsfind(ctx, which);
+	WS_Assert(ws);
+
+	return (WS_Overflowed(ws));
+}
+
+void
+vmod_workspace_overflow(VRT_CTX, VCL_ENUM which)
+{
+	struct ws *ws;
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+
+	ws = wsfind(ctx, which);
+	WS_Assert(ws);
+
+	WS_MarkOverflow(ws);
+}
-- 
2.1.4


-- 
Lasse Karstensen
Varnish Software AS



From lkarsten at varnish-software.com  Fri Jul 31 07:48:35 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Fri, 31 Jul 2015 09:48:35 +0200
Subject: [PATCH 2/4] Improve client ws overflow handling.
Message-ID: <20150731074833.GA17875@immer.varnish-software.com>

---
 bin/varnishd/cache/cache_deliver_proc.c | 5 ++++-
 bin/varnishd/cache/cache_range.c        | 8 ++++++--
 bin/varnishd/cache/cache_vrt_re.c       | 1 +
 3 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/bin/varnishd/cache/cache_deliver_proc.c b/bin/varnishd/cache/cache_deliver_proc.c
index 4fd666b..6366028 100644
--- a/bin/varnishd/cache/cache_deliver_proc.c
+++ b/bin/varnishd/cache/cache_deliver_proc.c
@@ -74,7 +74,10 @@ VDP_push(struct req *req, vdp_bytes *func, void *priv, int bottom)
 	VSB_delete(vsb);
 #endif
 	vdp = WS_Alloc(req->ws, sizeof *vdp);
-	AN(vdp);
+	if (vdp == NULL) {
+		WS_MarkOverflow(req->ws);
+		return;
+	}
 	INIT_OBJ(vdp, VDP_ENTRY_MAGIC);
 	vdp->func = func;
 	vdp->priv = priv;
diff --git a/bin/varnishd/cache/cache_range.c b/bin/varnishd/cache/cache_range.c
index 6122984..a109ba7 100644
--- a/bin/varnishd/cache/cache_range.c
+++ b/bin/varnishd/cache/cache_range.c
@@ -162,7 +162,9 @@ vrg_dorange(struct req *req, const char *r)
 	http_PutResponse(req->resp, "HTTP/1.1", 206, NULL);
 
 	vrg_priv = WS_Alloc(req->ws, sizeof *vrg_priv);
-	XXXAN(vrg_priv);
+	if (vrg_priv == NULL) {
+		return(-1);
+	}
 	INIT_OBJ(vrg_priv, VRG_PRIV_MAGIC);
 	vrg_priv->range_off = 0;
 	vrg_priv->range_low = low;
@@ -183,7 +185,9 @@ VRG_dorange(struct req *req, const char *r)
 	/* We must snapshot the length if we're streaming from the backend */
 
 	i = vrg_dorange(req, r);
-	if (i) {
+	if (i < 0)
+		WS_MarkOverflow(req->ws);
+	else if (i > 0) {
 		VSLb(req->vsl, SLT_Debug, "RANGE_FAIL line %d", i);
 		http_Unset(req->resp, H_Content_Length);
 		if (req->resp_len >= 0)
diff --git a/bin/varnishd/cache/cache_vrt_re.c b/bin/varnishd/cache/cache_vrt_re.c
index d96c4f4..c1ea167 100644
--- a/bin/varnishd/cache/cache_vrt_re.c
+++ b/bin/varnishd/cache/cache_vrt_re.c
@@ -171,6 +171,7 @@ VRT_regsub(VRT_CTX, int all, const char *str, void *re,
 	/* Copy suffix to match */
 	Tadd(&res_b, res_e, str + offset, 1 + len - offset);
 	if (res_b >= res_e) {
+		WS_MarkOverflow(ctx->ws);
 		WS_Release(ctx->ws, 0);
 		return (str);
 	}
-- 
2.1.4


-- 
Lasse Karstensen
Varnish Software AS



From lkarsten at varnish-software.com  Fri Jul 31 07:48:45 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Fri, 31 Jul 2015 09:48:45 +0200
Subject: [PATCH 3/4] Close with a 500 response on client ws overflow.
Message-ID: <20150731074843.GA17898@immer.varnish-software.com>

---
 bin/varnishd/cache/cache_req_fsm.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 46f777e..0a94850 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -112,6 +112,10 @@ static enum req_fsm_nxt
 cnt_deliver(struct worker *wrk, struct req *req)
 {
 	struct busyobj *bo;
+	char r_500[] =
+		"HTTP/1.1 500 Internal Server Error\r\n"
+		"Server: Varnish\r\n"
+		"Connection: close\r\n\r\n";
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
@@ -181,6 +185,28 @@ cnt_deliver(struct worker *wrk, struct req *req)
 
 	assert(wrk->handling == VCL_RET_DELIVER);
 
+	/* Make sure there is room in the ws left for delivery processors
+	 * (40 bytes each) and VSL buffering for Etag weakening and SetH().
+	 */
+	if (pdiff(req->ws->f, req->ws->e) < sizeof(struct vdp_entry)*3+128) {
+		VSLb(req->vsl, SLT_Error,
+		    "Inferred workspace_client overflow.");
+		WS_MarkOverflow(req->ws);
+	}
+
+	if (WS_Overflowed(req->ws)) {
+		VSLb(req->vsl, SLT_Error, "workspace_client overflow");
+		VSLb(req->vsl, SLT_RespProtocol, "HTTP/1.1");
+		VSLb(req->vsl, SLT_RespStatus, "500");
+		VSLb(req->vsl, SLT_RespReason, "Internal Server Error");
+
+		(void)write(req->sp->fd, r_500, strlen(r_500));
+		VSLb_ts_req(req, "Resp", W_TIM_real(wrk));
+		http_Teardown(req->resp);
+		SES_Close(req->sp, SC_OVERLOAD);
+		return (REQ_FSM_DONE);
+	}
+
 	if (!(req->objcore->flags & OC_F_PASS)
 	    && req->esi_level == 0
 	    && http_IsStatus(req->resp, 200)
-- 
2.1.4


-- 
Lasse Karstensen
Varnish Software AS



From lkarsten at varnish-software.com  Fri Jul 31 07:48:53 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Fri, 31 Jul 2015 09:48:53 +0200
Subject: [PATCH 4/4] Add test cases for overflow handling.
Message-ID: <20150731074851.GA17921@immer.varnish-software.com>

---
 bin/varnishtest/tests/c00070.vtc | 54 ++++++++++++++++++++++++++++++++++++++++
 bin/varnishtest/tests/c00071.vtc | 42 +++++++++++++++++++++++++++++++
 2 files changed, 96 insertions(+)
 create mode 100644 bin/varnishtest/tests/c00070.vtc
 create mode 100644 bin/varnishtest/tests/c00071.vtc

diff --git a/bin/varnishtest/tests/c00070.vtc b/bin/varnishtest/tests/c00070.vtc
new file mode 100644
index 0000000..1ab5bf3
--- /dev/null
+++ b/bin/varnishtest/tests/c00070.vtc
@@ -0,0 +1,54 @@
+varnishtest "Test ws overflow functions in vmod-debug"
+
+server s1 {
+	rxreq
+	txresp
+
+	rxreq
+	txresp
+} -start
+
+varnish v1 -vcl+backend {
+	import ${vmod_debug};
+	sub vcl_backend_response {
+		set beresp.http.free_backend = debug.workspace_free(backend);
+	}
+
+	sub vcl_deliver {
+		set resp.http.free_client = debug.workspace_free(client);
+		set resp.http.free_session = debug.workspace_free(session);
+		set resp.http.free_thread = debug.workspace_free(thread);
+
+		set resp.http.overflowed = debug.workspace_overflowed(client);
+		debug.workspace_allocate(client, 2048);
+
+		if (req.url == "/bar") {
+			debug.workspace_overflow(client);
+		}
+	}
+} -start
+
+logexpect l1 -v v1 -d 1 -g vxid -q "Error ~ 'overflow'" {
+        expect 0 *      Begin
+        expect * =      Error	"workspace_client overflow"
+        expect * =      End
+} -start
+
+client c1 {
+	txreq -url /foo
+	rxresp
+	expect resp.http.overflowed == "false"
+
+	expect resp.http.free_client > 57000
+	expect resp.http.free_backend > 57000
+	expect resp.http.free_session > 250
+	expect resp.http.free_thread > 2000
+} -run
+
+client c2 {
+	txreq -url /bar
+	rxresp
+	expect resp.status == 500
+} -run
+
+logexpect l1 -wait
diff --git a/bin/varnishtest/tests/c00071.vtc b/bin/varnishtest/tests/c00071.vtc
new file mode 100644
index 0000000..20e6ad9
--- /dev/null
+++ b/bin/varnishtest/tests/c00071.vtc
@@ -0,0 +1,42 @@
+varnishtest "Test actual client workspace overflow"
+
+server s1 {
+	rxreq
+	txresp
+
+	rxreq
+	txresp
+
+	rxreq
+	txresp
+} -start
+
+varnish v1 -vcl+backend {
+	import ${vmod_debug};
+	sub vcl_deliver {
+		debug.workspace_allocate(client, debug.workspace_free(client) - 278);
+		if (req.url ~ "/bar") {
+			set resp.http.x-foo = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
+		}
+		else if (req.url ~ "/baz") {
+			set resp.http.x-foo = regsub(req.url, "baz", "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaz");
+		}
+	}
+} -start
+
+client c1 {
+	txreq -url /foo
+	rxresp
+	expect resp.status == 200
+
+	txreq -url /bar
+	rxresp
+	expect resp.status == 500
+} -run
+
+client c2 {
+	txreq -url /baz
+	rxresp
+	expect resp.status == 500
+} -run
+
-- 
2.1.4


-- 
Lasse Karstensen
Varnish Software AS



From phk at phk.freebsd.dk  Fri Jul 31 08:22:57 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 31 Jul 2015 08:22:57 +0000
Subject: [PATCH] Remove unreachable sess_drop
In-Reply-To: <CAJV_h0beG7u613+Qy3THYS7cGtE6iawO50J0rBubAjJpT5K-oA@mail.gmail.com>
References: <CAJV_h0beG7u613+Qy3THYS7cGtE6iawO50J0rBubAjJpT5K-oA@mail.gmail.com>
Message-ID: <58317.1438330977@critter.freebsd.dk>

--------
In message <CAJV_h0beG7u613+Qy3THYS7cGtE6iawO50J0rBubAjJpT5K-oA at mail.gmail.com>
, Federico Schwindt writes:

>SES_New() will never return NULL. Replace noop with AN().

That is actually in preparation/reminder that it should be allowed
to fail, so I'd prefer to retain it.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From phk at phk.freebsd.dk  Fri Jul 31 08:24:32 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 31 Jul 2015 08:24:32 +0000
Subject: [PATCH] Kill unused session_max
In-Reply-To: <CAJV_h0ZGw1DLQkSCUMix6PsTrY_Qp3MLh+5TCLx7rpnMfwFR_Q@mail.gmail.com>
References: <CAJV_h0ZGw1DLQkSCUMix6PsTrY_Qp3MLh+5TCLx7rpnMfwFR_Q@mail.gmail.com>
Message-ID: <58347.1438331072@critter.freebsd.dk>

--------
In message <CAJV_h0ZGw1DLQkSCUMix6PsTrY_Qp3MLh+5TCLx7rpnMfwFR_Q at mail.gmail.com>
, Federico Schwindt writes:

>This looks like a relic from V3.

Sort of, as I said in the previous email, we should be able to fail
at creating more sessions (cant or wont) and this used to be the limit.

If anything, we should move the other way and make it work again.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From phk at phk.freebsd.dk  Fri Jul 31 10:12:39 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 31 Jul 2015 10:12:39 +0000
Subject: struct http_conn & HTTP Workshop
In-Reply-To: <CANTn4crN63L4HDECdoYN=ArVza6EpXn7LhPWZBLP=Y09L+bOLw@mail.gmail.com>
References: <CANTn4crN63L4HDECdoYN=ArVza6EpXn7LhPWZBLP=Y09L+bOLw@mail.gmail.com>
Message-ID: <58719.1438337559@critter.freebsd.dk>

--------
In message <CANTn4crN63L4HDECdoYN=ArVza6EpXn7LhPWZBLP=Y09L+bOLw at mail.gmail.com>
, Martin Blix Grydeland writes:

>Attached is a set of patches to fix up the connection handling in Varnish,
>making it more general for both client and backend connections.

I'm sort of torn about struct http_conn right now.

One one hand, it is a decent "pull stuff of a fd" handle, on the other
it seems to be a H1 dead end.

... Zooming out ...

I'm on my way back from the HTTP workshop and I think the short
summary of the workshop is that we are nowhere near done playing
with HTTP transport protocols and that we'll have "profiles" for
HTTP in the future.

The profiles will likely be along the lines of:

* "IOS"
	Internet Of Shit - minimal set for small-CPU with battery

* "API"
	Streamlined "RPC" sort of thing.  Main point: Requests are
	independent, and all the priority/push stuff does not apply.

* "Browser"
	Big, Fat and Heavy.  The full monty where requests depend
	on each other, have priorities, things gets pushed etc. etc.

That probably doesn't affect us much directly.

However transport protocols is a mess.

People have realized that putting H2 on a single TCP/TLS connection
was not the smart move, because everything stalls on packet loss.
Compare that to H1 where the other 5 TCP connections can while the
sixth is stalled.

What is absolutely clear is that TCP isn't a given, and that the
relatively clean layering of H1 over TLS isn't going to carry
forward.

Googles QUIC is a very interesting take on a transport for HTTP,
but it is unlikely to be the final word.

UDP seems to be a given for the "real thing", even if there is also
a layer of SCTP in there somewhere.  TCP is single-stream and the
"browser" profile just cannot be squeezed into that.

UDP means DNS work to do service-discovery and all the carriers who
shape UDP beause its "just P2P" will be upset etc.

TLS1.3 is the key technology here, it's all in the pot and needs to
simmer for some months.  Likely more once the cryptographers stick
their forks in it.

The good news is that TLS1.3 can be done in a lot less code than
OpenSSL.

The bad news is that they have not "as such" paid attention to
small/compact platforms (ie: IOS) and doesn't seem to care much for
it, focusing entirely on the "browser" profile.

There was widespread support for a "blind-caching" model, from
telco-kit people over CDNs to browsers and sites.

The model is essentially that an object is split in two parts.

The body is encrypted and put into a new object which gets no
'semantic' headers, ie: a random URL, a C-L and that's about it.

This object can be cached in caches nobody trust, like a
cell-tower cache or a corporate gateway cache.

The server can send an "indirect object" to a client, the headers
are the real thing, but it has "Content-Type: oob" or some such and
the body contains "fetch the real body using this URL and decrypt
it with this KEY".  The client then picks up URL via the cache.

This scheme saves bandwidth, but not latency, and it has a large
number of use-cases, including video-on-demand, DRM etc.

It won't exactly be our core-case, but VCL is a damn good place to
control the policies for such use.

Speaking of VCL:  Akamai is working on a "VCL to our own config"
converter.  If they succeed, VCL will de-facto become the CDN
language.  I wished them good luck and told them to keep in touch.

There is some hope that we can cooperate with various other people
on H2 test-tools but the bad news is that none of them had seen
anything like varnishtest before, so they hinted an expectation
that we would drag the heavy load since we're "clearly so much
ahead".

H2 plaintext is stuck in "nobody wants to be the first" mode, so
we may end up being the first to deal with H1->H2 upgrade.

I think I got my point about "proportional response" as a smarter
political strategy across and that TLS-everywhere might be counter
productive, but it's probably too late to get meaningful change.

There were lots of semi-heated discussions, but many of them ended
abrubtly with "Ohh, didn't think of *THAT*" when the other party
got a chance to elaborate and explain.

One example was that many browser/site people clearly didn't grasp
the high rates at which load-balancers and varnish operates.  Their
"A modern CPU can do $so_much crypto per core" arguments looked a
lot less impressive once they realized that $so_much were an order
of magnitude below our reality.

Overall it was a very good meeting, but 4 days of concentrated HTTP
was clearly an overdosis for most of us.  We will probably have
more of that kind of pow-wows in the future.

... Zooming back in ...

So it is not obvious to me that struct http_conn will be all that
general in the future.  It is a useful abstraction for "pull bytes
of a socket", and is used as such in H1 and PROXY, but it seems
pretty clear that H2 and later probably wont use it at all.

What we're really looking for here is the non-existent primitives
between the H2 semantic and H2 transport parts :-/

I'll try to come up with some more coherent thoughts.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From fgsch at lodoss.net  Fri Jul 31 12:59:19 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Fri, 31 Jul 2015 13:59:19 +0100
Subject: [PATCH] Kill unused session_max
In-Reply-To: <58347.1438331072@critter.freebsd.dk>
References: <CAJV_h0ZGw1DLQkSCUMix6PsTrY_Qp3MLh+5TCLx7rpnMfwFR_Q@mail.gmail.com>
 <58347.1438331072@critter.freebsd.dk>
Message-ID: <CAJV_h0ZuCUOkJS8MksvtDcHTo2Nz8P7w2rEXt0WZL_XY4GtM8g@mail.gmail.com>

Ok, thanks.

Now that I know we want to keep it I will try to bring it back.

On Fri, Jul 31, 2015 at 9:24 AM, Poul-Henning Kamp <phk at phk.freebsd.dk>
wrote:

> --------
> In message <
> CAJV_h0ZGw1DLQkSCUMix6PsTrY_Qp3MLh+5TCLx7rpnMfwFR_Q at mail.gmail.com>
> , Federico Schwindt writes:
>
> >This looks like a relic from V3.
>
> Sort of, as I said in the previous email, we should be able to fail
> at creating more sessions (cant or wont) and this used to be the limit.
>
> If anything, we should move the other way and make it work again.
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150731/fa5931c2/attachment.html>

From fgsch at lodoss.net  Fri Jul 31 13:00:19 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Fri, 31 Jul 2015 14:00:19 +0100
Subject: [PATCH] Remove unreachable sess_drop
In-Reply-To: <58317.1438330977@critter.freebsd.dk>
References: <CAJV_h0beG7u613+Qy3THYS7cGtE6iawO50J0rBubAjJpT5K-oA@mail.gmail.com>
 <58317.1438330977@critter.freebsd.dk>
Message-ID: <CAJV_h0a4ctQrt7qSP3XsOxcW8zSsWJtY3PdrpXarSw7VBR=R1w@mail.gmail.com>

Ok.

If that's the intention we should correct the doc for sess_drop at the very
least.

I will prepare a patch.

On Fri, Jul 31, 2015 at 9:22 AM, Poul-Henning Kamp <phk at phk.freebsd.dk>
wrote:

> --------
> In message <
> CAJV_h0beG7u613+Qy3THYS7cGtE6iawO50J0rBubAjJpT5K-oA at mail.gmail.com>
> , Federico Schwindt writes:
>
> >SES_New() will never return NULL. Replace noop with AN().
>
> That is actually in preparation/reminder that it should be allowed
> to fail, so I'd prefer to retain it.
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150731/33c44762/attachment.html>

From fgsch at lodoss.net  Fri Jul 31 13:01:32 2015
From: fgsch at lodoss.net (Federico Schwindt)
Date: Fri, 31 Jul 2015 14:01:32 +0100
Subject: Counters in master
In-Reply-To: <CAJV_h0aUvxrL5=0zH=RUS_hBdGtJjr=FNTU9qsBtN2V2-KJNcA@mail.gmail.com>
References: <CAJV_h0aUvxrL5=0zH=RUS_hBdGtJjr=FNTU9qsBtN2V2-KJNcA@mail.gmail.com>
Message-ID: <CAJV_h0YgA9EjBF8XYEZxsc=vhTYSbaCa1xCOkaFoaKyCk_9=YA@mail.gmail.com>

Anyone?

On Mon, Jul 13, 2015 at 7:35 PM, Federico Schwindt <fgsch at lodoss.net> wrote:

> Hi,
>
> Below is a summary of counters removed and renamed in master wrt 4.0 and
> IMO what we need to bring back for 4.1. Unless specifically mentioned they
> are relative to MAIN.
>
> Removed:
>
> backend_toolate - IIUC cannot happen anymore
> client_req_413 - unused in 4.0
> fetch_close
> fetch_oldhttp - used to indicate backends < 1.1
> VBE.<backend exp>.vcls - no longer needed
>
> Renamed:
>
> fetch_none (was fetch_zero)
> sc_pipe_overflow (was sess_pipe_overflow)
> rx_body (was client_req_411)
>
> Broken:
>
> backend_conn is never incremented (#1725)
>
> To be in parity with 4.0 we should address #1725.
> Not sure if there is any value in fetch_oldhttp or fetch_close.
>
> As for counters we might want to add, several people have mentioned the
> need for per-backend errors. There are 3 places in cache_backend.c that
> have the "XXX: Per backend stats ?" comment that we can start with and what
> the attached patch covers.
> That said, this might be post 4.1 material.
>
> Haven't looked at #1725 yet.
>
> f.-
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20150731/60a1c701/attachment.html>

From phk at phk.freebsd.dk  Fri Jul 31 15:07:33 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 31 Jul 2015 15:07:33 +0000
Subject: Counters in master
In-Reply-To: <CAJV_h0YgA9EjBF8XYEZxsc=vhTYSbaCa1xCOkaFoaKyCk_9=YA@mail.gmail.com>
References: <CAJV_h0aUvxrL5=0zH=RUS_hBdGtJjr=FNTU9qsBtN2V2-KJNcA@mail.gmail.com>
 <CAJV_h0YgA9EjBF8XYEZxsc=vhTYSbaCa1xCOkaFoaKyCk_9=YA@mail.gmail.com>
Message-ID: <1330.1438355253@critter.freebsd.dk>

--------
In message <CAJV_h0YgA9EjBF8XYEZxsc=vhTYSbaCa1xCOkaFoaKyCk_9=YA at mail.gmail.com>
, Federico Schwindt writes:

>Anyone?

I just came home from M?nster, and havn't even had a shower yet,
gimme a chance ?  :-)


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.