From phk at phk.freebsd.dk  Mon Sep  1 06:48:43 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 01 Sep 2014 06:48:43 +0000
Subject: what is the ttl?
In-Reply-To: <54008149.5090408@schokola.de>
References: <53FCCB1D.5070707@schokola.de>
 <CAOXZevBHCC=FpKfm7s9MVK==NFjObnKM0hYRhSPsdVfixSyehQ@mail.gmail.com>
 <53FE2354.1000908@schokola.de>
 <CAOXZevB4C2-V-MeUnkn46SfTNB+umS0vHtQWbcZYacjrMtK5GA@mail.gmail.com>
 <53FEC893.5060401@schokola.de> <53FECD9E.4040009@uplex.de>
 <53FECF80.8090506@schokola.de> <53FF168B.5020405@uplex.de>
 <1316.1409227062@critter.freebsd.dk>
 <CAOXZevDi6GpRw8Fd+5akxek+V295saV9KneuN1q1r3vKayy2og@mail.gmail.com>
 <54008149.5090408@schokola.de>
Message-ID: <49884.1409554123@critter.freebsd.dk>

--------
In message <54008149.5090408 at schokola.de>, Nils Goroll writes:
>After the helpful discussion on the topic I'm with phk now and I think we should
>just add (beresp|obj).age and leave the ttl as is.

Done.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From martin at varnish-software.com  Mon Sep  1 11:06:14 2014
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 1 Sep 2014 13:06:14 +0200
Subject: [PATCH] Make random/hash director exhaust the backend list when
 looking for healthy backend.
In-Reply-To: <CAJV_h0bWT-4r+OgP2t24n9S0e3nR_QCL0eyOf9mX=bUjvdejZg@mail.gmail.com>
References: <1409316510-23073-1-git-send-email-martin@varnish-software.com>
 <CAJV_h0bWT-4r+OgP2t24n9S0e3nR_QCL0eyOf9mX=bUjvdejZg@mail.gmail.com>
Message-ID: <CANTn4cp91s0PWBsdLVP4WdVjf7OTJEM3LkXN3p_dGo-t5KAK3A@mail.gmail.com>

On 29 August 2014 15:10, Federico Schwindt <fgsch at lodoss.net> wrote:

> Hi,
>
> Any reason not to use rr->vd->n_backend directly as the round-robin
> director does?
>

The n_backend of the vdir.[ch] is opaque to the director implementations. I
added the new attribute to keep it that way.

Martin


>
>
> On Fri, Aug 29, 2014 at 1:48 PM, Martin Blix Grydeland <
> martin at varnish-software.com> wrote:
>
>> Fixes: #1575
>> ---
>>  bin/varnishtest/tests/r01575.vtc | 56
>> ++++++++++++++++++++++++++++++++++++++++
>>  lib/libvmod_directors/hash.c     |  6 ++---
>>  lib/libvmod_directors/random.c   |  6 ++---
>>  lib/libvmod_directors/vdir.c     |  2 +-
>>  4 files changed, 63 insertions(+), 7 deletions(-)
>>  create mode 100644 bin/varnishtest/tests/r01575.vtc
>>
>> diff --git a/bin/varnishtest/tests/r01575.vtc
>> b/bin/varnishtest/tests/r01575.vtc
>> new file mode 100644
>> index 0000000..11c82c1
>> --- /dev/null
>> +++ b/bin/varnishtest/tests/r01575.vtc
>> @@ -0,0 +1,56 @@
>> +varnishtest "#1575 - random director exhaust backend list"
>> +
>> +# Add 5 backends to a random director, with the 5th having very low
>> weight.
>> +# Mark the first 4 sick, and make sure that the 5th will be selected.
>> +
>> +server s1 {
>> +       rxreq
>> +       txresp
>> +} -start
>> +
>> +server s2 {
>> +       rxreq
>> +       txresp
>> +} -start
>> +
>> +server s3 {
>> +       rxreq
>> +       txresp
>> +} -start
>> +
>> +server s4 {
>> +       rxreq
>> +       txresp
>> +} -start
>> +
>> +server s5 {
>> +       rxreq
>> +       txresp
>> +} -start
>> +
>> +varnish v1 -vcl+backend {
>> +       import ${vmod_directors};
>> +       sub vcl_init {
>> +               new rd = directors.random();
>> +               rd.add_backend(s1, 10000);
>> +               rd.add_backend(s2, 10000);
>> +               rd.add_backend(s3, 10000);
>> +               rd.add_backend(s4, 10000);
>> +               rd.add_backend(s5, 1);
>> +       }
>> +
>> +       sub vcl_backend_fetch {
>> +               set bereq.backend = rd.backend();
>> +       }
>> +} -start
>> +
>> +varnish v1 -cliok "backend.set_health s1 sick"
>> +varnish v1 -cliok "backend.set_health s2 sick"
>> +varnish v1 -cliok "backend.set_health s3 sick"
>> +varnish v1 -cliok "backend.set_health s4 sick"
>> +
>> +client c1 {
>> +       txreq
>> +       rxresp
>> +       expect resp.status == 200
>> +} -run
>> diff --git a/lib/libvmod_directors/hash.c b/lib/libvmod_directors/hash.c
>> index afef7ed..090039f 100644
>> --- a/lib/libvmod_directors/hash.c
>> +++ b/lib/libvmod_directors/hash.c
>> @@ -47,7 +47,7 @@ struct vmod_directors_hash {
>>         unsigned                                magic;
>>  #define VMOD_DIRECTORS_HASH_MAGIC              0xc08dd611
>>         struct vdir                             *vd;
>> -       unsigned                                nloops;
>> +       unsigned                                n_backend;
>>         struct vbitmap                          *vbm;
>>  };
>>
>> @@ -64,7 +64,6 @@ vmod_hash__init(const struct vrt_ctx *ctx, struct
>> vmod_directors_hash **rrp,
>>         AN(rr);
>>         rr->vbm = vbit_init(8);
>>         AN(rr->vbm);
>> -       rr->nloops = 3; //
>>         *rrp = rr;
>>         vdir_new(&rr->vd, vcl_name, NULL, NULL, rr);
>>  }
>> @@ -90,6 +89,7 @@ vmod_hash_add_backend(const struct vrt_ctx *ctx,
>>         CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
>>         CHECK_OBJ_NOTNULL(rr, VMOD_DIRECTORS_HASH_MAGIC);
>>         (void)vdir_add_backend(rr->vd, be, w);
>> +       rr->n_backend++;
>>  }
>>
>>  VCL_BACKEND __match_proto__()
>> @@ -120,6 +120,6 @@ vmod_hash_backend(const struct vrt_ctx *ctx, struct
>> vmod_directors_hash *rr,
>>         r = vbe32dec(sha256);
>>         r = scalbn(r, -32);
>>         assert(r >= 0 && r <= 1.0);
>> -       be = vdir_pick_be(rr->vd, r, rr->nloops);
>> +       be = vdir_pick_be(rr->vd, r, rr->n_backend);
>>         return (be);
>>  }
>> diff --git a/lib/libvmod_directors/random.c
>> b/lib/libvmod_directors/random.c
>> index 22f0bb9..8ae36a7 100644
>> --- a/lib/libvmod_directors/random.c
>> +++ b/lib/libvmod_directors/random.c
>> @@ -45,7 +45,7 @@ struct vmod_directors_random {
>>         unsigned                                magic;
>>  #define VMOD_DIRECTORS_RANDOM_MAGIC            0x4732d092
>>         struct vdir                             *vd;
>> -       unsigned                                nloops;
>> +       unsigned                                n_backend;
>>         struct vbitmap                          *vbm;
>>  };
>>
>> @@ -68,7 +68,7 @@ vmod_rr_getfd(const struct director *dir, struct
>> busyobj *bo)
>>         CAST_OBJ_NOTNULL(rr, dir->priv, VMOD_DIRECTORS_RANDOM_MAGIC);
>>         r = scalbn(random(), -31);
>>         assert(r >= 0 && r < 1.0);
>> -       be = vdir_pick_be(rr->vd, r, rr->nloops);
>> +       be = vdir_pick_be(rr->vd, r, rr->n_backend);
>>         if (be == NULL)
>>                 return (NULL);
>>         return (be->getfd(be, bo));
>> @@ -87,7 +87,6 @@ vmod_random__init(const struct vrt_ctx *ctx, struct
>> vmod_directors_random **rrp,
>>         AN(rr);
>>         rr->vbm = vbit_init(8);
>>         AN(rr->vbm);
>> -       rr->nloops = 3; //
>>         *rrp = rr;
>>         vdir_new(&rr->vd, vcl_name, vmod_rr_healthy, vmod_rr_getfd, rr);
>>  }
>> @@ -113,6 +112,7 @@ vmod_random_add_backend(const struct vrt_ctx *ctx,
>>         CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
>>         CHECK_OBJ_NOTNULL(rr, VMOD_DIRECTORS_RANDOM_MAGIC);
>>         (void)vdir_add_backend(rr->vd, be, w);
>> +       rr->n_backend++;
>>  }
>>
>>  VCL_BACKEND __match_proto__()
>> diff --git a/lib/libvmod_directors/vdir.c b/lib/libvmod_directors/vdir.c
>> index 12268eb..aae7a14 100644
>> --- a/lib/libvmod_directors/vdir.c
>> +++ b/lib/libvmod_directors/vdir.c
>> @@ -186,7 +186,7 @@ vdir_pick_be(struct vdir *vd, double w, unsigned
>> nloops)
>>         nbe = vd->n_backend;
>>         assert(w >= 0.0 && w < 1.0);
>>         vdir_lock(vd);
>> -       for (l = 0; nbe > 0 && tw > 0.0 && l <nloops; l++) {
>> +       for (l = 0; nbe > 0 && tw > 0.0 && l < nloops; l++) {
>>                 u = vdir_pick_by_weight(vd, w * tw, vbm);
>>                 be = vd->backend[u];
>>                 CHECK_OBJ_NOTNULL(be, DIRECTOR_MAGIC);
>> --
>> 2.1.0.rc1
>>
>>
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>
>
>


-- 
<http://varnish-software.com>*Martin Blix Grydeland*
Senior Developer | Varnish Software AS
Cell: +47 21 98 92 60
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140901/e7cf313d/attachment.html>

From fgsch at lodoss.net  Mon Sep  1 11:17:55 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Mon, 1 Sep 2014 12:17:55 +0100
Subject: [PATCH] Make random/hash director exhaust the backend list when
 looking for healthy backend.
In-Reply-To: <CANTn4cp91s0PWBsdLVP4WdVjf7OTJEM3LkXN3p_dGo-t5KAK3A@mail.gmail.com>
References: <1409316510-23073-1-git-send-email-martin@varnish-software.com>
 <CAJV_h0bWT-4r+OgP2t24n9S0e3nR_QCL0eyOf9mX=bUjvdejZg@mail.gmail.com>
 <CANTn4cp91s0PWBsdLVP4WdVjf7OTJEM3LkXN3p_dGo-t5KAK3A@mail.gmail.com>
Message-ID: <CAJV_h0aRo3g+hRA=UgHPH8jzLhM0CUKAjgqcq1FdW-mCki4Ujg@mail.gmail.com>

Good point.

Then this looks like I had in mind. OK for me.


On Mon, Sep 1, 2014 at 12:06 PM, Martin Blix Grydeland <
martin at varnish-software.com> wrote:

>
> On 29 August 2014 15:10, Federico Schwindt <fgsch at lodoss.net> wrote:
>
>> Hi,
>>
>> Any reason not to use rr->vd->n_backend directly as the round-robin
>> director does?
>>
>
> The n_backend of the vdir.[ch] is opaque to the director implementations.
> I added the new attribute to keep it that way.
>
> Martin
>
>
>>
>>
>> On Fri, Aug 29, 2014 at 1:48 PM, Martin Blix Grydeland <
>> martin at varnish-software.com> wrote:
>>
>>> Fixes: #1575
>>> ---
>>>  bin/varnishtest/tests/r01575.vtc | 56
>>> ++++++++++++++++++++++++++++++++++++++++
>>>  lib/libvmod_directors/hash.c     |  6 ++---
>>>  lib/libvmod_directors/random.c   |  6 ++---
>>>  lib/libvmod_directors/vdir.c     |  2 +-
>>>  4 files changed, 63 insertions(+), 7 deletions(-)
>>>  create mode 100644 bin/varnishtest/tests/r01575.vtc
>>>
>>> diff --git a/bin/varnishtest/tests/r01575.vtc
>>> b/bin/varnishtest/tests/r01575.vtc
>>> new file mode 100644
>>> index 0000000..11c82c1
>>> --- /dev/null
>>> +++ b/bin/varnishtest/tests/r01575.vtc
>>> @@ -0,0 +1,56 @@
>>> +varnishtest "#1575 - random director exhaust backend list"
>>> +
>>> +# Add 5 backends to a random director, with the 5th having very low
>>> weight.
>>> +# Mark the first 4 sick, and make sure that the 5th will be selected.
>>> +
>>> +server s1 {
>>> +       rxreq
>>> +       txresp
>>> +} -start
>>> +
>>> +server s2 {
>>> +       rxreq
>>> +       txresp
>>> +} -start
>>> +
>>> +server s3 {
>>> +       rxreq
>>> +       txresp
>>> +} -start
>>> +
>>> +server s4 {
>>> +       rxreq
>>> +       txresp
>>> +} -start
>>> +
>>> +server s5 {
>>> +       rxreq
>>> +       txresp
>>> +} -start
>>> +
>>> +varnish v1 -vcl+backend {
>>> +       import ${vmod_directors};
>>> +       sub vcl_init {
>>> +               new rd = directors.random();
>>> +               rd.add_backend(s1, 10000);
>>> +               rd.add_backend(s2, 10000);
>>> +               rd.add_backend(s3, 10000);
>>> +               rd.add_backend(s4, 10000);
>>> +               rd.add_backend(s5, 1);
>>> +       }
>>> +
>>> +       sub vcl_backend_fetch {
>>> +               set bereq.backend = rd.backend();
>>> +       }
>>> +} -start
>>> +
>>> +varnish v1 -cliok "backend.set_health s1 sick"
>>> +varnish v1 -cliok "backend.set_health s2 sick"
>>> +varnish v1 -cliok "backend.set_health s3 sick"
>>> +varnish v1 -cliok "backend.set_health s4 sick"
>>> +
>>> +client c1 {
>>> +       txreq
>>> +       rxresp
>>> +       expect resp.status == 200
>>> +} -run
>>> diff --git a/lib/libvmod_directors/hash.c b/lib/libvmod_directors/hash.c
>>> index afef7ed..090039f 100644
>>> --- a/lib/libvmod_directors/hash.c
>>> +++ b/lib/libvmod_directors/hash.c
>>> @@ -47,7 +47,7 @@ struct vmod_directors_hash {
>>>         unsigned                                magic;
>>>  #define VMOD_DIRECTORS_HASH_MAGIC              0xc08dd611
>>>         struct vdir                             *vd;
>>> -       unsigned                                nloops;
>>> +       unsigned                                n_backend;
>>>         struct vbitmap                          *vbm;
>>>  };
>>>
>>> @@ -64,7 +64,6 @@ vmod_hash__init(const struct vrt_ctx *ctx, struct
>>> vmod_directors_hash **rrp,
>>>         AN(rr);
>>>         rr->vbm = vbit_init(8);
>>>         AN(rr->vbm);
>>> -       rr->nloops = 3; //
>>>         *rrp = rr;
>>>         vdir_new(&rr->vd, vcl_name, NULL, NULL, rr);
>>>  }
>>> @@ -90,6 +89,7 @@ vmod_hash_add_backend(const struct vrt_ctx *ctx,
>>>         CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
>>>         CHECK_OBJ_NOTNULL(rr, VMOD_DIRECTORS_HASH_MAGIC);
>>>         (void)vdir_add_backend(rr->vd, be, w);
>>> +       rr->n_backend++;
>>>  }
>>>
>>>  VCL_BACKEND __match_proto__()
>>> @@ -120,6 +120,6 @@ vmod_hash_backend(const struct vrt_ctx *ctx, struct
>>> vmod_directors_hash *rr,
>>>         r = vbe32dec(sha256);
>>>         r = scalbn(r, -32);
>>>         assert(r >= 0 && r <= 1.0);
>>> -       be = vdir_pick_be(rr->vd, r, rr->nloops);
>>> +       be = vdir_pick_be(rr->vd, r, rr->n_backend);
>>>         return (be);
>>>  }
>>> diff --git a/lib/libvmod_directors/random.c
>>> b/lib/libvmod_directors/random.c
>>> index 22f0bb9..8ae36a7 100644
>>> --- a/lib/libvmod_directors/random.c
>>> +++ b/lib/libvmod_directors/random.c
>>> @@ -45,7 +45,7 @@ struct vmod_directors_random {
>>>         unsigned                                magic;
>>>  #define VMOD_DIRECTORS_RANDOM_MAGIC            0x4732d092
>>>         struct vdir                             *vd;
>>> -       unsigned                                nloops;
>>> +       unsigned                                n_backend;
>>>         struct vbitmap                          *vbm;
>>>  };
>>>
>>> @@ -68,7 +68,7 @@ vmod_rr_getfd(const struct director *dir, struct
>>> busyobj *bo)
>>>         CAST_OBJ_NOTNULL(rr, dir->priv, VMOD_DIRECTORS_RANDOM_MAGIC);
>>>         r = scalbn(random(), -31);
>>>         assert(r >= 0 && r < 1.0);
>>> -       be = vdir_pick_be(rr->vd, r, rr->nloops);
>>> +       be = vdir_pick_be(rr->vd, r, rr->n_backend);
>>>         if (be == NULL)
>>>                 return (NULL);
>>>         return (be->getfd(be, bo));
>>> @@ -87,7 +87,6 @@ vmod_random__init(const struct vrt_ctx *ctx, struct
>>> vmod_directors_random **rrp,
>>>         AN(rr);
>>>         rr->vbm = vbit_init(8);
>>>         AN(rr->vbm);
>>> -       rr->nloops = 3; //
>>>         *rrp = rr;
>>>         vdir_new(&rr->vd, vcl_name, vmod_rr_healthy, vmod_rr_getfd, rr);
>>>  }
>>> @@ -113,6 +112,7 @@ vmod_random_add_backend(const struct vrt_ctx *ctx,
>>>         CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
>>>         CHECK_OBJ_NOTNULL(rr, VMOD_DIRECTORS_RANDOM_MAGIC);
>>>         (void)vdir_add_backend(rr->vd, be, w);
>>> +       rr->n_backend++;
>>>  }
>>>
>>>  VCL_BACKEND __match_proto__()
>>> diff --git a/lib/libvmod_directors/vdir.c b/lib/libvmod_directors/vdir.c
>>> index 12268eb..aae7a14 100644
>>> --- a/lib/libvmod_directors/vdir.c
>>> +++ b/lib/libvmod_directors/vdir.c
>>> @@ -186,7 +186,7 @@ vdir_pick_be(struct vdir *vd, double w, unsigned
>>> nloops)
>>>         nbe = vd->n_backend;
>>>         assert(w >= 0.0 && w < 1.0);
>>>         vdir_lock(vd);
>>> -       for (l = 0; nbe > 0 && tw > 0.0 && l <nloops; l++) {
>>> +       for (l = 0; nbe > 0 && tw > 0.0 && l < nloops; l++) {
>>>                 u = vdir_pick_by_weight(vd, w * tw, vbm);
>>>                 be = vd->backend[u];
>>>                 CHECK_OBJ_NOTNULL(be, DIRECTOR_MAGIC);
>>> --
>>> 2.1.0.rc1
>>>
>>>
>>> _______________________________________________
>>> varnish-dev mailing list
>>> varnish-dev at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>>
>>
>>
>
>
> --
>  <http://varnish-software.com>*Martin Blix Grydeland*
> Senior Developer | Varnish Software AS
> Cell: +47 21 98 92 60
> We Make Websites Fly!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140901/d326813c/attachment.html>

From martin at varnish-software.com  Mon Sep  1 12:02:35 2014
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon,  1 Sep 2014 14:02:35 +0200
Subject: [PATCH 1/2] Enable batch record usage in logexpect
Message-ID: <1409572956-28584-1-git-send-email-martin@varnish-software.com>

This enables catching batch related log problems using logexpect.

This commit will make several test cases start failing until #1588 has been fixed.
---
 bin/varnishtest/vtc_logexp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bin/varnishtest/vtc_logexp.c b/bin/varnishtest/vtc_logexp.c
index d868a1b..a2b6323 100644
--- a/bin/varnishtest/vtc_logexp.c
+++ b/bin/varnishtest/vtc_logexp.c
@@ -330,7 +330,8 @@ logexp_start(struct logexp *le)
 	}
 	le->vsl = VSL_New();
 	AN(le->vsl);
-	c = VSL_CursorVSM(le->vsl, le->vsm, le->d_arg ? 0 : VSL_COPT_TAIL);
+	c = VSL_CursorVSM(le->vsl, le->vsm,
+	    (le->d_arg ? 0 : VSL_COPT_TAIL) | VSL_COPT_BATCH);
 	if (c == NULL) {
 		vtc_log(le->vl, 0, "VSL_CursorVSM: %s", VSL_Error(le->vsl));
 		logexp_close(le);
-- 
2.1.0




From martin at varnish-software.com  Mon Sep  1 12:02:36 2014
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon,  1 Sep 2014 14:02:36 +0200
Subject: [PATCH 2/2] Allocate request VXID only when it's needed and delay all
 logging until it's been allocated.
In-Reply-To: <1409572956-28584-1-git-send-email-martin@varnish-software.com>
References: <1409572956-28584-1-git-send-email-martin@varnish-software.com>
Message-ID: <1409572956-28584-2-git-send-email-martin@varnish-software.com>

This patch delays the VXID allocation until the time we know that we
will actually be logging anything for this request (not completely all
whitespace noise). This keeps the code in one place only (plus the ESI
mock request handling), instead of separated between the linger wait,
cache acceptor and pipelining code paths.

Coincidentally fixes the logging of timestamps problem in #1588.

Test case by daghf.

Fixes: #1581 #1588
---
 bin/varnishd/cache/cache_esi_deliver.c |  3 ++-
 bin/varnishd/cache/cache_fetch.c       |  4 ++--
 bin/varnishd/cache/cache_http1_fsm.c   | 39 +++++++++++++++-------------------
 bin/varnishd/cache/cache_session.c     |  4 ----
 bin/varnishtest/tests/r01581.vtc       | 23 ++++++++++++++++++++
 5 files changed, 44 insertions(+), 29 deletions(-)
 create mode 100644 bin/varnishtest/tests/r01581.vtc

diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c
index 55c8901..d272bfc 100644
--- a/bin/varnishd/cache/cache_esi_deliver.c
+++ b/bin/varnishd/cache/cache_esi_deliver.c
@@ -62,7 +62,8 @@ ved_include(struct req *preq, const char *src, const char *host)
 
 	req = SES_GetReq(wrk, preq->sp);
 	req->req_body_status = REQ_BODY_NONE;
-	AN(req->vsl->wid & VSL_CLIENTMARKER);
+	AZ(req->vsl->wid);
+	req->vsl->wid = VXID_Get(wrk, VSL_CLIENTMARKER);
 	VSLb(req->vsl, SLT_Begin, "req %u esi", VXID(preq->vsl->wid));
 	VSLb(preq->vsl, SLT_Link, "req %u esi", VXID(req->vsl->wid));
 	req->esi_level = preq->esi_level + 1;
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 001cd79..77c1e8c 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -822,8 +822,8 @@ VBF_Fetch(struct worker *wrk, struct req *req, struct objcore *oc,
 
 	bo = VBO_GetBusyObj(wrk, req);
 	CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC);
-	VSLb(bo->vsl, SLT_Begin, "bereq %u %s ", VXID(req->vsl->wid), how);
-	VSLb(req->vsl, SLT_Link, "bereq %u %s ", VXID(bo->vsl->wid), how);
+	VSLb(bo->vsl, SLT_Begin, "bereq %u %s", VXID(req->vsl->wid), how);
+	VSLb(req->vsl, SLT_Link, "bereq %u %s", VXID(bo->vsl->wid), how);
 
 	THR_SetBusyobj(bo);
 
diff --git a/bin/varnishd/cache/cache_http1_fsm.c b/bin/varnishd/cache/cache_http1_fsm.c
index b0c5299..98b3dcf 100644
--- a/bin/varnishd/cache/cache_http1_fsm.c
+++ b/bin/varnishd/cache/cache_http1_fsm.c
@@ -122,9 +122,9 @@ http1_wait(struct sess *sp, struct worker *wrk, struct req *req)
 		if (hs == HTTP1_COMPLETE) {
 			/* Got it, run with it */
 			if (isnan(req->t_first))
-				VSLb_ts_req(req, "Start", now);
-			VSLb_ts_req(req, "Req", now);
-			req->t_req = req->t_prev;
+				req->t_first = now;
+			if (isnan(req->t_req))
+				req->t_req = now;
 			req->acct.req_hdrbytes += Tlen(req->htc->rxbuf);
 			return (REQ_FSM_MORE);
 		} else if (hs == HTTP1_ERROR_EOF) {
@@ -151,8 +151,8 @@ http1_wait(struct sess *sp, struct worker *wrk, struct req *req)
 		} else {
 			/* Working on it */
 			if (isnan(req->t_first))
-				/* Timestamp Start on first byte received */
-				VSLb_ts_req(req, "Start", now);
+				/* Record first byte received time stamp */
+				req->t_first = now;
 			when = sp->t_idle + cache_param->timeout_req;
 			tmo = (int)(1e3 * (when - now));
 			if (when < now || tmo == 0) {
@@ -239,13 +239,7 @@ http1_cleanup(struct sess *sp, struct worker *wrk, struct req *req)
 
 	if (HTTP1_Reinit(req->htc) == HTTP1_COMPLETE) {
 		AZ(req->vsl->wid);
-		req->vsl->wid = VXID_Get(wrk, VSL_CLIENTMARKER);
-		VSLb(req->vsl, SLT_Begin, "req %u rxreq", VXID(req->sp->vxid));
-		VSL(SLT_Link, req->sp->vxid, "req %u rxreq",
-		    VXID(req->vsl->wid));
-		VSLb_ts_req(req, "Start", sp->t_idle);
-		VSLb_ts_req(req, "Req", sp->t_idle);
-		req->t_req = req->t_prev;
+		req->t_first = req->t_req = sp->t_idle;
 		wrk->stats.sess_pipeline++;
 		req->acct.req_hdrbytes += Tlen(req->htc->rxbuf);
 		return (SESS_DONE_RET_START);
@@ -305,16 +299,17 @@ http1_dissect(struct worker *wrk, struct req *req)
 
 	memset(&req->h1, 0, sizeof req->h1);
 
-	/*
-	 * Cache_req_fsm zeros the vxid once a requests is processed.
-	 * Allocate a new one only now that we know will need it.
-	 */
-	if (req->vsl->wid == 0) {
-		req->vsl->wid = VXID_Get(wrk, VSL_CLIENTMARKER);
-		VSLb(req->vsl, SLT_Begin, "req %u rxreq", VXID(req->sp->vxid));
-		VSL(SLT_Link, req->sp->vxid, "req %u rxreq",
-		    VXID(req->vsl->wid));
-	}
+	/* Allocate a new vxid now that we know we'll need it. */
+	AZ(req->vsl->wid);
+	req->vsl->wid = VXID_Get(wrk, VSL_CLIENTMARKER);
+
+	VSLb(req->vsl, SLT_Begin, "req %u rxreq", VXID(req->sp->vxid));
+	VSL(SLT_Link, req->sp->vxid, "req %u rxreq", VXID(req->vsl->wid));
+	AZ(isnan(req->t_first)); /* First byte timestamp set by http1_wait */
+	AZ(isnan(req->t_req));	 /* Complete req rcvd set by http1_wait */
+	req->t_prev = req->t_first;
+	VSLb_ts_req(req, "Start", req->t_first);
+	VSLb_ts_req(req, "Req", req->t_req);
 
 	/* Borrow VCL reference from worker thread */
 	VCL_Refresh(&wrk->vcl);
diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c
index 3b22ce7..bd48b45 100644
--- a/bin/varnishd/cache/cache_session.c
+++ b/bin/varnishd/cache/cache_session.c
@@ -133,9 +133,6 @@ ses_sess_pool_task(struct worker *wrk, void *arg)
 
 	req = SES_GetReq(wrk, sp);
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
-	AN(req->vsl->wid & VSL_CLIENTMARKER);
-	VSLb(req->vsl, SLT_Begin, "req %u rxreq", VXID(sp->vxid));
-	VSL(SLT_Link, sp->vxid, "req %u rxreq", VXID(req->vsl->wid));
 
 	sp->sess_step = S_STP_NEWREQ;
 	ses_req_pool_task(wrk, req);
@@ -362,7 +359,6 @@ SES_GetReq(struct worker *wrk, struct sess *sp)
 
 	sz = cache_param->vsl_buffer;
 	VSL_Setup(req->vsl, p, sz);
-	req->vsl->wid = VXID_Get(wrk, VSL_CLIENTMARKER);
 	p += sz;
 	p = (void*)PRNDUP(p);
 
diff --git a/bin/varnishtest/tests/r01581.vtc b/bin/varnishtest/tests/r01581.vtc
new file mode 100644
index 0000000..f04eb0e
--- /dev/null
+++ b/bin/varnishtest/tests/r01581.vtc
@@ -0,0 +1,23 @@
+varnishtest "Duplcate SLT_Begin"
+
+server s1 {
+	rxreq
+	txresp
+} -start
+
+varnish v1 -vcl+backend {
+} -start
+
+logexpect l1 -v v1 -g request {
+	  expect 0 1001 Begin
+	  expect * =	ReqStart
+	  expect * = 	End
+} -start
+
+client c1 {
+	delay 0.1
+	send "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n"
+	rxresp
+} -run
+
+logexpect l1 -wait
-- 
2.1.0




From slink at schokola.de  Mon Sep  1 17:01:44 2014
From: slink at schokola.de (Nils Goroll)
Date: Mon, 01 Sep 2014 19:01:44 +0200
Subject: proposal part 1 - VCL: edit header tokens/entities
In-Reply-To: <18938.1409209079@critter.freebsd.dk>
References: <53FCCB1D.5070707@schokola.de>
 <CAOXZevBHCC=FpKfm7s9MVK==NFjObnKM0hYRhSPsdVfixSyehQ@mail.gmail.com>
 <53FE2354.1000908@schokola.de>
 <CAOXZevB4C2-V-MeUnkn46SfTNB+umS0vHtQWbcZYacjrMtK5GA@mail.gmail.com>
 <53FEC893.5060401@schokola.de> <18938.1409209079@critter.freebsd.dk>
Message-ID: <5404A678.8010602@schokola.de>

On 28/08/14 08:57, Poul-Henning Kamp wrote:
> Btw, mildly related to this:  Long time ago we talked about being able
> to work on sub-parts of http headers:
> 
> 	if (INT(beresp.http.cache-control.max-age, 0) > 1200) {
> 		set beresp.http.cache-contro.max-age = 1200;
> 	}
> 
> We may want to reconsider this before we write too intricate VCL

phk had asked me to come up with a sketch of which VRT functions we'd need for
this, but I'd rather get some input about several ways to define the VCL first.

The general idea is that, while whole headers can be accessed using

	[un]set where.http.headername

tokens/entities in headers could be accessed using

	[un]set where.http.headername.token
and
	[un]set where.http.headername.token.option

VCL/VCC would know that for most headers elements are split by comma (,) and
key/value pairs are split by equals (=) and it would be aware of exceptions for
other well known headers (for instance the Cookie request header, if we take it
into scope).


The first issue here is how we denote token and option in VCL.

According to http://tools.ietf.org/html/rfc7230#section-3.2 we cannot assume
some "special" character not to be contained in a header token/entity name or value.

One suggestion we have discussed on irc was to use the array notation, but this
would make it necessary to quote [ and ], so

	set beresp.http.Foo.[g\[oo\]g\\l"e] = "watching";

would yield

	Foo: g[oo]g\l"e=watching

if there was no Foo header (and add it otherwise).


The best suggestion for a syntax I could think of so far is to use the existing
string notation, so

	set beresp.http.Foo.{"g[oo]g\l"e"} = "watching";

would yield the same result.

Some more examples (hopefully slightly more realistic) using this notation with
before and after

example #1

		Accept: text/plain; q=0.5, text/x-dvi, */*

	set req.http.Accept."text/plain"."q" = 3.14;

		-> Accept: text/plain; q=3.14, text/x-dvi, */*

	set req.http.Accept.{""quoted"/printable"};

		-> Accept: text/plain; q=3.14, text/x-dvi, */*, "quoted"/printable

	unset req.http.Accept."*/*";

		-> Accept: text/plain; q=3.14, text/x-dvi, "quoted"/printable


example #2

		Vary: Accept-Language

	set beresp.http.Vary."Accept-Encoding";

		-> Vary: Accept-Language, Accept-Encoding

	unset beresp.http.Vary."Accept-Language";

		-> Vary: Accept-Encoding

	unset beresp.http.Vary."Accept-Encoding";

		-> (void) - header removed

Feedback welcome.



From geoff at uplex.de  Tue Sep  2 06:33:51 2014
From: geoff at uplex.de (Geoff Simmons)
Date: Tue, 02 Sep 2014 08:33:51 +0200
Subject: proposal part 1 - VCL: edit header tokens/entities
In-Reply-To: <5404A678.8010602@schokola.de>
References: <53FCCB1D.5070707@schokola.de>
 <CAOXZevBHCC=FpKfm7s9MVK==NFjObnKM0hYRhSPsdVfixSyehQ@mail.gmail.com>
 <53FE2354.1000908@schokola.de>
 <CAOXZevB4C2-V-MeUnkn46SfTNB+umS0vHtQWbcZYacjrMtK5GA@mail.gmail.com>
 <53FEC893.5060401@schokola.de> <18938.1409209079@critter.freebsd.dk>
 <5404A678.8010602@schokola.de>
Message-ID: <540564CF.8080204@uplex.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 9/1/14 7:01 PM, Nils Goroll wrote:
> 
> According to http://tools.ietf.org/html/rfc7230#section-3.2 we
> cannot assume some "special" character not to be contained in a
> header token/entity name or value.

Maybe so, but in practice this sort of example is extremely unusual;
it would be OK if the syntax necessary for it is cumbersome.

> Foo: g[oo]g\l"e=watching

> The best suggestion for a syntax I could think of so far is to use
> the existing string notation, so
> 
> set beresp.http.Foo.{"g[oo]g\l"e"} = "watching";
> 
> would yield the same result.

It looks like this would work technically, but consider that VCL
strives for syntax that is familiar from other familiar languages. I
personally have never seen anything like that; "that looks weird"
might be a common user feedback.

Suggestion: stay with the quoted strings, but enclose them in square
brackets rather than use the dot notation. Then it looks the syntax
for associative arrays used by shells, Perl, Ruby, PHP and so forth.

set req.http.Accept["q"] = 3.14;
unset req.http.Accept["*/*"];
if (beresp.http.Surrogate-Control["max-age"] ~ "\d+")
if (beresp.http.Surrogate-Control["max-age"]) // boolean context

It gets a little repetitive with brackets when the string includes
quotation marks, then we'd have something like [{"..."}], but I think
that's OK.

I'm not a big fan of:

> Accept: text/plain; q=0.5, text/x-dvi, */*
> 
> set req.http.Accept."text/plain"."q" = 3.14;
> 
> -> Accept: text/plain; q=3.14, text/x-dvi, */*

I had to look at that several times to realize what's going on -- the
operation adds 'text/plain; q=3.14' to a comma-separated list. At
first glance it looked like the set operation was doing two things at
once, adding both 'text/plain', which is not set to anything, and 'q',
which is set to the right-hand side.

You'd think it should actually be (my notation suggestion this time):

set req.http.Accept["text/plain; q"] = 3.14:

Or of:

> set req.http.Accept.{""quoted"/printable"};
> 
> -> Accept: text/plain; q=3.14, text/x-dvi, */*, "quoted"/printable
> 
> Vary: Accept-Language
> 
> set beresp.http.Vary."Accept-Encoding";
> 
> -> Vary: Accept-Language, Accept-Encoding

I think we should have different operations, other than set and unset,
for comma-separated lists of scalar values that aren't of the form
x=y. Something like add and remove for lists. The VMOD header actually
comes very close to providing something like that (but not quite).

'set LHS;' without a RHS looks like one of those gotcha tests for C
programmers ("what does 'x;' do? did you know that it's even possible?").

These admittedly do look intuitive.

> -> Vary: Accept-Language, Accept-Encoding
> 
> unset beresp.http.Vary."Accept-Language";
> 
> -> Vary: Accept-Encoding
> 
> unset beresp.http.Vary."Accept-Encoding";
> 
> -> (void) - header removed


Best,
Geoff
- -- 
UPLEX Systemoptimierung
Scheffelstra?e 32
22301 Hamburg
http://uplex.de/
Mob: +49-176-63690917
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)

iQIcBAEBCAAGBQJUBWTOAAoJEOUwvh9pJNURXwMP/011D8XsoMzQ+CFFhkA1upQu
Cfd9uNePBvI3oDibJ+NDNH5f9vyXMcmrCnk6yvvRGO66X0B82PHANlzdXY5oG3gs
vBUHOfOvX9UBQjO6iWuHeogUoj7xD4wjCGVZFRgqE95F7c+hLCtY5UUoAn5J4lpf
UtoWsM046vAdiliiGjBX2QbFBoJ+li0s7Y2suLp1/fVaK8c+tRtV20ODKUwkDFFp
GsTihox6jZzdqVyYQeQOjlEk1FF01Y8FgYYETp3Rf9KXzKWbRSy3QCYIkyUzm0Dp
pseHrUdfNEUmRjsPKctmrsLG6mkqdV92LKzqhrgY28eE5miCX8PHqYaXLeaB7nTi
r9xnNkazSecBUmTkPwJkKjxflVhw1byG6xpHhsEC+JKmixxeQKhPP8Ug8vUA5XGx
2kDfduqudMaZ5rjd058Bjd/PVnzdvHKw7TxMSFeg82Hx+Uww6sg4TaJkgu8giCg/
P7ddbeOLsVF4OkESxXjcrJuSoVYXKKGZg2CnjLwmsgIYOl5d6nktmrhb8t/vs77w
Gk+SI5kFJ4+QJiE9KrNiWHl/J5TTCeku7sFwB94VWKU6iQ7SSPkqGoZ+G26Md/pg
ZKyEq1ueIphcKmnI+ZL/MsmOgkM0DaRBfHH0bFQI1B/j9FDaGLHfNsBkAS8dETWx
3JOtFUmT7SP2w+0+f5/R
=+NzM
-----END PGP SIGNATURE-----



From bilbo at hobbiton.org  Tue Sep  2 16:58:27 2014
From: bilbo at hobbiton.org (Leif Pedersen)
Date: Tue, 2 Sep 2014 11:58:27 -0500
Subject: proposal part 1 - VCL: edit header tokens/entities
In-Reply-To: <540564CF.8080204@uplex.de>
References: <53FCCB1D.5070707@schokola.de>
 <CAOXZevBHCC=FpKfm7s9MVK==NFjObnKM0hYRhSPsdVfixSyehQ@mail.gmail.com>
 <53FE2354.1000908@schokola.de>
 <CAOXZevB4C2-V-MeUnkn46SfTNB+umS0vHtQWbcZYacjrMtK5GA@mail.gmail.com>
 <53FEC893.5060401@schokola.de> <18938.1409209079@critter.freebsd.dk>
 <5404A678.8010602@schokola.de> <540564CF.8080204@uplex.de>
Message-ID: <CAK-wPOiCqmRHQk11vqzOVh4b-POyexxaiPo9xeoHaGMQ5JM9eg@mail.gmail.com>

On Tue, Sep 2, 2014 at 1:33 AM, Geoff Simmons <geoff at uplex.de> wrote:

>
> I'm not a big fan of:
>
> > Accept: text/plain; q=0.5, text/x-dvi, */*
> >
> > set req.http.Accept."text/plain"."q" = 3.14;
> >
> > -> Accept: text/plain; q=3.14, text/x-dvi, */*
>
> I had to look at that several times to realize what's going on -- the
> operation adds 'text/plain; q=3.14' to a comma-separated list. At
> first glance it looked like the set operation was doing two things at
> once, adding both 'text/plain', which is not set to anything, and 'q',
> which is set to the right-hand side.
>
> You'd think it should actually be (my notation suggestion this time):
>
> set req.http.Accept["text/plain; q"] = 3.14:
>
>
Just as a long-time user of Varnish, this caught my eye. I've worked around
this missing feature many times. Thank you for considering it.

I actually found the original proposal far more intuitive. Not only did I
immediately understand its intent, but it's clearer that VCL understands
the header's hierarchy intelligently. True, it'll automatically add
text/plain if missing for the purpose of setting q, but that's not uncommon
in other languages, nor is it ambiguous (what else could it reasonably
mean?). The counter-proposal looks to me like VCL does not understand the
semicolon. It's not clear that setting "text/plain; e" = 2.718 will result
in "text/plain; q=3.14; e=2.718" rather than "text/plain; q=3.14,
text/plain; e=2.718". (I'm fairly sure the former is right, but I admit
that a glance at RFC 2616 didn't clarify.) Nor is it clear that removing
text/plain will remove text/plain with a q-value, or that assigning
"text/plain; q" will replace "text/plain". I'd likely leave extra code in
my VCL to be sure, such as removing "text/plain; q" before setting
"text/plain". I'd wind up spending a lot of time checking these edge cases
to be sure Varnish handles it properly, whereas the original proposal
wouldn't provoke my skepticism.



-- 

As implied by email protocols, the information in this message is
not confidential.  Any middle-man or recipient may inspect, modify,
copy, forward, reply to, delete, or filter email for any purpose unless
said parties are otherwise obligated.  As the sender, I acknowledge that
I have a lower expectation of the control and privacy of this message
than I would a post-card.  Further, nothing in this message is
legally binding without cryptographic evidence of its integrity.

http://bilbo.hobbiton.org/wiki/Eat_My_Sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140902/c0e37999/attachment.html>

From slink at schokola.de  Tue Sep  2 17:12:33 2014
From: slink at schokola.de (Nils Goroll)
Date: Tue, 02 Sep 2014 19:12:33 +0200
Subject: stack ws workspace for pcre and others #1576
Message-ID: <5405FA81.7050808@schokola.de>

Hi,

on the weekend I have actually stumbled over the same cause that bug #1576 has
(but unfortunately have not immediately realized the connection) and have
changed our dcs_classifier vmod (which needs some reasonably large scratch
memory in the order of ~64k) to use the client workspace rather than stack.

Using the workspace seemed more reasonable as varnish core structures have been
moved away from the stack and I thought it would be more advantagous to increase
a workspace size than increase the stack size (which has a nice tiny default of
48k now).

But then Ferdericos comment in #1576 made me re-think this:

> Comment (by fgsch):
>
>  This is actually related to the thread_pool_stack size.
>  Bumping the value to 49k make the test work. Lowering pcre_match_limit
>  makes the match fail without crashing.
>
>  Should we reduce the pcre_match_limit to something more sensible? 10000
>  seems like a rather large value.

If we were to need a larger default stack for pcre anyway, it wouldn't make
sense to move scratch space to a WS.

On the other hand, we might as well consider to give pcre space from workspaces:

PCRESTACK(3):

   Compiling PCRE to use heap instead of stack for pcre[16]_exec()

       In environments where stack memory is constrained, you might want to
compile PCRE to use heap memory instead of stack for remember?
       ing  back-up  points  when pcre[16]_exec() is running. This makes it run
a lot more slowly, however.  Details of how to do this are
       given in the pcrebuild documentation. When built in this way, instead of
using the stack, PCRE obtains and frees memory by  calling
       the  functions  that are pointed to by the pcre[16]_stack_malloc and
pcre[16]_stack_free variables. By default, these point to mal?
       loc() and free(), but you can replace the pointers to cause PCRE to use
your own functions. Since the block sizes  are  always  the
       same,  and  are  always  freed in reverse order, it may be possible to
implement customized memory handlers that are more efficient
       than the standard functions.

So we could simply hand the free space of some ws to pcre via some
pcre_stack_(malloc|free).


Whatever we do, we should either try to minimize ws usage or stack usage, but do
it consequently for all components.

Nils



From fgsch at lodoss.net  Tue Sep  2 17:56:05 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Tue, 2 Sep 2014 18:56:05 +0100
Subject: stack ws workspace for pcre and others #1576
In-Reply-To: <5405FA81.7050808@schokola.de>
References: <5405FA81.7050808@schokola.de>
Message-ID: <CAJV_h0auUA7cD-kJSJXNUTdSD1Criznn5OYdKjDvHy-YtSpgww@mail.gmail.com>

Only problem is that it's a build option.

Increasing the stack space will only work as long as the regex doesn't
recurse more than the stack space allows.

Not sure what's the best solution here but we should avoid crashing.



On Tue, Sep 2, 2014 at 6:12 PM, Nils Goroll <slink at schokola.de> wrote:

> Hi,
>
> on the weekend I have actually stumbled over the same cause that bug #1576
> has
> (but unfortunately have not immediately realized the connection) and have
> changed our dcs_classifier vmod (which needs some reasonably large scratch
> memory in the order of ~64k) to use the client workspace rather than stack.
>
> Using the workspace seemed more reasonable as varnish core structures have
> been
> moved away from the stack and I thought it would be more advantagous to
> increase
> a workspace size than increase the stack size (which has a nice tiny
> default of
> 48k now).
>
> But then Ferdericos comment in #1576 made me re-think this:
>
> > Comment (by fgsch):
> >
> >  This is actually related to the thread_pool_stack size.
> >  Bumping the value to 49k make the test work. Lowering pcre_match_limit
> >  makes the match fail without crashing.
> >
> >  Should we reduce the pcre_match_limit to something more sensible? 10000
> >  seems like a rather large value.
>
> If we were to need a larger default stack for pcre anyway, it wouldn't make
> sense to move scratch space to a WS.
>
> On the other hand, we might as well consider to give pcre space from
> workspaces:
>
> PCRESTACK(3):
>
>    Compiling PCRE to use heap instead of stack for pcre[16]_exec()
>
>        In environments where stack memory is constrained, you might want to
> compile PCRE to use heap memory instead of stack for remember?
>        ing  back-up  points  when pcre[16]_exec() is running. This makes
> it run
> a lot more slowly, however.  Details of how to do this are
>        given in the pcrebuild documentation. When built in this way,
> instead of
> using the stack, PCRE obtains and frees memory by  calling
>        the  functions  that are pointed to by the pcre[16]_stack_malloc and
> pcre[16]_stack_free variables. By default, these point to mal?
>        loc() and free(), but you can replace the pointers to cause PCRE to
> use
> your own functions. Since the block sizes  are  always  the
>        same,  and  are  always  freed in reverse order, it may be possible
> to
> implement customized memory handlers that are more efficient
>        than the standard functions.
>
> So we could simply hand the free space of some ws to pcre via some
> pcre_stack_(malloc|free).
>
>
> Whatever we do, we should either try to minimize ws usage or stack usage,
> but do
> it consequently for all components.
>
> Nils
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140902/5edab457/attachment.html>

From slink at schokola.de  Tue Sep  2 18:02:13 2014
From: slink at schokola.de (Nils Goroll)
Date: Tue, 02 Sep 2014 20:02:13 +0200
Subject: stack vs workspace for pcre and others #1576
In-Reply-To: <CAJV_h0auUA7cD-kJSJXNUTdSD1Criznn5OYdKjDvHy-YtSpgww@mail.gmail.com>
References: <5405FA81.7050808@schokola.de>
 <CAJV_h0auUA7cD-kJSJXNUTdSD1Criznn5OYdKjDvHy-YtSpgww@mail.gmail.com>
Message-ID: <54060625.5060404@schokola.de>

On 02/09/14 19:56, Federico Schwindt wrote:
> Only problem is that it's a build option.

Yes. But I don't see this as a disadvantage. I think we need to make a decision.



From fgsch at lodoss.net  Tue Sep  2 18:06:36 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Tue, 2 Sep 2014 19:06:36 +0100
Subject: stack vs workspace for pcre and others #1576
In-Reply-To: <54060625.5060404@schokola.de>
References: <5405FA81.7050808@schokola.de>
 <CAJV_h0auUA7cD-kJSJXNUTdSD1Criznn5OYdKjDvHy-YtSpgww@mail.gmail.com>
 <54060625.5060404@schokola.de>
Message-ID: <CAJV_h0aJj5473i76V=JQtNWmOwDq9EaW1h4QgCvGNyj9qp85vA@mail.gmail.com>

What you mean not a disadvantage? Are you by any means suggesting to bundle
pcre with Varnish?

There is also this somewhat worrisome comment in the docs: PCRE runs
noticeably more slowly when built in this way.

I personally don't consider this an option at all.


On Tue, Sep 2, 2014 at 7:02 PM, Nils Goroll <slink at schokola.de> wrote:

> On 02/09/14 19:56, Federico Schwindt wrote:
> > Only problem is that it's a build option.
>
> Yes. But I don't see this as a disadvantage. I think we need to make a
> decision.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140902/e87786f0/attachment.html>

From slink at schokola.de  Tue Sep  2 19:24:51 2014
From: slink at schokola.de (Nils Goroll)
Date: Tue, 02 Sep 2014 21:24:51 +0200
Subject: stack vs workspace for pcre and others #1576
In-Reply-To: <CAJV_h0aJj5473i76V=JQtNWmOwDq9EaW1h4QgCvGNyj9qp85vA@mail.gmail.com>
References: <5405FA81.7050808@schokola.de>	<CAJV_h0auUA7cD-kJSJXNUTdSD1Criznn5OYdKjDvHy-YtSpgww@mail.gmail.com>	<54060625.5060404@schokola.de>
 <CAJV_h0aJj5473i76V=JQtNWmOwDq9EaW1h4QgCvGNyj9qp85vA@mail.gmail.com>
Message-ID: <54061983.9040301@schokola.de>

On 02/09/14 20:06, Federico Schwindt wrote:
> What you mean not a disadvantage? Are you by any means suggesting to bundle pcre
> with Varnish?

If it saved considerable amounts of per-thread memory then I'd consider the option.

Or, alternatively, realize that we need a larg-ish stack anyway and (re)adjust
the rest of the code following this insight.

> There is also this somewhat worrisome comment in the docs: PCRE runs noticeably
> more slowly when built in this way.

I thought the docs were referring to pcre_malloc and pcre_free pointing to
malloc() and free(), respectively and I'd expect the overhead to be not that
significant if we implemented pcre_malloc as moving the free pointer of a
workspace (pcre_free probably could be a noop).



From slink at schokola.de  Tue Sep  2 19:34:53 2014
From: slink at schokola.de (Nils Goroll)
Date: Tue, 02 Sep 2014 21:34:53 +0200
Subject: stack vs workspace for pcre and others #1576
In-Reply-To: <54061983.9040301@schokola.de>
References: <5405FA81.7050808@schokola.de>	<CAJV_h0auUA7cD-kJSJXNUTdSD1Criznn5OYdKjDvHy-YtSpgww@mail.gmail.com>	<54060625.5060404@schokola.de>
 <CAJV_h0aJj5473i76V=JQtNWmOwDq9EaW1h4QgCvGNyj9qp85vA@mail.gmail.com>
 <54061983.9040301@schokola.de>
Message-ID: <54061BDD.2000104@schokola.de>

On 02/09/14 21:24, Nils Goroll wrote:
> Or, alternatively, realize that we need a larg-ish stack anyway

For all those who have not yet read pcrestack(3):

       As a very rough rule of thumb, you should reckon on about 500 bytes per
       recursion. Thus, if you want to limit your stack  usage  to
       8Mb, you should set the limit at 16000 recursions. A 64Mb stack, on the
       other hand, can support around 128000 recursions.

pcre_match_limit_recursion 10000 (default)
thread_pool_stack          48k [bytes] (default)

So these obviously do not fit at all.

We we opt for keeping pcre on the stack, we should consider to auto-tune either
to sensible values.

Nils



From phk at phk.freebsd.dk  Tue Sep  2 19:45:19 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 02 Sep 2014 19:45:19 +0000
Subject: stack ws workspace for pcre and others #1576
In-Reply-To: <5405FA81.7050808@schokola.de>
References: <5405FA81.7050808@schokola.de>
Message-ID: <44559.1409687119@critter.freebsd.dk>

--------

I think moving pcre from stack to heap would hurt our performance
more than increasing the thread stack size.

What we really need is for pcre to tell us an estimate of how much
stack-space is needed during compilation of the REs, so that we
can refuse vcl.use if there isn't enough thread_stack for it.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From slink at schokola.de  Tue Sep  2 19:50:32 2014
From: slink at schokola.de (Nils Goroll)
Date: Tue, 02 Sep 2014 21:50:32 +0200
Subject: stack ws workspace for pcre and others #1576
In-Reply-To: <44559.1409687119@critter.freebsd.dk>
References: <5405FA81.7050808@schokola.de>
 <44559.1409687119@critter.freebsd.dk>
Message-ID: <54061F88.2080409@schokola.de>

On 02/09/14 21:45, Poul-Henning Kamp wrote:
> What we really need is for pcre to tell us an estimate of how much
> stack-space is needed during compilation

Is this possible? My understanding is that the recursion depth depends on the
data and thus all we can do is set a recursion limit for the stack size we have
available.

Nils



From phk at phk.freebsd.dk  Tue Sep  2 19:51:49 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 02 Sep 2014 19:51:49 +0000
Subject: stack ws workspace for pcre and others #1576
In-Reply-To: <54061F88.2080409@schokola.de>
References: <5405FA81.7050808@schokola.de>
 <44559.1409687119@critter.freebsd.dk> <54061F88.2080409@schokola.de>
Message-ID: <44632.1409687509@critter.freebsd.dk>

--------
In message <54061F88.2080409 at schokola.de>, Nils Goroll writes:
>On 02/09/14 21:45, Poul-Henning Kamp wrote:
>> What we really need is for pcre to tell us an estimate of how much
>> stack-space is needed during compilation
>
>Is this possible? My understanding is that the recursion depth depends on the
>data and thus all we can do is set a recursion limit for the stack size we have
>available.

You tell me, I'm no PCRE expert...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From slink at schokola.de  Tue Sep  2 19:53:31 2014
From: slink at schokola.de (Nils Goroll)
Date: Tue, 02 Sep 2014 21:53:31 +0200
Subject: stack ws workspace for pcre and others #1576
In-Reply-To: <44632.1409687509@critter.freebsd.dk>
References: <5405FA81.7050808@schokola.de>
 <44559.1409687119@critter.freebsd.dk> <54061F88.2080409@schokola.de>
 <44632.1409687509@critter.freebsd.dk>
Message-ID: <5406203B.1090002@schokola.de>

> You tell me, I'm no PCRE expert...

Neither am I :(



From slink at schokola.de  Tue Sep  2 20:08:20 2014
From: slink at schokola.de (Nils Goroll)
Date: Tue, 02 Sep 2014 22:08:20 +0200
Subject: stack ws workspace for pcre and others #1576
In-Reply-To: <54061F88.2080409@schokola.de>
References: <5405FA81.7050808@schokola.de>
 <44559.1409687119@critter.freebsd.dk> <54061F88.2080409@schokola.de>
Message-ID: <540623B4.6050507@schokola.de>



On 02/09/14 21:50, Nils Goroll wrote:
> On 02/09/14 21:45, Poul-Henning Kamp wrote:
>> What we really need is for pcre to tell us an estimate of how much
>> stack-space is needed during compilation
> 
> Is this possible? My understanding is that the recursion depth depends on the
> data and thus all we can do is set a recursion limit for the stack size we have
> available.

This sounds close to the answer we are looking for:

https://lists.exim.org/lurker/message/20130417.155604.5223f8b8.en.html


> Is there a way to get some "stats" for a regexp like max recursion
> depth, taken branches, steps needed for solution, ...?

The pcretest program has a facility for determining the max recursion
depth, given a pattern and a subject string. Grep for \M in the pcretest
man page.

pcretest(1):

 If \M is present, pcretest calls pcre_exec() several times, with different
values in the match_limit and match_limit_recursion fields of the pcre_extra
data structure, until it finds the minimum numbers for each parameter that allow
pcre_exec() to complete.




From fgsch at lodoss.net  Tue Sep  2 20:10:57 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Tue, 2 Sep 2014 21:10:57 +0100
Subject: stack vs workspace for pcre and others #1576
In-Reply-To: <54061983.9040301@schokola.de>
References: <5405FA81.7050808@schokola.de>
 <CAJV_h0auUA7cD-kJSJXNUTdSD1Criznn5OYdKjDvHy-YtSpgww@mail.gmail.com>
 <54060625.5060404@schokola.de>
 <CAJV_h0aJj5473i76V=JQtNWmOwDq9EaW1h4QgCvGNyj9qp85vA@mail.gmail.com>
 <54061983.9040301@schokola.de>
Message-ID: <CAJV_h0Y=itnKqsRdB48M4YFkU1_jhk3myy1dOFkrRz4UGd+kvw@mail.gmail.com>

I don't think you're considering all the implications.

Bundling something is no small feat, specially when bugs are found. The
libjemalloc problem is a good example.

Leaving that aside the NO_RECURSIVE code in pcre is not the default. This
code is likely much less exercised than the recursive one and bugs might be
lurking there. FWIW the last entry I can find in the changelog is from Dec
2010.

But most importantly this would be shifting the problem from the stack to
the workspace. This doesn't fix the problem, just makes it easier or
harder, we don't know, to reproduce.


On Tue, Sep 2, 2014 at 8:24 PM, Nils Goroll <slink at schokola.de> wrote:

> On 02/09/14 20:06, Federico Schwindt wrote:
> > What you mean not a disadvantage? Are you by any means suggesting to
> bundle pcre
> > with Varnish?
>
> If it saved considerable amounts of per-thread memory then I'd consider
> the option.
>
> Or, alternatively, realize that we need a larg-ish stack anyway and
> (re)adjust
> the rest of the code following this insight.
>
> > There is also this somewhat worrisome comment in the docs: PCRE runs
> noticeably
> > more slowly when built in this way.
>
> I thought the docs were referring to pcre_malloc and pcre_free pointing to
> malloc() and free(), respectively and I'd expect the overhead to be not
> that
> significant if we implemented pcre_malloc as moving the free pointer of a
> workspace (pcre_free probably could be a noop).
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140902/84c43b7c/attachment.html>

From slink at schokola.de  Tue Sep  2 20:19:27 2014
From: slink at schokola.de (Nils Goroll)
Date: Tue, 02 Sep 2014 22:19:27 +0200
Subject: stack vs workspace for pcre and others #1576
In-Reply-To: <CAJV_h0Y=itnKqsRdB48M4YFkU1_jhk3myy1dOFkrRz4UGd+kvw@mail.gmail.com>
References: <5405FA81.7050808@schokola.de>	<CAJV_h0auUA7cD-kJSJXNUTdSD1Criznn5OYdKjDvHy-YtSpgww@mail.gmail.com>	<54060625.5060404@schokola.de>	<CAJV_h0aJj5473i76V=JQtNWmOwDq9EaW1h4QgCvGNyj9qp85vA@mail.gmail.com>	<54061983.9040301@schokola.de>
 <CAJV_h0Y=itnKqsRdB48M4YFkU1_jhk3myy1dOFkrRz4UGd+kvw@mail.gmail.com>
Message-ID: <5406264F.3090209@schokola.de>

> I don't think you're considering all the implications.

let me pull this straight: No, I did not mean to say we should pull in pcre. I
only wanted to start a discussion about what is the best option, I have not
thought through all implications. Again, I am not a pcre expert.

I did mean to say though that if we wanted to minimize the stack size, we should
consider options to get memory for pcre from elsewhere.


And at this point it seems like keeping pcre on the stack is the best option, so
if we need a larg-ish stack anyway for pcre, I think we should re-think some of
the code in order to make good use of that (and possibly save memory elsewhere).

Also we definitely need to provide better defaults for
pcre_match_limit_recursion and thread_pool_stack




From phk at phk.freebsd.dk  Tue Sep  2 20:38:13 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 02 Sep 2014 20:38:13 +0000
Subject: stack vs workspace for pcre and others #1576
In-Reply-To: <5406264F.3090209@schokola.de>
References: <5405FA81.7050808@schokola.de>
 <CAJV_h0auUA7cD-kJSJXNUTdSD1Criznn5OYdKjDvHy-YtSpgww@mail.gmail.com>
 <54060625.5060404@schokola.de>
 <CAJV_h0aJj5473i76V=JQtNWmOwDq9EaW1h4QgCvGNyj9qp85vA@mail.gmail.com>
 <54061983.9040301@schokola.de>
 <CAJV_h0Y=itnKqsRdB48M4YFkU1_jhk3myy1dOFkrRz4UGd+kvw@mail.gmail.com>
 <5406264F.3090209@schokola.de>
Message-ID: <50696.1409690293@critter.freebsd.dk>

--------
In message <5406264F.3090209 at schokola.de>, Nils Goroll writes:
>> I don't think you're considering all the implications.
>
>let me pull this straight: No, I did not mean to say we should pull in pcre.

Pulling in PCRE to move things onto the heap doesn't solve the problem.

In fact it will probably increase our per-thread memory foot-print
more than increasing the stack will.

Move on, nothing to see here...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From fgsch at lodoss.net  Tue Sep  2 23:56:32 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Wed, 3 Sep 2014 00:56:32 +0100
Subject: stack ws workspace for pcre and others #1576
In-Reply-To: <540623B4.6050507@schokola.de>
References: <5405FA81.7050808@schokola.de>
 <44559.1409687119@critter.freebsd.dk>
 <54061F88.2080409@schokola.de> <540623B4.6050507@schokola.de>
Message-ID: <CAJV_h0ZfT-BwxxLf-ddz1Lb5C+W0SKvFNkKQS79CDMM9ANdnWA@mail.gmail.com>

Fix aside should we consider using sigaltstack to also catch these crashes?
I imagine something like the attached but allocated on startup and shared
among all workers.

This might be specific to pcre at the moment but I can see its usefulness
in the vmod world.




On Tue, Sep 2, 2014 at 9:08 PM, Nils Goroll <slink at schokola.de> wrote:

>
>
> On 02/09/14 21:50, Nils Goroll wrote:
> > On 02/09/14 21:45, Poul-Henning Kamp wrote:
> >> What we really need is for pcre to tell us an estimate of how much
> >> stack-space is needed during compilation
> >
> > Is this possible? My understanding is that the recursion depth depends
> on the
> > data and thus all we can do is set a recursion limit for the stack size
> we have
> > available.
>
> This sounds close to the answer we are looking for:
>
> https://lists.exim.org/lurker/message/20130417.155604.5223f8b8.en.html
>
>
> > Is there a way to get some "stats" for a regexp like max recursion
> > depth, taken branches, steps needed for solution, ...?
>
> The pcretest program has a facility for determining the max recursion
> depth, given a pattern and a subject string. Grep for \M in the pcretest
> man page.
>
> pcretest(1):
>
>  If \M is present, pcretest calls pcre_exec() several times, with different
> values in the match_limit and match_limit_recursion fields of the
> pcre_extra
> data structure, until it finds the minimum numbers for each parameter that
> allow
> pcre_exec() to complete.
>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140903/e7411f2b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 000-sigaltstack.patch
Type: text/x-patch
Size: 1543 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140903/e7411f2b/attachment.bin>

From phk at phk.freebsd.dk  Wed Sep  3 08:13:35 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 03 Sep 2014 08:13:35 +0000
Subject: stack ws workspace for pcre and others #1576
In-Reply-To: <CAJV_h0ZfT-BwxxLf-ddz1Lb5C+W0SKvFNkKQS79CDMM9ANdnWA@mail.gmail.com>
References: <5405FA81.7050808@schokola.de>
 <44559.1409687119@critter.freebsd.dk> <54061F88.2080409@schokola.de>
 <540623B4.6050507@schokola.de>
 <CAJV_h0ZfT-BwxxLf-ddz1Lb5C+W0SKvFNkKQS79CDMM9ANdnWA@mail.gmail.com>
Message-ID: <62879.1409732015@critter.freebsd.dk>

--------
In message <CAJV_h0ZfT-BwxxLf-ddz1Lb5C+W0SKvFNkKQS79CDMM9ANdnWA at mail.gmail.com>
, Federico Schwindt writes:

+#ifdef HAVE_SIGALTSTACK
+       if (cache_param->sigsegv_handler) {
+               char stackbuf[MINSIGSTKSZ];
+               stack_t ss;
+
+               ss.ss_size = sizeof(stackbuf);
+               ss.ss_sp = stackbuf;
+               ss.ss_flags = 0;
+               (void)sigaltstack(&ss, NULL);
+       }
+#endif

This will ad 2-4 KB to all threads.

Also, I'm not convinced about the wisdom of allocating the sigstack
on the normal stack...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From fgsch at lodoss.net  Wed Sep  3 11:09:53 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Wed, 3 Sep 2014 12:09:53 +0100
Subject: #1538
Message-ID: <CAJV_h0ahaa1YP_J8G1LUcn3cG-XjK+Ua74GT9Je9L9tniDSihQ@mail.gmail.com>

Compromise?

f.-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140903/a344a9c2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 000.vmod-check.patch
Type: text/x-patch
Size: 1819 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140903/a344a9c2/attachment.bin>

From fgsch at lodoss.net  Wed Sep  3 15:37:09 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Wed, 3 Sep 2014 16:37:09 +0100
Subject: stack ws workspace for pcre and others #1576
In-Reply-To: <62879.1409732015@critter.freebsd.dk>
References: <5405FA81.7050808@schokola.de>
 <44559.1409687119@critter.freebsd.dk>
 <54061F88.2080409@schokola.de> <540623B4.6050507@schokola.de>
 <CAJV_h0ZfT-BwxxLf-ddz1Lb5C+W0SKvFNkKQS79CDMM9ANdnWA@mail.gmail.com>
 <62879.1409732015@critter.freebsd.dk>
Message-ID: <CAJV_h0Zzc4x-5qQxSWWmbWS1r-z+R_JvvYvPx71YA7m4zd5MsQ@mail.gmail.com>

As discussed on irc the space can be allocated and shared among all threads
instead of using the stack.
I've tested it this morning and works fine. In both cases the stack trace
was present.

Without this code if we overflow the stack we just get a coredump.

Regardless of this we should lower the pcre limits for the time being based
on the information in pcrestack.


On Wed, Sep 3, 2014 at 9:13 AM, Poul-Henning Kamp <phk at phk.freebsd.dk>
wrote:

> --------
> In message <
> CAJV_h0ZfT-BwxxLf-ddz1Lb5C+W0SKvFNkKQS79CDMM9ANdnWA at mail.gmail.com>
> , Federico Schwindt writes:
>
> +#ifdef HAVE_SIGALTSTACK
> +       if (cache_param->sigsegv_handler) {
> +               char stackbuf[MINSIGSTKSZ];
> +               stack_t ss;
> +
> +               ss.ss_size = sizeof(stackbuf);
> +               ss.ss_sp = stackbuf;
> +               ss.ss_flags = 0;
> +               (void)sigaltstack(&ss, NULL);
> +       }
> +#endif
>
> This will ad 2-4 KB to all threads.
>
> Also, I'm not convinced about the wisdom of allocating the sigstack
> on the normal stack...
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140903/4d44aed4/attachment.html>

From slink at schokola.de  Wed Sep 10 04:39:20 2014
From: slink at schokola.de (Nils Goroll)
Date: Wed, 10 Sep 2014 06:39:20 +0200
Subject: 4.0 branch
Message-ID: <540FD5F8.6050403@schokola.de>

Hi,

I had started https://www.varnish-cache.org/trac/wiki/Merge4.0 and fgs has made
some additions since. But seeing Martin pushing useful fixes to 4.0, I wonder if
we should rather all try to commit required fixes to 4.0.

Lasse, what's your preference?

Nils



From Gauthier.Delacroix at coreye.fr  Wed Sep 10 13:28:36 2014
From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier)
Date: Wed, 10 Sep 2014 13:28:36 +0000
Subject: server.* in vcl_init{}
Message-ID: <5F530A9242E7F84F999DB40E0E268FBD45DC5001@mercalli.lild01.pictime.fr>

Hi,

As explained here (https://github.com/varnish/Varnish-Cache/blob/master/doc/changes.rst#vcl-2), server.* values aren't available in vcl_init{} and vcl_fini{} because they are "Request specific".

I understand why client.* is request specific, but server.* seems...server specific...

I'm using libvmod_statsd which allows to prefix all statsd requests through statsd.prefix() in vcl_init{}. I wish I could use server.identity as a prefix but I've been surprised server.* isn't available there (never used vcl_init{} before).

Am I mistaking something or shouldn't server.* be available in vcl_init{} ?

Thanks !

Gauthier Delacroix




From fgsch at lodoss.net  Wed Sep 17 11:04:05 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Wed, 17 Sep 2014 12:04:05 +0100
Subject: [PATCH] add VCL_BYTES
Message-ID: <CAJV_h0ZN8jPuBriVzhBNGDBgtGmOTn1WWsUmCPMi9cmWVJ1Gbg@mail.gmail.com>

Pretty trivial.
OK?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140917/3d13a446/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 000.add-vcl-bytes.patch
Type: text/x-patch
Size: 355 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140917/3d13a446/attachment.bin>

From fgsch at lodoss.net  Wed Sep 17 11:28:36 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Wed, 17 Sep 2014 12:28:36 +0100
Subject: [PATCH] stale-while-revalidate update
Message-ID: <CAJV_h0aPPmFNtpWN+24YasVmW0yBUJ47hDqarZ+BFwD8XsK=HA@mail.gmail.com>

Update diff after the discussion at vdd.

This will init grace with the s-w-r value regardless of the ttl.

Ok?

f.-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140917/1cfec359/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 000.s-w-r.patch
Type: text/x-patch
Size: 2882 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140917/1cfec359/attachment.bin>

From slink at schokola.de  Wed Sep 17 13:18:14 2014
From: slink at schokola.de (Nils Goroll)
Date: Wed, 17 Sep 2014 15:18:14 +0200
Subject: proposal part 1 - VCL: edit header tokens/entities
In-Reply-To: <5404A678.8010602@schokola.de>
References: <53FCCB1D.5070707@schokola.de>
 <CAOXZevBHCC=FpKfm7s9MVK==NFjObnKM0hYRhSPsdVfixSyehQ@mail.gmail.com>
 <53FE2354.1000908@schokola.de>
 <CAOXZevB4C2-V-MeUnkn46SfTNB+umS0vHtQWbcZYacjrMtK5GA@mail.gmail.com>
 <53FEC893.5060401@schokola.de> <18938.1409209079@critter.freebsd.dk>
 <5404A678.8010602@schokola.de>
Message-ID: <54198A16.8030805@schokola.de>

Status on this from https://www.varnish-cache.org/trac/wiki/VDD14Q3 :

We might want to write or at least prototype a vmod for it, but nothing is
pushing me personally to do this with high priority.

The main motivation for my proposal was that we probably need more token/entity
editing in core code anyway and so it was just an obvious idea to add the
functionality to VCC properly.

Nils



From fgsch at lodoss.net  Wed Sep 17 16:28:09 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Wed, 17 Sep 2014 17:28:09 +0100
Subject: [PATCH] stale-while-revalidate update
In-Reply-To: <CAJV_h0aPPmFNtpWN+24YasVmW0yBUJ47hDqarZ+BFwD8XsK=HA@mail.gmail.com>
References: <CAJV_h0aPPmFNtpWN+24YasVmW0yBUJ47hDqarZ+BFwD8XsK=HA@mail.gmail.com>
Message-ID: <CAJV_h0aK9WaapUD-RGC0tKR=DY3dJmO0yS1c2M4UGBgs7pHFvw@mail.gmail.com>

Actually after reviewing this I think we want to ignore s-w-r for a -1 ttl
so new diff attached.

Yes?

On Wed, Sep 17, 2014 at 12:28 PM, Federico Schwindt <fgsch at lodoss.net>
wrote:

> Update diff after the discussion at vdd.
>
> This will init grace with the s-w-r value regardless of the ttl.
>
> Ok?
>
> f.-
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140917/7af8ec6c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 000.s-w-r.patch
Type: text/x-patch
Size: 3026 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140917/7af8ec6c/attachment.bin>

From slink at schokola.de  Wed Sep 17 17:36:49 2014
From: slink at schokola.de (Nils Goroll)
Date: Wed, 17 Sep 2014 19:36:49 +0200
Subject: Update: [PATCH 1/2] Add a function to remove an element/token from
 a header, value (like Accept-Encoding from Vary:)
In-Reply-To: <53FE259A.30800@schokola.de>
References: <53FE259A.30800@schokola.de>
Message-ID: <5419C6B1.9010305@schokola.de>

With this change, http_RemoveHdrToken also removes any values of an element
token seperated by equals (like foo=bar)

phk, does this work for you?

Thanks, Nils
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Add-a-function-to-remove-an-element-token-from-a-hea.patch
Type: text/x-patch
Size: 3404 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140917/9b3d01a5/attachment.bin>

From slink at schokola.de  Wed Sep 17 18:00:16 2014
From: slink at schokola.de (Nils Goroll)
Date: Wed, 17 Sep 2014 20:00:16 +0200
Subject: [PATCH] stale-while-revalidate update
In-Reply-To: <CAJV_h0aK9WaapUD-RGC0tKR=DY3dJmO0yS1c2M4UGBgs7pHFvw@mail.gmail.com>
References: <CAJV_h0aPPmFNtpWN+24YasVmW0yBUJ47hDqarZ+BFwD8XsK=HA@mail.gmail.com>
 <CAJV_h0aK9WaapUD-RGC0tKR=DY3dJmO0yS1c2M4UGBgs7pHFvw@mail.gmail.com>
Message-ID: <5419CC30.6000602@schokola.de>

reviewed and tested, looks good to me.



From fgsch at lodoss.net  Thu Sep 18 18:58:46 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Thu, 18 Sep 2014 19:58:46 +0100
Subject: [PATCH] Surrogate-Control (max-age) support
Message-ID: <CAJV_h0YbmrnN7bCPz=qw+mkJ_q5175QmBH83JYdoo-SXwX-XfA@mail.gmail.com>

Hi,

I had some time on the plane today so here's a quick stab at getting
Surrogate-Control support as discussed during the vdd. Disabled by default.
Only max-age is supported with this patch.

f.-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140918/5263cc33/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 000.surrogate-control_support.patch
Type: text/x-patch
Size: 3961 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140918/5263cc33/attachment.bin>

From slink at schokola.de  Fri Sep 19 09:42:13 2014
From: slink at schokola.de (Nils Goroll)
Date: Fri, 19 Sep 2014 11:42:13 +0200
Subject: VRT_CacheReqBody without inline-C?
In-Reply-To: <32B2E540-68AE-460A-B544-0FEF4EAF29CB@gmail.com>
References: <53D692F4.7030904@schokola.de>
 <16357.1406707032@critter.freebsd.dk>
 <32B2E540-68AE-460A-B544-0FEF4EAF29CB@gmail.com>
Message-ID: <541BFA75.4050800@schokola.de>

-> cc2b4c4d1a14dcdfe1a9f01a9797cbea396c581b

On 30/07/14 17:00, Meng Zhang wrote:
> Add std.cache_req_body for VRT_CacheReqBody



From geoff at uplex.de  Fri Sep 19 11:26:49 2014
From: geoff at uplex.de (Geoff Simmons)
Date: Fri, 19 Sep 2014 13:26:49 +0200
Subject: [PATCH] Surrogate-Control (max-age) support
In-Reply-To: <CAJV_h0YbmrnN7bCPz=qw+mkJ_q5175QmBH83JYdoo-SXwX-XfA@mail.gmail.com>
References: <CAJV_h0YbmrnN7bCPz=qw+mkJ_q5175QmBH83JYdoo-SXwX-XfA@mail.gmail.com>
Message-ID: <541C12F9.80401@uplex.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 09/18/2014 08:58 PM, Federico Schwindt wrote:
> 
> I had some time on the plane today so here's a quick stab at
> getting Surrogate-Control support as discussed during the vdd.
> Disabled by default. Only max-age is supported with this patch.

We had some discussion about this in IRC, which I'll try to
recapitulate here.

My concern was that someone may already be using Surrogate-Control to
target a proxy that is downstream from Varnish. That would be broken
by the patch, since it unconditionally strips the header from the
response.

I believe we came around to a consensus to implement targeting, as
described in section 2.3 of http://www.w3.org/TR/edge-arch, so that
users may be able to target the S-C header to Varnish and/or another
proxy.

- - Varnish has a default token name, which may be overridden by a param
like: -p surrogate_token=<token>

- - When the feature is enabled, Varnish appends a
Surrogate-Capabilities request header, identifying itself with the
token, e.g.:

Surrogate-Capabilities: <token>="Surrogate/1.0"

The header is added if not already present in the request, otherwise
this value is appended to the existing header.

- - The Surrogate-Control response header is honored for the part that
targets the Varnish token, or for any part that contains no token, if
Varnish is not specifically targeted.

So a response may include:

  Surrogate-Control: max-age=60+30;myvarnish, no-store;mycdn

That would mean: the "myvarnish" instance of Varnish sets TTL=60 and
grace=30, while the no-store directive is targeted to "mycdn".

This would have the same effect (assuming the token is "myvarnish"):

  Surrogate-Control: max-age=60+30, no-store;mycdn

In that case, no-store is targeted to "mycdn", and the max-age
directive is targeted at any other proxy, and is therefore honored by
Varnish.

- - What exactly should the default token name be? That subject is ripe
for bikeshedding, so we can have a long, passionate discussion about
it, or just set it to "varnish", or to the value of server.id.

- - We can also evaluate no-store (fgs' patch just honors max-age for
simplicity, but no-store shouldn't be too hard).

- - Honoring no-store-remote should be a matter for VCL, since Varnishen
are not typically "remote" proxies.

- - All of this should be overridable in VCL.

- - MAYBE we remove the S-C response header if Varnish "consumes" it.
The TR says in section 2.2: "If no downstream surrogates have
identified themselves, the header should be stripped from responses."
Assuming that the lower-case "should" is just like an RFC SHOULD, we
can remove the header if no other downstream sent a
Surrogate-Capabilities header.

Since this is a "TR" and not a formal standard, and since it shows a
little sloppiness in some places (inconsistent spelling of the request
header, out-of-order numbering of chapter headings, examples that
don't match the prose), IMHO we don't need to be strict about
conformance. We can document the feature as "in partial fulfillment
of" and then describe exactly what it does, which is I think is good
enough for what we need: setting a TTL (and possibly grace) for
Varnish, without passing the same setting downstream (as is necessary
with Cache-Control and Expires).


Best,
Geoff
- -- 
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Icedove - http://www.enigmail.net/

iQIcBAEBCAAGBQJUHBL5AAoJEOUwvh9pJNURkcQP/jmUJnWdXL/g6to2JBiWI69u
tHTzomMM3H7kpalOStVVbhB+dpjs96AvSTtUn1hnRQkwIVIyW2sL5l64On+gvEfN
2S3YMWf4H7Hm7u0BlL0bTedv+bGCXkv1sQC/AY0ZxTlB6LxuSvpJEXYSVSRpoitU
NY9GAFtRBPQ8cwOlxetiT720ZeZ+GZo2khy32xUrnkBihOVafWznxdfK+wVJX7EZ
3aLMnJxlW9efzMtWsWB1wtqHy3slHfYC+8h/O7AsNF0JJ6mXZxZoqKmsJAZkjT0N
oi1MNk0crRgI4T96V0bjCpsijndohMCpRgV9q3tDfRLjdVpLQqi7PuM01budqjqC
gZqsh9iAmrfEA7XjLYOxSygZwB1kyoJIhTg+Tl5nBQIg+qS+D/WfLemzj8pBueiy
zo4zakIljmNoNU0bWReoUetoTXrbSld5LtXqjzHcixkVtExRdOvhM5RfiehgpSNQ
dLOMhU5nNpm+rGbRt20kz8ufhyzdFfwx53FpMV/NQKXxaubAO3ePEQp3OQDMYGSQ
SJ3m5KZK0XXBDCIkXGjzc4hMniE8eejoldLdso+EmLRsCy5FihQZTPV/8OBhMTwS
YknLMgsXiXtCEwdWtded610C0cB4uNnrmA7GpILfGivd1g3WOMy07/obXb7Rbzy3
2QuMNkTPXZhQp3bgEOXc
=u3a0
-----END PGP SIGNATURE-----



From perbu at varnish-software.com  Sat Sep 20 09:12:39 2014
From: perbu at varnish-software.com (Per Buer)
Date: Sat, 20 Sep 2014 11:12:39 +0200
Subject: [PATCH] substring matching in std
Message-ID: <CAOXZevApN8RhTJbSOmZZ5Wy+1ArWnKM0hzKuU2-OKSgkKSCOfw@mail.gmail.com>

 bin/varnishtest/tests/m00018.vtc | 34 ++++++++++++++++++++++++++++++++++
 lib/libvmod_std/vmod.vcc         | 12 +++++++++++-
 lib/libvmod_std/vmod_std.c       | 14 ++++++++++++++
 3 files changed, 59 insertions(+), 1 deletion(-)
 create mode 100644 bin/varnishtest/tests/m00018.vtc

diff --git a/bin/varnishtest/tests/m00018.vtc
b/bin/varnishtest/tests/m00018.vtc
new file mode 100644
index 0000000..af00716
--- /dev/null
+++ b/bin/varnishtest/tests/m00018.vtc
@@ -0,0 +1,34 @@
+varnishtest "Test substring matching in std"
+
+server s1 {
+       rxreq
+       txresp
+
+       rxreq
+       txresp
+} -start
+
+varnish v1 -vcl+backend {
+ import ${vmod_std};
+
+ sub vcl_deliver {
+    if (std.substr(req.url, "foo")) {
+ set resp.http.sub = "found";
+    } else {
+ set resp.http.sub = "not found";
+    }
+
+ }
+} -start
+
+client c1 {
+ txreq -url "/foobar"
+ rxresp
+ expect resp.http.sub == "found"
+
+
+ txreq -url "/quux"
+ rxresp
+ expect resp.http.sub == "not found"
+} -run
+
diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc
index fbc082b..e59c7a4 100644
--- a/lib/libvmod_std/vmod.vcc
+++ b/lib/libvmod_std/vmod.vcc
@@ -195,7 +195,6 @@ $Function STRING querysort(STRING)

 Description
         Sorts the querystring for cache normalization purposes.
-
 Example
         set req.url = std.querysort(req.url);

@@ -208,6 +207,17 @@ Example

        This will cache the req.body if its size is smaller than 1KB.

+$Function BOOL substr(STRING, STRING)
+
+Description
+ Returns true if the second string is a substring of the first
+ string. Note that the comparison is case sensitive. You can
+ use the tolower function on both strings if you want case
+ insensitivity.
+Example
+ if (std.substr(req.url, req.http.x-restrict))
+
+

 SEE ALSO
 ========
diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
index 8872a1c..8a89bb8 100644
--- a/lib/libvmod_std/vmod_std.c
+++ b/lib/libvmod_std/vmod_std.c
@@ -239,3 +239,17 @@ vmod_cache_req_body(const struct vrt_ctx *ctx,
VCL_BYTES size)
  result = VRT_CacheReqBody(ctx, size);
  VSLb(ctx->vsl, SLT_Debug,"VRT_CacheReqBody(%zu): %d", (size_t)size,
result);
 }
+
+VCL_BOOL __match_proto__(td_std_substr)
+vmod_substr(const struct vrt_ctx *ctx, VCL_STRING mstr, VCL_STRING msubstr)
+{
+
+        CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+        char *match =  strstr(mstr, msubstr);
+
+ if (match)
+  return(1);
+ else
+  return(0);
+}
+
-- 
1.9.3 (Apple Git-50)+GitX

-- 
*Per Buer*
CTO | Varnish Software AS
Cell: +47 95839117
We Make Websites Fly!
www.varnish-software.com
[image: Register now]
<http://info.varnish-software.com/varnish-summits-autumn-2014-registration>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140920/d0c917f1/attachment.html>

From varnish at bsdchicks.com  Sat Sep 20 09:30:34 2014
From: varnish at bsdchicks.com (Rogier 'DocWilco' Mulhuijzen)
Date: Sat, 20 Sep 2014 11:30:34 +0200
Subject: [PATCH] substring matching in std
In-Reply-To: <CAOXZevApN8RhTJbSOmZZ5Wy+1ArWnKM0hzKuU2-OKSgkKSCOfw@mail.gmail.com>
References: <CAOXZevApN8RhTJbSOmZZ5Wy+1ArWnKM0hzKuU2-OKSgkKSCOfw@mail.gmail.com>
Message-ID: <CANTTouXzDzUHuE4aP9S-ke=GYGX1nV4pvvsE3-P_ptvFNMjf2g@mail.gmail.com>

I'm not fond of using "substr" as the function name, as other languages use
this function with start/end or start/length parameters to create a new
string from the existing string.

I would suggest one of the following:

- strstr
- instr
- contains

Also, just out of curiosity, exactly how much faster is this than a regex
match?

On Sat, Sep 20, 2014 at 11:12 AM, Per Buer <perbu at varnish-software.com>
wrote:

>  bin/varnishtest/tests/m00018.vtc | 34 ++++++++++++++++++++++++++++++++++
>  lib/libvmod_std/vmod.vcc         | 12 +++++++++++-
>  lib/libvmod_std/vmod_std.c       | 14 ++++++++++++++
>  3 files changed, 59 insertions(+), 1 deletion(-)
>  create mode 100644 bin/varnishtest/tests/m00018.vtc
>
> diff --git a/bin/varnishtest/tests/m00018.vtc
> b/bin/varnishtest/tests/m00018.vtc
> new file mode 100644
> index 0000000..af00716
> --- /dev/null
> +++ b/bin/varnishtest/tests/m00018.vtc
> @@ -0,0 +1,34 @@
> +varnishtest "Test substring matching in std"
> +
> +server s1 {
> +       rxreq
> +       txresp
> +
> +       rxreq
> +       txresp
> +} -start
> +
> +varnish v1 -vcl+backend {
> + import ${vmod_std};
> +
> + sub vcl_deliver {
> +    if (std.substr(req.url, "foo")) {
> + set resp.http.sub = "found";
> +    } else {
> + set resp.http.sub = "not found";
> +    }
> +
> + }
> +} -start
> +
> +client c1 {
> + txreq -url "/foobar"
> + rxresp
> + expect resp.http.sub == "found"
> +
> +
> + txreq -url "/quux"
> + rxresp
> + expect resp.http.sub == "not found"
> +} -run
> +
> diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc
> index fbc082b..e59c7a4 100644
> --- a/lib/libvmod_std/vmod.vcc
> +++ b/lib/libvmod_std/vmod.vcc
> @@ -195,7 +195,6 @@ $Function STRING querysort(STRING)
>
>  Description
>          Sorts the querystring for cache normalization purposes.
> -
>  Example
>          set req.url = std.querysort(req.url);
>
> @@ -208,6 +207,17 @@ Example
>
>         This will cache the req.body if its size is smaller than 1KB.
>
> +$Function BOOL substr(STRING, STRING)
> +
> +Description
> + Returns true if the second string is a substring of the first
> + string. Note that the comparison is case sensitive. You can
> + use the tolower function on both strings if you want case
> + insensitivity.
> +Example
> + if (std.substr(req.url, req.http.x-restrict))
> +
> +
>
>  SEE ALSO
>  ========
> diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
> index 8872a1c..8a89bb8 100644
> --- a/lib/libvmod_std/vmod_std.c
> +++ b/lib/libvmod_std/vmod_std.c
> @@ -239,3 +239,17 @@ vmod_cache_req_body(const struct vrt_ctx *ctx,
> VCL_BYTES size)
>   result = VRT_CacheReqBody(ctx, size);
>   VSLb(ctx->vsl, SLT_Debug,"VRT_CacheReqBody(%zu): %d", (size_t)size,
> result);
>  }
> +
> +VCL_BOOL __match_proto__(td_std_substr)
> +vmod_substr(const struct vrt_ctx *ctx, VCL_STRING mstr, VCL_STRING
> msubstr)
> +{
> +
> +        CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
> +        char *match =  strstr(mstr, msubstr);
> +
> + if (match)
> +  return(1);
> + else
> +  return(0);
> +}
> +
> --
> 1.9.3 (Apple Git-50)+GitX
>
> --
> *Per Buer*
> CTO | Varnish Software AS
> Cell: +47 95839117
> We Make Websites Fly!
> www.varnish-software.com
> [image: Register now]
> <http://info.varnish-software.com/varnish-summits-autumn-2014-registration>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140920/7cc7df25/attachment.html>

From fgsch at lodoss.net  Sat Sep 20 11:00:00 2014
From: fgsch at lodoss.net (Federico Schwindt)
Date: Sat, 20 Sep 2014 12:00:00 +0100
Subject: [PATCH] substring matching in std
In-Reply-To: <CANTTouXzDzUHuE4aP9S-ke=GYGX1nV4pvvsE3-P_ptvFNMjf2g@mail.gmail.com>
References: <CAOXZevApN8RhTJbSOmZZ5Wy+1ArWnKM0hzKuU2-OKSgkKSCOfw@mail.gmail.com>
 <CANTTouXzDzUHuE4aP9S-ke=GYGX1nV4pvvsE3-P_ptvFNMjf2g@mail.gmail.com>
Message-ID: <CAJV_h0YQeAkLnZy4bj=sDwkEuYRtLh_xXFnEJCRTCAGeGwtPeQ@mail.gmail.com>

I'd prefer to keep substr (or rename it to strstr) but returning the
substring instead so you can also use the  string.

The reason for this is that regex match doesn't work on dynamic strings.
 On 20 Sep 2014 10:31, "Rogier 'DocWilco' Mulhuijzen" <varnish at bsdchicks.com>
wrote:

> I'm not fond of using "substr" as the function name, as other languages
> use this function with start/end or start/length parameters to create a new
> string from the existing string.
>
> I would suggest one of the following:
>
> - strstr
> - instr
> - contains
>
> Also, just out of curiosity, exactly how much faster is this than a regex
> match?
>
> On Sat, Sep 20, 2014 at 11:12 AM, Per Buer <perbu at varnish-software.com>
> wrote:
>
>>  bin/varnishtest/tests/m00018.vtc | 34 ++++++++++++++++++++++++++++++++++
>>  lib/libvmod_std/vmod.vcc         | 12 +++++++++++-
>>  lib/libvmod_std/vmod_std.c       | 14 ++++++++++++++
>>  3 files changed, 59 insertions(+), 1 deletion(-)
>>  create mode 100644 bin/varnishtest/tests/m00018.vtc
>>
>> diff --git a/bin/varnishtest/tests/m00018.vtc
>> b/bin/varnishtest/tests/m00018.vtc
>> new file mode 100644
>> index 0000000..af00716
>> --- /dev/null
>> +++ b/bin/varnishtest/tests/m00018.vtc
>> @@ -0,0 +1,34 @@
>> +varnishtest "Test substring matching in std"
>> +
>> +server s1 {
>> +       rxreq
>> +       txresp
>> +
>> +       rxreq
>> +       txresp
>> +} -start
>> +
>> +varnish v1 -vcl+backend {
>> + import ${vmod_std};
>> +
>> + sub vcl_deliver {
>> +    if (std.substr(req.url, "foo")) {
>> + set resp.http.sub = "found";
>> +    } else {
>> + set resp.http.sub = "not found";
>> +    }
>> +
>> + }
>> +} -start
>> +
>> +client c1 {
>> + txreq -url "/foobar"
>> + rxresp
>> + expect resp.http.sub == "found"
>> +
>> +
>> + txreq -url "/quux"
>> + rxresp
>> + expect resp.http.sub == "not found"
>> +} -run
>> +
>> diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc
>> index fbc082b..e59c7a4 100644
>> --- a/lib/libvmod_std/vmod.vcc
>> +++ b/lib/libvmod_std/vmod.vcc
>> @@ -195,7 +195,6 @@ $Function STRING querysort(STRING)
>>
>>  Description
>>          Sorts the querystring for cache normalization purposes.
>> -
>>  Example
>>          set req.url = std.querysort(req.url);
>>
>> @@ -208,6 +207,17 @@ Example
>>
>>         This will cache the req.body if its size is smaller than 1KB.
>>
>> +$Function BOOL substr(STRING, STRING)
>> +
>> +Description
>> + Returns true if the second string is a substring of the first
>> + string. Note that the comparison is case sensitive. You can
>> + use the tolower function on both strings if you want case
>> + insensitivity.
>> +Example
>> + if (std.substr(req.url, req.http.x-restrict))
>> +
>> +
>>
>>  SEE ALSO
>>  ========
>> diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
>> index 8872a1c..8a89bb8 100644
>> --- a/lib/libvmod_std/vmod_std.c
>> +++ b/lib/libvmod_std/vmod_std.c
>> @@ -239,3 +239,17 @@ vmod_cache_req_body(const struct vrt_ctx *ctx,
>> VCL_BYTES size)
>>   result = VRT_CacheReqBody(ctx, size);
>>   VSLb(ctx->vsl, SLT_Debug,"VRT_CacheReqBody(%zu): %d", (size_t)size,
>> result);
>>  }
>> +
>> +VCL_BOOL __match_proto__(td_std_substr)
>> +vmod_substr(const struct vrt_ctx *ctx, VCL_STRING mstr, VCL_STRING
>> msubstr)
>> +{
>> +
>> +        CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
>> +        char *match =  strstr(mstr, msubstr);
>> +
>> + if (match)
>> +  return(1);
>> + else
>> +  return(0);
>> +}
>> +
>> --
>> 1.9.3 (Apple Git-50)+GitX
>>
>> --
>> *Per Buer*
>> CTO | Varnish Software AS
>> Cell: +47 95839117
>> We Make Websites Fly!
>> www.varnish-software.com
>> [image: Register now]
>> <http://info.varnish-software.com/varnish-summits-autumn-2014-registration>
>>
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>
>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140920/609daf69/attachment.html>

From slink at schokola.de  Sat Sep 20 11:55:23 2014
From: slink at schokola.de (Nils Goroll)
Date: Sat, 20 Sep 2014 13:55:23 +0200
Subject: [PATCH] substring matching in std
In-Reply-To: <CAJV_h0YQeAkLnZy4bj=sDwkEuYRtLh_xXFnEJCRTCAGeGwtPeQ@mail.gmail.com>
References: <CAOXZevApN8RhTJbSOmZZ5Wy+1ArWnKM0hzKuU2-OKSgkKSCOfw@mail.gmail.com>
 <CANTTouXzDzUHuE4aP9S-ke=GYGX1nV4pvvsE3-P_ptvFNMjf2g@mail.gmail.com>
 <CAJV_h0YQeAkLnZy4bj=sDwkEuYRtLh_xXFnEJCRTCAGeGwtPeQ@mail.gmail.com>
Message-ID: <541D6B2B.4020400@schokola.de>

> The reason for this is that regex match doesn't work on dynamic strings.

https://code.uplex.de/uplex-varnish/libvmod-re



From perbu at varnish-software.com  Sat Sep 20 15:58:41 2014
From: perbu at varnish-software.com (Per Buer)
Date: Sat, 20 Sep 2014 17:58:41 +0200
Subject: [PATCH] substring matching in std
In-Reply-To: <CAJV_h0YQeAkLnZy4bj=sDwkEuYRtLh_xXFnEJCRTCAGeGwtPeQ@mail.gmail.com>
References: <CAOXZevApN8RhTJbSOmZZ5Wy+1ArWnKM0hzKuU2-OKSgkKSCOfw@mail.gmail.com>
 <CANTTouXzDzUHuE4aP9S-ke=GYGX1nV4pvvsE3-P_ptvFNMjf2g@mail.gmail.com>
 <CAJV_h0YQeAkLnZy4bj=sDwkEuYRtLh_xXFnEJCRTCAGeGwtPeQ@mail.gmail.com>
Message-ID: <CAOXZevBmLJ840-BHgUS5riFs+RHTj4tbr3geFoXVhCu0KW_ryw@mail.gmail.com>

Right. Renamed to strstr and returns the substring. The use is the more or
less the same.

>From 41498f59900079f8ec653ba517d867b35aea77cf Mon Sep 17 00:00:00 2001
Subject: [PATCH] add std.strstr

---
 bin/varnishtest/tests/m00018.vtc | 34 ++++++++++++++++++++++++++++++++++
 lib/libvmod_std/vmod.vcc         | 12 +++++++++++-
 lib/libvmod_std/vmod_std.c       | 11 +++++++++++
 3 files changed, 56 insertions(+), 1 deletion(-)
 create mode 100644 bin/varnishtest/tests/m00018.vtc

diff --git a/bin/varnishtest/tests/m00018.vtc
b/bin/varnishtest/tests/m00018.vtc
new file mode 100644
index 0000000..b51ed18
--- /dev/null
+++ b/bin/varnishtest/tests/m00018.vtc
@@ -0,0 +1,34 @@
+varnishtest "Test substring matching in std"
+
+server s1 {
+       rxreq
+       txresp
+
+       rxreq
+       txresp
+} -start
+
+varnish v1 -vcl+backend {
+ import ${vmod_std};
+
+ sub vcl_deliver {
+    if (std.strstr(req.url, "foo")) {
+ set resp.http.sub = "found";
+    } else {
+ set resp.http.sub = "not found";
+    }
+
+ }
+} -start
+
+client c1 {
+ txreq -url "/foobar"
+ rxresp
+ expect resp.http.sub == "found"
+
+
+ txreq -url "/quux"
+ rxresp
+ expect resp.http.sub == "not found"
+} -run
+
diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc
index fbc082b..455485a 100644
--- a/lib/libvmod_std/vmod.vcc
+++ b/lib/libvmod_std/vmod.vcc
@@ -195,7 +195,6 @@ $Function STRING querysort(STRING)

 Description
         Sorts the querystring for cache normalization purposes.
-
 Example
         set req.url = std.querysort(req.url);

@@ -208,6 +207,17 @@ Example

        This will cache the req.body if its size is smaller than 1KB.

+$Function STRING strstr(STRING, STRING)
+
+Description
+ Returns true if the second string is a substring of the first
+ string. Note that the comparison is case sensitive. You can
+ use the tolower function on both strings if you want case
+ insensitivity.
+Example
+ if (std.strstr(req.url, req.http.x-restrict))
+
+

 SEE ALSO
 ========
diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
index 8872a1c..898e8dd 100644
--- a/lib/libvmod_std/vmod_std.c
+++ b/lib/libvmod_std/vmod_std.c
@@ -239,3 +239,14 @@ vmod_cache_req_body(const struct vrt_ctx *ctx,
VCL_BYTES size)
  result = VRT_CacheReqBody(ctx, size);
  VSLb(ctx->vsl, SLT_Debug,"VRT_CacheReqBody(%zu): %d", (size_t)size,
result);
 }
+
+VCL_STRING __match_proto__(td_std_strstr)
+vmod_strstr(const struct vrt_ctx *ctx, VCL_STRING mstr, VCL_STRING msubstr)
+{
+
+        CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+        char *match =  strstr(mstr, msubstr);
+
+ return(match);
+}
+
-- 
1.9.3 (Apple Git-50)+GitX

On Sat, Sep 20, 2014 at 1:00 PM, Federico Schwindt <fgsch at lodoss.net> wrote:

> I'd prefer to keep substr (or rename it to strstr) but returning the
> substring instead so you can also use the  string.
>
> The reason for this is that regex match doesn't work on dynamic strings.
>  On 20 Sep 2014 10:31, "Rogier 'DocWilco' Mulhuijzen" <
> varnish at bsdchicks.com> wrote:
>
>> I'm not fond of using "substr" as the function name, as other languages
>> use this function with start/end or start/length parameters to create a new
>> string from the existing string.
>>
>> I would suggest one of the following:
>>
>> - strstr
>> - instr
>> - contains
>>
>> Also, just out of curiosity, exactly how much faster is this than a regex
>> match?
>>
>> On Sat, Sep 20, 2014 at 11:12 AM, Per Buer <perbu at varnish-software.com>
>> wrote:
>>
>>>  bin/varnishtest/tests/m00018.vtc | 34 ++++++++++++++++++++++++++++++++++
>>>  lib/libvmod_std/vmod.vcc         | 12 +++++++++++-
>>>  lib/libvmod_std/vmod_std.c       | 14 ++++++++++++++
>>>  3 files changed, 59 insertions(+), 1 deletion(-)
>>>  create mode 100644 bin/varnishtest/tests/m00018.vtc
>>>
>>> diff --git a/bin/varnishtest/tests/m00018.vtc
>>> b/bin/varnishtest/tests/m00018.vtc
>>> new file mode 100644
>>> index 0000000..af00716
>>> --- /dev/null
>>> +++ b/bin/varnishtest/tests/m00018.vtc
>>> @@ -0,0 +1,34 @@
>>> +varnishtest "Test substring matching in std"
>>> +
>>> +server s1 {
>>> +       rxreq
>>> +       txresp
>>> +
>>> +       rxreq
>>> +       txresp
>>> +} -start
>>> +
>>> +varnish v1 -vcl+backend {
>>> + import ${vmod_std};
>>> +
>>> + sub vcl_deliver {
>>> +    if (std.substr(req.url, "foo")) {
>>> + set resp.http.sub = "found";
>>> +    } else {
>>> + set resp.http.sub = "not found";
>>> +    }
>>> +
>>> + }
>>> +} -start
>>> +
>>> +client c1 {
>>> + txreq -url "/foobar"
>>> + rxresp
>>> + expect resp.http.sub == "found"
>>> +
>>> +
>>> + txreq -url "/quux"
>>> + rxresp
>>> + expect resp.http.sub == "not found"
>>> +} -run
>>> +
>>> diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc
>>> index fbc082b..e59c7a4 100644
>>> --- a/lib/libvmod_std/vmod.vcc
>>> +++ b/lib/libvmod_std/vmod.vcc
>>> @@ -195,7 +195,6 @@ $Function STRING querysort(STRING)
>>>
>>>  Description
>>>          Sorts the querystring for cache normalization purposes.
>>> -
>>>  Example
>>>          set req.url = std.querysort(req.url);
>>>
>>> @@ -208,6 +207,17 @@ Example
>>>
>>>         This will cache the req.body if its size is smaller than 1KB.
>>>
>>> +$Function BOOL substr(STRING, STRING)
>>> +
>>> +Description
>>> + Returns true if the second string is a substring of the first
>>> + string. Note that the comparison is case sensitive. You can
>>> + use the tolower function on both strings if you want case
>>> + insensitivity.
>>> +Example
>>> + if (std.substr(req.url, req.http.x-restrict))
>>> +
>>> +
>>>
>>>  SEE ALSO
>>>  ========
>>> diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
>>> index 8872a1c..8a89bb8 100644
>>> --- a/lib/libvmod_std/vmod_std.c
>>> +++ b/lib/libvmod_std/vmod_std.c
>>> @@ -239,3 +239,17 @@ vmod_cache_req_body(const struct vrt_ctx *ctx,
>>> VCL_BYTES size)
>>>   result = VRT_CacheReqBody(ctx, size);
>>>   VSLb(ctx->vsl, SLT_Debug,"VRT_CacheReqBody(%zu): %d", (size_t)size,
>>> result);
>>>  }
>>> +
>>> +VCL_BOOL __match_proto__(td_std_substr)
>>> +vmod_substr(const struct vrt_ctx *ctx, VCL_STRING mstr, VCL_STRING
>>> msubstr)
>>> +{
>>> +
>>> +        CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
>>> +        char *match =  strstr(mstr, msubstr);
>>> +
>>> + if (match)
>>> +  return(1);
>>> + else
>>> +  return(0);
>>> +}
>>> +
>>> --
>>> 1.9.3 (Apple Git-50)+GitX
>>>
>>> --
>>> *Per Buer*
>>> CTO | Varnish Software AS
>>> Cell: +47 95839117
>>> We Make Websites Fly!
>>> www.varnish-software.com
>>> [image: Register now]
>>> <http://info.varnish-software.com/varnish-summits-autumn-2014-registration>
>>>
>>> _______________________________________________
>>> varnish-dev mailing list
>>> varnish-dev at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>>
>>
>>
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>
>


-- 
*Per Buer*
CTO | Varnish Software AS
Cell: +47 95839117
We Make Websites Fly!
www.varnish-software.com
[image: Register now]
<http://info.varnish-software.com/varnish-summits-autumn-2014-registration>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140920/a1ffda8d/attachment.html>

From phk at phk.freebsd.dk  Mon Sep 22 07:09:59 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 22 Sep 2014 07:09:59 +0000
Subject: [PATCH] substring matching in std
In-Reply-To: <CAOXZevBmLJ840-BHgUS5riFs+RHTj4tbr3geFoXVhCu0KW_ryw@mail.gmail.com>
References: <CAOXZevApN8RhTJbSOmZZ5Wy+1ArWnKM0hzKuU2-OKSgkKSCOfw@mail.gmail.com>
 <CANTTouXzDzUHuE4aP9S-ke=GYGX1nV4pvvsE3-P_ptvFNMjf2g@mail.gmail.com>
 <CAJV_h0YQeAkLnZy4bj=sDwkEuYRtLh_xXFnEJCRTCAGeGwtPeQ@mail.gmail.com>
 <CAOXZevBmLJ840-BHgUS5riFs+RHTj4tbr3geFoXVhCu0KW_ryw@mail.gmail.com>
Message-ID: <63053.1411369799@critter.freebsd.dk>

--------
In message <CAOXZevBmLJ840-BHgUS5riFs+RHTj4tbr3geFoXVhCu0KW_ryw at mail.gmail.com>
, Per Buer writes:

>Right. Renamed to strstr and returns the substring. The use is the more or
>less the same.

Looks good, but two questions:

A) Should it use strcasestr() ?

or

B) should we have both std.strstr() an std.strcasestr() ?

Commit when you agree on the answer and have tested the resulting patch...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From phk at phk.freebsd.dk  Mon Sep 22 07:37:20 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 22 Sep 2014 07:37:20 +0000
Subject: Update: [PATCH 1/2] Add a function to remove an element/token
 from a header, value (like Accept-Encoding from Vary:)
In-Reply-To: <5419C6B1.9010305@schokola.de>
References: <53FE259A.30800@schokola.de> <5419C6B1.9010305@schokola.de>
Message-ID: <63683.1411371440@critter.freebsd.dk>

--------
In message <5419C6B1.9010305 at schokola.de>, Nils Goroll writes:

>With this change, http_RemoveHdrToken also removes any values of an element
>token seperated by equals (like foo=bar)
>
>phk, does this work for you?

Question:  What if the target token appears more than once ?

Overall I find the function much more complex than I like.  I think it
is because the answer http_GetHdrToken() gives is not what we need to
know.

I would do it something like this instead:

	did = 0
	Reserve workspace
	loop over tokens in src-header
		if token != target
			copy token to workspace
			did = 1
	if (did):
		release unused workspace
	else:
		return all reserved workspace
	return (did)

That would largely be a duplication of http_GetHdrToken() just a few
lines longer.

When we want to do more token-based functions, we should probably make
some foreach_token() infrastructure, but lets wait until #3 case crops
up...


Two security related observations:

>+	if (! (http_GetHdr(hp, hdr, &bp) &&
>+		http_GetHdrToken(hp, hdr, token, &tp)))
>+		return (0);
>+
>+	/* GetHdrToken returns a pointer to the next char after the token */
>+	tl = strlen(token);

Subtracting from pointers is a prime minefield for security issues,
and therefore I generally try to avoid it at all cost.  If I have
to do it, I try to plug in asserts which A) show that we know what
we're doing and B) stop us of we were wrong about that.

So here I would add:

	assert(tp - tl >= bp);

>+	tp -= tl;


This one is a big NO-NO:

>+		VSLb(hp->vsl, SLT_LostHeader, hdr);

You must *always* supply a const char* format string:

>+		VSLb(hp->vsl, SLT_LostHeader, "%s", hdr);

Otherwise a header with '%' in it will make the printf access random
memory.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From phk at phk.freebsd.dk  Mon Sep 22 07:46:30 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 22 Sep 2014 07:46:30 +0000
Subject: [PATCH] substring matching in std
In-Reply-To: <63053.1411369799@critter.freebsd.dk>
References: <CAOXZevApN8RhTJbSOmZZ5Wy+1ArWnKM0hzKuU2-OKSgkKSCOfw@mail.gmail.com>
 <CANTTouXzDzUHuE4aP9S-ke=GYGX1nV4pvvsE3-P_ptvFNMjf2g@mail.gmail.com>
 <CAJV_h0YQeAkLnZy4bj=sDwkEuYRtLh_xXFnEJCRTCAGeGwtPeQ@mail.gmail.com>
 <CAOXZevBmLJ840-BHgUS5riFs+RHTj4tbr3geFoXVhCu0KW_ryw@mail.gmail.com>
 <63053.1411369799@critter.freebsd.dk>
Message-ID: <63798.1411371990@critter.freebsd.dk>

--------
In message <63053.1411369799 at critter.freebsd.dk>, "Poul-Henning Kamp" writes:
>--------
>In message <CAOXZevBmLJ840-BHgUS5riFs+RHTj4tbr3geFoXVhCu0KW_ryw at mail.gmail.com>
>, Per Buer writes:
>
>>Right. Renamed to strstr and returns the substring. The use is the more or
>>less the same.
>
>Looks good, but two questions:
>
>A) Should it use strcasestr() ?
>
>or
>
>B) should we have both std.strstr() an std.strcasestr() ?
>
>Commit when you agree on the answer and have tested the resulting patch...

Ohh, and:  It needs to test for NULL inputs...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From slink at schokola.de  Mon Sep 22 10:59:58 2014
From: slink at schokola.de (Nils Goroll)
Date: Mon, 22 Sep 2014 12:59:58 +0200
Subject: Update: [PATCH 1/2] Add a function to remove an element/token
 from a header, value (like Accept-Encoding from Vary:)
In-Reply-To: <63683.1411371440@critter.freebsd.dk>
References: <53FE259A.30800@schokola.de> <5419C6B1.9010305@schokola.de>
 <63683.1411371440@critter.freebsd.dk>
Message-ID: <5420012E.7080908@schokola.de>

Phk,

this really is a helpful review, thank you for all the advice.

I will reiterate

Nils



From stanhope at gmail.com  Mon Sep 22 19:15:39 2014
From: stanhope at gmail.com (Phil Stanhope)
Date: Mon, 22 Sep 2014 15:15:39 -0400
Subject: Regression in std.syslog
Message-ID: <CADK3dNQudXr-GBpNdsZGoeXY-eecjDwGFaDS1nOd7F8hTybk+A@mail.gmail.com>

Hi all,

I've just found a regression in the syslog functionality in v4 std module.
Not sure when/where it emerged.

The log(...) method and the syslog(...) methods where updated to use
workspace. But the implementation is slightly different and the result is
that the syslog method writes an empty string to syslog.

...
CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
u = WS_Reserve(ctx->ws, 0);
p = ctx->ws->f;
va_start(ap, fmt);
p = VRT_StringList(p, u, fmt, ap);
va_end(ap);
if (p != NULL)
  syslog((int)fac, "%s", p);
WS_Release(ctx->ws, 0);
...

I think it should be:

...
* txt t;*
CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
u = WS_Reserve(ctx->ws, 0);
* t.b = ctx->ws->f;*
va_start(ap, fmt);
* t.e = VRT_StringList(t.b, u, fmt, ap);*
va_end(ap);
* if (t.e != NULL) {*
*  assert(t.e > t.b);*
*  t.e--;*
*  syslog((int)fac, "%s", t.b);*
* }*
WS_Release(ctx->ws, 0);
...

Thoughts?

-phil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140922/273e5531/attachment.html>

From apj at mutt.dk  Tue Sep 23 06:48:23 2014
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Tue, 23 Sep 2014 08:48:23 +0200
Subject: Regression in std.syslog
In-Reply-To: <CADK3dNQudXr-GBpNdsZGoeXY-eecjDwGFaDS1nOd7F8hTybk+A@mail.gmail.com>
References: <CADK3dNQudXr-GBpNdsZGoeXY-eecjDwGFaDS1nOd7F8hTybk+A@mail.gmail.com>
Message-ID: <20140923064823.GW19870@nerd.dk>

On Mon, Sep 22, 2014 at 03:15:39PM -0400, Phil Stanhope wrote:
> 
> I've just found a regression in the syslog functionality in v4 std module.
> Not sure when/where it emerged.
> 
> The log(...) method and the syslog(...) methods where updated to use
> workspace. But the implementation is slightly different and the result is
> that the syslog method writes an empty string to syslog.

Already fixed in master: https://www.varnish-cache.org/trac/ticket/1574

-- 
Andreas



From slink at schokola.de  Sun Sep 28 17:43:30 2014
From: slink at schokola.de (Nils Goroll)
Date: Sun, 28 Sep 2014 19:43:30 +0200
Subject: Would appreciate reciews of a header element parsing/modification
 prototype
In-Reply-To: <63683.1411371440@critter.freebsd.dk>
References: <53FE259A.30800@schokola.de> <5419C6B1.9010305@schokola.de>
 <63683.1411371440@critter.freebsd.dk>
Message-ID: <542848C2.7080900@schokola.de>

Hi,

On 22/09/14 09:37, Poul-Henning Kamp wrote:
> 
> I would do it something like this instead:
> 
> 	did = 0
> 	Reserve workspace
> 	loop over tokens in src-header
> 		if token != target
> 			copy token to workspace
> 			did = 1
> 	if (did):
> 		release unused workspace
> 	else:
> 		return all reserved workspace
> 	return (did)
> 
> That would largely be a duplication of http_GetHdrToken() just a few
> lines longer.

I actually have done an implementation along these lines, but I did not like it
at all, it got really lengthy, lacked generality etc etc.


With the idea in mind that we could have general header element editing
facilities (no matter in a vmod or core/vcl), I have written a more generic
prototype around the suggested http_RemoveHdrToken function following the design
example of struct http and the HTTP_* functions.

For the protoype I have worked in cache_http.c, the source structure and the
question of which functions to declare static are things I'd like to think about
later.

struct httphdr holds two txt arrays for the parse of a header line - one for
names and one for values (think "max-age=200")

HTTPHDR_estimate and HTTPHDR_create closely follow the HTTP_ examples.

HTTPHDR_Tokenize dissects a header line into element keys and values. The
seperating characters are to be generalized later.

HTTPHDR_Join constructs a header line from a struct httphdr on some buffer provided.

HTTPHDR_SameToken compares two header elemnts passed as (c)txt structs either
case sensitively for the quoted or case insensitively for the unquoted case.

HTTPHDR_RemoveToken nulls all instances of a token name in a struct httphdr

http_RemoveHdrToken (and the corresponding VRT) is the PoC implementation. It
builds the struct httphdr parse on the stack and only leaves a finalized header
on the workspace.

bin/varnishtest/tests/removehdrtoken.vtc is a working test for the remove header
token example.


I'd very much appreciate feedback on this.

Nils
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-prototype-header-element-parsing.patch
Type: text/x-patch
Size: 14712 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140928/c130549d/attachment.bin>

From slink at schokola.de  Sun Sep 28 18:25:18 2014
From: slink at schokola.de (Nils Goroll)
Date: Sun, 28 Sep 2014 20:25:18 +0200
Subject: Update1: Would appreciate reciews of a header element
 parsing/modification prototype
In-Reply-To: <63683.1411371440@critter.freebsd.dk>
References: <53FE259A.30800@schokola.de> <5419C6B1.9010305@schokola.de>
 <63683.1411371440@critter.freebsd.dk>
Message-ID: <5428528E.3020302@schokola.de>

Update after txt constification (my fault :P)

iow this patch is based upon e40edceffff77d0578237a85cf5109cd0d069599

	-

Hi,

On 22/09/14 09:37, Poul-Henning Kamp wrote:
> 
> I would do it something like this instead:
> 
> 	did = 0
> 	Reserve workspace
> 	loop over tokens in src-header
> 		if token != target
> 			copy token to workspace
> 			did = 1
> 	if (did):
> 		release unused workspace
> 	else:
> 		return all reserved workspace
> 	return (did)
> 
> That would largely be a duplication of http_GetHdrToken() just a few
> lines longer.

I actually have done an implementation along these lines, but I did not like it
at all, it got really lengthy, lacked generality etc etc.


With the idea in mind that we could have general header element editing
facilities (no matter in a vmod or core/vcl), I have written a more generic
prototype around the suggested http_RemoveHdrToken function following the design
example of struct http and the HTTP_* functions.

For the protoype I have worked in cache_http.c, the source structure and the
question of which functions to declare static are things I'd like to think about
later.

struct httphdr holds two txt arrays for the parse of a header line - one for
names and one for values (think "max-age=200")

HTTPHDR_estimate and HTTPHDR_create closely follow the HTTP_ examples.

HTTPHDR_Tokenize dissects a header line into element keys and values. The
seperating characters are to be generalized later.

HTTPHDR_Join constructs a header line from a struct httphdr on some buffer provided.

HTTPHDR_SameToken compares two header elemnts passed as (c)txt structs either
case sensitively for the quoted or case insensitively for the unquoted case.

HTTPHDR_RemoveToken nulls all instances of a token name in a struct httphdr

http_RemoveHdrToken (and the corresponding VRT) is the PoC implementation. It
builds the struct httphdr parse on the stack and only leaves a finalized header
on the workspace.

bin/varnishtest/tests/removehdrtoken.vtc is a working test for the remove header
token example.


I'd very much appreciate feedback on this.

Nils

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-prototype-header-element-parsing-update1.patch
Type: text/x-patch
Size: 14017 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20140928/4c73dd53/attachment.bin>