From guillaume at varnish-software.com  Tue Aug  1 07:34:39 2017
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Tue, 1 Aug 2017 09:34:39 +0200
Subject: [Varnish 4] Respecting client's Cache-Control: max-age= as TTL
In-Reply-To: <CAE35VmzDdg0wwwgtXGM8kgdLVXi6x+S_j91kFKvsKZX8VW0hFQ@mail.gmail.com>
References: <CAE35VmwyvgFiTj+aKp4LsgGPD4nCr4SNN9sjSe7n5bXS6t3Lxw@mail.gmail.com>
 <CAJ6ZYQzj1Y6pDDerhs8Mu5oWG6rCuO1uyWrQaM2TG20NzoP7dg@mail.gmail.com>
 <CAE35Vmzo2Bu5KQqVwkWSgSVCw2aoQ967Ujwafaw0VESH+74GgA@mail.gmail.com>
 <CAJ6ZYQxDaFmRnXxmBe7yU7iJz_UPJu15EwpT4HJRJPJn7j+=uQ@mail.gmail.com>
 <CAE35VmxXj1FFAjQQVLeUrk76j_bgHzzuiu1PoSgQ7cvEkUUCAQ@mail.gmail.com>
 <CAE35VmzDdg0wwwgtXGM8kgdLVXi6x+S_j91kFKvsKZX8VW0hFQ@mail.gmail.com>
Message-ID: <CAJ6ZYQx6eqxEL9GgM00PEeTCphxvxff27iONv_BrFg16CdxyZw@mail.gmail.com>

Ah, right, I totally forgot about that, sorry.

Sooooooo, there's no real clean way to do it (that I can see, smarter
people than me may have a solution), but here's what I can offer.

sub vcl_recv {
    if (req.restarts == 0) {
        unset req.http.force_miss;
    } else if (req.http.force_miss) {
        set req.hash_alway_miss
    }
}

sub vcl_hit {
    if (CONDITION TO BYPASS CACHE) {
        set req.http.force_miss = "yes";
        return(restart);
    }
}



-- 
Guillaume Quintard

On Tue, Aug 1, 2017 at 12:10 AM, Martynas Jusevi?ius <martynas at atomgraph.com
> wrote:

> Sorry, sent too soon. Here it goes:
>
> Thanks Guillaume.
>
> First I tried return(fetch) as you suggested
>
> sub vcl_hit {
>     if (req.http.Cache-Control ~ "max-age=[0-9]*") {
>         set req.http.Max-Age = regsub(req.http.Cache-Control,
> "max-age=([0-9]*)", "\1");
>         if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
>             std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
> req.http.Max-Age);
>             return(fetch);
>         }
>     }
>     ...
>
> but I got an error:
>
> -   VCL_call       HIT
> -   ReqHeader      Max-Age: 69
> -   VCL_Log        obj.age: 102.306 req.http.Max-Age: 69
> -   VCL_return     fetch
> -   VCL_Error      change return(fetch) to return(miss) in vcl_hit{}
> -   VCL_Error      vcl_hit{} returns miss without busy object.  Doing pass.
> -   VCL_call       PASS
> -   VCL_return     fetch
>
> I did as told and I tried return(miss)
>
> sub vcl_hit {
>     if (req.http.Cache-Control ~ "max-age=[0-9]*") {
>         set req.http.Max-Age = regsub(req.http.Cache-Control,
> "max-age=([0-9]*)", "\1");
>         if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
>             std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
> req.http.Max-Age);
>             return(miss);
>         }
>     }
>     ...
>
> but then I got another error:
>
> -   VCL_call       HIT
> -   ReqHeader      Max-Age: 69
> -   VCL_Log        obj.age: 195.391 req.http.Max-Age: 69
> -   VCL_return     miss
> -   VCL_Error      vcl_hit{} returns miss without busy object.  Doing pass.
> -   VCL_call       PASS
> -   VCL_return     fetch
>
> So it looks like the max-age logic is triggered correctly, but what is
> wrong with the return values?
>
> On Tue, Aug 1, 2017 at 12:01 AM, Martynas Jusevi?ius <
> martynas at atomgraph.com> wrote:
>
>> Thanks Guillaume.
>>
>> First I tried
>>
>> sub vcl_hit {
>>     if (req.http.Cache-Control ~ "max-age=[0-9]*") {
>>         set req.http.Max-Age = regsub(req.http.Cache-Control,
>> "max-age=([0-9]*)", "\1");
>>         if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
>>             std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
>> req.http.Max-Age);
>>             return(fetch);
>>         }
>>     }
>>     ...
>>
>> but I got an error:
>>
>> -   VCL_call       HIT
>> -   ReqHeader      Max-Age: 69
>> -   VCL_Log        obj.age: 102.306 req.http.Max-Age: 69
>> -   VCL_return     fetch
>> -   VCL_Error      change return(fetch) to return(miss) in vcl_hit{}
>> -   VCL_Error      vcl_hit{} returns miss without busy object.  Doing
>> pass.
>> -   VCL_call       PASS
>> -   VCL_return     fetch
>>
>> I did as told and I tried
>>
>>
>> On Mon, Jul 31, 2017 at 9:11 PM, Guillaume Quintard <
>> guillaume at varnish-software.com> wrote:
>>
>>> man vcl
>>>
>>> bereq is filtered to avoid side effects of the client forcing the ttl to
>>> the backed.
>>>
>>> Anyway, by the time you have access to bereq, it's too late for you
>>> since the decision to go to the backend has already been been made.
>>>
>>> --
>>> Guillaume Quintard
>>>
>>>
>>> On Jul 31, 2017 19:56, "Martynas Jusevi?ius" <martynas at atomgraph.com>
>>> wrote:
>>>
>>> Thanks. What was mostly unclear to me is passing the req header value
>>> all the way to where it's used to set TTL.
>>>
>>> Why doesn't bereq contain the req headers? At least Cache-Control is
>>> gone.
>>>
>>> But I guess that can be done using obj.ttl, which I didn't know about.
>>> Any documentation on that?
>>>
>>> On Mon, 31 Jul 2017 at 18.38, Guillaume Quintard <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> On github I pointed to the doc explaining how you can return(fetch) to
>>>> ignore a cached object, possibly based on ttl, so you already have half the
>>>> answer.
>>>>
>>>> The other part of the equation is just converting
>>>> req.http.cache-control to a duration and comparing that to obj.ttl. It will
>>>> be similar to what you have done on v3.
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Jul 31, 2017 18:25, "Martynas Jusevi?ius" <martynas at atomgraph.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have been reading quite a bit about Varnish and VCL but found almost
>>>>> no examples with Cache-Control coming from the client request [1].
>>>>>
>>>>> What I want to achieve: if the client sends Cache-Control: max-age=60,
>>>>> TTL becomes 60 s. If the cache hit is fresher than 60 s, deliver it,
>>>>> otherwise fetch a new response from backend (I hope I'm not misusing the
>>>>> VCL terms here) *and* cache it.
>>>>>
>>>>> I had hacked this together in the vcl_fetch section in Varnish 3.x by
>>>>> setting the req.http.Cache-Control max-age value as beresp.ttl, but
>>>>> vcl_fetch is gone in Varnish 4.x.
>>>>>
>>>>> I have received a suggestion to use vcl_hit and/or grace [2], but
>>>>> again -- no examples...
>>>>>
>>>>> Could anyone provide some VCL pseudo-code that
>>>>> uses req.http.Cache-Control value to override TTL? max-age number parsing
>>>>> not necessary, I have figure that out.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Martynas
>>>>>
>>>>> [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Header
>>>>> s/Cache-Control#Cache_request_directives
>>>>> [2] https://github.com/varnishcache/varnish-cache/issues/201
>>>>> 4#issuecomment-319096566
>>>>>
>>>>> _______________________________________________
>>>>> varnish-misc mailing list
>>>>> varnish-misc at varnish-cache.org
>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170801/1bf5b939/attachment.html>

From dridi at varni.sh  Tue Aug  1 09:04:16 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 1 Aug 2017 11:04:16 +0200
Subject: [Varnish 4] Respecting client's Cache-Control: max-age= as TTL
In-Reply-To: <CAJ6ZYQx6eqxEL9GgM00PEeTCphxvxff27iONv_BrFg16CdxyZw@mail.gmail.com>
References: <CAE35VmwyvgFiTj+aKp4LsgGPD4nCr4SNN9sjSe7n5bXS6t3Lxw@mail.gmail.com>
 <CAJ6ZYQzj1Y6pDDerhs8Mu5oWG6rCuO1uyWrQaM2TG20NzoP7dg@mail.gmail.com>
 <CAE35Vmzo2Bu5KQqVwkWSgSVCw2aoQ967Ujwafaw0VESH+74GgA@mail.gmail.com>
 <CAJ6ZYQxDaFmRnXxmBe7yU7iJz_UPJu15EwpT4HJRJPJn7j+=uQ@mail.gmail.com>
 <CAE35VmxXj1FFAjQQVLeUrk76j_bgHzzuiu1PoSgQ7cvEkUUCAQ@mail.gmail.com>
 <CAE35VmzDdg0wwwgtXGM8kgdLVXi6x+S_j91kFKvsKZX8VW0hFQ@mail.gmail.com>
 <CAJ6ZYQx6eqxEL9GgM00PEeTCphxvxff27iONv_BrFg16CdxyZw@mail.gmail.com>
Message-ID: <CABoVN9Dr05yYYt==YuBVfsLZ+rgKRNoYkH85v+GjJO_k4EnnfA@mail.gmail.com>

On Tue, Aug 1, 2017 at 9:34 AM, Guillaume Quintard
<guillaume at varnish-software.com> wrote:
> Ah, right, I totally forgot about that, sorry.
>
> Sooooooo, there's no real clean way to do it (that I can see, smarter people
> than me may have a solution), but here's what I can offer.

First, I would say that honoring a client's cache-control is a
terrible idea: it's a glorified DoS vector.

Now if you really want to do that (eg. you have a trustworthy client,
say a script, that drives caching decisions) you still can't outlive
the object's TTL (the one that was picked at the end of
vcl_backend_getch). Of course you need to assert the trustworthiness
of such requests too. It becomes quite cumbersome to allow clients to
make decisions.

Sorry, I can't help, I will only say "don't do it" :)

Cheers



From martynas at atomgraph.com  Tue Aug  1 17:17:15 2017
From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=)
Date: Tue, 1 Aug 2017 19:17:15 +0200
Subject: [Varnish 4] Respecting client's Cache-Control: max-age= as TTL
In-Reply-To: <CAJ6ZYQx6eqxEL9GgM00PEeTCphxvxff27iONv_BrFg16CdxyZw@mail.gmail.com>
References: <CAE35VmwyvgFiTj+aKp4LsgGPD4nCr4SNN9sjSe7n5bXS6t3Lxw@mail.gmail.com>
 <CAJ6ZYQzj1Y6pDDerhs8Mu5oWG6rCuO1uyWrQaM2TG20NzoP7dg@mail.gmail.com>
 <CAE35Vmzo2Bu5KQqVwkWSgSVCw2aoQ967Ujwafaw0VESH+74GgA@mail.gmail.com>
 <CAJ6ZYQxDaFmRnXxmBe7yU7iJz_UPJu15EwpT4HJRJPJn7j+=uQ@mail.gmail.com>
 <CAE35VmxXj1FFAjQQVLeUrk76j_bgHzzuiu1PoSgQ7cvEkUUCAQ@mail.gmail.com>
 <CAE35VmzDdg0wwwgtXGM8kgdLVXi6x+S_j91kFKvsKZX8VW0hFQ@mail.gmail.com>
 <CAJ6ZYQx6eqxEL9GgM00PEeTCphxvxff27iONv_BrFg16CdxyZw@mail.gmail.com>
Message-ID: <CAE35Vmx=nTeAW0eoJ14m9UvogKtqR0r4GyLYk5w3xpJqGRWF+A@mail.gmail.com>

Guillaume,

after I fixed a couple of typos, this *seems* to work now:

sub vcl_recv {
...
    if (req.restarts == 0) {
        unset req.http.force_miss;

if (req.http.x-forwarded-for) {
   set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
   set req.http.X-Forwarded-For = client.ip;
}
    }
    else if (req.http.force_miss) {
        set req.hash_always_miss = true;
    }
...


sub vcl_hit {
    if (req.http.Cache-Control ~ "max-age=[0-9]*") {
        set req.http.Max-Age = regsub(req.http.Cache-Control,
"max-age=([0-9]*)", "\1");
        if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {

   set req.http.force_miss = "yes";
            return(restart);
}
    }
...

After return(restart) in vcl_hit I can see a new BeReq which calls
BACKEND_FETCH, new BeResp which calls BACKEND_RESPONSE, and then restarted
Request which triggers hash_always_miss and then calls RECV, HASH, MISS,
and finally DELIVER.

If that looks right, it would be nice to have it documented somewhere for
future reference.

Thanks for your help.


On Tue, Aug 1, 2017 at 9:34 AM, Guillaume Quintard <
guillaume at varnish-software.com> wrote:

> Ah, right, I totally forgot about that, sorry.
>
> Sooooooo, there's no real clean way to do it (that I can see, smarter
> people than me may have a solution), but here's what I can offer.
>
> sub vcl_recv {
>     if (req.restarts == 0) {
>         unset req.http.force_miss;
>     } else if (req.http.force_miss) {
>         set req.hash_alway_miss
>     }
> }
>
> sub vcl_hit {
>     if (CONDITION TO BYPASS CACHE) {
>         set req.http.force_miss = "yes";
>         return(restart);
>     }
> }
>
>
>
> --
> Guillaume Quintard
>
> On Tue, Aug 1, 2017 at 12:10 AM, Martynas Jusevi?ius <
> martynas at atomgraph.com> wrote:
>
>> Sorry, sent too soon. Here it goes:
>>
>> Thanks Guillaume.
>>
>> First I tried return(fetch) as you suggested
>>
>> sub vcl_hit {
>>     if (req.http.Cache-Control ~ "max-age=[0-9]*") {
>>         set req.http.Max-Age = regsub(req.http.Cache-Control,
>> "max-age=([0-9]*)", "\1");
>>         if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
>>             std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
>> req.http.Max-Age);
>>             return(fetch);
>>         }
>>     }
>>     ...
>>
>> but I got an error:
>>
>> -   VCL_call       HIT
>> -   ReqHeader      Max-Age: 69
>> -   VCL_Log        obj.age: 102.306 req.http.Max-Age: 69
>> -   VCL_return     fetch
>> -   VCL_Error      change return(fetch) to return(miss) in vcl_hit{}
>> -   VCL_Error      vcl_hit{} returns miss without busy object.  Doing
>> pass.
>> -   VCL_call       PASS
>> -   VCL_return     fetch
>>
>> I did as told and I tried return(miss)
>>
>> sub vcl_hit {
>>     if (req.http.Cache-Control ~ "max-age=[0-9]*") {
>>         set req.http.Max-Age = regsub(req.http.Cache-Control,
>> "max-age=([0-9]*)", "\1");
>>         if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
>>             std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
>> req.http.Max-Age);
>>             return(miss);
>>         }
>>     }
>>     ...
>>
>> but then I got another error:
>>
>> -   VCL_call       HIT
>> -   ReqHeader      Max-Age: 69
>> -   VCL_Log        obj.age: 195.391 req.http.Max-Age: 69
>> -   VCL_return     miss
>> -   VCL_Error      vcl_hit{} returns miss without busy object.  Doing
>> pass.
>> -   VCL_call       PASS
>> -   VCL_return     fetch
>>
>> So it looks like the max-age logic is triggered correctly, but what is
>> wrong with the return values?
>>
>> On Tue, Aug 1, 2017 at 12:01 AM, Martynas Jusevi?ius <
>> martynas at atomgraph.com> wrote:
>>
>>> Thanks Guillaume.
>>>
>>> First I tried
>>>
>>> sub vcl_hit {
>>>     if (req.http.Cache-Control ~ "max-age=[0-9]*") {
>>>         set req.http.Max-Age = regsub(req.http.Cache-Control,
>>> "max-age=([0-9]*)", "\1");
>>>         if (obj.age > std.duration(req.http.Max-Age + "s", 1000000s)) {
>>>             std.log("obj.age: " + obj.age + " req.http.Max-Age: " +
>>> req.http.Max-Age);
>>>             return(fetch);
>>>         }
>>>     }
>>>     ...
>>>
>>> but I got an error:
>>>
>>> -   VCL_call       HIT
>>> -   ReqHeader      Max-Age: 69
>>> -   VCL_Log        obj.age: 102.306 req.http.Max-Age: 69
>>> -   VCL_return     fetch
>>> -   VCL_Error      change return(fetch) to return(miss) in vcl_hit{}
>>> -   VCL_Error      vcl_hit{} returns miss without busy object.  Doing
>>> pass.
>>> -   VCL_call       PASS
>>> -   VCL_return     fetch
>>>
>>> I did as told and I tried
>>>
>>>
>>> On Mon, Jul 31, 2017 at 9:11 PM, Guillaume Quintard <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> man vcl
>>>>
>>>> bereq is filtered to avoid side effects of the client forcing the ttl
>>>> to the backed.
>>>>
>>>> Anyway, by the time you have access to bereq, it's too late for you
>>>> since the decision to go to the backend has already been been made.
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>>
>>>> On Jul 31, 2017 19:56, "Martynas Jusevi?ius" <martynas at atomgraph.com>
>>>> wrote:
>>>>
>>>> Thanks. What was mostly unclear to me is passing the req header value
>>>> all the way to where it's used to set TTL.
>>>>
>>>> Why doesn't bereq contain the req headers? At least Cache-Control is
>>>> gone.
>>>>
>>>> But I guess that can be done using obj.ttl, which I didn't know about.
>>>> Any documentation on that?
>>>>
>>>> On Mon, 31 Jul 2017 at 18.38, Guillaume Quintard <
>>>> guillaume at varnish-software.com> wrote:
>>>>
>>>>> On github I pointed to the doc explaining how you can return(fetch) to
>>>>> ignore a cached object, possibly based on ttl, so you already have half the
>>>>> answer.
>>>>>
>>>>> The other part of the equation is just converting
>>>>> req.http.cache-control to a duration and comparing that to obj.ttl. It will
>>>>> be similar to what you have done on v3.
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>> On Jul 31, 2017 18:25, "Martynas Jusevi?ius" <martynas at atomgraph.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have been reading quite a bit about Varnish and VCL but found
>>>>>> almost no examples with Cache-Control coming from the client request [1].
>>>>>>
>>>>>> What I want to achieve: if the client sends Cache-Control:
>>>>>> max-age=60, TTL becomes 60 s. If the cache hit is fresher than 60 s,
>>>>>> deliver it, otherwise fetch a new response from backend (I hope I'm not
>>>>>> misusing the VCL terms here) *and* cache it.
>>>>>>
>>>>>> I had hacked this together in the vcl_fetch section in Varnish 3.x by
>>>>>> setting the req.http.Cache-Control max-age value as beresp.ttl, but
>>>>>> vcl_fetch is gone in Varnish 4.x.
>>>>>>
>>>>>> I have received a suggestion to use vcl_hit and/or grace [2], but
>>>>>> again -- no examples...
>>>>>>
>>>>>> Could anyone provide some VCL pseudo-code that
>>>>>> uses req.http.Cache-Control value to override TTL? max-age number parsing
>>>>>> not necessary, I have figure that out.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Martynas
>>>>>>
>>>>>> [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Header
>>>>>> s/Cache-Control#Cache_request_directives
>>>>>> [2] https://github.com/varnishcache/varnish-cache/issues/201
>>>>>> 4#issuecomment-319096566
>>>>>>
>>>>>> _______________________________________________
>>>>>> varnish-misc mailing list
>>>>>> varnish-misc at varnish-cache.org
>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170801/bda565b2/attachment.html>

From rbizzell at measinc.com  Thu Aug  3 19:17:48 2017
From: rbizzell at measinc.com (Rodney Bizzell)
Date: Thu, 3 Aug 2017 19:17:48 +0000
Subject: Socket error
Message-ID: <2377626b65be473a97bf634df3d31e93@mbx2serv.meas-inc.com>

Hello
What would possibly cause a could not get socket  cannot assign requested address in my config file I have the ip address of the web server assigned to port 80?


This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. |
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170803/7bc16660/attachment.html>

From lagged at gmail.com  Fri Aug  4 06:13:47 2017
From: lagged at gmail.com (Andrei)
Date: Fri, 4 Aug 2017 01:13:47 -0500
Subject: Socket error
In-Reply-To: <2377626b65be473a97bf634df3d31e93@mbx2serv.meas-inc.com>
References: <2377626b65be473a97bf634df3d31e93@mbx2serv.meas-inc.com>
Message-ID: <CAP+vvEuNvm9v6VbptM3xHcaJzjw2HgMUDkZ5uARwV=U6+OF-VA@mail.gmail.com>

Please provide more details regarding your setup, and the full error. If
you're certain there's nothing listening on the port, and you're still
getting the error, I'd check for selinux, portreserve, and straggling
semaphores

On Thu, Aug 3, 2017 at 2:17 PM, Rodney Bizzell <rbizzell at measinc.com> wrote:

> Hello
>
> What would possibly cause a could not get socket  cannot assign requested
> address in my config file I have the ip address of the web server assigned
> to port 80?
>
>
> This email (including any attachments) may contain confidential
> information intended solely for acknowledged recipients. If you think you
> have received this information in error, please reply to the sender and
> delete all copies from your system. Please note that unauthorized use,
> disclosure, or further distribution of this information is prohibited by
> the sender. Note also that we may monitor email directed to or originating
> from our network. Thank you for your consideration and assistance. |
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170804/05e77f8d/attachment.html>

From miguel_3_gonzalez at yahoo.es  Tue Aug  8 20:14:12 2017
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Tue, 8 Aug 2017 22:14:12 +0200
Subject: troubleshooting Varnish purges vs bans
Message-ID: <68e12477-a8f5-fb8e-d262-0804c71cc78d@yahoo.es>

Hi all,

  I have Wordpress sites server by an Apache Server (Cpanel) behind
Varnish.

  I?m trying to troubleshoot why Varnish http purge Wordpress plugin is
not purging well the Varnish cache when the clear cache button is pressed.

  However, when I run a ban manually from command line:

  echo "ban req.http.host ~ $URL" | varnishadm -S /etc/varnish/secret


 it seems to work clearing all content for that domain in the varnish cache.

 This is what I get when I purge the varnish cache from the plugin:


varnishlog -g request -q 'ReqMethod eq "PURGE"'

-   Begin          req 43813385 rxreq
-   Timestamp      Start: 1502221040.168021 0.000000 0.000000
-   Timestamp      Req: 1502221040.168021 0.000000 0.000000
-   ReqStart       178.33.117.62 34248
-   ReqMethod      PURGE
-   ReqURL         /.*
-   ReqProtocol    HTTP/1.1
-   ReqHeader      host: www.mydomain.com
-   ReqHeader      User-Agent: WordPress/4.8; https://www.mydomain.com
-   ReqHeader      Accept: */*
-   ReqHeader      Accept-Encoding: deflate, gzip
-   ReqHeader      Referer: http://www.mydomain.com/.*
-   ReqHeader      X-Purge-Method: regex
-   ReqHeader      Connection: close
-   ReqHeader      X-Forwarded-For: 178.33.117.62
-   VCL_call       RECV
-   ReqURL         /.*
-   ReqURL         /.*
-   ReqURL         /.*
-   ReqHeader      X-Port: 80
-   ReqUnset       X-Forwarded-For: 178.33.117.62
-   ReqHeader      X-Forwarded-For: 178.33.117.62
-   ReqUnset       host: www.mydomain.com
-   ReqHeader      Host: www.mydomain.com
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   VCL_acl        MATCH purge "178.33.117.62"
-   VCL_return     purge
-   ReqUnset       Accept-Encoding: deflate, gzip
-   ReqHeader      Accept-Encoding: gzip
-   VCL_call       HASH
-   VCL_return     lookup
-   VCL_call       PURGE
-   VCL_return     synth
-   Timestamp      Process: 1502221040.168079 0.000058 0.000058
-   RespHeader     Date: Tue, 08 Aug 2017 19:37:20 GMT
-   RespHeader     Server: Varnish
-   RespHeader     X-Varnish: 43813386
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     OK
-   RespReason     Purged
-   VCL_call       SYNTH
-   RespHeader     Content-Type: text/html; charset=utf-8
-   RespHeader     Retry-After: 5
-   VCL_return     deliver
-   RespHeader     Content-Length: 243
-   Storage        malloc Transient
-   RespHeader     Accept-Ranges: bytes
-   Debug          "RES_MODE 2"
-   RespHeader     Connection: close
-   Timestamp      Resp: 1502221040.168152 0.000131 0.000072
-   ReqAcct        223 0 223 216 243 459
-   End



------ manual ban ---------------

















From dridi at varni.sh  Wed Aug  9 09:54:30 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 9 Aug 2017 11:54:30 +0200
Subject: troubleshooting Varnish purges vs bans
In-Reply-To: <68e12477-a8f5-fb8e-d262-0804c71cc78d@yahoo.es>
References: <68e12477-a8f5-fb8e-d262-0804c71cc78d@yahoo.es>
Message-ID: <CABoVN9D5c6sZ=mvwWFnXtMgmg50FWViLBydeiZGeaXN8O8QRDQ@mail.gmail.com>

> -   ReqMethod      PURGE
> -   ReqURL         /.*

Hello Miguel,

A purge expects an exact match of an object hash, it doesn't work on
criteria like bans do. In order for a purge to succeed, you usually
need the exact Host header and URL in your purge request unless you
changed the default hash.

Cheers,
Dridi



From miguel_3_gonzalez at yahoo.es  Wed Aug  9 11:00:26 2017
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Wed, 9 Aug 2017 13:00:26 +0200
Subject: troubleshooting Varnish purges vs bans
In-Reply-To: <CABoVN9D5c6sZ=mvwWFnXtMgmg50FWViLBydeiZGeaXN8O8QRDQ@mail.gmail.com>
References: <68e12477-a8f5-fb8e-d262-0804c71cc78d@yahoo.es>
 <CABoVN9D5c6sZ=mvwWFnXtMgmg50FWViLBydeiZGeaXN8O8QRDQ@mail.gmail.com>
Message-ID: <c943402e-2747-cf7b-2ddd-98f24bcc03fe@yahoo.es>

On 08/09/17 11:54 AM, Dridi Boukelmoune wrote:
>> -   ReqMethod      PURGE
>> -   ReqURL         /.*
> 
> Hello Miguel,
> 
> A purge expects an exact match of an object hash, it doesn't work on
> criteria like bans do. In order for a purge to succeed, you usually
> need the exact Host header and URL in your purge request unless you
> changed the default hash.

This is not supposed to purge all objects for the whole site?

Regards,

Miguel



From dridi at varni.sh  Wed Aug  9 11:40:14 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 9 Aug 2017 13:40:14 +0200
Subject: troubleshooting Varnish purges vs bans
In-Reply-To: <c943402e-2747-cf7b-2ddd-98f24bcc03fe@yahoo.es>
References: <68e12477-a8f5-fb8e-d262-0804c71cc78d@yahoo.es>
 <CABoVN9D5c6sZ=mvwWFnXtMgmg50FWViLBydeiZGeaXN8O8QRDQ@mail.gmail.com>
 <c943402e-2747-cf7b-2ddd-98f24bcc03fe@yahoo.es>
Message-ID: <CABoVN9B48es79AaxxuisCkbFhRbqttErbUJTEt_e2TsOCUVeVQ@mail.gmail.com>

On Wed, Aug 9, 2017 at 1:00 PM, Miguel Gonz?lez
<miguel_3_gonzalez at yahoo.es> wrote:
> On 08/09/17 11:54 AM, Dridi Boukelmoune wrote:
>>> -   ReqMethod      PURGE
>>> -   ReqURL         /.*
>>
>> Hello Miguel,
>>
>> A purge expects an exact match of an object hash, it doesn't work on
>> criteria like bans do. In order for a purge to succeed, you usually
>> need the exact Host header and URL in your purge request unless you
>> changed the default hash.
>
> This is not supposed to purge all objects for the whole site?

This would only purge "www.mydomain.com/.*" interpreted as a string,
not a regular expression.

Dridi



From admin at beckspaced.com  Wed Aug  9 14:46:42 2017
From: admin at beckspaced.com (Admin Beckspaced)
Date: Wed, 9 Aug 2017 16:46:42 +0200
Subject: opensuse 42.2 systemd varnishlog.service fails to start after system
 reboot
Message-ID: <c5752e5d-8ed9-6885-d4fa-c9ee2fbf246a@beckspaced.com>

Hello Varnish Community,

I'm running an openSUSE 42.2 server with varnish 5.1.2

My problem is that varnishlog.service always fails starting after a 
system reboot.

cx40:~ # systemctl status varnishlog.service
? varnishlog.service - Varnish log generator
    Loaded: loaded (/etc/systemd/system/varnishlog.service; enabled; 
vendor preset: disabled)
    Active: failed (Result: exit-code) since Wed 2017-08-09 16:30:47 
CEST; 1min 7s ago
   Process: 1162 ExecStart=/usr/sbin/varnishncsa $VARNISHLOG_PARAMS -P 
/var/run/varnishlog.pid (code=exited, status=1/FAILURE)
  Main PID: 1162 (code=exited, status=1/FAILURE)

Aug 09 16:30:42 cx40 systemd[1]: Started Varnish log generator.
Aug 09 16:30:42 cx40 varnishncsa[1162]: Cannot open log - retrying for 5 
seconds
Aug 09 16:30:47 cx40 varnishncsa[1162]: Abandoned VSM file (Varnish not 
running?) /var/cache/varnish/cx40/_.vsm
Aug 09 16:30:47 cx40 systemd[1]: varnishlog.service: Main process 
exited, code=exited, status=1/FAILURE
Aug 09 16:30:47 cx40 systemd[1]: varnishlog.service: Unit entered failed 
state.
Aug 09 16:30:47 cx40 systemd[1]: varnishlog.service: Failed with result 
'exit-code'.

If I look at the varnish.service it is reported running

cx40:~ # systemctl status varnish.service
? varnish.service - Varnish HTTP accelerator/reverse proxy
    Loaded: loaded (/usr/lib/systemd/system/varnish.service; enabled; 
vendor preset: disabled)
    Active: active (running) since Wed 2017-08-09 16:30:48 CEST; 1min 
41s ago
  Main PID: 1579 (varnishd)
     Tasks: 218 (limit: 512)
    CGroup: /system.slice/varnish.service
            ??1579 /usr/sbin/varnishd -j unix,user=varnish -f 
/etc/varnish/main.vcl -a :80 -T localhost:6082 -s default=malloc,2G -s 
static=file,/var/cache/varnish,5G -P /var/run/varnishd.pid -F
            ??1672 /usr/sbin/varnishd -j unix,user=varnish -f 
/etc/varnish/main.vcl -a :80 -T localhost:6082 -s default=malloc,2G -s 
static=file,/var/cache/varnish,5G -P /var/run/varnishd.pid -F

Aug 09 16:30:48 cx40 systemd[1]: Started Varnish HTTP 
accelerator/reverse proxy.
Aug 09 16:30:49 cx40 varnishd[1579]: Platform: 
Linux,4.4.79-18.23-default,x86_64,-junix,-smalloc,-sfile,-smalloc,-hcritbit
Aug 09 16:30:49 cx40 varnishd[1579]: Debug: Platform: 
Linux,4.4.79-18.23-default,x86_64,-junix,-smalloc,-sfile,-smalloc,-hcritbit
Aug 09 16:30:49 cx40 varnishd[1579]: Child (1672) Started
Aug 09 16:30:49 cx40 varnishd[1579]: Debug: Child (1672) Started
Aug 09 16:30:49 cx40 varnishd[1579]: Child (1672) said Child starts
Aug 09 16:30:49 cx40 varnishd[1579]: Child (1672) said SMF.static 
mmap'ed 5368709120 bytes of 5368709120
Aug 09 16:30:49 cx40 varnishd[1579]: Info: Child (1672) said Child starts
Aug 09 16:30:49 cx40 varnishd[1579]: Info: Child (1672) said SMF.static 
mmap'ed 5368709120 bytes of 5368709120

If I start the varnishlog.service manually there's no problem at all

cx40:~ # systemctl start varnishlog.service
cx40:~ # systemctl status varnishlog.service
? varnishlog.service - Varnish log generator
    Loaded: loaded (/etc/systemd/system/varnishlog.service; enabled; 
vendor preset: disabled)
    Active: active (running) since Wed 2017-08-09 16:32:53 CEST; 4s ago
  Main PID: 3439 (varnishncsa)
     Tasks: 1 (limit: 512)
    CGroup: /system.slice/varnishlog.service
            ??3439 /usr/sbin/varnishncsa -f 
/etc/varnish/varnishncsa-log-format-string -a -w 
/var/log/varnish/varnish.log -P /var/run/varnishlog.pid

Aug 09 16:32:53 cx40 systemd[1]: Started Varnish log generator.

For me it looks like that varnishlog.service is waiting for 
varnish.service to start and the varnish.service seems to take a bit 
long to start up?
Is there a systemd service option to wait for a few seconds before 
starting varnishlog.service

something like a delayed start of the service?

I also override the default varnishlog.service in 
/usr/lib/systemd/system/varnishlog.service

with my own version in /etc/systemd/system/varnishlog.service

I do this because I need the [service] option WorkingDirectory

/etc/systemd/system/varnishlog.service

[Unit]
Description=Varnish log generator
Requires=varnish.service
#After= is not required

[Service]
WorkingDirectory=/var/log/varnish
EnvironmentFile=/etc/sysconfig/varnish
PIDFile=/var/run/varnishlog.pid
ExecStart=/usr/sbin/varnishncsa $VARNISHLOG_PARAMS -P 
/var/run/varnishlog.pid

[Install]
WantedBy=multi-user.target

The $VARNISHLOG_PARAMS are defined in /etc/sysconfig/varnish

VARNISHLOG_PARAMS="-f /etc/varnish/varnishncsa-log-format-string -a -w 
/var/log/varnish/varnish.log"

and the format string is defined in 
/etc/varnish/varnishncsa-log-format-string

%{VCL_Log:myhost}x %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"

All is working fine, except varnishlog.service won't properly start 
after a system reboot ;)

Any help is more than welcome ...

Thanks, greetings & best wishes
Becki




From dridi at varni.sh  Wed Aug  9 15:36:33 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 9 Aug 2017 17:36:33 +0200
Subject: opensuse 42.2 systemd varnishlog.service fails to start after
 system reboot
In-Reply-To: <c5752e5d-8ed9-6885-d4fa-c9ee2fbf246a@beckspaced.com>
References: <c5752e5d-8ed9-6885-d4fa-c9ee2fbf246a@beckspaced.com>
Message-ID: <CABoVN9CD5E9vdUFZ2KxCUzLMhE0cQNqYSDXKRZ9xEy0Tvkc_aQ@mail.gmail.com>

On Wed, Aug 9, 2017 at 4:46 PM, Admin Beckspaced <admin at beckspaced.com> wrote:
> Hello Varnish Community,
>
> I'm running an openSUSE 42.2 server with varnish 5.1.2

Update to 5.1.3, your Varnish instance can be DoS'd remotely!

> For me it looks like that varnishlog.service is waiting for varnish.service
> to start and the varnish.service seems to take a bit long to start up?

Correct.

> Is there a systemd service option to wait for a few seconds before starting
> varnishlog.service
> something like a delayed start of the service?

Not a systemd option, but you can use the `-t` option in varnishncsa
or varnishlog to increase the timeout (defaults to 5s).

> VARNISHLOG_PARAMS="-f /etc/varnish/varnishncsa-log-format-string -a -w
> /var/log/varnish/varnish.log"

This is where you want to add a timeout option. See man varnishncsa.

Cheers,
Dridi



From admin at beckspaced.com  Wed Aug  9 15:49:27 2017
From: admin at beckspaced.com (Admin Beckspaced)
Date: Wed, 9 Aug 2017 17:49:27 +0200
Subject: opensuse 42.2 systemd varnishlog.service fails to start after
 system reboot
In-Reply-To: <CABoVN9CD5E9vdUFZ2KxCUzLMhE0cQNqYSDXKRZ9xEy0Tvkc_aQ@mail.gmail.com>
References: <c5752e5d-8ed9-6885-d4fa-c9ee2fbf246a@beckspaced.com>
 <CABoVN9CD5E9vdUFZ2KxCUzLMhE0cQNqYSDXKRZ9xEy0Tvkc_aQ@mail.gmail.com>
Message-ID: <f8470e00-b1f3-8d20-2c63-aab760f02a04@beckspaced.com>

Hey Dridi,

thanks a lot for your input. Highly appreciated ;)

Will test out later and see if things work out ok.

Greetings
Becki


On 09.08.2017 17:36, Dridi Boukelmoune wrote:
> On Wed, Aug 9, 2017 at 4:46 PM, Admin Beckspaced <admin at beckspaced.com> wrote:
>> Hello Varnish Community,
>>
>> I'm running an openSUSE 42.2 server with varnish 5.1.2
> Update to 5.1.3, your Varnish instance can be DoS'd remotely!
>
>> For me it looks like that varnishlog.service is waiting for varnish.service
>> to start and the varnish.service seems to take a bit long to start up?
> Correct.
>
>> Is there a systemd service option to wait for a few seconds before starting
>> varnishlog.service
>> something like a delayed start of the service?
> Not a systemd option, but you can use the `-t` option in varnishncsa
> or varnishlog to increase the timeout (defaults to 5s).
>
>> VARNISHLOG_PARAMS="-f /etc/varnish/varnishncsa-log-format-string -a -w
>> /var/log/varnish/varnish.log"
> This is where you want to add a timeout option. See man varnishncsa.
>
> Cheers,
> Dridi
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>




From miguel_3_gonzalez at yahoo.es  Wed Aug  9 15:50:25 2017
From: miguel_3_gonzalez at yahoo.es (Miguel Gonzalez)
Date: Wed, 09 Aug 2017 17:50:25 +0200
Subject: troubleshooting Varnish purges vs bans
Message-ID: <3ivo1gyhfhrr15e2fhvvl608.1502293825989@email.android.com>

And what would be the right way to purge all the objects in the url? Or it can only be done with ban?

Regards,

Miguel

Dridi Boukelmoune <dridi at varni.sh> wrote:

>On Wed, Aug 9, 2017 at 1:00 PM, Miguel Gonz?lez
><miguel_3_gonzalez at yahoo.es> wrote:
>> On 08/09/17 11:54 AM, Dridi Boukelmoune wrote:
>>>> -   ReqMethod      PURGE
>>>> -   ReqURL         /.*
>>>
>>> Hello Miguel,
>>>
>>> A purge expects an exact match of an object hash, it doesn't work on
>>> criteria like bans do. In order for a purge to succeed, you usually
>>> need the exact Host header and URL in your purge request unless you
>>> changed the default hash.
>>
>> This is not supposed to purge all objects for the whole site?
>
>This would only purge "www.mydomain.com/.*" interpreted as a string,
>not a regular expression.
>
>Dridi
>
>_______________________________________________
>varnish-misc mailing list
>varnish-misc at varnish-cache.org
>https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

From Matthew.Pennington at colorado.edu  Thu Aug 10 04:52:25 2017
From: Matthew.Pennington at colorado.edu (Matthew Pennington)
Date: Wed, 9 Aug 2017 22:52:25 -0600
Subject: Varnish child failing due to new VMOD
Message-ID: <CAJhXkbjMf53+U+ezCvL6Yn0st19zOp6nHqxkknou3vHv1BMwzw@mail.gmail.com>

Hi All,

I have varnish 4 (4.1.8 as of today) installed on a RHEL 7 machine.  I have
seen a problem a few days after the past two updates that is causing
varnish to crash and not restart properly.  It looks like a child process
is failing but the manager is not trying to restart it.  I've included the
varnish system logs from the past 2 months.  The error in question is
"Loading VMOD std from /usr/lib64/varnish/vmods/libvmod_std.so: This is no
longer the same file seen by the VCL-compiler."  Does anyone have any ideas
what might be causing this or how to prevent it?

Thanks,

Matt


Jul 09 20:46:46 {servername} varnishd[35916]: Child (31586) not responding
to CLI, killed it.
Jul 09 20:46:46 {servername} varnishd[35916]: Unexpected reply from ping:
400 CLI communication error (hdr)
Jul 09 20:46:46 {servername} varnishd[35916]: Child (31586) died signal=11
Jul 09 20:46:46 {servername} varnishd[35916]: Child cleanup complete
Jul 09 20:46:46 {servername} varnishd[35916]: Child (7499) Started
Jul 09 20:46:46 {servername} varnishd[35916]: Child (7499) Pushing vcls
failed:
                                                              VCL "boot"
Failed initialization
                                                              Message:

Loading VMOD std from /usr/lib64/varnish/vmods/libvmod_std.so:
                                                              This is no
longer the same file seen by the VCL-compiler.
Jul 09 20:46:46 {servername} varnishd[35916]: Stopping Child
Jul 09 20:46:47 {servername} varnishd[35916]: Child (7499) ended
Jul 09 20:46:47 {servername} varnishd[35916]: Child (7499) said Child starts
Jul 09 20:46:47 {servername} varnishd[35916]: Child (7499) said Child dies
Jul 09 20:46:47 {servername} varnishd[35916]: Child cleanup complete
Jul 10 10:18:53 {servername} varnishd[35916]: Manager got SIGINT
Jul 10 10:18:53 {servername} systemd[1]: Stopping Varnish Cache, a
high-performance HTTP accelerator...
Jul 10 10:18:53 {servername} systemd[1]: Starting Varnish Cache, a
high-performance HTTP accelerator...
Jul 10 10:18:54 {servername} varnishd[20573]: Platform:
Linux,3.10.0-327.28.3.el7.x86_64,x86_64,-junix,-smalloc,-smal
Jul 10 10:18:54 {servername} systemd[1]: Started Varnish Cache, a
high-performance HTTP accelerator.
Jul 10 10:18:54 {servername} varnishd[20573]: Child (20574) Started
Jul 10 10:18:54 {servername} varnishd[20573]: Child (20574) said Child
starts
Jul 18 17:15:28 {servername} varnishd[20573]: Child (20574) died signal=11
Jul 18 17:15:28 {servername} varnishd[20573]: Child cleanup complete
Jul 18 17:15:28 {servername} varnishd[20573]: Child (14063) Started
Jul 18 17:15:28 {servername} varnishd[20573]: Child (14063) said Child
starts
Aug 09 22:16:04 {servername} varnishd[20573]: Child (14063) died signal=11
Aug 09 22:16:04 {servername} varnishd[20573]: Child cleanup complete
Aug 09 22:16:04 {servername} varnishd[20573]: Child (32837) Started
Aug 09 22:16:04 {servername} varnishd[20573]: Child (32837) Pushing vcls
failed:
                                                              VCL "boot"
Failed initialization
                                                              Message:

Loading VMOD std from /usr/lib64/varnish/vmods/libvmod_std.so:
                                                              This is no
longer the same file seen by the VCL-compiler.
Aug 09 22:16:04 {servername} varnishd[20573]: Stopping Child
Aug 09 22:16:05 {servername} varnishd[20573]: Child (32837) ended
Aug 09 22:16:06 {servername} varnishd[20573]: Child (32837) said Child
starts
Aug 09 22:16:06 {servername} varnishd[20573]: Child (32837) said Child dies
Aug 09 22:16:06 {servername} varnishd[20573]: Child cleanup complete
Aug 09 22:27:46 {servername} systemd[1]: Stopping Varnish Cache, a
high-performance HTTP accelerator...
Aug 09 22:27:46 {servername} varnishd[20573]: Manager got SIGINT
Aug 09 22:27:46 {servername} systemd[1]: Starting Varnish Cache, a
high-performance HTTP accelerator...
Aug 09 22:27:47 {servername} varnishd[34284]: Platform:
Linux,3.10.0-327.28.3.el7.x86_64,x86_64,-junix,-smalloc,-smal
Aug 09 22:27:47 {servername} systemd[1]: Started Varnish Cache, a
high-performance HTTP accelerator.
Aug 09 22:27:47 {servername} varnishd[34284]: Child (34285) Started
Aug 09 22:27:47 {servername} varnishd[34284]: Child (34285) said Child
starts
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170809/620447b8/attachment.html>

From admin at beckspaced.com  Thu Aug 10 06:40:50 2017
From: admin at beckspaced.com (Admin Beckspaced)
Date: Thu, 10 Aug 2017 08:40:50 +0200
Subject: opensuse 42.2 systemd varnishlog.service fails to start after
 system reboot
In-Reply-To: <CABoVN9CD5E9vdUFZ2KxCUzLMhE0cQNqYSDXKRZ9xEy0Tvkc_aQ@mail.gmail.com>
References: <c5752e5d-8ed9-6885-d4fa-c9ee2fbf246a@beckspaced.com>
 <CABoVN9CD5E9vdUFZ2KxCUzLMhE0cQNqYSDXKRZ9xEy0Tvkc_aQ@mail.gmail.com>
Message-ID: <eaf8857c-83f4-c38a-6a2d-bc1dc5de43f6@beckspaced.com>


On 09.08.2017 17:36, Dridi Boukelmoune wrote:
> On Wed, Aug 9, 2017 at 4:46 PM, Admin Beckspaced <admin at beckspaced.com> wrote:
>> Hello Varnish Community,
>>
>> I'm running an openSUSE 42.2 server with varnish 5.1.2
> Update to 5.1.3, your Varnish instance can be DoS'd remotely!
>
>> For me it looks like that varnishlog.service is waiting for varnish.service
>> to start and the varnish.service seems to take a bit long to start up?
> Correct.
>
>> Is there a systemd service option to wait for a few seconds before starting
>> varnishlog.service
>> something like a delayed start of the service?
> Not a systemd option, but you can use the `-t` option in varnishncsa
> or varnishlog to increase the timeout (defaults to 5s).
>
>> VARNISHLOG_PARAMS="-f /etc/varnish/varnishncsa-log-format-string -a -w
>> /var/log/varnish/varnish.log"
> This is where you want to add a timeout option. See man varnishncsa.
>
> Cheers,
> Dridi
Just to report back on the startup issue ...

added the timeout option to the startup via '-t 30'
and after a system reboot systemctl status reports

cx40:~ # systemctl status varnishlog.service
? varnishlog.service - Varnish log generator
    Loaded: loaded (/etc/systemd/system/varnishlog.service; enabled; 
vendor preset: disabled)
    Active: active (running) since Thu 2017-08-10 08:35:19 CEST; 1min 
35s ago
  Main PID: 1145 (varnishncsa)
     Tasks: 1 (limit: 512)
    CGroup: /system.slice/varnishlog.service
            ??1145 /usr/sbin/varnishncsa -t 30 -f 
/etc/varnish/varnishncsa-log-format-string -a -w 
/var/log/varnish/varnish.log -P /var/run/varnishlog.pid

Aug 10 08:35:19 cx40 systemd[1]: Started Varnish log generator.
Aug 10 08:35:19 cx40 varnishncsa[1145]: Cannot open log - retrying for 
30 seconds
Aug 10 08:35:28 cx40 varnishncsa[1145]: Log opened

so ... all has worked out fine ;)

Thanks & greetings
Becki



From dridi at varni.sh  Thu Aug 10 11:12:25 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 10 Aug 2017 13:12:25 +0200
Subject: troubleshooting Varnish purges vs bans
In-Reply-To: <3ivo1gyhfhrr15e2fhvvl608.1502293825989@email.android.com>
References: <3ivo1gyhfhrr15e2fhvvl608.1502293825989@email.android.com>
Message-ID: <CABoVN9A1swB5qe-fp+ndVWOSk0VmC0mKHt15LpTSU5h_q+Nyvg@mail.gmail.com>

On Wed, Aug 9, 2017 at 5:50 PM, Miguel Gonzalez
<miguel_3_gonzalez at yahoo.es> wrote:
> And what would be the right way to purge all the objects in the url? Or it can only be done with ban?

I'm not sure what you mean by "all the objects in the url", can you
please give me an example involving several objects?

Dridi



From miguel_3_gonzalez at yahoo.es  Thu Aug 10 17:15:10 2017
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Thu, 10 Aug 2017 19:15:10 +0200
Subject: troubleshooting Varnish purges vs bans
In-Reply-To: <CABoVN9A1swB5qe-fp+ndVWOSk0VmC0mKHt15LpTSU5h_q+Nyvg@mail.gmail.com>
References: <3ivo1gyhfhrr15e2fhvvl608.1502293825989@email.android.com>
 <CABoVN9A1swB5qe-fp+ndVWOSk0VmC0mKHt15LpTSU5h_q+Nyvg@mail.gmail.com>
Message-ID: <a3b23829-8bae-7dfa-f0b3-b04603f0bafd@yahoo.es>

On 08/10/17 1:12 PM, Dridi Boukelmoune wrote:
> On Wed, Aug 9, 2017 at 5:50 PM, Miguel Gonzalez
> <miguel_3_gonzalez at yahoo.es> wrote:
>> And what would be the right way to purge all the objects in the url? Or it can only be done with ban?
> 
> I'm not sure what you mean by "all the objects in the url", can you
> please give me an example involving several objects?
> 
> Dridi
> 

Hi,

 The Wordpress plugin is supposed to clean the whole cache for a website
when clicking on a button saying "Clear cache".

 You say from the varnishlog excerpt I sent that?s not the correct way
of doing so, or apparently what is logged by varnish is not a correct purge.

 So how do I tell the developer (or do it myself) to correct the code so
the whole cache for a given website is purged?.

 Regards,

 Miguel



From dridi at varni.sh  Fri Aug 11 10:22:19 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 11 Aug 2017 12:22:19 +0200
Subject: troubleshooting Varnish purges vs bans
In-Reply-To: <a3b23829-8bae-7dfa-f0b3-b04603f0bafd@yahoo.es>
References: <3ivo1gyhfhrr15e2fhvvl608.1502293825989@email.android.com>
 <CABoVN9A1swB5qe-fp+ndVWOSk0VmC0mKHt15LpTSU5h_q+Nyvg@mail.gmail.com>
 <a3b23829-8bae-7dfa-f0b3-b04603f0bafd@yahoo.es>
Message-ID: <CABoVN9DzSKQJRjaSsXiAh+W+Fr3RoWFyMUqEbiKsx-ZhDoAsag@mail.gmail.com>

>  The Wordpress plugin is supposed to clean the whole cache for a website
> when clicking on a button saying "Clear cache".
>
>  You say from the varnishlog excerpt I sent that?s not the correct way
> of doing so, or apparently what is logged by varnish is not a correct purge.
>
>  So how do I tell the developer (or do it myself) to correct the code so
> the whole cache for a given website is purged?.

Use the ban() function in your VCL in this case.

Dridi



From admin at beckspaced.com  Tue Aug 15 09:39:09 2017
From: admin at beckspaced.com (Admin Beckspaced)
Date: Tue, 15 Aug 2017 11:39:09 +0200
Subject: varnishlog client IP problem via Apache SSL reverse proxy
Message-ID: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>

Hello there ;)

I'm running varnish in front of my apache on port 80 without any issues 
so far.

Recently I decided to also use varnish for SSL connections

To do so I first do a http to https redirect within varnish VCL

if ( req.http.X-Forwarded-Proto !~ "(?i)https" ) {
     return (synth(750, ""));
}

then in vcl_synth()

sub vcl_synth {

             if (resp.status == 750) {
                 set resp.status = 301;
                 set resp.http.Location = "https://" + req.http.host + 
req.url;
                 return(deliver);
             }
}

This works fine and all http got redirected to https

Then on port 443 I got apache listening as a reverse proxy with the 
following config:

<VirtualHost *:443>

     ServerName somedomain.com
     ServerAlias *.somedomain.org

     SSLEngine on

     ... ssl cert stuff here ...

     ProxyPreserveHost On
     ProxyPass / http://127.0.0.1:80/
     ProxyPassReverse / http://127.0.0.1:80/
     RequestHeader set X-Forwarded-Port "443"
     RequestHeader set X-Forwarded-Proto "https"

</VirtualHost>

Also this works perfectly fine! Apache does the SSL termination and then 
reverse proxies everything back to varnish on port 80

If I have a look in the apache ssl log:

[15/Aug/2017:02:03:41 +0200] 35.190.201.122 TLSv1.2 
ECDHE-RSA-AES128-GCM-SHA256 "GET /feed/ HTTP/1.1" - 
"http://domain.org/feed/" "Go-http-client/1.1"
[15/Aug/2017:02:03:41 +0200] 35.190.201.122 TLSv1.2 
ECDHE-RSA-AES128-GCM-SHA256 "GET /feed HTTP/1.1" 10513 
"https://domain.org/feed/" "Go-http-client/1.1"

If I look in the varnishlog I see the following:

domain.org 35.190.201.122 - - [15/Aug/2017:02:03:41 +0200] "GET 
http://domain.org/feed/ HTTP/1.1" 301 0 "-" "Go-http-client/1.1"
domain.org 127.0.0.1 - - [15/Aug/2017:02:03:41 +0200] "GET 
http://domain.org/feed/ HTTP/1.1" 301 0 "http://domain.org/feed/" 
"Go-http-client/1.1"
domain.org 127.0.0.1 - - [15/Aug/2017:02:03:41 +0200] "GET 
http://domain.org/feed HTTP/1.1" 200 10513 "https://domain.org/feed/" 
"Go-http-client/1.1"

But in the process of Varnish -> Redirect http to https -> Apache 
Reverse Proxy -> Varnish I loose the client IP address in varnishlog
It jsut says 127.0.0.1

How can I forward the client IP to varnishlog in this process?

I need to have the client IP in varnishlog as I use those to generate 
statistics about the website.

any help, hints or insights would be awesome ;)

Thanks & greetings
Becki


-- 
Beckspaced - Server Administration
------------------------------------------------
Ralf Flederer
Marienplatz 9
97353 Wiesentheid
Tel.: 09383-9033825
Mobil: 01577-7258912
Internet: www.beckspaced.com
------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170815/3b2bee16/attachment.html>

From miguel_3_gonzalez at yahoo.es  Tue Aug 15 11:07:46 2017
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Tue, 15 Aug 2017 13:07:46 +0200
Subject: troubleshooting Varnish purges vs bans
In-Reply-To: <CABoVN9DzSKQJRjaSsXiAh+W+Fr3RoWFyMUqEbiKsx-ZhDoAsag@mail.gmail.com>
References: <3ivo1gyhfhrr15e2fhvvl608.1502293825989@email.android.com>
 <CABoVN9A1swB5qe-fp+ndVWOSk0VmC0mKHt15LpTSU5h_q+Nyvg@mail.gmail.com>
 <a3b23829-8bae-7dfa-f0b3-b04603f0bafd@yahoo.es>
 <CABoVN9DzSKQJRjaSsXiAh+W+Fr3RoWFyMUqEbiKsx-ZhDoAsag@mail.gmail.com>
Message-ID: <36d2b0ba-fd61-a326-e9d1-2528c05bcd59@yahoo.es>

On 08/11/17 12:22 PM, Dridi Boukelmoune wrote:
>>  The Wordpress plugin is supposed to clean the whole cache for a website
>> when clicking on a button saying "Clear cache".
>>
>>  You say from the varnishlog excerpt I sent that?s not the correct way
>> of doing so, or apparently what is logged by varnish is not a correct purge.
>>
>>  So how do I tell the developer (or do it myself) to correct the code so
>> the whole cache for a given website is purged?.
> 
> Use the ban() function in your VCL in this case.
> 
> Dridi
> 

I currently have in default.vcl this:

        # Allow purging from ACL
        if (req.method == "PURGE") {
                # If not allowed then a error 405 is returned
                if (!client.ip ~ purge) {
                        return(synth(405, "This IP is not allowed to
send PURGE requests."));
                }
                # If allowed, do a cache_lookup -> vlc_hit() or vlc_miss()
                return (purge);
        }


What do I have to change to use BAN? If I run ban at command line they
seem to work.

Do I need to change the Wordpress plugin to send BAN requests?

Thanks!

Miguel



From guillaume at varnish-software.com  Tue Aug 15 11:57:19 2017
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Tue, 15 Aug 2017 13:57:19 +0200
Subject: varnishlog client IP problem via Apache SSL reverse proxy
In-Reply-To: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>
References: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>
Message-ID: <CAJ6ZYQxRe9RTvxh44fr9myzHTy_VMGT5qV4CZKF_Jy4ANzzXtQ@mail.gmail.com>

So, if I understood, that's Apache in front of Varnish, not the other way
around. But let's not get lost on semantics

Varnish fanboy version: drop Apache, and use Hitch (http://hitch-tls.org/)
to handle SSL/TLS. That only works if Apache is only there for SSL
termination and not for other tasks not performed by Varnish (if so, which
ones?)
With it you can just test the server port used (std.port(server.ip) == 443)
to determine you are using https.

Apache clean version: use the PROXY protocol. Not sure Apache is able to do
it, haven't looked. That way, apache can behave like hitch and you don't
lose the client.ip info.

Apache dirty version: used the x-forwarded-for header to tell varnish who
sent the request, then use vmod_std to convert that string to an ip. Ugly,
but works.



-- 
Guillaume Quintard

On Tue, Aug 15, 2017 at 11:39 AM, Admin Beckspaced <admin at beckspaced.com>
wrote:

> Hello there ;)
>
> I'm running varnish in front of my apache on port 80 without any issues so
> far.
>
> Recently I decided to also use varnish for SSL connections
>
> To do so I first do a http to https redirect within varnish VCL
>
> if ( req.http.X-Forwarded-Proto !~ "(?i)https" ) {
>     return (synth(750, ""));
> }
>
> then in vcl_synth()
>
> sub vcl_synth {
>
>             if (resp.status == 750) {
>                 set resp.status = 301;
>                 set resp.http.Location = "https://" + req.http.host +
> req.url;
>                 return(deliver);
>             }
> }
>
> This works fine and all http got redirected to https
>
> Then on port 443 I got apache listening as a reverse proxy with the
> following config:
>
> <VirtualHost *:443>
>
>     ServerName somedomain.com
>     ServerAlias *.somedomain.org
>
>     SSLEngine on
>
>     ... ssl cert stuff here ...
>     ProxyPreserveHost On
>     ProxyPass / http://127.0.0.1:80/
>     ProxyPassReverse / http://127.0.0.1:80/
>     RequestHeader set X-Forwarded-Port "443"
>     RequestHeader set X-Forwarded-Proto "https"
>
> </VirtualHost>
>
> Also this works perfectly fine! Apache does the SSL termination and then
> reverse proxies everything back to varnish on port 80
>
> If I have a look in the apache ssl log:
>
> [15/Aug/2017:02:03:41 +0200] 35.190.201.122 TLSv1.2
> ECDHE-RSA-AES128-GCM-SHA256 "GET /feed/ HTTP/1.1" -
> "http://domain.org/feed/" <http://domain.org/feed/> "Go-http-client/1.1"
> [15/Aug/2017:02:03:41 +0200] 35.190.201.122 TLSv1.2
> ECDHE-RSA-AES128-GCM-SHA256 "GET /feed HTTP/1.1" 10513
> "https://domain.org/feed/" <https://domain.org/feed/> "Go-http-client/1.1"
>
> If I look in the varnishlog I see the following:
>
> domain.org 35.190.201.122 - - [15/Aug/2017:02:03:41 +0200] "GET
> http://domain.org/feed/ HTTP/1.1" 301 0 "-" "Go-http-client/1.1"
> domain.org 127.0.0.1 - - [15/Aug/2017:02:03:41 +0200] "GET
> http://domain.org/feed/ HTTP/1.1" 301 0 "http://domain.org/feed/"
> <http://domain.org/feed/> "Go-http-client/1.1"
> domain.org 127.0.0.1 - - [15/Aug/2017:02:03:41 +0200] "GET
> http://domain.org/feed HTTP/1.1" 200 10513 "https://domain.org/feed/"
> <https://domain.org/feed/> "Go-http-client/1.1"
>
> But in the process of Varnish -> Redirect http to https -> Apache Reverse
> Proxy -> Varnish I loose the client IP address in varnishlog
> It jsut says 127.0.0.1
>
> How can I forward the client IP to varnishlog in this process?
>
> I need to have the client IP in varnishlog as I use those to generate
> statistics about the website.
>
> any help, hints or insights would be awesome ;)
>
> Thanks & greetings
> Becki
>
>
> --
> Beckspaced - Server Administration
> ------------------------------------------------
> Ralf Flederer
> Marienplatz 9
> 97353 Wiesentheid
> Tel.: 09383-9033825
> Mobil: 01577-7258912
> Internet: www.beckspaced.com
> ------------------------------------------------
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170815/23da1221/attachment.html>

From jprins at betterbe.com  Tue Aug 15 20:04:23 2017
From: jprins at betterbe.com (Jan Hugo Prins | BetterBe)
Date: Tue, 15 Aug 2017 22:04:23 +0200
Subject: varnishlog client IP problem via Apache SSL reverse proxy
In-Reply-To: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>
References: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>
Message-ID: <d63c476c-4242-8f77-1996-6728ad60d31f@betterbe.com>

I would not do it like that.
Better is to use something like Hitch or HaProxy (my preference) and put
that in front of Varnish.
Then HaProxy / Hitch can terminate all SSL traffic, and HaProxy can also
do your redirect to SSL if needed.
Then in Varnish you use the Apache server as a backend and let it only
serve what it needs to serve.
Use the ProxyProtocol to send the client information from HaProxy to
Vernish.
In Varnish you need to put the client IP into the X-Forwarded-For header.
In Apache you can then use this header to have the real client IP address.

This way you have the real client IP information on all layers.

Jan Hugo Prins



On 08/15/2017 11:39 AM, Admin Beckspaced wrote:
>
> Hello there ;)
>
> I'm running varnish in front of my apache on port 80 without any
> issues so far.
>
> Recently I decided to also use varnish for SSL connections
>
> To do so I first do a http to https redirect within varnish VCL
>
> if ( req.http.X-Forwarded-Proto !~ "(?i)https" ) {
>     return (synth(750, ""));
> }
>
> then in vcl_synth()
>
> sub vcl_synth {
>
>             if (resp.status == 750) {
>                 set resp.status = 301;
>                 set resp.http.Location = "https://" + req.http.host +
> req.url;
>                 return(deliver);
>             }
> }
>
> This works fine and all http got redirected to https
>
> Then on port 443 I got apache listening as a reverse proxy with the
> following config:
>
> <VirtualHost *:443>
>
>     ServerName somedomain.com
>     ServerAlias *.somedomain.org
>
>     SSLEngine on
>
>     ... ssl cert stuff here ...
>
>     ProxyPreserveHost On
>     ProxyPass / http://127.0.0.1:80/
>     ProxyPassReverse / http://127.0.0.1:80/
>     RequestHeader set X-Forwarded-Port "443"
>     RequestHeader set X-Forwarded-Proto "https"
>
> </VirtualHost>
>
> Also this works perfectly fine! Apache does the SSL termination and
> then reverse proxies everything back to varnish on port 80
>
> If I have a look in the apache ssl log:
>
> [15/Aug/2017:02:03:41 +0200] 35.190.201.122 TLSv1.2
> ECDHE-RSA-AES128-GCM-SHA256 "GET /feed/ HTTP/1.1" -
> "http://domain.org/feed/" "Go-http-client/1.1"
> [15/Aug/2017:02:03:41 +0200] 35.190.201.122 TLSv1.2
> ECDHE-RSA-AES128-GCM-SHA256 "GET /feed HTTP/1.1" 10513
> "https://domain.org/feed/" "Go-http-client/1.1"
>
> If I look in the varnishlog I see the following:
>
> domain.org 35.190.201.122 - - [15/Aug/2017:02:03:41 +0200] "GET
> http://domain.org/feed/ HTTP/1.1" 301 0 "-" "Go-http-client/1.1"
> domain.org 127.0.0.1 - - [15/Aug/2017:02:03:41 +0200] "GET
> http://domain.org/feed/ HTTP/1.1" 301 0 "http://domain.org/feed/"
> "Go-http-client/1.1"
> domain.org 127.0.0.1 - - [15/Aug/2017:02:03:41 +0200] "GET
> http://domain.org/feed HTTP/1.1" 200 10513 "https://domain.org/feed/"
> "Go-http-client/1.1"
>
> But in the process of Varnish -> Redirect http to https -> Apache
> Reverse Proxy -> Varnish I loose the client IP address in varnishlog
> It jsut says 127.0.0.1
>
> How can I forward the client IP to varnishlog in this process?
>
> I need to have the client IP in varnishlog as I use those to generate
> statistics about the website.
>
> any help, hints or insights would be awesome ;)
>
> Thanks & greetings
> Becki
>
>
> -- 
> Beckspaced - Server Administration
> ------------------------------------------------
> Ralf Flederer
> Marienplatz 9
> 97353 Wiesentheid
> Tel.: 09383-9033825
> Mobil: 01577-7258912
> Internet: www.beckspaced.com
> ------------------------------------------------
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

-- 
Kind regards

Jan Hugo Prins
/DevOps Engineer/
<https://betterbe.com>
Auke Vleerstraat 140 E
7547 AN Enschede
CC no. 08097527
<https://www.kvk.nl/orderstraat/product-kiezen/?kvknummer=080975270000>
*T* +31 (0) 53 48 00 694 <tel:+31534800694>
*E* jprins at betterbe.com <mailto:jprins at betterbe.com>
*M* +31 (0)6 263 58 951 <tel:+31%20%280%296%20263%2058%20951>
www.betterbe.com <https://www.betterbe.com>
BetterBe accepts no liability for the content of this email, or for the
consequences of any actions taken on the basis
of the information provided, unless that information is subsequently
confirmed in writing. If you are not the intended
recipient you are notified that disclosing, copying, distributing or
taking any action in reliance on the contents of this
information is strictly prohibited.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170815/2ad54f2a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: abjappggihcebokc.png
Type: image/png
Size: 13988 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170815/2ad54f2a/attachment.png>

From lagged at gmail.com  Wed Aug 16 06:19:41 2017
From: lagged at gmail.com (Andrei)
Date: Wed, 16 Aug 2017 01:19:41 -0500
Subject: varnishlog client IP problem via Apache SSL reverse proxy
In-Reply-To: <d63c476c-4242-8f77-1996-6728ad60d31f@betterbe.com>
References: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>
 <d63c476c-4242-8f77-1996-6728ad60d31f@betterbe.com>
Message-ID: <CAP+vvEtSGbwwPT+3v8H1rQX1dMAiFqcaw_JGVAfezUU2ogx01Q@mail.gmail.com>

+1 for SSL with Hitch/HAProxy. The setup described with the Apache
runaround will more than likely tank as soon as large traffic spikes appear

On Tue, Aug 15, 2017 at 3:04 PM, Jan Hugo Prins | BetterBe <
jprins at betterbe.com> wrote:

> I would not do it like that.
> Better is to use something like Hitch or HaProxy (my preference) and put
> that in front of Varnish.
> Then HaProxy / Hitch can terminate all SSL traffic, and HaProxy can also
> do your redirect to SSL if needed.
> Then in Varnish you use the Apache server as a backend and let it only
> serve what it needs to serve.
> Use the ProxyProtocol to send the client information from HaProxy to
> Vernish.
> In Varnish you need to put the client IP into the X-Forwarded-For header.
> In Apache you can then use this header to have the real client IP address.
>
> This way you have the real client IP information on all layers.
>
> Jan Hugo Prins
>
>
>
>
> On 08/15/2017 11:39 AM, Admin Beckspaced wrote:
>
> Hello there ;)
>
> I'm running varnish in front of my apache on port 80 without any issues so
> far.
>
> Recently I decided to also use varnish for SSL connections
>
> To do so I first do a http to https redirect within varnish VCL
>
> if ( req.http.X-Forwarded-Proto !~ "(?i)https" ) {
>     return (synth(750, ""));
> }
>
> then in vcl_synth()
>
> sub vcl_synth {
>
>             if (resp.status == 750) {
>                 set resp.status = 301;
>                 set resp.http.Location = "https://" + req.http.host +
> req.url;
>                 return(deliver);
>             }
> }
>
> This works fine and all http got redirected to https
>
> Then on port 443 I got apache listening as a reverse proxy with the
> following config:
>
> <VirtualHost *:443>
>
>     ServerName somedomain.com
>     ServerAlias *.somedomain.org
>
>     SSLEngine on
>
>     ... ssl cert stuff here ...
>     ProxyPreserveHost On
>     ProxyPass / http://127.0.0.1:80/
>     ProxyPassReverse / http://127.0.0.1:80/
>     RequestHeader set X-Forwarded-Port "443"
>     RequestHeader set X-Forwarded-Proto "https"
>
> </VirtualHost>
>
> Also this works perfectly fine! Apache does the SSL termination and then
> reverse proxies everything back to varnish on port 80
>
> If I have a look in the apache ssl log:
>
> [15/Aug/2017:02:03:41 +0200] 35.190.201.122 TLSv1.2
> ECDHE-RSA-AES128-GCM-SHA256 "GET /feed/ HTTP/1.1" -
> "http://domain.org/feed/" <http://domain.org/feed/> "Go-http-client/1.1"
> [15/Aug/2017:02:03:41 +0200] 35.190.201.122 TLSv1.2
> ECDHE-RSA-AES128-GCM-SHA256 "GET /feed HTTP/1.1" 10513
> "https://domain.org/feed/" <https://domain.org/feed/> "Go-http-client/1.1"
>
> If I look in the varnishlog I see the following:
>
> domain.org 35.190.201.122 - - [15/Aug/2017:02:03:41 +0200] "GET
> http://domain.org/feed/ HTTP/1.1" 301 0 "-" "Go-http-client/1.1"
> domain.org 127.0.0.1 - - [15/Aug/2017:02:03:41 +0200] "GET
> http://domain.org/feed/ HTTP/1.1" 301 0 "http://domain.org/feed/"
> <http://domain.org/feed/> "Go-http-client/1.1"
> domain.org 127.0.0.1 - - [15/Aug/2017:02:03:41 +0200] "GET
> http://domain.org/feed HTTP/1.1" 200 10513 "https://domain.org/feed/"
> <https://domain.org/feed/> "Go-http-client/1.1"
>
> But in the process of Varnish -> Redirect http to https -> Apache Reverse
> Proxy -> Varnish I loose the client IP address in varnishlog
> It jsut says 127.0.0.1
>
> How can I forward the client IP to varnishlog in this process?
>
> I need to have the client IP in varnishlog as I use those to generate
> statistics about the website.
>
> any help, hints or insights would be awesome ;)
>
> Thanks & greetings
> Becki
>
>
> --
> Beckspaced - Server Administration
> ------------------------------------------------
> Ralf Flederer
> Marienplatz 9
> 97353 Wiesentheid
> Tel.: 09383-9033825
> Mobil: 01577-7258912
> Internet: www.beckspaced.com
> ------------------------------------------------
>
>
>
> _______________________________________________
> varnish-misc mailing listvarnish-misc at varnish-cache.orghttps://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
> --
> Kind regards
>
> Jan Hugo Prins
> *DevOps Engineer*
> <https://betterbe.com>
> Auke Vleerstraat 140 E
> 7547 AN Enschede
> CC no. 08097527
> <https://www.kvk.nl/orderstraat/product-kiezen/?kvknummer=080975270000>
> *T* +31 (0) 53 48 00 694 <+31534800694>
> *E* jprins at betterbe.com
> *M* +31 (0)6 263 58 951 <+31%20%280%296%20263%2058%20951> www.betterbe.com
> BetterBe accepts no liability for the content of this email, or for the
> consequences of any actions taken on the basis
> of the information provided, unless that information is subsequently
> confirmed in writing. If you are not the intended
> recipient you are notified that disclosing, copying, distributing or
> taking any action in reliance on the contents of this
> information is strictly prohibited.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170816/a096aace/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: abjappggihcebokc.png
Type: image/png
Size: 13988 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170816/a096aace/attachment.png>

From admin at beckspaced.com  Wed Aug 16 06:56:10 2017
From: admin at beckspaced.com (Admin Beckspaced)
Date: Wed, 16 Aug 2017 08:56:10 +0200
Subject: varnishlog client IP problem via Apache SSL reverse proxy
In-Reply-To: <d63c476c-4242-8f77-1996-6728ad60d31f@betterbe.com>
References: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>
 <d63c476c-4242-8f77-1996-6728ad60d31f@betterbe.com>
Message-ID: <5a696be0-c0a7-5317-a270-db1e7d1031c5@beckspaced.com>

Thanks a lot for your suggestion for using HaProxy ;)

My thinking was just: why install another bit of software when apache is 
able to do the SSL termination.
But like Andrei said, if traffic spikes hit the apache runaround will 
not be the optimal solution.

Do you guys have any recent up-to-date tutorials / howtos on setting up 
HaProxy as SSL terminator in front of varnish.
also doing the SSL redirects ...

Did look around for Hitch but wasn't very pleased with the info provided ;(

Any hints are welcome & thanks for your help & replies ;)

Greetings
Becki



On 15.08.2017 22:04, Jan Hugo Prins | BetterBe wrote:
> I would not do it like that.
> Better is to use something like Hitch or HaProxy (my preference) and 
> put that in front of Varnish.
> Then HaProxy / Hitch can terminate all SSL traffic, and HaProxy can 
> also do your redirect to SSL if needed.
> Then in Varnish you use the Apache server as a backend and let it only 
> serve what it needs to serve.
> Use the ProxyProtocol to send the client information from HaProxy to 
> Vernish.
> In Varnish you need to put the client IP into the X-Forwarded-For header.
> In Apache you can then use this header to have the real client IP address.
>
> This way you have the real client IP information on all layers.
>
> Jan Hugo Prins
>
>




From guillaume at varnish-software.com  Wed Aug 16 07:57:47 2017
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 16 Aug 2017 09:57:47 +0200
Subject: varnishlog client IP problem via Apache SSL reverse proxy
In-Reply-To: <5a696be0-c0a7-5317-a270-db1e7d1031c5@beckspaced.com>
References: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>
 <d63c476c-4242-8f77-1996-6728ad60d31f@betterbe.com>
 <5a696be0-c0a7-5317-a270-db1e7d1031c5@beckspaced.com>
Message-ID: <CAJ6ZYQyfR4i5JGHZvfEtC1r8obtY+-=i=e-52DXO9wiGQVezKw@mail.gmail.com>

At the risk of insisting, hitch is super easy to setup, once installed, you
just need to:
- Edit /etc/hitch/hitch.conf to
  - Set the front-end, usually *:443
  - Set the backend (where to send decrypted traffic), 127.0.0.1:8443
  - Set the pem-file line to point to a certificate
- Add "-a 127.0.0.1:8443,PROXY" to Varnish command.

The Varnish part will be needed anyway if you want to use the proxy
protocol.

The docs here
https://docs.varnish-software.com/varnish-cache-plus/features/client-ssl/
can help you (except that the name of the package differs) but the crux of
it is really what I listed above.

So we can do better next time, what didn't you like about the info you got
about hitch?

-- 
Guillaume Quintard

On Aug 16, 2017 09:29, "Admin Beckspaced" <admin at beckspaced.com> wrote:

> Thanks a lot for your suggestion for using HaProxy ;)
>
> My thinking was just: why install another bit of software when apache is
> able to do the SSL termination.
> But like Andrei said, if traffic spikes hit the apache runaround will not
> be the optimal solution.
>
> Do you guys have any recent up-to-date tutorials / howtos on setting up
> HaProxy as SSL terminator in front of varnish.
> also doing the SSL redirects ...
>
> Did look around for Hitch but wasn't very pleased with the info provided ;(
>
> Any hints are welcome & thanks for your help & replies ;)
>
> Greetings
> Becki
>
>
>
> On 15.08.2017 22:04, Jan Hugo Prins | BetterBe wrote:
>
>> I would not do it like that.
>> Better is to use something like Hitch or HaProxy (my preference) and put
>> that in front of Varnish.
>> Then HaProxy / Hitch can terminate all SSL traffic, and HaProxy can also
>> do your redirect to SSL if needed.
>> Then in Varnish you use the Apache server as a backend and let it only
>> serve what it needs to serve.
>> Use the ProxyProtocol to send the client information from HaProxy to
>> Vernish.
>> In Varnish you need to put the client IP into the X-Forwarded-For header.
>> In Apache you can then use this header to have the real client IP address.
>>
>> This way you have the real client IP information on all layers.
>>
>> Jan Hugo Prins
>>
>>
>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170816/25fd6f19/attachment.html>

From jprins at betterbe.com  Wed Aug 16 09:34:13 2017
From: jprins at betterbe.com (Jan Hugo Prins | BetterBe)
Date: Wed, 16 Aug 2017 11:34:13 +0200
Subject: varnishlog client IP problem via Apache SSL reverse proxy
In-Reply-To: <5a696be0-c0a7-5317-a270-db1e7d1031c5@beckspaced.com>
References: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>
 <d63c476c-4242-8f77-1996-6728ad60d31f@betterbe.com>
 <5a696be0-c0a7-5317-a270-db1e7d1031c5@beckspaced.com>
Message-ID: <d195d90f-c010-a7a3-e3a0-bc6ff0d77099@betterbe.com>

I think the choice between HaProxy and Hitch is probably one where you
have to look at what you need or might need in the future and what you
already know.
My decision to use HaProxy was based on several criteria:
- We already use HaProxy at other locations so I was familiar with the
product and it's configuration.
- Using Hitch would mean that I would need to maintain another part of
software.
- I wanted to be able to make traffic routing decisions before the
request hits Varnish. Send requests to different backend etc.
- I wanted to do some rewrites before the traffic hits Varnish.

Based on those criteria I decided to put HaProxy in front of Varnish.
Your criteria will very likely be different.

Jan Hugo Prins


On 08/16/2017 08:56 AM, Admin Beckspaced wrote:
> Thanks a lot for your suggestion for using HaProxy ;)
>
> My thinking was just: why install another bit of software when apache
> is able to do the SSL termination.
> But like Andrei said, if traffic spikes hit the apache runaround will
> not be the optimal solution.
>
> Do you guys have any recent up-to-date tutorials / howtos on setting
> up HaProxy as SSL terminator in front of varnish.
> also doing the SSL redirects ...
>
> Did look around for Hitch but wasn't very pleased with the info
> provided ;(
>
> Any hints are welcome & thanks for your help & replies ;)
>
> Greetings
> Becki
>
>
>
> On 15.08.2017 22:04, Jan Hugo Prins | BetterBe wrote:
>> I would not do it like that.
>> Better is to use something like Hitch or HaProxy (my preference) and
>> put that in front of Varnish.
>> Then HaProxy / Hitch can terminate all SSL traffic, and HaProxy can
>> also do your redirect to SSL if needed.
>> Then in Varnish you use the Apache server as a backend and let it
>> only serve what it needs to serve.
>> Use the ProxyProtocol to send the client information from HaProxy to
>> Vernish.
>> In Varnish you need to put the client IP into the X-Forwarded-For
>> header.
>> In Apache you can then use this header to have the real client IP
>> address.
>>
>> This way you have the real client IP information on all layers.
>>
>> Jan Hugo Prins
>>
>>
>

-- 
Kind regards

Jan Hugo Prins
/DevOps Engineer/
<https://betterbe.com>
Auke Vleerstraat 140 E
7547 AN Enschede
CC no. 08097527
<https://www.kvk.nl/orderstraat/product-kiezen/?kvknummer=080975270000>
*T* +31 (0) 53 48 00 694 <tel:+31534800694>
*E* jprins at betterbe.com <mailto:jprins at betterbe.com>
*M* +31 (0)6 263 58 951 <tel:+31%20%280%296%20263%2058%20951>
www.betterbe.com <https://www.betterbe.com>
BetterBe accepts no liability for the content of this email, or for the
consequences of any actions taken on the basis
of the information provided, unless that information is subsequently
confirmed in writing. If you are not the intended
recipient you are notified that disclosing, copying, distributing or
taking any action in reliance on the contents of this
information is strictly prohibited.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170816/56cca5da/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kinfkennhjlkhind.png
Type: image/png
Size: 13988 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170816/56cca5da/attachment.png>

From miguel_3_gonzalez at yahoo.es  Wed Aug 16 10:13:57 2017
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Wed, 16 Aug 2017 12:13:57 +0200
Subject: troubleshooting Varnish purges vs bans
In-Reply-To: <36d2b0ba-fd61-a326-e9d1-2528c05bcd59@yahoo.es>
References: <3ivo1gyhfhrr15e2fhvvl608.1502293825989@email.android.com>
 <CABoVN9A1swB5qe-fp+ndVWOSk0VmC0mKHt15LpTSU5h_q+Nyvg@mail.gmail.com>
 <a3b23829-8bae-7dfa-f0b3-b04603f0bafd@yahoo.es>
 <CABoVN9DzSKQJRjaSsXiAh+W+Fr3RoWFyMUqEbiKsx-ZhDoAsag@mail.gmail.com>
 <36d2b0ba-fd61-a326-e9d1-2528c05bcd59@yahoo.es>
Message-ID: <9ed6b032-de6c-93ba-3f41-310feecb5b66@yahoo.es>

I have found out how to make to work varnish http purge plugin with
Varnish 4.x in this article (funny it?s on the Varnish blog)

https://info.varnish-software.com/blog/step-step-speed-wordpress-varnish-software?success=true



I have to say If I added the snippet mentioned:

if (req.http.X-Purge-Method == "regex") {

ban("req.url ~ " + req.url + " && req.http.host ~ " + req.http.host);

return (synth(200, "Banned."));

} else {

return (purge);

}

It didn?t work out of the box. I got the following VCL_error:

- VCL_Error ban(): Expected && between conditions, found "&&"

I had to change it to:

if (req.http.X-Purge-Method == "regex") {

ban("req.url ~ " + req.url + " && req.http.host ~ " + req.http.host);

return (synth(200, "Banned."));

} else {

return (purge);

}

So maybe for other users using Wordpress with Varnish it might be useful

Regards,

Miguel


On 08/15/17 1:07 PM, Miguel Gonz?lez wrote:
> On 08/11/17 12:22 PM, Dridi Boukelmoune wrote:
>>>  The Wordpress plugin is supposed to clean the whole cache for a website
>>> when clicking on a button saying "Clear cache".
>>>
>>>  You say from the varnishlog excerpt I sent that?s not the correct way
>>> of doing so, or apparently what is logged by varnish is not a correct purge.
>>>
>>>  So how do I tell the developer (or do it myself) to correct the code so
>>> the whole cache for a given website is purged?.
>>
>> Use the ban() function in your VCL in this case.
>>
>> Dridi
>>
> 
> I currently have in default.vcl this:
> 
>         # Allow purging from ACL
>         if (req.method == "PURGE") {
>                 # If not allowed then a error 405 is returned
>                 if (!client.ip ~ purge) {
>                         return(synth(405, "This IP is not allowed to
> send PURGE requests."));
>                 }
>                 # If allowed, do a cache_lookup -> vlc_hit() or vlc_miss()
>                 return (purge);
>         }
> 
> 
> What do I have to change to use BAN? If I run ban at command line they
> seem to work.
> 
> Do I need to change the Wordpress plugin to send BAN requests?
> 
> Thanks!
> 
> Miguel
> 




From admin at beckspaced.com  Wed Aug 16 10:30:46 2017
From: admin at beckspaced.com (Admin Beckspaced)
Date: Wed, 16 Aug 2017 12:30:46 +0200
Subject: varnishlog client IP problem via Apache SSL reverse proxy
In-Reply-To: <CAJ6ZYQyfR4i5JGHZvfEtC1r8obtY+-=i=e-52DXO9wiGQVezKw@mail.gmail.com>
References: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>
 <d63c476c-4242-8f77-1996-6728ad60d31f@betterbe.com>
 <5a696be0-c0a7-5317-a270-db1e7d1031c5@beckspaced.com>
 <CAJ6ZYQyfR4i5JGHZvfEtC1r8obtY+-=i=e-52DXO9wiGQVezKw@mail.gmail.com>
Message-ID: <c2a13314-027c-3606-bb07-c3b2824e1b32@beckspaced.com>

Thanks Guillaume,

will then have a look into the info you provided and report back if I 
run into any trouble trying to setup hitch  ;)

What's your recommendation of up-to-date documents on how to setup hitch 
in front of varnish with multiple vhost SSL certificates?

So far I found:

https://github.com/varnish/hitch
https://hitch-tls.org/

Is there any docu elsewhere you can recommend?

Thanks a lot for your support!

Greetings
Becki


On 16.08.2017 09:57, Guillaume Quintard wrote:
> At the risk of insisting, hitch is super easy to setup, once 
> installed, you just need to:
> - Edit /etc/hitch/hitch.conf to
>   - Set the front-end, usually *:443
>   - Set the backend (where to send decrypted traffic), 127.0.0.1:8443 
> <http://127.0.0.1:8443>
>   - Set the pem-file line to point to a certificate
> - Add "-a 127.0.0.1:8443 <http://127.0.0.1:8443>,PROXY" to Varnish 
> command.
>
> The Varnish part will be needed anyway if you want to use the proxy 
> protocol.
>
> The docs here 
> https://docs.varnish-software.com/varnish-cache-plus/features/client-ssl/ 
> can help you (except that the name of the package differs) but the 
> crux of it is really what I listed above.
>
> So we can do better next time, what didn't you like about the info you 
> got about hitch?
>
> -- 
> Guillaume Quintard
>
> On Aug 16, 2017 09:29, "Admin Beckspaced" <admin at beckspaced.com 
> <mailto:admin at beckspaced.com>> wrote:
>
>     Thanks a lot for your suggestion for using HaProxy ;)
>
>     My thinking was just: why install another bit of software when
>     apache is able to do the SSL termination.
>     But like Andrei said, if traffic spikes hit the apache runaround
>     will not be the optimal solution.
>
>     Do you guys have any recent up-to-date tutorials / howtos on
>     setting up HaProxy as SSL terminator in front of varnish.
>     also doing the SSL redirects ...
>
>     Did look around for Hitch but wasn't very pleased with the info
>     provided ;(
>
>     Any hints are welcome & thanks for your help & replies ;)
>
>     Greetings
>     Becki
>
>
>
>     On 15.08.2017 22:04, Jan Hugo Prins | BetterBe wrote:
>
>         I would not do it like that.
>         Better is to use something like Hitch or HaProxy (my
>         preference) and put that in front of Varnish.
>         Then HaProxy / Hitch can terminate all SSL traffic, and
>         HaProxy can also do your redirect to SSL if needed.
>         Then in Varnish you use the Apache server as a backend and let
>         it only serve what it needs to serve.
>         Use the ProxyProtocol to send the client information from
>         HaProxy to Vernish.
>         In Varnish you need to put the client IP into the
>         X-Forwarded-For header.
>         In Apache you can then use this header to have the real client
>         IP address.
>
>         This way you have the real client IP information on all layers.
>
>         Jan Hugo Prins
>
>
>
>
>     _______________________________________________
>     varnish-misc mailing list
>     varnish-misc at varnish-cache.org <mailto:varnish-misc at varnish-cache.org>
>     https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>     <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>




From guillaume at varnish-software.com  Wed Aug 16 10:39:23 2017
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 16 Aug 2017 12:39:23 +0200
Subject: varnishlog client IP problem via Apache SSL reverse proxy
In-Reply-To: <c2a13314-027c-3606-bb07-c3b2824e1b32@beckspaced.com>
References: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>
 <d63c476c-4242-8f77-1996-6728ad60d31f@betterbe.com>
 <5a696be0-c0a7-5317-a270-db1e7d1031c5@beckspaced.com>
 <CAJ6ZYQyfR4i5JGHZvfEtC1r8obtY+-=i=e-52DXO9wiGQVezKw@mail.gmail.com>
 <c2a13314-027c-3606-bb07-c3b2824e1b32@beckspaced.com>
Message-ID: <CAJ6ZYQwQTfAnmkAhLDfYa1P_GtVTwiQw3Jk1WyA18Se24Z02fg@mail.gmail.com>

For multiple certificates, simply put multiple pem-file lines in
hitch.conf, and you're good to go :-)

-- 
Guillaume Quintard

On Aug 16, 2017 12:30, "Admin Beckspaced" <admin at beckspaced.com> wrote:

> Thanks Guillaume,
>
> will then have a look into the info you provided and report back if I run
> into any trouble trying to setup hitch  ;)
>
> What's your recommendation of up-to-date documents on how to setup hitch
> in front of varnish with multiple vhost SSL certificates?
>
> So far I found:
>
> https://github.com/varnish/hitch
> https://hitch-tls.org/
>
> Is there any docu elsewhere you can recommend?
>
> Thanks a lot for your support!
>
> Greetings
> Becki
>
>
> On 16.08.2017 09:57, Guillaume Quintard wrote:
>
>> At the risk of insisting, hitch is super easy to setup, once installed,
>> you just need to:
>> - Edit /etc/hitch/hitch.conf to
>>   - Set the front-end, usually *:443
>>   - Set the backend (where to send decrypted traffic), 127.0.0.1:8443 <
>> http://127.0.0.1:8443>
>>   - Set the pem-file line to point to a certificate
>> - Add "-a 127.0.0.1:8443 <http://127.0.0.1:8443>,PROXY" to Varnish
>> command.
>>
>> The Varnish part will be needed anyway if you want to use the proxy
>> protocol.
>>
>> The docs here https://docs.varnish-software.
>> com/varnish-cache-plus/features/client-ssl/ can help you (except that
>> the name of the package differs) but the crux of it is really what I listed
>> above.
>>
>> So we can do better next time, what didn't you like about the info you
>> got about hitch?
>>
>> --
>> Guillaume Quintard
>>
>> On Aug 16, 2017 09:29, "Admin Beckspaced" <admin at beckspaced.com <mailto:
>> admin at beckspaced.com>> wrote:
>>
>>     Thanks a lot for your suggestion for using HaProxy ;)
>>
>>     My thinking was just: why install another bit of software when
>>     apache is able to do the SSL termination.
>>     But like Andrei said, if traffic spikes hit the apache runaround
>>     will not be the optimal solution.
>>
>>     Do you guys have any recent up-to-date tutorials / howtos on
>>     setting up HaProxy as SSL terminator in front of varnish.
>>     also doing the SSL redirects ...
>>
>>     Did look around for Hitch but wasn't very pleased with the info
>>     provided ;(
>>
>>     Any hints are welcome & thanks for your help & replies ;)
>>
>>     Greetings
>>     Becki
>>
>>
>>
>>     On 15.08.2017 22:04, Jan Hugo Prins | BetterBe wrote:
>>
>>         I would not do it like that.
>>         Better is to use something like Hitch or HaProxy (my
>>         preference) and put that in front of Varnish.
>>         Then HaProxy / Hitch can terminate all SSL traffic, and
>>         HaProxy can also do your redirect to SSL if needed.
>>         Then in Varnish you use the Apache server as a backend and let
>>         it only serve what it needs to serve.
>>         Use the ProxyProtocol to send the client information from
>>         HaProxy to Vernish.
>>         In Varnish you need to put the client IP into the
>>         X-Forwarded-For header.
>>         In Apache you can then use this header to have the real client
>>         IP address.
>>
>>         This way you have the real client IP information on all layers.
>>
>>         Jan Hugo Prins
>>
>>
>>
>>
>>     _______________________________________________
>>     varnish-misc mailing list
>>     varnish-misc at varnish-cache.org <mailto:varnish-misc at varnish-cache.org
>> >
>>     https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>     <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170816/8c9bf430/attachment.html>

From quintinpar at gmail.com  Thu Aug 17 17:16:11 2017
From: quintinpar at gmail.com (Quintin Par)
Date: Thu, 17 Aug 2017 10:16:11 -0700
Subject: A better way to do Varnish hit rate?
Message-ID: <CACW9FdPhf4WXgXeCX+fe2=bgTW3y4Z2=Gy0AQpXzbjmVmUuq1g@mail.gmail.com>

I currently analyze varnish hit rate like this

http://d.pr/i/RswMeL
<https://mailtrack.io/trace/link/2a6a624a67c72c8b5fabc0f3ba4f44c93eb39964?url=http%3A%2F%2Fd.pr%2Fi%2FRswMeL&userId=74734&signature=167bba0ff7e3ed50>

While this is good, a major part of the request the server services is
non-cacheable. This brings down the hit rate drastically.

How can I modify this such that I count only requests that are candidates
for caching and calculate the rate? Which varnishstat metric should I be
looking?

- Quintin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170817/8e214f6d/attachment.html>

From admin at beckspaced.com  Fri Aug 18 07:08:36 2017
From: admin at beckspaced.com (Admin Beckspaced)
Date: Fri, 18 Aug 2017 09:08:36 +0200
Subject: varnishlog client IP problem via Apache SSL reverse proxy
In-Reply-To: <CAJ6ZYQyfR4i5JGHZvfEtC1r8obtY+-=i=e-52DXO9wiGQVezKw@mail.gmail.com>
References: <647da1e5-6b80-a52b-f957-60ed74cf0c95@beckspaced.com>
 <d63c476c-4242-8f77-1996-6728ad60d31f@betterbe.com>
 <5a696be0-c0a7-5317-a270-db1e7d1031c5@beckspaced.com>
 <CAJ6ZYQyfR4i5JGHZvfEtC1r8obtY+-=i=e-52DXO9wiGQVezKw@mail.gmail.com>
Message-ID: <e65213e9-7dd1-5f81-6269-9fd263d662be@beckspaced.com>

Good Morning Everyone ;)

Before doing some work today and installing Hitch as a SSL/TLS 
terminator in front of varnish on my production server ...

I would like to thank you all for your help & suggestions and especially 
@Guillaume for insisting to have a look at Hitch ;)

First, I want to take back my statement about hitch documentation. I 
found all in the docu that is needed to setup and configure hitch.
Perhaps I was a bit surprised about the 'tiny' amount of documentation 
but hey ... hitch is only a 'dumb' TLS proxy!
So there's nothing that much to document after all ;)

Also the redirect from http -> https works like a breeze ... though I 
had to do a bit of std.log("local port: "  + std.port(local.ip))
to understand the difference between remote, local, client, server 
variables ...

Another tiny bit of frustration I encountered via a typo copy & paste 
from github

https://github.com/varnish/hitch/issues/39

It's about setting the X-Forwarded-Proto https header in varnish VCL to 
let my wordpress know that it's running on https

and if you do a copy & paste of

sub vcl_recv {
    if (std.port(local.ip) == 80) {
      return (synth(700));
    } else {
      set req.http.X-Forwared-Proto = "https";
    }
}

and you don't have a close look that the 'd' is missing in 
req.http.X-Forwared-Proto
it should be req.http.X-Forwarded-Proto

you will scratch your head for quite some time and wonder why wordpress 
doesn't know that it is running on SSL ... oh my gosh!

anyway ... all worked out fine and now I got some hitch install to do on 
my production server ;)

Thanks & best wishes
Becki


On 16.08.2017 09:57, Guillaume Quintard wrote:
> At the risk of insisting, hitch is super easy to setup, once 
> installed, you just need to:
> - Edit /etc/hitch/hitch.conf to
>   - Set the front-end, usually *:443
>   - Set the backend (where to send decrypted traffic), 127.0.0.1:8443 
> <http://127.0.0.1:8443>
>   - Set the pem-file line to point to a certificate
> - Add "-a 127.0.0.1:8443 <http://127.0.0.1:8443>,PROXY" to Varnish 
> command.
>
> The Varnish part will be needed anyway if you want to use the proxy 
> protocol.
>
> The docs here 
> https://docs.varnish-software.com/varnish-cache-plus/features/client-ssl/ 
> can help you (except that the name of the package differs) but the 
> crux of it is really what I listed above.
>
> So we can do better next time, what didn't you like about the info you 
> got about hitch?
>
> -- 
> Guillaume Quintard
>
> On Aug 16, 2017 09:29, "Admin Beckspaced" <admin at beckspaced.com 
> <mailto:admin at beckspaced.com>> wrote:
>
>     Thanks a lot for your suggestion for using HaProxy ;)
>
>     My thinking was just: why install another bit of software when
>     apache is able to do the SSL termination.
>     But like Andrei said, if traffic spikes hit the apache runaround
>     will not be the optimal solution.
>
>     Do you guys have any recent up-to-date tutorials / howtos on
>     setting up HaProxy as SSL terminator in front of varnish.
>     also doing the SSL redirects ...
>
>     Did look around for Hitch but wasn't very pleased with the info
>     provided ;(
>
>     Any hints are welcome & thanks for your help & replies ;)
>
>     Greetings
>     Becki
>
>




From admin at beckspaced.com  Fri Aug 18 10:39:13 2017
From: admin at beckspaced.com (Admin Beckspaced)
Date: Fri, 18 Aug 2017 12:39:13 +0200
Subject: Hitch SSL handshake errors
Message-ID: <35c1c2b5-8d3b-3037-c44c-f682b6682ef1@beckspaced.com>

Hello again ;)

hitch is up and online on production server

seeing some SSL handshake errors in the logs:

Aug 18 12:32:47 cx40 hitch[19755]: 171.50.241.116:61452 :0 9:10 
Handshake failure: error:140760FC:SSL 
routines:SSL23_GET_CLIENT_HELLO:unknown protocol
Aug 18 12:32:53 cx40 hitch[19755]: 171.50.241.116:61480 :0 9:10 
Handshake failure: error:140760FC:SSL 
routines:SSL23_GET_CLIENT_HELLO:unknown protocol
Aug 18 12:32:58 cx40 hitch[19756]: 171.50.241.116:61505 :0 10:11 
Handshake failure: error:140760FC:SSL 
routines:SSL23_GET_CLIENT_HELLO:unknown protocol

I suspect it's because of an old / insecure SSL version

Do i need to worry about this and make it downwards compatible?

Or can I safely ignore the errros and enjoy my day?

Thanks & greetings
Becki





From lagged at gmail.com  Fri Aug 18 14:45:05 2017
From: lagged at gmail.com (Andrei)
Date: Fri, 18 Aug 2017 09:45:05 -0500
Subject: Hitch SSL handshake errors
In-Reply-To: <35c1c2b5-8d3b-3037-c44c-f682b6682ef1@beckspaced.com>
References: <35c1c2b5-8d3b-3037-c44c-f682b6682ef1@beckspaced.com>
Message-ID: <CAP+vvEuh5At+rQNppgdsLC=gVoxA_6TDkuViN6+BysL0DyCUig@mail.gmail.com>

Glad to see the progress :) They're safe to ignore.

On Fri, Aug 18, 2017 at 5:39 AM, Admin Beckspaced <admin at beckspaced.com>
wrote:

> Hello again ;)
>
> hitch is up and online on production server
>
> seeing some SSL handshake errors in the logs:
>
> Aug 18 12:32:47 cx40 hitch[19755]: 171.50.241.116:61452 :0 9:10 Handshake
> failure: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown
> protocol
> Aug 18 12:32:53 cx40 hitch[19755]: 171.50.241.116:61480 :0 9:10 Handshake
> failure: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown
> protocol
> Aug 18 12:32:58 cx40 hitch[19756]: 171.50.241.116:61505 :0 10:11
> Handshake failure: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown
> protocol
>
> I suspect it's because of an old / insecure SSL version
>
> Do i need to worry about this and make it downwards compatible?
>
> Or can I safely ignore the errros and enjoy my day?
>
> Thanks & greetings
> Becki
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170818/694be4b7/attachment.html>

From dridi at varni.sh  Mon Aug 21 07:00:25 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 21 Aug 2017 09:00:25 +0200
Subject: troubleshooting Varnish purges vs bans
In-Reply-To: <9ed6b032-de6c-93ba-3f41-310feecb5b66@yahoo.es>
References: <3ivo1gyhfhrr15e2fhvvl608.1502293825989@email.android.com>
 <CABoVN9A1swB5qe-fp+ndVWOSk0VmC0mKHt15LpTSU5h_q+Nyvg@mail.gmail.com>
 <a3b23829-8bae-7dfa-f0b3-b04603f0bafd@yahoo.es>
 <CABoVN9DzSKQJRjaSsXiAh+W+Fr3RoWFyMUqEbiKsx-ZhDoAsag@mail.gmail.com>
 <36d2b0ba-fd61-a326-e9d1-2528c05bcd59@yahoo.es>
 <9ed6b032-de6c-93ba-3f41-310feecb5b66@yahoo.es>
Message-ID: <CABoVN9CBvVbSHg6iHb2Sp6kri4+pGfqLvokOwx7SbajfAhV91g@mail.gmail.com>

On Wed, Aug 16, 2017 at 12:13 PM, Miguel Gonz?lez
<miguel_3_gonzalez at yahoo.es> wrote:
> I have found out how to make to work varnish http purge plugin with
> Varnish 4.x in this article (funny it?s on the Varnish blog)

Thanks for letting us know, and sorry for not replying earlier!

Dridi



From rbizzell at measinc.com  Mon Aug 21 18:52:21 2017
From: rbizzell at measinc.com (Rodney Bizzell)
Date: Mon, 21 Aug 2017 18:52:21 +0000
Subject: config error
Message-ID: <1503341541750.30859@measinc.com>

Hello,

I am not sure why I am getting a config error can't see what I am missing here is a copy of my config.


backend stgncwrite {
    .host = "stg.ncwrite.com";
    .port = "80";
    .connect_timeout = 5s;
    .first_byte_timeout = 60s;
    .between_bytes_timeout = 60s;
    .probe = {
        .url = "/";
        .timeout = 5s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
     }




backend stgpegwritingscholar {
    .host = "stg.pegwritingscholar.com";
    .port = "80";
    .connect_timeout = 5s;
    .first_byte_timeout = 60s;
    .between_bytes_timeout = 60s;
    .probe = {
        .url = "/";
        .timeout = 5s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
     }



sub vcl_recv {
    if(req.http.host == "stg.ncwrite.com"){
       set req.backend_hint = stgncwrite;
    } else if (req.http.host == "stg.pegwritingscholar.com"){
       set req.backend_hint = stgpegwritingscholar;
      return (hash);
    }



#sub vcl_pass {





sub vcl_backend_response {
  set beresp.grace = 6h;
  set beresp.ttl = 5m;
}



This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. |



From charles at beachcamera.com  Mon Aug 21 19:24:46 2017
From: charles at beachcamera.com (Bender, Charles)
Date: Mon, 21 Aug 2017 19:24:46 +0000
Subject: config error
References: <1503341541750.30859@measinc.com>
Message-ID: <7300EDCB79BBC8489A35D7A51478B8F778CDE59C@bcvmexmbox01.BEACHCAMERA.LOCAL>

Probably your vcl_recv should look like this-

sub vcl_recv {
    if (req.http.host == "stg.ncwrite.com") {
       set req.backend_hint = stgncwrite;
       }
    if (req.http.host == "stg.pegwritingscholar.com"){
       set req.backend_hint = stgpegwritingscholar;
       }
    return (hash);
    }

Also in my humble opinion its dangerous to use DNS names for backends. If DNS is slow/down then your backends won't be reachable. Better to use IP address when defining host and send the Host header from client or set in VCL


On 08/21/2017 03:11 PM, Rodney Bizzell wrote:

Hello,

I am not sure why I am getting a config error can't see what I am missing here is a copy of my config.


backend stgncwrite {
    .host = "stg.ncwrite.com";
    .port = "80";
    .connect_timeout = 5s;
    .first_byte_timeout = 60s;
    .between_bytes_timeout = 60s;
    .probe = {
        .url = "/";
        .timeout = 5s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
     }




backend stgpegwritingscholar {
    .host = "stg.pegwritingscholar.com";
    .port = "80";
    .connect_timeout = 5s;
    .first_byte_timeout = 60s;
    .between_bytes_timeout = 60s;
    .probe = {
        .url = "/";
        .timeout = 5s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
     }



sub vcl_recv {
    if(req.http.host == "stg.ncwrite.com"){
       set req.backend_hint = stgncwrite;
    } else if (req.http.host == "stg.pegwritingscholar.com"){
       set req.backend_hint = stgpegwritingscholar;
      return (hash);
    }



#sub vcl_pass {





sub vcl_backend_response {
  set beresp.grace = 6h;
  set beresp.ttl = 5m;
}



This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. |

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170821/2b38240d/attachment.html>

From rbizzell at measinc.com  Mon Aug 21 19:32:26 2017
From: rbizzell at measinc.com (Rodney Bizzell)
Date: Mon, 21 Aug 2017 19:32:26 +0000
Subject: config error
In-Reply-To: <7300EDCB79BBC8489A35D7A51478B8F778CDE59C@bcvmexmbox01.BEACHCAMERA.LOCAL>
References: <1503341541750.30859@measinc.com>,
 <7300EDCB79BBC8489A35D7A51478B8F778CDE59C@bcvmexmbox01.BEACHCAMERA.LOCAL>
Message-ID: <1503343946531.61459@measinc.com>

When I made the change an error comes up and says backend stgpegwritingscholar {

________________________________
From: Bender, Charles <charles at beachcamera.com>
Sent: Monday, August 21, 2017 3:24 PM
To: Rodney Bizzell; varnish-misc
Subject: Re: config error


Probably your vcl_recv should look like this-

sub vcl_recv {
    if (req.http.host == "stg.ncwrite.com") {
       set req.backend_hint = stgncwrite;
       }
    if (req.http.host == "stg.pegwritingscholar.com"){
       set req.backend_hint = stgpegwritingscholar;
       }
    return (hash);
    }

Also in my humble opinion its dangerous to use DNS names for backends. If DNS is slow/down then your backends won't be reachable. Better to use IP address when defining host and send the Host header from client or set in VCL


On 08/21/2017 03:11 PM, Rodney Bizzell wrote:

Hello,

I am not sure why I am getting a config error can't see what I am missing here is a copy of my config.


backend stgncwrite {
    .host = "stg.ncwrite.com";
    .port = "80";
    .connect_timeout = 5s;
    .first_byte_timeout = 60s;
    .between_bytes_timeout = 60s;
    .probe = {
        .url = "/";
        .timeout = 5s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
     }




backend stgpegwritingscholar {
    .host = "stg.pegwritingscholar.com";
    .port = "80";
    .connect_timeout = 5s;
    .first_byte_timeout = 60s;
    .between_bytes_timeout = 60s;
    .probe = {
        .url = "/";
        .timeout = 5s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
     }



sub vcl_recv {
    if(req.http.host == "stg.ncwrite.com"){
       set req.backend_hint = stgncwrite;
    } else if (req.http.host == "stg.pegwritingscholar.com"){
       set req.backend_hint = stgpegwritingscholar;
      return (hash);
    }



#sub vcl_pass {





sub vcl_backend_response {
  set beresp.grace = 6h;
  set beresp.ttl = 5m;
}



This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. |

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc






From miguel_3_gonzalez at yahoo.es  Mon Aug 21 19:36:53 2017
From: miguel_3_gonzalez at yahoo.es (Miguel Gonzalez)
Date: Mon, 21 Aug 2017 21:36:53 +0200
Subject: config error
Message-ID: <icfis5386u9d0u3jmkhdjb44.1503344213813@email.android.com>

Which error are you getting?

Rodney Bizzell <rbizzell at measinc.com> wrote:

>Hello,
>
>I am not sure why I am getting a config error can't see what I am missing here is a copy of my config.
>
>
>backend stgncwrite {
>    .host = "stg.ncwrite.com";
>    .port = "80";
>    .connect_timeout = 5s;
>    .first_byte_timeout = 60s;
>    .between_bytes_timeout = 60s;
>    .probe = {
>        .url = "/";
>        .timeout = 5s;
>        .interval = 5s;
>        .window = 5;
>        .threshold = 3;
>     }
>
>
>
>
>backend stgpegwritingscholar {
>    .host = "stg.pegwritingscholar.com";
>    .port = "80";
>    .connect_timeout = 5s;
>    .first_byte_timeout = 60s;
>    .between_bytes_timeout = 60s;
>    .probe = {
>        .url = "/";
>        .timeout = 5s;
>        .interval = 5s;
>        .window = 5;
>        .threshold = 3;
>     }
>
>
>
>sub vcl_recv {
>    if(req.http.host == "stg.ncwrite.com"){
>       set req.backend_hint = stgncwrite;
>    } else if (req.http.host == "stg.pegwritingscholar.com"){
>       set req.backend_hint = stgpegwritingscholar;
>      return (hash);
>    }
>
>
>
>#sub vcl_pass {
>
>
>
>
>
>sub vcl_backend_response {
>  set beresp.grace = 6h;
>  set beresp.ttl = 5m;
>}
>
>
>
>This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. |
>
>_______________________________________________
>varnish-misc mailing list
>varnish-misc at varnish-cache.org
>https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

From rbizzell at measinc.com  Mon Aug 21 19:47:41 2017
From: rbizzell at measinc.com (Rodney Bizzell)
Date: Mon, 21 Aug 2017 19:47:41 +0000
Subject: config error
In-Reply-To: <icfis5386u9d0u3jmkhdjb44.1503344213813@email.android.com>
References: <icfis5386u9d0u3jmkhdjb44.1503344213813@email.android.com>
Message-ID: <1503344861043.49980@measinc.com>

When I run varnishd -C -f /etc/varnish/default.vcl 
Expected '.' got 'backend'
(program line 94), at
('/etc/varnish/default.vcl' Line 36 Pos 1)
backend stgpegwritingscholar {
#######-----------------------

________________________________________
From: Miguel Gonzalez <miguel_3_gonzalez at yahoo.es>
Sent: Monday, August 21, 2017 3:36 PM
To: Rodney Bizzell; varnish-misc
Subject: Re: config error

Which error are you getting?

Rodney Bizzell <rbizzell at measinc.com> wrote:

>Hello,
>
>I am not sure why I am getting a config error can't see what I am missing here is a copy of my config.
>
>
>backend stgncwrite {
>    .host = "stg.ncwrite.com";
>    .port = "80";
>    .connect_timeout = 5s;
>    .first_byte_timeout = 60s;
>    .between_bytes_timeout = 60s;
>    .probe = {
>        .url = "/";
>        .timeout = 5s;
>        .interval = 5s;
>        .window = 5;
>        .threshold = 3;
>     }
>
>
>
>
>backend stgpegwritingscholar {
>    .host = "stg.pegwritingscholar.com";
>    .port = "80";
>    .connect_timeout = 5s;
>    .first_byte_timeout = 60s;
>    .between_bytes_timeout = 60s;
>    .probe = {
>        .url = "/";
>        .timeout = 5s;
>        .interval = 5s;
>        .window = 5;
>        .threshold = 3;
>     }
>
>
>
>sub vcl_recv {
>    if(req.http.host == "stg.ncwrite.com"){
>       set req.backend_hint = stgncwrite;
>    } else if (req.http.host == "stg.pegwritingscholar.com"){
>       set req.backend_hint = stgpegwritingscholar;
>      return (hash);
>    }
>
>
>
>#sub vcl_pass {
>
>
>
>
>
>sub vcl_backend_response {
>  set beresp.grace = 6h;
>  set beresp.ttl = 5m;
>}
>
>
>
>This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. |
>
>_______________________________________________
>varnish-misc mailing list
>varnish-misc at varnish-cache.org
>https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From adam.schumacher at flightaware.com  Mon Aug 21 21:22:58 2017
From: adam.schumacher at flightaware.com (Adam Schumacher)
Date: Mon, 21 Aug 2017 21:22:58 +0000
Subject: config error
In-Reply-To: <1503344861043.49980@measinc.com>
References: <icfis5386u9d0u3jmkhdjb44.1503344213813@email.android.com>
 <1503344861043.49980@measinc.com>
Message-ID: <5CC7B5AF-A04E-4858-A6E8-729DD5CD3869@flightaware.com>

Maybe this is just a copy/paste issue, but it looks like you are missing close brackets on a bunch of those statements: ?}?


::Adam


On 8/21/17, 2:47 PM, "varnish-misc-bounces+adam.schumacher=flightaware.com at varnish-cache.org on behalf of Rodney Bizzell" <varnish-misc-bounces+adam.schumacher=flightaware.com at varnish-cache.org on behalf of rbizzell at measinc.com> wrote:

    When I run varnishd -C -f /etc/varnish/default.vcl 
    Expected '.' got 'backend'
    (program line 94), at
    ('/etc/varnish/default.vcl' Line 36 Pos 1)
    backend stgpegwritingscholar {
    #######-----------------------
    
    ________________________________________
    From: Miguel Gonzalez <miguel_3_gonzalez at yahoo.es>
    Sent: Monday, August 21, 2017 3:36 PM
    To: Rodney Bizzell; varnish-misc
    Subject: Re: config error
    
    Which error are you getting?
    
    Rodney Bizzell <rbizzell at measinc.com> wrote:
    
    >Hello,
    >
    >I am not sure why I am getting a config error can't see what I am missing here is a copy of my config.
    >
    >
    >backend stgncwrite {
    >    .host = "stg.ncwrite.com";
    >    .port = "80";
    >    .connect_timeout = 5s;
    >    .first_byte_timeout = 60s;
    >    .between_bytes_timeout = 60s;
    >    .probe = {
    >        .url = "/";
    >        .timeout = 5s;
    >        .interval = 5s;
    >        .window = 5;
    >        .threshold = 3;
    >     }
    >
    >
    >
    >
    >backend stgpegwritingscholar {
    >    .host = "stg.pegwritingscholar.com";
    >    .port = "80";
    >    .connect_timeout = 5s;
    >    .first_byte_timeout = 60s;
    >    .between_bytes_timeout = 60s;
    >    .probe = {
    >        .url = "/";
    >        .timeout = 5s;
    >        .interval = 5s;
    >        .window = 5;
    >        .threshold = 3;
    >     }
    >
    >
    >
    >sub vcl_recv {
    >    if(req.http.host == "stg.ncwrite.com"){
    >       set req.backend_hint = stgncwrite;
    >    } else if (req.http.host == "stg.pegwritingscholar.com"){
    >       set req.backend_hint = stgpegwritingscholar;
    >      return (hash);
    >    }
    >
    >
    >
    >#sub vcl_pass {
    >
    >
    >
    >
    >
    >sub vcl_backend_response {
    >  set beresp.grace = 6h;
    >  set beresp.ttl = 5m;
    >}
    >
    >
    >
    >This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. |
    >
    >_______________________________________________
    >varnish-misc mailing list
    >varnish-misc at varnish-cache.org
    >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
    
    _______________________________________________
    varnish-misc mailing list
    varnish-misc at varnish-cache.org
    https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
    


From miguel_3_gonzalez at yahoo.es  Mon Aug 21 21:37:26 2017
From: miguel_3_gonzalez at yahoo.es (Miguel Gonzalez)
Date: Mon, 21 Aug 2017 23:37:26 +0200
Subject: config error
Message-ID: <2jl7x6bd5bmksuln3sxdovpv.1503351446226@email.android.com>

It looks like you are missing a closing bracket right before the second backend definition

??? backend stgpegwritingscholar {
??? #######-----------------------

Adam Schumacher <adam.schumacher at flightaware.com> wrote:

>Maybe this is just a copy/paste issue, but it looks like you are missing close brackets on a bunch of those statements: ?}?
>
>
>::Adam
>
>
>On 8/21/17, 2:47 PM, "varnish-misc-bounces+adam.schumacher=flightaware.com at varnish-cache.org on behalf of Rodney Bizzell" <varnish-misc-bounces+adam.schumacher=flightaware.com at varnish-cache.org on behalf of rbizzell at measinc.com> wrote:
>
>    When I run varnishd -C -f /etc/varnish/default.vcl 
>    Expected '.' got 'backend'
>    (program line 94), at
>    ('/etc/varnish/default.vcl' Line 36 Pos 1)
>    backend stgpegwritingscholar {
>    #######-----------------------
>    
>    ________________________________________
>    From: Miguel Gonzalez <miguel_3_gonzalez at yahoo.es>
>    Sent: Monday, August 21, 2017 3:36 PM
>    To: Rodney Bizzell; varnish-misc
>    Subject: Re: config error
>    
>    Which error are you getting?
>    
>    Rodney Bizzell <rbizzell at measinc.com> wrote:
>    
>    >Hello,
>    >
>    >I am not sure why I am getting a config error can't see what I am missing here is a copy of my config.
>    >
>    >
>    >backend stgncwrite {
>    >    .host = "stg.ncwrite.com";
>    >    .port = "80";
>    >    .connect_timeout = 5s;
>    >    .first_byte_timeout = 60s;
>    >    .between_bytes_timeout = 60s;
>    >    .probe = {
>    >        .url = "/";
>    >        .timeout = 5s;
>    >        .interval = 5s;
>    >        .window = 5;
>    >        .threshold = 3;
>    >     }
>    >
>    >
>    >
>    >
>    >backend stgpegwritingscholar {
>    >    .host = "stg.pegwritingscholar.com";
>    >    .port = "80";
>    >    .connect_timeout = 5s;
>    >    .first_byte_timeout = 60s;
>    >    .between_bytes_timeout = 60s;
>    >    .probe = {
>    >        .url = "/";
>    >        .timeout = 5s;
>    >        .interval = 5s;
>    >        .window = 5;
>    >        .threshold = 3;
>    >     }
>    >
>    >
>    >
>    >sub vcl_recv {
>    >    if(req.http.host == "stg.ncwrite.com"){
>    >       set req.backend_hint = stgncwrite;
>    >    } else if (req.http.host == "stg.pegwritingscholar.com"){
>    >       set req.backend_hint = stgpegwritingscholar;
>    >      return (hash);
>    >    }
>    >
>    >
>    >
>    >#sub vcl_pass {
>    >
>    >
>    >
>    >
>    >
>    >sub vcl_backend_response {
>    >  set beresp.grace = 6h;
>    >  set beresp.ttl = 5m;
>    >}
>    >
>    >
>    >
>    >This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. |
>    >
>    >_______________________________________________
>    >varnish-misc mailing list
>    >varnish-misc at varnish-cache.org
>    >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>    
>    _______________________________________________
>    varnish-misc mailing list
>    varnish-misc at varnish-cache.org
>    https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>    
>

From rbizzell at measinc.com  Tue Aug 22 12:49:05 2017
From: rbizzell at measinc.com (Rodney Bizzell)
Date: Tue, 22 Aug 2017 12:49:05 +0000
Subject: config error
In-Reply-To: <2jl7x6bd5bmksuln3sxdovpv.1503351446226@email.android.com>
References: <2jl7x6bd5bmksuln3sxdovpv.1503351446226@email.android.com>
Message-ID: <1503406145398.52111@measinc.com>

okay let me double check my config thanks!
________________________________________
From: Miguel Gonzalez <miguel_3_gonzalez at yahoo.es>
Sent: Monday, August 21, 2017 5:37 PM
To: Adam Schumacher; Rodney Bizzell; varnish-misc
Subject: Re: config error

It looks like you are missing a closing bracket right before the second backend definition

    backend stgpegwritingscholar {
    #######-----------------------

Adam Schumacher <adam.schumacher at flightaware.com> wrote:

>Maybe this is just a copy/paste issue, but it looks like you are missing close brackets on a bunch of those statements: ?}?
>
>
>::Adam
>
>
>On 8/21/17, 2:47 PM, "varnish-misc-bounces+adam.schumacher=flightaware.com at varnish-cache.org on behalf of Rodney Bizzell" <varnish-misc-bounces+adam.schumacher=flightaware.com at varnish-cache.org on behalf of rbizzell at measinc.com> wrote:
>
>    When I run varnishd -C -f /etc/varnish/default.vcl
>    Expected '.' got 'backend'
>    (program line 94), at
>    ('/etc/varnish/default.vcl' Line 36 Pos 1)
>    backend stgpegwritingscholar {
>    #######-----------------------
>
>    ________________________________________
>    From: Miguel Gonzalez <miguel_3_gonzalez at yahoo.es>
>    Sent: Monday, August 21, 2017 3:36 PM
>    To: Rodney Bizzell; varnish-misc
>    Subject: Re: config error
>
>    Which error are you getting?
>
>    Rodney Bizzell <rbizzell at measinc.com> wrote:
>
>    >Hello,
>    >
>    >I am not sure why I am getting a config error can't see what I am missing here is a copy of my config.
>    >
>    >
>    >backend stgncwrite {
>    >    .host = "stg.ncwrite.com";
>    >    .port = "80";
>    >    .connect_timeout = 5s;
>    >    .first_byte_timeout = 60s;
>    >    .between_bytes_timeout = 60s;
>    >    .probe = {
>    >        .url = "/";
>    >        .timeout = 5s;
>    >        .interval = 5s;
>    >        .window = 5;
>    >        .threshold = 3;
>    >     }
>    >
>    >
>    >
>    >
>    >backend stgpegwritingscholar {
>    >    .host = "stg.pegwritingscholar.com";
>    >    .port = "80";
>    >    .connect_timeout = 5s;
>    >    .first_byte_timeout = 60s;
>    >    .between_bytes_timeout = 60s;
>    >    .probe = {
>    >        .url = "/";
>    >        .timeout = 5s;
>    >        .interval = 5s;
>    >        .window = 5;
>    >        .threshold = 3;
>    >     }
>    >
>    >
>    >
>    >sub vcl_recv {
>    >    if(req.http.host == "stg.ncwrite.com"){
>    >       set req.backend_hint = stgncwrite;
>    >    } else if (req.http.host == "stg.pegwritingscholar.com"){
>    >       set req.backend_hint = stgpegwritingscholar;
>    >      return (hash);
>    >    }
>    >
>    >
>    >
>    >#sub vcl_pass {
>    >
>    >
>    >
>    >
>    >
>    >sub vcl_backend_response {
>    >  set beresp.grace = 6h;
>    >  set beresp.ttl = 5m;
>    >}
>    >
>    >
>    >
>    >This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. |
>    >
>    >_______________________________________________
>    >varnish-misc mailing list
>    >varnish-misc at varnish-cache.org
>    >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>    _______________________________________________
>    varnish-misc mailing list
>    varnish-misc at varnish-cache.org
>    https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>



From rbizzell at measinc.com  Thu Aug 24 15:41:20 2017
From: rbizzell at measinc.com (Rodney Bizzell)
Date: Thu, 24 Aug 2017 15:41:20 +0000
Subject: Estimated Memory allocation
Message-ID: <1503589280140.29379@measinc.com>

Hello,

I would like to know what is the best practices for estimating allocation of memory for varnish is there minimum based off of the number of backend web servers.Thanks!


This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. |



From kokoniimasu at gmail.com  Mon Aug 28 05:47:21 2017
From: kokoniimasu at gmail.com (kokoniimasu)
Date: Mon, 28 Aug 2017 14:47:21 +0900
Subject: Show source index in VCL_trace
Message-ID: <CAJihfGBSvUrOx5wp5f6DuP1PSfeJtxOZ0bqtZmKMJvNHMbQnGQ@mail.gmail.com>

Hello,

I'm using VCL_trace for debugging.
But, VCL_trace is not enough little info, if using multipe VCLs.


  24025048 VCL_trace      c 3 14.5
  24025048 VCL_trace      c 4 18.9
                            |  |  |
                            |  |  +- VCL program line position
                            |  +---- VCL program line number
                            +------- VCL trace point index


VCL_trace does not have identify source file way in VSL.
To identify, need to run varnishd "varnishd -C -f [file]"


  $varnishd -C -f /etc/varnish/default.vcl 2>&1 |less
  ...
  static struct vrt_ref VGC_ref[VGC_NREFS] = {
    [  1] = { 2,     1670,   66,   3,  "new" },
    [  2] = { 0,      143,    7,   3,  "new" },
    [  3] = { 0,      289,   14,   5,  "if" },
    [  4] = { 0,      392,   18,   9,  "set" },
  ...


This way have a problem, because it can't be guaranteed same as a
running config.

I thought 2 pattern solution.

- Add vcl source index in vcl_trace.
https://github.com/varnishcache/varnish-cache/compare/master...xcir:patch/modify_vcl_trace?expand=1

   32770 VCL_trace      c 1 2.3.3
   32770 VCL_trace      c 2 2.4.5
                          | | | |
                          | | | +---- VCL program line position
                          | | +------ VCL program line number
                          | +-------- VCL program source index
                          +---------- VCL trace point index



- Add -t(trace) option in vcl.show
https://github.com/varnishcache/varnish-cache/compare/master...xcir:patch/add_t_opt_cli?expand=1
This patch dumps VGC_ref.

   $ sudo varnishadm vcl.show -t boot
   index src  line  pos token
       1   2     3    3 "if"
       2   2     4    5 "std.log"
   ...
      36   1   189    5 "return"
      37   1   193    5 "return"

I think modify to vcl_trace is better.
But, VSL format change is painful.

I'll like to create either pull-request.
Can I get your opinion?

Regards,
--
Shohei Tanaka(@xcir)
http://blog.xcir.net/ (JP)



From jc at eworx.gr  Mon Aug 28 06:42:08 2017
From: jc at eworx.gr (John Cherouvim)
Date: Mon, 28 Aug 2017 09:42:08 +0300
Subject: varnishadm or varnish in weird state
Message-ID: <59A3BB40.80204@eworx.gr>

I have 2 Ubuntu 16 LTS servers provisioned using the same ansible
playbook and they both run varnish-4.1.1 with the exact same
configuration. The first server is production (receives a lot of
traffic) and the other one is staging (minimal traffic). They both have
~20 days of uptime with varnishstat reporting an 11 days uptime each.

On production I get weird results for trying to list the backends. Some
times I get this:
$ sudo varnishadm backend.list
PONG 1503057040 1.0

And some other times I get this:
$ sudo varnishadm backend.list
Backend name                   Admin      Probe
8440ffbd-e1de-4827-9f83-9096f5a97bf1.www probe      Healthy 5/5


but on staging I consistently get the following, which is what I've been
used to seeing in all other environments I've used varnish:
$ sudo varnishadm backend.list
Backend name                   Admin      Probe
boot.www                       probe      Healthy 5/5


My /etc/varnish/default.vcl starts like this on both servers:
> vcl 4.0;
> backend www {
>         .host = "localhost";
>         .port = "8888";
>         .connect_timeout = 60s;
>         .first_byte_timeout = 120s;
>         .between_bytes_timeout = 120s;
>         .max_connections = 256;
>         .probe = {
>                 .url = "/health-check";
>                 .timeout = 15s;
>                 .interval = 5s;
>                 .window = 5;
>                 .threshold = 2;
>         }
> }



For that kind of configuration I've never seen that hash like backend
name, nor that behavior where it randomly shows that or the "PONG" response.

Doing a "service varnish restart" fixed this.
But does anyone know why did this happen in the first place?

thanks




From dridi at varni.sh  Mon Aug 28 07:11:54 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 28 Aug 2017 09:11:54 +0200
Subject: varnishadm or varnish in weird state
In-Reply-To: <59A3BB40.80204@eworx.gr>
References: <59A3BB40.80204@eworx.gr>
Message-ID: <CABoVN9CLzOaBFL8OyQt+UOFqXcbU9xyeci+mcgyvS_=-YmfuJQ@mail.gmail.com>

> $ sudo varnishadm backend.list
> PONG 1503057040 1.0

Hello,

This is a known bug, fixed a while ago:

https://github.com/varnishcache/varnish-cache/issues/2010
https://github.com/varnishcache/varnish-cache/pull/2019

You should upgrade to 4.1.8, your version has a major vulnerability
and known bugs:

https://varnish-cache.org/security/VSV00001.html

Cheers



From dridi at varni.sh  Mon Aug 28 07:19:53 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 28 Aug 2017 09:19:53 +0200
Subject: Show source index in VCL_trace
In-Reply-To: <CAJihfGBSvUrOx5wp5f6DuP1PSfeJtxOZ0bqtZmKMJvNHMbQnGQ@mail.gmail.com>
References: <CAJihfGBSvUrOx5wp5f6DuP1PSfeJtxOZ0bqtZmKMJvNHMbQnGQ@mail.gmail.com>
Message-ID: <CABoVN9DWhqc=z9HEPBtPxodnfYpEUaG+NN4sbaVuEaKXZ0XfCw@mail.gmail.com>

> I thought 2 pattern solution.
>
> - Add vcl source index in vcl_trace.
> https://github.com/varnishcache/varnish-cache/compare/master...xcir:patch/modify_vcl_trace?expand=1
>
>    32770 VCL_trace      c 1 2.3.3
>    32770 VCL_trace      c 2 2.4.5
>                           | | | |
>                           | | | +---- VCL program line position
>                           | | +------ VCL program line number
>                           | +-------- VCL program source index
>                           +---------- VCL trace point index

Usually when we add a field to a record, it's appended to avoid
breaking existing programs already relying on the existing fields.

> - Add -t(trace) option in vcl.show
> https://github.com/varnishcache/varnish-cache/compare/master...xcir:patch/add_t_opt_cli?expand=1
> This patch dumps VGC_ref.
>
>    $ sudo varnishadm vcl.show -t boot
>    index src  line  pos token
>        1   2     3    3 "if"
>        2   2     4    5 "std.log"
>    ...
>       36   1   189    5 "return"
>       37   1   193    5 "return"
>
> I think modify to vcl_trace is better.
> But, VSL format change is painful.

I think we discussed this at some point and another solution was to
reference line numbers from the output of `vcl.show -v` that already
contains the whole VCL, including the built-in and the includes.

> I'll like to create either pull-request.
> Can I get your opinion?

I will mention this during today's bugwash, I'm not sure you can
attend given your timezone.

Cheers



From jc at eworx.gr  Mon Aug 28 07:21:53 2017
From: jc at eworx.gr (John Cherouvim)
Date: Mon, 28 Aug 2017 10:21:53 +0300
Subject: how can I read old varnish logs?
Message-ID: <59A3C491.2030504@eworx.gr>

In /var/log/varnish/ I have some old varnish logs, produced by 
varnishlog which, due to logrotated, are now in gz format:

 > -rw-r--r--  1 varnishlog varnish 143068514 Aug 10 23:59 
varnish.log.2017-08-10.gz
 > -rw-r--r--  1 varnishlog varnish 156373518 Aug 11 23:59 
varnish.log.2017-08-11.gz
 > -rw-r--r--  1 varnishlog varnish 134255825 Aug 12 23:59 
varnish.log.2017-08-12.gz
 > -rw-r--r--  1 varnishlog varnish 156992529 Aug 13 23:59 
varnish.log.2017-08-13.gz
 > -rw-r--r--  1 varnishlog varnish 176751837 Aug 14 23:59 
varnish.log.2017-08-14.gz
 > -rw-r--r--  1 varnishlog varnish 155948012 Aug 16 00:01 
varnish.log.2017-08-15.gz
 > -rw-r--r--  1 varnishlog varnish 169977134 Aug 17 00:01 
varnish.log.2017-08-16.gz

I've extracted those in another location and tried to view them via 
head/more but they look binary.

So I then tried opening them with varnishlog using either of the 
following parameters:

 > [-N filename]             VSM filename
 > [-r filename]             Binary file input

But that didn't work giving me:

 > Can't open log - retrying for 5 seconds

and:

 > Can't open log file (Not a VSL file:

Any idea on how can I inspect those historical logs from varnish?

p.s I use varnish-4.1.1



From guillaume at varnish-software.com  Mon Aug 28 07:31:47 2017
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Mon, 28 Aug 2017 09:31:47 +0200
Subject: Show source index in VCL_trace
In-Reply-To: <CAJihfGBSvUrOx5wp5f6DuP1PSfeJtxOZ0bqtZmKMJvNHMbQnGQ@mail.gmail.com>
References: <CAJihfGBSvUrOx5wp5f6DuP1PSfeJtxOZ0bqtZmKMJvNHMbQnGQ@mail.gmail.com>
Message-ID: <CAJ6ZYQwCzPChwqjSB9y3e4yK_zzJ_qsyDTPOYtwjL4LnVJ54WA@mail.gmail.com>

Hi,

We sort of already discussed that a few months ago and sort of agreed on
the first proposition (adding an vcl file index). I wanted to have a tool
actually using it before pushing any changes and that got put on the
backburner...

The VSL isn't painful, it's just a printf :-) I can help if you need a hand.

-- 
Guillaume Quintard

On Mon, Aug 28, 2017 at 7:47 AM, kokoniimasu <kokoniimasu at gmail.com> wrote:

> Hello,
>
> I'm using VCL_trace for debugging.
> But, VCL_trace is not enough little info, if using multipe VCLs.
>
>
>   24025048 VCL_trace      c 3 14.5
>   24025048 VCL_trace      c 4 18.9
>                             |  |  |
>                             |  |  +- VCL program line position
>                             |  +---- VCL program line number
>                             +------- VCL trace point index
>
>
> VCL_trace does not have identify source file way in VSL.
> To identify, need to run varnishd "varnishd -C -f [file]"
>
>
>   $varnishd -C -f /etc/varnish/default.vcl 2>&1 |less
>   ...
>   static struct vrt_ref VGC_ref[VGC_NREFS] = {
>     [  1] = { 2,     1670,   66,   3,  "new" },
>     [  2] = { 0,      143,    7,   3,  "new" },
>     [  3] = { 0,      289,   14,   5,  "if" },
>     [  4] = { 0,      392,   18,   9,  "set" },
>   ...
>
>
> This way have a problem, because it can't be guaranteed same as a
> running config.
>
> I thought 2 pattern solution.
>
> - Add vcl source index in vcl_trace.
> https://github.com/varnishcache/varnish-cache/compare/master...xcir:patch/
> modify_vcl_trace?expand=1
>
>    32770 VCL_trace      c 1 2.3.3
>    32770 VCL_trace      c 2 2.4.5
>                           | | | |
>                           | | | +---- VCL program line position
>                           | | +------ VCL program line number
>                           | +-------- VCL program source index
>                           +---------- VCL trace point index
>
>
>
> - Add -t(trace) option in vcl.show
> https://github.com/varnishcache/varnish-cache/compare/master...xcir:patch/
> add_t_opt_cli?expand=1
> This patch dumps VGC_ref.
>
>    $ sudo varnishadm vcl.show -t boot
>    index src  line  pos token
>        1   2     3    3 "if"
>        2   2     4    5 "std.log"
>    ...
>       36   1   189    5 "return"
>       37   1   193    5 "return"
>
> I think modify to vcl_trace is better.
> But, VSL format change is painful.
>
> I'll like to create either pull-request.
> Can I get your opinion?
>
> Regards,
> --
> Shohei Tanaka(@xcir)
> http://blog.xcir.net/ (JP)
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170828/4af3218f/attachment.html>

From kokoniimasu at gmail.com  Mon Aug 28 08:23:49 2017
From: kokoniimasu at gmail.com (kokoniimasu)
Date: Mon, 28 Aug 2017 17:23:49 +0900
Subject: Show source index in VCL_trace
In-Reply-To: <CAJ6ZYQwCzPChwqjSB9y3e4yK_zzJ_qsyDTPOYtwjL4LnVJ54WA@mail.gmail.com>
References: <CAJihfGBSvUrOx5wp5f6DuP1PSfeJtxOZ0bqtZmKMJvNHMbQnGQ@mail.gmail.com>
 <CAJ6ZYQwCzPChwqjSB9y3e4yK_zzJ_qsyDTPOYtwjL4LnVJ54WA@mail.gmail.com>
Message-ID: <CAJihfGAdXEO3-yoDw28-472hj1Z3uFSU8abWToSRD2teYe0AqA@mail.gmail.com>

Hi, Guillaume

Thank you for yur replay

> We sort of already discussed that a few months ago and sort of agreed on the first proposition (adding an vcl file index). I wanted to have a tool actually using it before pushing any changes and that got put on the backburner...

Good news.
I'll wait VSL format change!


2017-08-28 16:31 GMT+09:00 Guillaume Quintard <guillaume at varnish-software.com>:
> Hi,
>
> We sort of already discussed that a few months ago and sort of agreed on the
> first proposition (adding an vcl file index). I wanted to have a tool
> actually using it before pushing any changes and that got put on the
> backburner...
>
> The VSL isn't painful, it's just a printf :-) I can help if you need a hand.
>
> --
> Guillaume Quintard
>
> On Mon, Aug 28, 2017 at 7:47 AM, kokoniimasu <kokoniimasu at gmail.com> wrote:
>>
>> Hello,
>>
>> I'm using VCL_trace for debugging.
>> But, VCL_trace is not enough little info, if using multipe VCLs.
>>
>>
>>   24025048 VCL_trace      c 3 14.5
>>   24025048 VCL_trace      c 4 18.9
>>                             |  |  |
>>                             |  |  +- VCL program line position
>>                             |  +---- VCL program line number
>>                             +------- VCL trace point index
>>
>>
>> VCL_trace does not have identify source file way in VSL.
>> To identify, need to run varnishd "varnishd -C -f [file]"
>>
>>
>>   $varnishd -C -f /etc/varnish/default.vcl 2>&1 |less
>>   ...
>>   static struct vrt_ref VGC_ref[VGC_NREFS] = {
>>     [  1] = { 2,     1670,   66,   3,  "new" },
>>     [  2] = { 0,      143,    7,   3,  "new" },
>>     [  3] = { 0,      289,   14,   5,  "if" },
>>     [  4] = { 0,      392,   18,   9,  "set" },
>>   ...
>>
>>
>> This way have a problem, because it can't be guaranteed same as a
>> running config.
>>
>> I thought 2 pattern solution.
>>
>> - Add vcl source index in vcl_trace.
>>
>> https://github.com/varnishcache/varnish-cache/compare/master...xcir:patch/modify_vcl_trace?expand=1
>>
>>    32770 VCL_trace      c 1 2.3.3
>>    32770 VCL_trace      c 2 2.4.5
>>                           | | | |
>>                           | | | +---- VCL program line position
>>                           | | +------ VCL program line number
>>                           | +-------- VCL program source index
>>                           +---------- VCL trace point index
>>
>>
>>
>> - Add -t(trace) option in vcl.show
>>
>> https://github.com/varnishcache/varnish-cache/compare/master...xcir:patch/add_t_opt_cli?expand=1
>> This patch dumps VGC_ref.
>>
>>    $ sudo varnishadm vcl.show -t boot
>>    index src  line  pos token
>>        1   2     3    3 "if"
>>        2   2     4    5 "std.log"
>>    ...
>>       36   1   189    5 "return"
>>       37   1   193    5 "return"
>>
>> I think modify to vcl_trace is better.
>> But, VSL format change is painful.
>>
>> I'll like to create either pull-request.
>> Can I get your opinion?
>>
>> Regards,
>> --
>> Shohei Tanaka(@xcir)
>> http://blog.xcir.net/ (JP)
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>



From guillaume at varnish-software.com  Mon Aug 28 08:33:29 2017
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Mon, 28 Aug 2017 10:33:29 +0200
Subject: Show source index in VCL_trace
In-Reply-To: <CAJihfGAdXEO3-yoDw28-472hj1Z3uFSU8abWToSRD2teYe0AqA@mail.gmail.com>
References: <CAJihfGBSvUrOx5wp5f6DuP1PSfeJtxOZ0bqtZmKMJvNHMbQnGQ@mail.gmail.com>
 <CAJ6ZYQwCzPChwqjSB9y3e4yK_zzJ_qsyDTPOYtwjL4LnVJ54WA@mail.gmail.com>
 <CAJihfGAdXEO3-yoDw28-472hj1Z3uFSU8abWToSRD2teYe0AqA@mail.gmail.com>
Message-ID: <CAJ6ZYQwdtfzDhsgqGyAcsUK0TJtZ+23LovbwidZa244p8kvqxQ@mail.gmail.com>

Aaaaaaaaactually, I was more saying that we would be open to a PR :-)

the "appending to not break stuff" may not play here, with 6.0 and all
(specially since it's a rarely used interface)

Let's wait for the bugwash.

-- 
Guillaume Quintard

On Mon, Aug 28, 2017 at 10:23 AM, kokoniimasu <kokoniimasu at gmail.com> wrote:

> Hi, Guillaume
>
> Thank you for yur replay
>
> > We sort of already discussed that a few months ago and sort of agreed on
> the first proposition (adding an vcl file index). I wanted to have a tool
> actually using it before pushing any changes and that got put on the
> backburner...
>
> Good news.
> I'll wait VSL format change!
>
>
> 2017-08-28 16:31 GMT+09:00 Guillaume Quintard <guillaume at varnish-software.
> com>:
> > Hi,
> >
> > We sort of already discussed that a few months ago and sort of agreed on
> the
> > first proposition (adding an vcl file index). I wanted to have a tool
> > actually using it before pushing any changes and that got put on the
> > backburner...
> >
> > The VSL isn't painful, it's just a printf :-) I can help if you need a
> hand.
> >
> > --
> > Guillaume Quintard
> >
> > On Mon, Aug 28, 2017 at 7:47 AM, kokoniimasu <kokoniimasu at gmail.com>
> wrote:
> >>
> >> Hello,
> >>
> >> I'm using VCL_trace for debugging.
> >> But, VCL_trace is not enough little info, if using multipe VCLs.
> >>
> >>
> >>   24025048 VCL_trace      c 3 14.5
> >>   24025048 VCL_trace      c 4 18.9
> >>                             |  |  |
> >>                             |  |  +- VCL program line position
> >>                             |  +---- VCL program line number
> >>                             +------- VCL trace point index
> >>
> >>
> >> VCL_trace does not have identify source file way in VSL.
> >> To identify, need to run varnishd "varnishd -C -f [file]"
> >>
> >>
> >>   $varnishd -C -f /etc/varnish/default.vcl 2>&1 |less
> >>   ...
> >>   static struct vrt_ref VGC_ref[VGC_NREFS] = {
> >>     [  1] = { 2,     1670,   66,   3,  "new" },
> >>     [  2] = { 0,      143,    7,   3,  "new" },
> >>     [  3] = { 0,      289,   14,   5,  "if" },
> >>     [  4] = { 0,      392,   18,   9,  "set" },
> >>   ...
> >>
> >>
> >> This way have a problem, because it can't be guaranteed same as a
> >> running config.
> >>
> >> I thought 2 pattern solution.
> >>
> >> - Add vcl source index in vcl_trace.
> >>
> >> https://github.com/varnishcache/varnish-cache/
> compare/master...xcir:patch/modify_vcl_trace?expand=1
> >>
> >>    32770 VCL_trace      c 1 2.3.3
> >>    32770 VCL_trace      c 2 2.4.5
> >>                           | | | |
> >>                           | | | +---- VCL program line position
> >>                           | | +------ VCL program line number
> >>                           | +-------- VCL program source index
> >>                           +---------- VCL trace point index
> >>
> >>
> >>
> >> - Add -t(trace) option in vcl.show
> >>
> >> https://github.com/varnishcache/varnish-cache/
> compare/master...xcir:patch/add_t_opt_cli?expand=1
> >> This patch dumps VGC_ref.
> >>
> >>    $ sudo varnishadm vcl.show -t boot
> >>    index src  line  pos token
> >>        1   2     3    3 "if"
> >>        2   2     4    5 "std.log"
> >>    ...
> >>       36   1   189    5 "return"
> >>       37   1   193    5 "return"
> >>
> >> I think modify to vcl_trace is better.
> >> But, VSL format change is painful.
> >>
> >> I'll like to create either pull-request.
> >> Can I get your opinion?
> >>
> >> Regards,
> >> --
> >> Shohei Tanaka(@xcir)
> >> http://blog.xcir.net/ (JP)
> >>
> >> _______________________________________________
> >> varnish-misc mailing list
> >> varnish-misc at varnish-cache.org
> >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170828/51573892/attachment.html>

From dridi at varni.sh  Mon Aug 28 08:59:06 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 28 Aug 2017 10:59:06 +0200
Subject: how can I read old varnish logs?
In-Reply-To: <59A3C491.2030504@eworx.gr>
References: <59A3C491.2030504@eworx.gr>
Message-ID: <CABoVN9C=pONCej+rtk9CxczduhqQzuMpwEo0+esSssuBZeaxfA@mail.gmail.com>

On Mon, Aug 28, 2017 at 9:21 AM, John Cherouvim <jc at eworx.gr> wrote:
> In /var/log/varnish/ I have some old varnish logs, produced by varnishlog
> which, due to logrotated, are now in gz format:
>
>> -rw-r--r--  1 varnishlog varnish 143068514 Aug 10 23:59
>> varnish.log.2017-08-10.gz
>> -rw-r--r--  1 varnishlog varnish 156373518 Aug 11 23:59
>> varnish.log.2017-08-11.gz
>> -rw-r--r--  1 varnishlog varnish 134255825 Aug 12 23:59
>> varnish.log.2017-08-12.gz
>> -rw-r--r--  1 varnishlog varnish 156992529 Aug 13 23:59
>> varnish.log.2017-08-13.gz
>> -rw-r--r--  1 varnishlog varnish 176751837 Aug 14 23:59
>> varnish.log.2017-08-14.gz
>> -rw-r--r--  1 varnishlog varnish 155948012 Aug 16 00:01
>> varnish.log.2017-08-15.gz
>> -rw-r--r--  1 varnishlog varnish 169977134 Aug 17 00:01
>> varnish.log.2017-08-16.gz
>
> I've extracted those in another location and tried to view them via
> head/more but they look binary.
>
> So I then tried opening them with varnishlog using either of the following
> parameters:
>
>> [-N filename]             VSM filename
>> [-r filename]             Binary file input
>
> But that didn't work giving me:
>
>> Can't open log - retrying for 5 seconds
>
> and:
>
>> Can't open log file (Not a VSL file:
>
> Any idea on how can I inspect those historical logs from varnish?

What is the output of `file some_extracted_file`?

They are probably ascii dump of the logs if varnishlog is complaining.

Dridi



From jc at eworx.gr  Mon Aug 28 12:18:13 2017
From: jc at eworx.gr (John Cherouvim)
Date: Mon, 28 Aug 2017 15:18:13 +0300
Subject: how can I read old varnish logs?
In-Reply-To: <CABoVN9C=pONCej+rtk9CxczduhqQzuMpwEo0+esSssuBZeaxfA@mail.gmail.com>
References: <59A3C491.2030504@eworx.gr>
 <CABoVN9C=pONCej+rtk9CxczduhqQzuMpwEo0+esSssuBZeaxfA@mail.gmail.com>
Message-ID: <59A40A05.4040308@eworx.gr>

 > What is the output of `file some_extracted_file`?

It's "data".

The output of head looks like this:

 > )(?86 boot.www0P)(?Beresp: 1503698757.375353 119.998656 
119.998480-P)(?Error: 1503698757.375362 119.998665 0.000009 
))(?HTTP/1.1*)(?503+)(?Service Unavailable+)(?Backend fetch 
failed$,)(?Date: Fri, 25 Aug 2017 22:05:57 GMT,)(?Server: 
Varnish<)(?BACKEND_ERROR',)(?Content-Type: text/html; 
charset=utf-8,)(?Retry-After: >)(?deliverO)(?malloc Transient 
1)(?HTTP/1.12)(?5033)(?Backend fetch failed$4)(?Date: Fri, 25 Aug 2017 
22:05:57 GMT4)(?Server: Varnish'4)(?Content-Typ)(?284S)(?292 0 292 0 0 
0M)(?L)(@req 2631964 rxreq+P)(@Start: 1503698637.376097 0.000000 
0.000000)P)(@Req: 1503698637.376097 0.000000 0.000000?)(@54.162.61.100 
34377)(@GETg)(@/en/ )(@HTTP/1.1)(@Connection: closeG)(@User-Agent: 
Flamingo_SearchEngine (+http://www.flamingosearch.com/bot))(@Host: 
www.example.com)(@X-Forwarded-For: 
54.111.111.111<)(@RECV>)(@hash<)(@HASH>)(@lookup<)(@MISS>)(@fetchK)(@bereq 
2631966 fetch/P)(@Fetch: 1503698757.439626 120.063529 120.063529 
)(@HTTP/1.1)(@503Backend fetch failed$)(@Date: Fri, 25 Aug 2017 22:05:57 
GMT)(@Server: Varnish')(@Content-Type: text/html; 
charset=utf-8)(@Retry-After: 5)(@X-Varnish: 2631965)(@Age: 0)(@Via: 1.1 
varnish-v<)(@DELIVER)(@Via: 1.1 varnish-v>)(@deliver/P)(@Process: 
1503698757.439677 120.063580 0.000051)(@Content-Length: 284...



Dridi Boukelmoune wrote:
> On Mon, Aug 28, 2017 at 9:21 AM, John Cherouvim <jc at eworx.gr> wrote:
>   
>> In /var/log/varnish/ I have some old varnish logs, produced by varnishlog
>> which, due to logrotated, are now in gz format:
>>
>>     
>>> -rw-r--r--  1 varnishlog varnish 143068514 Aug 10 23:59
>>> varnish.log.2017-08-10.gz
>>> -rw-r--r--  1 varnishlog varnish 156373518 Aug 11 23:59
>>> varnish.log.2017-08-11.gz
>>> -rw-r--r--  1 varnishlog varnish 134255825 Aug 12 23:59
>>> varnish.log.2017-08-12.gz
>>> -rw-r--r--  1 varnishlog varnish 156992529 Aug 13 23:59
>>> varnish.log.2017-08-13.gz
>>> -rw-r--r--  1 varnishlog varnish 176751837 Aug 14 23:59
>>> varnish.log.2017-08-14.gz
>>> -rw-r--r--  1 varnishlog varnish 155948012 Aug 16 00:01
>>> varnish.log.2017-08-15.gz
>>> -rw-r--r--  1 varnishlog varnish 169977134 Aug 17 00:01
>>> varnish.log.2017-08-16.gz
>>>       
>> I've extracted those in another location and tried to view them via
>> head/more but they look binary.
>>
>> So I then tried opening them with varnishlog using either of the following
>> parameters:
>>
>>     
>>> [-N filename]             VSM filename
>>> [-r filename]             Binary file input
>>>       
>> But that didn't work giving me:
>>
>>     
>>> Can't open log - retrying for 5 seconds
>>>       
>> and:
>>
>>     
>>> Can't open log file (Not a VSL file:
>>>       
>> Any idea on how can I inspect those historical logs from varnish?
>>     
>
> What is the output of `file some_extracted_file`?
>
> They are probably ascii dump of the logs if varnishlog is complaining.
>
> Dridi
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>   




From dridi at varni.sh  Mon Aug 28 12:40:30 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 28 Aug 2017 14:40:30 +0200
Subject: how can I read old varnish logs?
In-Reply-To: <59A40A05.4040308@eworx.gr>
References: <59A3C491.2030504@eworx.gr>
 <CABoVN9C=pONCej+rtk9CxczduhqQzuMpwEo0+esSssuBZeaxfA@mail.gmail.com>
 <59A40A05.4040308@eworx.gr>
Message-ID: <CABoVN9CNPGGpF52Yw+eCc4Ut=79a-SSMwdrUUJ5VJ8ncdMwPrw@mail.gmail.com>

On Mon, Aug 28, 2017 at 2:18 PM, John Cherouvim <jc at eworx.gr> wrote:
>> What is the output of `file some_extracted_file`?
>
> It's "data".

Ouch, what is the output of `hexdump -C some_extracted_file | head -1` then?

Dridi



From jc at eworx.gr  Mon Aug 28 13:05:43 2017
From: jc at eworx.gr (John Cherouvim)
Date: Mon, 28 Aug 2017 16:05:43 +0300
Subject: how can I read old varnish logs?
In-Reply-To: <CABoVN9CNPGGpF52Yw+eCc4Ut=79a-SSMwdrUUJ5VJ8ncdMwPrw@mail.gmail.com>
References: <59A3C491.2030504@eworx.gr>
 <CABoVN9C=pONCej+rtk9CxczduhqQzuMpwEo0+esSssuBZeaxfA@mail.gmail.com>
 <59A40A05.4040308@eworx.gr>
 <CABoVN9CNPGGpF52Yw+eCc4Ut=79a-SSMwdrUUJ5VJ8ncdMwPrw@mail.gmail.com>
Message-ID: <59A41527.8080402@eworx.gr>

 > 00000000  12 00 00 4c 63 b6 4e 40  72 65 71 20 35 31 35 38  
|...Lc.N at req 5158|

Dridi Boukelmoune wrote:
> On Mon, Aug 28, 2017 at 2:18 PM, John Cherouvim <jc at eworx.gr> wrote:
>   
>>> What is the output of `file some_extracted_file`?
>>>       
>> It's "data".
>>     
>
> Ouch, what is the output of `hexdump -C some_extracted_file | head -1` then?
>
> Dridi
>
>   




From dridi at varni.sh  Mon Aug 28 13:50:55 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 28 Aug 2017 15:50:55 +0200
Subject: how can I read old varnish logs?
In-Reply-To: <59A41527.8080402@eworx.gr>
References: <59A3C491.2030504@eworx.gr>
 <CABoVN9C=pONCej+rtk9CxczduhqQzuMpwEo0+esSssuBZeaxfA@mail.gmail.com>
 <59A40A05.4040308@eworx.gr>
 <CABoVN9CNPGGpF52Yw+eCc4Ut=79a-SSMwdrUUJ5VJ8ncdMwPrw@mail.gmail.com>
 <59A41527.8080402@eworx.gr>
Message-ID: <CABoVN9BQwn3cGx5NNvjEUeZta0KWF5Eb+jOJo7oC22RNZzUOUA@mail.gmail.com>

On Mon, Aug 28, 2017 at 3:05 PM, John Cherouvim <jc at eworx.gr> wrote:
>> 00000000  12 00 00 4c 63 b6 4e 40  72 65 71 20 35 31 35 38  |...Lc.N at req
>> 5158|

This is weird, you are missing 4 magic bytes at the beginning of the
file. It could be that the logrotate integration is fundamentally
broken.

Does this work for you?

    $ (printf 'VSL\0' | cat - some_extracted_file) >some_fixed_file
    $ varnishlog -r some_fixed_file

Dridi



From jc at eworx.gr  Mon Aug 28 14:39:00 2017
From: jc at eworx.gr (John Cherouvim)
Date: Mon, 28 Aug 2017 17:39:00 +0300
Subject: how can I read old varnish logs?
In-Reply-To: <CABoVN9BQwn3cGx5NNvjEUeZta0KWF5Eb+jOJo7oC22RNZzUOUA@mail.gmail.com>
References: <59A3C491.2030504@eworx.gr>
 <CABoVN9C=pONCej+rtk9CxczduhqQzuMpwEo0+esSssuBZeaxfA@mail.gmail.com>
 <59A40A05.4040308@eworx.gr>
 <CABoVN9CNPGGpF52Yw+eCc4Ut=79a-SSMwdrUUJ5VJ8ncdMwPrw@mail.gmail.com>
 <59A41527.8080402@eworx.gr>
 <CABoVN9BQwn3cGx5NNvjEUeZta0KWF5Eb+jOJo7oC22RNZzUOUA@mail.gmail.com>
Message-ID: <59A42B04.5010506@eworx.gr>

Yes, that worked. I am sorry for the confusion but, yes, it seems that 
my modified logrotate configuration broke the log:

 > /var/log/varnish/varnish.log {
 >     compress
 >     compresscmd /bin/gzip
 >     uncompresscmd /bin/gunzip
 >     compressext gz
 >     dateext
 >     dateformat .%Y-%m-%d.
 >     daily
 >     rotate 7
 >     ifempty
 >     missingok
 >     copytruncate
 > }


Will fix that. Thanks for your time.




Dridi Boukelmoune wrote:
> On Mon, Aug 28, 2017 at 3:05 PM, John Cherouvim <jc at eworx.gr> wrote:
>   
>>> 00000000  12 00 00 4c 63 b6 4e 40  72 65 71 20 35 31 35 38  |...Lc.N at req
>>> 5158|
>>>       
>
> This is weird, you are missing 4 magic bytes at the beginning of the
> file. It could be that the logrotate integration is fundamentally
> broken.
>
> Does this work for you?
>
>     $ (printf 'VSL\0' | cat - some_extracted_file) >some_fixed_file
>     $ varnishlog -r some_fixed_file
>
> Dridi
>   




From dridi at varni.sh  Mon Aug 28 15:06:27 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 28 Aug 2017 17:06:27 +0200
Subject: how can I read old varnish logs?
In-Reply-To: <59A42B04.5010506@eworx.gr>
References: <59A3C491.2030504@eworx.gr>
 <CABoVN9C=pONCej+rtk9CxczduhqQzuMpwEo0+esSssuBZeaxfA@mail.gmail.com>
 <59A40A05.4040308@eworx.gr>
 <CABoVN9CNPGGpF52Yw+eCc4Ut=79a-SSMwdrUUJ5VJ8ncdMwPrw@mail.gmail.com>
 <59A41527.8080402@eworx.gr>
 <CABoVN9BQwn3cGx5NNvjEUeZta0KWF5Eb+jOJo7oC22RNZzUOUA@mail.gmail.com>
 <59A42B04.5010506@eworx.gr>
Message-ID: <CABoVN9AhoEtTciyr0kQodP_SbQBvO77d5pFe9nuQagM5VP1b8g@mail.gmail.com>

On Mon, Aug 28, 2017 at 4:39 PM, John Cherouvim <jc at eworx.gr> wrote:
> Yes, that worked. I am sorry for the confusion but, yes, it seems that my
> modified logrotate configuration broke the log:
>
>> /var/log/varnish/varnish.log {
>>     compress
>>     compresscmd /bin/gzip
>>     uncompresscmd /bin/gunzip
>>     compressext gz
>>     dateext
>>     dateformat .%Y-%m-%d.
>>     daily
>>     rotate 7
>>     ifempty
>>     missingok
>>     copytruncate
>> }
>
>
> Will fix that. Thanks for your time.

I'm not sure _your_ logrotate config is to blame, but glad to see it
worked out. I suppose this is something off, you are not sending the
SIGHUP signal (see man varnishlog).

Cheers



From olivier.hanesse at gmail.com  Tue Aug 29 16:19:13 2017
From: olivier.hanesse at gmail.com (Olivier Hanesse)
Date: Tue, 29 Aug 2017 18:19:13 +0200
Subject: Varnish Lurker is getting slower / Ban lists keeps increasing
Message-ID: <CABx4GKrZruf_oCtP_pJjkLFqYhM=htd8V_Jo0WieJgCV=Y9mjA@mail.gmail.com>

Hello,

We are running Varnish 5.1.3 revision 05c5ac6b9.

During business hours (peak traffic is around 300 req/s ), our ban list
keeps increasing to reach 100K objects (and sometimes more).
Ban traffic is around 4-5 ban/s during this period.  We are using only
"smart bans".

We are pushing varnishstat output to Graphite, and  we have noticed than
the lurker either stop working for a long period of time or is getting
really slow.

Here is a screenshot where you can see that the number of deleted BANs by
the lurker after 9am is decreasing and the BAN list keeps increasing (scale
is log base 10).
The blue bar shows that the lurker didn't delete any ban during 12:50 and
16:30

https://pasteboard.co/GHUPY6t.png

After business hours, everything returns to normal.

I've already tried to change (increase/decrease) both "ban_lurker_batch"
and "ban_lurker_sleep" parameters. Same things.

I don't know how I can get more lurker logs to debug this.
Is there a way to make it more aggressive ?

Any ideas ?

Regards

Olivier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170829/df108b2c/attachment.html>

From dridi at varni.sh  Tue Aug 29 17:04:54 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 29 Aug 2017 19:04:54 +0200
Subject: Varnish Lurker is getting slower / Ban lists keeps increasing
In-Reply-To: <CABx4GKrZruf_oCtP_pJjkLFqYhM=htd8V_Jo0WieJgCV=Y9mjA@mail.gmail.com>
References: <CABx4GKrZruf_oCtP_pJjkLFqYhM=htd8V_Jo0WieJgCV=Y9mjA@mail.gmail.com>
Message-ID: <CABoVN9D15wkK4AvgjZ2W7pdD+U+qh9Vu6anribZV-Pm7LmNKkg@mail.gmail.com>

> I've already tried to change (increase/decrease) both "ban_lurker_batch" and
> "ban_lurker_sleep" parameters. Same things.
>
> I don't know how I can get more lurker logs to debug this.
> Is there a way to make it more aggressive ?
>
> Any ideas ?

Did you look at the ban_cutoff parameter?

I think it landed in the 5.1 series.

Dridi



From olivier.hanesse at gmail.com  Wed Aug 30 06:47:41 2017
From: olivier.hanesse at gmail.com (Olivier Hanesse)
Date: Wed, 30 Aug 2017 08:47:41 +0200
Subject: Varnish Lurker is getting slower / Ban lists keeps increasing
In-Reply-To: <CABoVN9D15wkK4AvgjZ2W7pdD+U+qh9Vu6anribZV-Pm7LmNKkg@mail.gmail.com>
References: <CABx4GKrZruf_oCtP_pJjkLFqYhM=htd8V_Jo0WieJgCV=Y9mjA@mail.gmail.com>
 <CABoVN9D15wkK4AvgjZ2W7pdD+U+qh9Vu6anribZV-Pm7LmNKkg@mail.gmail.com>
Message-ID: <CABx4GKquYB1RPJFWNsv+NDCgJAdZSYsVhWyBz7sw9tJfj6FJWA@mail.gmail.com>

Hello Dridi,

Thanks for your reply.
I must admit I didn't understand perfectly the description of this option.

What will happen when the ban list hits the size of bans defined in
ban_cutoff value ?

Will the ban list be truncated and all the previous (oldest ?) bans will be
simply deleted without purging the cache ?
Or will varnish suspend other regular requests (increasing the lurker
priority thread , something like that ?) until the ban list size is under
this value ?

Regards

Olivier


2017-08-29 19:04 GMT+02:00 Dridi Boukelmoune <dridi at varni.sh>:

> > I've already tried to change (increase/decrease) both "ban_lurker_batch"
> and
> > "ban_lurker_sleep" parameters. Same things.
> >
> > I don't know how I can get more lurker logs to debug this.
> > Is there a way to make it more aggressive ?
> >
> > Any ideas ?
>
> Did you look at the ban_cutoff parameter?
>
> I think it landed in the 5.1 series.
>
> Dridi
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170830/bbc07ddf/attachment.html>

From jpotter-varnish at codepuppy.com  Wed Aug 30 07:24:04 2017
From: jpotter-varnish at codepuppy.com (jpotter-varnish at codepuppy.com)
Date: Wed, 30 Aug 2017 03:24:04 -0400
Subject: Limit requests to backend based on requested domain?
Message-ID: <237FC06B-BE63-4856-ACE5-A3B06886BE48@codepuppy.com>


Hi,

Is there a way to limit the number of concurrent requests sent to a backend based on the requested domain name? Or more broadly, based on some key/value?

I know I can limit the number of requests to a backend with max requests, but our backend services multiple domains, and I?m wanting to limit the number of requests per domain (so that too many requests to one doesn?t cut off resources for others). I?ve thought of maybe doing this by defining multiple backends (pointed at the same server), but then I worry that the health checks we run would stack up (i.e. if there are a 100 domains, going from 1 backend to 100 backends would presumable increase the health checks 100 times).

Thanks,
Jeff



From dridi at varni.sh  Wed Aug 30 08:16:37 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 30 Aug 2017 10:16:37 +0200
Subject: Limit requests to backend based on requested domain?
In-Reply-To: <237FC06B-BE63-4856-ACE5-A3B06886BE48@codepuppy.com>
References: <237FC06B-BE63-4856-ACE5-A3B06886BE48@codepuppy.com>
Message-ID: <CABoVN9CHX1v-3HTuwcm5Kd9ipLu2P710WVNJeyjkPvjTp2xg+g@mail.gmail.com>

On Wed, Aug 30, 2017 at 9:24 AM,  <jpotter-varnish at codepuppy.com> wrote:
>
> Hi,
>
> Is there a way to limit the number of concurrent requests sent to a backend based on the requested domain name? Or more broadly, based on some key/value?

Have a look at the vsthrottle VMOD, it's usually used for client
requests but you can probably use it on the backend side too.

https://github.com/varnish/varnish-modules
https://github.com/varnish/varnish-modules/blob/master/docs/vmod_vsthrottle.rst

Dridi



From guillaume at varnish-software.com  Wed Aug 30 08:22:03 2017
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 30 Aug 2017 10:22:03 +0200
Subject: Limit requests to backend based on requested domain?
In-Reply-To: <237FC06B-BE63-4856-ACE5-A3B06886BE48@codepuppy.com>
References: <237FC06B-BE63-4856-ACE5-A3B06886BE48@codepuppy.com>
Message-ID: <CAJ6ZYQz58Rp_iiq9LeOM-UiZ3CvrgPOpu_YSohZmt-RN_5mODg@mail.gmail.com>

Hi Jeff,

have a look at vmod_vsthrottle in varnish-modules, that should do it.

-- 
Guillaume Quintard

On Wed, Aug 30, 2017 at 9:24 AM, <jpotter-varnish at codepuppy.com> wrote:

>
> Hi,
>
> Is there a way to limit the number of concurrent requests sent to a
> backend based on the requested domain name? Or more broadly, based on some
> key/value?
>
> I know I can limit the number of requests to a backend with max requests,
> but our backend services multiple domains, and I?m wanting to limit the
> number of requests per domain (so that too many requests to one doesn?t cut
> off resources for others). I?ve thought of maybe doing this by defining
> multiple backends (pointed at the same server), but then I worry that the
> health checks we run would stack up (i.e. if there are a 100 domains, going
> from 1 backend to 100 backends would presumable increase the health checks
> 100 times).
>
> Thanks,
> Jeff
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170830/03359740/attachment.html>

From olivier.hanesse at gmail.com  Wed Aug 30 09:44:50 2017
From: olivier.hanesse at gmail.com (Olivier Hanesse)
Date: Wed, 30 Aug 2017 11:44:50 +0200
Subject: Varnish Lurker is getting slower / Ban lists keeps increasing
In-Reply-To: <CABx4GKquYB1RPJFWNsv+NDCgJAdZSYsVhWyBz7sw9tJfj6FJWA@mail.gmail.com>
References: <CABx4GKrZruf_oCtP_pJjkLFqYhM=htd8V_Jo0WieJgCV=Y9mjA@mail.gmail.com>
 <CABoVN9D15wkK4AvgjZ2W7pdD+U+qh9Vu6anribZV-Pm7LmNKkg@mail.gmail.com>
 <CABx4GKquYB1RPJFWNsv+NDCgJAdZSYsVhWyBz7sw9tJfj6FJWA@mail.gmail.com>
Message-ID: <CABx4GKpYT6zqyasWtJMsszLEKcUn_6zon-uYpDRmttS4oBZKtg@mail.gmail.com>

Hello,

Last night after your reply, I put a ban_cutoff value of 18500 according to
the definition (50ms of latency, 370K/s ban.lurker.tested) (I've restarted
varnish, "varnishadm param.show ban_cutoff" shows the right value)

This morning, nothing has changed :  ban lists is increasing (well over
18500).

https://pasteboard.co/GI1K682.png

Regards

Olivier





2017-08-30 8:47 GMT+02:00 Olivier Hanesse <olivier.hanesse at gmail.com>:

> Hello Dridi,
>
> Thanks for your reply.
> I must admit I didn't understand perfectly the description of this option.
>
> What will happen when the ban list hits the size of bans defined in
> ban_cutoff value ?
>
> Will the ban list be truncated and all the previous (oldest ?) bans will
> be simply deleted without purging the cache ?
> Or will varnish suspend other regular requests (increasing the lurker
> priority thread , something like that ?) until the ban list size is under
> this value ?
>
> Regards
>
> Olivier
>
>
> 2017-08-29 19:04 GMT+02:00 Dridi Boukelmoune <dridi at varni.sh>:
>
>> > I've already tried to change (increase/decrease) both
>> "ban_lurker_batch" and
>> > "ban_lurker_sleep" parameters. Same things.
>> >
>> > I don't know how I can get more lurker logs to debug this.
>> > Is there a way to make it more aggressive ?
>> >
>> > Any ideas ?
>>
>> Did you look at the ban_cutoff parameter?
>>
>> I think it landed in the 5.1 series.
>>
>> Dridi
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170830/1fe71c4b/attachment.html>

From slink at schokola.de  Wed Aug 30 12:37:09 2017
From: slink at schokola.de (Nils Goroll)
Date: Wed, 30 Aug 2017 14:37:09 +0200
Subject: Varnish Lurker is getting slower / Ban lists keeps increasing
In-Reply-To: <CABx4GKquYB1RPJFWNsv+NDCgJAdZSYsVhWyBz7sw9tJfj6FJWA@mail.gmail.com>
References: <CABx4GKrZruf_oCtP_pJjkLFqYhM=htd8V_Jo0WieJgCV=Y9mjA@mail.gmail.com>
 <CABoVN9D15wkK4AvgjZ2W7pdD+U+qh9Vu6anribZV-Pm7LmNKkg@mail.gmail.com>
 <CABx4GKquYB1RPJFWNsv+NDCgJAdZSYsVhWyBz7sw9tJfj6FJWA@mail.gmail.com>
Message-ID: <e5acdaa0-589f-12dd-7f3b-b1465ade01a3@schokola.de>

Hi Olivier,

I'm responding to the last two emails from you in one go

On 30/08/17 08:47, Olivier Hanesse wrote:
> What will happen when the ban list hits the size of bans defined in ban_cutoff
> value ?

The ban lurker still works the list of bans as before, but when having reached
the <ban_cutoff>th ban, we kill all objects hanging off these bans without
testing the ban condition.

This way, actively used objects (which get tested against the ban list at
request time and will end up hanging off some ban near the top of the ban list)
will not get killed, but rather only those which were least frequently accessed
(iow the long tail).

On 30/08/17 11:44, Olivier Hanesse wrote:
> Last night after your reply, I put a ban_cutoff value of 18500 according to
> the definition (50ms of latency, 370K/s ban.lurker.tested) (I've restarted
> varnish, "varnishadm param.show ban_cutoff" shows the right value)
>
> This morning, nothing has changed :  ban lists is increasing (well over
> 18500).

One obvious explanation would be that the lurker had not got to the cutoff value.

But I wonder what exactly you are measuring here. In your first email you wrote

On 29/08/17 18:19, Olivier Hanesse wrote:
> our ban list keeps increasing to reach 100K objects (and sometimes more).

This makes me guess that maybe you'd be graphing the number of objects hanging
off the bans. Quick reminder:

* the second column in the varnishadm ban.list output is the number of
  objects associated with this ban (objects, for which this ban has
  last been tested)

* what the ban_cutoff parameter is limiting is the number of bans
  (that would be varnishadm ban.list | wc -l minus 2)

So can you please double check that you are graphing the latter and not the
former for ban.list?

If you'd actually be graphing the former, then we don't have a problem as this
will just be the total number of objects in your cache.

Thanks, Nils




From olivier.hanesse at gmail.com  Wed Aug 30 13:05:50 2017
From: olivier.hanesse at gmail.com (Olivier Hanesse)
Date: Wed, 30 Aug 2017 15:05:50 +0200
Subject: Varnish Lurker is getting slower / Ban lists keeps increasing
In-Reply-To: <e5acdaa0-589f-12dd-7f3b-b1465ade01a3@schokola.de>
References: <CABx4GKrZruf_oCtP_pJjkLFqYhM=htd8V_Jo0WieJgCV=Y9mjA@mail.gmail.com>
 <CABoVN9D15wkK4AvgjZ2W7pdD+U+qh9Vu6anribZV-Pm7LmNKkg@mail.gmail.com>
 <CABx4GKquYB1RPJFWNsv+NDCgJAdZSYsVhWyBz7sw9tJfj6FJWA@mail.gmail.com>
 <e5acdaa0-589f-12dd-7f3b-b1465ade01a3@schokola.de>
Message-ID: <CABx4GKoxs3FgT=2O292cmQ=3-=J5F0rKGEyL9k9s3UtvpOF_0g@mail.gmail.com>

Hello Nils,

Thanks for your reply. Now I understand better the definition of this
parameter (cache efficiency vs response time).

I've checked and I am really graphing the size of the ban list (using the
value "MAIN.bans" from varnishstat ). This value is the same that the
"varnishadm
ban.list | wc -l" output.

I've just see that there is a "new" value in varnishstat
 "MAIN.bans_lurker_obj_killed_cutoff". I will add this value to my
monitoring plugin.

Olivier

2017-08-30 14:37 GMT+02:00 Nils Goroll <slink at schokola.de>:

> Hi Olivier,
>
> I'm responding to the last two emails from you in one go
>
> On 30/08/17 08:47, Olivier Hanesse wrote:
> > What will happen when the ban list hits the size of bans defined in
> ban_cutoff
> > value ?
>
> The ban lurker still works the list of bans as before, but when having
> reached
> the <ban_cutoff>th ban, we kill all objects hanging off these bans without
> testing the ban condition.
>
> This way, actively used objects (which get tested against the ban list at
> request time and will end up hanging off some ban near the top of the ban
> list)
> will not get killed, but rather only those which were least frequently
> accessed
> (iow the long tail).
>
> On 30/08/17 11:44, Olivier Hanesse wrote:
> > Last night after your reply, I put a ban_cutoff value of 18500 according
> to
> > the definition (50ms of latency, 370K/s ban.lurker.tested) (I've
> restarted
> > varnish, "varnishadm param.show ban_cutoff" shows the right value)
> >
> > This morning, nothing has changed :  ban lists is increasing (well over
> > 18500).
>
> One obvious explanation would be that the lurker had not got to the cutoff
> value.
>
> But I wonder what exactly you are measuring here. In your first email you
> wrote
>
> On 29/08/17 18:19, Olivier Hanesse wrote:
> > our ban list keeps increasing to reach 100K objects (and sometimes more).
>
> This makes me guess that maybe you'd be graphing the number of objects
> hanging
> off the bans. Quick reminder:
>
> * the second column in the varnishadm ban.list output is the number of
>   objects associated with this ban (objects, for which this ban has
>   last been tested)
>
> * what the ban_cutoff parameter is limiting is the number of bans
>   (that would be varnishadm ban.list | wc -l minus 2)
>
> So can you please double check that you are graphing the latter and not the
> former for ban.list?
>
> If you'd actually be graphing the former, then we don't have a problem as
> this
> will just be the total number of objects in your cache.
>
> Thanks, Nils
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170830/94b0648e/attachment.html>

From slink at schokola.de  Wed Aug 30 13:14:13 2017
From: slink at schokola.de (Nils Goroll)
Date: Wed, 30 Aug 2017 15:14:13 +0200
Subject: Varnish Lurker is getting slower / Ban lists keeps increasing
In-Reply-To: <CABx4GKpYT6zqyasWtJMsszLEKcUn_6zon-uYpDRmttS4oBZKtg@mail.gmail.com>
References: <CABx4GKrZruf_oCtP_pJjkLFqYhM=htd8V_Jo0WieJgCV=Y9mjA@mail.gmail.com>
 <CABoVN9D15wkK4AvgjZ2W7pdD+U+qh9Vu6anribZV-Pm7LmNKkg@mail.gmail.com>
 <CABx4GKquYB1RPJFWNsv+NDCgJAdZSYsVhWyBz7sw9tJfj6FJWA@mail.gmail.com>
 <CABx4GKpYT6zqyasWtJMsszLEKcUn_6zon-uYpDRmttS4oBZKtg@mail.gmail.com>
Message-ID: <12ca81a6-44fa-bfa0-046d-b051cabc92b6@schokola.de>


On 30/08/17 11:44, Olivier Hanesse wrote:
> 50ms of latency, 370K/s ban.lurker.tested

BTW, the value to be used for the calculation is rate(bans_lurker_tests_tested)
not rate(bans_lurker_tested)

I've also just improved the documentation of ban_cutoff in master.



From olivier.hanesse at gmail.com  Wed Aug 30 13:41:44 2017
From: olivier.hanesse at gmail.com (Olivier Hanesse)
Date: Wed, 30 Aug 2017 15:41:44 +0200
Subject: Varnish Lurker is getting slower / Ban lists keeps increasing
In-Reply-To: <12ca81a6-44fa-bfa0-046d-b051cabc92b6@schokola.de>
References: <CABx4GKrZruf_oCtP_pJjkLFqYhM=htd8V_Jo0WieJgCV=Y9mjA@mail.gmail.com>
 <CABoVN9D15wkK4AvgjZ2W7pdD+U+qh9Vu6anribZV-Pm7LmNKkg@mail.gmail.com>
 <CABx4GKquYB1RPJFWNsv+NDCgJAdZSYsVhWyBz7sw9tJfj6FJWA@mail.gmail.com>
 <CABx4GKpYT6zqyasWtJMsszLEKcUn_6zon-uYpDRmttS4oBZKtg@mail.gmail.com>
 <12ca81a6-44fa-bfa0-046d-b051cabc92b6@schokola.de>
Message-ID: <CABx4GKoc-jehC3B=bCtgfGniXW8ay=VDXFObJTJn5U-BEkeg7Q@mail.gmail.com>

Ok, so I could upgrade the cutoff parameter to 70K (bans_lurker_tests_tested
is around 1.4M)

I will keep it at 18500 to debug this ban list behaviour.

2017-08-30 15:14 GMT+02:00 Nils Goroll <slink at schokola.de>:

>
> On 30/08/17 11:44, Olivier Hanesse wrote:
> > 50ms of latency, 370K/s ban.lurker.tested
>
> BTW, the value to be used for the calculation is
> rate(bans_lurker_tests_tested)
> not rate(bans_lurker_tested)
>
> I've also just improved the documentation of ban_cutoff in master.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170830/14e94df5/attachment.html>

From jpotter-varnish at codepuppy.com  Wed Aug 30 16:28:34 2017
From: jpotter-varnish at codepuppy.com (jpotter-varnish at codepuppy.com)
Date: Wed, 30 Aug 2017 12:28:34 -0400
Subject: Limit requests to backend based on requested domain?
In-Reply-To: <CAJ6ZYQz58Rp_iiq9LeOM-UiZ3CvrgPOpu_YSohZmt-RN_5mODg@mail.gmail.com>
References: <237FC06B-BE63-4856-ACE5-A3B06886BE48@codepuppy.com>
 <CAJ6ZYQz58Rp_iiq9LeOM-UiZ3CvrgPOpu_YSohZmt-RN_5mODg@mail.gmail.com>
Message-ID: <5E7EEF9A-7912-48B9-BBE4-7CBA8045E929@codepuppy.com>


Thanks, Guillaume and Dridi ? I?m not seeing a way to limit concurrent requests however. We?re using vsthrottle to limit the total number of requests to a domain ? i.e. ?100 requests in 10 seconds; 600 requests in 5 minutes?; but that doesn?t guard against the case of someone?s PHP script blocking (say, external API request from some wordpress plugin), and when that happens, those PHP workers pile up pretty fast. Normally, requests to the backend at the rate of something like 100 per second in total would be reasonable, but in the case of bad backend PHP code in one domain, well; it?d resource starve workers for all the domains on that backend server (shared hosting).

Perhaps this is something that?s similar enough to what vsthrottle does that I should fork it and figure out how to implement it as a counter instead?

Thanks,
Jeff

> On Aug 30, 2017, at 4:22 AM, Guillaume Quintard <guillaume at varnish-software.com> wrote:
> 
> Hi Jeff,
> 
> have a look at vmod_vsthrottle in varnish-modules, that should do it.
> 
> -- 
> Guillaume Quintard
> 
> On Wed, Aug 30, 2017 at 9:24 AM, <jpotter-varnish at codepuppy.com <mailto:jpotter-varnish at codepuppy.com>> wrote:
> 
> Hi,
> 
> Is there a way to limit the number of concurrent requests sent to a backend based on the requested domain name? Or more broadly, based on some key/value?
> 
> I know I can limit the number of requests to a backend with max requests, but our backend services multiple domains, and I?m wanting to limit the number of requests per domain (so that too many requests to one doesn?t cut off resources for others). I?ve thought of maybe doing this by defining multiple backends (pointed at the same server), but then I worry that the health checks we run would stack up (i.e. if there are a 100 domains, going from 1 backend to 100 backends would presumable increase the health checks 100 times).
> 
> Thanks,
> Jeff
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org <mailto:varnish-misc at varnish-cache.org>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20170830/037a1df6/attachment.html>

From dridi at varni.sh  Wed Aug 30 16:55:34 2017
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 30 Aug 2017 18:55:34 +0200
Subject: Limit requests to backend based on requested domain?
In-Reply-To: <5E7EEF9A-7912-48B9-BBE4-7CBA8045E929@codepuppy.com>
References: <237FC06B-BE63-4856-ACE5-A3B06886BE48@codepuppy.com>
 <CAJ6ZYQz58Rp_iiq9LeOM-UiZ3CvrgPOpu_YSohZmt-RN_5mODg@mail.gmail.com>
 <5E7EEF9A-7912-48B9-BBE4-7CBA8045E929@codepuppy.com>
Message-ID: <CABoVN9DFNgOFtyJcLjdXKLdyrgNZNr8ZdXLZbpdKzUOtdWWLGQ@mail.gmail.com>

On Wed, Aug 30, 2017 at 6:28 PM,  <jpotter-varnish at codepuppy.com> wrote:
>
> Thanks, Guillaume and Dridi ? I?m not seeing a way to limit concurrent
> requests however. We?re using vsthrottle to limit the total number of
> requests to a domain ? i.e. ?100 requests in 10 seconds; 600 requests in 5
> minutes?; but that doesn?t guard against the case of someone?s PHP script
> blocking (say, external API request from some wordpress plugin), and when
> that happens, those PHP workers pile up pretty fast. Normally, requests to
> the backend at the rate of something like 100 per second in total would be
> reasonable, but in the case of bad backend PHP code in one domain, well;
> it?d resource starve workers for all the domains on that backend server
> (shared hosting).

Correct.

> Perhaps this is something that?s similar enough to what vsthrottle does that
> I should fork it and figure out how to implement it as a counter instead?

Browse the issue tracker, we may have discussed that at some point but
I'm not sure. Please also have a look at the old repositories before
it got bundled to varnish-modules. We are close to the next Varnish
release and I can't spend much time on this.

https://github.com/varnish/libvmod-vsthrottle

The problem with explicit inc/dec operations is that they may not be
balanced, so you'll need a PRIV_TASK to keep track of that and clear
the state if the transaction ends with a positive balance.

Dridi