From fgtham at gmail.com  Thu Dec  1 09:47:01 2016
From: fgtham at gmail.com (Florian Tham)
Date: Thu, 1 Dec 2016 10:47:01 +0100
Subject: Varnish5 PROXY protocol and health probes
Message-ID: <CAKKwObZUT0=TM_=Oa2vToLMnWiq+fyjtRUymBLWY7ThJDFRTGA@mail.gmail.com>

Hi,

I'm using varnish-5.0.0 with PROXY protocol enabled on the backends
thanks to the new .proxy_header attribute. This works great as long as
the backends don't have health probes defined. When a backend has both
a .proxy_header attribute and a probe defined, varnish fails to
properly detect the backend health state and marks the backend as
sick.

The problem seems to be that varnish does not use the PROXY protocol
when sending health probes. If the backend correctly implements the
PROXY protocol, it is required to drop the connection, as it cannot
identify it as valid PROXY protocol v1 nor v2. Thus, from varnish's
pov the backend is unreachable.

I think it would be pretty useful to have health probes handle the
PROXY protocol as well. What do you think? Are there any plans to
enhance varnish accordingly?

Best regards,

Florian



From phk at phk.freebsd.dk  Thu Dec  1 09:55:08 2016
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 01 Dec 2016 09:55:08 +0000
Subject: Varnish5 PROXY protocol and health probes
In-Reply-To: <CAKKwObZUT0=TM_=Oa2vToLMnWiq+fyjtRUymBLWY7ThJDFRTGA@mail.gmail.com>
References: <CAKKwObZUT0=TM_=Oa2vToLMnWiq+fyjtRUymBLWY7ThJDFRTGA@mail.gmail.com>
Message-ID: <22342.1480586108@critter.freebsd.dk>

--------
In message <CAKKwObZUT0=TM_=Oa2vToLMnWiq+fyjtRUymBLWY7ThJDFRTGA at mail.gmail.com>
, Florian Tham writes:

That's a bug, please open a ticket:

	https://github.com/varnishcache/varnish-cache


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From fgtham at gmail.com  Thu Dec  1 10:15:25 2016
From: fgtham at gmail.com (Florian Tham)
Date: Thu, 1 Dec 2016 11:15:25 +0100
Subject: Varnish5 PROXY protocol and health probes
In-Reply-To: <22342.1480586108@critter.freebsd.dk>
References: <CAKKwObZUT0=TM_=Oa2vToLMnWiq+fyjtRUymBLWY7ThJDFRTGA@mail.gmail.com>
 <22342.1480586108@critter.freebsd.dk>
Message-ID: <CAKKwObahfpXDWpKYt31sYMhu9hCLxLyqnZNihdZM7Z6=R4WcmQ@mail.gmail.com>

For reference, ticket is here:
https://github.com/varnishcache/varnish-cache/issues/2151

Best regards,

Florian

On Thu, Dec 1, 2016 at 10:55 AM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> --------
> In message <CAKKwObZUT0=TM_=Oa2vToLMnWiq+fyjtRUymBLWY7ThJDFRTGA at mail.gmail.com>
> , Florian Tham writes:
>
> That's a bug, please open a ticket:
>
>         https://github.com/varnishcache/varnish-cache
>
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.



From stardothosting at gmail.com  Thu Dec  1 22:23:19 2016
From: stardothosting at gmail.com (Star Dot)
Date: Thu, 1 Dec 2016 17:23:19 -0500
Subject: Is Varnish 4 able to follow a redirect and cache the destination?
Message-ID: <CAFu1TJYFayq=0xsC7WSZ2VhQmgQgn=VCnhw=2TvLg+xYUnYw8Q@mail.gmail.com>

Hello!

I am dealing with Varnish 4 piping requests to a backend server. The
backend server processes the request and redirects to Amazon S3 to serve
the actual file.

So in short :

Request -> Varnish -> Nginx 302 Redirect -> Amazon S3 file

Varnish is happily caching the 302 response only, but I'm curious if I can
somehow follow the redirect and cache the destination file completely? This
will alleviate load off the nginx server obviously.

I've seen several topics dance around this issue but I'm curious if osmeone
can point me in the right direction!

Thanks
--


StackStar Managed Hosting Services : https://www.stackstar.com
Shift8 Web Design in Toronto : https://www.shift8web.ca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161201/a0fd55db/attachment.html>

From guillaume at varnish-software.com  Thu Dec  1 22:57:13 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Thu, 1 Dec 2016 23:57:13 +0100
Subject: Is Varnish 4 able to follow a redirect and cache the destination?
In-Reply-To: <CAFu1TJYFayq=0xsC7WSZ2VhQmgQgn=VCnhw=2TvLg+xYUnYw8Q@mail.gmail.com>
References: <CAFu1TJYFayq=0xsC7WSZ2VhQmgQgn=VCnhw=2TvLg+xYUnYw8Q@mail.gmail.com>
Message-ID: <CAJ6ZYQzA81+Xpnm0BCBWroYR2mU7D5Em4nwTFxD=2rtcuj6h7g@mail.gmail.com>

It can, you will have to play a bit with the retry/restart mechanics after
changing and cleaning bereq.url.

On Dec 1, 2016 23:49, "Star Dot" <stardothosting at gmail.com> wrote:

> Hello!
>
> I am dealing with Varnish 4 piping requests to a backend server. The
> backend server processes the request and redirects to Amazon S3 to serve
> the actual file.
>
> So in short :
>
> Request -> Varnish -> Nginx 302 Redirect -> Amazon S3 file
>
> Varnish is happily caching the 302 response only, but I'm curious if I can
> somehow follow the redirect and cache the destination file completely? This
> will alleviate load off the nginx server obviously.
>
> I've seen several topics dance around this issue but I'm curious if
> osmeone can point me in the right direction!
>
> Thanks
> --
>
>
> StackStar Managed Hosting Services : https://www.stackstar.com
> Shift8 Web Design in Toronto : https://www.shift8web.ca
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161201/374ccafd/attachment.html>

From peterloron at gmail.com  Fri Dec  2 08:11:37 2016
From: peterloron at gmail.com (Peter Loron)
Date: Fri, 2 Dec 2016 00:11:37 -0800
Subject: Grace mode with 404 responses?
Message-ID: <57811D62-4862-4953-A5D1-A4B4D12BF631@gmail.com>

Hello, folks. I?m testing a Varnish 4.1.1 setup. Specifically I?m working with shielding a client service from an unreliable remote service by proxying and caching responses from the remote system.

Grace mode is working fine for situations where the remote server is unresponsive (e.g. apache turned off), but does not seem to work when a 404 is returned for a resource. I immediately get the 404 on the client when the TTL (not the TTL + grace) is expired.

I?m assuming this is something I?m not doing right in my VCL file, but I can?t figure out what.

Is there a canonical example of how to handle this properly?

Thanks.

-Pete


From apj at mutt.dk  Fri Dec  2 08:51:38 2016
From: apj at mutt.dk (Andreas Plesner)
Date: Fri, 2 Dec 2016 09:51:38 +0100
Subject: Is Varnish 4 able to follow a redirect and cache the destination?
In-Reply-To: <CAFu1TJYFayq=0xsC7WSZ2VhQmgQgn=VCnhw=2TvLg+xYUnYw8Q@mail.gmail.com>
References: <CAFu1TJYFayq=0xsC7WSZ2VhQmgQgn=VCnhw=2TvLg+xYUnYw8Q@mail.gmail.com>
Message-ID: <20161202085138.GG22017@nerd.dk>

On Thu, Dec 01, 2016 at 05:23:19PM -0500, Star Dot wrote:

> Varnish is happily caching the 302 response only, but I'm curious if I can
> somehow follow the redirect and cache the destination file completely? This
> will alleviate load off the nginx server obviously.

Yes. This is what we do:

sub vcl_deliver {
  if ((resp.status == 301) || (resp.status == 302)) {
    set req.url = regsub(resp.http.Location,"^http://[^/]+(.*)","\1");
    return(restart);
  }
}

Handling of the host header (and all the possible backends) is left as an
exercise for the reader.

What this does is that it caches the redirect as well as the destination, thus
sort-of normalizing the request, while still caching the ultimate destination
only once.

-- 
Andreas



From stardothosting at gmail.com  Fri Dec  2 13:36:04 2016
From: stardothosting at gmail.com (Star Dot)
Date: Fri, 2 Dec 2016 08:36:04 -0500
Subject: Is Varnish 4 able to follow a redirect and cache the destination?
In-Reply-To: <20161202085138.GG22017@nerd.dk>
References: <CAFu1TJYFayq=0xsC7WSZ2VhQmgQgn=VCnhw=2TvLg+xYUnYw8Q@mail.gmail.com>
 <20161202085138.GG22017@nerd.dk>
Message-ID: <CAFu1TJa0hs9B05p8JB2O97_WUG0jTxPB9-WXc_8oqXuw=kgR4w@mail.gmail.com>

On Fri, Dec 2, 2016 at 3:51 AM, Andreas Plesner <apj at mutt.dk> wrote:

>
> Yes. This is what we do:
>
> sub vcl_deliver {
>   if ((resp.status == 301) || (resp.status == 302)) {
>     set req.url = regsub(resp.http.Location,"^http://[^/]+(.*)","\1");
>     return(restart);
>   }
> }
>
> Handling of the host header (and all the possible backends) is left as an
> exercise for the reader.
>
> What this does is that it caches the redirect as well as the destination,
> thus
> sort-of normalizing the request, while still caching the ultimate
> destination
> only once.
>
>

Tentatively your rules work for me. Is there any way for varnish to cache
the 302 redirect if its on a completely different host? In my example we
are hitting media.domain.com which hits an nginx server that redirects it
to a completely different URL (S3) that varnish is not caching. After
testing your rule, I see that its definitely skipping a step and providing
the destination URL (S3) to the browser. But that destination is not being
cached.

Is there a way to hide that hostname/url change completely so its
transparent to the end user?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161202/f6f13a53/attachment.html>

From cservin-varnish at cromagnon.com  Fri Dec  2 15:25:12 2016
From: cservin-varnish at cromagnon.com (Craig Servin)
Date: Fri, 02 Dec 2016 09:25:12 -0600
Subject: n_lru_nuked  per stevedore
Message-ID: <2c6c71543b0a5413699b06d13f723bc7@www.cromagnon.com>

Hi All,

Is there a way to get at the number of lru evictions per stevedore?

Thanks,

Craig



From dridi at varni.sh  Fri Dec  2 16:09:05 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 2 Dec 2016 17:09:05 +0100
Subject: n_lru_nuked per stevedore
In-Reply-To: <2c6c71543b0a5413699b06d13f723bc7@www.cromagnon.com>
References: <2c6c71543b0a5413699b06d13f723bc7@www.cromagnon.com>
Message-ID: <CABoVN9CptFQ2S5hv02Z994+7hQ3QVehyn_ebjjwM0=Q73F4Ukg@mail.gmail.com>

On Fri, Dec 2, 2016 at 4:25 PM, Craig Servin
<cservin-varnish at cromagnon.com> wrote:
> Hi All,
>
> Is there a way to get at the number of lru evictions per stevedore?

Hi,

Unfortunately no, although it has been discussed at least once by the
dev team. There's nothing against having them but it's currently not
on the agenda.

You can sort-of get this information from varnishlog, but to get a
result similar to varnishstat you'd need to write something on top to
crunch the data.

Cheers



From admin at beckspaced.com  Fri Dec  2 18:30:01 2016
From: admin at beckspaced.com (Admin Beckspaced)
Date: Fri, 2 Dec 2016 19:30:01 +0100
Subject: varnishncsa logs split per domain
In-Reply-To: <912a8f40-e02b-d578-585c-e3d9b8fdd0a2@beckspaced.com>
References: <d0e759e1-a91c-5632-bac4-c98710839391@beckspaced.com>
 <a3d31652-9a07-45cb-53e1-46c2faaeff48@gurutek.biz>
 <976bff90-7c3e-e9c2-3517-6bc30a937de5@beckspaced.com>
 <f27bdfd8-cbf5-1f8b-df2c-2ab09de426a2@gurutek.biz>
 <4aea1d32-f9b8-ada4-fb9d-532f605ae5da@beckspaced.com>
 <CAP+vvEtzMD4=Nc_kKtMoWaGeTbmtdp181MEyFb_sF78rg0U6rQ@mail.gmail.com>
 <5aa0e368-0121-8542-70bc-27aeae96e4e9@beckspaced.com>
 <CABoVN9AzmgyN1oqHmixKz-i4Y3u+JRLKTs2h0NLbfT8yfhfPPA@mail.gmail.com>
 <d2946717-21ed-5060-c058-24cb43262429@beckspaced.com>
 <CABoVN9DkEGiMKi+=x6W-a5g01XpdZcdRrhRPf-Zq0pC=5spyPA@mail.gmail.com>
 <912a8f40-e02b-d578-585c-e3d9b8fdd0a2@beckspaced.com>
Message-ID: <39257342-24c8-5dd8-9158-6b150de8893a@beckspaced.com>


On 15.11.2016 17:57, Admin Beckspaced wrote:
>
> Am 15.11.2016 um 16:51 schrieb Dridi Boukelmoune:
>>> a bit more hints and info would be nice ;)
>> man vmod_std
>> man varnishncsa
>>
>> That's how much "nice" I'm willing to do :p
>>
>> Dridi
>>
>>
> ok. first I want to say thanks for being nice and pointing me to the 
> man pages.
>
> after a bit of reading I finally found the parts I was looking for:
>
>     import std;
>
>     sub vcl_recv {
>
>     std.log(?myhost:? + regsub(req.http.Host, "^www\.", "") );
>
>     }
>
> in varnishncsa:
>
>      # varnishncsa -F ?%h %l %u %t ?%r? %s %b ?%{Referer}i? 
> ?%{User-agent}i %{VCL_Log:myhost}x?
>
>
> not yet tested but I think this is what Dridi was pointing to?
>
Hello again,

sorry, it has been a while but I just thought to finish the thread I 
started and point to the solution I decided to go with at last.

The question in the beginning was: How can I split the varnishncsa logs 
per domain.

My first thinking was to use the query -q option, e.g. varnishncsa -q 
"ReqHeader ~ '^Host: .*example.com'"

But this approach would end up in a lot varnishncsa instances, as 
pointed out by Andrei, and also the problem with not being able to 
normalize the Request Host header.
Then Dridi pointed me to man vmod_std and using std.log in VCL, which 
was the final bit needed ;)

So here's my current solution:

I run a single instance of varnishncsa with the following params:

VARNISHLOG_PARAMS="-f /etc/varnish/varnishncsa-log-format-string -a -w 
/var/log/varnish/varnish.log"

the varnishncsa-log-format-string is as follows:

%{VCL_Log:myhost}x %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"

at the beginning VCL_Log:key     The value set by std.log("key:value") 
in VCL, more on that later

My varnish sits in front of Apache with 30 something different domains. 
Currently I don't use varnish for all domains, but have setup varnish 
VCL in such a way that I can filter on the domains and decide to cache 
with varnish or just skip caching and pass to the apache backend.

My varnish VCL is based on the varnish boilerplate which I found on the net:

http://verticalprogramming.com/2013/09/15/varnish-virtual-host-boilerplate

So if I decide to cache a particular domain with varnish I can normalize 
the Request Host header in VCL

sub vcl_recv {

     if (req.http.host ~ "somedomain\.com") {

         # tilde ~ uses regex
         if (req.http.host ~ "^(www.)?somedomain\.com$") {

             //normalize the req.http.host
             set req.http.host = regsub(req.http.Host, "^www\.", "");

             std.log("myhost:" + req.http.Host );
...

so this std.log(myhost:somedomian.com) gets picked up by varnishncsa and 
the custom format string, see above.

which then produces a nice & steady varnish.log file with a normalized 
host at the beginning for domains I want to cache and an empty space if 
I don't want to.

then we got the split-logfile from apache:

https://httpd.apache.org/docs/2.4/programs/split-logfile.html

which was exactly made for a setup with the host names at the very 
beginning of the log file. Only thing someone needs to take care of is 
that the logfiles will get created in the directory where the script is 
run. so therefore I created a small bash script wrapper split-logfile.sh 
which first changes to the right working directory:

#!/bin/bash
cd /var/log/varnish
/usr/bin/split-logfile < varnish.log

and on the daily logrotate on the /var/log/varnish/varnish.log I added 
the following:

/var/log/varnish/varnish.log {
     ...
     prerotate
      /var/log/varnish/split-logfile.sh
     endscript
     ...
}

so before the varnish.log gets rotated split-logfile.sh gets called and 
creates the different log files per normalized host and an access.log 
for all the requests without a hostname at the beginning.
After a view logrotate runs the /var/log/varnish/ could look like that:

mydomain.com.log
myotherdomain.com.log
access.log
varnish.log
varnish.log-20161130
varnish.log-20161201
varnish.log-20161202

which finally give me exactly what I wanted! A single instance of 
varnishncsa producing a varnish.log for all domains.
a per domain split via split-logfile.sh on each logrotate run resulting 
in log files per domain ready to use with webalizer, which gets also 
called on prerotate the logfile:

/var/log/varnish/mydomain.com.log {
     ...
     prerotate
      /usr/bin/webalizer -qc /etc/webalizer/mydomain.conf
     endscript
     ...
}

perhaps this might help someone here looking for something similar?

thanks & greetings
becki





From guillaume at varnish-software.com  Fri Dec  2 19:49:35 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 2 Dec 2016 20:49:35 +0100
Subject: varnishncsa logs split per domain
In-Reply-To: <39257342-24c8-5dd8-9158-6b150de8893a@beckspaced.com>
References: <d0e759e1-a91c-5632-bac4-c98710839391@beckspaced.com>
 <a3d31652-9a07-45cb-53e1-46c2faaeff48@gurutek.biz>
 <976bff90-7c3e-e9c2-3517-6bc30a937de5@beckspaced.com>
 <f27bdfd8-cbf5-1f8b-df2c-2ab09de426a2@gurutek.biz>
 <4aea1d32-f9b8-ada4-fb9d-532f605ae5da@beckspaced.com>
 <CAP+vvEtzMD4=Nc_kKtMoWaGeTbmtdp181MEyFb_sF78rg0U6rQ@mail.gmail.com>
 <5aa0e368-0121-8542-70bc-27aeae96e4e9@beckspaced.com>
 <CABoVN9AzmgyN1oqHmixKz-i4Y3u+JRLKTs2h0NLbfT8yfhfPPA@mail.gmail.com>
 <d2946717-21ed-5060-c058-24cb43262429@beckspaced.com>
 <CABoVN9DkEGiMKi+=x6W-a5g01XpdZcdRrhRPf-Zq0pC=5spyPA@mail.gmail.com>
 <912a8f40-e02b-d578-585c-e3d9b8fdd0a2@beckspaced.com>
 <39257342-24c8-5dd8-9158-6b150de8893a@beckspaced.com>
Message-ID: <CAJ6ZYQy33rKZpCPUOc-A-9XBmGLz16Np-yzZ6a2fHrXGBD52xg@mail.gmail.com>

Thanks for reporting back, great job!

On Dec 2, 2016 19:57, "Admin Beckspaced" <admin at beckspaced.com> wrote:

>
> On 15.11.2016 17:57, Admin Beckspaced wrote:
>
>>
>> Am 15.11.2016 um 16:51 schrieb Dridi Boukelmoune:
>>
>>> a bit more hints and info would be nice ;)
>>>>
>>> man vmod_std
>>> man varnishncsa
>>>
>>> That's how much "nice" I'm willing to do :p
>>>
>>> Dridi
>>>
>>>
>>> ok. first I want to say thanks for being nice and pointing me to the man
>> pages.
>>
>> after a bit of reading I finally found the parts I was looking for:
>>
>>     import std;
>>
>>     sub vcl_recv {
>>
>>     std.log(?myhost:? + regsub(req.http.Host, "^www\.", "") );
>>
>>     }
>>
>> in varnishncsa:
>>
>>      # varnishncsa -F ?%h %l %u %t ?%r? %s %b ?%{Referer}i?
>> ?%{User-agent}i %{VCL_Log:myhost}x?
>>
>>
>> not yet tested but I think this is what Dridi was pointing to?
>>
>> Hello again,
>
> sorry, it has been a while but I just thought to finish the thread I
> started and point to the solution I decided to go with at last.
>
> The question in the beginning was: How can I split the varnishncsa logs
> per domain.
>
> My first thinking was to use the query -q option, e.g. varnishncsa -q
> "ReqHeader ~ '^Host: .*example.com'"
>
> But this approach would end up in a lot varnishncsa instances, as pointed
> out by Andrei, and also the problem with not being able to normalize the
> Request Host header.
> Then Dridi pointed me to man vmod_std and using std.log in VCL, which was
> the final bit needed ;)
>
> So here's my current solution:
>
> I run a single instance of varnishncsa with the following params:
>
> VARNISHLOG_PARAMS="-f /etc/varnish/varnishncsa-log-format-string -a -w
> /var/log/varnish/varnish.log"
>
> the varnishncsa-log-format-string is as follows:
>
> %{VCL_Log:myhost}x %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"
>
> at the beginning VCL_Log:key     The value set by std.log("key:value") in
> VCL, more on that later
>
> My varnish sits in front of Apache with 30 something different domains.
> Currently I don't use varnish for all domains, but have setup varnish VCL
> in such a way that I can filter on the domains and decide to cache with
> varnish or just skip caching and pass to the apache backend.
>
> My varnish VCL is based on the varnish boilerplate which I found on the
> net:
>
> http://verticalprogramming.com/2013/09/15/varnish-virtual-host-boilerplate
>
> So if I decide to cache a particular domain with varnish I can normalize
> the Request Host header in VCL
>
> sub vcl_recv {
>
>     if (req.http.host ~ "somedomain\.com") {
>
>         # tilde ~ uses regex
>         if (req.http.host ~ "^(www.)?somedomain\.com$") {
>
>             //normalize the req.http.host
>             set req.http.host = regsub(req.http.Host, "^www\.", "");
>
>             std.log("myhost:" + req.http.Host );
> ...
>
> so this std.log(myhost:somedomian.com) gets picked up by varnishncsa and
> the custom format string, see above.
>
> which then produces a nice & steady varnish.log file with a normalized
> host at the beginning for domains I want to cache and an empty space if I
> don't want to.
>
> then we got the split-logfile from apache:
>
> https://httpd.apache.org/docs/2.4/programs/split-logfile.html
>
> which was exactly made for a setup with the host names at the very
> beginning of the log file. Only thing someone needs to take care of is that
> the logfiles will get created in the directory where the script is run. so
> therefore I created a small bash script wrapper split-logfile.sh which
> first changes to the right working directory:
>
> #!/bin/bash
> cd /var/log/varnish
> /usr/bin/split-logfile < varnish.log
>
> and on the daily logrotate on the /var/log/varnish/varnish.log I added the
> following:
>
> /var/log/varnish/varnish.log {
>     ...
>     prerotate
>      /var/log/varnish/split-logfile.sh
>     endscript
>     ...
> }
>
> so before the varnish.log gets rotated split-logfile.sh gets called and
> creates the different log files per normalized host and an access.log for
> all the requests without a hostname at the beginning.
> After a view logrotate runs the /var/log/varnish/ could look like that:
>
> mydomain.com.log
> myotherdomain.com.log
> access.log
> varnish.log
> varnish.log-20161130
> varnish.log-20161201
> varnish.log-20161202
>
> which finally give me exactly what I wanted! A single instance of
> varnishncsa producing a varnish.log for all domains.
> a per domain split via split-logfile.sh on each logrotate run resulting in
> log files per domain ready to use with webalizer, which gets also called on
> prerotate the logfile:
>
> /var/log/varnish/mydomain.com.log {
>     ...
>     prerotate
>      /usr/bin/webalizer -qc /etc/webalizer/mydomain.conf
>     endscript
>     ...
> }
>
> perhaps this might help someone here looking for something similar?
>
> thanks & greetings
> becki
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161202/01f7e59d/attachment.html>

From admin at beckspaced.com  Mon Dec  5 17:36:59 2016
From: admin at beckspaced.com (Admin Beckspaced)
Date: Mon, 5 Dec 2016 18:36:59 +0100
Subject: Varnishncsa log file size
Message-ID: <509ce6aa-cfa0-3b41-e6af-f758b6ec04f1@beckspaced.com>

Hello Varnish-Users,

recently I have setup varnish logging via varnishncsa.
If I look at the varnish log file size it's currently around 50 MB per 
day which gets rotate daily via logrotate.

How big can the log file size grow without any performance issues?
Is 100 MB still a valid log file size? even 500 MB perhaps?

The more domains I add the bigger the log file will grow per day. Any 
hints are welcome ;)

Thanks & greetings
Becki



From A.Hongens at netmatch.nl  Mon Dec  5 18:27:11 2016
From: A.Hongens at netmatch.nl (=?iso-8859-1?Q?Angelo_H=F6ngens?=)
Date: Mon, 5 Dec 2016 18:27:11 +0000
Subject: Varnishncsa log file size
In-Reply-To: <509ce6aa-cfa0-3b41-e6af-f758b6ec04f1@beckspaced.com>
References: <509ce6aa-cfa0-3b41-e6af-f758b6ec04f1@beckspaced.com>
Message-ID: <6A7ABA19243F1E4EADD8BB1563CDDCCB8AD80D54@TIL-EXCH-05.netmatch.local>

I sometimes have a few gigabytes of logfile per day. I don't notice any performance issues at all, so don't worry about that. 

-- 
 
With kind regards,
 
 
Angelo H?ngens
 
Systems Administrator
 
------------------------------------------
NetMatch
travel technology solutions

Professor Donderstraat 46
5017 HL Tilburg
T: +31 (0)13 5811088
F: +31 (0)13 5821239
 
mailto:A.Hongens at netmatch.nl
http://www.netmatch.nl
------------------------------------------
Disclaimer
 
Deze e-mail is vertrouwelijk en uitsluitend bedoeld voor geadresseerde(n) en de organisatie van geadresseerde(n) en mag niet openbaar worden gemaakt aan derde partijen This e-mail is confidential and may not be disclosed to third parties since this e-mail is only intended for the addressee and the organization the addressee represents.


-----Original Message-----
From: varnish-misc-bounces+a.hongens=netmatch.nl at varnish-cache.org [mailto:varnish-misc-bounces+a.hongens=netmatch.nl at varnish-cache.org] On Behalf Of Admin Beckspaced
Sent: Monday, 5 December, 2016 18:37
To: varnish-misc at varnish-cache.org
Subject: Varnishncsa log file size

Hello Varnish-Users,

recently I have setup varnish logging via varnishncsa.
If I look at the varnish log file size it's currently around 50 MB per day which gets rotate daily via logrotate.

How big can the log file size grow without any performance issues?
Is 100 MB still a valid log file size? even 500 MB perhaps?

The more domains I add the bigger the log file will grow per day. Any hints are welcome ;)

Thanks & greetings
Becki

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From cservin-varnish at cromagnon.com  Mon Dec  5 19:48:03 2016
From: cservin-varnish at cromagnon.com (Craig Servin)
Date: Mon, 05 Dec 2016 13:48:03 -0600
Subject: Varnishncsa log file size
In-Reply-To: <509ce6aa-cfa0-3b41-e6af-f758b6ec04f1@beckspaced.com>
References: <509ce6aa-cfa0-3b41-e6af-f758b6ec04f1@beckspaced.com>
Message-ID: <ed6cf6c2ad5499fe0cc80a98e96bcc80@www.cromagnon.com>

Just make sure you don't have logrotate doing a copytruncate on the 
varnishlog.  Let it rename and then send varnishncsa a HUP and you will 
have no performance issues.

Cheers,

Craig


On 2016-12-05 11:36, Admin Beckspaced wrote:
> Hello Varnish-Users,
> 
> recently I have setup varnish logging via varnishncsa.
> If I look at the varnish log file size it's currently around 50 MB per
> day which gets rotate daily via logrotate.
> 
> How big can the log file size grow without any performance issues?
> Is 100 MB still a valid log file size? even 500 MB perhaps?
> 
> The more domains I add the bigger the log file will grow per day. Any
> hints are welcome ;)
> 
> Thanks & greetings
> Becki
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc




From t.cramer at beslist.nl  Wed Dec  7 08:29:40 2016
From: t.cramer at beslist.nl (Thijs Cramer)
Date: Wed, 7 Dec 2016 09:29:40 +0100
Subject: Serving static html when all backends are sick
Message-ID: <1481099380.8276.9.camel@beslist.nl>

Given the following code:

=========================================
sub vcl_synth {
? if (resp.status == 999) {
????set resp.status = 200;
????set resp.http.Content-Type = "text/html; charset=utf-8";
????synthetic( {"Static HTML"} );
? ? return(deliver);
? }
}

# Check if backend is healthy, otherwise, redirect
if (!std.healthy(req.backend_hint)) {
? return (synth(999, "All backends down."));
}

=========================================

Will the static page be served if ALL backends are down, or just when
the specific backend it got back from the backend_hint is down?

We want to be able to serve static html as a fallback if all backends
are down.

Is this the proper way to do this?

Thanks in advance!

- Thijs



From justinl at arena.net  Wed Dec  7 21:22:05 2016
From: justinl at arena.net (Justin Lloyd)
Date: Wed, 7 Dec 2016 21:22:05 +0000
Subject: Hit ratio dropped significantly after recent upgrades
Message-ID: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>

Hello,

I use Varnish on four load-balanced web servers running Apache with several virtual hosts providing MediaWiki wikis. Last week I upgraded the web servers to support upgrading MediaWiki from 1.26 to 1.27 due to PHP requirements. This meant upgrading (reinstalling) Ubuntu 12.04 to 16.04, meaning I went from PHP 5.3 to 7, Apache 2.4 + mod_php to Apache 2.4 + PHP-FPM, and Varnish 3 to 4.

For the upgrade from Varnish 3 to 4, I adapted my VCL code<https://gist.github.com/Calygos/105957a997ea3bde6b8257a1f34bbd20> accordingly, based on Mediawiki Varnish sample code<https://www.mediawiki.org/wiki/Manual:Varnish_caching>, but for reasons I don't understand, my Varnish hit ratio dropped from around 86% to ~20-37%. This picture<http://imgur.com/jX4iu6s> shows a graph of the last 10 days hit ratio calculations for the four web servers. The graph is calculated using metrics from the Collectd Varnish plugin as (conns - bereqs / conns), i.e.:

asPercent(
    diffSeries(
        linux.hostname.varnish-default-connections.connections-received,
        linux.hostname.varnish-default-backend.http_requests-requests
    ),
   linux.hostname.varnish-default-connections.connections-received
)

I've also tried the Varnish hit and miss metrics but those give similar results. So I'm not sure what changed to give such a poor hit ratio, despite otherwise good performance on large, busy wikis. It's possible it could be something in MediaWiki's cookie and/or session handling and I've been investigating that possibility as well, but I figured this would be a good place to ask, too.

Thanks,
Justin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161207/3c76ccf5/attachment.html>

From japrice at gmail.com  Thu Dec  8 13:35:04 2016
From: japrice at gmail.com (Jason Price)
Date: Thu, 8 Dec 2016 08:35:04 -0500
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>

I think we're going to need something a little more specific to go on.
That is a mile of changes all at once.

Finding a single request that should be cached, but isn't and producing the
varnish log for that request will probably help illuminate what's going on.

On Wed, Dec 7, 2016 at 4:22 PM, Justin Lloyd <justinl at arena.net> wrote:

> Hello,
>
>
>
> I use Varnish on four load-balanced web servers running Apache with
> several virtual hosts providing MediaWiki wikis. Last week I upgraded the
> web servers to support upgrading MediaWiki from 1.26 to 1.27 due to PHP
> requirements. This meant upgrading (reinstalling) Ubuntu 12.04 to 16.04,
> meaning I went from PHP 5.3 to 7, Apache 2.4 + mod_php to Apache 2.4 +
> PHP-FPM, and Varnish 3 to 4.
>
>
>
> For the upgrade from Varnish 3 to 4, I adapted my VCL code
> <https://gist.github.com/Calygos/105957a997ea3bde6b8257a1f34bbd20>
> accordingly, based on Mediawiki Varnish sample code
> <https://www.mediawiki.org/wiki/Manual:Varnish_caching>, but for reasons
> I don?t understand, my Varnish hit ratio dropped from around 86% to
> ~20-37%. This picture <http://imgur.com/jX4iu6s> shows a graph of the
> last 10 days hit ratio calculations for the four web servers. The graph is
> calculated using metrics from the Collectd Varnish plugin as (conns -
> bereqs / conns), i.e.:
>
>
>
> asPercent(
>
>     diffSeries(
>
>         linux.*hostname*.varnish-default-connections.connections-received,
>
>         linux.*hostname*.varnish-default-backend.http_requests-requests
>
>     ),
>
>    linux.*hostname*.varnish-default-connections.connections-received
>
> )
>
>
>
> I?ve also tried the Varnish hit and miss metrics but those give similar
> results. So I?m not sure what changed to give such a poor hit ratio,
> despite otherwise good performance on large, busy wikis. It?s possible it
> could be something in MediaWiki?s cookie and/or session handling and I?ve
> been investigating that possibility as well, but I figured this would be a
> good place to ask, too.
>
>
>
> Thanks,
>
> Justin
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161208/f065fe67/attachment.html>

From dridi at varni.sh  Thu Dec  8 14:36:35 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 8 Dec 2016 15:36:35 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <CABoVN9BVh3A5oYqahduYgYVw9ixaB+s7Nn27yVdU8mM47P+CDA@mail.gmail.com>

> results. So I?m not sure what changed to give such a poor hit ratio, despite
> otherwise good performance on large, busy wikis. It?s possible it could be

Hello Justin,

I'm afraid too many things changed at once, with so many moving parts
it will be hard to find the cause. I suggest you start with transaction logs,
looking essentially at what the backend is responding and how your VCL
is handling it.

Dridi



From dridi at varni.sh  Thu Dec  8 14:48:39 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 8 Dec 2016 15:48:39 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
Message-ID: <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>

On Thu, Dec 8, 2016 at 2:35 PM, Jason Price <japrice at gmail.com> wrote:
> I think we're going to need something a little more specific to go on.  That
> is a mile of changes all at once.

Yes: varnishlog, coffee, and a lot of patience.

> Finding a single request that should be cached, but isn't and producing the
> varnish log for that request will probably help illuminate what's going on.

There's currently no way to query the transaction log of a specific request:
https://github.com/varnishcache/varnish-cache/issues/2154

I'm just saying...

Dridi



From guillaume at varnish-software.com  Thu Dec  8 16:08:40 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Thu, 8 Dec 2016 17:08:40 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
Message-ID: <CAJ6ZYQy3YMRFxe=65rJR6TkY97LizHVCyLn1pCe3S0nhOF2mnQ@mail.gmail.com>

On Thu, Dec 8, 2016 at 3:48 PM, Dridi Boukelmoune <dridi at varni.sh> wrote:
>
> There's currently no way to query the transaction log of a specific
> request:
> https://github.com/varnishcache/varnish-cache/issues/2154


awk is love, awk is life:

varnishlog -d -v -g raw | awk '$1 == XXXXXX'

still better than nothing.

-- 
Guillaume Quintard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161208/fca36e4f/attachment.html>

From justinl at arena.net  Thu Dec  8 20:26:37 2016
From: justinl at arena.net (Justin Lloyd)
Date: Thu, 8 Dec 2016 20:26:37 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
Message-ID: <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>

I have been doing a lot of digging with varnishtop and varnishlog, and our VCL really didn?t change from this upgrade except as needed to migrate from Varnish 3 to 4. As I mentioned, our web app is MediaWiki so we don't control its caching requirements and recommendations, so what I'm trying to understand is whether the drop in the hit rate is due to some change(s) in MediaWiki's cookie and/or cache handling (e.g. via Cache-Control and Set-Cookie headers) or if something in Varnish changed that affects how it determines  things. For example, a while back I had been using the Varnish hit and miss metrics in Collectd to calculate the ratio but apparently how those values are calculated with respect to purges changed so the hit ratio dropped, causing me to change the ratio calculation to use incoming connections and backend requests instead.

That said, based on my varnishlog and varnishtop testing, I have a strong feeling that the biggest part of the problem is thumbnail images. If you look again at my VCL code (https://gist.github.com/Calygos/105957a997ea3bde6b8257a1f34bbd20), you can see I strip cookies from thumbnails so they should get cached, but I seem to get a lot more misses than hits when watching for thumbnail URL requests through varnishtop. I give 8 GB to Varnish and its process is typically only around 1 to 2 GB when previous it would be at 8 GB with frequent nukes and the occasional spike of expires that would temporarily eliminate nukes while memory filled up again. For what it's worth, I added the thumbnail stripping a couple of years ago due to a performance issue and it helped tremendously, so I don't know why it would become problematic with these latest upgrades.

Justin

-----Original Message-----
From: Dridi Boukelmoune [mailto:dridi at varni.sh] 
Sent: Thursday, December 8, 2016 6:49 AM
To: Jason Price <japrice at gmail.com>
Cc: Justin Lloyd <justinl at arena.net>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

On Thu, Dec 8, 2016 at 2:35 PM, Jason Price <japrice at gmail.com> wrote:
> I think we're going to need something a little more specific to go on.  
> That is a mile of changes all at once.

Yes: varnishlog, coffee, and a lot of patience.

> Finding a single request that should be cached, but isn't and 
> producing the varnish log for that request will probably help illuminate what's going on.

There's currently no way to query the transaction log of a specific request:
https://github.com/varnishcache/varnish-cache/issues/2154

I'm just saying...

Dridi

From daghf at varnish-software.com  Fri Dec  9 12:46:48 2016
From: daghf at varnish-software.com (Dag Haavi Finstad)
Date: Fri, 9 Dec 2016 13:46:48 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>

Hi

Is this Varnish 4.1 ?

We have an unsolved bug open describing something very similar,
https://github.com/varnishcache/varnish-cache/issues/1859

On Thu, Dec 8, 2016 at 9:26 PM, Justin Lloyd <justinl at arena.net> wrote:
> I have been doing a lot of digging with varnishtop and varnishlog, and our VCL really didn?t change from this upgrade except as needed to migrate from Varnish 3 to 4. As I mentioned, our web app is MediaWiki so we don't control its caching requirements and recommendations, so what I'm trying to understand is whether the drop in the hit rate is due to some change(s) in MediaWiki's cookie and/or cache handling (e.g. via Cache-Control and Set-Cookie headers) or if something in Varnish changed that affects how it determines  things. For example, a while back I had been using the Varnish hit and miss metrics in Collectd to calculate the ratio but apparently how those values are calculated with respect to purges changed so the hit ratio dropped, causing me to change the ratio calculation to use incoming connections and backend requests instead.
>
> That said, based on my varnishlog and varnishtop testing, I have a strong feeling that the biggest part of the problem is thumbnail images. If you look again at my VCL code (https://gist.github.com/Calygos/105957a997ea3bde6b8257a1f34bbd20), you can see I strip cookies from thumbnails so they should get cached, but I seem to get a lot more misses than hits when watching for thumbnail URL requests through varnishtop. I give 8 GB to Varnish and its process is typically only around 1 to 2 GB when previous it would be at 8 GB with frequent nukes and the occasional spike of expires that would temporarily eliminate nukes while memory filled up again. For what it's worth, I added the thumbnail stripping a couple of years ago due to a performance issue and it helped tremendously, so I don't know why it would become problematic with these latest upgrades.
>
> Justin
>
> -----Original Message-----
> From: Dridi Boukelmoune [mailto:dridi at varni.sh]
> Sent: Thursday, December 8, 2016 6:49 AM
> To: Jason Price <japrice at gmail.com>
> Cc: Justin Lloyd <justinl at arena.net>; varnish-misc at varnish-cache.org
> Subject: Re: Hit ratio dropped significantly after recent upgrades
>
> On Thu, Dec 8, 2016 at 2:35 PM, Jason Price <japrice at gmail.com> wrote:
>> I think we're going to need something a little more specific to go on.
>> That is a mile of changes all at once.
>
> Yes: varnishlog, coffee, and a lot of patience.
>
>> Finding a single request that should be cached, but isn't and
>> producing the varnish log for that request will probably help illuminate what's going on.
>
> There's currently no way to query the transaction log of a specific request:
> https://github.com/varnishcache/varnish-cache/issues/2154
>
> I'm just saying...
>
> Dridi
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



-- 
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!



From justinl at arena.net  Fri Dec  9 13:43:38 2016
From: justinl at arena.net (Justin Lloyd)
Date: Fri, 9 Dec 2016 13:43:38 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
Message-ID: <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>

Hello! Yes, it is Varnish 4.1.1-1 from the Ubuntu 16.04 repo. I'll look at the issue you've linked and see if I can match it to our situation. Thanks!

Justin

-----Original Message-----
From: Dag Haavi Finstad [mailto:daghf at varnish-software.com] 
Sent: Friday, December 9, 2016 4:47 AM
To: Justin Lloyd <justinl at arena.net>
Cc: Dridi Boukelmoune <dridi at varni.sh>; Jason Price <japrice at gmail.com>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

Hi

Is this Varnish 4.1 ?

We have an unsolved bug open describing something very similar,
https://github.com/varnishcache/varnish-cache/issues/1859

On Thu, Dec 8, 2016 at 9:26 PM, Justin Lloyd <justinl at arena.net> wrote:
> I have been doing a lot of digging with varnishtop and varnishlog, and our VCL really didn?t change from this upgrade except as needed to migrate from Varnish 3 to 4. As I mentioned, our web app is MediaWiki so we don't control its caching requirements and recommendations, so what I'm trying to understand is whether the drop in the hit rate is due to some change(s) in MediaWiki's cookie and/or cache handling (e.g. via Cache-Control and Set-Cookie headers) or if something in Varnish changed that affects how it determines  things. For example, a while back I had been using the Varnish hit and miss metrics in Collectd to calculate the ratio but apparently how those values are calculated with respect to purges changed so the hit ratio dropped, causing me to change the ratio calculation to use incoming connections and backend requests instead.
>
> That said, based on my varnishlog and varnishtop testing, I have a strong feeling that the biggest part of the problem is thumbnail images. If you look again at my VCL code (https://gist.github.com/Calygos/105957a997ea3bde6b8257a1f34bbd20), you can see I strip cookies from thumbnails so they should get cached, but I seem to get a lot more misses than hits when watching for thumbnail URL requests through varnishtop. I give 8 GB to Varnish and its process is typically only around 1 to 2 GB when previous it would be at 8 GB with frequent nukes and the occasional spike of expires that would temporarily eliminate nukes while memory filled up again. For what it's worth, I added the thumbnail stripping a couple of years ago due to a performance issue and it helped tremendously, so I don't know why it would become problematic with these latest upgrades.
>
> Justin
>
> -----Original Message-----
> From: Dridi Boukelmoune [mailto:dridi at varni.sh]
> Sent: Thursday, December 8, 2016 6:49 AM
> To: Jason Price <japrice at gmail.com>
> Cc: Justin Lloyd <justinl at arena.net>; varnish-misc at varnish-cache.org
> Subject: Re: Hit ratio dropped significantly after recent upgrades
>
> On Thu, Dec 8, 2016 at 2:35 PM, Jason Price <japrice at gmail.com> wrote:
>> I think we're going to need something a little more specific to go on.
>> That is a mile of changes all at once.
>
> Yes: varnishlog, coffee, and a lot of patience.
>
>> Finding a single request that should be cached, but isn't and 
>> producing the varnish log for that request will probably help illuminate what's going on.
>
> There's currently no way to query the transaction log of a specific request:
> https://github.com/varnishcache/varnish-cache/issues/2154
>
> I'm just saying...
>
> Dridi
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



--
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!

From justinl at arena.net  Fri Dec  9 17:08:27 2016
From: justinl at arena.net (Justin Lloyd)
Date: Fri, 9 Dec 2016 17:08:27 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>

We may be seeing this issue though I can't confirm since I all of my servers are running 4.1.1 now, but I calculate my hit ratio based on the total number of Varnish connections and how many are sent to Apache, per server. Again, for reference from my first email, here's my Graphite dashboard function for generating my hit ratio graph:

asPercent(
    diffSeries(
        linux.hostname.varnish-default-connections.connections-received,
        linux.hostname.varnish-default-backend.http_requests-requests
    ),
   linux.hostname.varnish-default-connections.connections-received
)

I also look at varnishstat to confirm, here's about a 10 minute sample:

    NAME                 CURRENT        CHANGE       AVERAGE        AVG_10       AVG_100      AVG_1000 
MAIN.client_req          7937622          0.00         47.00          5.27         83.94         60.92
MAIN.backend_req         6341302         51.91         37.00         38.97         53.59         43.80
MAIN.cache_hit           2267216          0.00         13.00          1.08         25.91         18.19
MAIN.cache_miss          4906402          0.00         29.00          1.50         43.93         37.26

So I don't think our problem is a cache_hit/miss value calculation issue since the number of client vs. backend requests is similar and very underperforming.

To reiterate on a point in another of my responses in this thread, I think it may be something about MediaWiki thumbnail images not being cached properly despite our current VCL in that regard not having changed from how it worked prior to the upgrade during which time we were seeing a very high (86%-ish) hit ratio from the same formula.

-----Original Message-----
From: varnish-misc-bounces+justinl=arena.net at varnish-cache.org [mailto:varnish-misc-bounces+justinl=arena.net at varnish-cache.org] On Behalf Of Justin Lloyd
Sent: Friday, December 9, 2016 5:44 AM
To: Dag Haavi Finstad <daghf at varnish-software.com>
Cc: varnish-misc at varnish-cache.org
Subject: RE: Hit ratio dropped significantly after recent upgrades

Hello! Yes, it is Varnish 4.1.1-1 from the Ubuntu 16.04 repo. I'll look at the issue you've linked and see if I can match it to our situation. Thanks!

Justin

-----Original Message-----
From: Dag Haavi Finstad [mailto:daghf at varnish-software.com]
Sent: Friday, December 9, 2016 4:47 AM
To: Justin Lloyd <justinl at arena.net>
Cc: Dridi Boukelmoune <dridi at varni.sh>; Jason Price <japrice at gmail.com>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

Hi

Is this Varnish 4.1 ?

We have an unsolved bug open describing something very similar,
https://github.com/varnishcache/varnish-cache/issues/1859

On Thu, Dec 8, 2016 at 9:26 PM, Justin Lloyd <justinl at arena.net> wrote:
> I have been doing a lot of digging with varnishtop and varnishlog, and our VCL really didn?t change from this upgrade except as needed to migrate from Varnish 3 to 4. As I mentioned, our web app is MediaWiki so we don't control its caching requirements and recommendations, so what I'm trying to understand is whether the drop in the hit rate is due to some change(s) in MediaWiki's cookie and/or cache handling (e.g. via Cache-Control and Set-Cookie headers) or if something in Varnish changed that affects how it determines  things. For example, a while back I had been using the Varnish hit and miss metrics in Collectd to calculate the ratio but apparently how those values are calculated with respect to purges changed so the hit ratio dropped, causing me to change the ratio calculation to use incoming connections and backend requests instead.
>
> That said, based on my varnishlog and varnishtop testing, I have a strong feeling that the biggest part of the problem is thumbnail images. If you look again at my VCL code (https://gist.github.com/Calygos/105957a997ea3bde6b8257a1f34bbd20), you can see I strip cookies from thumbnails so they should get cached, but I seem to get a lot more misses than hits when watching for thumbnail URL requests through varnishtop. I give 8 GB to Varnish and its process is typically only around 1 to 2 GB when previous it would be at 8 GB with frequent nukes and the occasional spike of expires that would temporarily eliminate nukes while memory filled up again. For what it's worth, I added the thumbnail stripping a couple of years ago due to a performance issue and it helped tremendously, so I don't know why it would become problematic with these latest upgrades.
>
> Justin
>
> -----Original Message-----
> From: Dridi Boukelmoune [mailto:dridi at varni.sh]
> Sent: Thursday, December 8, 2016 6:49 AM
> To: Jason Price <japrice at gmail.com>
> Cc: Justin Lloyd <justinl at arena.net>; varnish-misc at varnish-cache.org
> Subject: Re: Hit ratio dropped significantly after recent upgrades
>
> On Thu, Dec 8, 2016 at 2:35 PM, Jason Price <japrice at gmail.com> wrote:
>> I think we're going to need something a little more specific to go on.
>> That is a mile of changes all at once.
>
> Yes: varnishlog, coffee, and a lot of patience.
>
>> Finding a single request that should be cached, but isn't and 
>> producing the varnish log for that request will probably help illuminate what's going on.
>
> There's currently no way to query the transaction log of a specific request:
> https://github.com/varnishcache/varnish-cache/issues/2154
>
> I'm just saying...
>
> Dridi
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



--
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!
_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

From dridi at varni.sh  Fri Dec  9 18:10:42 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 9 Dec 2016 19:10:42 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>

> To reiterate on a point in another of my responses in this thread, I think it may be something about MediaWiki thumbnail images not being cached properly despite our current VCL in that regard not having changed from how it worked prior to the upgrade during which time we were seeing a very high (86%-ish) hit ratio from the same formula.

To reiterate on a point I made on a couple occasions, it's time to
give varnishlog a spin. Too much focus on VCL, and not enough on
what's happening.

Dridi



From justinl at arena.net  Fri Dec  9 19:19:19 2016
From: justinl at arena.net (Justin Lloyd)
Date: Fri, 9 Dec 2016 19:19:19 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
Message-ID: <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>

I really am looking at what's happening as well. I have been looking at both varnishlog and varnishtop and I see a lot of thumbnail image requests being sent to the backend when there is still plenty of room for them in the cache, so even though there are a lot of thumbnail images, I shouldn't see so many backend requests for them. As I previously mentioned, I give Varnish 8 GB and it used to stay full (based on RSS usage and looking at nukes vs. expires) but now it hovers around only about 2 GB used. A related statistics is that there used to be 600-700k objects in Varnish (based on our graphs of MAIN.n_object via Collectd's varnish-default-struct.objects-object metric) but now there are only roughly 40-70k objects in Varnish at any given time. So it's definitely caching a lot fewer things than it was before the upgrade, and most of the requested URLs for requests that have cookies are for a lot of images and thumbnails. Images shouldn't be cached due to size and overall volume but thumbnails should, which is why I strip cookies from the thumbnails. These varnishtop commands break out /images and /images/thumb client requests, showing IMHO too many regular images being cached and nowhere near enough thumbnails:

# varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/" and not ReqURL ~ "/images/thumb"'

   349.47 VCL_call       HASH
   349.47 VCL_call       RECV
   349.47 VCL_call       DELIVER
   207.22 VCL_call       HIT
   116.40 VCL_call       MISS
   116.30 VCL_call       PASS

# varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/thumb"'                            

  1859.60 VCL_call       HASH
  1859.60 VCL_call       RECV
  1859.60 VCL_call       DELIVER
  1424.83 VCL_call       MISS
   422.84 VCL_call       HIT
   218.82 VCL_call       PASS

I'm still poking around trying to correlate caching of other types of URLs based on whether or not the requests have cookies, if Cache-Control gets returned, etc. but I just wanted to reply with this info. I do appreciate the responses I'm getting! :)


-----Original Message-----
From: Dridi Boukelmoune [mailto:dridi at varni.sh] 
Sent: Friday, December 9, 2016 10:11 AM
To: Justin Lloyd <justinl at arena.net>
Cc: Dag Haavi Finstad <daghf at varnish-software.com>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

> To reiterate on a point in another of my responses in this thread, I think it may be something about MediaWiki thumbnail images not being cached properly despite our current VCL in that regard not having changed from how it worked prior to the upgrade during which time we were seeing a very high (86%-ish) hit ratio from the same formula.

To reiterate on a point I made on a couple occasions, it's time to give varnishlog a spin. Too much focus on VCL, and not enough on what's happening.

Dridi

From guillaume at varnish-software.com  Mon Dec 12 18:51:55 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Mon, 12 Dec 2016 19:51:55 +0100
Subject: Serving static html when all backends are sick
In-Reply-To: <1481099380.8276.9.camel@beslist.nl>
References: <1481099380.8276.9.camel@beslist.nl>
Message-ID: <CAJ6ZYQySN1ydQ-4=a-UViSJ7MquixPkH2H4-iXoQVB0hJguisA@mail.gmail.com>

A director is considered healthy is any of its backend is healthy, so if
you put a director in req.backend_hint, you're good.

-- 
Guillaume Quintard

On Wed, Dec 7, 2016 at 9:29 AM, Thijs Cramer <t.cramer at beslist.nl> wrote:

> Given the following code:
>
> =========================================
> sub vcl_synth {
>   if (resp.status == 999) {
>     set resp.status = 200;
>     set resp.http.Content-Type = "text/html; charset=utf-8";
>     synthetic( {"Static HTML"} );
>     return(deliver);
>   }
> }
>
> # Check if backend is healthy, otherwise, redirect
> if (!std.healthy(req.backend_hint)) {
>   return (synth(999, "All backends down."));
> }
>
> =========================================
>
> Will the static page be served if ALL backends are down, or just when
> the specific backend it got back from the backend_hint is down?
>
> We want to be able to serve static html as a fallback if all backends
> are down.
>
> Is this the proper way to do this?
>
> Thanks in advance!
>
> - Thijs
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161212/5b3e3db3/attachment.html>

From justinl at arena.net  Tue Dec 13 02:37:06 2016
From: justinl at arena.net (Justin Lloyd)
Date: Tue, 13 Dec 2016 02:37:06 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>

To follow up on my last email from Friday, at this point the problem boils down to one thing that I've not been able to determine: Why are far fewer things being cached now than before the upgrade?

1. Cookies don't seem to be the problem. Most appear to be Google Analytics (as opposed to session), which are being unset by vcl_recv.

2. varnishlog/varnishtop shows many thumbnail URLs being missed and virtually none are requested with a no-cache cache-control header. Is it possible to use these tools determine if they (or any URLs for that matter) are being cached following a miss-deliver sequence? There are about 1.5m thumbnail files totaling around 30 GB, which prior to the upgrades wasn't an issue, and I don't think it is now since there are only a few expires and purges per minute and no nukes at all. Varnish is only using about 2 GB out of the 8 GB allocated to it, where it used to use all 8 GB and have lots of nukes and far fewer expires, so it's not a memory constraint.

Could there be some other resource limitation I'm hitting without knowing it (nothing in any logs I've seen)? Everything else I could think of so far seems fine, e.g. open files, threads, tcp connections.


-----Original Message-----
From: varnish-misc-bounces+justinl=arena.net at varnish-cache.org [mailto:varnish-misc-bounces+justinl=arena.net at varnish-cache.org] On Behalf Of Justin Lloyd
Sent: Friday, December 9, 2016 11:19 AM
To: Dridi Boukelmoune <dridi at varni.sh>
Cc: varnish-misc at varnish-cache.org
Subject: RE: Hit ratio dropped significantly after recent upgrades

I really am looking at what's happening as well. I have been looking at both varnishlog and varnishtop and I see a lot of thumbnail image requests being sent to the backend when there is still plenty of room for them in the cache, so even though there are a lot of thumbnail images, I shouldn't see so many backend requests for them. As I previously mentioned, I give Varnish 8 GB and it used to stay full (based on RSS usage and looking at nukes vs. expires) but now it hovers around only about 2 GB used. A related statistics is that there used to be 600-700k objects in Varnish (based on our graphs of MAIN.n_object via Collectd's varnish-default-struct.objects-object metric) but now there are only roughly 40-70k objects in Varnish at any given time. So it's definitely caching a lot fewer things than it was before the upgrade, and most of the requested URLs for requests that have cookies are for a lot of images and thumbnails. Images shouldn't be cached due to size and overall volume but thumbnails should, which is why I strip cookies from the thumbnails. These varnishtop commands break out /images and /images/thumb client requests, showing IMHO too many regular images being cached and nowhere near enough thumbnails:

# varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/" and not ReqURL ~ "/images/thumb"'

   349.47 VCL_call       HASH
   349.47 VCL_call       RECV
   349.47 VCL_call       DELIVER
   207.22 VCL_call       HIT
   116.40 VCL_call       MISS
   116.30 VCL_call       PASS

# varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/thumb"'                            

  1859.60 VCL_call       HASH
  1859.60 VCL_call       RECV
  1859.60 VCL_call       DELIVER
  1424.83 VCL_call       MISS
   422.84 VCL_call       HIT
   218.82 VCL_call       PASS

I'm still poking around trying to correlate caching of other types of URLs based on whether or not the requests have cookies, if Cache-Control gets returned, etc. but I just wanted to reply with this info. I do appreciate the responses I'm getting! :)


-----Original Message-----
From: Dridi Boukelmoune [mailto:dridi at varni.sh] 
Sent: Friday, December 9, 2016 10:11 AM
To: Justin Lloyd <justinl at arena.net>
Cc: Dag Haavi Finstad <daghf at varnish-software.com>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

> To reiterate on a point in another of my responses in this thread, I think it may be something about MediaWiki thumbnail images not being cached properly despite our current VCL in that regard not having changed from how it worked prior to the upgrade during which time we were seeing a very high (86%-ish) hit ratio from the same formula.

To reiterate on a point I made on a couple occasions, it's time to give varnishlog a spin. Too much focus on VCL, and not enough on what's happening.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From dridi at varni.sh  Tue Dec 13 08:16:05 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 13 Dec 2016 09:16:05 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <CABoVN9BWkcerE7J_ey+mf8oCuH8gkkJzteAMPpqBM6Xon3ZXYA@mail.gmail.com>

> Could there be some other resource limitation I'm hitting without knowing it (nothing in any logs I've seen)? Everything else I could think of so far seems fine, e.g. open files, threads, tcp connections.

We can't tell if we can't have a look at a transaction's log so please
find one thumbnail's req+bereq log with varnishlog and post it on the
list. Check that it doesn't contain sensitive information and otherwise
sanitize it before posting.

Dridi



From guillaume at varnish-software.com  Tue Dec 13 08:17:13 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Tue, 13 Dec 2016 09:17:13 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>

Can you pastebin the req+bereq transactions in varnishlog, related to such
a miss?

-- 
Guillaume Quintard

On Tue, Dec 13, 2016 at 3:37 AM, Justin Lloyd <justinl at arena.net> wrote:

> To follow up on my last email from Friday, at this point the problem boils
> down to one thing that I've not been able to determine: Why are far fewer
> things being cached now than before the upgrade?
>
> 1. Cookies don't seem to be the problem. Most appear to be Google
> Analytics (as opposed to session), which are being unset by vcl_recv.
>
> 2. varnishlog/varnishtop shows many thumbnail URLs being missed and
> virtually none are requested with a no-cache cache-control header. Is it
> possible to use these tools determine if they (or any URLs for that matter)
> are being cached following a miss-deliver sequence? There are about 1.5m
> thumbnail files totaling around 30 GB, which prior to the upgrades wasn't
> an issue, and I don't think it is now since there are only a few expires
> and purges per minute and no nukes at all. Varnish is only using about 2 GB
> out of the 8 GB allocated to it, where it used to use all 8 GB and have
> lots of nukes and far fewer expires, so it's not a memory constraint.
>
> Could there be some other resource limitation I'm hitting without knowing
> it (nothing in any logs I've seen)? Everything else I could think of so far
> seems fine, e.g. open files, threads, tcp connections.
>
>
> -----Original Message-----
> From: varnish-misc-bounces+justinl=arena.net at varnish-cache.org [mailto:
> varnish-misc-bounces+justinl=arena.net at varnish-cache.org] On Behalf Of
> Justin Lloyd
> Sent: Friday, December 9, 2016 11:19 AM
> To: Dridi Boukelmoune <dridi at varni.sh>
> Cc: varnish-misc at varnish-cache.org
> Subject: RE: Hit ratio dropped significantly after recent upgrades
>
> I really am looking at what's happening as well. I have been looking at
> both varnishlog and varnishtop and I see a lot of thumbnail image requests
> being sent to the backend when there is still plenty of room for them in
> the cache, so even though there are a lot of thumbnail images, I shouldn't
> see so many backend requests for them. As I previously mentioned, I give
> Varnish 8 GB and it used to stay full (based on RSS usage and looking at
> nukes vs. expires) but now it hovers around only about 2 GB used. A related
> statistics is that there used to be 600-700k objects in Varnish (based on
> our graphs of MAIN.n_object via Collectd's varnish-default-struct.objects-object
> metric) but now there are only roughly 40-70k objects in Varnish at any
> given time. So it's definitely caching a lot fewer things than it was
> before the upgrade, and most of the requested URLs for requests that have
> cookies are for a lot of images and thumbnails. Images shouldn't be cached
> due to size and overall volume but thumbnails should, which is why I strip
> cookies from the thumbnails. These varnishtop commands break out /images
> and /images/thumb client requests, showing IMHO too many regular images
> being cached and nowhere near enough thumbnails:
>
> # varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/" and not ReqURL ~
> "/images/thumb"'
>
>    349.47 VCL_call       HASH
>    349.47 VCL_call       RECV
>    349.47 VCL_call       DELIVER
>    207.22 VCL_call       HIT
>    116.40 VCL_call       MISS
>    116.30 VCL_call       PASS
>
> # varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/thumb"'
>
>   1859.60 VCL_call       HASH
>   1859.60 VCL_call       RECV
>   1859.60 VCL_call       DELIVER
>   1424.83 VCL_call       MISS
>    422.84 VCL_call       HIT
>    218.82 VCL_call       PASS
>
> I'm still poking around trying to correlate caching of other types of URLs
> based on whether or not the requests have cookies, if Cache-Control gets
> returned, etc. but I just wanted to reply with this info. I do appreciate
> the responses I'm getting! :)
>
>
> -----Original Message-----
> From: Dridi Boukelmoune [mailto:dridi at varni.sh]
> Sent: Friday, December 9, 2016 10:11 AM
> To: Justin Lloyd <justinl at arena.net>
> Cc: Dag Haavi Finstad <daghf at varnish-software.com>;
> varnish-misc at varnish-cache.org
> Subject: Re: Hit ratio dropped significantly after recent upgrades
>
> > To reiterate on a point in another of my responses in this thread, I
> think it may be something about MediaWiki thumbnail images not being cached
> properly despite our current VCL in that regard not having changed from how
> it worked prior to the upgrade during which time we were seeing a very high
> (86%-ish) hit ratio from the same formula.
>
> To reiterate on a point I made on a couple occasions, it's time to give
> varnishlog a spin. Too much focus on VCL, and not enough on what's
> happening.
>
> Dridi
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161213/751b69cb/attachment.html>

From justinl at arena.net  Tue Dec 13 14:23:56 2016
From: justinl at arena.net (Justin Lloyd)
Date: Tue, 13 Dec 2016 14:23:56 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
Message-ID: <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>

Here?s a typical varnishlog miss for a thumbnail image, appropriately sanitized. I can provide more if it helps

https://gist.github.com/Calygos/ca7906da005569046a7031d1fcaa6372


From: Guillaume Quintard [mailto:guillaume at varnish-software.com]
Sent: Tuesday, December 13, 2016 12:17 AM
To: Justin Lloyd <justinl at arena.net>
Cc: Dridi Boukelmoune <dridi at varni.sh>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

Can you pastebin the req+bereq transactions in varnishlog, related to such a miss?

--
Guillaume Quintard

On Tue, Dec 13, 2016 at 3:37 AM, Justin Lloyd <justinl at arena.net<mailto:justinl at arena.net>> wrote:
To follow up on my last email from Friday, at this point the problem boils down to one thing that I've not been able to determine: Why are far fewer things being cached now than before the upgrade?

1. Cookies don't seem to be the problem. Most appear to be Google Analytics (as opposed to session), which are being unset by vcl_recv.

2. varnishlog/varnishtop shows many thumbnail URLs being missed and virtually none are requested with a no-cache cache-control header. Is it possible to use these tools determine if they (or any URLs for that matter) are being cached following a miss-deliver sequence? There are about 1.5m thumbnail files totaling around 30 GB, which prior to the upgrades wasn't an issue, and I don't think it is now since there are only a few expires and purges per minute and no nukes at all. Varnish is only using about 2 GB out of the 8 GB allocated to it, where it used to use all 8 GB and have lots of nukes and far fewer expires, so it's not a memory constraint.

Could there be some other resource limitation I'm hitting without knowing it (nothing in any logs I've seen)? Everything else I could think of so far seems fine, e.g. open files, threads, tcp connections.


-----Original Message-----
From: varnish-misc-bounces+justinl=arena.net at varnish-cache.org<mailto:arena.net at varnish-cache.org> [mailto:varnish-misc-bounces+justinl<mailto:varnish-misc-bounces%2Bjustinl>=arena.net at varnish-cache.org<mailto:arena.net at varnish-cache.org>] On Behalf Of Justin Lloyd
Sent: Friday, December 9, 2016 11:19 AM
To: Dridi Boukelmoune <dridi at varni.sh<mailto:dridi at varni.sh>>
Cc: varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
Subject: RE: Hit ratio dropped significantly after recent upgrades

I really am looking at what's happening as well. I have been looking at both varnishlog and varnishtop and I see a lot of thumbnail image requests being sent to the backend when there is still plenty of room for them in the cache, so even though there are a lot of thumbnail images, I shouldn't see so many backend requests for them. As I previously mentioned, I give Varnish 8 GB and it used to stay full (based on RSS usage and looking at nukes vs. expires) but now it hovers around only about 2 GB used. A related statistics is that there used to be 600-700k objects in Varnish (based on our graphs of MAIN.n_object via Collectd's varnish-default-struct.objects-object metric) but now there are only roughly 40-70k objects in Varnish at any given time. So it's definitely caching a lot fewer things than it was before the upgrade, and most of the requested URLs for requests that have cookies are for a lot of images and thumbnails. Images shouldn't be cached due to size and overall volume but thumbnails should, which is why I strip cookies from the thumbnails. These varnishtop commands break out /images and /images/thumb client requests, showing IMHO too many regular images being cached and nowhere near enough thumbnails:

# varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/" and not ReqURL ~ "/images/thumb"'

   349.47 VCL_call       HASH
   349.47 VCL_call       RECV
   349.47 VCL_call       DELIVER
   207.22 VCL_call       HIT
   116.40 VCL_call       MISS
   116.30 VCL_call       PASS

# varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/thumb"'

  1859.60 VCL_call       HASH
  1859.60 VCL_call       RECV
  1859.60 VCL_call       DELIVER
  1424.83 VCL_call       MISS
   422.84 VCL_call       HIT
   218.82 VCL_call       PASS

I'm still poking around trying to correlate caching of other types of URLs based on whether or not the requests have cookies, if Cache-Control gets returned, etc. but I just wanted to reply with this info. I do appreciate the responses I'm getting! :)


-----Original Message-----
From: Dridi Boukelmoune [mailto:dridi at varni.sh<mailto:dridi at varni.sh>]
Sent: Friday, December 9, 2016 10:11 AM
To: Justin Lloyd <justinl at arena.net<mailto:justinl at arena.net>>
Cc: Dag Haavi Finstad <daghf at varnish-software.com<mailto:daghf at varnish-software.com>>; varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
Subject: Re: Hit ratio dropped significantly after recent upgrades

> To reiterate on a point in another of my responses in this thread, I think it may be something about MediaWiki thumbnail images not being cached properly despite our current VCL in that regard not having changed from how it worked prior to the upgrade during which time we were seeing a very high (86%-ish) hit ratio from the same formula.

To reiterate on a point I made on a couple occasions, it's time to give varnishlog a spin. Too much focus on VCL, and not enough on what's happening.

Dridi
_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161213/ea502dfe/attachment.html>

From dridi at varni.sh  Tue Dec 13 14:39:52 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 13 Dec 2016 15:39:52 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <CABoVN9AmehF22ZLxjWpzYuk2HizzqvKpOt3Jz36zWJ6oDnYqXA@mail.gmail.com>

On Tue, Dec 13, 2016 at 3:23 PM, Justin Lloyd <justinl at arena.net> wrote:
> Here?s a typical varnishlog miss for a thumbnail image, appropriately
> sanitized. I can provide more if it helps
>
> https://gist.github.com/Calygos/ca7906da005569046a7031d1fcaa6372

After a quick look at the log, I was able to find which line [1] of
VCL is responsible for this behavior. Your backend doesn't say that
the resource is public, so you may have to configure mediawiki to mark
thumbnails as public. This is not something I can help with.

Cheers

[1] https://gist.github.com/Calygos/105957a997ea3bde6b8257a1f34bbd20#file-wiki-vcl-L144



From justinl at arena.net  Tue Dec 13 14:46:41 2016
From: justinl at arena.net (Justin Lloyd)
Date: Tue, 13 Dec 2016 14:46:41 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <CABoVN9AmehF22ZLxjWpzYuk2HizzqvKpOt3Jz36zWJ6oDnYqXA@mail.gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9AmehF22ZLxjWpzYuk2HizzqvKpOt3Jz36zWJ6oDnYqXA@mail.gmail.com>
Message-ID: <DM5PR10MB16275250A090B33C767FB786A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>

Hmm, interesting. I've never made the connection to that line, which is part of the recommended Varnish configuration for MediaWiki[1], but it's not been a problem in the past as far as I know. I'll definitely investigate this more deeply. Thank you!

[1] https://www.mediawiki.org/wiki/Manual:Varnish_caching


-----Original Message-----
From: Dridi Boukelmoune [mailto:dridi at varni.sh] 
Sent: Tuesday, December 13, 2016 6:40 AM
To: Justin Lloyd <justinl at arena.net>
Cc: Guillaume Quintard <guillaume at varnish-software.com>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

On Tue, Dec 13, 2016 at 3:23 PM, Justin Lloyd <justinl at arena.net> wrote:
> Here?s a typical varnishlog miss for a thumbnail image, appropriately 
> sanitized. I can provide more if it helps
>
> https://gist.github.com/Calygos/ca7906da005569046a7031d1fcaa6372

After a quick look at the log, I was able to find which line [1] of VCL is responsible for this behavior. Your backend doesn't say that the resource is public, so you may have to configure mediawiki to mark thumbnails as public. This is not something I can help with.

Cheers

[1] https://gist.github.com/Calygos/105957a997ea3bde6b8257a1f34bbd20#file-wiki-vcl-L144

From justinl at arena.net  Tue Dec 13 15:01:59 2016
From: justinl at arena.net (Justin Lloyd)
Date: Tue, 13 Dec 2016 15:01:59 +0000
Subject: Hit ratio dropped significantly after recent upgrades
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9AmehF22ZLxjWpzYuk2HizzqvKpOt3Jz36zWJ6oDnYqXA@mail.gmail.com> 
Message-ID: <DM5PR10MB16273E562869C86688B829D3A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>

Actually, unless I'm missing something, that line would only matter for requests with Authorization headers, which none of our responses have since we don?t use http authentication.

-----Original Message-----
From: Justin Lloyd 
Sent: Tuesday, December 13, 2016 6:47 AM
To: 'Dridi Boukelmoune' <dridi at varni.sh>
Cc: Guillaume Quintard <guillaume at varnish-software.com>; varnish-misc at varnish-cache.org
Subject: RE: Hit ratio dropped significantly after recent upgrades

Hmm, interesting. I've never made the connection to that line, which is part of the recommended Varnish configuration for MediaWiki[1], but it's not been a problem in the past as far as I know. I'll definitely investigate this more deeply. Thank you!

[1] https://www.mediawiki.org/wiki/Manual:Varnish_caching


-----Original Message-----
From: Dridi Boukelmoune [mailto:dridi at varni.sh]
Sent: Tuesday, December 13, 2016 6:40 AM
To: Justin Lloyd <justinl at arena.net>
Cc: Guillaume Quintard <guillaume at varnish-software.com>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

On Tue, Dec 13, 2016 at 3:23 PM, Justin Lloyd <justinl at arena.net> wrote:
> Here?s a typical varnishlog miss for a thumbnail image, appropriately 
> sanitized. I can provide more if it helps
>
> https://gist.github.com/Calygos/ca7906da005569046a7031d1fcaa6372

After a quick look at the log, I was able to find which line [1] of VCL is responsible for this behavior. Your backend doesn't say that the resource is public, so you may have to configure mediawiki to mark thumbnails as public. This is not something I can help with.

Cheers

[1] https://gist.github.com/Calygos/105957a997ea3bde6b8257a1f34bbd20#file-wiki-vcl-L144

From dridi at varni.sh  Tue Dec 13 15:10:01 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 13 Dec 2016 16:10:01 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB16273E562869C86688B829D3A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9AmehF22ZLxjWpzYuk2HizzqvKpOt3Jz36zWJ6oDnYqXA@mail.gmail.com>
 <DM5PR10MB16273E562869C86688B829D3A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <CABoVN9Av3FW-yMuX5s+gZjFV-D=Rd3OcvzYMcNYfKCyzcjYVvA@mail.gmail.com>

On Tue, Dec 13, 2016 at 4:01 PM, Justin Lloyd <justinl at arena.net> wrote:
> Actually, unless I'm missing something, that line would only matter for requests with Authorization headers, which none of our responses have since we don?t use http authentication.

I still have trouble getting the hang of reading. And I had seen this
one but for some reason missed it...



From dridi at varni.sh  Tue Dec 13 15:13:02 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 13 Dec 2016 16:13:02 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <CABoVN9Av3FW-yMuX5s+gZjFV-D=Rd3OcvzYMcNYfKCyzcjYVvA@mail.gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9AmehF22ZLxjWpzYuk2HizzqvKpOt3Jz36zWJ6oDnYqXA@mail.gmail.com>
 <DM5PR10MB16273E562869C86688B829D3A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9Av3FW-yMuX5s+gZjFV-D=Rd3OcvzYMcNYfKCyzcjYVvA@mail.gmail.com>
Message-ID: <CABoVN9BnJ9aJUQDknLRmPqV7z36YoJE83DryFBhR-M=2gj-Y1A@mail.gmail.com>

> I still have trouble getting the hang of reading. And I had seen this
> one but for some reason missed it...

Actually, this looks like the response is properly cached, otherwise
it'd be using the Transient storage.

Dridi



From justinl at arena.net  Tue Dec 13 15:20:02 2016
From: justinl at arena.net (Justin Lloyd)
Date: Tue, 13 Dec 2016 15:20:02 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <CABoVN9BnJ9aJUQDknLRmPqV7z36YoJE83DryFBhR-M=2gj-Y1A@mail.gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9AmehF22ZLxjWpzYuk2HizzqvKpOt3Jz36zWJ6oDnYqXA@mail.gmail.com>
 <DM5PR10MB16273E562869C86688B829D3A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9Av3FW-yMuX5s+gZjFV-D=Rd3OcvzYMcNYfKCyzcjYVvA@mail.gmail.com>
 <CABoVN9BnJ9aJUQDknLRmPqV7z36YoJE83DryFBhR-M=2gj-Y1A@mail.gmail.com>
Message-ID: <DM5PR10MB1627D3D0BEF5A368FC5644ACA19B0@DM5PR10MB1627.namprd10.prod.outlook.com>

That's really the crux of my problem, as I tried to summarize in my last big email though it was longer than it probably should have been. Is it possible to determine from varnishlog whether/which requests are being missed and fetched but not cached? There's plenty of room in the cache, so I'm at a loss to determine why more stuff isn't being or staying cached long enough to result in a good hit rate like I saw prior to my recent upgrades.

I get the feeling that this will turn out to be either something very simple in the Varnish and/or MediaWiki configuration or else a bug somewhere, though I'd lean towards the former as being more likely. 

-----Original Message-----
From: Dridi Boukelmoune [mailto:dridi at varni.sh] 
Sent: Tuesday, December 13, 2016 7:13 AM
To: Justin Lloyd <justinl at arena.net>
Cc: Guillaume Quintard <guillaume at varnish-software.com>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

> I still have trouble getting the hang of reading. And I had seen this 
> one but for some reason missed it...

Actually, this looks like the response is properly cached, otherwise it'd be using the Transient storage.

Dridi

From dridi at varni.sh  Tue Dec 13 15:42:01 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 13 Dec 2016 16:42:01 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB1627D3D0BEF5A368FC5644ACA19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9AmehF22ZLxjWpzYuk2HizzqvKpOt3Jz36zWJ6oDnYqXA@mail.gmail.com>
 <DM5PR10MB16273E562869C86688B829D3A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9Av3FW-yMuX5s+gZjFV-D=Rd3OcvzYMcNYfKCyzcjYVvA@mail.gmail.com>
 <CABoVN9BnJ9aJUQDknLRmPqV7z36YoJE83DryFBhR-M=2gj-Y1A@mail.gmail.com>
 <DM5PR10MB1627D3D0BEF5A368FC5644ACA19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <CABoVN9C58h6+4VddK6j_CF5fiXfavq2kJBhUT6duS6jCU-JTUg@mail.gmail.com>

On Tue, Dec 13, 2016 at 4:20 PM, Justin Lloyd <justinl at arena.net> wrote:
> That's really the crux of my problem, as I tried to summarize in my last big email though it was longer than it probably should have been. Is it possible to determine from varnishlog whether/which requests are being missed and fetched but not cached?

There are more things you can see with varnishlog like hashes for
instance, but you could start by logging markers in your VCL to see
which branches you are taking.

Example:

    if (beresp.ttl <= 0s) {
        std.log("negative ttl");
        set beresp.uncacheable = true;
        return (deliver);
    }

And do the same for all branches, and see which code is executed.

> I get the feeling that this will turn out to be either something very simple in the Varnish and/or MediaWiki configuration or else a bug somewhere, though I'd lean towards the former as being more likely.

If you can reproduce it in lab conditions where you are the only user,
look for a transaction containing:

    - VCL_return hash
    - VCL_call HASH
    - VCL_return lookup
    - VCL_call PASS

And then look for the previous transaction on the same resource.

Dridi



From justinl at arena.net  Tue Dec 13 16:36:51 2016
From: justinl at arena.net (Justin Lloyd)
Date: Tue, 13 Dec 2016 16:36:51 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <CABoVN9C58h6+4VddK6j_CF5fiXfavq2kJBhUT6duS6jCU-JTUg@mail.gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9AmehF22ZLxjWpzYuk2HizzqvKpOt3Jz36zWJ6oDnYqXA@mail.gmail.com>
 <DM5PR10MB16273E562869C86688B829D3A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9Av3FW-yMuX5s+gZjFV-D=Rd3OcvzYMcNYfKCyzcjYVvA@mail.gmail.com>
 <CABoVN9BnJ9aJUQDknLRmPqV7z36YoJE83DryFBhR-M=2gj-Y1A@mail.gmail.com>
 <DM5PR10MB1627D3D0BEF5A368FC5644ACA19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9C58h6+4VddK6j_CF5fiXfavq2kJBhUT6duS6jCU-JTUg@mail.gmail.com>
Message-ID: <DM5PR10MB1627322D029A65E6133FB910A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>

Yep! Getting tons of uncacheable responses, including lots of /images/thumb URLs, due to negative TTLs. Not sure yet what the best way to handle this is since I'm sure this is done for a reason, so I'll need to dig back in to the MW code and possibly hit up the devs on IRC again. Thanks! I'll let you know how this goes.

Justin

-----Original Message-----
From: Dridi Boukelmoune [mailto:dridi at varni.sh] 
Sent: Tuesday, December 13, 2016 7:42 AM
To: Justin Lloyd <justinl at arena.net>
Cc: Guillaume Quintard <guillaume at varnish-software.com>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

On Tue, Dec 13, 2016 at 4:20 PM, Justin Lloyd <justinl at arena.net> wrote:
> That's really the crux of my problem, as I tried to summarize in my last big email though it was longer than it probably should have been. Is it possible to determine from varnishlog whether/which requests are being missed and fetched but not cached?

There are more things you can see with varnishlog like hashes for instance, but you could start by logging markers in your VCL to see which branches you are taking.

Example:

    if (beresp.ttl <= 0s) {
        std.log("negative ttl");
        set beresp.uncacheable = true;
        return (deliver);
    }

And do the same for all branches, and see which code is executed.

> I get the feeling that this will turn out to be either something very simple in the Varnish and/or MediaWiki configuration or else a bug somewhere, though I'd lean towards the former as being more likely.

If you can reproduce it in lab conditions where you are the only user, look for a transaction containing:

    - VCL_return hash
    - VCL_call HASH
    - VCL_return lookup
    - VCL_call PASS

And then look for the previous transaction on the same resource.

Dridi

From fgtham at gmail.com  Tue Dec 13 20:13:18 2016
From: fgtham at gmail.com (Florian Tham)
Date: Tue, 13 Dec 2016 21:13:18 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <158f9d15f30.275e.6038e5157c3d20148272845607c0f0ea@gmail.com>

The log shows that the fetched object is introduced into the cache with 
both TTL and grace time set to 120s each:

    --  VCL_call       BACKEND_RESPONSE
    --  TTL            VCL 120 120 0 1481637557
    --  VCL_return     deliver
    --  Storage        malloc s0

It would be interesting to see if a subsequent request to the same URL 
within less than 4 minutes would yield another miss or not.

Regards,

Florian


Am 13. Dezember 2016 15:27:16 schrieb Justin Lloyd <justinl at arena.net>:

> Here?s a typical varnishlog miss for a thumbnail image, appropriately 
> sanitized. I can provide more if it helps
>
> https://gist.github.com/Calygos/ca7906da005569046a7031d1fcaa6372
>
>
> From: Guillaume Quintard [mailto:guillaume at varnish-software.com]
> Sent: Tuesday, December 13, 2016 12:17 AM
> To: Justin Lloyd <justinl at arena.net>
> Cc: Dridi Boukelmoune <dridi at varni.sh>; varnish-misc at varnish-cache.org
> Subject: Re: Hit ratio dropped significantly after recent upgrades
>
> Can you pastebin the req+bereq transactions in varnishlog, related to such 
> a miss?
>
> --
> Guillaume Quintard
>
> On Tue, Dec 13, 2016 at 3:37 AM, Justin Lloyd 
> <justinl at arena.net<mailto:justinl at arena.net>> wrote:
> To follow up on my last email from Friday, at this point the problem boils 
> down to one thing that I've not been able to determine: Why are far fewer 
> things being cached now than before the upgrade?
>
> 1. Cookies don't seem to be the problem. Most appear to be Google Analytics 
> (as opposed to session), which are being unset by vcl_recv.
>
> 2. varnishlog/varnishtop shows many thumbnail URLs being missed and 
> virtually none are requested with a no-cache cache-control header. Is it 
> possible to use these tools determine if they (or any URLs for that matter) 
> are being cached following a miss-deliver sequence? There are about 1.5m 
> thumbnail files totaling around 30 GB, which prior to the upgrades wasn't 
> an issue, and I don't think it is now since there are only a few expires 
> and purges per minute and no nukes at all. Varnish is only using about 2 GB 
> out of the 8 GB allocated to it, where it used to use all 8 GB and have 
> lots of nukes and far fewer expires, so it's not a memory constraint.
>
> Could there be some other resource limitation I'm hitting without knowing 
> it (nothing in any logs I've seen)? Everything else I could think of so far 
> seems fine, e.g. open files, threads, tcp connections.
>
>
> -----Original Message-----
> From: 
> varnish-misc-bounces+justinl=arena.net at varnish-cache.org<mailto:arena.net at varnish-cache.org> 
> [mailto:varnish-misc-bounces+justinl<mailto:varnish-misc-bounces%2Bjustinl>=arena.net at varnish-cache.org<mailto:arena.net at varnish-cache.org>] 
> On Behalf Of Justin Lloyd
> Sent: Friday, December 9, 2016 11:19 AM
> To: Dridi Boukelmoune <dridi at varni.sh<mailto:dridi at varni.sh>>
> Cc: varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> Subject: RE: Hit ratio dropped significantly after recent upgrades
>
> I really am looking at what's happening as well. I have been looking at 
> both varnishlog and varnishtop and I see a lot of thumbnail image requests 
> being sent to the backend when there is still plenty of room for them in 
> the cache, so even though there are a lot of thumbnail images, I shouldn't 
> see so many backend requests for them. As I previously mentioned, I give 
> Varnish 8 GB and it used to stay full (based on RSS usage and looking at 
> nukes vs. expires) but now it hovers around only about 2 GB used. A related 
> statistics is that there used to be 600-700k objects in Varnish (based on 
> our graphs of MAIN.n_object via Collectd's 
> varnish-default-struct.objects-object metric) but now there are only 
> roughly 40-70k objects in Varnish at any given time. So it's definitely 
> caching a lot fewer things than it was before the upgrade, and most of the 
> requested URLs for requests that have cookies are for a lot of images and 
> thumbnails. Images shouldn't be cached due to size and overall volume but 
> thumbnails should, which is why I strip cookies from the thumbnails. These 
> varnishtop commands break out /images and /images/thumb client requests, 
> showing IMHO too many regular images being cached and nowhere near enough 
> thumbnails:
>
> # varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/" and not ReqURL ~ 
> "/images/thumb"'
>
>    349.47 VCL_call       HASH
>    349.47 VCL_call       RECV
>    349.47 VCL_call       DELIVER
>    207.22 VCL_call       HIT
>    116.40 VCL_call       MISS
>    116.30 VCL_call       PASS
>
> # varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/thumb"'
>
>   1859.60 VCL_call       HASH
>   1859.60 VCL_call       RECV
>   1859.60 VCL_call       DELIVER
>   1424.83 VCL_call       MISS
>    422.84 VCL_call       HIT
>    218.82 VCL_call       PASS
>
> I'm still poking around trying to correlate caching of other types of URLs 
> based on whether or not the requests have cookies, if Cache-Control gets 
> returned, etc. but I just wanted to reply with this info. I do appreciate 
> the responses I'm getting! :)
>
>
> -----Original Message-----
> From: Dridi Boukelmoune [mailto:dridi at varni.sh<mailto:dridi at varni.sh>]
> Sent: Friday, December 9, 2016 10:11 AM
> To: Justin Lloyd <justinl at arena.net<mailto:justinl at arena.net>>
> Cc: Dag Haavi Finstad 
> <daghf at varnish-software.com<mailto:daghf at varnish-software.com>>; 
> varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> Subject: Re: Hit ratio dropped significantly after recent upgrades
>
>> To reiterate on a point in another of my responses in this thread, I think 
>> it may be something about MediaWiki thumbnail images not being cached 
>> properly despite our current VCL in that regard not having changed from how 
>> it worked prior to the upgrade during which time we were seeing a very high 
>> (86%-ish) hit ratio from the same formula.
>
> To reiterate on a point I made on a couple occasions, it's time to give 
> varnishlog a spin. Too much focus on VCL, and not enough on what's happening.
>
> Dridi
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> ----------
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc





From justinl at arena.net  Tue Dec 13 20:19:12 2016
From: justinl at arena.net (Justin Lloyd)
Date: Tue, 13 Dec 2016 20:19:12 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <158f9d15f30.275e.6038e5157c3d20148272845607c0f0ea@gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <158f9d15f30.275e.6038e5157c3d20148272845607c0f0ea@gmail.com>
Message-ID: <DM5PR10MB162718E58B8CEEA06CF5DDF5A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>

Based on this conversation, I added a 1h TTL to thumbnail images in vcl_backend_response and that has gotten my hit ratio up to about 55-60% depending on how you calculate it (hit/miss values vs. frontend/backend connections), with up to about 72k objects in memory, up from about 60k max before, though before the upgrades it was more like 600-700k objects.

It's been an hour now and I'm seeing a spike in expired objects and a drop in the number of objects, so I'll probably increase the TTL until I find a sweet spot. I don't think there's any risk since thumbnails don't change often, so even a max of 48h may be reasonable.  So I'll do more testing today and see how things go.

Thanks!


-----Original Message-----
From: Florian Tham [mailto:fgtham at gmail.com] 
Sent: Tuesday, December 13, 2016 12:13 PM
To: Justin Lloyd <justinl at arena.net>
Cc: varnish-misc at varnish-cache.org
Subject: RE: Hit ratio dropped significantly after recent upgrades

The log shows that the fetched object is introduced into the cache with both TTL and grace time set to 120s each:

    --  VCL_call       BACKEND_RESPONSE
    --  TTL            VCL 120 120 0 1481637557
    --  VCL_return     deliver
    --  Storage        malloc s0

It would be interesting to see if a subsequent request to the same URL within less than 4 minutes would yield another miss or not.

Regards,

Florian


Am 13. Dezember 2016 15:27:16 schrieb Justin Lloyd <justinl at arena.net>:

> Here?s a typical varnishlog miss for a thumbnail image, appropriately 
> sanitized. I can provide more if it helps
>
> https://gist.github.com/Calygos/ca7906da005569046a7031d1fcaa6372
>
>
> From: Guillaume Quintard [mailto:guillaume at varnish-software.com]
> Sent: Tuesday, December 13, 2016 12:17 AM
> To: Justin Lloyd <justinl at arena.net>
> Cc: Dridi Boukelmoune <dridi at varni.sh>; varnish-misc at varnish-cache.org
> Subject: Re: Hit ratio dropped significantly after recent upgrades
>
> Can you pastebin the req+bereq transactions in varnishlog, related to 
> such a miss?
>
> --
> Guillaume Quintard
>
> On Tue, Dec 13, 2016 at 3:37 AM, Justin Lloyd 
> <justinl at arena.net<mailto:justinl at arena.net>> wrote:
> To follow up on my last email from Friday, at this point the problem 
> boils down to one thing that I've not been able to determine: Why are 
> far fewer things being cached now than before the upgrade?
>
> 1. Cookies don't seem to be the problem. Most appear to be Google 
> Analytics (as opposed to session), which are being unset by vcl_recv.
>
> 2. varnishlog/varnishtop shows many thumbnail URLs being missed and 
> virtually none are requested with a no-cache cache-control header. Is 
> it possible to use these tools determine if they (or any URLs for that 
> matter) are being cached following a miss-deliver sequence? There are 
> about 1.5m thumbnail files totaling around 30 GB, which prior to the 
> upgrades wasn't an issue, and I don't think it is now since there are 
> only a few expires and purges per minute and no nukes at all. Varnish 
> is only using about 2 GB out of the 8 GB allocated to it, where it 
> used to use all 8 GB and have lots of nukes and far fewer expires, so it's not a memory constraint.
>
> Could there be some other resource limitation I'm hitting without 
> knowing it (nothing in any logs I've seen)? Everything else I could 
> think of so far seems fine, e.g. open files, threads, tcp connections.
>
>
> -----Original Message-----
> From: 
> varnish-misc-bounces+justinl=arena.net at varnish-cache.org<mailto:arena.
> varnish-misc-bounces+net at varnish-cache.org>
> [mailto:varnish-misc-bounces+justinl<mailto:varnish-misc-bounces%2Bjus
> tinl>=arena.net at varnish-cache.org<mailto:arena.net at varnish-cache.org>]
> On Behalf Of Justin Lloyd
> Sent: Friday, December 9, 2016 11:19 AM
> To: Dridi Boukelmoune <dridi at varni.sh<mailto:dridi at varni.sh>>
> Cc: 
> varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> Subject: RE: Hit ratio dropped significantly after recent upgrades
>
> I really am looking at what's happening as well. I have been looking 
> at both varnishlog and varnishtop and I see a lot of thumbnail image 
> requests being sent to the backend when there is still plenty of room 
> for them in the cache, so even though there are a lot of thumbnail 
> images, I shouldn't see so many backend requests for them. As I 
> previously mentioned, I give Varnish 8 GB and it used to stay full 
> (based on RSS usage and looking at nukes vs. expires) but now it 
> hovers around only about 2 GB used. A related statistics is that there 
> used to be 600-700k objects in Varnish (based on our graphs of 
> MAIN.n_object via Collectd's varnish-default-struct.objects-object 
> metric) but now there are only roughly 40-70k objects in Varnish at 
> any given time. So it's definitely caching a lot fewer things than it 
> was before the upgrade, and most of the requested URLs for requests 
> that have cookies are for a lot of images and thumbnails. Images 
> shouldn't be cached due to size and overall volume but thumbnails 
> should, which is why I strip cookies from the thumbnails. These 
> varnishtop commands break out /images and /images/thumb client 
> requests, showing IMHO too many regular images being cached and 
> nowhere near enough
> thumbnails:
>
> # varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/" and not ReqURL ~ 
> "/images/thumb"'
>
>    349.47 VCL_call       HASH
>    349.47 VCL_call       RECV
>    349.47 VCL_call       DELIVER
>    207.22 VCL_call       HIT
>    116.40 VCL_call       MISS
>    116.30 VCL_call       PASS
>
> # varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/thumb"'
>
>   1859.60 VCL_call       HASH
>   1859.60 VCL_call       RECV
>   1859.60 VCL_call       DELIVER
>   1424.83 VCL_call       MISS
>    422.84 VCL_call       HIT
>    218.82 VCL_call       PASS
>
> I'm still poking around trying to correlate caching of other types of 
> URLs based on whether or not the requests have cookies, if 
> Cache-Control gets returned, etc. but I just wanted to reply with this 
> info. I do appreciate the responses I'm getting! :)
>
>
> -----Original Message-----
> From: Dridi Boukelmoune [mailto:dridi at varni.sh<mailto:dridi at varni.sh>]
> Sent: Friday, December 9, 2016 10:11 AM
> To: Justin Lloyd <justinl at arena.net<mailto:justinl at arena.net>>
> Cc: Dag Haavi Finstad
> <daghf at varnish-software.com<mailto:daghf at varnish-software.com>>;
> varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> Subject: Re: Hit ratio dropped significantly after recent upgrades
>
>> To reiterate on a point in another of my responses in this thread, I 
>> think it may be something about MediaWiki thumbnail images not being 
>> cached properly despite our current VCL in that regard not having 
>> changed from how it worked prior to the upgrade during which time we 
>> were seeing a very high
>> (86%-ish) hit ratio from the same formula.
>
> To reiterate on a point I made on a couple occasions, it's time to 
> give varnishlog a spin. Too much focus on VCL, and not enough on what's happening.
>
> Dridi
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> ----------
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From justinl at arena.net  Wed Dec 14 13:58:08 2016
From: justinl at arena.net (Justin Lloyd)
Date: Wed, 14 Dec 2016 13:58:08 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB162718E58B8CEEA06CF5DDF5A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <158f9d15f30.275e.6038e5157c3d20148272845607c0f0ea@gmail.com>
 <DM5PR10MB162718E58B8CEEA06CF5DDF5A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <DM5PR10MB1627377C2EDB651E5C650820A19A0@DM5PR10MB1627.namprd10.prod.outlook.com>

So after increasing the TTL for thumbnail images to 4 hours, the hit ratio got to 65-70%, objects in memory got up to around 150k before tapering off (due to lots of expirations starting after 4 hours, as to be expected) and slowly dipping back down to around 100k before starting on the upswing again. I'm continuing to test increasing the TTL, setting it to 24h and we'll see if we have any problems reported as a result in case that's too long, but at any rate we definitely appear to have found the smoking gun and I know where to tinker to try to better optimize things.

I'm not sure why this changed with the upgrades, whether it was something in MediaWiki or Varnish, but at least I know where to spend cycles on optimizations.

Thank you all very much for the help!

Justin
 

-----Original Message-----
From: varnish-misc-bounces+justinl=arena.net at varnish-cache.org [mailto:varnish-misc-bounces+justinl=arena.net at varnish-cache.org] On Behalf Of Justin Lloyd
Sent: Tuesday, December 13, 2016 12:19 PM
To: Florian Tham <fgtham at gmail.com>
Cc: varnish-misc at varnish-cache.org
Subject: RE: Hit ratio dropped significantly after recent upgrades

Based on this conversation, I added a 1h TTL to thumbnail images in vcl_backend_response and that has gotten my hit ratio up to about 55-60% depending on how you calculate it (hit/miss values vs. frontend/backend connections), with up to about 72k objects in memory, up from about 60k max before, though before the upgrades it was more like 600-700k objects.

It's been an hour now and I'm seeing a spike in expired objects and a drop in the number of objects, so I'll probably increase the TTL until I find a sweet spot. I don't think there's any risk since thumbnails don't change often, so even a max of 48h may be reasonable.  So I'll do more testing today and see how things go.

Thanks!


-----Original Message-----
From: Florian Tham [mailto:fgtham at gmail.com]
Sent: Tuesday, December 13, 2016 12:13 PM
To: Justin Lloyd <justinl at arena.net>
Cc: varnish-misc at varnish-cache.org
Subject: RE: Hit ratio dropped significantly after recent upgrades

The log shows that the fetched object is introduced into the cache with both TTL and grace time set to 120s each:

    --  VCL_call       BACKEND_RESPONSE
    --  TTL            VCL 120 120 0 1481637557
    --  VCL_return     deliver
    --  Storage        malloc s0

It would be interesting to see if a subsequent request to the same URL within less than 4 minutes would yield another miss or not.

Regards,

Florian


Am 13. Dezember 2016 15:27:16 schrieb Justin Lloyd <justinl at arena.net>:

> Here?s a typical varnishlog miss for a thumbnail image, appropriately 
> sanitized. I can provide more if it helps
>
> https://gist.github.com/Calygos/ca7906da005569046a7031d1fcaa6372
>
>
> From: Guillaume Quintard [mailto:guillaume at varnish-software.com]
> Sent: Tuesday, December 13, 2016 12:17 AM
> To: Justin Lloyd <justinl at arena.net>
> Cc: Dridi Boukelmoune <dridi at varni.sh>; varnish-misc at varnish-cache.org
> Subject: Re: Hit ratio dropped significantly after recent upgrades
>
> Can you pastebin the req+bereq transactions in varnishlog, related to 
> such a miss?
>
> --
> Guillaume Quintard
>
> On Tue, Dec 13, 2016 at 3:37 AM, Justin Lloyd 
> <justinl at arena.net<mailto:justinl at arena.net>> wrote:
> To follow up on my last email from Friday, at this point the problem 
> boils down to one thing that I've not been able to determine: Why are 
> far fewer things being cached now than before the upgrade?
>
> 1. Cookies don't seem to be the problem. Most appear to be Google 
> Analytics (as opposed to session), which are being unset by vcl_recv.
>
> 2. varnishlog/varnishtop shows many thumbnail URLs being missed and 
> virtually none are requested with a no-cache cache-control header. Is 
> it possible to use these tools determine if they (or any URLs for that
> matter) are being cached following a miss-deliver sequence? There are 
> about 1.5m thumbnail files totaling around 30 GB, which prior to the 
> upgrades wasn't an issue, and I don't think it is now since there are 
> only a few expires and purges per minute and no nukes at all. Varnish 
> is only using about 2 GB out of the 8 GB allocated to it, where it 
> used to use all 8 GB and have lots of nukes and far fewer expires, so it's not a memory constraint.
>
> Could there be some other resource limitation I'm hitting without 
> knowing it (nothing in any logs I've seen)? Everything else I could 
> think of so far seems fine, e.g. open files, threads, tcp connections.
>
>
> -----Original Message-----
> From: 
> varnish-misc-bounces+justinl=arena.net at varnish-cache.org<mailto:arena.
> varnish-misc-bounces+net at varnish-cache.org>
> [mailto:varnish-misc-bounces+justinl<mailto:varnish-misc-bounces%2Bjus
> tinl>=arena.net at varnish-cache.org<mailto:arena.net at varnish-cache.org>]
> On Behalf Of Justin Lloyd
> Sent: Friday, December 9, 2016 11:19 AM
> To: Dridi Boukelmoune <dridi at varni.sh<mailto:dridi at varni.sh>>
> Cc: 
> varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> Subject: RE: Hit ratio dropped significantly after recent upgrades
>
> I really am looking at what's happening as well. I have been looking 
> at both varnishlog and varnishtop and I see a lot of thumbnail image 
> requests being sent to the backend when there is still plenty of room 
> for them in the cache, so even though there are a lot of thumbnail 
> images, I shouldn't see so many backend requests for them. As I 
> previously mentioned, I give Varnish 8 GB and it used to stay full 
> (based on RSS usage and looking at nukes vs. expires) but now it 
> hovers around only about 2 GB used. A related statistics is that there 
> used to be 600-700k objects in Varnish (based on our graphs of 
> MAIN.n_object via Collectd's varnish-default-struct.objects-object
> metric) but now there are only roughly 40-70k objects in Varnish at 
> any given time. So it's definitely caching a lot fewer things than it 
> was before the upgrade, and most of the requested URLs for requests 
> that have cookies are for a lot of images and thumbnails. Images 
> shouldn't be cached due to size and overall volume but thumbnails 
> should, which is why I strip cookies from the thumbnails. These 
> varnishtop commands break out /images and /images/thumb client 
> requests, showing IMHO too many regular images being cached and 
> nowhere near enough
> thumbnails:
>
> # varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/" and not ReqURL ~ 
> "/images/thumb"'
>
>    349.47 VCL_call       HASH
>    349.47 VCL_call       RECV
>    349.47 VCL_call       DELIVER
>    207.22 VCL_call       HIT
>    116.40 VCL_call       MISS
>    116.30 VCL_call       PASS
>
> # varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/thumb"'
>
>   1859.60 VCL_call       HASH
>   1859.60 VCL_call       RECV
>   1859.60 VCL_call       DELIVER
>   1424.83 VCL_call       MISS
>    422.84 VCL_call       HIT
>    218.82 VCL_call       PASS
>
> I'm still poking around trying to correlate caching of other types of 
> URLs based on whether or not the requests have cookies, if 
> Cache-Control gets returned, etc. but I just wanted to reply with this 
> info. I do appreciate the responses I'm getting! :)
>
>
> -----Original Message-----
> From: Dridi Boukelmoune [mailto:dridi at varni.sh<mailto:dridi at varni.sh>]
> Sent: Friday, December 9, 2016 10:11 AM
> To: Justin Lloyd <justinl at arena.net<mailto:justinl at arena.net>>
> Cc: Dag Haavi Finstad
> <daghf at varnish-software.com<mailto:daghf at varnish-software.com>>;
> varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> Subject: Re: Hit ratio dropped significantly after recent upgrades
>
>> To reiterate on a point in another of my responses in this thread, I 
>> think it may be something about MediaWiki thumbnail images not being 
>> cached properly despite our current VCL in that regard not having 
>> changed from how it worked prior to the upgrade during which time we 
>> were seeing a very high
>> (86%-ish) hit ratio from the same formula.
>
> To reiterate on a point I made on a couple occasions, it's time to 
> give varnishlog a spin. Too much focus on VCL, and not enough on what's happening.
>
> Dridi
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> ----------
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

From dridi at varni.sh  Wed Dec 14 15:20:00 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 14 Dec 2016 16:20:00 +0100
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB1627377C2EDB651E5C650820A19A0@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <158f9d15f30.275e.6038e5157c3d20148272845607c0f0ea@gmail.com>
 <DM5PR10MB162718E58B8CEEA06CF5DDF5A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627377C2EDB651E5C650820A19A0@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <CABoVN9BYfpw99WpZYwhu20p8UNCn1WXOfTzMDFcB1C5pH=NQ5w@mail.gmail.com>

On Wed, Dec 14, 2016 at 2:58 PM, Justin Lloyd <justinl at arena.net> wrote:
> So after increasing the TTL for thumbnail images to 4 hours, the hit ratio got to 65-70%, objects in memory got up to around 150k before tapering off (due to lots of expirations starting after 4 hours, as to be expected) and slowly dipping back down to around 100k before starting on the upswing again. I'm continuing to test increasing the TTL, setting it to 24h and we'll see if we have any problems reported as a result in case that's too long, but at any rate we definitely appear to have found the smoking gun and I know where to tinker to try to better optimize things.
>
> I'm not sure why this changed with the upgrades, whether it was something in MediaWiki or Varnish, but at least I know where to spend cycles on optimizations.
>
> Thank you all very much for the help!

Hi,

Thanks for the feedback and glad to see things back to normal.
Consider sharing your findings with the MediaWiki folks as they will
likely know better how to deal with the thumbnails.

If images/thumbnails aren't changing often, you can safely increase
the TTL (ideally directly from MediaWiki) as long as you have an
invalidation strategy in place.

Cheers



From ronald.ploeger at gmx.de  Thu Dec 15 08:10:01 2016
From: ronald.ploeger at gmx.de (=?UTF-8?B?UGzDtmdlciwgUm9uYWxk?=)
Date: Thu, 15 Dec 2016 09:10:01 +0100
Subject: How to compare 'bereq.backend' and 'req.backend_hint' against a
 director/backend in Varnish 4.1
Message-ID: <a58d5212-156f-780e-f9fe-e083e3fd15cf@gmx.de>

We are in upgrading from Varnish 3 to Varnish 4.1 and we have 
experienced a problem when comparing 'bereq.backend' or 
'req.backend_hint' to a director or a backend.

We created Varnish Test Cases to demonstrate this behavior.

In Varnish 3 this tests runs successfully:

------------------------------------------------------------------------------------------------------
varnishtest "Test backends and directors"

server s1 {
   rxreq
   txresp -status 200
} -start


varnish v1 -vcl {
   backend b_1 {
     .host = "${s1_addr}";
     .port = "${s1_port}";
   }

   director d1 round-robin {
     { .backend = b_1; }
   }

   sub vcl_recv {
     set req.backend = d1;
   }

   sub vcl_fetch {
     if ( req.backend == d1 ) {
       set beresp.http.X-Backend-Fetch = "d1";
     }
   }

   sub vcl_deliver {
     if ( req.backend == d1 ) {
       set resp.http.X-Backend-Deliver = "d1";
     }
   }

} -start

client c1 {
   txreq
   rxresp
   expect resp.status == 200
   expect resp.http.X-Backend-Fetch == "d1"
   expect resp.http.X-Backend-Deliver == "d1"
} -run
------------------------------------------------------------------------------------------------------

In Varnish 4 the following adapted test:

------------------------------------------------------------------------------------------------------
varnishtest "Test backends and directors"

server s1 {
   rxreq
   txresp -status 200
} -start


varnish v1 -vcl {
   import std;
   import directors;
   backend b1 {
     .host = "${s1_addr}";
     .port = "${s1_port}";
   }

   sub vcl_init {
     new d1 = directors.round_robin();
     d1.add_backend(b1);
   }

   sub vcl_recv {
     set req.backend_hint = d1.backend();
   }

   sub vcl_backend_response {
     if ( bereq.backend == d1 ) {
       set beresp.http.X-Backend-Response = "d1";
     }
   }

   sub vcl_deliver {
     if ( req.backend_hint == d1 ) {
       set resp.http.X-Backend-Deliver = "d1";
     }
   }

} -start

client c1 {
   txreq
   rxresp
   expect resp.status == 200
   expect resp.http.X-Backend-Response == "d1"
   expect resp.http.X-Backend-Deliver == "d1"
} -run
------------------------------------------------------------------------------------------------------

fails with this message:

**** v1    0.4 CLI RX| Message from VCC-compiler:\n
**** v1    0.4 CLI RX| Backend not found: 'd1'\n
**** v1    0.4 CLI RX| ('<vcl.inline>' Line 21 Pos 27)\n
**** v1    0.4 CLI RX|     if ( bereq.backend == d1 ) {\n
**** v1    0.4 CLI RX| --------------------------##----\n

and when we comment out the 'vcl_backend_response' sub it fails with 
this message:

**** v1    0.5 CLI RX| Message from VCC-compiler:\n
**** v1    0.5 CLI RX| Backend not found: 'd1'\n
**** v1    0.5 CLI RX| ('<vcl.inline>' Line 28 Pos 30)\n
**** v1    0.5 CLI RX|     if ( req.backend_hint == d1 ) {\n
**** v1    0.5 CLI RX| -----------------------------##----\n


When we change this test to compare against the backend directly and not 
against the director the test compiles

------------------------------------------------------------------------------------------------------
varnishtest "Test backends and directors"

server s1 {
   rxreq
   txresp -status 200
} -start


varnish v1 -vcl {
   import std;
   import directors;
   backend b1 {
     .host = "${s1_addr}";
     .port = "${s1_port}";
   }

   sub vcl_init {
     new d1 = directors.round_robin();
     d1.add_backend(b1);
   }

   sub vcl_recv {
     set req.backend_hint = d1.backend();
   }

   sub vcl_backend_response {
     std.log("xxxxxxxx-vcl_backend_response: " + bereq.backend);
     if ( bereq.backend == b1 ) {
       set beresp.http.X-Backend-Response = "d1";
     }
   }

   sub vcl_deliver {
     std.log("xxxxxxxx-vcl_deliver: " + req.backend_hint);
     if ( req.backend_hint == b1 ) {
       set resp.http.X-Backend-Deliver = "d1";
     }
   }

} -start

client c1 {
   txreq
   rxresp
   expect resp.status == 200
   expect resp.http.X-Backend-Response == "d1"
   expect resp.http.X-Backend-Deliver == "d1"
} -run
------------------------------------------------------------------------------------------------------

The test also fails because the expectations of the client are not met:

**   c1    1.0 === expect resp.http.X-Backend-Response == "d1"
---- c1    1.0 EXPECT resp.http.X-Backend-Response (<undef>) == "d1" failed

and we can see in the log that 'bereq.backend' and 'req.backend_hint' 
are set to 'd1'

...
**** v1    1.0 vsl|       1002 VCL_call        b BACKEND_RESPONSE
**** v1    1.0 vsl|       1002 VCL_Log         b 
xxxxxxxx-vcl_backend_response: d1
**** v1    1.0 vsl|       1002 VCL_return      b deliver
...
**** v1    1.0 vsl|       1001 VCL_call        c DELIVER
**** v1    1.0 vsl|       1001 VCL_Log         c xxxxxxxx-vcl_deliver: d1
**** v1    1.0 vsl|       1001 VCL_return      c deliver
...

The question is how to compare 'bereq.backend' and 'req.backend_hint' 
properly against a director/backend.
Are we missing something or is this a bug?


Thanks,

Ronald




From a.rosenhagen at ffuenf.de  Thu Dec 15 09:07:52 2016
From: a.rosenhagen at ffuenf.de (Achim Rosenhagen)
Date: Thu, 15 Dec 2016 10:07:52 +0100
Subject: package sources for Ubuntu 16.04 LTS (xenial)
Message-ID: <58525D68.8080903@ffuenf.de>

Hi,

atm I don't see package sources for Ubuntu 16.04 LTS (xenial) at
https://repo.varnish-cache.org/ubuntu/dists/
Any ETA when those will be added?

Cheers,
Achim



From dridi at varni.sh  Thu Dec 15 09:26:12 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 15 Dec 2016 10:26:12 +0100
Subject: How to compare 'bereq.backend' and 'req.backend_hint' against a
 director/backend in Varnish 4.1
In-Reply-To: <a58d5212-156f-780e-f9fe-e083e3fd15cf@gmx.de>
References: <a58d5212-156f-780e-f9fe-e083e3fd15cf@gmx.de>
Message-ID: <CABoVN9C+po7F63SXHGmRM-isMybPWW=Y3Ua3mPoOFVsOhb47SA@mail.gmail.com>

On Thu, Dec 15, 2016 at 9:10 AM, Pl?ger, Ronald <ronald.ploeger at gmx.de> wrote:
> We are in upgrading from Varnish 3 to Varnish 4.1 and we have experienced a
> problem when comparing 'bereq.backend' or 'req.backend_hint' to a director
> or a backend.

Hi Ronald,

Nice to see some varnishtest usage! The main problem with your code is
that you are confusing directors with VMOD objects. Because the whole
thing is confusing, and we really need to come with good docs because
I can't keep on explaining it. We have bad docs [1] on the topic if you
are interested in the details, but since Varnish 4.0 the things that were
confusing in the code base have become visible to end-users...

> fails with this message:
>
> **** v1    0.4 CLI RX| Message from VCC-compiler:\n
> **** v1    0.4 CLI RX| Backend not found: 'd1'\n
> **** v1    0.4 CLI RX| ('<vcl.inline>' Line 21 Pos 27)\n
> **** v1    0.4 CLI RX|     if ( bereq.backend == d1 ) {\n
> **** v1    0.4 CLI RX| --------------------------##----\n

The type system:

- b1 is a backend
- d1 is an object, not a backend
- d1.backend() is a director (and also a backend)
- req.backend_hint takes a director (or a backend)
- bereq.backend takes a backend (or a director)

Why did I distinguish between the last two? Because of how backend
resolution [1] works by default.

What you can do instead is this:

    if ( bereq.backend == d1.backend() ) {
        ...
    }

It will work depending on the semantics of the VMOD object. For
objects defined in VMOD directors, it should work with all of them
except the hash director. It appears to fail in Varnish 4.1.3 on my
system. Please check with the latest 4.1 and open a bug if that's
still the case. It works on the master branch, I haven't tried the 5.0
release.

> The question is how to compare 'bereq.backend' and 'req.backend_hint'
> properly against a director/backend.
> Are we missing something or is this a bug?

See above, probably a bit of both!

Also please note that you can avoid the comparisons altogether if it's
only about keeping track of backend selection. You should have seen in
your logs that req.backend_hint would give you the director "d1" and
bereq.backend would give you the selected backend "b1".

This test cases passes on both 4.1.3 and master:

    varnishtest "Test backends and directors"

    server s1 {
        rxreq
        txresp
    } -start

    server s2 {
        rxreq
        txresp
    } -start

    varnish v1 -vcl+backend {
        import directors;

        sub vcl_init {
            new d1 = directors.round_robin();
            d1.add_backend(s1);
            d1.add_backend(s2);
        }

        sub vcl_recv {
            set req.backend_hint = d1.backend();
        }

        sub vcl_backend_response {
            set beresp.http.X-Backend-Response = beresp.backend;
        }

        sub vcl_deliver {
            set resp.http.X-Backend-Deliver = req.backend_hint;
        }
    } -start

    client c1 {
        txreq -url "/foo"
        rxresp
        expect resp.status == 200
        expect resp.http.X-Backend-Response == "s1"
        expect resp.http.X-Backend-Deliver  == "d1"

        txreq -url "/bar"
        rxresp
        expect resp.status == 200
        expect resp.http.X-Backend-Response == "s2"
        expect resp.http.X-Backend-Deliver  == "d1"
    } -run

Cheers,
Dridi

Head of the Backends&Directors department

[1] https://www.varnish-cache.org/docs/4.1/reference/directors.html



From dridi at varni.sh  Thu Dec 15 12:00:12 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 15 Dec 2016 13:00:12 +0100
Subject: package sources for Ubuntu 16.04 LTS (xenial)
In-Reply-To: <58525D68.8080903@ffuenf.de>
References: <58525D68.8080903@ffuenf.de>
Message-ID: <CABoVN9DRVtLvyjPrrhJOhsOdUVGm0py=zuDRtFmzOdjh7jswPQ@mail.gmail.com>

On Thu, Dec 15, 2016 at 10:07 AM, Achim Rosenhagen
<a.rosenhagen at ffuenf.de> wrote:
> Hi,
>
> atm I don't see package sources for Ubuntu 16.04 LTS (xenial) at
> https://repo.varnish-cache.org/ubuntu/dists/
> Any ETA when those will be added?

Hello,

No ETA that I know of, we may start shipping them with Varnish 5.1 but
I can't confirm.

Dridi



From ronald.ploeger at gmx.de  Fri Dec 16 07:37:52 2016
From: ronald.ploeger at gmx.de (=?UTF-8?B?UGzDtmdlciwgUm9uYWxk?=)
Date: Fri, 16 Dec 2016 08:37:52 +0100
Subject: How to compare 'bereq.backend' and 'req.backend_hint' against a
 director/backend in Varnish 4.1
Message-ID: <18d88903-cf69-79dd-8650-43d110d8589b@gmx.de>

Hi Dridi,

thank you for your quick answer.

 > Nice to see some varnishtest usage!

We do love varnishtest :-)

 > What you can do instead is this:
 >
 >     if ( bereq.backend == d1.backend() ) {
 >         ...
 >     }

 >
 > It will work depending on the semantics of the VMOD object. For
 > objects defined in VMOD directors, it should work with all of them
 > except the hash director. It appears to fail in Varnish 4.1.3 on my
 > system. Please check with the latest 4.1 and open a bug if that's
 > still the case. It works on the master branch, I haven't tried the 5.0
 > release.

We are testing against "varnishd (varnish-4.1.4 revision 4529ff7)" from 
"https://repo.varnish-cache.org/ubuntu/" and I adapted the test to use 
"d1.backend()"

------------------------------------------------------------------------------------------------------

varnishtest "Test backends and directors"

server s1 {
   rxreq
   txresp -status 200
} -start


varnish v1 -vcl {
   import std;
   import directors;
   backend b1 {
     .host = "${s1_addr}";
     .port = "${s1_port}";
   }

   sub vcl_init {
     new d1 = directors.round_robin();
     d1.add_backend(b1);
   }

   sub vcl_recv {
     set req.backend_hint = d1.backend();
   }

   sub vcl_backend_response {
     std.log("xxxxxxxx-vcl_backend_response-bereq: " + bereq.backend);
     std.log("xxxxxxxx-vcl_backend_response-beresp: " + beresp.backend);
     if ( bereq.backend == d1.backend() ) {
       set beresp.http.X-Backend-Response = "d1";
     }
   }

   sub vcl_deliver {
     std.log("xxxxxxxx-vcl_deliver: " + req.backend_hint);
     if ( req.backend_hint == d1.backend() ) {
       set resp.http.X-Backend-Deliver = "d1";
     }
   }

} -start

client c1 {
   txreq
   rxresp
   expect resp.status == 200
   expect resp.http.X-Backend-Response == "d1"
   expect resp.http.X-Backend-Deliver == "d1"
} -run
------------------------------------------------------------------------------------------------------

But this also does not compile:

**** v1    0.5 CLI RX| Message from VCC-compiler:\n
**** v1    0.5 CLI RX| Backend not found: 'd1.backend'\n
**** v1    0.5 CLI RX| ('<vcl.inline>' Line 22 Pos 27)\n
**** v1    0.5 CLI RX|     if ( bereq.backend == d1.backend() ) {\n
**** v1    0.5 CLI RX| --------------------------##########------\n

So I am going to open a bug/issue here 
"https://github.com/varnishcache/varnish-cache", right?


Thanks again and best regards,

Ronald





From dridi at varni.sh  Fri Dec 16 09:56:39 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 16 Dec 2016 10:56:39 +0100
Subject: How to compare 'bereq.backend' and 'req.backend_hint' against a
 director/backend in Varnish 4.1
In-Reply-To: <18d88903-cf69-79dd-8650-43d110d8589b@gmx.de>
References: <18d88903-cf69-79dd-8650-43d110d8589b@gmx.de>
Message-ID: <CABoVN9BZhwQ2vsH-hBRzGCAXpE2iKc0N9WD+ujMS9LPpMKq86A@mail.gmail.com>

> So I am going to open a bug/issue here
> "https://github.com/varnishcache/varnish-cache", right?

Yes please, and mention that it does work on the master branch. You
can quote me as @dridi on github.

I'd like to stress one more time that comparing bereq.backend to
d1.backend() is likely to always fail because in the backend context
you will should the resolved backend.

Cheers



From ronald.ploeger at gmx.de  Fri Dec 16 13:22:17 2016
From: ronald.ploeger at gmx.de (=?UTF-8?B?UGzDtmdlciwgUm9uYWxk?=)
Date: Fri, 16 Dec 2016 14:22:17 +0100
Subject: How to compare 'bereq.backend' and 'req.backend_hint' against a
 director/backend in Varnish 4.1
Message-ID: <d1a74b96-f596-a694-5d07-054ef63786a1@gmx.de>

Hi Dridi,

thanks for your comment.

> I'd like to stress one more time that comparing bereq.backend to
> d1.backend() is likely to always fail because in the backend context
> you will should the resolved backend.


 From the log output for:

std.log("xxxxxxxx-vcl_backend_response-bereq: " + bereq.backend);
std.log("xxxxxxxx-vcl_backend_response-beresp: " + beresp.backend);
...	
std.log("xxxxxxxx-vcl_deliver: " + req.backend_hint);


**** v1    1.0 vsl|       1002 VCL_Log         b xxxxxxxx-vcl_backend_response-bereq: d1
**** v1    1.0 vsl|       1002 VCL_Log         b xxxxxxxx-vcl_backend_response-beresp: b1
...
**** v1    1.0 vsl|       1001 VCL_Log         c xxxxxxxx-vcl_deliver: d1

I got the impression that both "req.backend_hint" and "bereq.backend" hold the director (because they both output "d1") and that only "beresp.backend" holds the backend (because it outputs "b1").
Therefore I assumed that a comparison against "d1.backend()" would work for "req.backend_hint" and "bereq.backend" but not for  "beresp.backend".

Best,
Ronald

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161216/760cab08/attachment.html>

From dridi at varni.sh  Fri Dec 16 14:16:14 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 16 Dec 2016 15:16:14 +0100
Subject: How to compare 'bereq.backend' and 'req.backend_hint' against a
 director/backend in Varnish 4.1
In-Reply-To: <d1a74b96-f596-a694-5d07-054ef63786a1@gmx.de>
References: <d1a74b96-f596-a694-5d07-054ef63786a1@gmx.de>
Message-ID: <CABoVN9DasS17twFBeO3i-q6CO5H+CHw6Z_9LrDi7hhqs=Pn1Sw@mail.gmail.com>

> I got the impression that both "req.backend_hint" and "bereq.backend" hold
> the director (because they both output "d1") and that only "beresp.backend"
> holds the backend (because it outputs "b1").

You're right, I meant beresp.backend and confused it with bereq.backend
in my previous response. I somehow mixed up your usage of bereq.backend
and my own test case using beresp.backend...

> Therefore I assumed that a comparison against "d1.backend()" would work for
> "req.backend_hint" and "bereq.backend" but not for  "beresp.backend".

That is correct!

Cheers

PS. I had to proof-read this one twice to avoid further confusion



From justinl at arena.net  Fri Dec 16 15:00:12 2016
From: justinl at arena.net (Justin Lloyd)
Date: Fri, 16 Dec 2016 15:00:12 +0000
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <CABoVN9BYfpw99WpZYwhu20p8UNCn1WXOfTzMDFcB1C5pH=NQ5w@mail.gmail.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <158f9d15f30.275e.6038e5157c3d20148272845607c0f0ea@gmail.com>
 <DM5PR10MB162718E58B8CEEA06CF5DDF5A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627377C2EDB651E5C650820A19A0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9BYfpw99WpZYwhu20p8UNCn1WXOfTzMDFcB1C5pH=NQ5w@mail.gmail.com>
Message-ID: <DM5PR10MB162777B878E6B6580B5C20FDA19C0@DM5PR10MB1627.namprd10.prod.outlook.com>

I actually have the TTL set to 48h now and it seems to work well, but I'm not sure if there's anything specific I should do regarding invalidation. In general, wikis don't allow for deleting or even changing images; you just upload new ones and the wiki software keeps previous versions in the revision history. However, I am concerned about invalidating cached images when new versions are uploaded so that the previous version is purged from the cache, but I'm not sure about the most appropriate way to handle that from a Varnish or MediaWiki perspective. If you have any thoughts on that, I'd appreciate it.

-----Original Message-----
From: Dridi Boukelmoune [mailto:dridi at varni.sh] 
Sent: Wednesday, December 14, 2016 7:20 AM
To: Justin Lloyd <justinl at arena.net>
Cc: Florian Tham <fgtham at gmail.com>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

On Wed, Dec 14, 2016 at 2:58 PM, Justin Lloyd <justinl at arena.net> wrote:
> So after increasing the TTL for thumbnail images to 4 hours, the hit ratio got to 65-70%, objects in memory got up to around 150k before tapering off (due to lots of expirations starting after 4 hours, as to be expected) and slowly dipping back down to around 100k before starting on the upswing again. I'm continuing to test increasing the TTL, setting it to 24h and we'll see if we have any problems reported as a result in case that's too long, but at any rate we definitely appear to have found the smoking gun and I know where to tinker to try to better optimize things.
>
> I'm not sure why this changed with the upgrades, whether it was something in MediaWiki or Varnish, but at least I know where to spend cycles on optimizations.
>
> Thank you all very much for the help!

Hi,

Thanks for the feedback and glad to see things back to normal.
Consider sharing your findings with the MediaWiki folks as they will likely know better how to deal with the thumbnails.

If images/thumbnails aren't changing often, you can safely increase the TTL (ideally directly from MediaWiki) as long as you have an invalidation strategy in place.

Cheers

From lagged at gmail.com  Fri Dec 16 19:26:52 2016
From: lagged at gmail.com (Andrei)
Date: Fri, 16 Dec 2016 13:26:52 -0600
Subject: Hit ratio dropped significantly after recent upgrades
In-Reply-To: <DM5PR10MB162777B878E6B6580B5C20FDA19C0@DM5PR10MB1627.namprd10.prod.outlook.com>
References: <DM5PR10MB1627825C3690AD1C8587F675A1850@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAChvjRC4aEW8r1yPnY3Qze15n7rNaznKy8xUz4Rx_T=QGy8puA@mail.gmail.com>
 <CABoVN9BZZUzdOLBcn8k4QX+HwehXpdGnGfNE8X-j9c5s8ksGNg@mail.gmail.com>
 <DM5PR10MB16275FC22B5EDEDA20D24755A1840@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAEh05VY7Jd_Pa1P8bejM1iem_+nPDaqrpcqrPUn6OP7z-xt7qQ@mail.gmail.com>
 <DM5PR10MB1627FE8DAB99DF289F6660B6A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB16276CA9AFC56B194658D378A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9B=CES6GogZSrkdWjEiS5ycm=7-0E56t=izd=E9epmbdA@mail.gmail.com>
 <DM5PR10MB1627F84BE5E7E35ABA1F8F32A1870@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627ECC0E6BF365F249A1C33A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CAJ6ZYQzSq3eWB9usabyF7pmRDdy5jr9G5bOEhpDWB81Bb+D79w@mail.gmail.com>
 <DM5PR10MB1627EC5D585E0B70F5438354A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <158f9d15f30.275e.6038e5157c3d20148272845607c0f0ea@gmail.com>
 <DM5PR10MB162718E58B8CEEA06CF5DDF5A19B0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <DM5PR10MB1627377C2EDB651E5C650820A19A0@DM5PR10MB1627.namprd10.prod.outlook.com>
 <CABoVN9BYfpw99WpZYwhu20p8UNCn1WXOfTzMDFcB1C5pH=NQ5w@mail.gmail.com>
 <DM5PR10MB162777B878E6B6580B5C20FDA19C0@DM5PR10MB1627.namprd10.prod.outlook.com>
Message-ID: <CAP+vvEtDPqG+Kad41UvTTczb7uEZR9_f+Km1WpprZBupEeOiVA@mail.gmail.com>

The best thing to do (imo) is to trigger the purges directly in media wiki.
I only did a quick search but they seem to already integrate easily with
Varnish, to send cache purge requests on updates/changes
https://m.mediawiki.org/wiki/Manual:Varnish_caching hope it helps!

On Dec 16, 2016 17:42, "Justin Lloyd" <justinl at arena.net> wrote:

> I actually have the TTL set to 48h now and it seems to work well, but I'm
> not sure if there's anything specific I should do regarding invalidation.
> In general, wikis don't allow for deleting or even changing images; you
> just upload new ones and the wiki software keeps previous versions in the
> revision history. However, I am concerned about invalidating cached images
> when new versions are uploaded so that the previous version is purged from
> the cache, but I'm not sure about the most appropriate way to handle that
> from a Varnish or MediaWiki perspective. If you have any thoughts on that,
> I'd appreciate it.
>
> -----Original Message-----
> From: Dridi Boukelmoune [mailto:dridi at varni.sh]
> Sent: Wednesday, December 14, 2016 7:20 AM
> To: Justin Lloyd <justinl at arena.net>
> Cc: Florian Tham <fgtham at gmail.com>; varnish-misc at varnish-cache.org
> Subject: Re: Hit ratio dropped significantly after recent upgrades
>
> On Wed, Dec 14, 2016 at 2:58 PM, Justin Lloyd <justinl at arena.net> wrote:
> > So after increasing the TTL for thumbnail images to 4 hours, the hit
> ratio got to 65-70%, objects in memory got up to around 150k before
> tapering off (due to lots of expirations starting after 4 hours, as to be
> expected) and slowly dipping back down to around 100k before starting on
> the upswing again. I'm continuing to test increasing the TTL, setting it to
> 24h and we'll see if we have any problems reported as a result in case
> that's too long, but at any rate we definitely appear to have found the
> smoking gun and I know where to tinker to try to better optimize things.
> >
> > I'm not sure why this changed with the upgrades, whether it was
> something in MediaWiki or Varnish, but at least I know where to spend
> cycles on optimizations.
> >
> > Thank you all very much for the help!
>
> Hi,
>
> Thanks for the feedback and glad to see things back to normal.
> Consider sharing your findings with the MediaWiki folks as they will
> likely know better how to deal with the thumbnails.
>
> If images/thumbnails aren't changing often, you can safely increase the
> TTL (ideally directly from MediaWiki) as long as you have an invalidation
> strategy in place.
>
> Cheers
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161216/e4b816c0/attachment.html>

From ronald.ploeger at gmx.de  Sat Dec 17 09:44:01 2016
From: ronald.ploeger at gmx.de (Ronald)
Date: Sat, 17 Dec 2016 10:44:01 +0100
Subject: Status of zipnish
Message-ID: <8410f80d-73da-ceda-4990-285a85c2155d@gmx.de>

Hi,

about a year ago Varnish Software announces "zipnish": 
https://www.varnish-software.com/varnish-software-launches-zipnish/

The last commit on the project 
https://github.com/varnish/zipnish/commits/master was more than half a 
year ago.

Is "zipnish" already dead? Does anyone can comment on the status?

Thanks,

Ronald




From bubonic.pestilence at gmail.com  Sun Dec 18 00:05:48 2016
From: bubonic.pestilence at gmail.com (Anton Berezhkov)
Date: Sun, 18 Dec 2016 03:05:48 +0300
Subject: Deliver on HIT, otherwise redirect using "503; Location: ..."
Message-ID: <EDC53F74-567A-433D-B191-41F209EB1632@gmail.com>

Hello.

Switched to Varnish from Nginx for additional functionality and better control of handling requests.
But still can't implement what i want. And I want simple behaviour "Redirect on MISS/PASS".
I want to use VC for deploying quick "cdn" servers for our mp4-video-servers (used for HTML5 players), without need to store all files on this quick (ssd, upto 2x480GB space, full database about 6TB).

Currently we have 6 servers with SATA HDDs and hitting iowait like a trucks :) 

Examples:
- Request -> Varnish -> HIT: serve it using Varnish.
- Request -> Varnish -> MISS: start caching data from backend, and instantly reply to client: `Location: http://backend/$req.url"
- Request -> Varnish -> UPDATE: see `-> MISS` behaviour.

From my perspective, i should do this "detach & reply redirect" somewhere in `vcl_miss` OR `vcl_backend_fetch`, because if i understood correctly https://www.varnish-cache.org/docs/4.1/reference/states.html, i need vcl_backend_response to keep run in background (as additional thread) while doing return(synth(...)) to redirect user.

Similiar thing is "hitting stale content while object is updating".
But in my case "replying redirect while object is updating".

Also, i pray to implement this without writing additional scripts, why? I could do external php/ruby checker/cache-pusher with nginx & etc. But scared by performance downgrade :(


From mark.staudinger at nyi.net  Sun Dec 18 00:58:29 2016
From: mark.staudinger at nyi.net (Mark Staudinger)
Date: Sat, 17 Dec 2016 19:58:29 -0500
Subject: Deliver on HIT, otherwise redirect using "503; Location: ..."
In-Reply-To: <EDC53F74-567A-433D-B191-41F209EB1632@gmail.com>
References: <EDC53F74-567A-433D-B191-41F209EB1632@gmail.com>
Message-ID: <op.ysm6brnnk1fmft@markair.local>

Hi Anton,

Have you looked into the "do_stream" feature of Varnish?  This will begin  
serving the content to the visitor without waiting for the entire object  
to be downloaded and stored in cache.  Set in vcl_backend_response.

https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl

Cheers,
Mark

On Sat, 17 Dec 2016 19:05:48 -0500, Anton Berezhkov  
<bubonic.pestilence at gmail.com> wrote:

> Hello.
>
> Switched to Varnish from Nginx for additional functionality and better  
> control of handling requests.
> But still can't implement what i want. And I want simple behaviour  
> "Redirect on MISS/PASS".
> I want to use VC for deploying quick "cdn" servers for our  
> mp4-video-servers (used for HTML5 players), without need to store all  
> files on this quick (ssd, upto 2x480GB space, full database about 6TB).
>
> Currently we have 6 servers with SATA HDDs and hitting iowait like a  
> trucks :)
>
> Examples:
> - Request -> Varnish -> HIT: serve it using Varnish.
> - Request -> Varnish -> MISS: start caching data from backend, and  
> instantly reply to client: `Location: http://backend/$req.url"
> - Request -> Varnish -> UPDATE: see `-> MISS` behaviour.
>
> From my perspective, i should do this "detach & reply redirect"  
> somewhere in `vcl_miss` OR `vcl_backend_fetch`, because if i understood  
> correctly https://www.varnish-cache.org/docs/4.1/reference/states.html,  
> i need vcl_backend_response to keep run in background (as additional  
> thread) while doing return(synth(...)) to redirect user.
>
> Similiar thing is "hitting stale content while object is updating".
> But in my case "replying redirect while object is updating".
>
> Also, i pray to implement this without writing additional scripts, why?  
> I could do external php/ruby checker/cache-pusher with nginx & etc. But  
> scared by performance downgrade :(
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From bubonic.pestilence at gmail.com  Sun Dec 18 01:05:39 2016
From: bubonic.pestilence at gmail.com (Anton Berezhkov)
Date: Sun, 18 Dec 2016 04:05:39 +0300
Subject: Deliver on HIT, otherwise redirect using "503; Location: ..."
In-Reply-To: <op.ysm6brnnk1fmft@markair.local>
References: <EDC53F74-567A-433D-B191-41F209EB1632@gmail.com>
 <op.ysm6brnnk1fmft@markair.local>
Message-ID: <97AA9480-E989-4E3C-88A3-0D58A251D5A9@gmail.com>

Hello. Yes, i already did it, and noticed change. But this not what i'm looking for.

As this option doesn't provide: "instantly reply to client Location...", i'm forced to return(deliver).

I'll repeat:
 - If MISS:
 - - Thread #1: start caching data into storage
 - - Main thread: reply to client: synth(503, "Moved Temporarily")

 - If UPDAING:
 - - Main thread: reply to client: synth(503, "Moved Temporarily")

 - If HIT:
 - - Main thread: serve cached data

And beresp.do_stream is just non-blocking IO. While i'm looking for "Detaching backend-to-cache-routine into different thread, while main thread responses to client 'Location: http://backend/$req.url'"

> On Dec 18, 2016, at 3:58 AM, Mark Staudinger <mark.staudinger at nyi.net> wrote:
> 
> Hi Anton,
> 
> Have you looked into the "do_stream" feature of Varnish?  This will begin serving the content to the visitor without waiting for the entire object to be downloaded and stored in cache.  Set in vcl_backend_response.
> 
> https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl
> 
> Cheers,
> Mark
> 
> On Sat, 17 Dec 2016 19:05:48 -0500, Anton Berezhkov <bubonic.pestilence at gmail.com> wrote:
> 
>> Hello.
>> 
>> Switched to Varnish from Nginx for additional functionality and better control of handling requests.
>> But still can't implement what i want. And I want simple behaviour "Redirect on MISS/PASS".
>> I want to use VC for deploying quick "cdn" servers for our mp4-video-servers (used for HTML5 players), without need to store all files on this quick (ssd, upto 2x480GB space, full database about 6TB).
>> 
>> Currently we have 6 servers with SATA HDDs and hitting iowait like a trucks :)
>> 
>> Examples:
>> - Request -> Varnish -> HIT: serve it using Varnish.
>> - Request -> Varnish -> MISS: start caching data from backend, and instantly reply to client: `Location: http://backend/$req.url"
>> - Request -> Varnish -> UPDATE: see `-> MISS` behaviour.
>> 
>> From my perspective, i should do this "detach & reply redirect" somewhere in `vcl_miss` OR `vcl_backend_fetch`, because if i understood correctly https://www.varnish-cache.org/docs/4.1/reference/states.html, i need vcl_backend_response to keep run in background (as additional thread) while doing return(synth(...)) to redirect user.
>> 
>> Similiar thing is "hitting stale content while object is updating".
>> But in my case "replying redirect while object is updating".
>> 
>> Also, i pray to implement this without writing additional scripts, why? I could do external php/ruby checker/cache-pusher with nginx & etc. But scared by performance downgrade :(
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc




From bubonic.pestilence at gmail.com  Sun Dec 18 01:09:31 2016
From: bubonic.pestilence at gmail.com (Anton Berezhkov)
Date: Sun, 18 Dec 2016 04:09:31 +0300
Subject: Deliver on HIT, otherwise redirect using "503; Location: ..."
In-Reply-To: <op.ysm6brnnk1fmft@markair.local>
References: <EDC53F74-567A-433D-B191-41F209EB1632@gmail.com>
 <op.ysm6brnnk1fmft@markair.local>
Message-ID: <C862F1C0-E0C3-4E53-88B1-8A25F180CF61@gmail.com>

This is how I semi-implemented: http://pastebin.com/drDP8JxP
Now i need to use script which will run "curi -I -X PUT <url-to-put-into-cache>".


> On Dec 18, 2016, at 3:58 AM, Mark Staudinger <mark.staudinger at nyi.net> wrote:
> 
> Hi Anton,
> 
> Have you looked into the "do_stream" feature of Varnish?  This will begin serving the content to the visitor without waiting for the entire object to be downloaded and stored in cache.  Set in vcl_backend_response.
> 
> https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl
> 
> Cheers,
> Mark
> 
> On Sat, 17 Dec 2016 19:05:48 -0500, Anton Berezhkov <bubonic.pestilence at gmail.com> wrote:
> 
>> Hello.
>> 
>> Switched to Varnish from Nginx for additional functionality and better control of handling requests.
>> But still can't implement what i want. And I want simple behaviour "Redirect on MISS/PASS".
>> I want to use VC for deploying quick "cdn" servers for our mp4-video-servers (used for HTML5 players), without need to store all files on this quick (ssd, upto 2x480GB space, full database about 6TB).
>> 
>> Currently we have 6 servers with SATA HDDs and hitting iowait like a trucks :)
>> 
>> Examples:
>> - Request -> Varnish -> HIT: serve it using Varnish.
>> - Request -> Varnish -> MISS: start caching data from backend, and instantly reply to client: `Location: http://backend/$req.url"
>> - Request -> Varnish -> UPDATE: see `-> MISS` behaviour.
>> 
>> From my perspective, i should do this "detach & reply redirect" somewhere in `vcl_miss` OR `vcl_backend_fetch`, because if i understood correctly https://www.varnish-cache.org/docs/4.1/reference/states.html, i need vcl_backend_response to keep run in background (as additional thread) while doing return(synth(...)) to redirect user.
>> 
>> Similiar thing is "hitting stale content while object is updating".
>> But in my case "replying redirect while object is updating".
>> 
>> Also, i pray to implement this without writing additional scripts, why? I could do external php/ruby checker/cache-pusher with nginx & etc. But scared by performance downgrade :(
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc




From japrice at gmail.com  Sun Dec 18 18:48:08 2016
From: japrice at gmail.com (Jason Price)
Date: Sun, 18 Dec 2016 13:48:08 -0500
Subject: Deliver on HIT, otherwise redirect using "503; Location: ..."
In-Reply-To: <C862F1C0-E0C3-4E53-88B1-8A25F180CF61@gmail.com>
References: <EDC53F74-567A-433D-B191-41F209EB1632@gmail.com>
 <op.ysm6brnnk1fmft@markair.local>
 <C862F1C0-E0C3-4E53-88B1-8A25F180CF61@gmail.com>
Message-ID: <CAChvjRCMyTGKJ1X-iVAsz_YD4FyhdyquV+q9CutmU9AqiLHPTQ@mail.gmail.com>

It would be possible to do this with varnish... but I have to ask... why
bother?

If the purpose is to offload the IO load, then varnish is good, but you
need to prime the cache... TBH, what I'd do first is put one or a pair of
varnish boxes really close to the overloaded box, and force all traffic to
that server through the close varnish boxes... using the do_stream feature,
you'll get stuff out there fairly quickly.

After that is working nicely, I'd layer in the further out varnish boxes
which interact with the near-varnish boxes to get their data.

This works well at scale since the local caches offer whatever's useful
local to them, and the 'near-varnish' boxes handle the 'global caching'
world.

This was how I arranged it at $PreviousGig and the outer CDN was getting a
85-90% cache hit ratio, and the inner tier was seeing 60% cache hit
ratio's.  (The inner tier's ratio will depend heavily on how many outer
tier's there are...)

On Sat, Dec 17, 2016 at 8:09 PM, Anton Berezhkov <
bubonic.pestilence at gmail.com> wrote:

> This is how I semi-implemented: http://pastebin.com/drDP8JxP
> Now i need to use script which will run "curi -I -X PUT
> <url-to-put-into-cache>".
>
>
> > On Dec 18, 2016, at 3:58 AM, Mark Staudinger <mark.staudinger at nyi.net>
> wrote:
> >
> > Hi Anton,
> >
> > Have you looked into the "do_stream" feature of Varnish?  This will
> begin serving the content to the visitor without waiting for the entire
> object to be downloaded and stored in cache.  Set in vcl_backend_response.
> >
> > https://github.com/mattiasgeniar/varnish-4.0-
> configuration-templates/blob/master/default.vcl
> >
> > Cheers,
> > Mark
> >
> > On Sat, 17 Dec 2016 19:05:48 -0500, Anton Berezhkov <
> bubonic.pestilence at gmail.com> wrote:
> >
> >> Hello.
> >>
> >> Switched to Varnish from Nginx for additional functionality and better
> control of handling requests.
> >> But still can't implement what i want. And I want simple behaviour
> "Redirect on MISS/PASS".
> >> I want to use VC for deploying quick "cdn" servers for our
> mp4-video-servers (used for HTML5 players), without need to store all files
> on this quick (ssd, upto 2x480GB space, full database about 6TB).
> >>
> >> Currently we have 6 servers with SATA HDDs and hitting iowait like a
> trucks :)
> >>
> >> Examples:
> >> - Request -> Varnish -> HIT: serve it using Varnish.
> >> - Request -> Varnish -> MISS: start caching data from backend, and
> instantly reply to client: `Location: http://backend/$req.url"
> >> - Request -> Varnish -> UPDATE: see `-> MISS` behaviour.
> >>
> >> From my perspective, i should do this "detach & reply redirect"
> somewhere in `vcl_miss` OR `vcl_backend_fetch`, because if i understood
> correctly https://www.varnish-cache.org/docs/4.1/reference/states.html, i
> need vcl_backend_response to keep run in background (as additional thread)
> while doing return(synth(...)) to redirect user.
> >>
> >> Similiar thing is "hitting stale content while object is updating".
> >> But in my case "replying redirect while object is updating".
> >>
> >> Also, i pray to implement this without writing additional scripts, why?
> I could do external php/ruby checker/cache-pusher with nginx & etc. But
> scared by performance downgrade :(
> >> _______________________________________________
> >> varnish-misc mailing list
> >> varnish-misc at varnish-cache.org
> >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161218/0bb52549/attachment.html>

From guillaume at varnish-software.com  Sun Dec 18 19:59:26 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Sun, 18 Dec 2016 20:59:26 +0100
Subject: Deliver on HIT, otherwise redirect using "503; Location: ..."
In-Reply-To: <CAChvjRCMyTGKJ1X-iVAsz_YD4FyhdyquV+q9CutmU9AqiLHPTQ@mail.gmail.com>
References: <EDC53F74-567A-433D-B191-41F209EB1632@gmail.com>
 <op.ysm6brnnk1fmft@markair.local>
 <C862F1C0-E0C3-4E53-88B1-8A25F180CF61@gmail.com>
 <CAChvjRCMyTGKJ1X-iVAsz_YD4FyhdyquV+q9CutmU9AqiLHPTQ@mail.gmail.com>
Message-ID: <CAJ6ZYQx5GuGcYFj7TwOmhMk-9NCpE+FOiH7McQdkcQfo1kz3hg@mail.gmail.com>

I think Jason is right in asking "why?". What do you want to achieve
specifically with this behavior?

Varnish has streaming and request coalescing, meaning a request can be
served as soon as data starts being available AND the backend doesn't
suffer from simultaneous misses on the same object. I feel that should
cover almost all your needs, so I'm curious about the use-case.

On Dec 18, 2016 20:27, "Jason Price" <japrice at gmail.com> wrote:

> It would be possible to do this with varnish... but I have to ask... why
> bother?
>
> If the purpose is to offload the IO load, then varnish is good, but you
> need to prime the cache... TBH, what I'd do first is put one or a pair of
> varnish boxes really close to the overloaded box, and force all traffic to
> that server through the close varnish boxes... using the do_stream feature,
> you'll get stuff out there fairly quickly.
>
> After that is working nicely, I'd layer in the further out varnish boxes
> which interact with the near-varnish boxes to get their data.
>
> This works well at scale since the local caches offer whatever's useful
> local to them, and the 'near-varnish' boxes handle the 'global caching'
> world.
>
> This was how I arranged it at $PreviousGig and the outer CDN was getting a
> 85-90% cache hit ratio, and the inner tier was seeing 60% cache hit
> ratio's.  (The inner tier's ratio will depend heavily on how many outer
> tier's there are...)
>
> On Sat, Dec 17, 2016 at 8:09 PM, Anton Berezhkov <
> bubonic.pestilence at gmail.com> wrote:
>
>> This is how I semi-implemented: http://pastebin.com/drDP8JxP
>> Now i need to use script which will run "curi -I -X PUT
>> <url-to-put-into-cache>".
>>
>>
>> > On Dec 18, 2016, at 3:58 AM, Mark Staudinger <mark.staudinger at nyi.net>
>> wrote:
>> >
>> > Hi Anton,
>> >
>> > Have you looked into the "do_stream" feature of Varnish?  This will
>> begin serving the content to the visitor without waiting for the entire
>> object to be downloaded and stored in cache.  Set in vcl_backend_response.
>> >
>> > https://github.com/mattiasgeniar/varnish-4.0-configuration-
>> templates/blob/master/default.vcl
>> >
>> > Cheers,
>> > Mark
>> >
>> > On Sat, 17 Dec 2016 19:05:48 -0500, Anton Berezhkov <
>> bubonic.pestilence at gmail.com> wrote:
>> >
>> >> Hello.
>> >>
>> >> Switched to Varnish from Nginx for additional functionality and better
>> control of handling requests.
>> >> But still can't implement what i want. And I want simple behaviour
>> "Redirect on MISS/PASS".
>> >> I want to use VC for deploying quick "cdn" servers for our
>> mp4-video-servers (used for HTML5 players), without need to store all files
>> on this quick (ssd, upto 2x480GB space, full database about 6TB).
>> >>
>> >> Currently we have 6 servers with SATA HDDs and hitting iowait like a
>> trucks :)
>> >>
>> >> Examples:
>> >> - Request -> Varnish -> HIT: serve it using Varnish.
>> >> - Request -> Varnish -> MISS: start caching data from backend, and
>> instantly reply to client: `Location: http://backend/$req.url"
>> >> - Request -> Varnish -> UPDATE: see `-> MISS` behaviour.
>> >>
>> >> From my perspective, i should do this "detach & reply redirect"
>> somewhere in `vcl_miss` OR `vcl_backend_fetch`, because if i understood
>> correctly https://www.varnish-cache.org/docs/4.1/reference/states.html,
>> i need vcl_backend_response to keep run in background (as additional
>> thread) while doing return(synth(...)) to redirect user.
>> >>
>> >> Similiar thing is "hitting stale content while object is updating".
>> >> But in my case "replying redirect while object is updating".
>> >>
>> >> Also, i pray to implement this without writing additional scripts,
>> why? I could do external php/ruby checker/cache-pusher with nginx & etc.
>> But scared by performance downgrade :(
>> >> _______________________________________________
>> >> varnish-misc mailing list
>> >> varnish-misc at varnish-cache.org
>> >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161218/463704a4/attachment.html>

From bubonic.pestilence at gmail.com  Sun Dec 18 20:21:41 2016
From: bubonic.pestilence at gmail.com (Anton Berezhkov)
Date: Sun, 18 Dec 2016 23:21:41 +0300
Subject: Deliver on HIT, otherwise redirect using "503; Location: ..."
In-Reply-To: <CAJ6ZYQx5GuGcYFj7TwOmhMk-9NCpE+FOiH7McQdkcQfo1kz3hg@mail.gmail.com>
References: <EDC53F74-567A-433D-B191-41F209EB1632@gmail.com>
 <op.ysm6brnnk1fmft@markair.local>
 <C862F1C0-E0C3-4E53-88B1-8A25F180CF61@gmail.com>
 <CAChvjRCMyTGKJ1X-iVAsz_YD4FyhdyquV+q9CutmU9AqiLHPTQ@mail.gmail.com>
 <CAJ6ZYQx5GuGcYFj7TwOmhMk-9NCpE+FOiH7McQdkcQfo1kz3hg@mail.gmail.com>
Message-ID: <EC9357AD-0F1F-4B9C-B1A2-BCC1DD94287F@gmail.com>

We have 6 servers with 6TB on each one (same video files). Currently they're hitting iowait limit of SATA disk (240ops total). At same time, each server can provide 500Mbit of guaranteed bandwidth.

With HDD restriction, each server provides about 320Mbit. There is also problem with fragmentation, which is caused by nature of HTML5 video players & HTTP (allowing to request partial data with Range header).

Before this moment, we were scaling horizontally, by duplicating this servers.

There is also option to get same server with 2x480GB SSDs. As i reasearched from nginx logs, 98% of daily traffic lays in ?800GB of files.

What i want to achieve: To build a varnish server with 2x480GB SSDs(no raid), with storage for varnish about 800GBs. Which will guarantedly fill all available bandwidth for a server.

Also, I built simple load-balancer, which monitors each server for current eth0 load (in Mbps) and decide to which one redirect (using HTTP Location header).

Request for Video -> LBServer: Find lowest loaded(1 of 6) & Redirect to LBNode -> LBN: serve request

To add new HDD-LBN, i need to setup server, sync videos, setup some additional software.

My wish: add new SSD-LBN, setup & sync varnish config, and it will build cached pool itself.

Why i need redirect?
1. It will offload bandwidth of SSD-LBN, pass-through will take bandwidth of both servers + still cause iowait problems on HDD-LBN.
2. It will "prove" that uncached video will be take from HDD-LBN which always have all videos.

Currently all LBN servers are hosted on OVH and we're good with them, especially because of low price :)

If you have any suggestions, i'll be glad to hear them :)

> On Dec 18, 2016, at 10:59 PM, Guillaume Quintard <guillaume at varnish-software.com> wrote:
> 
> I think Jason is right in asking "why?". What do you want to achieve specifically with this behavior? 
> 
> Varnish has streaming and request coalescing, meaning a request can be served as soon as data starts being available AND the backend doesn't suffer from simultaneous misses on the same object. I feel that should cover almost all your needs, so I'm curious about the use-case.
> 
> On Dec 18, 2016 20:27, "Jason Price" <japrice at gmail.com> wrote:
> It would be possible to do this with varnish... but I have to ask... why bother?
> 
> If the purpose is to offload the IO load, then varnish is good, but you need to prime the cache... TBH, what I'd do first is put one or a pair of varnish boxes really close to the overloaded box, and force all traffic to that server through the close varnish boxes... using the do_stream feature, you'll get stuff out there fairly quickly.
> 
> After that is working nicely, I'd layer in the further out varnish boxes which interact with the near-varnish boxes to get their data.
> 
> This works well at scale since the local caches offer whatever's useful local to them, and the 'near-varnish' boxes handle the 'global caching' world.
> 
> This was how I arranged it at $PreviousGig and the outer CDN was getting a 85-90% cache hit ratio, and the inner tier was seeing 60% cache hit ratio's.  (The inner tier's ratio will depend heavily on how many outer tier's there are...)
> 
> On Sat, Dec 17, 2016 at 8:09 PM, Anton Berezhkov <bubonic.pestilence at gmail.com> wrote:
> This is how I semi-implemented: http://pastebin.com/drDP8JxP
> Now i need to use script which will run "curi -I -X PUT <url-to-put-into-cache>".
> 
> 
> > On Dec 18, 2016, at 3:58 AM, Mark Staudinger <mark.staudinger at nyi.net> wrote:
> >
> > Hi Anton,
> >
> > Have you looked into the "do_stream" feature of Varnish?  This will begin serving the content to the visitor without waiting for the entire object to be downloaded and stored in cache.  Set in vcl_backend_response.
> >
> > https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl
> >
> > Cheers,
> > Mark
> >
> > On Sat, 17 Dec 2016 19:05:48 -0500, Anton Berezhkov <bubonic.pestilence at gmail.com> wrote:
> >
> >> Hello.
> >>
> >> Switched to Varnish from Nginx for additional functionality and better control of handling requests.
> >> But still can't implement what i want. And I want simple behaviour "Redirect on MISS/PASS".
> >> I want to use VC for deploying quick "cdn" servers for our mp4-video-servers (used for HTML5 players), without need to store all files on this quick (ssd, upto 2x480GB space, full database about 6TB).
> >>
> >> Currently we have 6 servers with SATA HDDs and hitting iowait like a trucks :)
> >>
> >> Examples:
> >> - Request -> Varnish -> HIT: serve it using Varnish.
> >> - Request -> Varnish -> MISS: start caching data from backend, and instantly reply to client: `Location: http://backend/$req.url"
> >> - Request -> Varnish -> UPDATE: see `-> MISS` behaviour.
> >>
> >> From my perspective, i should do this "detach & reply redirect" somewhere in `vcl_miss` OR `vcl_backend_fetch`, because if i understood correctly https://www.varnish-cache.org/docs/4.1/reference/states.html, i need vcl_backend_response to keep run in background (as additional thread) while doing return(synth(...)) to redirect user.
> >>
> >> Similiar thing is "hitting stale content while object is updating".
> >> But in my case "replying redirect while object is updating".
> >>
> >> Also, i pray to implement this without writing additional scripts, why? I could do external php/ruby checker/cache-pusher with nginx & etc. But scared by performance downgrade :(
> >> _______________________________________________
> >> varnish-misc mailing list
> >> varnish-misc at varnish-cache.org
> >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> 
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> 
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc




From mathias.fuerstenau at gfz-potsdam.de  Tue Dec 20 07:30:32 2016
From: mathias.fuerstenau at gfz-potsdam.de (=?utf-8?Q?Mathias_F=C3=BCrstenau?=)
Date: Tue, 20 Dec 2016 08:30:32 +0100
Subject: Varnish on Solaris 11
Message-ID: <2C6170FA-C932-46AA-A65A-BEB16772FAFA@gfz-potsdam.de>

Hello everyone!

I have a little bit trouble with Varnish on Solaris 11. The problem is that I have no config file for Varnish. I always have to start with "varnishd -f /etc/varnish/default.vcl -s malloc,1G -a 0.0.0.0:80 -p thread_pool_min=400?. I also have no Varnish service or something else. There is no varnish.service or varnish.params. There is also no varnish in init.d. It is very suspicious. I hope you can help.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161220/4a43a076/attachment.html>

From dridi at varni.sh  Tue Dec 20 08:50:55 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 20 Dec 2016 09:50:55 +0100
Subject: Varnish on Solaris 11
In-Reply-To: <2C6170FA-C932-46AA-A65A-BEB16772FAFA@gfz-potsdam.de>
References: <2C6170FA-C932-46AA-A65A-BEB16772FAFA@gfz-potsdam.de>
Message-ID: <CABoVN9BU1biVKr670LFv69hhOx=-H4Q24KwZEYhNub1a5nPAEQ@mail.gmail.com>

On Tue, Dec 20, 2016 at 8:30 AM, Mathias F?rstenau
<mathias.fuerstenau at gfz-potsdam.de> wrote:
> Hello everyone!
>
> I have a little bit trouble with Varnish on Solaris 11. The problem is that
> I have no config file for Varnish. I always have to start with "varnishd -f
> /etc/varnish/default.vcl -s malloc,1G -a 0.0.0.0:80 -p thread_pool_min=400?.
> I also have no Varnish service or something else. There is no
> varnish.service or varnish.params. There is also no varnish in init.d. It is
> very suspicious. I hope you can help.

Hello Mathias,

Yes, service files are located in the pkg-varnish-cache repo, and are
specific to Debian and Red Hat derivatives. I'm afraid we don't have
any for Solaris.

Dridi



From A.Hongens at netmatch.nl  Tue Dec 20 14:37:24 2016
From: A.Hongens at netmatch.nl (=?iso-8859-1?Q?Angelo_H=F6ngens?=)
Date: Tue, 20 Dec 2016 14:37:24 +0000
Subject: stale-if-error in varnish 4.1?
Message-ID: <cdda09905af442c083a8c3b113a8c6fb@netmatch.nl>

Hey guys and girls,

I got a question from one of my webdevs if and how varnish can do stale-if-error.

I can't find much about the subject. I found a document[1] stating: "Varnish 4.1 implements stale-while-revalidate for the first time, but not stale-if-error."

I found an article on fastly.com [2] dating from 2014 that says: "Fastly is excited to announce that as of today, we support stale-while-revalidate and stale-if-error.", so that would seem to contradict the previous statement. Fastly uses normal varnish right?

Googling some more I found another page on docs.fastly.com [3] that explains how to do stale-if-error: do some custom VCL hacking. They do stuff with 'stale.exists' and restarts. I've been working with varnish for a while, but it looks like my varnish 4.1 instance doesn't know the 'stale.exists' thingy and I have no idea how restarts work.

2 questions for you smart guys and girls:

- I native stale-if-error on the roadmap?
- How can I hack stale-if-error in my VCL using my varnish41 machines?



References: 

[1] https://varnishfoo.info/varnishfoo.pdf
[2] https://www.fastly.com/blog/stale-while-revalidate-stale-if-error-available-today
[3] https://docs.fastly.com/guides/performance-tuning/serving-stale-content

-- 
 
With kind regards,
 
 
Angelo H?ngens
 
Systems Administrator
 
------------------------------------------
NetMatch
travel technology solutions

Professor Donderstraat 46
5017 HL Tilburg
T: +31 (0)13 5811088
F: +31 (0)13 5821239
 
mailto:A.Hongens at netmatch.nl
http://www.netmatch.nl
------------------------------------------
Disclaimer
 
Deze e-mail is vertrouwelijk en uitsluitend bedoeld voor geadresseerde(n) en de organisatie van geadresseerde(n) en mag niet openbaar worden gemaakt aan derde partijen This e-mail is confidential and may not be disclosed to third parties since this e-mail is only intended for the addressee and the organization the addressee represents.





From dridi at varni.sh  Tue Dec 20 15:28:28 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 20 Dec 2016 16:28:28 +0100
Subject: stale-if-error in varnish 4.1?
In-Reply-To: <cdda09905af442c083a8c3b113a8c6fb@netmatch.nl>
References: <cdda09905af442c083a8c3b113a8c6fb@netmatch.nl>
Message-ID: <CABoVN9Bd5FPZXvgrH409kbrhZhozoRJhabrxTYe94=qiosq50A@mail.gmail.com>

>  Fastly uses normal varnish right?

A fork of Varnish, for various reasons.

> 2 questions for you smart guys and girls:
>
> - I native stale-if-error on the roadmap?

I think not.

> - How can I hack stale-if-error in my VCL using my varnish41 machines?

Fed's already done it [1] so you should be able to reuse his snippet.

Cheers

[1] https://github.com/fgsch/vcl-snippets/blob/master/v4/stale-if-error.vcl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161220/8307a611/attachment.html>

From guillaume at varnish-software.com  Tue Dec 20 16:02:33 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Tue, 20 Dec 2016 17:02:33 +0100
Subject: Deliver on HIT, otherwise redirect using "503; Location: ..."
In-Reply-To: <EC9357AD-0F1F-4B9C-B1A2-BCC1DD94287F@gmail.com>
References: <EDC53F74-567A-433D-B191-41F209EB1632@gmail.com>
 <op.ysm6brnnk1fmft@markair.local>
 <C862F1C0-E0C3-4E53-88B1-8A25F180CF61@gmail.com>
 <CAChvjRCMyTGKJ1X-iVAsz_YD4FyhdyquV+q9CutmU9AqiLHPTQ@mail.gmail.com>
 <CAJ6ZYQx5GuGcYFj7TwOmhMk-9NCpE+FOiH7McQdkcQfo1kz3hg@mail.gmail.com>
 <EC9357AD-0F1F-4B9C-B1A2-BCC1DD94287F@gmail.com>
Message-ID: <CAJ6ZYQyQr47cCDkV-kqgadFk8fv7Oeyi1ZBtZL0rGAF7XGbAhQ@mail.gmail.com>

Thanks for the context. So, if I get what you're writing, the goal is too
redirect to a node that has the object in cache?

Question is, what's the time needed to:
- send a request to the server on the LAN and receive the object
- send the redirect across the web, and wait for the client to send a new
request, again across the web.

If the former is not at least an order of magnitude larger than the latter,
I wouldn't bother.

The issues I have with your redirection scheme are that:
- IIUC, you are basically explaining to people where the backend is,
instead of shielding it with Varnish
- it doesn't lower the backend traffic
- as said, I'm not even sure the user-experience is better/faster

-- 
Guillaume Quintard

On Sun, Dec 18, 2016 at 9:21 PM, Anton Berezhkov <
bubonic.pestilence at gmail.com> wrote:

> We have 6 servers with 6TB on each one (same video files). Currently
> they're hitting iowait limit of SATA disk (240ops total). At same time,
> each server can provide 500Mbit of guaranteed bandwidth.
>
> With HDD restriction, each server provides about 320Mbit. There is also
> problem with fragmentation, which is caused by nature of HTML5 video
> players & HTTP (allowing to request partial data with Range header).
>
> Before this moment, we were scaling horizontally, by duplicating this
> servers.
>
> There is also option to get same server with 2x480GB SSDs. As i
> reasearched from nginx logs, 98% of daily traffic lays in ?800GB of files.
>
> What i want to achieve: To build a varnish server with 2x480GB SSDs(no
> raid), with storage for varnish about 800GBs. Which will guarantedly fill
> all available bandwidth for a server.
>
> Also, I built simple load-balancer, which monitors each server for current
> eth0 load (in Mbps) and decide to which one redirect (using HTTP Location
> header).
>
> Request for Video -> LBServer: Find lowest loaded(1 of 6) & Redirect to
> LBNode -> LBN: serve request
>
> To add new HDD-LBN, i need to setup server, sync videos, setup some
> additional software.
>
> My wish: add new SSD-LBN, setup & sync varnish config, and it will build
> cached pool itself.
>
> Why i need redirect?
> 1. It will offload bandwidth of SSD-LBN, pass-through will take bandwidth
> of both servers + still cause iowait problems on HDD-LBN.
> 2. It will "prove" that uncached video will be take from HDD-LBN which
> always have all videos.
>
> Currently all LBN servers are hosted on OVH and we're good with them,
> especially because of low price :)
>
> If you have any suggestions, i'll be glad to hear them :)
>
> > On Dec 18, 2016, at 10:59 PM, Guillaume Quintard <
> guillaume at varnish-software.com> wrote:
> >
> > I think Jason is right in asking "why?". What do you want to achieve
> specifically with this behavior?
> >
> > Varnish has streaming and request coalescing, meaning a request can be
> served as soon as data starts being available AND the backend doesn't
> suffer from simultaneous misses on the same object. I feel that should
> cover almost all your needs, so I'm curious about the use-case.
> >
> > On Dec 18, 2016 20:27, "Jason Price" <japrice at gmail.com> wrote:
> > It would be possible to do this with varnish... but I have to ask... why
> bother?
> >
> > If the purpose is to offload the IO load, then varnish is good, but you
> need to prime the cache... TBH, what I'd do first is put one or a pair of
> varnish boxes really close to the overloaded box, and force all traffic to
> that server through the close varnish boxes... using the do_stream feature,
> you'll get stuff out there fairly quickly.
> >
> > After that is working nicely, I'd layer in the further out varnish boxes
> which interact with the near-varnish boxes to get their data.
> >
> > This works well at scale since the local caches offer whatever's useful
> local to them, and the 'near-varnish' boxes handle the 'global caching'
> world.
> >
> > This was how I arranged it at $PreviousGig and the outer CDN was getting
> a 85-90% cache hit ratio, and the inner tier was seeing 60% cache hit
> ratio's.  (The inner tier's ratio will depend heavily on how many outer
> tier's there are...)
> >
> > On Sat, Dec 17, 2016 at 8:09 PM, Anton Berezhkov <
> bubonic.pestilence at gmail.com> wrote:
> > This is how I semi-implemented: http://pastebin.com/drDP8JxP
> > Now i need to use script which will run "curi -I -X PUT
> <url-to-put-into-cache>".
> >
> >
> > > On Dec 18, 2016, at 3:58 AM, Mark Staudinger <mark.staudinger at nyi.net>
> wrote:
> > >
> > > Hi Anton,
> > >
> > > Have you looked into the "do_stream" feature of Varnish?  This will
> begin serving the content to the visitor without waiting for the entire
> object to be downloaded and stored in cache.  Set in vcl_backend_response.
> > >
> > > https://github.com/mattiasgeniar/varnish-4.0-
> configuration-templates/blob/master/default.vcl
> > >
> > > Cheers,
> > > Mark
> > >
> > > On Sat, 17 Dec 2016 19:05:48 -0500, Anton Berezhkov <
> bubonic.pestilence at gmail.com> wrote:
> > >
> > >> Hello.
> > >>
> > >> Switched to Varnish from Nginx for additional functionality and
> better control of handling requests.
> > >> But still can't implement what i want. And I want simple behaviour
> "Redirect on MISS/PASS".
> > >> I want to use VC for deploying quick "cdn" servers for our
> mp4-video-servers (used for HTML5 players), without need to store all files
> on this quick (ssd, upto 2x480GB space, full database about 6TB).
> > >>
> > >> Currently we have 6 servers with SATA HDDs and hitting iowait like a
> trucks :)
> > >>
> > >> Examples:
> > >> - Request -> Varnish -> HIT: serve it using Varnish.
> > >> - Request -> Varnish -> MISS: start caching data from backend, and
> instantly reply to client: `Location: http://backend/$req.url"
> > >> - Request -> Varnish -> UPDATE: see `-> MISS` behaviour.
> > >>
> > >> From my perspective, i should do this "detach & reply redirect"
> somewhere in `vcl_miss` OR `vcl_backend_fetch`, because if i understood
> correctly https://www.varnish-cache.org/docs/4.1/reference/states.html, i
> need vcl_backend_response to keep run in background (as additional thread)
> while doing return(synth(...)) to redirect user.
> > >>
> > >> Similiar thing is "hitting stale content while object is updating".
> > >> But in my case "replying redirect while object is updating".
> > >>
> > >> Also, i pray to implement this without writing additional scripts,
> why? I could do external php/ruby checker/cache-pusher with nginx & etc.
> But scared by performance downgrade :(
> > >> _______________________________________________
> > >> varnish-misc mailing list
> > >> varnish-misc at varnish-cache.org
> > >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161220/64b55019/attachment.html>

From sander at hoentjen.eu  Fri Dec 23 10:18:23 2016
From: sander at hoentjen.eu (Sander Hoentjen)
Date: Fri, 23 Dec 2016 11:18:23 +0100
Subject: Proxy Protocol - CLIENT_SSL
Message-ID: <57c9c467-b4d1-ae6a-0328-acf8c35ded5e@hoentjen.eu>

Hi list,

I have a questioned about both Hitch and Varnish:
Does hitch support (defines) PP2_CLIENT_SSL from proxy-protocol [1]?
The follow-up question is: Can Varnish proxy this information (in
essence just keep the proxy header as-is)

Regards,
Sander



From guillaume at varnish-software.com  Sun Dec 25 09:46:46 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Sun, 25 Dec 2016 10:46:46 +0100
Subject: Grace mode with 404 responses?
In-Reply-To: <57811D62-4862-4953-A5D1-A4B4D12BF631@gmail.com>
References: <57811D62-4862-4953-A5D1-A4B4D12BF631@gmail.com>
Message-ID: <CAJ6ZYQwa=z4egC0go1KUr7DoKKMj=1D0kgKO97H=VxCimmRirQ@mail.gmail.com>

Hi Peter,

what is not working in the case of 404?

-- 
Guillaume Quintard

On Fri, Dec 2, 2016 at 9:11 AM, Peter Loron <peterloron at gmail.com> wrote:

> Hello, folks. I?m testing a Varnish 4.1.1 setup. Specifically I?m working
> with shielding a client service from an unreliable remote service by
> proxying and caching responses from the remote system.
>
> Grace mode is working fine for situations where the remote server is
> unresponsive (e.g. apache turned off), but does not seem to work when a 404
> is returned for a resource. I immediately get the 404 on the client when
> the TTL (not the TTL + grace) is expired.
>
> I?m assuming this is something I?m not doing right in my VCL file, but I
> can?t figure out what.
>
> Is there a canonical example of how to handle this properly?
>
> Thanks.
>
> -Pete
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161225/ffc59907/attachment.html>

From jack.xsuperman at gmail.com  Wed Dec 28 09:12:54 2016
From: jack.xsuperman at gmail.com (JackDrogon)
Date: Wed, 28 Dec 2016 17:12:54 +0800
Subject: Varnish backend timeout doesn't work with keep alive backend
Message-ID: <etPan.58638216.5e5629b5.7c41@gmail.com>

Hi,All:
	I set varnish backend that:
	backend default { .host = "127.0.0.1"; .port = "4567"; .connect_timeout = 1s; .first_byte_timeout = 2s; .between_bytes_timeout = 1s; }
	The first timeout is right, but if the prev connection is right, varnish use keepalive to deal with the next connection with same url and the timeout conf doesn't wok.
	How can make timeout work with all connections even with backend keepalive.

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161228/45d9a815/attachment.html>

From apj at mutt.dk  Wed Dec 28 09:24:43 2016
From: apj at mutt.dk (Andreas Plesner)
Date: Wed, 28 Dec 2016 10:24:43 +0100
Subject: Varnish backend timeout doesn't work with keep alive backend
In-Reply-To: <etPan.58638216.5e5629b5.7c41@gmail.com>
References: <etPan.58638216.5e5629b5.7c41@gmail.com>
Message-ID: <20161228092443.GZ22017@nerd.dk>

On Wed, Dec 28, 2016 at 05:12:54PM +0800, JackDrogon wrote:

> 	I set varnish backend that:
> 	backend default { .host = "127.0.0.1"; .port = "4567"; .connect_timeout
> 	= 1s; .first_byte_timeout = 2s; .between_bytes_timeout = 1s; }
> 	The first timeout is right, but if the prev connection is right,
> 	varnish use keepalive to deal with the next connection with same url
> 	and the timeout conf doesn't wok.

Please elaborate. Describe the situation that doesn't work as you expect, and what you would expect to happen.

-- 
Andreas



From dridi at varni.sh  Wed Dec 28 09:55:09 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 28 Dec 2016 10:55:09 +0100
Subject: Varnish backend timeout doesn't work with keep alive backend
In-Reply-To: <20161228092443.GZ22017@nerd.dk>
References: <etPan.58638216.5e5629b5.7c41@gmail.com>
 <20161228092443.GZ22017@nerd.dk>
Message-ID: <CABoVN9BSPCN9dBfbxC+Jz8ie=c1hXApDb0oqOvO1sc_zA=PMTA@mail.gmail.com>

On Wed, Dec 28, 2016 at 10:24 AM, Andreas Plesner <apj at mutt.dk> wrote:
> On Wed, Dec 28, 2016 at 05:12:54PM +0800, JackDrogon wrote:
>
>>       I set varnish backend that:
>>       backend default { .host = "127.0.0.1"; .port = "4567"; .connect_timeout
>>       = 1s; .first_byte_timeout = 2s; .between_bytes_timeout = 1s; }
>>       The first timeout is right, but if the prev connection is right,
>>       varnish use keepalive to deal with the next connection with same url
>>       and the timeout conf doesn't wok.
>
> Please elaborate. Describe the situation that doesn't work as you expect, and what you would expect to happen.

Possibly a known problem [1] for first_byte_timeout, apparently still
an open issue.

Dridi

[1] https://github.com/varnishcache/varnish-cache/issues/1772



From jack.xsuperman at gmail.com  Wed Dec 28 10:45:51 2016
From: jack.xsuperman at gmail.com (JackDrogon)
Date: Wed, 28 Dec 2016 18:45:51 +0800
Subject: Varnish backend timeout doesn't work with keep alive
 backend
In-Reply-To: <CABoVN9BSPCN9dBfbxC+Jz8ie=c1hXApDb0oqOvO1sc_zA=PMTA@mail.gmail.com>
References: <etPan.58638216.5e5629b5.7c41@gmail.com>
 <20161228092443.GZ22017@nerd.dk>
 <CABoVN9BSPCN9dBfbxC+Jz8ie=c1hXApDb0oqOvO1sc_zA=PMTA@mail.gmail.com>
Message-ID: <etPan.586397e0.6c765835.7c41@gmail.com>

OK.
?I run a go server, such as:

func handler(res http.ResponseWriter, req *http.Request) {
? ? ? ? atomic.AddUint32(&times, 1)
? ? ? ? number := rand.Intn(100)
? ? ? ? fmt.Printf("Random number: %d.\n", number)
? ? ? ? t := atomic.LoadUint32(&times)
? ? ? ? if t%4 == 1 {
? ? ? ? ? ? ? ? time.Sleep(7 * time.Second)
? ? ? ? }
? ? ? ? fmt.Fprintf(res, "Random number: %d.\n", number)
? ? ? ? fmt.Println("=========================")
}


I use this as varnish backend, and setting hash_always_miss
so I curl 127.0.0.1:8000 for 5 times.

curl 127.0.0.1:8000 ?0.00s user 0.00s system 0% cpu 2 total
curl 127.0.0.1:8000 ?0.00s user 0.00s system 77% cpu 0.005 total
curl 127.0.0.1:8000 ?0.00s user 0.00s system 66% cpu 0.005 total
curl 127.0.0.1:8000 ?0.00s user 0.00s system 68% cpu 0.004 total
curl 127.0.0.1:8000 ?0.00s user 0.00s system 0% cpu 7.005 total?

I think the fifth requet's time usage is 2s, but It's 7s. How can I set varnish to make it be 2s

On 28 December 2016 at 18:19:21, Boukelmoune Dridi (dridi at varni.sh) wrote:

On Wed, Dec 28, 2016 at 10:24 AM, Andreas Plesner <apj at mutt.dk> wrote:  
> On Wed, Dec 28, 2016 at 05:12:54PM +0800, JackDrogon wrote:  
>  
>> I set varnish backend that:  
>> backend default { .host = "127.0.0.1"; .port = "4567"; .connect_timeout  
>> = 1s; .first_byte_timeout = 2s; .between_bytes_timeout = 1s; }  
>> The first timeout is right, but if the prev connection is right,  
>> varnish use keepalive to deal with the next connection with same url  
>> and the timeout conf doesn't wok.  
>  
> Please elaborate. Describe the situation that doesn't work as you expect, and what you would expect to happen.  

Possibly a known problem [1] for first_byte_timeout, apparently still  
an open issue.  

Dridi  

[1] https://github.com/varnishcache/varnish-cache/issues/1772  

_______________________________________________  
varnish-misc mailing list  
varnish-misc at varnish-cache.org  
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161228/c676364d/attachment.html>

From dridi at varni.sh  Wed Dec 28 12:54:42 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 28 Dec 2016 13:54:42 +0100
Subject: Varnish backend timeout doesn't work with keep alive backend
In-Reply-To: <etPan.586397e0.6c765835.7c41@gmail.com>
References: <etPan.58638216.5e5629b5.7c41@gmail.com>
 <20161228092443.GZ22017@nerd.dk>
 <CABoVN9BSPCN9dBfbxC+Jz8ie=c1hXApDb0oqOvO1sc_zA=PMTA@mail.gmail.com>
 <etPan.586397e0.6c765835.7c41@gmail.com>
Message-ID: <CABoVN9Dss4=7bzasLKzKLVxw3QGr-WVqWbzzS3EVsAYZq3=iQA@mail.gmail.com>

On Wed, Dec 28, 2016 at 11:45 AM, JackDrogon <jack.xsuperman at gmail.com> wrote:
> OK.
>  I run a go server, such as:
>
> func handler(res http.ResponseWriter, req *http.Request) {
>         atomic.AddUint32(&times, 1)
>         number := rand.Intn(100)
>         fmt.Printf("Random number: %d.\n", number)
>         t := atomic.LoadUint32(&times)
>         if t%4 == 1 {
>                 time.Sleep(7 * time.Second)
>         }
>         fmt.Fprintf(res, "Random number: %d.\n", number)
>         fmt.Println("=========================")
> }
>
>
> I use this as varnish backend, and setting hash_always_miss
> so I curl 127.0.0.1:8000 for 5 times.
>
> curl 127.0.0.1:8000  0.00s user 0.00s system 0% cpu 2 total
> curl 127.0.0.1:8000  0.00s user 0.00s system 77% cpu 0.005 total
> curl 127.0.0.1:8000  0.00s user 0.00s system 66% cpu 0.005 total
> curl 127.0.0.1:8000  0.00s user 0.00s system 68% cpu 0.004 total
> curl 127.0.0.1:8000  0.00s user 0.00s system 0% cpu 7.005 total
>
> I think the fifth requet's time usage is 2s, but It's 7s. How can I set
> varnish to make it be 2s

I thought this bug had been fixed, but apparently not. A workaround
would be to disable keep-alive by forcefully closing backend
connections. With the obvious performance implication of having to
reconnect for every single request.

I'll try to remember to discuss this during next Monday's bugwash.

Dridi



From jack.xsuperman at gmail.com  Thu Dec 29 02:44:24 2016
From: jack.xsuperman at gmail.com (JackDrogon)
Date: Thu, 29 Dec 2016 10:44:24 +0800
Subject: Varnish backend timeout doesn't work with keep alive
 backend
In-Reply-To: <CABoVN9Dss4=7bzasLKzKLVxw3QGr-WVqWbzzS3EVsAYZq3=iQA@mail.gmail.com>
References: <etPan.58638216.5e5629b5.7c41@gmail.com>
 <20161228092443.GZ22017@nerd.dk>
 <CABoVN9BSPCN9dBfbxC+Jz8ie=c1hXApDb0oqOvO1sc_zA=PMTA@mail.gmail.com>
 <etPan.586397e0.6c765835.7c41@gmail.com>
 <CABoVN9Dss4=7bzasLKzKLVxw3QGr-WVqWbzzS3EVsAYZq3=iQA@mail.gmail.com>
Message-ID: <etPan.58647889.15a5174.7c41@gmail.com>

How to disable backend?keep-alive. I want to test it.


On 28 December 2016 at 20:55:23, Boukelmoune Dridi (dridi at varni.sh) wrote:
implication
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161229/79634906/attachment.html>

From dridi at varni.sh  Thu Dec 29 08:21:06 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 29 Dec 2016 09:21:06 +0100
Subject: Varnish backend timeout doesn't work with keep alive backend
In-Reply-To: <etPan.58647889.15a5174.7c41@gmail.com>
References: <etPan.58638216.5e5629b5.7c41@gmail.com>
 <20161228092443.GZ22017@nerd.dk>
 <CABoVN9BSPCN9dBfbxC+Jz8ie=c1hXApDb0oqOvO1sc_zA=PMTA@mail.gmail.com>
 <etPan.586397e0.6c765835.7c41@gmail.com>
 <CABoVN9Dss4=7bzasLKzKLVxw3QGr-WVqWbzzS3EVsAYZq3=iQA@mail.gmail.com>
 <etPan.58647889.15a5174.7c41@gmail.com>
Message-ID: <CABoVN9DQPM1mB0wpT0c7o3n_5+J_ajqpjSrz9kBUgjq8zFUvrA@mail.gmail.com>

On Thu, Dec 29, 2016 at 3:44 AM, JackDrogon <jack.xsuperman at gmail.com> wrote:
> How to disable backend keep-alive. I want to test it.

In vcl_backend_fetch, you can do:

    set bereq.http.connection = "close";

That's how it's specified in HTTP/1.1, by default a connection is persistent.

Dridi



From sander at hoentjen.eu  Thu Dec 29 15:01:28 2016
From: sander at hoentjen.eu (Sander Hoentjen)
Date: Thu, 29 Dec 2016 16:01:28 +0100
Subject: Proxy Protocol - CLIENT_SSL
In-Reply-To: <57c9c467-b4d1-ae6a-0328-acf8c35ded5e@hoentjen.eu>
References: <57c9c467-b4d1-ae6a-0328-acf8c35ded5e@hoentjen.eu>
Message-ID: <79f63449-300a-5f07-79ca-44fae6fa4150@hoentjen.eu>

On 12/23/2016 11:18 AM, Sander Hoentjen wrote:
> Hi list,
>
> I have a questioned about both Hitch and Varnish:
> Does hitch support (defines) PP2_CLIENT_SSL from proxy-protocol [1]?
> The follow-up question is: Can Varnish proxy this information (in
> essence just keep the proxy header as-is)
>
> Regards,
> Sander
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
Hmm, it seems I forgot the link to proxy-protocol [1]:
http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
And then specifically I am talking about the binary header format
(version 2).

"""

If the length specified in the PROXY protocol header indicates that additional
bytes are part of the header beyond the address information, a receiver may
choose to skip over and ignore those bytes, or attempt to interpret those
bytes.

The information in those bytes will be arranged in Type-Length-Value (TLV
vectors) in the following format.  The first byte is the Type of the vector.
The second two bytes represent the length in bytes of the value (not included
the Type and Length bytes), and following the length field is the number of
bytes specified by the length.

        struct pp2_tlv {
            uint8_t type;
            uint8_t length_hi;
            uint8_t length_lo;
            uint8_t value[0];
        };

The following types have already been registered for the <type> field :

        #define PP2_TYPE_ALPN           0x01
        #define PP2_TYPE_AUTHORITY      0x02
        #define PP2_TYPE_SSL            0x20
        #define PP2_SUBTYPE_SSL_VERSION 0x21
        #define PP2_SUBTYPE_SSL_CN      0x22
        #define PP2_TYPE_NETNS          0x30
"""

It would be very nice if Hitch supports this, but I can't find any info
on it. If this is not the right mailing list to ask, it would be nice if
someone can point me in the right direction.

Regards,
Sander



From jack.xsuperman at gmail.com  Fri Dec 30 10:10:24 2016
From: jack.xsuperman at gmail.com (JackDrogon)
Date: Fri, 30 Dec 2016 18:10:24 +0800
Subject: Varnish backend timeout doesn't work with keep alive
 backend
In-Reply-To: <CABoVN9DQPM1mB0wpT0c7o3n_5+J_ajqpjSrz9kBUgjq8zFUvrA@mail.gmail.com>
References: <etPan.58638216.5e5629b5.7c41@gmail.com>
 <20161228092443.GZ22017@nerd.dk>
 <CABoVN9BSPCN9dBfbxC+Jz8ie=c1hXApDb0oqOvO1sc_zA=PMTA@mail.gmail.com>
 <etPan.586397e0.6c765835.7c41@gmail.com>
 <CABoVN9Dss4=7bzasLKzKLVxw3QGr-WVqWbzzS3EVsAYZq3=iQA@mail.gmail.com>
 <etPan.58647889.15a5174.7c41@gmail.com>
 <CABoVN9DQPM1mB0wpT0c7o3n_5+J_ajqpjSrz9kBUgjq8zFUvrA@mail.gmail.com>
Message-ID: <etPan.58663290.4379e816.7c41@gmail.com>

It's so diffcult for me to tracing the timeout problem.




I painted a map of the backend request.
I tracing varnish, found that?cache/cache_obj.c:ObjWaitState lock for waiting backend response. But the vwe_thread epoll wait time is not right? it is default time 100s
When epoll_wait is run, it wait for backend response first_bytes_timeout, which in my case is 7s.
If I set cache_waiter_epoll.c:vwe_thread => ?vwe->next = now + 1, all is right. The timeout time became 2s.

I read some varnish code, not full. I want know where is the code that varnish add the backend response header read event to epoll when keepalive is enable.
I guess It's cnt_miss in my case, but I can't see the code, it's in vgc.so. How can see the vgc.so's origin code which is generated by libvcc.

Thanks.

On 29 December 2016 at 16:21:47, Boukelmoune Dridi (dridi at varni.sh) wrote:

On Thu, Dec 29, 2016 at 3:44 AM, JackDrogon <jack.xsuperman at gmail.com> wrote:  
> How to disable backend keep-alive. I want to test it.  

In vcl_backend_fetch, you can do:  

set bereq.http.connection = "close";  

That's how it's specified in HTTP/1.1, by default a connection is persistent.  

Dridi  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161230/c8d2f426/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2016-12-30 at 14.15.13.png
Type: image/png
Size: 85600 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161230/c8d2f426/attachment.png>

From dridi at varni.sh  Fri Dec 30 11:13:01 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 30 Dec 2016 12:13:01 +0100
Subject: Varnish backend timeout doesn't work with keep alive backend
In-Reply-To: <etPan.58663290.4379e816.7c41@gmail.com>
References: <etPan.58638216.5e5629b5.7c41@gmail.com>
 <20161228092443.GZ22017@nerd.dk>
 <CABoVN9BSPCN9dBfbxC+Jz8ie=c1hXApDb0oqOvO1sc_zA=PMTA@mail.gmail.com>
 <etPan.586397e0.6c765835.7c41@gmail.com>
 <CABoVN9Dss4=7bzasLKzKLVxw3QGr-WVqWbzzS3EVsAYZq3=iQA@mail.gmail.com>
 <etPan.58647889.15a5174.7c41@gmail.com>
 <CABoVN9DQPM1mB0wpT0c7o3n_5+J_ajqpjSrz9kBUgjq8zFUvrA@mail.gmail.com>
 <etPan.58663290.4379e816.7c41@gmail.com>
Message-ID: <CABoVN9C+ns8QVwLUnymMRh4yisducmn8vzhPNW2jHwoi=xN_LA@mail.gmail.com>

On Fri, Dec 30, 2016 at 11:10 AM, JackDrogon <jack.xsuperman at gmail.com> wrote:
>
> It's so diffcult for me to tracing the timeout problem.
>
>
>
> I painted a map of the backend request.
> I tracing varnish, found that cache/cache_obj.c:ObjWaitState lock for waiting backend response. But the vwe_thread epoll wait time is not right? it is default time 100s
> When epoll_wait is run, it wait for backend response first_bytes_timeout, which in my case is 7s.
> If I set cache_waiter_epoll.c:vwe_thread =>  vwe->next = now + 1, all is right. The timeout time became 2s.

Feel free to comment on github in the issue I mentioned if you think
you have a lead. I thought I had seen a commit fix this in the not so
recent past, but I must have confused it with another ticket (probably
1806). As I said, I plan to mention this during the next bugwash.

> I read some varnish code, not full. I want know where is the code that varnish add the backend response header read event to epoll when keepalive is enable.
> I guess It's cnt_miss in my case, but I can't see the code, it's in vgc.so. How can see the vgc.so's origin code which is generated by libvcc.

To translate your VCL code to C, see varnishd -C (in the varnishd
man). But this C code will not touch the waiting code, it's only code
that is called by varnishd for each step (sub) in the VCL engine,
executing your VCL logic (reading/setting variables, picking
transitions etc).

Dridi



From jack.xsuperman at gmail.com  Fri Dec 30 11:51:40 2016
From: jack.xsuperman at gmail.com (JackDrogon)
Date: Fri, 30 Dec 2016 19:51:40 +0800
Subject: Varnish backend timeout doesn't work with keep alive
 backend
In-Reply-To: <CABoVN9C+ns8QVwLUnymMRh4yisducmn8vzhPNW2jHwoi=xN_LA@mail.gmail.com>
References: <etPan.58638216.5e5629b5.7c41@gmail.com>
 <20161228092443.GZ22017@nerd.dk>
 <CABoVN9BSPCN9dBfbxC+Jz8ie=c1hXApDb0oqOvO1sc_zA=PMTA@mail.gmail.com>
 <etPan.586397e0.6c765835.7c41@gmail.com>
 <CABoVN9Dss4=7bzasLKzKLVxw3QGr-WVqWbzzS3EVsAYZq3=iQA@mail.gmail.com>
 <etPan.58647889.15a5174.7c41@gmail.com>
 <CABoVN9DQPM1mB0wpT0c7o3n_5+J_ajqpjSrz9kBUgjq8zFUvrA@mail.gmail.com>
 <etPan.58663290.4379e816.7c41@gmail.com>
 <CABoVN9C+ns8QVwLUnymMRh4yisducmn8vzhPNW2jHwoi=xN_LA@mail.gmail.com>
Message-ID: <etPan.58664a4c.43464cb3.7c41@gmail.com>

OK.
I will discuss about the problem on github. The problem is different from #1806.


On 30 December 2016 at 19:13:42, Boukelmoune Dridi (dridi at varni.sh) wrote:

but I must have confused it with another ticket (probably?
1806)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20161230/d5277761/attachment.html>