From andrew.langhorn at digital.cabinet-office.gov.uk  Sat May  2 23:10:56 2015
From: andrew.langhorn at digital.cabinet-office.gov.uk (Andrew Langhorn)
Date: Sun, 3 May 2015 00:10:56 +0100
Subject: Using multiple scenarios inside a conditional VCL statement
Message-ID: <CANCa_Wa=GgJBaaMsnCN1cfuy81QO7OZrxh5x2WhO02ZeZE493A@mail.gmail.com>

Hi,

When doing a comparison inside a conditional statement, what is the
accepted way to provide multiple scenarios to test against inside the
conditional?

For instance, if I set up multiple ACLs, and then check client.ip against
them using the following syntax, is it likely to work?

if (client.ip ~ acl1 || acl2 || acl3 ) {
 ...
}

If not, does anyone know what will work?

Thanks,

Andrew Langhorn
Web Operations
Government Digital Service

e: andrew.langhorn at digital.cabinet-office.gov.uk
a: 6th Floor, Aviation House, 125 Kingsway, London, WC2B 6NH
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150503/3b37a143/attachment.html>

From dridi at varni.sh  Mon May  4 09:50:32 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 4 May 2015 11:50:32 +0200
Subject: Don't cache large files
In-Reply-To: <5CF29039-FABC-4571-AE9A-3556FCA59B52@psu.edu>
References: <D187E8F3-7A28-4500-BFD8-07D61C655394@psu.edu>
 <CABoVN9DW+E=w-yd8osg-aUvLhMs+u_MHmODp9U3Nh6sS5SWY6g@mail.gmail.com>
 <5CF29039-FABC-4571-AE9A-3556FCA59B52@psu.edu>
Message-ID: <CABoVN9AvHn1bVbhHtXzo=KMTqwGYxJ4aarpokDfkMimaF3rJ3Q@mail.gmail.com>

Hi Jason,

The pipe mode is designed for non-HTTP traffic. For example, you
switch to the WebSocket protocol, Varnish can no longer proxy anything
and simply let data go directly back and forth between the client and
the backend. But to use WebSocket, you first need to make an HTTP
request.

https://en.wikipedia.org/wiki/WebSocket#WebSocket_protocol_handshake

Cheers,
Dridi

On Fri, Apr 17, 2015 at 2:38 PM, Jason Heffner <jdh132 at psu.edu> wrote:
> Thanks Dridi. I had switched to just marking the content uncacheable and using return(deliver), per the new v4 method, temporarily. I wasn?t sure if there was any advantage to using pipe anymore for files ranging from ~10MB - 500MB in size on the server?
>
> Jason
>
>> On Apr 17, 2015, at 2:58 AM, Dridi Boukelmoune <dridi at varni.sh> wrote:
>>
>> Why pipe when you could simply have a hit-for-pass object?
>>
>> In vcl_backend_response you could simply set beresp.uncacheable to
>> true (return(pass) in vcl_fetch for v3). With such a config you'd hit
>> your backend twice for all responses of more than 10MB.
>>
>> Dridi
>>
>> On Thu, Apr 16, 2015 at 6:05 PM, Jason Heffner <jdh132 at psu.edu> wrote:
>>> I?m working on updating our Varnish 3 config to Varnish 4 and I was able to figure out how to have the same functionality for almost all of the config. I wanted to find opinions on how to handle large files. We don?t want to cache any large files and had this in place for varnish 3. I?ve read streaming instead of piping was a solution but found nothing to determine this based solely on file size from the backend.
>>>
>>> # Varnish 3 config
>>> added to vcl_recv:
>>>  /* Bypass cache for large files.  The x-pipe header is
>>>     set in vcl_fetch when a too large file is detected. */
>>>  if (req.http.x-pipe && req.restarts > 0) {
>>>    remove req.http.x-pipe;
>>>    return (pipe);
>>>  }
>>>
>>> added to vcl_fetch:
>>>  # don't cache files larger than 10MB
>>>  /* Don't try to cache too large files.  It appears
>>>     Varnish just crashes if we don't filter them. */
>>>  if (beresp.http.Content-Length ~ "[0-9]{8,}" ) {
>>>    set req.http.x-pipe = "1";
>>>    return (restart);
>>>  }
>>>
>>> Since req.* is no longer available in vcl_recv this code doesn?t function anymore.
>>>
>>> Thanks,
>>> Jason
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



From dridi at varni.sh  Mon May  4 09:56:22 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 4 May 2015 11:56:22 +0200
Subject: Using multiple scenarios inside a conditional VCL statement
In-Reply-To: <CANCa_Wa=GgJBaaMsnCN1cfuy81QO7OZrxh5x2WhO02ZeZE493A@mail.gmail.com>
References: <CANCa_Wa=GgJBaaMsnCN1cfuy81QO7OZrxh5x2WhO02ZeZE493A@mail.gmail.com>
Message-ID: <CABoVN9CXR4R=R+B6gciXNiN=AD-QkuoOVwW3Eq2EbR40skHMww@mail.gmail.com>

Hi Andrew,

The syntax would be a bit more verbose:

if (client.ip ~ acl1 || client.ip ~ acl2 || client.ip ~ acl3 ) {
    ...
}

Cheers,
Dridi

On Sun, May 3, 2015 at 1:10 AM, Andrew Langhorn
<andrew.langhorn at digital.cabinet-office.gov.uk> wrote:
> Hi,
>
> When doing a comparison inside a conditional statement, what is the accepted
> way to provide multiple scenarios to test against inside the conditional?
>
> For instance, if I set up multiple ACLs, and then check client.ip against
> them using the following syntax, is it likely to work?
>
> if (client.ip ~ acl1 || acl2 || acl3 ) {
>  ...
> }
>
> If not, does anyone know what will work?
>
> Thanks,
>
> Andrew Langhorn
> Web Operations
> Government Digital Service
>
> e: andrew.langhorn at digital.cabinet-office.gov.uk
> a: 6th Floor, Aviation House, 125 Kingsway, London, WC2B 6NH
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From elaine at fastly.com  Tue May  5 20:53:37 2015
From: elaine at fastly.com (Elaine Greenberg)
Date: Tue, 5 May 2015 13:53:37 -0700
Subject: Varnish Meetup in NYC Next Week
Message-ID: <CANeg4+e5LLr9W33visOrw=ziX=9LidAnCD31mYBi7DRB81QDMg@mail.gmail.com>

Hi all,

We're hosting a Varnish Meetup at iHeartMedia next week on Thursday, May 14
in New York, featuring Kenton Jacobsen <https://twitter.com/kentonjacobsen>,
Head of Engineering at Conde Nast for Vogue and Style Properties, and
Fastly's Stephen Basile <https://twitter.com/stephenb>. More info on talk
topics to come, but we'll definitely be coving little-known Varnish tips
and tricks. We'd love to see you there!

You can RSVP here:
http://www.meetup.com/NYC-Varnish-Meetup/events/222078403/

Food and beer will be provided :)

Please let me know if you have any questions! And tell your friends!

Thanks,
Elaine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150505/6f855738/attachment.html>

From krishna.ku at flipkart.com  Fri May  8 08:43:05 2015
From: krishna.ku at flipkart.com (Krishna Kumar (Engineering))
Date: Fri, 8 May 2015 14:13:05 +0530
Subject: Configuration file error
Message-ID: <CACycfVY8PrVqFr-V2BCiTFtLCqQAubTUXeWSU_h4iJkYcdehZg@mail.gmail.com>

Hi,

I am testing varnish as a backend to haproxy. varnish has 2 backend's
for cache misses. But starting varnish with this configuration gives this
error (on Debian Wheezy):

directors are now in directors VMOD.
('input' Line 13 Pos 1)
director cache_director round-robin {
########-----------------------------

The configuration file is:

vcl 4.0;

backend server1 {
    .host = "<ip1 here>";
    .port = "80";
}

backend server2 {
    .host = "<ip2 here>";
    .port = "80";
}

director cache_director round-robin {
    { .backend = server1; }
    { .backend = server2; }
}

sub vcl_recv {
    set req.backend = cache_director;
}

sub vcl_backend_response {
}

sub vcl_deliver {
}

Could someone help on what is required?

Thanks,
- Krishna Kumar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150508/5e5a9f7b/attachment.html>

From kokoniimasu at gmail.com  Fri May  8 08:55:22 2015
From: kokoniimasu at gmail.com (kokoniimasu)
Date: Fri, 8 May 2015 17:55:22 +0900
Subject: Configuration file error
In-Reply-To: <CACycfVY8PrVqFr-V2BCiTFtLCqQAubTUXeWSU_h4iJkYcdehZg@mail.gmail.com>
References: <CACycfVY8PrVqFr-V2BCiTFtLCqQAubTUXeWSU_h4iJkYcdehZg@mail.gmail.com>
Message-ID: <CAJihfGD7Uv1Su9Yo9CAB4nwXtLYrGxxcHOdWGFjsGmc9AjP=7g@mail.gmail.com>

Hi,

Directors moved to VMOD at Varnish4.
  ref: https://www.varnish-cache.org/docs/4.0/whats-new/upgrading.html#directors-have-been-moved-to-the-vmod-directors

Regards,
--
Shohei Tanaka(@xcir)
http://xcir.net/

2015-05-08 17:43 GMT+09:00 Krishna Kumar (Engineering)
<krishna.ku at flipkart.com>:
> Hi,
>
> I am testing varnish as a backend to haproxy. varnish has 2 backend's
> for cache misses. But starting varnish with this configuration gives this
> error (on Debian Wheezy):
>
> directors are now in directors VMOD.
> ('input' Line 13 Pos 1)
> director cache_director round-robin {
> ########-----------------------------
>
> The configuration file is:
>
> vcl 4.0;
>
> backend server1 {
>     .host = "<ip1 here>";
>     .port = "80";
> }
>
> backend server2 {
>     .host = "<ip2 here>";
>     .port = "80";
> }
>
> director cache_director round-robin {
>     { .backend = server1; }
>     { .backend = server2; }
> }
>
> sub vcl_recv {
>     set req.backend = cache_director;
> }
>
> sub vcl_backend_response {
> }
>
> sub vcl_deliver {
> }
>
> Could someone help on what is required?
>
> Thanks,
> - Krishna Kumar
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From krishna.ku at flipkart.com  Fri May  8 09:07:48 2015
From: krishna.ku at flipkart.com (Krishna Kumar (Engineering))
Date: Fri, 8 May 2015 14:37:48 +0530
Subject: Configuration file error
In-Reply-To: <CAJihfGD7Uv1Su9Yo9CAB4nwXtLYrGxxcHOdWGFjsGmc9AjP=7g@mail.gmail.com>
References: <CACycfVY8PrVqFr-V2BCiTFtLCqQAubTUXeWSU_h4iJkYcdehZg@mail.gmail.com>
 <CAJihfGD7Uv1Su9Yo9CAB4nwXtLYrGxxcHOdWGFjsGmc9AjP=7g@mail.gmail.com>
Message-ID: <CACycfVaM5_yMihBNkeD+4WUNjzp8zftzOEUHhDbzZXXEyDyCnw@mail.gmail.com>

Hi,

Thanks for the link. I followed the directions there and still get an error:

# cat /etc/varnish/default.vcl
vcl 4.0;

backend server1 {
    .host = "<ip1 here>";
    .port = "80";
}

backend server2 {
    .host = "<ip2 here>";
    .port = "80";
}

sub vcl_init {
    new cluster1 = directors.round_robin();
    cluster1.add_backend(server1, 1.0);
    cluster1.add_backend(server2, 1.0);
}

sub vcl_recv {
    set req.backend_hint = cluster1.backend();
}

sub vcl_backend_response {
}

sub vcl_deliver {
}

# sh /etc/init.d/varnish restart
Message from VCC-compiler:
Symbol not found: 'directors.round_robin' at ('input' Line 14 Pos 24)
        new cluster1 = directors.round_robin();
-----------------------#####################---

Thanks,
- Krishna Kumar

On Fri, May 8, 2015 at 2:25 PM, kokoniimasu <kokoniimasu at gmail.com> wrote:

> Hi,
>
> Directors moved to VMOD at Varnish4.
>   ref:
> https://www.varnish-cache.org/docs/4.0/whats-new/upgrading.html#directors-have-been-moved-to-the-vmod-directors
>
> Regards,
> --
> Shohei Tanaka(@xcir)
> http://xcir.net/
>
> 2015-05-08 17:43 GMT+09:00 Krishna Kumar (Engineering)
> <krishna.ku at flipkart.com>:
> > Hi,
> >
> > I am testing varnish as a backend to haproxy. varnish has 2 backend's
> > for cache misses. But starting varnish with this configuration gives this
> > error (on Debian Wheezy):
> >
> > directors are now in directors VMOD.
> > ('input' Line 13 Pos 1)
> > director cache_director round-robin {
> > ########-----------------------------
> >
> > The configuration file is:
> >
> > vcl 4.0;
> >
> > backend server1 {
> >     .host = "<ip1 here>";
> >     .port = "80";
> > }
> >
> > backend server2 {
> >     .host = "<ip2 here>";
> >     .port = "80";
> > }
> >
> > director cache_director round-robin {
> >     { .backend = server1; }
> >     { .backend = server2; }
> > }
> >
> > sub vcl_recv {
> >     set req.backend = cache_director;
> > }
> >
> > sub vcl_backend_response {
> > }
> >
> > sub vcl_deliver {
> > }
> >
> > Could someone help on what is required?
> >
> > Thanks,
> > - Krishna Kumar
> >
> >
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150508/6b20a304/attachment.html>

From kokoniimasu at gmail.com  Fri May  8 09:36:01 2015
From: kokoniimasu at gmail.com (kokoniimasu)
Date: Fri, 8 May 2015 18:36:01 +0900
Subject: Configuration file error
In-Reply-To: <CACycfVaM5_yMihBNkeD+4WUNjzp8zftzOEUHhDbzZXXEyDyCnw@mail.gmail.com>
References: <CACycfVY8PrVqFr-V2BCiTFtLCqQAubTUXeWSU_h4iJkYcdehZg@mail.gmail.com>
 <CAJihfGD7Uv1Su9Yo9CAB4nwXtLYrGxxcHOdWGFjsGmc9AjP=7g@mail.gmail.com>
 <CACycfVaM5_yMihBNkeD+4WUNjzp8zftzOEUHhDbzZXXEyDyCnw@mail.gmail.com>
Message-ID: <CAJihfGDFGVS-FoSS0CwM8360ZyUQF0bXZ-s-0hgaKK5LewmMYQ@mail.gmail.com>

Hi,


vcl 4.0;

import directors; // <- load the vmod_directors

backend server1 {
...

Regards,
--
Shohei Tanaka(@xcir)
http://xcir.net/

2015-05-08 18:07 GMT+09:00 Krishna Kumar (Engineering)
<krishna.ku at flipkart.com>:
> Hi,
>
> Thanks for the link. I followed the directions there and still get an error:
>
> # cat /etc/varnish/default.vcl
> vcl 4.0;
>
> backend server1 {
>     .host = "<ip1 here>";
>     .port = "80";
> }
>
> backend server2 {
>     .host = "<ip2 here>";
>     .port = "80";
> }
>
> sub vcl_init {
>     new cluster1 = directors.round_robin();
>     cluster1.add_backend(server1, 1.0);
>     cluster1.add_backend(server2, 1.0);
> }
>
> sub vcl_recv {
>     set req.backend_hint = cluster1.backend();
> }
>
> sub vcl_backend_response {
> }
>
> sub vcl_deliver {
> }
>
> # sh /etc/init.d/varnish restart
> Message from VCC-compiler:
> Symbol not found: 'directors.round_robin' at ('input' Line 14 Pos 24)
>         new cluster1 = directors.round_robin();
> -----------------------#####################---
>
> Thanks,
> - Krishna Kumar
>
> On Fri, May 8, 2015 at 2:25 PM, kokoniimasu <kokoniimasu at gmail.com> wrote:
>>
>> Hi,
>>
>> Directors moved to VMOD at Varnish4.
>>   ref:
>> https://www.varnish-cache.org/docs/4.0/whats-new/upgrading.html#directors-have-been-moved-to-the-vmod-directors
>>
>> Regards,
>> --
>> Shohei Tanaka(@xcir)
>> http://xcir.net/
>>
>> 2015-05-08 17:43 GMT+09:00 Krishna Kumar (Engineering)
>> <krishna.ku at flipkart.com>:
>> > Hi,
>> >
>> > I am testing varnish as a backend to haproxy. varnish has 2 backend's
>> > for cache misses. But starting varnish with this configuration gives
>> > this
>> > error (on Debian Wheezy):
>> >
>> > directors are now in directors VMOD.
>> > ('input' Line 13 Pos 1)
>> > director cache_director round-robin {
>> > ########-----------------------------
>> >
>> > The configuration file is:
>> >
>> > vcl 4.0;
>> >
>> > backend server1 {
>> >     .host = "<ip1 here>";
>> >     .port = "80";
>> > }
>> >
>> > backend server2 {
>> >     .host = "<ip2 here>";
>> >     .port = "80";
>> > }
>> >
>> > director cache_director round-robin {
>> >     { .backend = server1; }
>> >     { .backend = server2; }
>> > }
>> >
>> > sub vcl_recv {
>> >     set req.backend = cache_director;
>> > }
>> >
>> > sub vcl_backend_response {
>> > }
>> >
>> > sub vcl_deliver {
>> > }
>> >
>> > Could someone help on what is required?
>> >
>> > Thanks,
>> > - Krishna Kumar
>> >
>> >
>> >
>> > _______________________________________________
>> > varnish-misc mailing list
>> > varnish-misc at varnish-cache.org
>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>



From krishna.ku at flipkart.com  Fri May  8 09:59:08 2015
From: krishna.ku at flipkart.com (Krishna Kumar (Engineering))
Date: Fri, 8 May 2015 15:29:08 +0530
Subject: Configuration file error
In-Reply-To: <CAJihfGDFGVS-FoSS0CwM8360ZyUQF0bXZ-s-0hgaKK5LewmMYQ@mail.gmail.com>
References: <CACycfVY8PrVqFr-V2BCiTFtLCqQAubTUXeWSU_h4iJkYcdehZg@mail.gmail.com>
 <CAJihfGD7Uv1Su9Yo9CAB4nwXtLYrGxxcHOdWGFjsGmc9AjP=7g@mail.gmail.com>
 <CACycfVaM5_yMihBNkeD+4WUNjzp8zftzOEUHhDbzZXXEyDyCnw@mail.gmail.com>
 <CAJihfGDFGVS-FoSS0CwM8360ZyUQF0bXZ-s-0hgaKK5LewmMYQ@mail.gmail.com>
Message-ID: <CACycfVYnm69cs6gK+RbCnu6bYhO0eKJr829L=OOMr1M_49yD7w@mail.gmail.com>

Hi,

Thank you very much - that worked. I also had to remove the ", 1.0"
string in the "cluster1.add_backend(server1);" line, since that gave an
error: "Expected ')' got ','"

Thank you.
- Krishna Kumar


On Fri, May 8, 2015 at 3:06 PM, kokoniimasu <kokoniimasu at gmail.com> wrote:

> Hi,
>
>
> vcl 4.0;
>
> import directors; // <- load the vmod_directors
>
> backend server1 {
> ...
>
> Regards,
> --
> Shohei Tanaka(@xcir)
> http://xcir.net/
>
> 2015-05-08 18:07 GMT+09:00 Krishna Kumar (Engineering)
> <krishna.ku at flipkart.com>:
> > Hi,
> >
> > Thanks for the link. I followed the directions there and still get an
> error:
> >
> > # cat /etc/varnish/default.vcl
> > vcl 4.0;
> >
> > backend server1 {
> >     .host = "<ip1 here>";
> >     .port = "80";
> > }
> >
> > backend server2 {
> >     .host = "<ip2 here>";
> >     .port = "80";
> > }
> >
> > sub vcl_init {
> >     new cluster1 = directors.round_robin();
> >     cluster1.add_backend(server1, 1.0);
> >     cluster1.add_backend(server2, 1.0);
> > }
> >
> > sub vcl_recv {
> >     set req.backend_hint = cluster1.backend();
> > }
> >
> > sub vcl_backend_response {
> > }
> >
> > sub vcl_deliver {
> > }
> >
> > # sh /etc/init.d/varnish restart
> > Message from VCC-compiler:
> > Symbol not found: 'directors.round_robin' at ('input' Line 14 Pos 24)
> >         new cluster1 = directors.round_robin();
> > -----------------------#####################---
> >
> > Thanks,
> > - Krishna Kumar
> >
> > On Fri, May 8, 2015 at 2:25 PM, kokoniimasu <kokoniimasu at gmail.com>
> wrote:
> >>
> >> Hi,
> >>
> >> Directors moved to VMOD at Varnish4.
> >>   ref:
> >>
> https://www.varnish-cache.org/docs/4.0/whats-new/upgrading.html#directors-have-been-moved-to-the-vmod-directors
> >>
> >> Regards,
> >> --
> >> Shohei Tanaka(@xcir)
> >> http://xcir.net/
> >>
> >> 2015-05-08 17:43 GMT+09:00 Krishna Kumar (Engineering)
> >> <krishna.ku at flipkart.com>:
> >> > Hi,
> >> >
> >> > I am testing varnish as a backend to haproxy. varnish has 2 backend's
> >> > for cache misses. But starting varnish with this configuration gives
> >> > this
> >> > error (on Debian Wheezy):
> >> >
> >> > directors are now in directors VMOD.
> >> > ('input' Line 13 Pos 1)
> >> > director cache_director round-robin {
> >> > ########-----------------------------
> >> >
> >> > The configuration file is:
> >> >
> >> > vcl 4.0;
> >> >
> >> > backend server1 {
> >> >     .host = "<ip1 here>";
> >> >     .port = "80";
> >> > }
> >> >
> >> > backend server2 {
> >> >     .host = "<ip2 here>";
> >> >     .port = "80";
> >> > }
> >> >
> >> > director cache_director round-robin {
> >> >     { .backend = server1; }
> >> >     { .backend = server2; }
> >> > }
> >> >
> >> > sub vcl_recv {
> >> >     set req.backend = cache_director;
> >> > }
> >> >
> >> > sub vcl_backend_response {
> >> > }
> >> >
> >> > sub vcl_deliver {
> >> > }
> >> >
> >> > Could someone help on what is required?
> >> >
> >> > Thanks,
> >> > - Krishna Kumar
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > varnish-misc mailing list
> >> > varnish-misc at varnish-cache.org
> >> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150508/08a446d6/attachment.html>

From miguel_3_gonzalez at yahoo.es  Tue May 12 18:19:27 2015
From: miguel_3_gonzalez at yahoo.es (Miguel Gonzalez)
Date: Tue, 12 May 2015 18:19:27 +0000 (UTC)
Subject: normalizing ?ver= css and js
Message-ID: <71244718.1036810.1431454767053.JavaMail.yahoo@mail.yahoo.com>

Dear all,
? I have Varnish 4 installed, in vcl_recv I have added this line:
? set req.url = regsub(req.url, "\?ver=.*$", "");
? 
? which supposedly normalizes all css and js ?ver=xxxx strings. 
?? However tools.pingdom.com reports there are still css and js files being delivered with those strings.
? What am I doing wrong?
? Thanks!? 
?? Miguel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150512/5c25902a/attachment.html>

From info at massivescale.net  Tue May 12 19:03:41 2015
From: info at massivescale.net (Andrzej Godziuk)
Date: Tue, 12 May 2015 21:03:41 +0200
Subject: normalizing ?ver= css and js
In-Reply-To: <71244718.1036810.1431454767053.JavaMail.yahoo@mail.yahoo.com>
References: <71244718.1036810.1431454767053.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <20150512210341.13e50343@gdr-desktop.gdr.name>

On Tue, 12 May 2015 18:19:27 +0000 (UTC)
Miguel Gonzalez <miguel_3_gonzalez at yahoo.es> wrote:

> Dear all,
> ? I have Varnish 4 installed, in vcl_recv I have added this line:
> ? set req.url = regsub(req.url, "\?ver=.*$", "");
> ? 
> ? which supposedly normalizes all css and js ?ver=xxxx strings. 
> ?? However tools.pingdom.com reports there are still css and js files
> being delivered with those strings. What am I doing wrong?
> ? Thanks!? 
> ?? Miguel
>   

Hi,

Adding this line to your configuration did not remove ?ver=xxx from
HTML produced by your application. You can't do it with Varnish (well,
technically you can do everything with Varnish but it would require a
fair amount of C programming).

You should fix your application to not produce such strings. 

What this snipped of VCL code does is it keeps a single copy of said
JS/CSS files, so that no matter what the value of xxx is, it's always
served from the same cache entry.

Andrzej Godziuk
http://massivescale.net/



From raygo22 at gmail.com  Wed May 13 15:33:34 2015
From: raygo22 at gmail.com (Raymond Gonzalez)
Date: Wed, 13 May 2015 11:33:34 -0400
Subject: Caching cookies with Varnish 4
Message-ID: <CAGyfp2guy01cB2jJOZJCGrzsSBPRE__Cu6+0c3q28Xo0aXSR3A@mail.gmail.com>

I want to be able to cache pages that have a cookie. Let's say I have a
cookie called "testcookie" that I set up with php.

If testcookie = 1 I want a page (cached), if testcookie = 2 it will have a
different page ( i want to cache it) , and like that N times. This is my
current varnish vcl

____________


vcl 4.0;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "127.0.0.1";
    .port = "8080";
}



sub vcl_hash {
hash_data(regsub(req.http.Cookie,"^.*?testvarnish=([^;]*);*.*$" , "\1"));
}

sub vcl_recv {
    # Happens before we check if we have this in cache already.
    #
    # Typically you clean up the request here, removing cookies you don't
need,
    # rewriting the request, etc.

}

sub vcl_backend_response {
    # Happens after we have read the response headers from the backend.
    #
    # Here you clean the response headers, removing silly Set-Cookie headers
    # and other mistakes your backend does.

}

sub vcl_deliver {
    # Happens when we have all the pieces we need, and are about to send the
    # response to the client.
    #
    # You can do accounting or modifying the final object here.
}

________________

Any idea? Thanks!

-- 
Raymond Gonzalez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150513/da87e5a1/attachment.html>

From thomas.lecomte at virtual-expo.com  Wed May 13 21:30:27 2015
From: thomas.lecomte at virtual-expo.com (Thomas Lecomte)
Date: Wed, 13 May 2015 23:30:27 +0200
Subject: Caching cookies with Varnish 4
In-Reply-To: <CAGyfp2guy01cB2jJOZJCGrzsSBPRE__Cu6+0c3q28Xo0aXSR3A@mail.gmail.com>
References: <CAGyfp2guy01cB2jJOZJCGrzsSBPRE__Cu6+0c3q28Xo0aXSR3A@mail.gmail.com>
Message-ID: <CAJban9BmV7e4pat_11MDtpUMGF4iYakhO5dW=NLrDaZx+b0+Xg@mail.gmail.com>

On Wed, May 13, 2015 at 5:33 PM, Raymond Gonzalez <raygo22 at gmail.com> wrote:
> I want to be able to cache pages that have a cookie. Let's say I have a
> cookie called "testcookie" that I set up with php.
>
> [...]
>
> Any idea? Thanks!

Hello Raymond,

Since your VCL code doesn't return anything itself, the default VCL
implementation is run after your own.
The default vcl_recv subroutine will return (pass) if req.http.Cookie
is present, which is the first reason why your page isn't going to be
cached (in fact, it won't even be looked up).
Also, in vcl_backend_response, the returned object won't be cached if
it contains a Set-Cookie HTTP header.

You might want to take care of these cases yourself if you want to
cache those kind of pages, instead of leaving it to the default VCL
implementation.

Regards,

-- 
Thomas Lecomte



From raygo22 at gmail.com  Wed May 13 21:34:37 2015
From: raygo22 at gmail.com (Raymond Gonzalez)
Date: Wed, 13 May 2015 17:34:37 -0400
Subject: Caching cookies with Varnish 4
In-Reply-To: <CAJban9BmV7e4pat_11MDtpUMGF4iYakhO5dW=NLrDaZx+b0+Xg@mail.gmail.com>
References: <CAGyfp2guy01cB2jJOZJCGrzsSBPRE__Cu6+0c3q28Xo0aXSR3A@mail.gmail.com>
 <CAJban9BmV7e4pat_11MDtpUMGF4iYakhO5dW=NLrDaZx+b0+Xg@mail.gmail.com>
Message-ID: <3AD28F4D-4895-4DFE-B1AB-7498FE9A16BB@gmail.com>

Hi Thomas. Thanks for that. No, I'm very new at working with varnish (1 day old) so don't really know how to do what you are telling me. Could you please let me know what I should put or where can I find a tutorial to do this? A lot of what I've found online is not for V4 so it's not working. Thanks!


> On May 13, 2015, at 5:30 PM, Thomas Lecomte <thomas.lecomte at virtual-expo.com> wrote:
> 
>> On Wed, May 13, 2015 at 5:33 PM, Raymond Gonzalez <raygo22 at gmail.com> wrote:
>> I want to be able to cache pages that have a cookie. Let's say I have a
>> cookie called "testcookie" that I set up with php.
>> 
>> [...]
>> 
>> Any idea? Thanks!
> 
> Hello Raymond,
> 
> Since your VCL code doesn't return anything itself, the default VCL
> implementation is run after your own.
> The default vcl_recv subroutine will return (pass) if req.http.Cookie
> is present, which is the first reason why your page isn't going to be
> cached (in fact, it won't even be looked up).
> Also, in vcl_backend_response, the returned object won't be cached if
> it contains a Set-Cookie HTTP header.
> 
> You might want to take care of these cases yourself if you want to
> cache those kind of pages, instead of leaving it to the default VCL
> implementation.
> 
> Regards,
> 
> -- 
> Thomas Lecomte



From thomas.lecomte at virtual-expo.com  Wed May 13 21:37:14 2015
From: thomas.lecomte at virtual-expo.com (Thomas Lecomte)
Date: Wed, 13 May 2015 23:37:14 +0200
Subject: Caching cookies with Varnish 4
In-Reply-To: <3AD28F4D-4895-4DFE-B1AB-7498FE9A16BB@gmail.com>
References: <CAGyfp2guy01cB2jJOZJCGrzsSBPRE__Cu6+0c3q28Xo0aXSR3A@mail.gmail.com>
 <CAJban9BmV7e4pat_11MDtpUMGF4iYakhO5dW=NLrDaZx+b0+Xg@mail.gmail.com>
 <3AD28F4D-4895-4DFE-B1AB-7498FE9A16BB@gmail.com>
Message-ID: <CAJban9A_F9tx2JLhBvwKk-_vs_x5KixBcHfqbTHvV-vMhDjkrQ@mail.gmail.com>

On Wed, May 13, 2015 at 11:34 PM, Raymond Gonzalez <raygo22 at gmail.com> wrote:
> Hi Thomas. Thanks for that. No, I'm very new at working with varnish (1 day old) so don't really know how to do what you are telling me. Could you please let me know what I should put or where can I find a tutorial to do this? A lot of what I've found online is not for V4 so it's not working. Thanks!

Have you checked out this page?

https://www.varnish-cache.org/trac/wiki/VCLExampleCacheCookies

It seems to be Varnish 4-compliant.

Hope this helps,

-- 
Thomas Lecomte



From raygo22 at gmail.com  Wed May 13 21:45:35 2015
From: raygo22 at gmail.com (Raymond Gonzalez)
Date: Wed, 13 May 2015 17:45:35 -0400
Subject: Caching cookies with Varnish 4
In-Reply-To: <CAJban9A_F9tx2JLhBvwKk-_vs_x5KixBcHfqbTHvV-vMhDjkrQ@mail.gmail.com>
References: <CAGyfp2guy01cB2jJOZJCGrzsSBPRE__Cu6+0c3q28Xo0aXSR3A@mail.gmail.com>
 <CAJban9BmV7e4pat_11MDtpUMGF4iYakhO5dW=NLrDaZx+b0+Xg@mail.gmail.com>
 <3AD28F4D-4895-4DFE-B1AB-7498FE9A16BB@gmail.com>
 <CAJban9A_F9tx2JLhBvwKk-_vs_x5KixBcHfqbTHvV-vMhDjkrQ@mail.gmail.com>
Message-ID: <59A09BDB-9406-4D3D-82CD-81C64C7422DF@gmail.com>

I did but it only says what to put in vcl_hash

> On May 13, 2015, at 5:37 PM, Thomas Lecomte <thomas.lecomte at virtual-expo.com> wrote:
> 
>> On Wed, May 13, 2015 at 11:34 PM, Raymond Gonzalez <raygo22 at gmail.com> wrote:
>> Hi Thomas. Thanks for that. No, I'm very new at working with varnish (1 day old) so don't really know how to do what you are telling me. Could you please let me know what I should put or where can I find a tutorial to do this? A lot of what I've found online is not for V4 so it's not working. Thanks!
> 
> Have you checked out this page?
> 
> https://www.varnish-cache.org/trac/wiki/VCLExampleCacheCookies
> 
> It seems to be Varnish 4-compliant.
> 
> Hope this helps,
> 
> -- 
> Thomas Lecomte



From raygo22 at gmail.com  Wed May 13 22:50:18 2015
From: raygo22 at gmail.com (Raymond Gonzalez)
Date: Wed, 13 May 2015 18:50:18 -0400
Subject: Caching cookies with Varnish 4
In-Reply-To: <59A09BDB-9406-4D3D-82CD-81C64C7422DF@gmail.com>
References: <CAGyfp2guy01cB2jJOZJCGrzsSBPRE__Cu6+0c3q28Xo0aXSR3A@mail.gmail.com>
 <CAJban9BmV7e4pat_11MDtpUMGF4iYakhO5dW=NLrDaZx+b0+Xg@mail.gmail.com>
 <3AD28F4D-4895-4DFE-B1AB-7498FE9A16BB@gmail.com>
 <CAJban9A_F9tx2JLhBvwKk-_vs_x5KixBcHfqbTHvV-vMhDjkrQ@mail.gmail.com>
 <59A09BDB-9406-4D3D-82CD-81C64C7422DF@gmail.com>
Message-ID: <CAGyfp2izMKwyUFCso1OxJiNz2=FMcYvmVnckaJVOdhp=bmi6cg@mail.gmail.com>

This is my current vcl, can you see what am I doing wrong? Thanks!:

sub vcl_recv {
    # Happens before we check if we have this in cache already.
    #
    # Typically you clean up the request here, removing cookies you don't
need,
    # rewriting the request, etc.


 if (req.method != "GET" &&
       req.method != "HEAD" &&
       req.method != "PUT" &&
       req.method != "POST" &&
       req.method != "TRACE" &&
       req.method != "OPTIONS" &&
       req.method != "DELETE") {
         /* Non-RFC2616 or CONNECT which is weird. */
         return (pipe);
     }

     if (req.method != "GET" && req.method != "HEAD") {
         /* We only deal with GET and HEAD by default */
         return (pass);
     }


#  if (req.http.Authorization ) {
#        /* Not cacheable by default */
#       return (pass);
#   }

        if (req.url ~ "test" && req.http.Cookie) {
         return (hash);
        }


        if ( req.http.Cookie) {
                 return (hash);
        }

       return (hash);
}


sub vcl_backend_response {
    # Happens after we have read the response headers from the backend.
    #
    # Here you clean the response headers, removing silly Set-Cookie headers
    # and other mistakes your backend does.
}

sub vcl_deliver {
    # Happens when we have all the pieces we need, and are about to send the
    # response to the client.
    #
    # You can do accounting or modifying the final object here.

  if (obj.hits > 0) {
       set resp.http.X-Cache = "HIT";
    } else {
       set resp.http.X-Cache = "MISS";
    }

}

sub vcl_hash {

     hash_data(req.url);
     if (req.http.host) {
         hash_data(req.http.host);
     } else {
         hash_data(server.ip);
     }



hash_data(req.http.Cookie);


# if( req.http.Cookie ~ "testvarnish" ) {
#        set req.http.X-Varnish-Hashed-On = regsub( req.http.Cookie,
"^.*?testvarnish=([^;]*);*.*$", "\1" );
#    }


# if(  req.http.X-Varnish-Hashed-On ) {
#       hash_data( req.http.X-Varnish-Hashed-On);
#    }

     return (lookup);
}

sub vcl_pass {
  # Called upon entering pass mode. In this mode, the request is passed on
to the backend, and the
  # backend's response is passed on to the client, but is not entered into
the cache. Subsequent
  # requests submitted over the same client connection are handled normally.
}



On Wed, May 13, 2015 at 5:45 PM, Raymond Gonzalez <raygo22 at gmail.com> wrote:

> I did but it only says what to put in vcl_hash
>
> > On May 13, 2015, at 5:37 PM, Thomas Lecomte <
> thomas.lecomte at virtual-expo.com> wrote:
> >
> >> On Wed, May 13, 2015 at 11:34 PM, Raymond Gonzalez <raygo22 at gmail.com>
> wrote:
> >> Hi Thomas. Thanks for that. No, I'm very new at working with varnish (1
> day old) so don't really know how to do what you are telling me. Could you
> please let me know what I should put or where can I find a tutorial to do
> this? A lot of what I've found online is not for V4 so it's not working.
> Thanks!
> >
> > Have you checked out this page?
> >
> > https://www.varnish-cache.org/trac/wiki/VCLExampleCacheCookies
> >
> > It seems to be Varnish 4-compliant.
> >
> > Hope this helps,
> >
> > --
> > Thomas Lecomte
>



-- 
Raymond Gonzalez
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150513/8380bc18/attachment.html>

From dridi at varni.sh  Sat May 16 13:19:07 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Sat, 16 May 2015 15:19:07 +0200
Subject: varnish content length replaced
In-Reply-To: <20150507063138.27163.qmail@pro-236-32.rediffmailpro.com>
References: <20150507063138.27163.qmail@pro-236-32.rediffmailpro.com>
Message-ID: <CABoVN9DjN8WY2qhK4O59MPWTZ19JZ2RfXbHR0dwkCpuwqFa1Sw@mail.gmail.com>

Hi,

Could you please send enough VCL to reproduce?

Please find below (and attached) a test case that doesn't reproduce
your issue. Since I only had the backend request, I inferred the
client request from it. As you can see, Varnish 4 (4.0.3 on my
machine) built-in VCL does nothing by default.

--->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8---
server s1 {
    rxreq
    txresp -hdr "Server: Apache" \
           -hdr {P3P: CP="ALL DSP COR LAW CUR DEVi TAI PSAi PSD
                          IVA IVD CONo HIS TELo OUR DEL SAM BUS LOC"
              policyref="http://xxx.domain.com/w3c/p3p.xml"} \
           -hdr "Last-Modified: Wed, 06 May 2015 06:54:03 GMT" \
           -hdr "Accept-Ranges: bytes" \
           -hdr "Content-Type: application/font-woff" \
           -bodylen 25008
} -start

varnish v1 -vcl+backend {
    # built-in VCL
} -start

client c1 {
    txreq -url "/ajaxprism/js_jq_1_0/css/fonts/roboto-regular-webfont.woff" \
          -hdr "Accept: application/font-woff;q=0.9,*/*;q=0.8" \
          -hdr "Accept-Language: en-US,en;q=0.5" \
          -hdr "Origin: http://xxx.domain.com" \
          -hdr "Pragma: no-cache" \
          -hdr "Referer:http://xxx.domain.com/ajaxprism/js_jq_1_0/css/style_2.css"\
          -hdr {User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:37.0) Gecko/20100101
                            Firefox/37.0} \
          -hdr "host: xxx.domain.com" \
          -hdr "Accept-Encoding: gzip"
    rxresp
    expect resp.http.content-length != <undef>
    expect resp.http.content-length == "25008"
} -run
--->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8---

I also don't understand how it's a problem to not have a C-L header
for chunked encoding.

Regards,
Dridi

On Thu, May 7, 2015 at 8:31 AM, Anand Shah <anand at rediff-inc.com> wrote:
> Hello,
>
>
> Using Varnish 4, I have a set of backends that're responding with a valid
> Content-Length header and no Transfer-Encoding header. On the first hit from
> a client, rather than responding to the client with those headers, Varnish
> is dropping the Content-Length header and adding Transfer-Encoding: chunked
> to the response. This is especially happening for fonts. The same data when
> fetched from apache and nginx works like a charm and continue to be
> problematic when used varnish in front.
>
> I've tried a number of semi-obvious things like below but all in vain:
>
> beresp.do_stream = true
> beresp.do_gzip = false
> unset req.http.Accept-Encoding
>
>
> Sample varnishlog <backend> for your reference.
>
>
>
> *   << BeReq    >> 247282680
> -   Begin          bereq 247282679 fetch
> -   Timestamp      Start: 1430979942.850688 0.000000 0.000000
> -   BereqMethod    GET
> -   BereqURL
> /ajaxprism/js_jq_1_0/css/fonts/roboto-regular-webfont.woff
> -   BereqProtocol  HTTP/1.1
> -   BereqHeader    Accept: application/font-woff;q=0.9,*/*;q=0.8
> -   BereqHeader    Accept-Language: en-US,en;q=0.5
> -   BereqHeader    Origin: http://xxx.domain.com
> -   BereqHeader    Pragma: no-cache
> -   BereqHeader    Referer:
> http://xxx.domain.com/ajaxprism/js_jq_1_0/css/style_2.css
> -   BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:37.0)
> Gecko/20100101 Firefox/37.0
> -   BereqHeader    X-Forwarded-For: ::1
> -   BereqHeader    host: xxx.domain.com
> -   BereqHeader    Accept-Encoding: gzip
> -   BereqHeader    X-Varnish: 247282680
> -   VCL_call       BACKEND_FETCH
> -   VCL_return     fetch
> -   BackendClose   24 jdelivery(xxx.xxx.xxx.xxx,,80) toolate
> -   BackendOpen    24 jdelivery(xxx.xxx.xxx.xxx,,80) xxx.xxx.xxx.xxx 35680
> -   Backend        24 jdelivery jdelivery(xxx.xxx.xxx.xxx,,80)
> -   Timestamp      Bereq: 1430979942.852142 0.001454 0.001454
> -   Timestamp      Beresp: 1430979942.855756 0.005068 0.003614
> -   BerespProtocol HTTP/1.1
> -   BerespStatus   200
> -   BerespReason   OK
> -   BerespHeader   Date: Thu, 07 May 2015 06:25:43 GMT
> -   BerespHeader   Server: Apache
> -   BerespHeader   P3P: CP="ALL DSP COR LAW CUR DEVi TAI PSAi PSD IVA IVD
> CONo HIS TELo OUR DEL SAM BUS LOC"
> policyref="http://xxx.domain.com/w3c/p3p.xml"
> -   BerespHeader   Last-Modified: Wed, 06 May 2015 06:54:03 GMT
> -   BerespHeader   Accept-Ranges: bytes
> -   BerespHeader   Content-Length: 25008
> -   BerespHeader   Content-Type: application/font-woff
> -   TTL            RFC 120 -1 -1 1430979943 1430979943 1430979943 0 0
> -   VCL_call       BACKEND_RESPONSE
> -   TTL            VCL 120 604800 0 1430979943
> -   VCL_return     deliver
> -   Storage        file s0
> -   ObjProtocol    HTTP/1.1
> -   ObjStatus      200
> -   ObjReason      OK
> -   ObjHeader      Date: Thu, 07 May 2015 06:25:43 GMT
> -   ObjHeader      Server: Apache
> -   ObjHeader      P3P: CP="ALL DSP COR LAW CUR DEVi TAI PSAi PSD IVA IVD
> CONo HIS TELo OUR DEL SAM BUS LOC"
> policyref="http://xxx.domain.com/w3c/p3p.xml"
> -   ObjHeader      Last-Modified: Wed, 06 May 2015 06:54:03 GMT
> -   ObjHeader      Content-Type: application/font-woff
> -   Fetch_Body     3 length stream
> -   BackendReuse   24 jdelivery(xxx.xxx.xxx.xxx,,80)
> -   Timestamp      BerespBody: 1430979942.859386 0.008698 0.003630
> -   Length         25008
> -   BereqAcct      449 0 449 337 25008 25345
> -   End
>
>
> Regards,
> Anand
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: woff.vtc
Type: application/octet-stream
Size: 1124 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150516/4a8bf500/attachment.obj>

From reader_1000 at hotmail.com  Sun May 17 08:58:49 2015
From: reader_1000 at hotmail.com (kioto mitsubisi)
Date: Sun, 17 May 2015 08:58:49 +0000
Subject: hit-for-pass clarification
In-Reply-To: <BLU182-W234FCD092246FDB80187FA94C50@phx.gbl>
References: <BLU182-W234FCD092246FDB80187FA94C50@phx.gbl>
Message-ID: <BLU182-W23D181F9CE99CA6DDE032294C50@phx.gbl>

Hi,

I am testing Varnish and really liked it. Thanks a lot for this great software.
While reading documentation and mailing-lists, I got confused by the concept of hit-for-pass.
When there is no cache object, Varnish will queue the incoming objects and send only one request to the backend,
and return the result. However if the response from backend is not cacheable, then a hit-for-pass object is generated and all the request will go directly to the backend. My question is that the given example for this scenario is response with set-cookie header. However, I put the following line in vecl_recv to remove cookies.

? ? unset req.http.cookie;

So in this case, if backend returns a response with set-cookie, will there be a hit-for-pass? Do I need to remove set-cookie in?vcl_backend_response in order to prevent that? Could you please help me to understand the concept?

Thanks in advance? 		 	   		  


From reader_1000 at hotmail.com  Sun May 17 09:59:18 2015
From: reader_1000 at hotmail.com (kioto mitsubisi)
Date: Sun, 17 May 2015 09:59:18 +0000
Subject: pass uses higher cpu than hit
Message-ID: <BLU182-W752497E40DDD7433CE323C94C50@phx.gbl>

Hi,

I was trying to load test my Varnish setup with wrk[1]. I have Varnish and Apache http server (backend) on same machine (RedHat 6).
I am testing for two scenarios.

First, I am making requests to page which Varnish caches and CPU usage of Varnish is around 17% on average:
wrk -t50 -c200 -d90s --latency? http://host/cached/hit_page.html

Second, I am making requests to page which Varnish passes to the backend since it is a static page and I cannot purge it timely when it changes therefore I prefer Varnish not to cache it. However, for this scenario CPU usage of Varnish is around 78% on average:
wrk -t50 -c200 -d90s --latency? http://host/nocache/pass_page.txt

It seemed a little bit strange to since I directly pass the request to the backend without any additonal processing. Could you please help me to find the reason? Where should I look for: operating system, backend, Varnish parameters? I put my VCL, varnishstat -1 output below. I also tried the "perf record" system-wide and put its output but I am not sure it is useful without having kernel debug packages.
Thanks in advance.

[1] https://github.com/wg/wrk

sub vcl_recv {
??? if (req.url !~ "^/cached"){
??????? return (pass);
??? }

??? unset req.http.cookie;

??? if (req.http.host ~ "(?i)^(www.)?example.com") {
??????? set req.http.host = "example.com";
??? }

??? # Strip hash, server doesn't need it.
??? if (req.url ~ "\#") {
??????? set req.url = regsub(req.url, "\#.*$", "");
??? }
??? 
??? # remove query string
??? set req.url = regsub(req.url, "\?.*$", "");
??? # remove trailing slash
??? set req.url = regsub(req.url, "/$", "");
??? 
??? if (req.method == "PURGE") {
??????? if (!client.ip ~ purgers) {
??????????? return(synth(405,"Not allowed"));
??????? }
??????? return (purge);
??? }

}

varnishstat -1:

MAIN.uptime????????????? 47764???????? 1.00 Child process uptime
MAIN.sess_conn???????????? 848???????? 0.02 Sessions accepted
MAIN.sess_drop?????????????? 0???????? 0.00 Sessions dropped
MAIN.sess_fail?????????????? 0???????? 0.00 Session accept failures
MAIN.sess_pipe_overflow??????????? 0???????? 0.00 Session pipe overflow
MAIN.client_req_400??????????????? 0???????? 0.00 Client requests received, subject to 400 errors
MAIN.client_req_411??????????????? 0???????? 0.00 Client requests received, subject to 411 errors
MAIN.client_req_413??????????????? 0???????? 0.00 Client requests received, subject to 413 errors
MAIN.client_req_417??????????????? 0???????? 0.00 Client requests received, subject to 417 errors
MAIN.client_req?????????????? 427203???????? 8.94 Good client requests received
MAIN.cache_hit???????????????? 72990???????? 1.53 Cache hits
MAIN.cache_hitpass???????????????? 0???????? 0.00 Cache hits for pass
MAIN.cache_miss????????????????? 152???????? 0.00 Cache misses
MAIN.backend_conn?????????????? 3537???????? 0.07 Backend conn. success
MAIN.backend_unhealthy???????????? 0???????? 0.00 Backend conn. not attempted
MAIN.backend_busy????????????????? 0???????? 0.00 Backend conn. too many
MAIN.backend_fail????????????????? 0???????? 0.00 Backend conn. failures
MAIN.backend_reuse??????????? 350666???????? 7.34 Backend conn. reuses
MAIN.backend_toolate????????????? 30???????? 0.00 Backend conn. was closed
MAIN.backend_recycle????????? 350705???????? 7.34 Backend conn. recycles
MAIN.backend_retry???????????????? 0???????? 0.00 Backend conn. retry
MAIN.fetch_head??????????????????? 0???????? 0.00 Fetch no body (HEAD)
MAIN.fetch_length???????????? 354193???????? 7.42 Fetch with Length
MAIN.fetch_chunked??????????????? 13???????? 0.00 Fetch chunked
MAIN.fetch_eof???????????????????? 0???????? 0.00 Fetch EOF
MAIN.fetch_bad???????????????????? 0???????? 0.00 Fetch bad T-E
MAIN.fetch_close?????????????????? 0???????? 0.00 Fetch wanted close
MAIN.fetch_oldhttp???????????????? 0???????? 0.00 Fetch pre HTTP/1.1 closed
MAIN.fetch_zero??????????????????? 0???????? 0.00 Fetch zero len body
MAIN.fetch_1xx???????????????????? 0???????? 0.00 Fetch no body (1xx)
MAIN.fetch_204???????????????????? 0???????? 0.00 Fetch no body (204)
MAIN.fetch_304???????????????????? 0???????? 0.00 Fetch no body (304)
MAIN.fetch_failed????????????????? 5???????? 0.00 Fetch failed (all causes)
MAIN.fetch_no_thread?????????????? 5???????? 0.00 Fetch failed (no thread)
MAIN.pools???????????????????????? 2????????? .?? Number of thread pools
MAIN.threads???????????????????? 100????????? .?? Total number of threads
MAIN.threads_limited?????????????? 2???????? 0.00 Threads hit max
MAIN.threads_created???????????? 453???????? 0.01 Threads created
MAIN.threads_destroyed?????????? 353???????? 0.01 Threads destroyed
MAIN.threads_failed??????????????? 0???????? 0.00 Thread creation failed
MAIN.thread_queue_len????????????? 0????????? .?? Length of session queue
MAIN.busy_sleep??????????????????? 0???????? 0.00 Number of requests sent to sleep on busy objhdr
MAIN.busy_wakeup?????????????????? 0???????? 0.00 Number of requests woken after sleep on busy objhdr
MAIN.sess_queued???????????????? 313???????? 0.01 Sessions queued for thread
MAIN.sess_dropped????????????????? 5???????? 0.00 Sessions dropped for thread
MAIN.n_object???????????????????? 89????????? .?? object structs made
MAIN.n_vampireobject?????????????? 0????????? .?? unresurrected objects
MAIN.n_objectcore??????????????? 115????????? .?? objectcore structs made
MAIN.n_objecthead??????????????? 115????????? .?? objecthead structs made
MAIN.n_waitinglist??????????????? 26????????? .?? waitinglist structs made
MAIN.n_backend???????????????????? 1????????? .?? Number of backends
MAIN.n_expired??????????????????? 61????????? .?? Number of expired objects
MAIN.n_lru_nuked?????????????????? 0????????? .?? Number of LRU nuked objects
MAIN.n_lru_moved???????????????? 120????????? .?? Number of LRU moved objects
MAIN.losthdr?????????????????????? 0???????? 0.00 HTTP header overflows
MAIN.s_sess????????????????????? 848???????? 0.02 Total sessions seen
MAIN.s_req??????????????????? 427203???????? 8.94 Total requests seen
MAIN.s_pipe??????????????????????? 0???????? 0.00 Total pipe sessions seen
MAIN.s_pass?????????????????? 354059???????? 7.41 Total pass-ed requests seen
MAIN.s_fetch????????????????? 354211???????? 7.42 Total backend fetches initiated
MAIN.s_synth?????????????????????? 7???????? 0.00 Total synthethic responses made
MAIN.s_req_hdrbytes???????? 23873180?????? 499.82 Request header bytes
MAIN.s_req_bodybytes?????????????? 0???????? 0.00 Request body bytes
MAIN.s_resp_hdrbytes?????? 109855624????? 2299.97 Response header bytes
MAIN.s_resp_bodybytes???? 6030354123??? 126253.12 Response body bytes
MAIN.s_pipe_hdrbytes?????????????? 0???????? 0.00 Pipe request header bytes
MAIN.s_pipe_in???????????????????? 0???????? 0.00 Piped bytes from client
MAIN.s_pipe_out??????????????????? 0???????? 0.00 Piped bytes to client
MAIN.sess_closed?????????????????? 0???????? 0.00 Session Closed
MAIN.sess_pipeline???????????????? 0???????? 0.00 Session Pipeline
MAIN.sess_readahead??????????????? 0???????? 0.00 Session Read Ahead
MAIN.sess_herd??????????????? 426561???????? 8.93 Session herd
MAIN.shm_records??????????? 33794054?????? 707.52 SHM records
MAIN.shm_writes????????????? 1598314??????? 33.46 SHM writes
MAIN.shm_flushes?????????????????? 0???????? 0.00 SHM flushes due to overflow
MAIN.shm_cont?????????????????? 3312???????? 0.07 SHM MTX contention
MAIN.shm_cycles?????????????????? 12???????? 0.00 SHM cycles through buffer
MAIN.sms_nreq????????????????????? 0???????? 0.00 SMS allocator requests
MAIN.sms_nobj????????????????????? 0????????? .?? SMS outstanding allocations
MAIN.sms_nbytes??????????????????? 0????????? .?? SMS outstanding bytes
MAIN.sms_balloc??????????????????? 0????????? .?? SMS bytes allocated
MAIN.sms_bfree???????????????????? 0????????? .?? SMS bytes freed
MAIN.backend_req????????????? 354204???????? 7.42 Backend requests made
MAIN.n_vcl???????????????????????? 2???????? 0.00 Number of loaded VCLs in total
MAIN.n_vcl_avail?????????????????? 2???????? 0.00 Number of VCLs available
MAIN.n_vcl_discard???????????????? 0???????? 0.00 Number of discarded VCLs
MAIN.bans????????????????????????? 1????????? .?? Count of bans
MAIN.bans_completed??????????????? 1????????? .?? Number of bans marked 'completed'
MAIN.bans_obj????????????????????? 0????????? .?? Number of bans using obj.*
MAIN.bans_req????????????????????? 0????????? .?? Number of bans using req.*
MAIN.bans_added??????????????????? 1???????? 0.00 Bans added
MAIN.bans_deleted????????????????? 0???????? 0.00 Bans deleted
MAIN.bans_tested?????????????????? 0???????? 0.00 Bans tested against objects (lookup)
MAIN.bans_obj_killed?????????????? 0???????? 0.00 Objects killed by bans (lookup)
MAIN.bans_lurker_tested??????????? 0???????? 0.00 Bans tested against objects (lurker)
MAIN.bans_tests_tested???????????? 0???????? 0.00 Ban tests tested against objects (lookup)
MAIN.bans_lurker_tests_tested??????????? 0???????? 0.00 Ban tests tested against objects (lurker)
MAIN.bans_lurker_obj_killed????????????? 0???????? 0.00 Objects killed by bans (lurker)
MAIN.bans_dups?????????????????????????? 0???????? 0.00 Bans superseded by other bans
MAIN.bans_lurker_contention????????????? 0???????? 0.00 Lurker gave way for lookup
MAIN.bans_persisted_bytes?????????????? 13????????? .?? Bytes used by the persisted ban lists
MAIN.bans_persisted_fragmentation??????????? 0????????? .?? Extra bytes in persisted ban lists due to fragmentation
MAIN.n_purges??????????????????????????????? 2????????? .?? Number of purge operations executed
MAIN.n_obj_purged??????????????????????????? 2????????? .?? Number of purged objects
MAIN.exp_mailed??????????????????????????? 154???????? 0.00 Number of objects mailed to expiry thread
MAIN.exp_received????????????????????????? 154???????? 0.00 Number of objects received by expiry thread
MAIN.hcb_nolock????????????????????????? 73144???????? 1.53 HCB Lookups without lock
MAIN.hcb_lock????????????????????????????? 152???????? 0.00 HCB Lookups with lock
MAIN.hcb_insert??????????????????????????? 152???????? 0.00 HCB Inserts
MAIN.esi_errors????????????????????????????? 0???????? 0.00 ESI parse errors (unlock)
MAIN.esi_warnings??????????????????????????? 0???????? 0.00 ESI parse warnings (unlock)
MAIN.vmods?????????????????????????????????? 0????????? .?? Loaded VMODs
MAIN.n_gzip????????????????????????????????? 0???????? 0.00 Gzip operations
MAIN.n_gunzip??????????????????????????????? 0???????? 0.00 Gunzip operations
MAIN.vsm_free?????????????????????????? 972544????????? .?? Free VSM space
MAIN.vsm_used???????????????????????? 83962064????????? .?? Used VSM space
MAIN.vsm_cooling???????????????????????????? 0????????? .?? Cooling VSM space
MAIN.vsm_overflow??????????????????????????? 0????????? .?? Overflow VSM space
MAIN.vsm_overflowed????????????????????????? 0???????? 0.00 Overflowed VSM space
MGT.uptime?????????????????????????????? 47764???????? 1.00 Management process uptime
MGT.child_start????????????????????????????? 1???????? 0.00 Child process started
MGT.child_exit?????????????????????????????? 0???????? 0.00 Child process normal exit
MGT.child_stop?????????????????????????????? 0???????? 0.00 Child process unexpected exit
MGT.child_died?????????????????????????????? 0???????? 0.00 Child process died (signal)
MGT.child_dump?????????????????????????????? 0???????? 0.00 Child process core dumped
MGT.child_panic????????????????????????????? 0???????? 0.00 Child process panic
MEMPOOL.vbc.live???????????????????????????? 6????????? .?? In use
MEMPOOL.vbc.pool??????????????????????????? 10????????? .?? In Pool
MEMPOOL.vbc.sz_wanted?????????????????????? 88????????? .?? Size requested
MEMPOOL.vbc.sz_needed????????????????????? 120????????? .?? Size allocated
MEMPOOL.vbc.allocs??????????????????????? 3537???????? 0.07 Allocations
MEMPOOL.vbc.frees???????????????????????? 3531???????? 0.07 Frees
MEMPOOL.vbc.recycle?????????????????????? 3469???????? 0.07 Recycled from pool
MEMPOOL.vbc.timeout??????????????????????? 182???????? 0.00 Timed out from pool
MEMPOOL.vbc.toosmall???????????????????????? 0???????? 0.00 Too small to recycle
MEMPOOL.vbc.surplus????????????????????????? 0???????? 0.00 Too many for pool
MEMPOOL.vbc.randry????????????????????????? 68???????? 0.00 Pool ran dry
MEMPOOL.busyobj.live???????????????????????? 0????????? .?? In use
MEMPOOL.busyobj.pool??????????????????????? 10????????? .?? In Pool
MEMPOOL.busyobj.sz_wanted??????????????? 65536????????? .?? Size requested
MEMPOOL.busyobj.sz_needed??????????????? 65568????????? .?? Size allocated
MEMPOOL.busyobj.allocs????????????????? 354211???????? 7.42 Allocations
MEMPOOL.busyobj.frees?????????????????? 354211???????? 7.42 Frees
MEMPOOL.busyobj.recycle???????????????? 354063???????? 7.41 Recycled from pool
MEMPOOL.busyobj.timeout??????????????????? 212???????? 0.00 Timed out from pool
MEMPOOL.busyobj.toosmall???????????????????? 0???????? 0.00 Too small to recycle
MEMPOOL.busyobj.surplus????????????????????? 0???????? 0.00 Too many for pool
MEMPOOL.busyobj.randry???????????????????? 148???????? 0.00 Pool ran dry
MEMPOOL.req0.live??????????????????????????? 0????????? .?? In use
MEMPOOL.req0.pool?????????????????????????? 10????????? .?? In Pool
MEMPOOL.req0.sz_wanted?????????????????? 65536????????? .?? Size requested
MEMPOOL.req0.sz_needed?????????????????? 65568????????? .?? Size allocated
MEMPOOL.req0.allocs???????????????????? 249219???????? 5.22 Allocations
MEMPOOL.req0.frees????????????????????? 249219???????? 5.22 Frees
MEMPOOL.req0.recycle??????????????????? 248702???????? 5.21 Recycled from pool
MEMPOOL.req0.timeout?????????????????????? 559???????? 0.01 Timed out from pool
MEMPOOL.req0.toosmall??????????????????????? 0???????? 0.00 Too small to recycle
MEMPOOL.req0.surplus??????????????????????? 40???????? 0.00 Too many for pool
MEMPOOL.req0.randry??????????????????????? 517???????? 0.01 Pool ran dry
MEMPOOL.sess0.live?????????????????????????? 0????????? .?? In use
MEMPOOL.sess0.pool????????????????????????? 10????????? .?? In Pool
MEMPOOL.sess0.sz_wanted??????????????????? 384????????? .?? Size requested
MEMPOOL.sess0.sz_needed??????????????????? 416????????? .?? Size allocated
MEMPOOL.sess0.allocs?????????????????????? 441???????? 0.01 Allocations
MEMPOOL.sess0.frees??????????????????????? 441???????? 0.01 Frees
MEMPOOL.sess0.recycle?????????????????????? 64???????? 0.00 Recycled from pool
MEMPOOL.sess0.timeout????????????????????? 369???????? 0.01 Timed out from pool
MEMPOOL.sess0.toosmall?????????????????????? 0???????? 0.00 Too small to recycle
MEMPOOL.sess0.surplus?????????????????????? 67???????? 0.00 Too many for pool
MEMPOOL.sess0.randry?????????????????????? 377???????? 0.01 Pool ran dry
MEMPOOL.req1.live??????????????????????????? 0????????? .?? In use
MEMPOOL.req1.pool?????????????????????????? 10????????? .?? In Pool
MEMPOOL.req1.sz_wanted?????????????????? 65536????????? .?? Size requested
MEMPOOL.req1.sz_needed?????????????????? 65568????????? .?? Size allocated
MEMPOOL.req1.allocs???????????????????? 178151???????? 3.73 Allocations
MEMPOOL.req1.frees????????????????????? 178151???????? 3.73 Frees
MEMPOOL.req1.recycle??????????????????? 177736???????? 3.72 Recycled from pool
MEMPOOL.req1.timeout?????????????????????? 481???????? 0.01 Timed out from pool
MEMPOOL.req1.toosmall??????????????????????? 0???????? 0.00 Too small to recycle
MEMPOOL.req1.surplus??????????????????????? 23???????? 0.00 Too many for pool
MEMPOOL.req1.randry??????????????????????? 415???????? 0.01 Pool ran dry
MEMPOOL.sess1.live?????????????????????????? 0????????? .?? In use
MEMPOOL.sess1.pool????????????????????????? 10????????? .?? In Pool
MEMPOOL.sess1.sz_wanted??????????????????? 384????????? .?? Size requested
MEMPOOL.sess1.sz_needed??????????????????? 416????????? .?? Size allocated
MEMPOOL.sess1.allocs?????????????????????? 407???????? 0.01 Allocations
MEMPOOL.sess1.frees??????????????????????? 407???????? 0.01 Frees
MEMPOOL.sess1.recycle?????????????????????? 64???????? 0.00 Recycled from pool
MEMPOOL.sess1.timeout????????????????????? 339???????? 0.01 Timed out from pool
MEMPOOL.sess1.toosmall?????????????????????? 0???????? 0.00 Too small to recycle
MEMPOOL.sess1.surplus?????????????????????? 64???????? 0.00 Too many for pool
MEMPOOL.sess1.randry?????????????????????? 343???????? 0.01 Pool ran dry
SMA.s0.c_req?????????????????????????????? 327???????? 0.01 Allocator requests
SMA.s0.c_fail??????????????????????????????? 0???????? 0.00 Allocator failures
SMA.s0.c_bytes???????????????????????? 5656978?????? 118.44 Bytes allocated
SMA.s0.c_freed???????????????????????? 2492444??????? 52.18 Bytes freed
SMA.s0.g_alloc???????????????????????????? 194????????? .?? Allocations outstanding
SMA.s0.g_bytes???????????????????????? 3164534????????? .?? Bytes outstanding
SMA.s0.g_space????????????????????? 1070577290????????? .?? Bytes available
SMA.Transient.c_req???????????????????? 708108??????? 14.83 Allocator requests
SMA.Transient.c_fail???????????????????????? 0???????? 0.00 Allocator failures
SMA.Transient.c_bytes?????????????? 2703443424???? 56600.02 Bytes allocated
SMA.Transient.c_freed?????????????? 2703443424???? 56600.02 Bytes freed
SMA.Transient.g_alloc??????????????????????? 0????????? .?? Allocations outstanding
SMA.Transient.g_bytes??????????????????????? 0????????? .?? Bytes outstanding
SMA.Transient.g_space??????????????????????? 0????????? .?? Bytes available
VBE.default(127.0.0.1,,8080).vcls??????????? 2????????? .?? VCL references
VBE.default(127.0.0.1,,8080).happy??????????? 0????????? .?? Happy health probes
VBE.default(127.0.0.1,,8080).bereq_hdrbytes???? 47588248?????? 996.32 Request header bytes
VBE.default(127.0.0.1,,8080).bereq_bodybytes??????????? 0???????? 0.00 Request body bytes
VBE.default(127.0.0.1,,8080).beresp_hdrbytes???? 72693732????? 1521.94 Response header bytes
VBE.default(127.0.0.1,,8080).beresp_bodybytes?? 2473817104???? 51792.50 Response body bytes
VBE.default(127.0.0.1,,8080).pipe_hdrbytes?????????????? 0???????? 0.00 Pipe request header bytes
VBE.default(127.0.0.1,,8080).pipe_out??????????????????? 0???????? 0.00 Piped bytes to backend
VBE.default(127.0.0.1,,8080).pipe_in???????????????????? 0???????? 0.00 Piped bytes from backend
LCK.sms.creat??????????????????????????????????????????? 0???????? 0.00 Created locks
LCK.sms.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.sms.locks??????????????????????????????????????????? 0???????? 0.00 Lock Operations
LCK.smp.creat??????????????????????????????????????????? 0???????? 0.00 Created locks
LCK.smp.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.smp.locks??????????????????????????????????????????? 0???????? 0.00 Lock Operations
LCK.sma.creat??????????????????????????????????????????? 2???????? 0.00 Created locks
LCK.sma.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.sma.locks????????????????????????????????????? 1416676??????? 29.66 Lock Operations
LCK.smf.creat??????????????????????????????????????????? 0???????? 0.00 Created locks
LCK.smf.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.smf.locks??????????????????????????????????????????? 0???????? 0.00 Lock Operations
LCK.hsl.creat??????????????????????????????????????????? 0???????? 0.00 Created locks
LCK.hsl.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.hsl.locks??????????????????????????????????????????? 0???????? 0.00 Lock Operations
LCK.hcb.creat??????????????????????????????????????????? 1???????? 0.00 Created locks
LCK.hcb.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.hcb.locks????????????????????????????????????????? 481???????? 0.01 Lock Operations
LCK.hcl.creat??????????????????????????????????????????? 0???????? 0.00 Created locks
LCK.hcl.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.hcl.locks??????????????????????????????????????????? 0???????? 0.00 Lock Operations
LCK.vcl.creat??????????????????????????????????????????? 1???????? 0.00 Created locks
LCK.vcl.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.vcl.locks?????????????????????????????????????? 710015??????? 14.87 Lock Operations
LCK.sessmem.creat??????????????????????????????????????? 0???????? 0.00 Created locks
LCK.sessmem.destroy????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.sessmem.locks??????????????????????????????????????? 0???????? 0.00 Lock Operations
LCK.sess.creat???????????????????????????????????????? 848???????? 0.02 Created locks
LCK.sess.destroy?????????????????????????????????????? 848???????? 0.02 Destroyed locks
LCK.sess.locks?????????????????????????????????????????? 0???????? 0.00 Lock Operations
LCK.wstat.creat????????????????????????????????????????? 1???????? 0.00 Created locks
LCK.wstat.destroy??????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.wstat.locks???????????????????????????????????? 791316??????? 16.57 Lock Operations
LCK.herder.creat???????????????????????????????????????? 0???????? 0.00 Created locks
LCK.herder.destroy?????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.herder.locks???????????????????????????????????????? 0???????? 0.00 Lock Operations
LCK.wq.creat???????????????????????????????????????????? 3???????? 0.00 Created locks
LCK.wq.destroy?????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.wq.locks?????????????????????????????????????? 2365705??????? 49.53 Lock Operations
LCK.objhdr.creat?????????????????????????????????????? 353???????? 0.01 Created locks
LCK.objhdr.destroy???????????????????????????????????? 237???????? 0.00 Destroyed locks
LCK.objhdr.locks?????????????????????????????????? 4623042??????? 96.79 Lock Operations
LCK.exp.creat??????????????????????????????????????????? 1???????? 0.00 Created locks
LCK.exp.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.exp.locks??????????????????????????????????????? 10003???????? 0.21 Lock Operations
LCK.lru.creat??????????????????????????????????????????? 2???????? 0.00 Created locks
LCK.lru.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.lru.locks????????????????????????????????????????? 505???????? 0.01 Lock Operations
LCK.cli.creat??????????????????????????????????????????? 1???????? 0.00 Created locks
LCK.cli.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.cli.locks??????????????????????????????????????? 15939???????? 0.33 Lock Operations
LCK.ban.creat??????????????????????????????????????????? 1???????? 0.00 Created locks
LCK.ban.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.ban.locks?????????????????????????????????????? 157052???????? 3.29 Lock Operations
LCK.vbp.creat??????????????????????????????????????????? 1???????? 0.00 Created locks
LCK.vbp.destroy????????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.vbp.locks??????????????????????????????????????????? 0???????? 0.00 Lock Operations
LCK.backend.creat??????????????????????????????????????? 1???????? 0.00 Created locks
LCK.backend.destroy????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.backend.locks?????????????????????????????????? 712009??????? 14.91 Lock Operations
LCK.vcapace.creat??????????????????????????????????????? 1???????? 0.00 Created locks
LCK.vcapace.destroy????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.vcapace.locks??????????????????????????????????????? 0???????? 0.00 Lock Operations
LCK.nbusyobj.creat?????????????????????????????????????? 0???????? 0.00 Created locks
LCK.nbusyobj.destroy???????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.nbusyobj.locks?????????????????????????????????????? 0???????? 0.00 Lock Operations
LCK.busyobj.creat?????????????????????????????????? 354194???????? 7.42 Created locks
LCK.busyobj.destroy???????????????????????????????? 354208???????? 7.42 Destroyed locks
LCK.busyobj.locks????????????????????????????????? 2658757??????? 55.66 Lock Operations
LCK.mempool.creat??????????????????????????????????????? 6???????? 0.00 Created locks
LCK.mempool.destroy????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.mempool.locks????????????????????????????????? 1828426??????? 38.28 Lock Operations
LCK.vxid.creat?????????????????????????????????????????? 1???????? 0.00 Created locks
LCK.vxid.destroy???????????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.vxid.locks???????????????????????????????????????? 417???????? 0.01 Lock Operations
LCK.pipestat.creat?????????????????????????????????????? 1???????? 0.00 Created locks
LCK.pipestat.destroy???????????????????????????????????? 0???????? 0.00 Destroyed locks
LCK.pipestat.locks?????????????????????????????????????? 0???????? 0.00 Lock Operations

perf report:
-? 40.48%???????????? init? [kernel.kallsyms]????????????? [k] native_safe_halt??????????????????????????????????????????????????????????????????????????? 
???? native_safe_halt????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
-? 20.92%????????? swapper? [kernel.kallsyms]????????????? [k] native_safe_halt??????????????????????????????????????????????????????????????????????????? 
???? native_safe_halt????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
-?? 2.33%???????? varnishd? [kernel.kallsyms]????????????? [k] _spin_unlock_irqrestore???????????????????????????????????????????????????????????????????? 
?? - _spin_unlock_irqrestore?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 30.72% pthread_cond_broadcast@@GLIBC_2.3.2???????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 28.50% pthread_cond_signal@@GLIBC_2.3.2??????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 15.73% __libc_writev?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 11.28% 0x351940e6fd??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 3.27% pthread_cond_timedwait@@GLIBC_2.3.2????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 3.09% __poll?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 2.78% 0x351940e75d???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 1.41% epoll_wait?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 0.75% __lll_unlock_wake??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
-?? 1.06%???????? varnishd? libc-2.12.so?????????????????? [.] __memset_sse2?????????????????????????????????????????????????????????????????????????????? 
???? __memset_sse2???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
-?? 0.88%???????? varnishd? [kernel.kallsyms]????????????? [k] finish_task_switch????????????????????????????????????????????????????????????????????????? 
?? - finish_task_switch??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 28.86% pthread_cond_wait@@GLIBC_2.3.2????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 23.55% pthread_cond_timedwait@@GLIBC_2.3.2???????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 16.71% 0x351940e75d??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 15.35% epoll_wait????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
??????? 7.22% pthread_cond_signal@@GLIBC_2.3.2???????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 7.22% __poll?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
????? + 0.74% __lll_lock_wait????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
-?? 0.88%???????? varnishd? libc-2.12.so?????????????????? [.] vfprintf??????????????????????????????????????????????????????????????????????????????????? 
???? vfprintf????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
-?? 0.81%???????? varnishd? libc-2.12.so?????????????????? [.] __printf_fp???????????????????????????????????????????????????????????????????????????????? 
???? __printf_fp?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 
+?? 0.66%???????? varnishd? varnishd?????????????????????? [.] 0x00000000000200fd????????????????????????????????????????????????????????????????????????? 
+?? 0.59%??????????? httpd? [kernel.kallsyms]????????????? [k] _spin_unlock_irqrestore???????????????????????????????????????????????????????????????????? 
+?? 0.54%???????? varnishd? [kernel.kallsyms]????????????? [k] native_read_tsc??? 
 		 	   		  


From raymond.jennings at nytimes.com  Mon May 18 14:55:53 2015
From: raymond.jennings at nytimes.com (Jennings III, Raymond)
Date: Mon, 18 May 2015 10:55:53 -0400
Subject: Problem when logrotate runs on Varnish
Message-ID: <CADFjDbMXVN=jx-9rCCA9HYMPzCEYrVVup9fORxebF17kc-dXuQ@mail.gmail.com>

I get an outage at midnight when logrotate runs on the Varnish logs.  I
don't see anything in syslog but I do get various alerts from Nagios, etc.

I have the following logrotate for Varnish.  I see others where they use
USR1 instead of HUP - not sure if that makes a difference.  Anything else I
could look for.  It smells of a "timing issue" but not much to go on or
prove my hunch.



/var/nyt/logs/varnish/*_log {
    daily
    missingok
    rotate 7
    compress
    dateext
    compresscmd /usr/bin/bzip2
    compressext .bz2
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/varnishlog.pid 2>/dev/null` 2>
/dev/null || true
        /bin/kill -HUP `cat /var/run/varnishncsa.pid 2>/dev/null` 2>
/dev/null || true
    endscript
    lastaction
        /usr/bin/php /var/nyt/bin/varnish/varnish-copylog-S3.php
>>/tmp/varnish-log-S3.txt 2>&1
    endscript
}


Raymond Jennings III
*nytimes.com <http://nytimes.com/>*
*Office: 212.556.7786 <212-556-7786>*

*iPhone: 914.330.5074 <914-330-5074>E-mail: Raymond.Jennings at nytimes.com
<Raymond.Jennings at nytimes.com>FaceTime: Raymond.Jennings at nytimes.com
<Raymond.Jennings at nytimes.com>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150518/837e5700/attachment.html>

From viktor.villafuerte at optusnet.com.au  Mon May 18 23:22:56 2015
From: viktor.villafuerte at optusnet.com.au (Viktor Villafuerte)
Date: Tue, 19 May 2015 09:22:56 +1000
Subject: varnish content length replaced
In-Reply-To: <CABoVN9DjN8WY2qhK4O59MPWTZ19JZ2RfXbHR0dwkCpuwqFa1Sw@mail.gmail.com>
References: <20150507063138.27163.qmail@pro-236-32.rediffmailpro.com>
 <CABoVN9DjN8WY2qhK4O59MPWTZ19JZ2RfXbHR0dwkCpuwqFa1Sw@mail.gmail.com>
Message-ID: <20150518232256.GC13253@optusnet.com.au>

Hi,

On Sat 16 May 2015 15:19:07, Dridi Boukelmoune wrote:
> Hi,
> 
> Could you please send enough VCL to reproduce?
> 
> Please find below (and attached) a test case that doesn't reproduce
> your issue. Since I only had the backend request, I inferred the
> client request from it. As you can see, Varnish 4 (4.0.3 on my
> machine) built-in VCL does nothing by default.
> 
> --->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8---
> server s1 {
>     rxreq
>     txresp -hdr "Server: Apache" \
>            -hdr {P3P: CP="ALL DSP COR LAW CUR DEVi TAI PSAi PSD
>                           IVA IVD CONo HIS TELo OUR DEL SAM BUS LOC"
>               policyref="http://xxx.domain.com/w3c/p3p.xml"} \
>            -hdr "Last-Modified: Wed, 06 May 2015 06:54:03 GMT" \
>            -hdr "Accept-Ranges: bytes" \
>            -hdr "Content-Type: application/font-woff" \
>            -bodylen 25008
> } -start
> 
> varnish v1 -vcl+backend {
>     # built-in VCL
> } -start
> 
> client c1 {
>     txreq -url "/ajaxprism/js_jq_1_0/css/fonts/roboto-regular-webfont.woff" \
>           -hdr "Accept: application/font-woff;q=0.9,*/*;q=0.8" \
>           -hdr "Accept-Language: en-US,en;q=0.5" \
>           -hdr "Origin: http://xxx.domain.com" \
>           -hdr "Pragma: no-cache" \
>           -hdr "Referer:http://xxx.domain.com/ajaxprism/js_jq_1_0/css/style_2.css"\
>           -hdr {User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:37.0) Gecko/20100101
>                             Firefox/37.0} \
>           -hdr "host: xxx.domain.com" \
>           -hdr "Accept-Encoding: gzip"
>     rxresp
>     expect resp.http.content-length != <undef>
>     expect resp.http.content-length == "25008"
> } -run
> --->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8-------->8---
> 
> I also don't understand how it's a problem to not have a C-L header
> for chunked encoding.


I think what's Anand says it's not that he's missing C-L header but that
Varnish adds it by itself..


http://stackoverflow.com/questions/23643233/how-do-i-disable-transfer-encoding-chunked-encoding-in-varnish

this worked for me - see the last post and 'do_esi' setting.


thanks

v




> 
> Regards,
> Dridi
> 
> On Thu, May 7, 2015 at 8:31 AM, Anand Shah <anand at rediff-inc.com> wrote:
> > Hello,
> >
> >
> > Using Varnish 4, I have a set of backends that're responding with a valid
> > Content-Length header and no Transfer-Encoding header. On the first hit from
> > a client, rather than responding to the client with those headers, Varnish
> > is dropping the Content-Length header and adding Transfer-Encoding: chunked
> > to the response. This is especially happening for fonts. The same data when
> > fetched from apache and nginx works like a charm and continue to be
> > problematic when used varnish in front.
> >
> > I've tried a number of semi-obvious things like below but all in vain:
> >
> > beresp.do_stream = true
> > beresp.do_gzip = false
> > unset req.http.Accept-Encoding
> >
> >
> > Sample varnishlog <backend> for your reference.
> >
> >
> >
> > *   << BeReq    >> 247282680
> > -   Begin          bereq 247282679 fetch
> > -   Timestamp      Start: 1430979942.850688 0.000000 0.000000
> > -   BereqMethod    GET
> > -   BereqURL
> > /ajaxprism/js_jq_1_0/css/fonts/roboto-regular-webfont.woff
> > -   BereqProtocol  HTTP/1.1
> > -   BereqHeader    Accept: application/font-woff;q=0.9,*/*;q=0.8
> > -   BereqHeader    Accept-Language: en-US,en;q=0.5
> > -   BereqHeader    Origin: http://xxx.domain.com
> > -   BereqHeader    Pragma: no-cache
> > -   BereqHeader    Referer:
> > http://xxx.domain.com/ajaxprism/js_jq_1_0/css/style_2.css
> > -   BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:37.0)
> > Gecko/20100101 Firefox/37.0
> > -   BereqHeader    X-Forwarded-For: ::1
> > -   BereqHeader    host: xxx.domain.com
> > -   BereqHeader    Accept-Encoding: gzip
> > -   BereqHeader    X-Varnish: 247282680
> > -   VCL_call       BACKEND_FETCH
> > -   VCL_return     fetch
> > -   BackendClose   24 jdelivery(xxx.xxx.xxx.xxx,,80) toolate
> > -   BackendOpen    24 jdelivery(xxx.xxx.xxx.xxx,,80) xxx.xxx.xxx.xxx 35680
> > -   Backend        24 jdelivery jdelivery(xxx.xxx.xxx.xxx,,80)
> > -   Timestamp      Bereq: 1430979942.852142 0.001454 0.001454
> > -   Timestamp      Beresp: 1430979942.855756 0.005068 0.003614
> > -   BerespProtocol HTTP/1.1
> > -   BerespStatus   200
> > -   BerespReason   OK
> > -   BerespHeader   Date: Thu, 07 May 2015 06:25:43 GMT
> > -   BerespHeader   Server: Apache
> > -   BerespHeader   P3P: CP="ALL DSP COR LAW CUR DEVi TAI PSAi PSD IVA IVD
> > CONo HIS TELo OUR DEL SAM BUS LOC"
> > policyref="http://xxx.domain.com/w3c/p3p.xml"
> > -   BerespHeader   Last-Modified: Wed, 06 May 2015 06:54:03 GMT
> > -   BerespHeader   Accept-Ranges: bytes
> > -   BerespHeader   Content-Length: 25008
> > -   BerespHeader   Content-Type: application/font-woff
> > -   TTL            RFC 120 -1 -1 1430979943 1430979943 1430979943 0 0
> > -   VCL_call       BACKEND_RESPONSE
> > -   TTL            VCL 120 604800 0 1430979943
> > -   VCL_return     deliver
> > -   Storage        file s0
> > -   ObjProtocol    HTTP/1.1
> > -   ObjStatus      200
> > -   ObjReason      OK
> > -   ObjHeader      Date: Thu, 07 May 2015 06:25:43 GMT
> > -   ObjHeader      Server: Apache
> > -   ObjHeader      P3P: CP="ALL DSP COR LAW CUR DEVi TAI PSAi PSD IVA IVD
> > CONo HIS TELo OUR DEL SAM BUS LOC"
> > policyref="http://xxx.domain.com/w3c/p3p.xml"
> > -   ObjHeader      Last-Modified: Wed, 06 May 2015 06:54:03 GMT
> > -   ObjHeader      Content-Type: application/font-woff
> > -   Fetch_Body     3 length stream
> > -   BackendReuse   24 jdelivery(xxx.xxx.xxx.xxx,,80)
> > -   Timestamp      BerespBody: 1430979942.859386 0.008698 0.003630
> > -   Length         25008
> > -   BereqAcct      449 0 449 337 25008 25345
> > -   End
> >
> >
> > Regards,
> > Anand
> > _______________________________________________
> > varnish-dev mailing list
> > varnish-dev at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev


> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


-- 
Regards

Viktor Villafuerte
Optus Internet Engineering
t: 02 808-25265



From hage.jeremy at gmail.com  Tue May 19 12:36:13 2015
From: hage.jeremy at gmail.com (=?UTF-8?B?asOpcsOpbXkgSEFHRQ==?=)
Date: Tue, 19 May 2015 14:36:13 +0200
Subject: Pipe issue
Message-ID: <CAPC9jbGk_0pLQ48nyuuY9pzYLXT8BFBHzuyA7r6mimuUEmvvJg@mail.gmail.com>

Hello,

I am encountering an issue with varnish and pipes. When the number of
backend connections achieves max_connections, varnish doesn't send any
data.

The error message from curl is "curl: (52) Empty reply from server". That
seems to be a bug because, in my opinion Varnish is supposed to reply a 503
error at least.

It is easy to reproduce. I installed the official debian package (wheezy)
of varnish 4. Then I put this little configuration.

vcl 4.0;
backend default {
    .host = "127.0.0.1";
    .port = "80";
    .max_connections = 1;
}

sub vcl_recv {
    return (pipe);
}

Then I made 2 simultaneous connections. To make it as simplest as possible,
I installed Apache as backend with mod_php and I added a script which only
makes a "sleep(10)". When I make 2 curl simultaneously, the second gives me
this error "curl: (52) Empty reply from server" .

Is it an expected behavior?

Best regards,
--
J?r?my HAGE
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150519/c63bcc09/attachment.html>

From rlane at ahbelo.com  Tue May 19 13:30:16 2015
From: rlane at ahbelo.com (Lane, Richard)
Date: Tue, 19 May 2015 08:30:16 -0500
Subject: Problem when logrotate runs on Varnish
Message-ID: <CADt-aG1BCmRBkL0qReRs60podaZ7ehLAP0934puLs-y2Wx6kMg@mail.gmail.com>

Raymond,

We found that using the option 'delaycompress' helps ensure the Varnish
process is able to release the file before it is compressed. It seems the
Varnish process is having to wait or retry if we don't delay compression.
Then we use 'lastaction' to script the compress afterwards.

Cheers,

Richard Lane
AHBelo / The Dallas Morning News


---
Date: Mon, 18 May 2015 10:55:53 -0400
From: "Jennings III, Raymond" <raymond.jennings at nytimes.com>
To: varnish-misc <varnish-misc at varnish-cache.org>
Subject: Problem when logrotate runs on Varnish
Message-ID:
        <CADFjDbMXVN=jx-9rCCA9HYMPzCEYrVVup9fORxebF17kc-dXuQ at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I get an outage at midnight when logrotate runs on the Varnish logs.  I
don't see anything in syslog but I do get various alerts from Nagios, etc.

I have the following logrotate for Varnish.  I see others where they use
USR1 instead of HUP - not sure if that makes a difference.  Anything else I
could look for.  It smells of a "timing issue" but not much to go on or
prove my hunch.



/var/nyt/logs/varnish/*_log {
    daily
    missingok
    rotate 7
    compress
    dateext
    compresscmd /usr/bin/bzip2
    compressext .bz2
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/varnishlog.pid 2>/dev/null` 2>
/dev/null || true
        /bin/kill -HUP `cat /var/run/varnishncsa.pid 2>/dev/null` 2>
/dev/null || true
    endscript
    lastaction
        /usr/bin/php /var/nyt/bin/varnish/varnish-copylog-S3.php
>>/tmp/varnish-log-S3.txt 2>&1
    endscript
}


Raymond Jennings III
*nytimes.com <http://nytimes.com/>*
*Office: 212.556.7786 <212-556-7786>*

*iPhone: 914.330.5074 <914-330-5074>E-mail: Raymond.Jennings at nytimes.com
<Raymond.Jennings at nytimes.com>FaceTime: Raymond.Jennings at nytimes.com
<Raymond.Jennings at nytimes.com>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150519/fde9414d/attachment.html>

From dridi at varni.sh  Tue May 19 14:04:45 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 19 May 2015 16:04:45 +0200
Subject: Pipe issue
In-Reply-To: <CAPC9jbGk_0pLQ48nyuuY9pzYLXT8BFBHzuyA7r6mimuUEmvvJg@mail.gmail.com>
References: <CAPC9jbGk_0pLQ48nyuuY9pzYLXT8BFBHzuyA7r6mimuUEmvvJg@mail.gmail.com>
Message-ID: <CABoVN9BLeLtzv0s-qZqDXwQgoh-b0VMxpaZJf=tOeYnc=JHSWg@mail.gmail.com>

Hi,

Please find attached a test case that reproduces the behaviour.

Here's what happens for c2:

$ varnishtest -v 1pipe.vtc | awk '{ if ($2 == "c2") print}'
**   c2    0.4 Starting client
**   c2    0.4 Waiting for client
**   c2    0.4 Started (5 iterations)
***  c2    0.4 Connect to 127.0.0.1 34647
***  c2    0.4 connected fd 10 from 127.0.0.1 45762 to 127.0.0.1 34647
***  c2    0.4 txreq
**** c2    0.4 txreq| GET / HTTP/1.1\r\n
**** c2    0.4 txreq| \r\n
***  c2    0.4 rxresp
---- c2    0.4 HTTP rx EOF (fd:10 read: Success)

Should I file an issue in trac? It doesn't feel right.

Best Regards,
Dridi

On Tue, May 19, 2015 at 2:36 PM, j?r?my HAGE <hage.jeremy at gmail.com> wrote:
> Hello,
>
> I am encountering an issue with varnish and pipes. When the number of
> backend connections achieves max_connections, varnish doesn't send any data.
>
> The error message from curl is "curl: (52) Empty reply from server". That
> seems to be a bug because, in my opinion Varnish is supposed to reply a 503
> error at least.
>
> It is easy to reproduce. I installed the official debian package (wheezy) of
> varnish 4. Then I put this little configuration.
>
> vcl 4.0;
> backend default {
>     .host = "127.0.0.1";
>     .port = "80";
>     .max_connections = 1;
> }
>
> sub vcl_recv {
>     return (pipe);
> }
>
> Then I made 2 simultaneous connections. To make it as simplest as possible,
> I installed Apache as backend with mod_php and I added a script which only
> makes a "sleep(10)". When I make 2 curl simultaneously, the second gives me
> this error "curl: (52) Empty reply from server" .
>
> Is it an expected behavior?
>
> Best regards,
> --
> J?r?my HAGE
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1pipe.vtc
Type: application/octet-stream
Size: 359 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150519/2c562a3f/attachment.obj>

From dridi at varni.sh  Tue May 19 15:16:17 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 19 May 2015 17:16:17 +0200
Subject: Pipe issue
In-Reply-To: <CABoVN9BLeLtzv0s-qZqDXwQgoh-b0VMxpaZJf=tOeYnc=JHSWg@mail.gmail.com>
References: <CAPC9jbGk_0pLQ48nyuuY9pzYLXT8BFBHzuyA7r6mimuUEmvvJg@mail.gmail.com>
 <CABoVN9BLeLtzv0s-qZqDXwQgoh-b0VMxpaZJf=tOeYnc=JHSWg@mail.gmail.com>
Message-ID: <CABoVN9D9-DzxA=fJNGEiNmiPyEDMDsH2syiCV1j-VOX1occL8w@mail.gmail.com>

Hi J?r?my,

It actually makes sense both according to the documentation (the state
machines), and the goal of vcl_pipe.

If your request was asking for a protocol upgrade, you can no longer
respond with http. If you are piping http traffic, you should instead
rely on return(pass).

Best Regards,
Dridi

On Tue, May 19, 2015 at 4:04 PM, Dridi Boukelmoune <dridi at varni.sh> wrote:
> Hi,
>
> Please find attached a test case that reproduces the behaviour.
>
> Here's what happens for c2:
>
> $ varnishtest -v 1pipe.vtc | awk '{ if ($2 == "c2") print}'
> **   c2    0.4 Starting client
> **   c2    0.4 Waiting for client
> **   c2    0.4 Started (5 iterations)
> ***  c2    0.4 Connect to 127.0.0.1 34647
> ***  c2    0.4 connected fd 10 from 127.0.0.1 45762 to 127.0.0.1 34647
> ***  c2    0.4 txreq
> **** c2    0.4 txreq| GET / HTTP/1.1\r\n
> **** c2    0.4 txreq| \r\n
> ***  c2    0.4 rxresp
> ---- c2    0.4 HTTP rx EOF (fd:10 read: Success)
>
> Should I file an issue in trac? It doesn't feel right.
>
> Best Regards,
> Dridi
>
> On Tue, May 19, 2015 at 2:36 PM, j?r?my HAGE <hage.jeremy at gmail.com> wrote:
>> Hello,
>>
>> I am encountering an issue with varnish and pipes. When the number of
>> backend connections achieves max_connections, varnish doesn't send any data.
>>
>> The error message from curl is "curl: (52) Empty reply from server". That
>> seems to be a bug because, in my opinion Varnish is supposed to reply a 503
>> error at least.
>>
>> It is easy to reproduce. I installed the official debian package (wheezy) of
>> varnish 4. Then I put this little configuration.
>>
>> vcl 4.0;
>> backend default {
>>     .host = "127.0.0.1";
>>     .port = "80";
>>     .max_connections = 1;
>> }
>>
>> sub vcl_recv {
>>     return (pipe);
>> }
>>
>> Then I made 2 simultaneous connections. To make it as simplest as possible,
>> I installed Apache as backend with mod_php and I added a script which only
>> makes a "sleep(10)". When I make 2 curl simultaneously, the second gives me
>> this error "curl: (52) Empty reply from server" .
>>
>> Is it an expected behavior?
>>
>> Best regards,
>> --
>> J?r?my HAGE
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From varnishlist at amanzi.co.nz  Wed May 20 09:04:01 2015
From: varnishlist at amanzi.co.nz (stuartm)
Date: Wed, 20 May 2015 21:04:01 +1200
Subject: Varnish 4 on Debian 8 issue
Message-ID: <CAL2qbRG2F6B6Av+0XxSZTxRb3F0OuKALXtVyyvR0V-BqOAa2Bg@mail.gmail.com>

Hi there,

I've been trying to figure out an issue with Varnish 4 on Debian 8 and have
found a reproducible problem that I'd like to report... What I found was
that the Varnish service was starting okay but it wasn't using any of the
parameters I had set in /etc/default/varnish. For example, I had set the
listening port to be 80, but Varnish would always start with port 6081.

After reading this blog post:
https://ma.ttias.be/debug-varnish-4-x-on-systemd-that-fails-to-start, I
realised that the Varnish service (/lib/systemd/system/varnish.service)
wasn't using any parameter files - instead it was hard coded to start with
the default values only. Here's the output of the
/lib/systemd/system/varnish.service file:

[Unit]
Description=Varnish HTTP accelerator

[Service]
Type=forking
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStartPre=/usr/sbin/varnishd -C -f /etc/varnish/default.vcl
ExecStart=/usr/sbin/varnishd -a :6081 -T localhost:6082 -f
/etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
ExecReload=/usr/share/varnish/reload-vcl

[Install]
WantedBy=multi-user.target

This blog post: https://ma.ttias.be/running-varnish-4-x-on-systemd/,
indicated that there should be an environments file located here:
/etc/varnish/varnish.params and this file would be referenced in the
varnish.service file on this line:
EnvironmentFile=/etc/varnish/varnish.params. But as you can see the
varnish.service script doesn't include that line and the parameters file
doesn't exist on my Debian 8 system.

This issue is reproducible as I've just built another clean system and
tried again using both the Debian package as well as the Varnish-hosted
package which I installed using these instructions:
https://www.varnish-cache.org/installation/debian

My temporary fix is to edit the varnish.service file and manually configure
the parameters to match my needs, but this isn't ideal.

Hope this report helps.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150520/c93ccfea/attachment.html>

From dridi at varni.sh  Wed May 20 09:22:57 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 20 May 2015 11:22:57 +0200
Subject: Varnish 4 on Debian 8 issue
In-Reply-To: <CAL2qbRG2F6B6Av+0XxSZTxRb3F0OuKALXtVyyvR0V-BqOAa2Bg@mail.gmail.com>
References: <CAL2qbRG2F6B6Av+0XxSZTxRb3F0OuKALXtVyyvR0V-BqOAa2Bg@mail.gmail.com>
Message-ID: <CABoVN9AThaODFm-4cGOm=w2GGZf2qoA7H8WxTv6=nk3VpP0_pw@mail.gmail.com>

Hi,

On Wed, May 20, 2015 at 11:04 AM, stuartm <varnishlist at amanzi.co.nz> wrote:
> Hi there,
>
> I've been trying to figure out an issue with Varnish 4 on Debian 8 and have
> found a reproducible problem that I'd like to report... What I found was
> that the Varnish service was starting okay but it wasn't using any of the
> parameters I had set in /etc/default/varnish. For example, I had set the
> listening port to be 80, but Varnish would always start with port 6081.

Debian 8 ships varnish with both systemd and classic sysvinit
services. The file /etc/default/varnish is probably used by
/etc/init.d/varnish.

See this list of files:
https://packages.debian.org/jessie/amd64/varnish/filelist

> After reading this blog post:
> https://ma.ttias.be/debug-varnish-4-x-on-systemd-that-fails-to-start, I
> realised that the Varnish service (/lib/systemd/system/varnish.service)
> wasn't using any parameter files - instead it was hard coded to start with
> the default values only. Here's the output of the
> /lib/systemd/system/varnish.service file:
>
> [Unit]
> Description=Varnish HTTP accelerator
>
> [Service]
> Type=forking
> LimitNOFILE=131072
> LimitMEMLOCK=82000
> ExecStartPre=/usr/sbin/varnishd -C -f /etc/varnish/default.vcl
> ExecStart=/usr/sbin/varnishd -a :6081 -T localhost:6082 -f
> /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
> ExecReload=/usr/share/varnish/reload-vcl
>
> [Install]
> WantedBy=multi-user.target
>
> This blog post: https://ma.ttias.be/running-varnish-4-x-on-systemd/,
> indicated that there should be an environments file located here:
> /etc/varnish/varnish.params and this file would be referenced in the
> varnish.service file on this line:
> EnvironmentFile=/etc/varnish/varnish.params. But as you can see the
> varnish.service script doesn't include that line and the parameters file
> doesn't exist on my Debian 8 system.

This is how it was done in Fedora, but you can override the default
systemd service by adding your own varnish.service file in
/etc/systemd/system if you want to use an external files for
parameters or simply change the default ones.

See systemd's documentation:
http://www.freedesktop.org/software/systemd/man/systemd.unit.html

> This issue is reproducible as I've just built another clean system and tried
> again using both the Debian package as well as the Varnish-hosted package
> which I installed using these instructions:
> https://www.varnish-cache.org/installation/debian
>
> My temporary fix is to edit the varnish.service file and manually configure
> the parameters to match my needs, but this isn't ideal.
>
> Hope this report helps.

I hope my response will help too :)

Cheers,
Dridi



From reader_1000 at hotmail.com  Wed May 20 15:08:53 2015
From: reader_1000 at hotmail.com (kioto mitsubisi)
Date: Wed, 20 May 2015 15:08:53 +0000
Subject: hit-for-pass clarification
In-Reply-To: <BLU182-W23D181F9CE99CA6DDE032294C50@phx.gbl>
References: <BLU182-W234FCD092246FDB80187FA94C50@phx.gbl>,
 <BLU182-W23D181F9CE99CA6DDE032294C50@phx.gbl>
Message-ID: <BLU182-W13A1E8F9D4705BC99D45FA94C20@phx.gbl>

Hi,

Using a simple cgi script (as varnish book examples), I tested the scenario and it seems that I also need to remove set-cookie header from backend response, otherwise it create a hit-for-pass object.

#! /bin/sh
sleep 5
echo "Content-type: text/plain"
#echo "Cache-control: max-age=0"
echo "Set-Cookie: dummy=var; Path=/; HttpOnly"
echo
echo "Hello world"
date


 		 	   		  


From carlos.abalde at gmail.com  Wed May 20 15:21:28 2015
From: carlos.abalde at gmail.com (Carlos Abalde)
Date: Wed, 20 May 2015 17:21:28 +0200
Subject: hit-for-pass clarification
In-Reply-To: <BLU182-W13A1E8F9D4705BC99D45FA94C20@phx.gbl>
References: <BLU182-W234FCD092246FDB80187FA94C50@phx.gbl>
 <BLU182-W23D181F9CE99CA6DDE032294C50@phx.gbl>
 <BLU182-W13A1E8F9D4705BC99D45FA94C20@phx.gbl>
Message-ID: <3143C4E6-2E5A-4ABF-B4C6-531882D0A842@gmail.com>


> On 20 May 2015, at 17:08, kioto mitsubisi <reader_1000 at hotmail.com> wrote:
> 
> Hi,
> 
> Using a simple cgi script (as varnish book examples), I tested the scenario and it seems that I also need to remove set-cookie header from backend response, otherwise it create a hit-for-pass object.
> 
> #! /bin/sh
> sleep 5
> echo "Content-type: text/plain"
> #echo "Cache-control: max-age=0"
> echo "Set-Cookie: dummy=var; Path=/; HttpOnly"
> echo
> echo "Hello world"
> date

Hi,

'vcl_backend_response' in built-in VCL (https://github.com/varnish/Varnish-Cache/blob/master/bin/varnishd/builtin.vcl) creates a hit-for-pass object when the backend response contains a Set-Cookie header:

sub vcl_backend_response {
    if (beresp.ttl <= 0s ||
        beresp.http.Set-Cookie ||
        beresp.http.Surrogate-control ~ "no-store" ||
        (!beresp.http.Surrogate-Control &&
         beresp.http.Cache-Control ~ "no-cache|no-store|private") ||
        beresp.http.Vary == "*") {
        /*
         * Mark as "Hit-For-Pass" for the next 2 minutes
         */
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }
    return (deliver);
}

Please, check out https://github.com/varnish/Varnish-Book/blob/Varnish-Book-v4/varnish_book.rst#hit-for-pass for details.

Cheers,

--
Carlos Abalde




From reader_1000 at hotmail.com  Wed May 20 17:16:17 2015
From: reader_1000 at hotmail.com (kioto mitsubisi)
Date: Wed, 20 May 2015 17:16:17 +0000
Subject: hit-for-pass clarification
In-Reply-To: <3143C4E6-2E5A-4ABF-B4C6-531882D0A842@gmail.com>
References: <BLU182-W234FCD092246FDB80187FA94C50@phx.gbl>
 <BLU182-W23D181F9CE99CA6DDE032294C50@phx.gbl>
 <BLU182-W13A1E8F9D4705BC99D45FA94C20@phx.gbl>,
 <3143C4E6-2E5A-4ABF-B4C6-531882D0A842@gmail.com>
Message-ID: <BLU182-W85AD725E2CB8BA495A862094C20@phx.gbl>

Thank you for explanation and links.

 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150520/ff38b134/attachment.html>

From reader_1000 at hotmail.com  Wed May 20 17:20:15 2015
From: reader_1000 at hotmail.com (kioto mitsubisi)
Date: Wed, 20 May 2015 17:20:15 +0000
Subject: pass uses higher cpu than hit
In-Reply-To: <BLU182-W752497E40DDD7433CE323C94C50@phx.gbl>
References: <BLU182-W752497E40DDD7433CE323C94C50@phx.gbl>
Message-ID: <BLU182-W83DAC79C74D4C606FBADAC94C20@phx.gbl>

> From: reader_1000 at hotmail.com
> Hi,
> 
> First, I am making requests to page which Varnish caches and CPU usage of Varnish is around 17% on average:
> 
> Second, I am making requests to page which Varnish passes to the backend since it is a static page and I cannot purge it timely when it changes therefore I prefer Varnish not to cache it. However, for this scenario CPU usage of Varnish is around 78% on average:
> 
> It seemed a little bit strange to since I directly pass the request to the backend without any additonal processing. Could you please help me to find the reason? Where should I look for: operating system, backend, Varnish parameters? 

Hi,

Has anyone experienced a similar issue? Is it related to tcp connection establishment cost? What is the recommended keepalive value for backend which resides on the same machine with Varnish?

Thanks. 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150520/e58959c3/attachment.html>

From gabster at lelutin.ca  Wed May 20 18:45:12 2015
From: gabster at lelutin.ca (Gabriel Filion)
Date: Wed, 20 May 2015 14:45:12 -0400
Subject: Varnish 4 on Debian 8 issue
In-Reply-To: <CAL2qbRG2F6B6Av+0XxSZTxRb3F0OuKALXtVyyvR0V-BqOAa2Bg@mail.gmail.com>
References: <CAL2qbRG2F6B6Av+0XxSZTxRb3F0OuKALXtVyyvR0V-BqOAa2Bg@mail.gmail.com>
Message-ID: <555CD638.5040504@lelutin.ca>

fwiw I've seen the same issue. there's a bug report on debian about this
and the maintainer says the proper way to do things is to override the
.service file:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=749272

imho it makes it harder to maintain throughout upgrades :\ but it would
seem as though "that's how things are done with systemd"

On 20/05/15 05:04 AM, stuartm wrote:
> I've been trying to figure out an issue with Varnish 4 on Debian 8 and
> have found a reproducible problem that I'd like to report... What I
> found was that the Varnish service was starting okay but it wasn't using
> any of the parameters I had set in /etc/default/varnish. For example, I
> had set the listening port to be 80, but Varnish would always start with
> port 6081.
> 
> After reading this blog
> post: https://ma.ttias.be/debug-varnish-4-x-on-systemd-that-fails-to-start,
> I realised that the Varnish service
> (/lib/systemd/system/varnish.service) wasn't using any parameter files -
> instead it was hard coded to start with the default values only. Here's
> the output of the /lib/systemd/system/varnish.service file:
> 
> [Unit]
> Description=Varnish HTTP accelerator
> 
> [Service]
> Type=forking
> LimitNOFILE=131072
> LimitMEMLOCK=82000
> ExecStartPre=/usr/sbin/varnishd -C -f /etc/varnish/default.vcl
> ExecStart=/usr/sbin/varnishd -a :6081 -T localhost:6082 -f
> /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
> ExecReload=/usr/share/varnish/reload-vcl
> 
> [Install]
> WantedBy=multi-user.target
> 
> This blog post: https://ma.ttias.be/running-varnish-4-x-on-systemd/,
> indicated that there should be an environments file located here:
> /etc/varnish/varnish.params and this file would be referenced in the
> varnish.service file on this line:
> EnvironmentFile=/etc/varnish/varnish.params. But as you can see the
> varnish.service script doesn't include that line and the parameters file
> doesn't exist on my Debian 8 system.
> 
> This issue is reproducible as I've just built another clean system and
> tried again using both the Debian package as well as the Varnish-hosted
> package which I installed using these
> instructions: https://www.varnish-cache.org/installation/debian
> 
> My temporary fix is to edit the varnish.service file and manually
> configure the parameters to match my needs, but this isn't ideal.

-- 
Gabriel Filion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150520/f8268a09/attachment.bin>

From perbu at varnish-software.com  Thu May 21 06:48:03 2015
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 21 May 2015 08:48:03 +0200
Subject: pass uses higher cpu than hit
In-Reply-To: <BLU182-W83DAC79C74D4C606FBADAC94C20@phx.gbl>
References: <BLU182-W752497E40DDD7433CE323C94C50@phx.gbl>
 <BLU182-W83DAC79C74D4C606FBADAC94C20@phx.gbl>
Message-ID: <CAOXZevCEtQ376N23oDJtnOdkoU9MEAhKH379xwhWgy2s5g2vBQ@mail.gmail.com>

On Wed, May 20, 2015 at 7:20 PM, kioto mitsubisi <reader_1000 at hotmail.com>
wrote:

> (the worst email quoting I've even seen :-)
>


The reason why Varnish uses more CPU when pass'ing requests is because it
is a lot more work. A cache hit is delivered directly from (virtual) memory
- in most cases physical memory. So, a cache hit is just a writev() and
that is it, more or less.

With a pass you have to fetch it over the network and that involves another
TCP connection, talking to a remote server, parsing stuff, storing it
memory and then dumping it onto the connection.

So this behavior is expected.

-- 
*Per Buer*
CTO | Varnish Software AS
Cell: +47 95839117
We Make Websites Fly!
www.varnish-software.com
[image: Register now]
<http://info.varnish-software.com/varnish-summits-autumn-2014-registration>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150521/e76f32b2/attachment.html>

From reader_1000 at hotmail.com  Thu May 21 16:30:04 2015
From: reader_1000 at hotmail.com (kioto mitsubisi)
Date: Thu, 21 May 2015 16:30:04 +0000
Subject: pass uses higher cpu than hit
In-Reply-To: <CAOXZevCEtQ376N23oDJtnOdkoU9MEAhKH379xwhWgy2s5g2vBQ@mail.gmail.com>
References: <BLU182-W752497E40DDD7433CE323C94C50@phx.gbl>
 <BLU182-W83DAC79C74D4C606FBADAC94C20@phx.gbl>,
 <CAOXZevCEtQ376N23oDJtnOdkoU9MEAhKH379xwhWgy2s5g2vBQ@mail.gmail.com>
Message-ID: <BLU182-W29103FB11BD306214B4B2094C10@phx.gbl>

Hi
________________________________
> From: perbu at varnish-software.com 
> (the worst email quoting I've even seen :-) 

Sorry, I am trying to learn to post maillist and not very good at it :)

>  
>  
> The reason why Varnish uses more CPU when pass'ing requests is because  
> it is a lot more work. A cache hit is delivered directly from (virtual)  
> memory - in most cases physical memory. So, a cache hit is just a  
> writev() and that is it, more or less. 
>  
> With a pass you have to fetch it over the network and that involves  
> another TCP connection, talking to a remote server, parsing stuff,  
> storing it memory and then dumping it onto the connection. 
>  
> So this behavior is expected. 

I was confused  because when running tests without Varnish, Apache used a lot less CPU than Varnish but it seems that same things apply there, it is serving a static file which cached by Linux filesystem. I also thought my VCL which works only for hit objects would have more overhead, but I was wrong. Thanks for explanation.
 		 	   		  


From webteknik.tcmb at gmail.com  Mon May 25 15:12:15 2015
From: webteknik.tcmb at gmail.com (Web Teknik)
Date: Mon, 25 May 2015 18:12:15 +0300
Subject: /var/lib/varnish tmpfs recommended size and multiple _.vsm files
Message-ID: <CA+6ErLsjOif5U12e1NrtF4N__kfCPC-EF5WbY136=UWMhFA5NA@mail.gmail.com>

Hi,

After reading Varnish Book and Varnish wiki, we decided to put
/var/lib/varnish directory to tmpfs with following line in /etc/fstab

tmpfs                   /var/lib/varnish        tmpfs
defaults,noatime,size=128M        0 0


However, while trying restarting varnish, another shmlog file which is
named _.vsm.4311 is also stored under /var/lib/varnish/<host> whose
size is 42MB. As a result Varnish refused to start since disk is full.
We couldn't reproduce the issue that _.vsm.somenumber (PID?) is
created. Under which conditions multiple _.vsm files are created and
in order to prevent failure, what is the recommended size for
/var/lib/varnish directory?

Thanks in advance.



From webteknik.tcmb at gmail.com  Wed May 27 20:19:30 2015
From: webteknik.tcmb at gmail.com (Web Teknik)
Date: Wed, 27 May 2015 23:19:30 +0300
Subject: /var/lib/varnish tmpfs recommended size and multiple _.vsm files
In-Reply-To: <CA+6ErLsjOif5U12e1NrtF4N__kfCPC-EF5WbY136=UWMhFA5NA@mail.gmail.com>
References: <CA+6ErLsjOif5U12e1NrtF4N__kfCPC-EF5WbY136=UWMhFA5NA@mail.gmail.com>
Message-ID: <CA+6ErLssC123rPBGXhpH6_NmTf2BhQYpCY7PBo2A5-Gz9NaNdQ@mail.gmail.com>

Hi,

Any suggestions on size of /var/lib/varnish while mounted as tmpfs? We
only have one instance, therefore we thought it would be enough to set
it slightly bigger than 82MB, but while restarting Varnish, some other
shmlog file named _.vsm.4311 got created and Varnish failed to start
because of full disk. Therefore we could not be sure about size of
tmpfs.

Thanks

On Mon, May 25, 2015 at 6:12 PM, Web Teknik <webteknik.tcmb at gmail.com> wrote:
> Hi,
>
> After reading Varnish Book and Varnish wiki, we decided to put
> /var/lib/varnish directory to tmpfs with following line in /etc/fstab
>
> tmpfs                   /var/lib/varnish        tmpfs
> defaults,noatime,size=128M        0 0
>
>
> However, while trying restarting varnish, another shmlog file which is
> named _.vsm.4311 is also stored under /var/lib/varnish/<host> whose
> size is 42MB. As a result Varnish refused to start since disk is full.
> We couldn't reproduce the issue that _.vsm.somenumber (PID?) is
> created. Under which conditions multiple _.vsm files are created and
> in order to prevent failure, what is the recommended size for
> /var/lib/varnish directory?
>
> Thanks in advance.



From R.Guest at gns.cri.nz  Wed May 27 21:54:59 2015
From: R.Guest at gns.cri.nz (Richard Guest)
Date: Thu, 28 May 2015 09:54:59 +1200
Subject: /var/lib/varnish tmpfs recommended size and multiple _.vsm files
In-Reply-To: <CA+6ErLssC123rPBGXhpH6_NmTf2BhQYpCY7PBo2A5-Gz9NaNdQ@mail.gmail.com>
References: <CA+6ErLsjOif5U12e1NrtF4N__kfCPC-EF5WbY136=UWMhFA5NA@mail.gmail.com>
 <CA+6ErLssC123rPBGXhpH6_NmTf2BhQYpCY7PBo2A5-Gz9NaNdQ@mail.gmail.com>
Message-ID: <OF3B95F4EE.F83C8520-ONCC257E52.007773D5-CC257E52.00786416@gns.cri.nz>

Hi,

I'm afraid I can't help here, but I would like to pipe up and say that I 
have
noticed this behaviour on some of our varnish hosts also.

If someone more intimately familiar with the relevant code could chime in 
and
shed some light on this, then I'm sure we could figure out a sane number
fairly quickly.

My best guess (and I stress guess here, because I've done literally zero
investigation) is it's related to a client still reading from the SHM log, 
so
it's politely moved to one side, rather than just being destroyed.
Does that sound plausible? Anyone...?

Cheers,
Rich

> Hi,
> 
> Any suggestions on size of /var/lib/varnish while mounted as tmpfs? We
> only have one instance, therefore we thought it would be enough to set
> it slightly bigger than 82MB, but while restarting Varnish, some other
> shmlog file named _.vsm.4311 got created and Varnish failed to start
> because of full disk. Therefore we could not be sure about size of
> tmpfs.
> 
> Thanks
> 
> On Mon, May 25, 2015 at 6:12 PM, Web Teknik <webteknik.tcmb at gmail.com> 
wrote:
> > Hi,
> >
> > After reading Varnish Book and Varnish wiki, we decided to put
> > /var/lib/varnish directory to tmpfs with following line in /etc/fstab
> >
> > tmpfs                   /var/lib/varnish        tmpfs
> > defaults,noatime,size=128M        0 0
> >
> >
> > However, while trying restarting varnish, another shmlog file which is
> > named _.vsm.4311 is also stored under /var/lib/varnish/<host> whose
> > size is 42MB. As a result Varnish refused to start since disk is full.
> > We couldn't reproduce the issue that _.vsm.somenumber (PID?) is
> > created. Under which conditions multiple _.vsm files are created and
> > in order to prevent failure, what is the recommended size for
> > /var/lib/varnish directory?
> >
> > Thanks in advance.


Notice: This email and any attachments are confidential.
If received in error please destroy and immediately notify us.
Do not copy or disclose the contents.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150528/997e035b/attachment.html>

From daghf at varnish-software.com  Thu May 28 12:01:08 2015
From: daghf at varnish-software.com (Dag Haavi Finstad)
Date: Thu, 28 May 2015 14:01:08 +0200
Subject: /var/lib/varnish tmpfs recommended size and multiple _.vsm files
In-Reply-To: <OF3B95F4EE.F83C8520-ONCC257E52.007773D5-CC257E52.00786416@gns.cri.nz>
References: <CA+6ErLsjOif5U12e1NrtF4N__kfCPC-EF5WbY136=UWMhFA5NA@mail.gmail.com>
 <CA+6ErLssC123rPBGXhpH6_NmTf2BhQYpCY7PBo2A5-Gz9NaNdQ@mail.gmail.com>
 <OF3B95F4EE.F83C8520-ONCC257E52.007773D5-CC257E52.00786416@gns.cri.nz>
Message-ID: <CAEh05VYtToAxB2698Ary10SE=5wVe5GVUgLxrkbCdX6-uSZD1g@mail.gmail.com>

Hi

The situation is that some of the tools keep the old VSM file open
(varnishlog, varnishncsa, ...) which will prevent it from getting
cleaned up prior to Varnish creating a new one.

Sizing the tmpfs to fit two full VSM files should do the trick (at least ~170M).

Regards,
Dag

On Wed, May 27, 2015 at 11:54 PM, Richard Guest <R.Guest at gns.cri.nz> wrote:
> Hi,
>
> I'm afraid I can't help here, but I would like to pipe up and say that I
> have
> noticed this behaviour on some of our varnish hosts also.
>
> If someone more intimately familiar with the relevant code could chime in
> and
> shed some light on this, then I'm sure we could figure out a sane number
> fairly quickly.
>
> My best guess (and I stress guess here, because I've done literally zero
> investigation) is it's related to a client still reading from the SHM log,
> so
> it's politely moved to one side, rather than just being destroyed.
> Does that sound plausible? Anyone...?
>
> Cheers,
> Rich
>
>> Hi,
>>
>> Any suggestions on size of /var/lib/varnish while mounted as tmpfs? We
>> only have one instance, therefore we thought it would be enough to set
>> it slightly bigger than 82MB, but while restarting Varnish, some other
>> shmlog file named _.vsm.4311 got created and Varnish failed to start
>> because of full disk. Therefore we could not be sure about size of
>> tmpfs.
>>
>> Thanks
>>
>> On Mon, May 25, 2015 at 6:12 PM, Web Teknik <webteknik.tcmb at gmail.com>
>> wrote:
>> > Hi,
>> >
>> > After reading Varnish Book and Varnish wiki, we decided to put
>> > /var/lib/varnish directory to tmpfs with following line in /etc/fstab
>> >
>> > tmpfs                   /var/lib/varnish        tmpfs
>> > defaults,noatime,size=128M        0 0
>> >
>> >
>> > However, while trying restarting varnish, another shmlog file which is
>> > named _.vsm.4311 is also stored under /var/lib/varnish/<host> whose
>> > size is 42MB. As a result Varnish refused to start since disk is full.
>> > We couldn't reproduce the issue that _.vsm.somenumber (PID?) is
>> > created. Under which conditions multiple _.vsm files are created and
>> > in order to prevent failure, what is the recommended size for
>> > /var/lib/varnish directory?
>> >
>> > Thanks in advance.
>
> Notice: This email and any attachments are confidential. If received in
> error please destroy and immediately notify us. Do not copy or disclose the
> contents.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



-- 
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!



From geoff at uplex.de  Thu May 28 18:25:23 2015
From: geoff at uplex.de (Geoff Simmons)
Date: Thu, 28 May 2015 20:25:23 +0200
Subject: Workspace exhaustion with Varnish 4 under load
Message-ID: <55675D93.60801@uplex.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hello all,

We're getting our first experience with testing Varnish 4 under load
(v4.0.3), and we're having severe problems with workspace exhaustion
- -- LostHeader in the logs, lostheader stats counter increasing, and
various VMODs reporting insufficient workspace errors in the log. The
lostheader stat increased in bursts of thousands per second, mostly
around 2K/s, max ca. 9K/s. The lost headers and VMOD errors make the
proxies broken, and we've only seen the problem under load (not at all
with Varnish3).

On V3 we have sess_workspace=256KB, so on V4 we started out with
workspace_client=workspace_backend=256KB. We tried doubling the value
on each test to address the problem, but as of 16MB (64 times the
value in V3), it still wasn't enough. At 32MB, varnishd filled up the
RAM and crashed. With V3, we run well under 50% of available RAM on
same-sized machines.

Workspace config is different in V4, and memory pools are altogether
new, so I'm wondering what our mistake is.

Do we need to tune the memory pools? My reading of the code makes me
think that it won't help -- if workspace is too small, nothing about
the mempools can change that. Unless there are situations in which a
thread can't get workspace from a mempool?

I'm also learning how to read the new MEMPOOL.* stats. The code makes
me think that MEMPOOL.*.randry > 0 and MEMPOOL.*.timeouts > 0 are not
a problem (?). Does MEMPOOL.*.surplus > 0 indicate a problem?

It wouldn't surprise me if we need larger workspaces in V4 than in V3,
just not >64 times as much.

We're using new VMODs in V4 (necessarily since they have to be
changed), and we've tried commenting out / working around some of them
that we thought were possible suspects for using excessive workspace,
but so far it hasn't helped.

The logs show that the offending requests are *always* ESI-included.
We have some deep ESI nesting, up to at least esi_level==7. And we
also have some retry/restart logic for error responses, all of which
uses the same workspace, and I see that some of these show workspace
exhaustion. But that's no different in our V3 setup -- the backend
apps are the same, so the ESI nesting is as well, and 256KB of
workspace is enough.

I'd be very grateful for any pointers about where we should be looking.


Thanks,
Geoff
- -- 
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBCAAGBQJVZ12IAAoJEOUwvh9pJNURQcoP/2DO/e/E3TcDPeg9/UbzZ2l2
uaoTEGfkqVguF2UTynnpYCfJlWd3WOC1+8EOUg7QUXPTZ+Va/MVOkZFtij2ETZkh
gaXjj8UfFP3udyNtdagDQwLDdmrG+dcCS1epAXXKoLAq90dvJ+E3xiHeJpvjyetZ
RP9PyNnUb+tL8ozstWOJBsHoJ9sHufQ6nqvhk7ytOmg1XtWz52riViP8zh47UDKb
H+K2IOxaheMcHjFZt+idek99RfWFY1DQDGPuVzD5dqlvkowboZEkdA2G28iPJcHY
55sn6hbao5SDSZK1VLlq/xAj/T6iHkZo0L3UoPzAO13VHI8V28YNpq+DlnhnBfx6
gP9/N/WB9vwmYZQbRayBJecGUCH1vuO52QLZOnREh54kwDYPlyMfuehiZroRBH6N
3ok2VDV/SvT7S0kPC9vtBc2u1fr6WIAq1n6uwpABPbzmDHfs/H84nf+oSulHi70Y
jnsNX98lDQTE746Q92WdgrH+pShZuukUoP39kkz/4J7TjniGVNGYPFTqa3eSu9P4
U7BGwVucxC60RkDF6AM0zcsplYGp86c+UhSfYxradi6qDmY+LI+uVOQzfrwSEOVU
z5Wbo1R+S3/qdTgkrJSDUnHY4wsXcr1x/KhWSiYKdspzVA36uSoF5TtgDJvr9pCn
oYmwG+PZ+YB85mnkXnQ6
=u8Vg
-----END PGP SIGNATURE-----



From phk at phk.freebsd.dk  Thu May 28 21:26:54 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 28 May 2015 21:26:54 +0000
Subject: Workspace exhaustion with Varnish 4 under load
In-Reply-To: <55675D93.60801@uplex.de>
References: <55675D93.60801@uplex.de>
Message-ID: <7912.1432848414@critter.freebsd.dk>

--------

I have seen very little info/experience on tuning V4 yet, and have
no realistic setup to collect info/experience on myself.

Your workspace issue sounds like a clear bug to me, and ESI sounds
very probable as the culprit, sounds ticket-ish to me.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From geoff at uplex.de  Fri May 29 05:23:43 2015
From: geoff at uplex.de (Geoff Simmons)
Date: Fri, 29 May 2015 07:23:43 +0200
Subject: Workspace exhaustion with Varnish 4 under load
In-Reply-To: <7912.1432848414@critter.freebsd.dk>
References: <55675D93.60801@uplex.de> <7912.1432848414@critter.freebsd.dk>
Message-ID: <5567F7DF.4080303@uplex.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 05/28/2015 11:26 PM, Poul-Henning Kamp wrote:
> --------
> 
> I have seen very little info/experience on tuning V4 yet, and have 
> no realistic setup to collect info/experience on myself.
> 
> Your workspace issue sounds like a clear bug to me, and ESI sounds 
> very probable as the culprit, sounds ticket-ish to me.

Thanks phk, I've posted ticket #1743:

https://www.varnish-cache.org/trac/ticket/1743

- -- 
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBCAAGBQJVZ/ffAAoJEOUwvh9pJNUR0KAQAJPn748TC3Sayl+mHL7Uuc8k
EaSI38PF8Ti2pb7nDdwktZIOBiGxsvNl5H3G3X9yUh9KJVOH00Q8lD+gxG48xDis
c5u13/vCLepwomwJXauHLAYelJ5GgWvxtAUUxfHUE3InkK0Ypvl7UJ7mmmD1iiSj
EoHUgzj4mrZ5oIPdkr5c7EdSMnZzGiS+pQ0kc0tSLt16wklwMkqQvfAe+Oja/YhH
LWLG9vbFj0nCLESDXjJRTLabvGZy+TREsSKzpKJRCXkNQSZDkYIQn+qWif2FJTOt
GpdodJu4zETUUnBmz8t3d8CX677k27G7A17uOnqwyFEJ6WUx94Plawz2P0xxRCX+
Iy9tnIDz3dOjR7HFspTlIU0XwbMfJOq+ykY0nfZggeCd5DEpaH8JUeBbCEExK1Mp
uSVv6EJLFixkV4iuhBdYeXwUrR6gmdpWBz72Otrnk4dZMMCRZnPCAHgggSEd8V90
bjWedPgqaaadVpBGdMVs1TDJc7sTe7CaPAZnJd0rHSx+wHqBth45XE3xrtD0znNh
S82Ewm0znvTZ/rdH/jfWnWU6Cz9K9cfxF86Mc2WUFSK0jlzWZmvRM8sMGds0qSM+
Nd5c9Z+Pux0Fcv6jj1LLuxWxTcbQTibVpeRg0TgJaEyuo9HydsQXnN42ZICYuOtt
xHeuqkv27ixNZXw9orBi
=chTs
-----END PGP SIGNATURE-----



From reader_1000 at hotmail.com  Sat May 30 20:41:03 2015
From: reader_1000 at hotmail.com (kioto mitsubisi)
Date: Sat, 30 May 2015 20:41:03 +0000
Subject: HTTP 416 Content-Range:bytes */0 issue?
Message-ID: <BLU182-W3997A65DDBC30CBB16A24794C80@phx.gbl>

Hi,

>From time to time, Varnish 4 respondes the the client with HTTP 416 even though backend successfully returns HTTP 200.
The problem occurs when a pdf document is opened in Chromium and it send the "Content-Range:bytes */0" header.
Anyone experienced the issue before? Do we need to something about Accept-Range etc in my VCL? Currently we have a very simple VCL that removes cookies, lowers the URL and removes query string.

Thanks in advance.
 		 	   		  


From reader_1000 at hotmail.com  Sat May 30 20:54:07 2015
From: reader_1000 at hotmail.com (kioto mitsubisi)
Date: Sat, 30 May 2015 20:54:07 +0000
Subject: HTTP 416 Content-Range:bytes */0 issue?
In-Reply-To: <BLU182-W3997A65DDBC30CBB16A24794C80@phx.gbl>
References: <BLU182-W3997A65DDBC30CBB16A24794C80@phx.gbl>
Message-ID: <BLU182-W18795F97EE35DD5EC6BF5994C80@phx.gbl>

I am sorry, I wrote headers wrong. The request and response headers when we face the issue are below:

Request:
GET /url/to/pdf HTTP/1.1
Host: host
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/41.0.2272.76 Chrome/41.0.2272.76 Safari/537.36
Accept: */*
DNT: 1
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: _ga=GA1.3.208602568.1420406175
Range: bytes=0-32767


Response
HTTP/1.1 416 Requested Range Not Satisfiable
Date: Sat, 30 May 2015 20:40:27 GMT
Cache-Control: public
Expires: Sat, 30 May 2015 21:40:27 GMT
ETag: "343188177"
Content-Type: application/pdf
Content-Language: en-US
Age: 0
Connection: keep-alive
Accept-Ranges: bytes
Content-Range: bytes */0
 		 	   		  


From reader_1000 at hotmail.com  Sat May 30 22:17:34 2015
From: reader_1000 at hotmail.com (kioto mitsubisi)
Date: Sat, 30 May 2015 22:17:34 +0000
Subject: HTTP 416 Content-Range:bytes */0 issue?
In-Reply-To: <BLU182-W18795F97EE35DD5EC6BF5994C80@phx.gbl>
References: <BLU182-W3997A65DDBC30CBB16A24794C80@phx.gbl>,
 <BLU182-W18795F97EE35DD5EC6BF5994C80@phx.gbl>
Message-ID: <BLU182-W4387471D37D451EF83CC1D94C80@phx.gbl>

It seems that commenter 14 [1] from #1506 ticket has the same issue with us and it seems fixed now [2]. Is there a workaround we can use until the version this fix included is released?

[1] https://www.varnish-cache.org/trac/ticket/1506#comment:14

[2] https://www.varnish-cache.org/trac/changeset/97434c7cef70d99533f1fb7f78419dd5c9d36d23

Thanks in advance.
 		 	   		  


From phk at phk.freebsd.dk  Sat May 30 22:31:12 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 30 May 2015 22:31:12 +0000
Subject: HTTP 416 Content-Range:bytes */0 issue?
In-Reply-To: <BLU182-W4387471D37D451EF83CC1D94C80@phx.gbl>
References: <BLU182-W3997A65DDBC30CBB16A24794C80@phx.gbl>,
 <BLU182-W18795F97EE35DD5EC6BF5994C80@phx.gbl>
 <BLU182-W4387471D37D451EF83CC1D94C80@phx.gbl>
Message-ID: <76408.1433025072@critter.freebsd.dk>

--------
In message <BLU182-W4387471D37D451EF83CC1D94C80 at phx.gbl>, kioto mitsubisi write
s:
>It seems that commenter 14 [1] from #1506 ticket has the same issue
>with us and it seems fixed now [2]. Is there a workaround we can
>use until the version this fix included is released?

The easiest is probably to remove the affected Range: headers
in your vcl_recv{}


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From reader_1000 at hotmail.com  Sun May 31 08:05:35 2015
From: reader_1000 at hotmail.com (kioto mitsubisi)
Date: Sun, 31 May 2015 08:05:35 +0000
Subject: HTTP 416 Content-Range:bytes */0 issue?
In-Reply-To: <76408.1433025072@critter.freebsd.dk>
References: <BLU182-W3997A65DDBC30CBB16A24794C80@phx.gbl>,
 <BLU182-W18795F97EE35DD5EC6BF5994C80@phx.gbl>
 <BLU182-W4387471D37D451EF83CC1D94C80@phx.gbl>,
 <76408.1433025072@critter.freebsd.dk>
Message-ID: <BLU182-W28F8F8D8CEC556D587A9D694B70@phx.gbl>

Thanks you for your help phk, can I ask one more question if you don't mind? In another ticket (#1618) that your fix refers to, reporter said that putting the code below worked for him. ?Is this a valid workaround also?

sub vcl_backend_response {
? ? set beresp.do_stream = false;?
}

Thanks again

----------------------------------------
> To: reader_1000 at hotmail.com
> CC: varnish-misc at varnish-cache.org
> Subject: Re: HTTP 416 Content-Range:bytes */0 issue?
> From: phk at phk.freebsd.dk
> Date: Sat, 30 May 2015 22:31:12 +0000
>
> --------
> In message <BLU182-W4387471D37D451EF83CC1D94C80 at phx.gbl>, kioto mitsubisi write
> s:
>>It seems that commenter 14 [1] from #1506 ticket has the same issue
>>with us and it seems fixed now [2]. Is there a workaround we can
>>use until the version this fix included is released?
>
> The easiest is probably to remove the affected Range: headers
> in your vcl_recv{}
>
>
> --
> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG | TCP/IP since RFC 956
> FreeBSD committer | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
 		 	   		  


From phk at phk.freebsd.dk  Sun May 31 08:41:35 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 31 May 2015 08:41:35 +0000
Subject: HTTP 416 Content-Range:bytes */0 issue?
In-Reply-To: <BLU182-W28F8F8D8CEC556D587A9D694B70@phx.gbl>
References: <BLU182-W3997A65DDBC30CBB16A24794C80@phx.gbl>,
 <BLU182-W18795F97EE35DD5EC6BF5994C80@phx.gbl>
 <BLU182-W4387471D37D451EF83CC1D94C80@phx.gbl>,
 <76408.1433025072@critter.freebsd.dk>
 <BLU182-W28F8F8D8CEC556D587A9D694B70@phx.gbl>
Message-ID: <33497.1433061695@critter.freebsd.dk>

--------
In message <BLU182-W28F8F8D8CEC556D587A9D694B70 at phx.gbl>, kioto mitsubisi write
s:
>Thanks you for your help phk, can I ask one more question if you don't mind?
>In another ticket (#1618) that your fix refers to, reporter said that
> putting the code below worked for him. =A0Is this a valid workaround also?
>
>
>sub vcl_backend_response {
>   set beresp.do_stream = false;
>}

Yes, that should also work.

But you probably do not want to do that for all your traffic, but only
in case of the troublesome Range headers.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From reader_1000 at hotmail.com  Sun May 31 09:11:09 2015
From: reader_1000 at hotmail.com (kioto mitsubisi)
Date: Sun, 31 May 2015 09:11:09 +0000
Subject: HTTP 416 Content-Range:bytes */0 issue?
In-Reply-To: <33497.1433061695@critter.freebsd.dk>
References: <BLU182-W3997A65DDBC30CBB16A24794C80@phx.gbl>,
 <BLU182-W18795F97EE35DD5EC6BF5994C80@phx.gbl>
 <BLU182-W4387471D37D451EF83CC1D94C80@phx.gbl>,
 <76408.1433025072@critter.freebsd.dk>
 <BLU182-W28F8F8D8CEC556D587A9D694B70@phx.gbl>,
 <33497.1433061695@critter.freebsd.dk>
Message-ID: <BLU182-W164FF7034BEBB65386932E94B70@phx.gbl>


> Yes, that should also work.
>
> But you probably do not want to do that for all your traffic, but only
> in case of the troublesome Range headers.
>

We will edit our VCL according to your suggestions. Thank you very much for your help and explanations.