From sujithnss at gmail.com  Thu Sep  1 07:31:47 2016
From: sujithnss at gmail.com (sujith pv)
Date: Thu, 1 Sep 2016 13:01:47 +0530
Subject: Varnish malloc issue
Message-ID: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>

Hi All

I had already mailed this query long back but this time putting in a
different manner.


   - We are using Varnish 4.0 in our end. We have a machine with memory of
   16G with 8G being malloc for Varnish. We have a TTL for 2 hrs as well.
   - During our peak traffic, when we see the total memory of the machine
   reaching 90% and like varnishd process is taking some 89% .
   - So I'm just confused even though we had allocated just 8G malloc which
   is like 50% of the total memory, how the process is eating up 89% of memory
   and the memory is not releasing even after TTL.


Any help please...

Best Regards
Sujith P V
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160901/aba92c67/attachment.html>

From guillaume at varnish-software.com  Thu Sep  1 07:53:22 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Thu, 1 Sep 2016 09:53:22 +0200
Subject: Varnish malloc issue
In-Reply-To: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
Message-ID: <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>

Is that virtual or real memory?
Try to cap your Transient storage to 1GB, you may be overloaded with short
lived objects.

On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com> wrote:

> Hi All
>
> I had already mailed this query long back but this time putting in a
> different manner.
>
>
>    - We are using Varnish 4.0 in our end. We have a machine with memory
>    of 16G with 8G being malloc for Varnish. We have a TTL for 2 hrs as well.
>    - During our peak traffic, when we see the total memory of the machine
>    reaching 90% and like varnishd process is taking some 89% .
>    - So I'm just confused even though we had allocated just 8G malloc
>    which is like 50% of the total memory, how the process is eating up 89% of
>    memory and the memory is not releasing even after TTL.
>
>
> Any help please...
>
> Best Regards
> Sujith P V
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160901/bd39da62/attachment.html>

From apj at mutt.dk  Thu Sep  1 08:40:22 2016
From: apj at mutt.dk (Andreas Plesner)
Date: Thu, 1 Sep 2016 10:40:22 +0200
Subject: Varnish malloc issue
In-Reply-To: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
Message-ID: <20160901084022.GD23724@nerd.dk>

On Thu, Sep 01, 2016 at 01:01:47PM +0530, sujith pv wrote:
> 
> I had already mailed this query long back but this time putting in a
> different manner.

You still haven't answered the questions in the original thread, please don't
start a new one.

-- 
Andreas



From sujithnss at gmail.com  Thu Sep  1 09:11:30 2016
From: sujithnss at gmail.com (sujith pv)
Date: Thu, 1 Sep 2016 14:41:30 +0530
Subject: Varnish malloc issue
In-Reply-To: <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
Message-ID: <CAFzBPcWeVPVSwg-g0T8EyaM+by_qDuQtX3NLu9yCC9KAATff7g@mail.gmail.com>

Hi

Please find the below memory of Varnishd in my case (result from TOP
command)

[image: Inline image 3]
Here 14.9g being VIRT and 11g being RES.

@Andreas : Apologies for starting a new thread as previous thread got
diverted in between by a different query by someone. Actually I'm very
newbie to Varnish and just learning and trying to sort out issues.

Best Regards
Sujith P V

On Thu, Sep 1, 2016 at 1:23 PM, Guillaume Quintard <
guillaume at varnish-software.com> wrote:

> Is that virtual or real memory?
> Try to cap your Transient storage to 1GB, you may be overloaded with short
> lived objects.
>
> On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com> wrote:
>
>> Hi All
>>
>> I had already mailed this query long back but this time putting in a
>> different manner.
>>
>>
>>    - We are using Varnish 4.0 in our end. We have a machine with memory
>>    of 16G with 8G being malloc for Varnish. We have a TTL for 2 hrs as well.
>>    - During our peak traffic, when we see the total memory of the
>>    machine reaching 90% and like varnishd process is taking some 89% .
>>    - So I'm just confused even though we had allocated just 8G malloc
>>    which is like 50% of the total memory, how the process is eating up 89% of
>>    memory and the memory is not releasing even after TTL.
>>
>>
>> Any help please...
>>
>> Best Regards
>> Sujith P V
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160901/c8139659/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mem.png
Type: image/png
Size: 1287 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160901/c8139659/attachment.png>

From apj at mutt.dk  Thu Sep  1 09:52:00 2016
From: apj at mutt.dk (Andreas Plesner)
Date: Thu, 1 Sep 2016 11:52:00 +0200
Subject: Varnish malloc issue
In-Reply-To: <CAFzBPcWeVPVSwg-g0T8EyaM+by_qDuQtX3NLu9yCC9KAATff7g@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <CAFzBPcWeVPVSwg-g0T8EyaM+by_qDuQtX3NLu9yCC9KAATff7g@mail.gmail.com>
Message-ID: <20160901095200.GE23724@nerd.dk>

On Thu, Sep 01, 2016 at 02:41:30PM +0530, sujith pv wrote:
> 
> @Andreas : Apologies for starting a new thread as previous thread got
> diverted in between by a different query by someone. Actually I'm very
> newbie to Varnish and just learning and trying to sort out issues.

Please answer the previous questions: Did you compile varnish yourself? Is it
linked to libjemalloc?

-- 
Andreas



From albert.tollkuci at gmail.com  Thu Sep  1 09:57:53 2016
From: albert.tollkuci at gmail.com (=?UTF-8?Q?Albert_Tollku=C3=A7i?=)
Date: Thu, 1 Sep 2016 11:57:53 +0200
Subject: Caching for some cookies?
Message-ID: <CACqaJ6jSqZAHe9p53r=L-j=emP5PE6ZzmQGHEdymJmmjtJkMcw@mail.gmail.com>

Hello,
I've started using Varnish a few months ago and my vcl files has started to
get complicated. I'm serving 5 different sites with the same varnish
instance (using different backends) and this part is working.

Now I have some cases that for some sites I need to keep some cookies and
cache different versions based on them. Basic scenario is to show the user
a message only the first time he sees the page. Without varnish if the
cookie is not there I set it and show the message. Next time the cookie is
there and no message is shown.

With varnish my logic is to include the cookie in hash and cache two
different versions. The challenge is to make sure cookies are set correctly
and here I'm kind of stack. I've added some debug messages in my vcl and
for some reason I'm not seeing the cookies I set in my server in
vcl_backend_response. Inside the routine, I have the following code in the
beginning:

       # AT: DEBUG
       if(bereq.http.x-at-debug == "1") {
         std.log("AT DEBUG - bereq URL: " + bereq.url + ". Cookies: '" +
bereq.http.Cookie + "'");
         std.log("AT DEBUG - beresp set-cookie: '" +
 beresp.http.set-cookie + "'");
       }

I set x-at-debug header in vcl_recv based on my IP, so I can track only my
requests. In the server side I see the cookie "__IKUB_AL_Myikub_PopUp"
being set, but in vcl_backend_response is not.

Is there anything in between that can strip the cookie?

Thank you,
Albert



-- 
Web: http://www.tollkuci.com
Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
<https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
<https://www.facebook.com/albert.tollkuci> Twitter
<https://twitter.com/AlbertTollkuci> Career 2.0
<http://careers.stackoverflow.com/atollkuci>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160901/9c0c4661/attachment.html>

From sujithnss at gmail.com  Thu Sep  1 09:59:03 2016
From: sujithnss at gmail.com (sujith pv)
Date: Thu, 1 Sep 2016 15:29:03 +0530
Subject: Varnish malloc issue
In-Reply-To: <20160901095200.GE23724@nerd.dk>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <CAFzBPcWeVPVSwg-g0T8EyaM+by_qDuQtX3NLu9yCC9KAATff7g@mail.gmail.com>
 <20160901095200.GE23724@nerd.dk>
Message-ID: <CAFzBPcWe3EksqqO46L=qFwMz1JeitAKUS21Y8zQO-XXbQRdH-A@mail.gmail.com>

Here they have done it using tar.gz.(tar -zxvf varnish-4.0.3-rc1.tar.gz)
followed by make and make install
Sorry Im not sure around libjemalloc.

On Thu, Sep 1, 2016 at 3:22 PM, Andreas Plesner <apj at mutt.dk> wrote:

> On Thu, Sep 01, 2016 at 02:41:30PM +0530, sujith pv wrote:
> >
> > @Andreas : Apologies for starting a new thread as previous thread got
> > diverted in between by a different query by someone. Actually I'm very
> > newbie to Varnish and just learning and trying to sort out issues.
>
> Please answer the previous questions: Did you compile varnish yourself? Is
> it
> linked to libjemalloc?
>
> --
> Andreas
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160901/387803e6/attachment.html>

From ianmac51 at gmail.com  Thu Sep  1 17:58:55 2016
From: ianmac51 at gmail.com (Ian Macdonald)
Date: Thu, 1 Sep 2016 18:58:55 +0100
Subject: Sending a ? to the backend
Message-ID: <CAFeBtokBFHb809tnqtg4=i4faj9DfeCQuy=WnrMW1Q4_O77upg@mail.gmail.com>

Hi All,

I am having trouble rewriting a url I need to send a question mark to
the backend and it just does not seem to work I have tried escaping
the ?, long strings, regsub() and all seem to fail.  here is a snippet
if anyone can point me in the right direction

vcl_recv {
...

if (req.url ~ "^/myurl$") {

                        set req.url = {"/search?q=term"};
                        return (pass);




...

}

quick httpie check gets seems to show anything from the ? goes missing

 <p>The requested URL <code>/search</code> was not found on this
server.  <ins>That?s all we know.</ins>

varnish is 4.1.3 on centos 6.8

Cheers

Ian



From viktor.villafuerte at optusnet.com.au  Thu Sep  1 23:18:32 2016
From: viktor.villafuerte at optusnet.com.au (Viktor Villafuerte)
Date: Fri, 2 Sep 2016 09:18:32 +1000
Subject: Varnish malloc issue
In-Reply-To: <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
Message-ID: <20160901231832.GA3267@optusnet.com.au>

Hi Guillaume,

On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
> Is that virtual or real memory?
> Try to cap your Transient storage to 1GB, you may be overloaded with short
> lived objects.

Could you elaborate on this bit more? I've got very similar problem
here. Eg: Total memory 384g

VIRT: 487g
RES:  311g

free says: 46g free

but SWAP is 91.3% used :(


How can this be?


thanks

v


> 
> On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com> wrote:
> 
> > Hi All
> >
> > I had already mailed this query long back but this time putting in a
> > different manner.
> >
> >
> >    - We are using Varnish 4.0 in our end. We have a machine with memory
> >    of 16G with 8G being malloc for Varnish. We have a TTL for 2 hrs as well.
> >    - During our peak traffic, when we see the total memory of the machine
> >    reaching 90% and like varnishd process is taking some 89% .
> >    - So I'm just confused even though we had allocated just 8G malloc
> >    which is like 50% of the total memory, how the process is eating up 89% of
> >    memory and the memory is not releasing even after TTL.
> >
> >
> > Any help please...
> >
> > Best Regards
> > Sujith P V
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >

> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


-- 
Regards

Viktor Villafuerte
Optus Internet Engineering
t: +61 2 80825265



From guillaume at varnish-software.com  Fri Sep  2 05:51:51 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 2 Sep 2016 07:51:51 +0200
Subject: Sending a ? to the backend
In-Reply-To: <CAFeBtokBFHb809tnqtg4=i4faj9DfeCQuy=WnrMW1Q4_O77upg@mail.gmail.com>
References: <CAFeBtokBFHb809tnqtg4=i4faj9DfeCQuy=WnrMW1Q4_O77upg@mail.gmail.com>
Message-ID: <CAJ6ZYQzSnx+W3jxh0HTE3qvUALnb4ZdTL3eCGBmnWCd9wkEkxQ@mail.gmail.com>

Did should work, have a look at varnishlog (Req AND corresponding Bereq),
it will give you a better view of what is going on. If in doubt, pastebin
it, and send us the link.

And obviously, make sure you have nothing else in you vcl that could
interfer with that if.

-- 
Guillaume Quintard

On Thu, Sep 1, 2016 at 7:58 PM, Ian Macdonald <ianmac51 at gmail.com> wrote:

> Hi All,
>
> I am having trouble rewriting a url I need to send a question mark to
> the backend and it just does not seem to work I have tried escaping
> the ?, long strings, regsub() and all seem to fail.  here is a snippet
> if anyone can point me in the right direction
>
> vcl_recv {
> ...
>
> if (req.url ~ "^/myurl$") {
>
>                         set req.url = {"/search?q=term"};
>                         return (pass);
>
>
>
>
> ...
>
> }
>
> quick httpie check gets seems to show anything from the ? goes missing
>
>  <p>The requested URL <code>/search</code> was not found on this
> server.  <ins>That?s all we know.</ins>
>
> varnish is 4.1.3 on centos 6.8
>
> Cheers
>
> Ian
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160902/5c15eac7/attachment.html>

From guillaume at varnish-software.com  Fri Sep  2 05:54:47 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 2 Sep 2016 07:54:47 +0200
Subject: Varnish malloc issue
In-Reply-To: <20160901231832.GA3267@optusnet.com.au>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <20160901231832.GA3267@optusnet.com.au>
Message-ID: <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>

Hi Viktor,

Have a look here
https://www.varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage

Sujith, please try to use a packaged version from your distribution.

-- 
Guillaume Quintard

On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
viktor.villafuerte at optusnet.com.au> wrote:

> Hi Guillaume,
>
> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
> > Is that virtual or real memory?
> > Try to cap your Transient storage to 1GB, you may be overloaded with
> short
> > lived objects.
>
> Could you elaborate on this bit more? I've got very similar problem
> here. Eg: Total memory 384g
>
> VIRT: 487g
> RES:  311g
>
> free says: 46g free
>
> but SWAP is 91.3% used :(
>
>
> How can this be?
>
>
> thanks
>
> v
>
>
> >
> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com> wrote:
> >
> > > Hi All
> > >
> > > I had already mailed this query long back but this time putting in a
> > > different manner.
> > >
> > >
> > >    - We are using Varnish 4.0 in our end. We have a machine with memory
> > >    of 16G with 8G being malloc for Varnish. We have a TTL for 2 hrs as
> well.
> > >    - During our peak traffic, when we see the total memory of the
> machine
> > >    reaching 90% and like varnishd process is taking some 89% .
> > >    - So I'm just confused even though we had allocated just 8G malloc
> > >    which is like 50% of the total memory, how the process is eating up
> 89% of
> > >    memory and the memory is not releasing even after TTL.
> > >
> > >
> > > Any help please...
> > >
> > > Best Regards
> > > Sujith P V
> > >
> > > _______________________________________________
> > > varnish-misc mailing list
> > > varnish-misc at varnish-cache.org
> > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> > >
>
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
> --
> Regards
>
> Viktor Villafuerte
> Optus Internet Engineering
> t: +61 2 80825265
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160902/a1b33ccb/attachment.html>

From guillaume at varnish-software.com  Fri Sep  2 08:23:48 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 2 Sep 2016 10:23:48 +0200
Subject: Varnish Cache & Ad Impressions
In-Reply-To: <CAPQGzE0YpfjGtRTviLvqkE9cRoLt6PMBFEDrjuV_q8sdyKPUdg@mail.gmail.com>
References: <CAPQGzE1Dewfi2Aq+u3WbYWZXT+szX=g6UTAv+uyD2Hp5dpmkLA@mail.gmail.com>
 <CAJ6ZYQxdnJ9TwZvHZ2V+kCQNqWwKKsr-rRKFv5UHbcg+xgCnrQ@mail.gmail.com>
 <CAPQGzE2VyX5TmPmXzBNqgMAFRqs2AFxVhWNMXTQvftxPBo+fXw@mail.gmail.com>
 <CAJ6ZYQxpLy-wFL0=vmN=mFmuC_tQyrpfzK-4mSzOFBB0W1iZcQ@mail.gmail.com>
 <CAPQGzE1kWyZwP7EqGJRTzUDxvKAJN=ZoQGsC3Tn8zx7DzxBCtg@mail.gmail.com>
 <CAPQGzE0YpfjGtRTviLvqkE9cRoLt6PMBFEDrjuV_q8sdyKPUdg@mail.gmail.com>
Message-ID: <CAJ6ZYQzJP7B8DfBU2BQH3b=auaujOJmj9TwxFO6k-JQmDCR+Cw@mail.gmail.com>

Try removing the headers bits and the accept-encoding parts, varnish
default behavior should be enough.

And if that doesn't help, you really need to understand what is going on on
the advertising side of things.

On Aug 31, 2016 21:42, "Ayberk Kimsesiz" <ayberk.kimsesiz at gmail.com> wrote:

> Hi Guillaume,
>
> Varnish truly has a serious affect on impressions of banners. This month
> was a substantial lost. What is your advise on this?
> We are very pleased with Varnish but also experiencing a loss of income.
>
> Ad code:
>
> <script data-adfscript="adx.adform.net/adx/?mid=220975&rnd=%%ADFRND%%
> "></script>
> <script src="//s1.adform.net/banners/scripts/adx.js" async defer></script>
>
> Thanks,
> Ayberk
>
>
> 2016-08-22 14:10 GMT+03:00 Ayberk Kimsesiz <ayberk.kimsesiz at gmail.com>:
>
>> Hi,
>>
>> We are using the default.vcl file as follows. Our website is getting
>> higher traffic each day but there is a decline in ads' impression values.
>> We are using .js in ad codes. Do you think the following settings might be
>> the problem?
>>
>> /* SET THE HOST AND PORT OF WORDPRESS
>>  * *********************************************************/
>> vcl 4.0;
>> import std;
>>
>> backend default {
>>   .host = "********";
>>   .port = "8080";
>>   .connect_timeout = 600s;
>>   .first_byte_timeout = 600s;
>>   .between_bytes_timeout = 600s;
>>   .max_connections = 800;
>>     }
>>
>>   # SET THE ALLOWED IP OF PURGE REQUESTS
>>   # ##########################################################
>>    acl purge {
>>   "localhost";
>>   "127.0.0.1";
>>     }
>>
>>    #THE RECV FUNCTION
>>     # ##########################################################
>>    sub vcl_recv {
>>    if (req.http.Host == "****.com/forum" || req.url ~ "forum") {
>> return (pass);
>> }
>>
>>    # set realIP by trimming CloudFlare IP which will be used for various
>> checks
>>     set req.http.X-Actual-IP = regsub(req.http.X-Forwarded-For, "[,
>> ].*$", "");
>>
>>         # FORWARD THE IP OF THE REQUEST
>>   if (req.restarts == 0) {
>>     if (req.http.x-forwarded-for) {
>>       set req.http.X-Forwarded-For =
>>       req.http.X-Forwarded-For + ", " + client.ip;
>>     } else {
>>       set req.http.X-Forwarded-For = client.ip;
>>     }
>>   }
>>
>>  # Purge request check sections for hash_always_miss, purge and ban
>>  # BLOCK IF NOT IP is not in purge acl
>>  # ##########################################################
>>
>>   # Enable smart refreshing using hash_always_miss
>>     if (req.http.Cache-Control ~ "no-cache") {
>>     if (client.ip ~ purge || !std.ip(req.http.X-Actual-IP, "1.2.3.4") ~
>> purge) {
>>          set req.hash_always_miss = true;
>>     }
>>     }
>>
>>      if (req.method == "PURGE") {
>>     if (!client.ip ~ purge || !std.ip(req.http.X-Actual-IP, "1.2.3.4") ~
>> purge) {
>>         return(synth(405,"Not allowed."));
>>         }
>>     return (purge);
>>
>>     }
>>      if (req.method == "BAN") {
>>         # Same ACL check as above:
>>         if (!client.ip ~ purge || !std.ip(req.http.X-Actual-IP,
>> "1.2.3.4") ~ purge) {
>>                         return(synth(403, "Not allowed."));
>>         }
>>         ban("req.http.host == " + req.http.host +
>>                   " && req.url == " + req.url);
>>
>>         # Throw a synthetic page so the
>>         # request won't go to the backend.
>>         return(synth(200, "Ban added"));
>> }
>>
>>
>> # Unset cloudflare cookies
>> # Remove has_js and CloudFlare/Google Analytics __* cookies.
>>       set req.http.Cookie = regsuball(req.http.Cookie,
>> "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
>>       # Remove a ";" prefix, if present.
>>      set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
>>
>>   # For Testing: If you want to test with Varnish passing (not caching)
>> uncomment
>>   # return( pass );
>>
>>   # FORWARD THE IP OF THE REQUEST
>>   if (req.restarts == 0) {
>>     if (req.http.x-forwarded-for) {
>>       set req.http.X-Forwarded-For =
>>       req.http.X-Forwarded-For + ", " + client.ip;
>>     } else {
>>       set req.http.X-Forwarded-For = client.ip;
>>     }
>>   }
>>
>>    # DO NOT CACHE RSS FEED
>>    if (req.url ~ "/feed(/)?") {
>>     return ( pass );
>>    }
>>
>>     ## Do not cache search results, comment these 3 lines if you do want
>> to cache them
>>
>>     if (req.url ~ "/\?s\=") {
>>     return ( pass );
>>   }
>>
>> # CLEAN UP THE ENCODING HEADER.
>>   # SET TO GZIP, DEFLATE, OR REMOVE ENTIRELY.  WITH VARY ACCEPT-ENCODING
>>   # VARNISH WILL CREATE SEPARATE CACHES FOR EACH
>>   # DO NOT ACCEPT-ENCODING IMAGES, ZIPPED FILES, AUDIO, ETC.
>>   # ##########################################################
>>   if (req.http.Accept-Encoding) {
>>     if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
>>       # No point in compressing these
>>       unset req.http.Accept-Encoding;
>>     } elsif (req.http.Accept-Encoding ~ "gzip") {
>>       set req.http.Accept-Encoding = "gzip";
>>     } elsif (req.http.Accept-Encoding ~ "deflate") {
>>       set req.http.Accept-Encoding = "deflate";
>>     } else {
>>       # unknown algorithm
>>       unset req.http.Accept-Encoding;
>>     }
>>   }
>>
>>   # PIPE ALL NON-STANDARD REQUESTS
>>   # ##########################################################
>>   if (req.method != "GET" &&
>>     req.method != "HEAD" &&
>>     req.method != "PUT" &&
>>     req.method != "POST" &&
>>     req.method != "TRACE" &&
>>     req.method != "OPTIONS" &&
>>     req.method != "DELETE") {
>>       return (pipe);
>>   }
>>
>>   # ONLY CACHE GET AND HEAD REQUESTS
>>   # ##########################################################
>>   if (req.method != "GET" && req.method != "HEAD") {
>>     return (pass);
>>   }
>>
>>   # OPTIONAL: DO NOT CACHE LOGGED IN USERS (THIS OCCURS IN FETCH TOO,
>> EITHER
>>   # COMMENT OR UNCOMMENT BOTH
>>   # ##########################################################
>>   if ( req.http.cookie ~ "wordpress_logged_in" ) {
>>     return( pass );
>>   }
>>
>>   # IF THE REQUEST IS NOT FOR A PREVIEW, WP-ADMIN OR WP-LOGIN
>>   # THEN UNSET THE COOKIES
>>   # ##########################################################
>>   if (!(req.url ~ "wp-(login|admin)")
>>     && !(req.url ~ "&preview=true" )
>>   ){
>>     unset req.http.cookie;
>>   }
>>
>>   # IF BASIC AUTH IS ON THEN DO NOT CACHE
>>   # ##########################################################
>>   if (req.http.Authorization || req.http.Cookie) {
>>     return (pass);
>>   }
>>
>>   # IF YOU GET HERE THEN THIS REQUEST SHOULD BE CACHED
>>   # ##########################################################
>>   return (hash);
>>   # This is for phpmyadmin
>>   if (req.http.Host == "ki1.org") {
>>   return (pass);
>>   }
>>
>>   if (req.http.Host == "mysql.ki1.org") {
>>   return (pass);
>>   }
>>
>>   }
>>
>> # HIT FUNCTION
>> # ##########################################################
>> sub vcl_hit {
>>   # IF THIS IS A PURGE REQUEST THEN DO THE PURGE
>>   # ##########################################################
>>   if (req.method == "PURGE") {
>>     #
>>     # This is now handled in vcl_recv.
>>     #
>>     # purge;
>>     return (synth(200, "Purged."));
>>   }
>>   return (deliver);
>> }
>>
>> # MISS FUNCTION
>> # ##########################################################
>> sub vcl_miss {
>>   if (req.method == "PURGE") {
>>     #
>>     # This is now handled in vcl_recv.
>>     #
>>     # purge;
>>     return (synth(200, "Purged."));
>>   }
>>   return (fetch);
>> }
>>
>> # FETCH FUNCTION
>> # ##########################################################
>> sub vcl_backend_response {
>>   # I SET THE VARY TO ACCEPT-ENCODING, THIS OVERRIDES W3TC
>>   # TENDANCY TO SET VARY USER-AGENT.  YOU MAY OR MAY NOT WANT
>>   # TO DO THIS
>>   # ##########################################################
>>   set beresp.http.Vary = "Accept-Encoding";
>>
>>   # IF NOT WP-ADMIN THEN UNSET COOKIES AND SET THE AMOUNT OF
>>   # TIME THIS PAGE WILL STAY CACHED (TTL)
>>   # ##########################################################
>>    if (!(bereq.url ~ "wp-(login|admin)|forum") && !bereq.http.cookie ~
>> "wordpress_logged_in" && !bereq.http.host == "******.com/forum" ) {
>>     unset beresp.http.set-cookie;
>>     set beresp.ttl = 52w;
>> #    set beresp.grace =1w;
>>   }
>>
>>
>>
>>   if (beresp.ttl <= 0s ||
>>     beresp.http.Set-Cookie ||
>>     beresp.http.Vary == "*") {
>>       set beresp.ttl = 120 s;
>>       # set beresp.ttl = 120s;
>>       set beresp.uncacheable = true;
>>       return (deliver);
>>   }
>>
>>   return (deliver);
>> }
>>
>> # DELIVER FUNCTION
>> # ##########################################################
>> sub vcl_deliver {
>>   # IF THIS PAGE IS ALREADY CACHED THEN RETURN A 'HIT' TEXT
>>   # IN THE HEADER (GREAT FOR DEBUGGING)
>>   # ##########################################################
>>   if (obj.hits > 0) {
>>     set resp.http.X-Cache = "HIT";
>>   # IF THIS IS A MISS RETURN THAT IN THE HEADER
>>   # ##########################################################
>>   } else {
>>     set resp.http.X-Cache = "MISS";
>>   }
>> }
>>
>>
>> 2016-08-16 19:42 GMT+03:00 Guillaume Quintard <
>> guillaume at varnish-software.com>:
>>
>>> Maybe, what is it supposed to do, why is it there and is any of your ad
>>> frameworks affected by this?
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160902/dff4cfc3/attachment.html>

From apj at mutt.dk  Fri Sep  2 08:45:51 2016
From: apj at mutt.dk (Andreas Plesner)
Date: Fri, 2 Sep 2016 10:45:51 +0200
Subject: Varnish Cache & Ad Impressions
In-Reply-To: <CAPQGzE0YpfjGtRTviLvqkE9cRoLt6PMBFEDrjuV_q8sdyKPUdg@mail.gmail.com>
References: <CAPQGzE1Dewfi2Aq+u3WbYWZXT+szX=g6UTAv+uyD2Hp5dpmkLA@mail.gmail.com>
 <CAJ6ZYQxdnJ9TwZvHZ2V+kCQNqWwKKsr-rRKFv5UHbcg+xgCnrQ@mail.gmail.com>
 <CAPQGzE2VyX5TmPmXzBNqgMAFRqs2AFxVhWNMXTQvftxPBo+fXw@mail.gmail.com>
 <CAJ6ZYQxpLy-wFL0=vmN=mFmuC_tQyrpfzK-4mSzOFBB0W1iZcQ@mail.gmail.com>
 <CAPQGzE1kWyZwP7EqGJRTzUDxvKAJN=ZoQGsC3Tn8zx7DzxBCtg@mail.gmail.com>
 <CAPQGzE0YpfjGtRTviLvqkE9cRoLt6PMBFEDrjuV_q8sdyKPUdg@mail.gmail.com>
Message-ID: <20160902084551.GF23724@nerd.dk>

On Wed, Aug 31, 2016 at 10:42:19PM +0300, Ayberk Kimsesiz wrote:
> 
> <script data-adfscript="adx.adform.net/adx/?mid=220975&rnd=%%ADFRND%%
> "></script>
> <script src="//s1.adform.net/banners/scripts/adx.js" async defer></script>

Is %%ADFRND%% a variable that must be different for each request? If so, you
can't cache it.

-- 
Andreas



From guillaume at varnish-software.com  Fri Sep  2 08:53:16 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 2 Sep 2016 10:53:16 +0200
Subject: Caching for some cookies?
In-Reply-To: <CACqaJ6jSqZAHe9p53r=L-j=emP5PE6ZzmQGHEdymJmmjtJkMcw@mail.gmail.com>
References: <CACqaJ6jSqZAHe9p53r=L-j=emP5PE6ZzmQGHEdymJmmjtJkMcw@mail.gmail.com>
Message-ID: <CAJ6ZYQxzAnkvgHJeYHq=ad7gqDcQxq46Co1+BkTu5dYxkLvRdg@mail.gmail.com>

Hi,

Look at the varnishlog, there should be a trace of that unset, once
confirmed, find it in your vcl.

Know that if you don't return from a subroutine, the built-in vcl is
executed ("locate builtin.vcl" to find a commented version of it).

On Sep 1, 2016 12:59, "Albert Tollku?i" <albert.tollkuci at gmail.com> wrote:

> Hello,
> I've started using Varnish a few months ago and my vcl files has started
> to get complicated. I'm serving 5 different sites with the same varnish
> instance (using different backends) and this part is working.
>
> Now I have some cases that for some sites I need to keep some cookies and
> cache different versions based on them. Basic scenario is to show the user
> a message only the first time he sees the page. Without varnish if the
> cookie is not there I set it and show the message. Next time the cookie is
> there and no message is shown.
>
> With varnish my logic is to include the cookie in hash and cache two
> different versions. The challenge is to make sure cookies are set correctly
> and here I'm kind of stack. I've added some debug messages in my vcl and
> for some reason I'm not seeing the cookies I set in my server in
> vcl_backend_response. Inside the routine, I have the following code in the
> beginning:
>
>        # AT: DEBUG
>        if(bereq.http.x-at-debug == "1") {
>          std.log("AT DEBUG - bereq URL: " + bereq.url + ". Cookies: '" +
> bereq.http.Cookie + "'");
>          std.log("AT DEBUG - beresp set-cookie: '" +
>  beresp.http.set-cookie + "'");
>        }
>
> I set x-at-debug header in vcl_recv based on my IP, so I can track only my
> requests. In the server side I see the cookie "__IKUB_AL_Myikub_PopUp"
> being set, but in vcl_backend_response is not.
>
> Is there anything in between that can strip the cookie?
>
> Thank you,
> Albert
>
>
>
> --
> Web: http://www.tollkuci.com
> Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
> <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
> <https://www.facebook.com/albert.tollkuci> Twitter
> <https://twitter.com/AlbertTollkuci> Career 2.0
> <http://careers.stackoverflow.com/atollkuci>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160902/04bcb78a/attachment.html>

From ayberk.kimsesiz at gmail.com  Fri Sep  2 10:13:33 2016
From: ayberk.kimsesiz at gmail.com (Ayberk Kimsesiz)
Date: Fri, 2 Sep 2016 13:13:33 +0300
Subject: Varnish Cache & Ad Impressions
In-Reply-To: <20160902084551.GF23724@nerd.dk>
References: <CAPQGzE1Dewfi2Aq+u3WbYWZXT+szX=g6UTAv+uyD2Hp5dpmkLA@mail.gmail.com>
 <CAJ6ZYQxdnJ9TwZvHZ2V+kCQNqWwKKsr-rRKFv5UHbcg+xgCnrQ@mail.gmail.com>
 <CAPQGzE2VyX5TmPmXzBNqgMAFRqs2AFxVhWNMXTQvftxPBo+fXw@mail.gmail.com>
 <CAJ6ZYQxpLy-wFL0=vmN=mFmuC_tQyrpfzK-4mSzOFBB0W1iZcQ@mail.gmail.com>
 <CAPQGzE1kWyZwP7EqGJRTzUDxvKAJN=ZoQGsC3Tn8zx7DzxBCtg@mail.gmail.com>
 <CAPQGzE0YpfjGtRTviLvqkE9cRoLt6PMBFEDrjuV_q8sdyKPUdg@mail.gmail.com>
 <20160902084551.GF23724@nerd.dk>
Message-ID: <CAPQGzE3-B4LquvKTsNo4kSiV0akijHqpvmXNU5yFhj0+cw6+Og@mail.gmail.com>

Hi Andreas,

Yes, %%ADFRND%% sends different requests because there are more than one
advertiser. You meant this, is that right?
How can i block it so it won't cache?

Thanks,
Ayberk

2016-09-02 11:45 GMT+03:00 Andreas Plesner <apj at mutt.dk>:

> On Wed, Aug 31, 2016 at 10:42:19PM +0300, Ayberk Kimsesiz wrote:
> >
> > <script data-adfscript="adx.adform.net/adx/?mid=220975&rnd=%%ADFRND%%
> > "></script>
> > <script src="//s1.adform.net/banners/scripts/adx.js" async
> defer></script>
>
> Is %%ADFRND%% a variable that must be different for each request? If so,
> you
> can't cache it.
>
> --
> Andreas
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160902/f59a1443/attachment.html>

From ayberk.kimsesiz at gmail.com  Fri Sep  2 11:08:47 2016
From: ayberk.kimsesiz at gmail.com (Ayberk Kimsesiz)
Date: Fri, 2 Sep 2016 14:08:47 +0300
Subject: Varnish Cache & Ad Impressions
In-Reply-To: <CAPQGzE3-B4LquvKTsNo4kSiV0akijHqpvmXNU5yFhj0+cw6+Og@mail.gmail.com>
References: <CAPQGzE1Dewfi2Aq+u3WbYWZXT+szX=g6UTAv+uyD2Hp5dpmkLA@mail.gmail.com>
 <CAJ6ZYQxdnJ9TwZvHZ2V+kCQNqWwKKsr-rRKFv5UHbcg+xgCnrQ@mail.gmail.com>
 <CAPQGzE2VyX5TmPmXzBNqgMAFRqs2AFxVhWNMXTQvftxPBo+fXw@mail.gmail.com>
 <CAJ6ZYQxpLy-wFL0=vmN=mFmuC_tQyrpfzK-4mSzOFBB0W1iZcQ@mail.gmail.com>
 <CAPQGzE1kWyZwP7EqGJRTzUDxvKAJN=ZoQGsC3Tn8zx7DzxBCtg@mail.gmail.com>
 <CAPQGzE0YpfjGtRTviLvqkE9cRoLt6PMBFEDrjuV_q8sdyKPUdg@mail.gmail.com>
 <20160902084551.GF23724@nerd.dk>
 <CAPQGzE3-B4LquvKTsNo4kSiV0akijHqpvmXNU5yFhj0+cw6+Og@mail.gmail.com>
Message-ID: <CAPQGzE1GDKjbdV8j-iMCtphvRC6X=0qQogsS9Y6=7xijZ4Gg-w@mail.gmail.com>

Hi Andreas,

Yes, %%ADFRND%% sends different requests because there are more than one
advertiser. You meant this, is that right?
How can i block it so it won't cache?

Thanks,
Ayberk

2016-09-02 13:13 GMT+03:00 Ayberk Kimsesiz <ayberk.kimsesiz at gmail.com>:

> Hi Andreas,
>
> Yes, %%ADFRND%% sends different requests because there are more than one
> advertiser. You meant this, is that right?
> How can i block it so it won't cache?
>
> Thanks,
> Ayberk
>
> 2016-09-02 11:45 GMT+03:00 Andreas Plesner <apj at mutt.dk>:
>
>> On Wed, Aug 31, 2016 at 10:42:19PM +0300, Ayberk Kimsesiz wrote:
>> >
>> > <script data-adfscript="adx.adform.net/adx/?mid=220975&rnd=%%ADFRND%%
>> > "></script>
>> > <script src="//s1.adform.net/banners/scripts/adx.js" async
>> defer></script>
>>
>> Is %%ADFRND%% a variable that must be different for each request? If so,
>> you
>> can't cache it.
>>
>> --
>> Andreas
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160902/a19a2196/attachment.html>

From albert.tollkuci at gmail.com  Fri Sep  2 16:28:54 2016
From: albert.tollkuci at gmail.com (=?UTF-8?Q?Albert_Tollku=C3=A7i?=)
Date: Fri, 2 Sep 2016 18:28:54 +0200
Subject: Caching for some cookies?
In-Reply-To: <CAJ6ZYQxzAnkvgHJeYHq=ad7gqDcQxq46Co1+BkTu5dYxkLvRdg@mail.gmail.com>
References: <CACqaJ6jSqZAHe9p53r=L-j=emP5PE6ZzmQGHEdymJmmjtJkMcw@mail.gmail.com>
 <CAJ6ZYQxzAnkvgHJeYHq=ad7gqDcQxq46Co1+BkTu5dYxkLvRdg@mail.gmail.com>
Message-ID: <CACqaJ6gF2yQmfm9qzMgwQVA3wbk8iJXEh8gp9Fw8Fjf=OCX87Q@mail.gmail.com>

Thank you Guillaume,
probably I'm missing something, but can't find any unset.

Is there anything running before vcl_backend_response? I see the cookie in
the backend side and is not there anymore in vcl_backend_response?

Albert

On Fri, Sep 2, 2016 at 10:53 AM, Guillaume Quintard <
guillaume at varnish-software.com> wrote:

> Hi,
>
> Look at the varnishlog, there should be a trace of that unset, once
> confirmed, find it in your vcl.
>
> Know that if you don't return from a subroutine, the built-in vcl is
> executed ("locate builtin.vcl" to find a commented version of it).
>
> On Sep 1, 2016 12:59, "Albert Tollku?i" <albert.tollkuci at gmail.com> wrote:
>
>> Hello,
>> I've started using Varnish a few months ago and my vcl files has started
>> to get complicated. I'm serving 5 different sites with the same varnish
>> instance (using different backends) and this part is working.
>>
>> Now I have some cases that for some sites I need to keep some cookies and
>> cache different versions based on them. Basic scenario is to show the user
>> a message only the first time he sees the page. Without varnish if the
>> cookie is not there I set it and show the message. Next time the cookie is
>> there and no message is shown.
>>
>> With varnish my logic is to include the cookie in hash and cache two
>> different versions. The challenge is to make sure cookies are set correctly
>> and here I'm kind of stack. I've added some debug messages in my vcl and
>> for some reason I'm not seeing the cookies I set in my server in
>> vcl_backend_response. Inside the routine, I have the following code in the
>> beginning:
>>
>>        # AT: DEBUG
>>        if(bereq.http.x-at-debug == "1") {
>>          std.log("AT DEBUG - bereq URL: " + bereq.url + ". Cookies: '" +
>> bereq.http.Cookie + "'");
>>          std.log("AT DEBUG - beresp set-cookie: '" +
>>  beresp.http.set-cookie + "'");
>>        }
>>
>> I set x-at-debug header in vcl_recv based on my IP, so I can track only
>> my requests. In the server side I see the cookie "__IKUB_AL_Myikub_PopUp"
>> being set, but in vcl_backend_response is not.
>>
>> Is there anything in between that can strip the cookie?
>>
>> Thank you,
>> Albert
>>
>>
>>
>> --
>> Web: http://www.tollkuci.com
>> Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci>
>> Google+ <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
>> <https://www.facebook.com/albert.tollkuci> Twitter
>> <https://twitter.com/AlbertTollkuci> Career 2.0
>> <http://careers.stackoverflow.com/atollkuci>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>


-- 
Web: http://www.tollkuci.com
Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
<https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
<https://www.facebook.com/albert.tollkuci> Twitter
<https://twitter.com/AlbertTollkuci> Career 2.0
<http://careers.stackoverflow.com/atollkuci>
------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160902/45f4eebd/attachment.html>

From albert.tollkuci at gmail.com  Fri Sep  2 17:05:50 2016
From: albert.tollkuci at gmail.com (=?UTF-8?Q?Albert_Tollku=C3=A7i?=)
Date: Fri, 2 Sep 2016 19:05:50 +0200
Subject: Caching for some cookies?
Message-ID: <CACqaJ6hqbnCUxeijhpU-zuH0iw8kkEr=TM21g9uMryv6=LM4gg@mail.gmail.com>

Sorry for the long post, but I'm stuck here. Below is the full log from
varnishlog for a MISS request (so I can check what's happening with the
backend). In bold there are the important bits:

- In the original request I have "*-   Link           req 294922 restart*".
Is the restart normal, because of the miss?

- In the backend response I can see the set cookie header: *-
BerespHeader   Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu,
24-Aug-2017 16:48:53 GMT; path=/*

- My log doesn't show the set-cookie header (*-   VCL_Log        AT DEBUG -
beresp set-cookie: 'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/;
HttpOnly'*). Is that because there are two set-cookie headers and it is
showing only the first?

- Immediately after the log there's a line "*-   BerespUnset    Set-Cookie:
__IKUB_AL_Myikub_PopUp=1; expires=Thu, 24-Aug-2017 16:48:53 GMT; path=/"*.
It looks like this is responsible for un-setting the cookie, is that right?

- If so the cookie is striped between my first and second log and the code
in between is the following:

*    if(beresp.http.set-cookie)*
*    {*
*      std.log("AT Vanish - mobile.ikub.al <http://mobile.ikub.al> original
backend set cookie: '" + beresp.http.set-cookie + "'");*
*    }*

*    # AT: We have to keep all cookies starting with __IKUB*
*    set beresp.http.set-cookie = ";" + beresp.http.set-cookie;*
*    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "; +",
";");*
*    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
";(__IKUB.*)=", "; \1=");*
*    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";[^
][^;]*", "");*
*    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "^[;
]+|[; ]+$", "");*

*    # Clean empty cookies*
*    if (beresp.http.set-cookie ~ "^\s*$") {*
*      set beresp.http.set-cookie = "";*
*    }*

*    if (beresp.http.set-cookie == ""){*
*      unset beresp.http.set-cookie;*
*    }*
*    else*
*    {*
*      std.log("AT Varnish - mobile.ikub.al <http://mobile.ikub.al>
processed backend cookie: '" + beresp.http.set-cookie + "'");*
*    }*

I have the same regex in the request processing and they're working, but
maybe for some reason not working here...


Full log from varnishlog:

*   << Request  >> 294921
-   Begin          req 294920 rxreq
-   Timestamp      Start: 1472834933.190575 0.000000 0.000000
-   Timestamp      Req: 1472834933.190575 0.000000 0.000000
-   ReqStart       185.158.1.35 5547
-   ReqMethod      GET
-   ReqURL         /Default2.aspx
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Host: mobile.ikub.al
-   ReqHeader      Connection: keep-alive
-   ReqHeader      Pragma: no-cache
-   ReqHeader      Cache-Control: no-cache
-   ReqHeader      Upgrade-Insecure-Requests: 1
-   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
-   ReqHeader      Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
-   ReqHeader      Accept-Encoding: gzip, deflate, sdch
-   ReqHeader      Accept-Language: en-US,en;q=0.8,sq;q=0.6
-   ReqHeader      Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
_gat=1; __utmt=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
__utma=1.12079592.1472832572.1
-   ReqHeader      X-Forwarded-For: 185.158.1.35
-   VCL_call       RECV
-   VCL_Log        AT DEBUG - recv URL: /Default2.aspx. Cookies:
'__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
_gat=1; __utmt=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.147
-   ReqHeader      x-at-debug: 1
-   ReqUnset       Host: mobile.ikub.al
-   ReqHeader      Host: mobile.ikub.al
-   ReqURL         /Default2.aspx
-   ReqUnset       Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
_gat=1; __utmt=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
__utma=1.12079592.1472832572.1
-   ReqHeader      Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
_gat=1; __utmt=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
__utma=1.12079592.1472832572.1
-   ReqUnset       Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
_gat=1; __utmt=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
__utma=1.12079592.1472832572.1
-   ReqHeader      Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
_gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
-   ReqUnset       Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
_gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
-   ReqHeader      Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
_gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
_gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie:
__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   VCL_return     purge
-   ReqUnset       Accept-Encoding: gzip, deflate, sdch
-   ReqHeader      Accept-Encoding: gzip
-   VCL_call       HASH
-   VCL_return     lookup
-   VCL_call       PURGE
-   ReqHeader      X-Purge: Yes
-   VCL_return     restart
-   Timestamp      Restart: 1472834933.190646 0.000071 0.000071
*-   Link           req 294922 restart*
-   End




*   << BeReq    >> 294923
-   Begin          bereq 294922 fetch
-   Timestamp      Start: 1472834933.190706 0.000000 0.000000
-   BereqMethod    GET
-   BereqURL       /Default2.aspx
-   BereqProtocol  HTTP/1.1
-   BereqHeader    Pragma: no-cache
-   BereqHeader    Upgrade-Insecure-Requests: 1
-   BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
-   BereqHeader    Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
-   BereqHeader    Accept-Language: en-US,en;q=0.8,sq;q=0.6
-   BereqHeader    X-Forwarded-For: 185.158.1.35
-   BereqHeader    Accept-Encoding: gzip
-   BereqHeader    X-Purge: Yes
-   BereqHeader    x-at-debug: 1
-   BereqHeader    Host: mobile.ikub.al
-   BereqHeader    x-state: MISS
-   BereqHeader    X-Varnish: 294923
-   VCL_call       BACKEND_FETCH
-   VCL_Log        AT DEBUG - fetch bereq URL: /Default2.aspx. Cookies: ''
-   VCL_return     fetch
-   Backend        43 ikubmobileserver ikubmobileserver(13.94.212.211,,80)
-   Timestamp      Bereq: 1472834933.190737 0.000031 0.000031
-   Timestamp      Beresp: 1472834933.830661 0.639955 0.639924
-   BerespProtocol HTTP/1.1
-   BerespStatus   200
-   BerespReason   OK
-   BerespHeader   Cache-Control: private
-   BerespHeader   Content-Type: text/html; charset=utf-8
-   BerespHeader   Content-Encoding: gzip
-   BerespHeader   Vary: Accept-Encoding
-   BerespHeader   Server: Microsoft-IIS/8.5
-   BerespHeader   X-AspNet-Version: 2.0.50727
-   BerespHeader   Set-Cookie: ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;
path=/; HttpOnly
*-   BerespHeader   Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu,
24-Aug-2017 16:48:53 GMT; path=/*
-   BerespHeader   X-Powered-By: ASP.NET
-   BerespHeader   Date: Fri, 02 Sep 2016 16:48:53 GMT
-   BerespHeader   Content-Length: 52233
-   TTL            RFC 120 -1 -1 1472834934 1472834934 1472834933 0 0
-   VCL_call       BACKEND_RESPONSE
-   VCL_Log        AT DEBUG - bereq URL: /Default2.aspx. Cookies: ''
*-   VCL_Log        AT DEBUG - beresp set-cookie:
'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'*
-   VCL_Log        AT Vanish - mobile.ikub.al original backend set cookie:
'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'
-   BerespUnset    Set-Cookie: ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;
path=/; HttpOnly
*-   BerespUnset    Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu,
24-Aug-2017 16:48:53 GMT; path=/*
-   BerespHeader   set-cookie: ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;
path=/; HttpOnly
-   BerespUnset    set-cookie: ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;
path=/; HttpOnly
-   BerespHeader   set-cookie:
;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
-   BerespUnset    set-cookie:
;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
-   BerespHeader   set-cookie:
;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
-   BerespUnset    set-cookie:
;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
-   BerespHeader   set-cookie:
-   BerespUnset    set-cookie:
-   BerespHeader   set-cookie:
-   BerespUnset    set-cookie:
-   BerespHeader   set-cookie:
-   BerespUnset    set-cookie:
-   TTL            VCL 900 10 0 1472834934
-   VCL_Log        AT DEBUG - bereq2 URL: /Default2.aspx. Cookies: ''
-   VCL_Log        AT DEBUG - beresp2 set-cookie: ''
-   TTL            VCL 900 21600 0 1472834934
-   VCL_Log        AT DEBUG - bereq3 URL: /Default2.aspx. Cookies: ''
-   VCL_Log        AT DEBUG - beresp3 set-cookie: ''
-   VCL_return     deliver
-   Storage        malloc s0
-   ObjProtocol    HTTP/1.1
-   ObjStatus      200
-   ObjReason      OK
-   ObjHeader      Cache-Control: private
-   ObjHeader      Content-Type: text/html; charset=utf-8
-   ObjHeader      Content-Encoding: gzip
-   ObjHeader      Vary: Accept-Encoding
-   ObjHeader      Server: Microsoft-IIS/8.5
-   ObjHeader      X-AspNet-Version: 2.0.50727
-   ObjHeader      X-Powered-By: ASP.NET
-   ObjHeader      Date: Fri, 02 Sep 2016 16:48:53 GMT
-   ObjHeader      Content-Length: 52233
-   Fetch_Body     3 length stream
-   Gzip           u F - 52233 194069 80 80 417793
-   BackendReuse   43 ikubmobileserver(13.94.212.211,,80)
-   Timestamp      BerespBody: 1472834933.831743 0.641037 0.001082
-   Length         52233
-   BereqAcct      473 0 473 428 52233 52661
-   End

*   << Request  >> 294922
-   Begin          req 294921 restart
-   Timestamp      Start: 1472834933.190646 0.000071 0.000000
-   ReqStart       185.158.1.35 5547
-   ReqMethod      GET
-   ReqURL         /Default2.aspx
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Connection: keep-alive
-   ReqHeader      Pragma: no-cache
-   ReqHeader      Cache-Control: no-cache
-   ReqHeader      Upgrade-Insecure-Requests: 1
-   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
-   ReqHeader      Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
-   ReqHeader      Accept-Language: en-US,en;q=0.8,sq;q=0.6
-   ReqHeader      X-Forwarded-For: 185.158.1.35
-   ReqHeader      x-at-debug: 1
-   ReqHeader      Host: mobile.ikub.al
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Accept-Encoding: gzip
-   ReqHeader      X-Purge: Yes
-   VCL_call       RECV
-   VCL_Log        AT DEBUG - recv URL: /Default2.aspx. Cookies:
'OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d; '
-   ReqUnset       x-at-debug: 1
-   ReqHeader      x-at-debug: 1
-   ReqUnset       Host: mobile.ikub.al
-   ReqHeader      Host: mobile.ikub.al
-   ReqURL         /Default2.aspx
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   VCL_Log        AT DEBUG - recv2 URL: /Default2.aspx. Cookies:
'OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
__auc=c970f060156eba988423264f56d; '
-   ReqUnset       Cookie: OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: ;OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie: ;OX_plg=swf|shk|pm;
__asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie:
;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie:
;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie:
;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
-   ReqUnset       Cookie:
;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
-   ReqHeader      Cookie: ;
-   ReqUnset       Cookie: ;
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   VCL_Log        AT DEBUG - recv3 URL: /Default2.aspx. Cookies: ''
-   VCL_return     hash
-   VCL_call       HASH
-   VCL_return     lookup
-   Debug          "XXXX MISS"
-   VCL_call       MISS
-   ReqHeader      x-state: MISS
-   VCL_return     fetch
-   Link           bereq 294923 fetch
-   Timestamp      Fetch: 1472834933.831768 0.641192 0.641122
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     OK
-   RespHeader     Cache-Control: private
-   RespHeader     Content-Type: text/html; charset=utf-8
-   RespHeader     Content-Encoding: gzip
-   RespHeader     Vary: Accept-Encoding
-   RespHeader     Server: Microsoft-IIS/8.5
-   RespHeader     X-AspNet-Version: 2.0.50727
-   RespHeader     X-Powered-By: ASP.NET
-   RespHeader     Date: Fri, 02 Sep 2016 16:48:53 GMT
-   RespHeader     Content-Length: 52233
-   RespHeader     X-Varnish: 294922
-   RespHeader     Age: 0
-   RespHeader     Via: 1.1 varnish-v4
-   VCL_call       DELIVER
-   RespHeader     X-Cache: MISS
-   RespHeader     X-Cache-Hits: 0
-   RespUnset      X-Powered-By: ASP.NET
-   RespUnset      Server: Microsoft-IIS/8.5
-   RespUnset      X-Varnish: 294922
-   RespUnset      Via: 1.1 varnish-v4
-   RespUnset      X-AspNet-Version: 2.0.50727
-   VCL_Log        AT DEBUG - deliver URL: /Default2.aspx. Cookies: ''
-   VCL_Log        AT DEBUG - deliver set-cookie: ''
-   VCL_return     deliver
-   Timestamp      Process: 1472834933.831794 0.641218 0.000026
-   Debug          "RES_MODE 2"
-   RespHeader     Connection: keep-alive
-   RespHeader     Accept-Ranges: bytes
-   Timestamp      Resp: 1472834933.831830 0.641255 0.000036
-   Debug          "XXX REF 2"
-   ReqAcct        828 0 828 276 52233 52509
-   End


On Fri, Sep 2, 2016 at 6:28 PM, Albert Tollku?i <albert.tollkuci at gmail.com>
wrote:

> Thank you Guillaume,
> probably I'm missing something, but can't find any unset.
>
> Is there anything running before vcl_backend_response? I see the cookie in
> the backend side and is not there anymore in vcl_backend_response?
>
> Albert
>
> On Fri, Sep 2, 2016 at 10:53 AM, Guillaume Quintard <
> guillaume at varnish-software.com> wrote:
>
>> Hi,
>>
>> Look at the varnishlog, there should be a trace of that unset, once
>> confirmed, find it in your vcl.
>>
>> Know that if you don't return from a subroutine, the built-in vcl is
>> executed ("locate builtin.vcl" to find a commented version of it).
>>
>> On Sep 1, 2016 12:59, "Albert Tollku?i" <albert.tollkuci at gmail.com>
>> wrote:
>>
>>> Hello,
>>> I've started using Varnish a few months ago and my vcl files has started
>>> to get complicated. I'm serving 5 different sites with the same varnish
>>> instance (using different backends) and this part is working.
>>>
>>> Now I have some cases that for some sites I need to keep some cookies
>>> and cache different versions based on them. Basic scenario is to show the
>>> user a message only the first time he sees the page. Without varnish if the
>>> cookie is not there I set it and show the message. Next time the cookie is
>>> there and no message is shown.
>>>
>>> With varnish my logic is to include the cookie in hash and cache two
>>> different versions. The challenge is to make sure cookies are set correctly
>>> and here I'm kind of stack. I've added some debug messages in my vcl and
>>> for some reason I'm not seeing the cookies I set in my server in
>>> vcl_backend_response. Inside the routine, I have the following code in the
>>> beginning:
>>>
>>>        # AT: DEBUG
>>>        if(bereq.http.x-at-debug == "1") {
>>>          std.log("AT DEBUG - bereq URL: " + bereq.url + ". Cookies: '" +
>>> bereq.http.Cookie + "'");
>>>          std.log("AT DEBUG - beresp set-cookie: '" +
>>>  beresp.http.set-cookie + "'");
>>>        }
>>>
>>> I set x-at-debug header in vcl_recv based on my IP, so I can track only
>>> my requests. In the server side I see the cookie "__IKUB_AL_Myikub_PopUp"
>>> being set, but in vcl_backend_response is not.
>>>
>>> Is there anything in between that can strip the cookie?
>>>
>>> Thank you,
>>> Albert
>>>
>>>
>>>
>>> --
>>> Web: http://www.tollkuci.com
>>> Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci>
>>> Google+ <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
>>> <https://www.facebook.com/albert.tollkuci> Twitter
>>> <https://twitter.com/AlbertTollkuci> Career 2.0
>>> <http://careers.stackoverflow.com/atollkuci>
>>>
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>
>
> --
> Web: http://www.tollkuci.com
> Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
> <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
> <https://www.facebook.com/albert.tollkuci> Twitter
> <https://twitter.com/AlbertTollkuci> Career 2.0
> <http://careers.stackoverflow.com/atollkuci>
> ------------------------------
>
>


-- 
Web: http://www.tollkuci.com
Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
<https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
<https://www.facebook.com/albert.tollkuci> Twitter
<https://twitter.com/AlbertTollkuci> Career 2.0
<http://careers.stackoverflow.com/atollkuci>
------------------------------

Imagination is more important than knowledge
*Albert Einstein*

The three chief virtues of a programmer are: Laziness, Impatience and Hubris
*Larry Wall*

Men are basically smart or dumb and lazy or ambitious. The dumb and
ambitious ones are dangerous and I get rid of them. The dumb and lazy ones
I give mundane duties. The smart ambitious ones I put on my staff. The
smart and lazy ones I make my commanders
*Erwin Rommel*

The best programmers are not marginally better than merely good ones.  They
are an order-of-magnitude better, measured by whatever standard: conceptual
creativity, speed, ingenuity of design, or problem-solving ability.
*Randall E. Stross*

Measuring programming progress by lines of code is like measuring aircraft
building progress by weight.
*Bill Gates*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160902/57c9ae2b/attachment.html>

From albert.tollkuci at gmail.com  Sat Sep  3 09:16:35 2016
From: albert.tollkuci at gmail.com (=?UTF-8?Q?Albert_Tollku=C3=A7i?=)
Date: Sat, 3 Sep 2016 11:16:35 +0200
Subject: Caching for some cookies?
In-Reply-To: <CACqaJ6hqbnCUxeijhpU-zuH0iw8kkEr=TM21g9uMryv6=LM4gg@mail.gmail.com>
References: <CACqaJ6hqbnCUxeijhpU-zuH0iw8kkEr=TM21g9uMryv6=LM4gg@mail.gmail.com>
Message-ID: <CACqaJ6gHEQNYbdOf=Bt7SLYDdTmxKH0nRa6dFO++VwH7rgQisg@mail.gmail.com>

After digging more it seems I have to install varnish modules from
https://github.com/varnish/varnish-modules. The instructions to install
them are:

sudo apt-get install libvarnishapi-dev || sudo yum install varnish-libs-devel
./bootstrap   # If running from git.
./configure
make
make check   # optional
sudo make install

I was able to successfully perform the steps until make. When I try to make
I get the message:

*make  all-recursive*
*make[1]: Entering directory
'/home/atollkuci/varnish-4.0.3/varnish-modules'*
*Making all in src*
*make[2]: Entering directory
'/home/atollkuci/varnish-4.0.3/varnish-modules/src'*
*/bin/bash: PYTHON@: command not found*
*Makefile:803: recipe for target 'vcc_cookie_if.h' failed*
*make[2]: *** [vcc_cookie_if.h] Error 127*
*make[2]: Leaving directory
'/home/atollkuci/varnish-4.0.3/varnish-modules/src'*
*Makefile:485: recipe for target 'all-recursive' failed*
*make[1]: *** [all-recursive] Error 1*
*make[1]: Leaving directory '/home/atollkuci/varnish-4.0.3/varnish-modules'*
*Makefile:396: recipe for target 'all' failed*
*make: *** [all] Error 2*

It looks like it can't find python, but if I run from bash python or
python3 I get the python interpreter.

Any idea how to proceed?

I'm running Ubuntu 15.10 and varnish 4.0.3.

Thank you,
Albert










On Fri, Sep 2, 2016 at 7:05 PM, Albert Tollku?i <albert.tollkuci at gmail.com>
wrote:

> Sorry for the long post, but I'm stuck here. Below is the full log from
> varnishlog for a MISS request (so I can check what's happening with the
> backend). In bold there are the important bits:
>
> - In the original request I have "*-   Link           req 294922 restart*".
> Is the restart normal, because of the miss?
>
> - In the backend response I can see the set cookie header: *-
> BerespHeader   Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu,
> 24-Aug-2017 16:48:53 GMT; path=/*
>
> - My log doesn't show the set-cookie header (*-   VCL_Log        AT DEBUG
> - beresp set-cookie: 'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/;
> HttpOnly'*). Is that because there are two set-cookie headers and it is
> showing only the first?
>
> - Immediately after the log there's a line "*-   BerespUnset
>  Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu, 24-Aug-2017 16:48:53
> GMT; path=/"*. It looks like this is responsible for un-setting the
> cookie, is that right?
>
> - If so the cookie is striped between my first and second log and the code
> in between is the following:
>
> *    if(beresp.http.set-cookie)*
> *    {*
> *      std.log("AT Vanish - mobile.ikub.al <http://mobile.ikub.al>
> original backend set cookie: '" + beresp.http.set-cookie + "'");*
> *    }*
>
> *    # AT: We have to keep all cookies starting with __IKUB*
> *    set beresp.http.set-cookie = ";" + beresp.http.set-cookie;*
> *    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "; +",
> ";");*
> *    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
> ";(__IKUB.*)=", "; \1=");*
> *    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";[^
> ][^;]*", "");*
> *    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "^[;
> ]+|[; ]+$", "");*
>
> *    # Clean empty cookies*
> *    if (beresp.http.set-cookie ~ "^\s*$") {*
> *      set beresp.http.set-cookie = "";*
> *    }*
>
> *    if (beresp.http.set-cookie == ""){*
> *      unset beresp.http.set-cookie;*
> *    }*
> *    else*
> *    {*
> *      std.log("AT Varnish - mobile.ikub.al <http://mobile.ikub.al>
> processed backend cookie: '" + beresp.http.set-cookie + "'");*
> *    }*
>
> I have the same regex in the request processing and they're working, but
> maybe for some reason not working here...
>
>
> Full log from varnishlog:
>
> *   << Request  >> 294921
> -   Begin          req 294920 rxreq
> -   Timestamp      Start: 1472834933.190575 0.000000 0.000000
> -   Timestamp      Req: 1472834933.190575 0.000000 0.000000
> -   ReqStart       185.158.1.35 5547
> -   ReqMethod      GET
> -   ReqURL         /Default2.aspx
> -   ReqProtocol    HTTP/1.1
> -   ReqHeader      Host: mobile.ikub.al
> -   ReqHeader      Connection: keep-alive
> -   ReqHeader      Pragma: no-cache
> -   ReqHeader      Cache-Control: no-cache
> -   ReqHeader      Upgrade-Insecure-Requests: 1
> -   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
> -   ReqHeader      Accept: text/html,application/xhtml+
> xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> -   ReqHeader      Accept-Encoding: gzip, deflate, sdch
> -   ReqHeader      Accept-Language: en-US,en;q=0.8,sq;q=0.6
> -   ReqHeader      Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; _gat=1; __utmt=1;
> OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
> _ga=GA1.2.12079592.1472832572; __utma=1.12079592.1472832572.1
> -   ReqHeader      X-Forwarded-For: 185.158.1.35
> -   VCL_call       RECV
> -   VCL_Log        AT DEBUG - recv URL: /Default2.aspx. Cookies:
> '__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
> _gat=1; __utmt=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.147
> -   ReqHeader      x-at-debug: 1
> -   ReqUnset       Host: mobile.ikub.al
> -   ReqHeader      Host: mobile.ikub.al
> -   ReqURL         /Default2.aspx
> -   ReqUnset       Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; _gat=1; __utmt=1;
> OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
> _ga=GA1.2.12079592.1472832572; __utma=1.12079592.1472832572.1
> -   ReqHeader      Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; _gat=1; __utmt=1;
> OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
> _ga=GA1.2.12079592.1472832572; __utma=1.12079592.1472832572.1
> -   ReqUnset       Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; _gat=1; __utmt=1;
> OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
> _ga=GA1.2.12079592.1472832572; __utma=1.12079592.1472832572.1
> -   ReqHeader      Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; _gat=1;
> OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
> _ga=GA1.2.12079592.1472832572;
> -   ReqUnset       Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; _gat=1;
> OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
> _ga=GA1.2.12079592.1472832572;
> -   ReqHeader      Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; _gat=1;
> OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; _gat=1;
> OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: __gads=ID=c8c7e856e0927271:T=
> 1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   VCL_return     purge
> -   ReqUnset       Accept-Encoding: gzip, deflate, sdch
> -   ReqHeader      Accept-Encoding: gzip
> -   VCL_call       HASH
> -   VCL_return     lookup
> -   VCL_call       PURGE
> -   ReqHeader      X-Purge: Yes
> -   VCL_return     restart
> -   Timestamp      Restart: 1472834933.190646 0.000071 0.000071
> *-   Link           req 294922 restart*
> -   End
>
>
>
>
> *   << BeReq    >> 294923
> -   Begin          bereq 294922 fetch
> -   Timestamp      Start: 1472834933.190706 0.000000 0.000000
> -   BereqMethod    GET
> -   BereqURL       /Default2.aspx
> -   BereqProtocol  HTTP/1.1
> -   BereqHeader    Pragma: no-cache
> -   BereqHeader    Upgrade-Insecure-Requests: 1
> -   BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
> -   BereqHeader    Accept: text/html,application/xhtml+
> xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> -   BereqHeader    Accept-Language: en-US,en;q=0.8,sq;q=0.6
> -   BereqHeader    X-Forwarded-For: 185.158.1.35
> -   BereqHeader    Accept-Encoding: gzip
> -   BereqHeader    X-Purge: Yes
> -   BereqHeader    x-at-debug: 1
> -   BereqHeader    Host: mobile.ikub.al
> -   BereqHeader    x-state: MISS
> -   BereqHeader    X-Varnish: 294923
> -   VCL_call       BACKEND_FETCH
> -   VCL_Log        AT DEBUG - fetch bereq URL: /Default2.aspx. Cookies: ''
> -   VCL_return     fetch
> -   Backend        43 ikubmobileserver ikubmobileserver(13.94.212.211,,80)
> -   Timestamp      Bereq: 1472834933.190737 0.000031 0.000031
> -   Timestamp      Beresp: 1472834933.830661 0.639955 0.639924
> -   BerespProtocol HTTP/1.1
> -   BerespStatus   200
> -   BerespReason   OK
> -   BerespHeader   Cache-Control: private
> -   BerespHeader   Content-Type: text/html; charset=utf-8
> -   BerespHeader   Content-Encoding: gzip
> -   BerespHeader   Vary: Accept-Encoding
> -   BerespHeader   Server: Microsoft-IIS/8.5
> -   BerespHeader   X-AspNet-Version: 2.0.50727
> -   BerespHeader   Set-Cookie: ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;
> path=/; HttpOnly
> *-   BerespHeader   Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu,
> 24-Aug-2017 16:48:53 GMT; path=/*
> -   BerespHeader   X-Powered-By: ASP.NET
> -   BerespHeader   Date: Fri, 02 Sep 2016 16:48:53 GMT
> -   BerespHeader   Content-Length: 52233
> -   TTL            RFC 120 -1 -1 1472834934 1472834934 1472834933 0 0
> -   VCL_call       BACKEND_RESPONSE
> -   VCL_Log        AT DEBUG - bereq URL: /Default2.aspx. Cookies: ''
> *-   VCL_Log        AT DEBUG - beresp set-cookie:
> 'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'*
> -   VCL_Log        AT Vanish - mobile.ikub.al original backend set
> cookie: 'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'
> -   BerespUnset    Set-Cookie: ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;
> path=/; HttpOnly
> *-   BerespUnset    Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu,
> 24-Aug-2017 16:48:53 GMT; path=/*
> -   BerespHeader   set-cookie: ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;
> path=/; HttpOnly
> -   BerespUnset    set-cookie: ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;
> path=/; HttpOnly
> -   BerespHeader   set-cookie: ;ASP.NET_SessionId=
> pkkav5uflq02es45dktlo03w;path=/;HttpOnly
> -   BerespUnset    set-cookie: ;ASP.NET_SessionId=
> pkkav5uflq02es45dktlo03w;path=/;HttpOnly
> -   BerespHeader   set-cookie: ;ASP.NET_SessionId=
> pkkav5uflq02es45dktlo03w;path=/;HttpOnly
> -   BerespUnset    set-cookie: ;ASP.NET_SessionId=
> pkkav5uflq02es45dktlo03w;path=/;HttpOnly
> -   BerespHeader   set-cookie:
> -   BerespUnset    set-cookie:
> -   BerespHeader   set-cookie:
> -   BerespUnset    set-cookie:
> -   BerespHeader   set-cookie:
> -   BerespUnset    set-cookie:
> -   TTL            VCL 900 10 0 1472834934
> -   VCL_Log        AT DEBUG - bereq2 URL: /Default2.aspx. Cookies: ''
> -   VCL_Log        AT DEBUG - beresp2 set-cookie: ''
> -   TTL            VCL 900 21600 0 1472834934
> -   VCL_Log        AT DEBUG - bereq3 URL: /Default2.aspx. Cookies: ''
> -   VCL_Log        AT DEBUG - beresp3 set-cookie: ''
> -   VCL_return     deliver
> -   Storage        malloc s0
> -   ObjProtocol    HTTP/1.1
> -   ObjStatus      200
> -   ObjReason      OK
> -   ObjHeader      Cache-Control: private
> -   ObjHeader      Content-Type: text/html; charset=utf-8
> -   ObjHeader      Content-Encoding: gzip
> -   ObjHeader      Vary: Accept-Encoding
> -   ObjHeader      Server: Microsoft-IIS/8.5
> -   ObjHeader      X-AspNet-Version: 2.0.50727
> -   ObjHeader      X-Powered-By: ASP.NET
> -   ObjHeader      Date: Fri, 02 Sep 2016 16:48:53 GMT
> -   ObjHeader      Content-Length: 52233
> -   Fetch_Body     3 length stream
> -   Gzip           u F - 52233 194069 80 80 417793
> -   BackendReuse   43 ikubmobileserver(13.94.212.211,,80)
> -   Timestamp      BerespBody: 1472834933.831743 0.641037 0.001082
> -   Length         52233
> -   BereqAcct      473 0 473 428 52233 52661
> -   End
>
> *   << Request  >> 294922
> -   Begin          req 294921 restart
> -   Timestamp      Start: 1472834933.190646 0.000071 0.000000
> -   ReqStart       185.158.1.35 5547
> -   ReqMethod      GET
> -   ReqURL         /Default2.aspx
> -   ReqProtocol    HTTP/1.1
> -   ReqHeader      Connection: keep-alive
> -   ReqHeader      Pragma: no-cache
> -   ReqHeader      Cache-Control: no-cache
> -   ReqHeader      Upgrade-Insecure-Requests: 1
> -   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
> -   ReqHeader      Accept: text/html,application/xhtml+
> xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> -   ReqHeader      Accept-Language: en-US,en;q=0.8,sq;q=0.6
> -   ReqHeader      X-Forwarded-For: 185.158.1.35
> -   ReqHeader      x-at-debug: 1
> -   ReqHeader      Host: mobile.ikub.al
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Accept-Encoding: gzip
> -   ReqHeader      X-Purge: Yes
> -   VCL_call       RECV
> -   VCL_Log        AT DEBUG - recv URL: /Default2.aspx. Cookies:
> 'OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
> '
> -   ReqUnset       x-at-debug: 1
> -   ReqHeader      x-at-debug: 1
> -   ReqUnset       Host: mobile.ikub.al
> -   ReqHeader      Host: mobile.ikub.al
> -   ReqURL         /Default2.aspx
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   VCL_Log        AT DEBUG - recv2 URL: /Default2.aspx. Cookies:
> 'OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d;
> '
> -   ReqUnset       Cookie: OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: ;OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: ;OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> __auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: ;OX_plg=swf|shk|pm;__asc=
> c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: ;OX_plg=swf|shk|pm;__asc=
> c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: ;OX_plg=swf|shk|pm;__asc=
> c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
> -   ReqUnset       Cookie: ;OX_plg=swf|shk|pm;__asc=
> c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
> -   ReqHeader      Cookie: ;
> -   ReqUnset       Cookie: ;
> -   ReqHeader      Cookie:
> -   ReqUnset       Cookie:
> -   ReqHeader      Cookie:
> -   ReqUnset       Cookie:
> -   VCL_Log        AT DEBUG - recv3 URL: /Default2.aspx. Cookies: ''
> -   VCL_return     hash
> -   VCL_call       HASH
> -   VCL_return     lookup
> -   Debug          "XXXX MISS"
> -   VCL_call       MISS
> -   ReqHeader      x-state: MISS
> -   VCL_return     fetch
> -   Link           bereq 294923 fetch
> -   Timestamp      Fetch: 1472834933.831768 0.641192 0.641122
> -   RespProtocol   HTTP/1.1
> -   RespStatus     200
> -   RespReason     OK
> -   RespHeader     Cache-Control: private
> -   RespHeader     Content-Type: text/html; charset=utf-8
> -   RespHeader     Content-Encoding: gzip
> -   RespHeader     Vary: Accept-Encoding
> -   RespHeader     Server: Microsoft-IIS/8.5
> -   RespHeader     X-AspNet-Version: 2.0.50727
> -   RespHeader     X-Powered-By: ASP.NET
> -   RespHeader     Date: Fri, 02 Sep 2016 16:48:53 GMT
> -   RespHeader     Content-Length: 52233
> -   RespHeader     X-Varnish: 294922
> -   RespHeader     Age: 0
> -   RespHeader     Via: 1.1 varnish-v4
> -   VCL_call       DELIVER
> -   RespHeader     X-Cache: MISS
> -   RespHeader     X-Cache-Hits: 0
> -   RespUnset      X-Powered-By: ASP.NET
> -   RespUnset      Server: Microsoft-IIS/8.5
> -   RespUnset      X-Varnish: 294922
> -   RespUnset      Via: 1.1 varnish-v4
> -   RespUnset      X-AspNet-Version: 2.0.50727
> -   VCL_Log        AT DEBUG - deliver URL: /Default2.aspx. Cookies: ''
> -   VCL_Log        AT DEBUG - deliver set-cookie: ''
> -   VCL_return     deliver
> -   Timestamp      Process: 1472834933.831794 0.641218 0.000026
> -   Debug          "RES_MODE 2"
> -   RespHeader     Connection: keep-alive
> -   RespHeader     Accept-Ranges: bytes
> -   Timestamp      Resp: 1472834933.831830 0.641255 0.000036
> -   Debug          "XXX REF 2"
> -   ReqAcct        828 0 828 276 52233 52509
> -   End
>
>
> On Fri, Sep 2, 2016 at 6:28 PM, Albert Tollku?i <albert.tollkuci at gmail.com
> > wrote:
>
>> Thank you Guillaume,
>> probably I'm missing something, but can't find any unset.
>>
>> Is there anything running before vcl_backend_response? I see the cookie
>> in the backend side and is not there anymore in vcl_backend_response?
>>
>> Albert
>>
>> On Fri, Sep 2, 2016 at 10:53 AM, Guillaume Quintard <
>> guillaume at varnish-software.com> wrote:
>>
>>> Hi,
>>>
>>> Look at the varnishlog, there should be a trace of that unset, once
>>> confirmed, find it in your vcl.
>>>
>>> Know that if you don't return from a subroutine, the built-in vcl is
>>> executed ("locate builtin.vcl" to find a commented version of it).
>>>
>>> On Sep 1, 2016 12:59, "Albert Tollku?i" <albert.tollkuci at gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>> I've started using Varnish a few months ago and my vcl files has
>>>> started to get complicated. I'm serving 5 different sites with the same
>>>> varnish instance (using different backends) and this part is working.
>>>>
>>>> Now I have some cases that for some sites I need to keep some cookies
>>>> and cache different versions based on them. Basic scenario is to show the
>>>> user a message only the first time he sees the page. Without varnish if the
>>>> cookie is not there I set it and show the message. Next time the cookie is
>>>> there and no message is shown.
>>>>
>>>> With varnish my logic is to include the cookie in hash and cache two
>>>> different versions. The challenge is to make sure cookies are set correctly
>>>> and here I'm kind of stack. I've added some debug messages in my vcl and
>>>> for some reason I'm not seeing the cookies I set in my server in
>>>> vcl_backend_response. Inside the routine, I have the following code in the
>>>> beginning:
>>>>
>>>>        # AT: DEBUG
>>>>        if(bereq.http.x-at-debug == "1") {
>>>>          std.log("AT DEBUG - bereq URL: " + bereq.url + ". Cookies: '"
>>>> + bereq.http.Cookie + "'");
>>>>          std.log("AT DEBUG - beresp set-cookie: '" +
>>>>  beresp.http.set-cookie + "'");
>>>>        }
>>>>
>>>> I set x-at-debug header in vcl_recv based on my IP, so I can track only
>>>> my requests. In the server side I see the cookie "__IKUB_AL_Myikub_PopUp"
>>>> being set, but in vcl_backend_response is not.
>>>>
>>>> Is there anything in between that can strip the cookie?
>>>>
>>>> Thank you,
>>>> Albert
>>>>
>>>>
>>>>
>>>> --
>>>> Web: http://www.tollkuci.com
>>>> Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci>
>>>> Google+ <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
>>>> <https://www.facebook.com/albert.tollkuci> Twitter
>>>> <https://twitter.com/AlbertTollkuci> Career 2.0
>>>> <http://careers.stackoverflow.com/atollkuci>
>>>>
>>>>
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc at varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>
>>>
>>
>>
>> --
>> Web: http://www.tollkuci.com
>> Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci>
>> Google+ <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
>> <https://www.facebook.com/albert.tollkuci> Twitter
>> <https://twitter.com/AlbertTollkuci> Career 2.0
>> <http://careers.stackoverflow.com/atollkuci>
>> ------------------------------
>>
>>
>
>
> --
> Web: http://www.tollkuci.com
> Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
> <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
> <https://www.facebook.com/albert.tollkuci> Twitter
> <https://twitter.com/AlbertTollkuci> Career 2.0
> <http://careers.stackoverflow.com/atollkuci>
> ------------------------------
>



-- 
Web: http://www.tollkuci.com
Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
<https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
<https://www.facebook.com/albert.tollkuci> Twitter
<https://twitter.com/AlbertTollkuci> Career 2.0
<http://careers.stackoverflow.com/atollkuci>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160903/255dede2/attachment.html>

From miguel_3_gonzalez at yahoo.es  Sat Sep  3 09:59:42 2016
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Sat, 3 Sep 2016 11:59:42 +0200
Subject: bitrate and memory allocated
Message-ID: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>

Dear all,

  I have Varnish 4.0.3 in front of a WHM/Cpanel hosting around 40
Wordpress sites. Only 4-5 sites are quite visited but the rest are blogs
that don?t have many visits.

  Varnish seems to work fine, I have increased the RAM of the VPS from 7
to 21 Gb of RAM because from time to time the RAM of the system got
exhausted and I had to restart the OS.

  I have signed up in datadoghq to get a handle of how well Varnish is
doing and try to squeeze the performance of it. Mostly because I have
realized that when I test any site with a tool like pingdom, the first
test gets a worse performance than consecutive tests (when I assume
Varnish starts caching).

  I have assigned in malloc 6 Gb of RAM but I have been checking the
numbers provided by varnishstat and after 12 hours I don?t even get
close to 1 Gb of RAM (I assume c_bytes is what stands for) used by varnish:

 varnishstat -1 |grep SMA
SMA.s0.c_req                             33969         0.76 Allocator
requests
SMA.s0.c_fail                                0         0.00 Allocator
failures
SMA.s0.c_bytes                       984052835     21896.06 Bytes allocated
SMA.s0.c_freed                       395844358      8807.89 Bytes freed
SMA.s0.g_alloc                           19833          .   Allocations
outstanding
SMA.s0.g_bytes                       588208477          .   Bytes
outstanding
SMA.s0.g_space                      5854242467          .   Bytes available

 I had a ttl of 2 hours and now I have changed it to 24 hours, maybe I
will increase it if I see the RAM allocated to Varnish is not entirely
used. The bitrate is erratic, reaching almost 1 and then dropping to
less than 0.5.

Are my assumptions right?

Another question, I know that running varnishtop I get the current most
visited site, but how do I aggregate that info over the hours? I would
like to get a picture of which are the most used sites (I?m considering
separating storages and dedicate only RAM for the most used and have
disk storage for the rest).

Thanks in advance

Miguel

ps: Here is my default.vcl

#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at
https://www.varnish-cache.org/docs/
# and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.

# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;

import std;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "MYIP";
    .port = "82";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;


}

acl purge {
        "localhost";
        "127.0.0.1";
        "MYIP";
}

# This function is used when a request is send by a HTTP client (Browser)
sub vcl_recv {

        # remove ?ver=xxxxx strings from urls so css and js files are
cached.
        # Watch out when upgrading WordPress, need to restart Varnish or
flush cache.
        set req.url = regsub(req.url, "\?ver=.*$", "");

        # Remove "replytocom" from requests to make caching better.
        set req.url = regsub(req.url, "\?replytocom=.*$", "");

        #We pass real IP to the backend

        if (req.restarts == 0) {
            if (req.http.X-Forwarded-For) {
               set req.http.X-Forwarded-For = req.http.X-Forwarded-For +
", " + client.ip;
           } else {
            set req.http.X-Forwarded-For = client.ip;
           }
        }


        # Normalize the header, remove the port (in case you're testing
this on various TCP ports)

        set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

        # Remove has_js and CloudFlare/Google Analytics __* cookies.
        set req.http.Cookie = regsuball(req.http.Cookie,
"(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
        # Remove a ";" prefix, if present.
        set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");


        # Allow purging from ACL
        if (req.method == "PURGE") {
                # If not allowed then a error 405 is returned
                if (!client.ip ~ purge) {
                        return(synth(405, "This IP is not allowed to
send PURGE requests."));
                }
                # If allowed, do a cache_lookup -> vlc_hit() or vlc_miss()
                return (purge);
        }

        # Post requests will not be cached
        #if (req.http.Authorization || req.method == "POST") {
        #       return (pass);
        #}

        # Pass anything other than GET and HEAD directly.
        if (req.method != "GET" && req.method != "HEAD") {
                return( pass );
        }      /* We only deal with GET and HEAD by default */

        # --- WordPress specific configuration

        # Did not cache the admin and login pages
        if (req.url ~
"nocache|cart|my-account|checkout|addons|tienda|carro|wp-admin|wp-(comments-post|login|signup|activate|mail|cron)\.php|preview\=true|admin-ajax\.php|xmlrpc\.php|bb-admin|whm-server-status|server-status|control\.php|bb-login\.php|bb-reset-password\.php|register\.php")
{
            return (pass);
        }

        if (req.url ~ "(ajax|dynamic|custom)") {
             return(pass);
        }

        # Remove the "has_js" cookie
        set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(;
)?", "");

        # Remove any Google Analytics based cookies
        set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(;
)?", "");

        # Remove the Quant Capital cookies (added by some plugin, all __qca)
        set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(;
)?", "");

        # Remove the wp-settings-1 cookie
        set req.http.Cookie = regsuball(req.http.Cookie,
"wp-settings-1=[^;]+(; )?", "");

        # Remove the wp-settings-time-1 cookie
        set req.http.Cookie = regsuball(req.http.Cookie,
"wp-settings-time-1=[^;]+(; )?", "");

        # Remove the wp test cookie
        set req.http.Cookie = regsuball(req.http.Cookie,
"wordpress_test_cookie=[^;]+(; )?", "");

        # Are there cookies left with only spaces or that are empty?
        if (req.http.cookie ~ "^ *$") {
                    unset req.http.cookie;
        }

        # Cache the following files extensions
        if (req.url ~ "\.(txt|css|js|png|gif|jp(e)?g|swf|ico)") {
                unset req.http.cookie;
        }

        # Normalize Accept-Encoding header and compression
        # https://www.varnish-cache.org/docs/3.0/tutorial/vary.html
        if (req.http.Accept-Encoding) {
                # Do no compress compressed files...
                if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
                                unset req.http.Accept-Encoding;
                } elsif (req.http.Accept-Encoding ~ "gzip") {
                        set req.http.Accept-Encoding = "gzip";
                } elsif (req.http.Accept-Encoding ~ "deflate") {
                        set req.http.Accept-Encoding = "deflate";
                } else {
                        unset req.http.Accept-Encoding;
                }
        }

        # Check the cookies for wordpress-specific items
        if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~
"comment_") {
                return (pass);
        }
        if (!req.http.cookie) {
                unset req.http.cookie;
        }

        # --- End of WordPress specific configuration

        # Did not cache HTTP authentication and HTTP Cookie
        if (req.http.Authorization || req.http.Cookie) {
                # Not cacheable by default
                return (pass);
        }

        # Cache all others requests
        return (hash);
}

sub vcl_pipe {
        return (pipe);
}

sub vcl_pass {
        return (fetch);
}

# The data on which the hashing will take place
sub vcl_hash {
        hash_data(req.url);
        if (req.http.host) {
        hash_data(req.http.host);
        } else {
        hash_data(server.ip);
        }

        # If the client supports compression, keep that in a different cache
        if (req.http.Accept-Encoding) {
                hash_data(req.http.Accept-Encoding);
        }

        return (lookup);
}

# This function is used when a request is sent by our backend (Nginx server)
sub vcl_backend_response {
        # Remove some headers we never want to see
        unset beresp.http.Server;
        unset beresp.http.X-Powered-By;

        # For static content strip all backend cookies
        if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") {
                unset beresp.http.cookie;
        }
        # Don't store backend
        if (bereq.url ~ "wp-(login|admin)" || bereq.url ~ "preview=true") {
                set beresp.uncacheable = true;
                set beresp.ttl = 30s;
                return (deliver);
        }

        # Only allow cookies to be set if we're in admin area
                if (!(bereq.url ~
"(wp-login|cart|my-account|checkout|addons|tienda|carro|wp-admin|preview=true)"))
{
                unset beresp.http.set-cookie;
        }

        # don't cache response to posted requests or those with basic auth
        if ( bereq.method == "POST" || bereq.http.Authorization ) {
                set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
        }

        # don't cache search results
        if ( bereq.url ~ "\?s=" ){
                set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
        }

        # only cache status ok
        if ( beresp.status != 200 ) {
                set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
        }

        # A TTL of 24h
        set beresp.ttl = 24h;
        # Define the default grace period to serve cached content
        #set beresp.grace = 30s;
        set beresp.grace = 1h;

        return (deliver);
}

# The routine when we deliver the HTTP request to the user
# Last chance to modify headers that are sent to the client
sub vcl_deliver {
        if (obj.hits > 0) {
                set resp.http.X-Cache = "cached";
        } else {
                set resp.http.x-Cache = "uncached";
        }

        # Remove some headers: PHP version
        unset resp.http.X-Powered-By;

        # Remove some headers: Apache version & OS
        unset resp.http.Server;

        # Remove some heanders: Varnish
        unset resp.http.Via;
        unset resp.http.X-Varnish;

        return (deliver);
}

sub vcl_init {
        return (ok);
}

sub vcl_fini {
        return (ok);
}











From miguel_3_gonzalez at yahoo.es  Sat Sep  3 10:14:39 2016
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Sat, 3 Sep 2016 12:14:39 +0200
Subject: bitrate and memory allocated
In-Reply-To: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>
References: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>
Message-ID: <c4b627c6-67e0-eddc-cb97-e059addd45a9@yahoo.es>

Why I don?t get a copy of my messages and I do get the rest of emails? I
have just double checked my settings and I should get a copy of my messages

Miguel

On 09/03/16 11:59 AM, Miguel Gonz?lez wrote:
> Dear all,
> 
>   I have Varnish 4.0.3 in front of a WHM/Cpanel hosting around 40
> Wordpress sites. Only 4-5 sites are quite visited but the rest are blogs
> that don?t have many visits.
> 
>   Varnish seems to work fine, I have increased the RAM of the VPS from 7
> to 21 Gb of RAM because from time to time the RAM of the system got
> exhausted and I had to restart the OS.
> 
>   I have signed up in datadoghq to get a handle of how well Varnish is
> doing and try to squeeze the performance of it. Mostly because I have
> realized that when I test any site with a tool like pingdom, the first
> test gets a worse performance than consecutive tests (when I assume
> Varnish starts caching).
> 
>   I have assigned in malloc 6 Gb of RAM but I have been checking the
> numbers provided by varnishstat and after 12 hours I don?t even get
> close to 1 Gb of RAM (I assume c_bytes is what stands for) used by varnish:
> 
>  varnishstat -1 |grep SMA
> SMA.s0.c_req                             33969         0.76 Allocator
> requests
> SMA.s0.c_fail                                0         0.00 Allocator
> failures
> SMA.s0.c_bytes                       984052835     21896.06 Bytes allocated
> SMA.s0.c_freed                       395844358      8807.89 Bytes freed
> SMA.s0.g_alloc                           19833          .   Allocations
> outstanding
> SMA.s0.g_bytes                       588208477          .   Bytes
> outstanding
> SMA.s0.g_space                      5854242467          .   Bytes available
> 
>  I had a ttl of 2 hours and now I have changed it to 24 hours, maybe I
> will increase it if I see the RAM allocated to Varnish is not entirely
> used. The bitrate is erratic, reaching almost 1 and then dropping to
> less than 0.5.
> 
> Are my assumptions right?
> 
> Another question, I know that running varnishtop I get the current most
> visited site, but how do I aggregate that info over the hours? I would
> like to get a picture of which are the most used sites (I?m considering
> separating storages and dedicate only RAM for the most used and have
> disk storage for the rest).
> 
> Thanks in advance
> 
> Miguel
> 
> ps: Here is my default.vcl
> 
> #
> # This is an example VCL file for Varnish.
> #
> # It does not do anything by default, delegating control to the
> # builtin VCL. The builtin VCL is called when there is no explicit
> # return statement.
> #
> # See the VCL chapters in the Users Guide at
> https://www.varnish-cache.org/docs/
> # and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.
> 
> # Marker to tell the VCL compiler that this VCL has been adapted to the
> # new 4.0 format.
> vcl 4.0;
> 
> import std;
> 
> # Default backend definition. Set this to point to your content server.
> backend default {
>     .host = "MYIP";
>     .port = "82";
>     .connect_timeout = 600s;
>     .first_byte_timeout = 600s;
>     .between_bytes_timeout = 600s;
> 
> 
> }
> 
> acl purge {
>         "localhost";
>         "127.0.0.1";
>         "MYIP";
> }
> 
> # This function is used when a request is send by a HTTP client (Browser)
> sub vcl_recv {
> 
>         # remove ?ver=xxxxx strings from urls so css and js files are
> cached.
>         # Watch out when upgrading WordPress, need to restart Varnish or
> flush cache.
>         set req.url = regsub(req.url, "\?ver=.*$", "");
> 
>         # Remove "replytocom" from requests to make caching better.
>         set req.url = regsub(req.url, "\?replytocom=.*$", "");
> 
>         #We pass real IP to the backend
> 
>         if (req.restarts == 0) {
>             if (req.http.X-Forwarded-For) {
>                set req.http.X-Forwarded-For = req.http.X-Forwarded-For +
> ", " + client.ip;
>            } else {
>             set req.http.X-Forwarded-For = client.ip;
>            }
>         }
> 
> 
>         # Normalize the header, remove the port (in case you're testing
> this on various TCP ports)
> 
>         set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
> 
>         # Remove has_js and CloudFlare/Google Analytics __* cookies.
>         set req.http.Cookie = regsuball(req.http.Cookie,
> "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
>         # Remove a ";" prefix, if present.
>         set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
> 
> 
>         # Allow purging from ACL
>         if (req.method == "PURGE") {
>                 # If not allowed then a error 405 is returned
>                 if (!client.ip ~ purge) {
>                         return(synth(405, "This IP is not allowed to
> send PURGE requests."));
>                 }
>                 # If allowed, do a cache_lookup -> vlc_hit() or vlc_miss()
>                 return (purge);
>         }
> 
>         # Post requests will not be cached
>         #if (req.http.Authorization || req.method == "POST") {
>         #       return (pass);
>         #}
> 
>         # Pass anything other than GET and HEAD directly.
>         if (req.method != "GET" && req.method != "HEAD") {
>                 return( pass );
>         }      /* We only deal with GET and HEAD by default */
> 
>         # --- WordPress specific configuration
> 
>         # Did not cache the admin and login pages
>         if (req.url ~
> "nocache|cart|my-account|checkout|addons|tienda|carro|wp-admin|wp-(comments-post|login|signup|activate|mail|cron)\.php|preview\=true|admin-ajax\.php|xmlrpc\.php|bb-admin|whm-server-status|server-status|control\.php|bb-login\.php|bb-reset-password\.php|register\.php")
> {
>             return (pass);
>         }
> 
>         if (req.url ~ "(ajax|dynamic|custom)") {
>              return(pass);
>         }
> 
>         # Remove the "has_js" cookie
>         set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(;
> )?", "");
> 
>         # Remove any Google Analytics based cookies
>         set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(;
> )?", "");
> 
>         # Remove the Quant Capital cookies (added by some plugin, all __qca)
>         set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(;
> )?", "");
> 
>         # Remove the wp-settings-1 cookie
>         set req.http.Cookie = regsuball(req.http.Cookie,
> "wp-settings-1=[^;]+(; )?", "");
> 
>         # Remove the wp-settings-time-1 cookie
>         set req.http.Cookie = regsuball(req.http.Cookie,
> "wp-settings-time-1=[^;]+(; )?", "");
> 
>         # Remove the wp test cookie
>         set req.http.Cookie = regsuball(req.http.Cookie,
> "wordpress_test_cookie=[^;]+(; )?", "");
> 
>         # Are there cookies left with only spaces or that are empty?
>         if (req.http.cookie ~ "^ *$") {
>                     unset req.http.cookie;
>         }
> 
>         # Cache the following files extensions
>         if (req.url ~ "\.(txt|css|js|png|gif|jp(e)?g|swf|ico)") {
>                 unset req.http.cookie;
>         }
> 
>         # Normalize Accept-Encoding header and compression
>         # https://www.varnish-cache.org/docs/3.0/tutorial/vary.html
>         if (req.http.Accept-Encoding) {
>                 # Do no compress compressed files...
>                 if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
>                                 unset req.http.Accept-Encoding;
>                 } elsif (req.http.Accept-Encoding ~ "gzip") {
>                         set req.http.Accept-Encoding = "gzip";
>                 } elsif (req.http.Accept-Encoding ~ "deflate") {
>                         set req.http.Accept-Encoding = "deflate";
>                 } else {
>                         unset req.http.Accept-Encoding;
>                 }
>         }
> 
>         # Check the cookies for wordpress-specific items
>         if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~
> "comment_") {
>                 return (pass);
>         }
>         if (!req.http.cookie) {
>                 unset req.http.cookie;
>         }
> 
>         # --- End of WordPress specific configuration
> 
>         # Did not cache HTTP authentication and HTTP Cookie
>         if (req.http.Authorization || req.http.Cookie) {
>                 # Not cacheable by default
>                 return (pass);
>         }
> 
>         # Cache all others requests
>         return (hash);
> }
> 
> sub vcl_pipe {
>         return (pipe);
> }
> 
> sub vcl_pass {
>         return (fetch);
> }
> 
> # The data on which the hashing will take place
> sub vcl_hash {
>         hash_data(req.url);
>         if (req.http.host) {
>         hash_data(req.http.host);
>         } else {
>         hash_data(server.ip);
>         }
> 
>         # If the client supports compression, keep that in a different cache
>         if (req.http.Accept-Encoding) {
>                 hash_data(req.http.Accept-Encoding);
>         }
> 
>         return (lookup);
> }
> 
> # This function is used when a request is sent by our backend (Nginx server)
> sub vcl_backend_response {
>         # Remove some headers we never want to see
>         unset beresp.http.Server;
>         unset beresp.http.X-Powered-By;
> 
>         # For static content strip all backend cookies
>         if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") {
>                 unset beresp.http.cookie;
>         }
>         # Don't store backend
>         if (bereq.url ~ "wp-(login|admin)" || bereq.url ~ "preview=true") {
>                 set beresp.uncacheable = true;
>                 set beresp.ttl = 30s;
>                 return (deliver);
>         }
> 
>         # Only allow cookies to be set if we're in admin area
>                 if (!(bereq.url ~
> "(wp-login|cart|my-account|checkout|addons|tienda|carro|wp-admin|preview=true)"))
> {
>                 unset beresp.http.set-cookie;
>         }
> 
>         # don't cache response to posted requests or those with basic auth
>         if ( bereq.method == "POST" || bereq.http.Authorization ) {
>                 set beresp.uncacheable = true;
>                 set beresp.ttl = 120s;
>                 return (deliver);
>         }
> 
>         # don't cache search results
>         if ( bereq.url ~ "\?s=" ){
>                 set beresp.uncacheable = true;
>                 set beresp.ttl = 120s;
>                 return (deliver);
>         }
> 
>         # only cache status ok
>         if ( beresp.status != 200 ) {
>                 set beresp.uncacheable = true;
>                 set beresp.ttl = 120s;
>                 return (deliver);
>         }
> 
>         # A TTL of 24h
>         set beresp.ttl = 24h;
>         # Define the default grace period to serve cached content
>         #set beresp.grace = 30s;
>         set beresp.grace = 1h;
> 
>         return (deliver);
> }
> 
> # The routine when we deliver the HTTP request to the user
> # Last chance to modify headers that are sent to the client
> sub vcl_deliver {
>         if (obj.hits > 0) {
>                 set resp.http.X-Cache = "cached";
>         } else {
>                 set resp.http.x-Cache = "uncached";
>         }
> 
>         # Remove some headers: PHP version
>         unset resp.http.X-Powered-By;
> 
>         # Remove some headers: Apache version & OS
>         unset resp.http.Server;
> 
>         # Remove some heanders: Varnish
>         unset resp.http.Via;
>         unset resp.http.X-Varnish;
> 
>         return (deliver);
> }
> 
> sub vcl_init {
>         return (ok);
> }
> 
> sub vcl_fini {
>         return (ok);
> }
> 
> 
> 
> 
> 
> 
> 
> 




From miguel_3_gonzalez at yahoo.es  Sat Sep  3 17:23:01 2016
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Sat, 3 Sep 2016 19:23:01 +0200
Subject: Caching for some cookies?
In-Reply-To: <CACqaJ6gHEQNYbdOf=Bt7SLYDdTmxKH0nRa6dFO++VwH7rgQisg@mail.gmail.com>
References: <CACqaJ6hqbnCUxeijhpU-zuH0iw8kkEr=TM21g9uMryv6=LM4gg@mail.gmail.com>
 <CACqaJ6gHEQNYbdOf=Bt7SLYDdTmxKH0nRa6dFO++VwH7rgQisg@mail.gmail.com>
Message-ID: <33062d91-f14b-2df6-a039-2f5ad33bc268@yahoo.es>

Those vmods are for varnish open source version?

ps: Why my messages are stored in the archives but are not getting
through by email?



On 09/03/16 11:16 AM, Albert Tollku?i wrote:
> After digging more it seems I have to install varnish modules
> from https://github.com/varnish/varnish-modules. The instructions to
> install them are:
> 
> sudo apt-get install libvarnishapi-dev || sudo yum install varnish-libs-devel
> ./bootstrap   # If running from git.
> ./configure
> make
> make check   # optional
> sudo make install
> 
> I was able to successfully perform the steps until make. When I try to
> make I get the message:
> 
> /make  all-recursive/
> /make[1]: Entering directory
> '/home/atollkuci/varnish-4.0.3/varnish-modules'/
> /Making all in src/
> /make[2]: Entering directory
> '/home/atollkuci/varnish-4.0.3/varnish-modules/src'/
> //bin/bash: PYTHON@: command not found/
> /Makefile:803: recipe for target 'vcc_cookie_if.h' failed/
> /make[2]: *** [vcc_cookie_if.h] Error 127/
> /make[2]: Leaving directory
> '/home/atollkuci/varnish-4.0.3/varnish-modules/src'/
> /Makefile:485: recipe for target 'all-recursive' failed/
> /make[1]: *** [all-recursive] Error 1/
> /make[1]: Leaving directory '/home/atollkuci/varnish-4.0.3/varnish-modules'/
> /Makefile:396: recipe for target 'all' failed/
> /make: *** [all] Error 2/
> 
> It looks like it can't find python, but if I run from bash python or
> python3 I get the python interpreter.
> 
> Any idea how to proceed?
> 
> I'm running Ubuntu 15.10 and varnish 4.0.3.
> 
> Thank you,
> Albert
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Fri, Sep 2, 2016 at 7:05 PM, Albert Tollku?i
> <albert.tollkuci at gmail.com <mailto:albert.tollkuci at gmail.com>> wrote:
> 
>     Sorry for the long post, but I'm stuck here. Below is the full log
>     from varnishlog for a MISS request (so I can check what's happening
>     with the backend). In bold there are the important bits:
> 
>     - In the original request I have "*-   Link           req 294922
>     restart*". Is the restart normal, because of the miss?
> 
>     - In the backend response I can see the set cookie header: *-  
>     BerespHeader   Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu,
>     24-Aug-2017 16:48:53 GMT; path=/*
>     *
>     *
>     - My log doesn't show the set-cookie header (*-   VCL_Log        AT
>     DEBUG - beresp set-cookie:
>     'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'*). Is
>     that because there are two set-cookie headers and it is showing only
>     the first?
> 
>     - Immediately after the log there's a line "*-   BerespUnset  
>      Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu, 24-Aug-2017
>     16:48:53 GMT; path=/"*. It looks like this is responsible for
>     un-setting the cookie, is that right?
> 
>     - If so the cookie is striped between my first and second log and
>     the code in between is the following:
> 
>     /    if(beresp.http.set-cookie)/
>     /    {/
>     /      std.log("AT Vanish - mobile.ikub.al <http://mobile.ikub.al>
>     original backend set cookie: '" + beresp.http.set-cookie + "'");/
>     /    }/
>     /
>     /
>     /    # AT: We have to keep all cookies starting with __IKUB/
>     /    set beresp.http.set-cookie = ";" + beresp.http.set-cookie;/
>     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
>     "; +", ";");/
>     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
>     ";(__IKUB.*)=", "; \1=");/
>     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
>     ";[^ ][^;]*", "");/
>     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
>     "^[; ]+|[; ]+$", "");/
>     /
>     /
>     /    # Clean empty cookies/
>     /    if (beresp.http.set-cookie ~ "^\s*$") {/
>     /      set beresp.http.set-cookie = "";/
>     /    }/
>     /
>     /
>     /    if (beresp.http.set-cookie == ""){/
>     /      unset beresp.http.set-cookie;/
>     /    }/
>     /    else/
>     /    {/
>     /      std.log("AT Varnish - mobile.ikub.al <http://mobile.ikub.al>
>     processed backend cookie: '" + beresp.http.set-cookie + "'");/
>     /    }/
> 
>     I have the same regex in the request processing and they're working,
>     but maybe for some reason not working here...
> 
> 
>     Full log from varnishlog:
> 
>     *   << Request  >> 294921    
>     -   Begin          req 294920 rxreq
>     -   Timestamp      Start: 1472834933.190575 0.000000 0.000000
>     -   Timestamp      Req: 1472834933.190575 0.000000 0.000000
>     -   ReqStart       185.158.1.35 5547
>     -   ReqMethod      GET
>     -   ReqURL         /Default2.aspx
>     -   ReqProtocol    HTTP/1.1
>     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
>     -   ReqHeader      Connection: keep-alive
>     -   ReqHeader      Pragma: no-cache
>     -   ReqHeader      Cache-Control: no-cache
>     -   ReqHeader      Upgrade-Insecure-Requests: 1
>     -   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64;
>     x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116
>     Safari/537.36
>     -   ReqHeader      Accept:
>     text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
>     -   ReqHeader      Accept-Encoding: gzip, deflate, sdch
>     -   ReqHeader      Accept-Language: en-US,en;q=0.8,sq;q=0.6
>     -   ReqHeader      Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
>     __utma=1.12079592.1472832572.1
>     -   ReqHeader      X-Forwarded-For: 185.158.1.35
>     -   VCL_call       RECV
>     -   VCL_Log        AT DEBUG - recv URL: /Default2.aspx. Cookies:
>     '__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.147
>     -   ReqHeader      x-at-debug: 1
>     -   ReqUnset       Host: mobile.ikub.al <http://mobile.ikub.al>
>     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
>     -   ReqURL         /Default2.aspx
>     -   ReqUnset       Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
>     __utma=1.12079592.1472832572.1
>     -   ReqHeader      Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
>     __utma=1.12079592.1472832572.1
>     -   ReqUnset       Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
>     __utma=1.12079592.1472832572.1
>     -   ReqHeader      Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572; 
>     -   ReqUnset       Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572; 
>     -   ReqHeader      Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie:
>     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   VCL_return     purge
>     -   ReqUnset       Accept-Encoding: gzip, deflate, sdch
>     -   ReqHeader      Accept-Encoding: gzip
>     -   VCL_call       HASH
>     -   VCL_return     lookup
>     -   VCL_call       PURGE
>     -   ReqHeader      X-Purge: Yes
>     -   VCL_return     restart
>     -   Timestamp      Restart: 1472834933.190646 0.000071 0.000071
>     *-   Link           req 294922 restart*
>     -   End            
> 
> 
> 
> 
>     *   << BeReq    >> 294923    
>     -   Begin          bereq 294922 fetch
>     -   Timestamp      Start: 1472834933.190706 0.000000 0.000000
>     -   BereqMethod    GET
>     -   BereqURL       /Default2.aspx
>     -   BereqProtocol  HTTP/1.1
>     -   BereqHeader    Pragma: no-cache
>     -   BereqHeader    Upgrade-Insecure-Requests: 1
>     -   BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64;
>     x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116
>     Safari/537.36
>     -   BereqHeader    Accept:
>     text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
>     -   BereqHeader    Accept-Language: en-US,en;q=0.8,sq;q=0.6
>     -   BereqHeader    X-Forwarded-For: 185.158.1.35
>     -   BereqHeader    Accept-Encoding: gzip
>     -   BereqHeader    X-Purge: Yes
>     -   BereqHeader    x-at-debug: 1
>     -   BereqHeader    Host: mobile.ikub.al <http://mobile.ikub.al>
>     -   BereqHeader    x-state: MISS
>     -   BereqHeader    X-Varnish: 294923
>     -   VCL_call       BACKEND_FETCH
>     -   VCL_Log        AT DEBUG - fetch bereq URL: /Default2.aspx.
>     Cookies: ''
>     -   VCL_return     fetch
>     -   Backend        43 ikubmobileserver
>     ikubmobileserver(13.94.212.211,,80)
>     -   Timestamp      Bereq: 1472834933.190737 0.000031 0.000031
>     -   Timestamp      Beresp: 1472834933.830661 0.639955 0.639924
>     -   BerespProtocol HTTP/1.1
>     -   BerespStatus   200
>     -   BerespReason   OK
>     -   BerespHeader   Cache-Control: private
>     -   BerespHeader   Content-Type: text/html; charset=utf-8
>     -   BerespHeader   Content-Encoding: gzip
>     -   BerespHeader   Vary: Accept-Encoding
>     -   BerespHeader   Server: Microsoft-IIS/8.5
>     -   BerespHeader   X-AspNet-Version: 2.0.50727
>     -   BerespHeader   Set-Cookie:
>     ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
>     *-   BerespHeader   Set-Cookie: __IKUB_AL_Myikub_PopUp=1;
>     expires=Thu, 24-Aug-2017 16:48:53 GMT; path=/*
>     -   BerespHeader   X-Powered-By: ASP.NET <http://ASP.NET>
>     -   BerespHeader   Date: Fri, 02 Sep 2016 16:48:53 GMT
>     -   BerespHeader   Content-Length: 52233
>     -   TTL            RFC 120 -1 -1 1472834934 1472834934 1472834933 0 0
>     -   VCL_call       BACKEND_RESPONSE
>     -   VCL_Log        AT DEBUG - bereq URL: /Default2.aspx. Cookies: ''
>     *-   VCL_Log        AT DEBUG - beresp set-cookie:
>     'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'*
>     -   VCL_Log        AT Vanish - mobile.ikub.al
>     <http://mobile.ikub.al> original backend set cookie:
>     'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'
>     -   BerespUnset    Set-Cookie:
>     ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
>     *-   BerespUnset    Set-Cookie: __IKUB_AL_Myikub_PopUp=1;
>     expires=Thu, 24-Aug-2017 16:48:53 GMT; path=/*
>     -   BerespHeader   set-cookie:
>     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
>     -   BerespUnset    set-cookie:
>     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
>     -   BerespHeader   set-cookie:
>     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
>     -   BerespUnset    set-cookie:
>     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
>     -   BerespHeader   set-cookie:
>     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
>     -   BerespUnset    set-cookie:
>     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
>     -   BerespHeader   set-cookie: 
>     -   BerespUnset    set-cookie: 
>     -   BerespHeader   set-cookie: 
>     -   BerespUnset    set-cookie: 
>     -   BerespHeader   set-cookie: 
>     -   BerespUnset    set-cookie: 
>     -   TTL            VCL 900 10 0 1472834934
>     -   VCL_Log        AT DEBUG - bereq2 URL: /Default2.aspx. Cookies: ''
>     -   VCL_Log        AT DEBUG - beresp2 set-cookie: ''
>     -   TTL            VCL 900 21600 0 1472834934
>     -   VCL_Log        AT DEBUG - bereq3 URL: /Default2.aspx. Cookies: ''
>     -   VCL_Log        AT DEBUG - beresp3 set-cookie: ''
>     -   VCL_return     deliver
>     -   Storage        malloc s0
>     -   ObjProtocol    HTTP/1.1
>     -   ObjStatus      200
>     -   ObjReason      OK
>     -   ObjHeader      Cache-Control: private
>     -   ObjHeader      Content-Type: text/html; charset=utf-8
>     -   ObjHeader      Content-Encoding: gzip
>     -   ObjHeader      Vary: Accept-Encoding
>     -   ObjHeader      Server: Microsoft-IIS/8.5
>     -   ObjHeader      X-AspNet-Version: 2.0.50727
>     -   ObjHeader      X-Powered-By: ASP.NET <http://ASP.NET>
>     -   ObjHeader      Date: Fri, 02 Sep 2016 16:48:53 GMT
>     -   ObjHeader      Content-Length: 52233
>     -   Fetch_Body     3 length stream
>     -   Gzip           u F - 52233 194069 80 80 417793
>     -   BackendReuse   43 ikubmobileserver(13.94.212.211,,80)
>     -   Timestamp      BerespBody: 1472834933.831743 0.641037 0.001082
>     -   Length         52233
>     -   BereqAcct      473 0 473 428 52233 52661
>     -   End            
> 
>     *   << Request  >> 294922    
>     -   Begin          req 294921 restart
>     -   Timestamp      Start: 1472834933.190646 0.000071 0.000000
>     -   ReqStart       185.158.1.35 5547
>     -   ReqMethod      GET
>     -   ReqURL         /Default2.aspx
>     -   ReqProtocol    HTTP/1.1
>     -   ReqHeader      Connection: keep-alive
>     -   ReqHeader      Pragma: no-cache
>     -   ReqHeader      Cache-Control: no-cache
>     -   ReqHeader      Upgrade-Insecure-Requests: 1
>     -   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64;
>     x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116
>     Safari/537.36
>     -   ReqHeader      Accept:
>     text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
>     -   ReqHeader      Accept-Language: en-US,en;q=0.8,sq;q=0.6
>     -   ReqHeader      X-Forwarded-For: 185.158.1.35
>     -   ReqHeader      x-at-debug: 1
>     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Accept-Encoding: gzip
>     -   ReqHeader      X-Purge: Yes
>     -   VCL_call       RECV
>     -   VCL_Log        AT DEBUG - recv URL: /Default2.aspx. Cookies:
>     'OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; '
>     -   ReqUnset       x-at-debug: 1
>     -   ReqHeader      x-at-debug: 1
>     -   ReqUnset       Host: mobile.ikub.al <http://mobile.ikub.al>
>     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
>     -   ReqURL         /Default2.aspx
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   VCL_Log        AT DEBUG - recv2 URL: /Default2.aspx. Cookies:
>     'OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>     __auc=c970f060156eba988423264f56d; '
>     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie: ;OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqUnset       Cookie: ;OX_plg=swf|shk|pm;
>     __asc=c970f060156eba988423264f56d; __auc=c970f060156eba988423264f56d; 
>     -   ReqHeader      Cookie:
>     ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
>     -   ReqUnset       Cookie:
>     ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
>     -   ReqHeader      Cookie:
>     ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
>     -   ReqUnset       Cookie:
>     ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
>     -   ReqHeader      Cookie: ;
>     -   ReqUnset       Cookie: ;
>     -   ReqHeader      Cookie: 
>     -   ReqUnset       Cookie: 
>     -   ReqHeader      Cookie: 
>     -   ReqUnset       Cookie: 
>     -   VCL_Log        AT DEBUG - recv3 URL: /Default2.aspx. Cookies: ''
>     -   VCL_return     hash
>     -   VCL_call       HASH
>     -   VCL_return     lookup
>     -   Debug          "XXXX MISS"
>     -   VCL_call       MISS
>     -   ReqHeader      x-state: MISS
>     -   VCL_return     fetch
>     -   Link           bereq 294923 fetch
>     -   Timestamp      Fetch: 1472834933.831768 0.641192 0.641122
>     -   RespProtocol   HTTP/1.1
>     -   RespStatus     200
>     -   RespReason     OK
>     -   RespHeader     Cache-Control: private
>     -   RespHeader     Content-Type: text/html; charset=utf-8
>     -   RespHeader     Content-Encoding: gzip
>     -   RespHeader     Vary: Accept-Encoding
>     -   RespHeader     Server: Microsoft-IIS/8.5
>     -   RespHeader     X-AspNet-Version: 2.0.50727
>     -   RespHeader     X-Powered-By: ASP.NET <http://ASP.NET>
>     -   RespHeader     Date: Fri, 02 Sep 2016 16:48:53 GMT
>     -   RespHeader     Content-Length: 52233
>     -   RespHeader     X-Varnish: 294922
>     -   RespHeader     Age: 0
>     -   RespHeader     Via: 1.1 varnish-v4
>     -   VCL_call       DELIVER
>     -   RespHeader     X-Cache: MISS
>     -   RespHeader     X-Cache-Hits: 0
>     -   RespUnset      X-Powered-By: ASP.NET <http://ASP.NET>
>     -   RespUnset      Server: Microsoft-IIS/8.5
>     -   RespUnset      X-Varnish: 294922
>     -   RespUnset      Via: 1.1 varnish-v4
>     -   RespUnset      X-AspNet-Version: 2.0.50727
>     -   VCL_Log        AT DEBUG - deliver URL: /Default2.aspx. Cookies: ''
>     -   VCL_Log        AT DEBUG - deliver set-cookie: ''
>     -   VCL_return     deliver
>     -   Timestamp      Process: 1472834933.831794 0.641218 0.000026
>     -   Debug          "RES_MODE 2"
>     -   RespHeader     Connection: keep-alive
>     -   RespHeader     Accept-Ranges: bytes
>     -   Timestamp      Resp: 1472834933.831830 0.641255 0.000036
>     -   Debug          "XXX REF 2"
>     -   ReqAcct        828 0 828 276 52233 52509
>     -   End            
> 
> 
>     On Fri, Sep 2, 2016 at 6:28 PM, Albert Tollku?i
>     <albert.tollkuci at gmail.com <mailto:albert.tollkuci at gmail.com>> wrote:
> 
>         Thank you Guillaume,
>         probably I'm missing something, but can't find any unset.
> 
>         Is there anything running before vcl_backend_response? I see the
>         cookie in the backend side and is not there anymore in
>         vcl_backend_response?
> 
>         Albert
> 
>         On Fri, Sep 2, 2016 at 10:53 AM, Guillaume Quintard
>         <guillaume at varnish-software.com
>         <mailto:guillaume at varnish-software.com>> wrote:
> 
>             Hi,
> 
>             Look at the varnishlog, there should be a trace of that
>             unset, once confirmed, find it in your vcl.
> 
>             Know that if you don't return from a subroutine, the
>             built-in vcl is executed ("locate builtin.vcl" to find a
>             commented version of it).
> 
> 
>             On Sep 1, 2016 12:59, "Albert Tollku?i"
>             <albert.tollkuci at gmail.com
>             <mailto:albert.tollkuci at gmail.com>> wrote:
> 
>                 Hello,
>                 I've started using Varnish a few months ago and my vcl
>                 files has started to get complicated. I'm serving 5
>                 different sites with the same varnish instance (using
>                 different backends) and this part is working.
> 
>                 Now I have some cases that for some sites I need to keep
>                 some cookies and cache different versions based on them.
>                 Basic scenario is to show the user a message only the
>                 first time he sees the page. Without varnish if the
>                 cookie is not there I set it and show the message. Next
>                 time the cookie is there and no message is shown.
> 
>                 With varnish my logic is to include the cookie in hash
>                 and cache two different versions. The challenge is to
>                 make sure cookies are set correctly and here I'm kind of
>                 stack. I've added some debug messages in my vcl and for
>                 some reason I'm not seeing the cookies I set in my
>                 server in vcl_backend_response. Inside the routine, I
>                 have the following code in the beginning:
> 
>                        # AT: DEBUG
>                        if(bereq.http.x-at-debug == "1") {
>                          std.log("AT DEBUG - bereq URL: " + bereq.url +
>                 ". Cookies: '" + bereq.http.Cookie + "'");
>                          std.log("AT DEBUG - beresp set-cookie: '" +
>                  beresp.http.set-cookie + "'");
>                        }
> 
>                 I set x-at-debug header in vcl_recv based on my IP, so I
>                 can track only my requests. In the server side I see the
>                 cookie "__IKUB_AL_Myikub_PopUp" being set, but in
>                 vcl_backend_response is not.
> 
>                 Is there anything in between that can strip the cookie?
> 
>                 Thank you,
>                 Albert
> 
> 
> 
>                 -- 
>                 Web: http://www.tollkuci.com <http://www.tollkuci.com/>
>                 Follow me on: LinkedIn
>                 <http://www.linkedin.com/in/alberttollkuci> Google+
>                 <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
>                 <https://www.facebook.com/albert.tollkuci> Twitter
>                 <https://twitter.com/AlbertTollkuci> Career 2.0
>                 <http://careers.stackoverflow.com/atollkuci>
> 
> 
>                 _______________________________________________
>                 varnish-misc mailing list
>                 varnish-misc at varnish-cache.org
>                 <mailto:varnish-misc at varnish-cache.org>
>                 https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>                 <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
> 
> 
> 
> 
>         -- 
>         Web: http://www.tollkuci.com <http://www.tollkuci.com/>
>         Follow me on: LinkedIn
>         <http://www.linkedin.com/in/alberttollkuci> Google+
>         <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
>         <https://www.facebook.com/albert.tollkuci> Twitter
>         <https://twitter.com/AlbertTollkuci> Career 2.0
>         <http://careers.stackoverflow.com/atollkuci>
>         ------------------------------------------------------------------------
> 
> 
> 
> 
>     -- 
>     Web: http://www.tollkuci.com <http://www.tollkuci.com/>
>     Follow me on: LinkedIn
>     <http://www.linkedin.com/in/alberttollkuci> Google+
>     <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
>     <https://www.facebook.com/albert.tollkuci> Twitter
>     <https://twitter.com/AlbertTollkuci> Career 2.0
>     <http://careers.stackoverflow.com/atollkuci>
>     ------------------------------------------------------------------------
> 
> 
> 
> 
> -- 
> Web: http://www.tollkuci.com <http://www.tollkuci.com/>
> Follow me on: LinkedIn
> <http://www.linkedin.com/in/alberttollkuci> Google+
> <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
> <https://www.facebook.com/albert.tollkuci> Twitter
> <https://twitter.com/AlbertTollkuci> Career 2.0
> <http://careers.stackoverflow.com/atollkuci>
> 
> 
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> 




From albert.tollkuci at gmail.com  Sat Sep  3 18:29:56 2016
From: albert.tollkuci at gmail.com (=?UTF-8?Q?Albert_Tollku=C3=A7i?=)
Date: Sat, 3 Sep 2016 20:29:56 +0200
Subject: Caching for some cookies?
In-Reply-To: <33062d91-f14b-2df6-a039-2f5ad33bc268@yahoo.es>
References: <CACqaJ6hqbnCUxeijhpU-zuH0iw8kkEr=TM21g9uMryv6=LM4gg@mail.gmail.com>
 <CACqaJ6gHEQNYbdOf=Bt7SLYDdTmxKH0nRa6dFO++VwH7rgQisg@mail.gmail.com>
 <33062d91-f14b-2df6-a039-2f5ad33bc268@yahoo.es>
Message-ID: <CACqaJ6g1AMuBUSd8QGve7BBoiTA7zZF1Oi36tGcnhRn6e4pOrA@mail.gmail.com>

Update: The modules require at least varnish 4.1 so I did upgrade to latest
version from github (varnish-trunk revision c99382d). However I'm still
getting an error in make:

In file included from /usr/local/include/varnish/common/common.h:39:0,
                 from /usr/local/include/varnish/cache/cache.h:39,
                 from vmod_softpurge.c:30:
vmod_softpurge.c: In function 'vmod_softpurge':
/usr/local/include/varnish/vqueue.h:540:39: error: 'struct objcore' has no
member named 'list'
 #define VTAILQ_NEXT(elm, field) ((elm)->field.vtqe_next)
                                       ^
/usr/local/include/varnish/vqueue.h:460:14: note: in expansion of macro
'VTAILQ_NEXT'
      (var) = VTAILQ_NEXT((var), field))
              ^
vmod_softpurge.c:63:2: note: in expansion of macro 'VTAILQ_FOREACH'
  VTAILQ_FOREACH(oc, &oh->objcs, list) {
  ^
vmod_softpurge.c:68:23: error: 'OC_EF_DYING' undeclared (first use in this
function)
   if (oc->exp_flags & OC_EF_DYING)
                       ^
vmod_softpurge.c:68:23: note: each undeclared identifier is reported only
once for each function it appears in
vmod_softpurge.c:85:27: error: 'struct objcore' has no member named 'exp'
   EXP_Rearm(oc, now, 0, oc->exp.grace, oc->exp.keep);
                           ^
vmod_softpurge.c:85:42: error: 'struct objcore' has no member named 'exp'
   EXP_Rearm(oc, now, 0, oc->exp.grace, oc->exp.keep);
                                          ^
Makefile:579: recipe for target 'vmod_softpurge.lo' failed

Stuck completely now...

On Sat, Sep 3, 2016 at 7:23 PM, Miguel Gonz?lez <miguel_3_gonzalez at yahoo.es>
wrote:

> Those vmods are for varnish open source version?
>
> ps: Why my messages are stored in the archives but are not getting
> through by email?
>
>
>
> On 09/03/16 11:16 AM, Albert Tollku?i wrote:
> > After digging more it seems I have to install varnish modules
> > from https://github.com/varnish/varnish-modules. The instructions to
> > install them are:
> >
> > sudo apt-get install libvarnishapi-dev || sudo yum install
> varnish-libs-devel
> > ./bootstrap   # If running from git.
> > ./configure
> > make
> > make check   # optional
> > sudo make install
> >
> > I was able to successfully perform the steps until make. When I try to
> > make I get the message:
> >
> > /make  all-recursive/
> > /make[1]: Entering directory
> > '/home/atollkuci/varnish-4.0.3/varnish-modules'/
> > /Making all in src/
> > /make[2]: Entering directory
> > '/home/atollkuci/varnish-4.0.3/varnish-modules/src'/
> > //bin/bash: PYTHON@: command not found/
> > /Makefile:803: recipe for target 'vcc_cookie_if.h' failed/
> > /make[2]: *** [vcc_cookie_if.h] Error 127/
> > /make[2]: Leaving directory
> > '/home/atollkuci/varnish-4.0.3/varnish-modules/src'/
> > /Makefile:485: recipe for target 'all-recursive' failed/
> > /make[1]: *** [all-recursive] Error 1/
> > /make[1]: Leaving directory '/home/atollkuci/varnish-4.0.
> 3/varnish-modules'/
> > /Makefile:396: recipe for target 'all' failed/
> > /make: *** [all] Error 2/
> >
> > It looks like it can't find python, but if I run from bash python or
> > python3 I get the python interpreter.
> >
> > Any idea how to proceed?
> >
> > I'm running Ubuntu 15.10 and varnish 4.0.3.
> >
> > Thank you,
> > Albert
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Sep 2, 2016 at 7:05 PM, Albert Tollku?i
> > <albert.tollkuci at gmail.com <mailto:albert.tollkuci at gmail.com>> wrote:
> >
> >     Sorry for the long post, but I'm stuck here. Below is the full log
> >     from varnishlog for a MISS request (so I can check what's happening
> >     with the backend). In bold there are the important bits:
> >
> >     - In the original request I have "*-   Link           req 294922
> >     restart*". Is the restart normal, because of the miss?
> >
> >     - In the backend response I can see the set cookie header: *-
> >     BerespHeader   Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu,
> >     24-Aug-2017 16:48:53 GMT; path=/*
> >     *
> >     *
> >     - My log doesn't show the set-cookie header (*-   VCL_Log        AT
> >     DEBUG - beresp set-cookie:
> >     'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'*). Is
> >     that because there are two set-cookie headers and it is showing only
> >     the first?
> >
> >     - Immediately after the log there's a line "*-   BerespUnset
> >      Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu, 24-Aug-2017
> >     16:48:53 GMT; path=/"*. It looks like this is responsible for
> >     un-setting the cookie, is that right?
> >
> >     - If so the cookie is striped between my first and second log and
> >     the code in between is the following:
> >
> >     /    if(beresp.http.set-cookie)/
> >     /    {/
> >     /      std.log("AT Vanish - mobile.ikub.al <http://mobile.ikub.al>
> >     original backend set cookie: '" + beresp.http.set-cookie + "'");/
> >     /    }/
> >     /
> >     /
> >     /    # AT: We have to keep all cookies starting with __IKUB/
> >     /    set beresp.http.set-cookie = ";" + beresp.http.set-cookie;/
> >     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
> >     "; +", ";");/
> >     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
> >     ";(__IKUB.*)=", "; \1=");/
> >     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
> >     ";[^ ][^;]*", "");/
> >     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
> >     "^[; ]+|[; ]+$", "");/
> >     /
> >     /
> >     /    # Clean empty cookies/
> >     /    if (beresp.http.set-cookie ~ "^\s*$") {/
> >     /      set beresp.http.set-cookie = "";/
> >     /    }/
> >     /
> >     /
> >     /    if (beresp.http.set-cookie == ""){/
> >     /      unset beresp.http.set-cookie;/
> >     /    }/
> >     /    else/
> >     /    {/
> >     /      std.log("AT Varnish - mobile.ikub.al <http://mobile.ikub.al>
> >     processed backend cookie: '" + beresp.http.set-cookie + "'");/
> >     /    }/
> >
> >     I have the same regex in the request processing and they're working,
> >     but maybe for some reason not working here...
> >
> >
> >     Full log from varnishlog:
> >
> >     *   << Request  >> 294921
> >     -   Begin          req 294920 rxreq
> >     -   Timestamp      Start: 1472834933.190575 0.000000 0.000000
> >     -   Timestamp      Req: 1472834933.190575 0.000000 0.000000
> >     -   ReqStart       185.158.1.35 5547
> >     -   ReqMethod      GET
> >     -   ReqURL         /Default2.aspx
> >     -   ReqProtocol    HTTP/1.1
> >     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
> >     -   ReqHeader      Connection: keep-alive
> >     -   ReqHeader      Pragma: no-cache
> >     -   ReqHeader      Cache-Control: no-cache
> >     -   ReqHeader      Upgrade-Insecure-Requests: 1
> >     -   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64;
> >     x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116
> >     Safari/537.36
> >     -   ReqHeader      Accept:
> >     text/html,application/xhtml+xml,application/xml;q=0.9,
> image/webp,*/*;q=0.8
> >     -   ReqHeader      Accept-Encoding: gzip, deflate, sdch
> >     -   ReqHeader      Accept-Language: en-US,en;q=0.8,sq;q=0.6
> >     -   ReqHeader      Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
> >     __utma=1.12079592.1472832572.1
> >     -   ReqHeader      X-Forwarded-For: 185.158.1.35
> >     -   VCL_call       RECV
> >     -   VCL_Log        AT DEBUG - recv URL: /Default2.aspx. Cookies:
> >     '__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.147
> >     -   ReqHeader      x-at-debug: 1
> >     -   ReqUnset       Host: mobile.ikub.al <http://mobile.ikub.al>
> >     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
> >     -   ReqURL         /Default2.aspx
> >     -   ReqUnset       Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
> >     __utma=1.12079592.1472832572.1
> >     -   ReqHeader      Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
> >     __utma=1.12079592.1472832572.1
> >     -   ReqUnset       Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
> >     __utma=1.12079592.1472832572.1
> >     -   ReqHeader      Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
> >     -   ReqUnset       Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
> >     -   ReqHeader      Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie:
> >     __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-
> HgsPe72zKoyugqygtg;
> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   VCL_return     purge
> >     -   ReqUnset       Accept-Encoding: gzip, deflate, sdch
> >     -   ReqHeader      Accept-Encoding: gzip
> >     -   VCL_call       HASH
> >     -   VCL_return     lookup
> >     -   VCL_call       PURGE
> >     -   ReqHeader      X-Purge: Yes
> >     -   VCL_return     restart
> >     -   Timestamp      Restart: 1472834933.190646 0.000071 0.000071
> >     *-   Link           req 294922 restart*
> >     -   End
> >
> >
> >
> >
> >     *   << BeReq    >> 294923
> >     -   Begin          bereq 294922 fetch
> >     -   Timestamp      Start: 1472834933.190706 0.000000 0.000000
> >     -   BereqMethod    GET
> >     -   BereqURL       /Default2.aspx
> >     -   BereqProtocol  HTTP/1.1
> >     -   BereqHeader    Pragma: no-cache
> >     -   BereqHeader    Upgrade-Insecure-Requests: 1
> >     -   BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64;
> >     x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116
> >     Safari/537.36
> >     -   BereqHeader    Accept:
> >     text/html,application/xhtml+xml,application/xml;q=0.9,
> image/webp,*/*;q=0.8
> >     -   BereqHeader    Accept-Language: en-US,en;q=0.8,sq;q=0.6
> >     -   BereqHeader    X-Forwarded-For: 185.158.1.35
> >     -   BereqHeader    Accept-Encoding: gzip
> >     -   BereqHeader    X-Purge: Yes
> >     -   BereqHeader    x-at-debug: 1
> >     -   BereqHeader    Host: mobile.ikub.al <http://mobile.ikub.al>
> >     -   BereqHeader    x-state: MISS
> >     -   BereqHeader    X-Varnish: 294923
> >     -   VCL_call       BACKEND_FETCH
> >     -   VCL_Log        AT DEBUG - fetch bereq URL: /Default2.aspx.
> >     Cookies: ''
> >     -   VCL_return     fetch
> >     -   Backend        43 ikubmobileserver
> >     ikubmobileserver(13.94.212.211,,80)
> >     -   Timestamp      Bereq: 1472834933.190737 0.000031 0.000031
> >     -   Timestamp      Beresp: 1472834933.830661 0.639955 0.639924
> >     -   BerespProtocol HTTP/1.1
> >     -   BerespStatus   200
> >     -   BerespReason   OK
> >     -   BerespHeader   Cache-Control: private
> >     -   BerespHeader   Content-Type: text/html; charset=utf-8
> >     -   BerespHeader   Content-Encoding: gzip
> >     -   BerespHeader   Vary: Accept-Encoding
> >     -   BerespHeader   Server: Microsoft-IIS/8.5
> >     -   BerespHeader   X-AspNet-Version: 2.0.50727
> >     -   BerespHeader   Set-Cookie:
> >     ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
> >     *-   BerespHeader   Set-Cookie: __IKUB_AL_Myikub_PopUp=1;
> >     expires=Thu, 24-Aug-2017 16:48:53 GMT; path=/*
> >     -   BerespHeader   X-Powered-By: ASP.NET <http://ASP.NET>
> >     -   BerespHeader   Date: Fri, 02 Sep 2016 16:48:53 GMT
> >     -   BerespHeader   Content-Length: 52233
> >     -   TTL            RFC 120 -1 -1 1472834934 1472834934 1472834933 0 0
> >     -   VCL_call       BACKEND_RESPONSE
> >     -   VCL_Log        AT DEBUG - bereq URL: /Default2.aspx. Cookies: ''
> >     *-   VCL_Log        AT DEBUG - beresp set-cookie:
> >     'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'*
> >     -   VCL_Log        AT Vanish - mobile.ikub.al
> >     <http://mobile.ikub.al> original backend set cookie:
> >     'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'
> >     -   BerespUnset    Set-Cookie:
> >     ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
> >     *-   BerespUnset    Set-Cookie: __IKUB_AL_Myikub_PopUp=1;
> >     expires=Thu, 24-Aug-2017 16:48:53 GMT; path=/*
> >     -   BerespHeader   set-cookie:
> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
> >     -   BerespUnset    set-cookie:
> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
> >     -   BerespHeader   set-cookie:
> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
> >     -   BerespUnset    set-cookie:
> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
> >     -   BerespHeader   set-cookie:
> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
> >     -   BerespUnset    set-cookie:
> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
> >     -   BerespHeader   set-cookie:
> >     -   BerespUnset    set-cookie:
> >     -   BerespHeader   set-cookie:
> >     -   BerespUnset    set-cookie:
> >     -   BerespHeader   set-cookie:
> >     -   BerespUnset    set-cookie:
> >     -   TTL            VCL 900 10 0 1472834934
> >     -   VCL_Log        AT DEBUG - bereq2 URL: /Default2.aspx. Cookies: ''
> >     -   VCL_Log        AT DEBUG - beresp2 set-cookie: ''
> >     -   TTL            VCL 900 21600 0 1472834934
> >     -   VCL_Log        AT DEBUG - bereq3 URL: /Default2.aspx. Cookies: ''
> >     -   VCL_Log        AT DEBUG - beresp3 set-cookie: ''
> >     -   VCL_return     deliver
> >     -   Storage        malloc s0
> >     -   ObjProtocol    HTTP/1.1
> >     -   ObjStatus      200
> >     -   ObjReason      OK
> >     -   ObjHeader      Cache-Control: private
> >     -   ObjHeader      Content-Type: text/html; charset=utf-8
> >     -   ObjHeader      Content-Encoding: gzip
> >     -   ObjHeader      Vary: Accept-Encoding
> >     -   ObjHeader      Server: Microsoft-IIS/8.5
> >     -   ObjHeader      X-AspNet-Version: 2.0.50727
> >     -   ObjHeader      X-Powered-By: ASP.NET <http://ASP.NET>
> >     -   ObjHeader      Date: Fri, 02 Sep 2016 16:48:53 GMT
> >     -   ObjHeader      Content-Length: 52233
> >     -   Fetch_Body     3 length stream
> >     -   Gzip           u F - 52233 194069 80 80 417793
> >     -   BackendReuse   43 ikubmobileserver(13.94.212.211,,80)
> >     -   Timestamp      BerespBody: 1472834933.831743 0.641037 0.001082
> >     -   Length         52233
> >     -   BereqAcct      473 0 473 428 52233 52661
> >     -   End
> >
> >     *   << Request  >> 294922
> >     -   Begin          req 294921 restart
> >     -   Timestamp      Start: 1472834933.190646 0.000071 0.000000
> >     -   ReqStart       185.158.1.35 5547
> >     -   ReqMethod      GET
> >     -   ReqURL         /Default2.aspx
> >     -   ReqProtocol    HTTP/1.1
> >     -   ReqHeader      Connection: keep-alive
> >     -   ReqHeader      Pragma: no-cache
> >     -   ReqHeader      Cache-Control: no-cache
> >     -   ReqHeader      Upgrade-Insecure-Requests: 1
> >     -   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64;
> >     x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116
> >     Safari/537.36
> >     -   ReqHeader      Accept:
> >     text/html,application/xhtml+xml,application/xml;q=0.9,
> image/webp,*/*;q=0.8
> >     -   ReqHeader      Accept-Language: en-US,en;q=0.8,sq;q=0.6
> >     -   ReqHeader      X-Forwarded-For: 185.158.1.35
> >     -   ReqHeader      x-at-debug: 1
> >     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Accept-Encoding: gzip
> >     -   ReqHeader      X-Purge: Yes
> >     -   VCL_call       RECV
> >     -   VCL_Log        AT DEBUG - recv URL: /Default2.aspx. Cookies:
> >     'OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d; '
> >     -   ReqUnset       x-at-debug: 1
> >     -   ReqHeader      x-at-debug: 1
> >     -   ReqUnset       Host: mobile.ikub.al <http://mobile.ikub.al>
> >     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
> >     -   ReqURL         /Default2.aspx
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   VCL_Log        AT DEBUG - recv2 URL: /Default2.aspx. Cookies:
> >     'OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
> >     __auc=c970f060156eba988423264f56d; '
> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: ;OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie: ;OX_plg=swf|shk|pm;
> >     __asc=c970f060156eba988423264f56d; __auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie:
> >     ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie:
> >     ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie:
> >     ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=
> c970f060156eba988423264f56d;
> >     -   ReqUnset       Cookie:
> >     ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=
> c970f060156eba988423264f56d;
> >     -   ReqHeader      Cookie: ;
> >     -   ReqUnset       Cookie: ;
> >     -   ReqHeader      Cookie:
> >     -   ReqUnset       Cookie:
> >     -   ReqHeader      Cookie:
> >     -   ReqUnset       Cookie:
> >     -   VCL_Log        AT DEBUG - recv3 URL: /Default2.aspx. Cookies: ''
> >     -   VCL_return     hash
> >     -   VCL_call       HASH
> >     -   VCL_return     lookup
> >     -   Debug          "XXXX MISS"
> >     -   VCL_call       MISS
> >     -   ReqHeader      x-state: MISS
> >     -   VCL_return     fetch
> >     -   Link           bereq 294923 fetch
> >     -   Timestamp      Fetch: 1472834933.831768 0.641192 0.641122
> >     -   RespProtocol   HTTP/1.1
> >     -   RespStatus     200
> >     -   RespReason     OK
> >     -   RespHeader     Cache-Control: private
> >     -   RespHeader     Content-Type: text/html; charset=utf-8
> >     -   RespHeader     Content-Encoding: gzip
> >     -   RespHeader     Vary: Accept-Encoding
> >     -   RespHeader     Server: Microsoft-IIS/8.5
> >     -   RespHeader     X-AspNet-Version: 2.0.50727
> >     -   RespHeader     X-Powered-By: ASP.NET <http://ASP.NET>
> >     -   RespHeader     Date: Fri, 02 Sep 2016 16:48:53 GMT
> >     -   RespHeader     Content-Length: 52233
> >     -   RespHeader     X-Varnish: 294922
> >     -   RespHeader     Age: 0
> >     -   RespHeader     Via: 1.1 varnish-v4
> >     -   VCL_call       DELIVER
> >     -   RespHeader     X-Cache: MISS
> >     -   RespHeader     X-Cache-Hits: 0
> >     -   RespUnset      X-Powered-By: ASP.NET <http://ASP.NET>
> >     -   RespUnset      Server: Microsoft-IIS/8.5
> >     -   RespUnset      X-Varnish: 294922
> >     -   RespUnset      Via: 1.1 varnish-v4
> >     -   RespUnset      X-AspNet-Version: 2.0.50727
> >     -   VCL_Log        AT DEBUG - deliver URL: /Default2.aspx. Cookies:
> ''
> >     -   VCL_Log        AT DEBUG - deliver set-cookie: ''
> >     -   VCL_return     deliver
> >     -   Timestamp      Process: 1472834933.831794 0.641218 0.000026
> >     -   Debug          "RES_MODE 2"
> >     -   RespHeader     Connection: keep-alive
> >     -   RespHeader     Accept-Ranges: bytes
> >     -   Timestamp      Resp: 1472834933.831830 0.641255 0.000036
> >     -   Debug          "XXX REF 2"
> >     -   ReqAcct        828 0 828 276 52233 52509
> >     -   End
> >
> >
> >     On Fri, Sep 2, 2016 at 6:28 PM, Albert Tollku?i
> >     <albert.tollkuci at gmail.com <mailto:albert.tollkuci at gmail.com>>
> wrote:
> >
> >         Thank you Guillaume,
> >         probably I'm missing something, but can't find any unset.
> >
> >         Is there anything running before vcl_backend_response? I see the
> >         cookie in the backend side and is not there anymore in
> >         vcl_backend_response?
> >
> >         Albert
> >
> >         On Fri, Sep 2, 2016 at 10:53 AM, Guillaume Quintard
> >         <guillaume at varnish-software.com
> >         <mailto:guillaume at varnish-software.com>> wrote:
> >
> >             Hi,
> >
> >             Look at the varnishlog, there should be a trace of that
> >             unset, once confirmed, find it in your vcl.
> >
> >             Know that if you don't return from a subroutine, the
> >             built-in vcl is executed ("locate builtin.vcl" to find a
> >             commented version of it).
> >
> >
> >             On Sep 1, 2016 12:59, "Albert Tollku?i"
> >             <albert.tollkuci at gmail.com
> >             <mailto:albert.tollkuci at gmail.com>> wrote:
> >
> >                 Hello,
> >                 I've started using Varnish a few months ago and my vcl
> >                 files has started to get complicated. I'm serving 5
> >                 different sites with the same varnish instance (using
> >                 different backends) and this part is working.
> >
> >                 Now I have some cases that for some sites I need to keep
> >                 some cookies and cache different versions based on them.
> >                 Basic scenario is to show the user a message only the
> >                 first time he sees the page. Without varnish if the
> >                 cookie is not there I set it and show the message. Next
> >                 time the cookie is there and no message is shown.
> >
> >                 With varnish my logic is to include the cookie in hash
> >                 and cache two different versions. The challenge is to
> >                 make sure cookies are set correctly and here I'm kind of
> >                 stack. I've added some debug messages in my vcl and for
> >                 some reason I'm not seeing the cookies I set in my
> >                 server in vcl_backend_response. Inside the routine, I
> >                 have the following code in the beginning:
> >
> >                        # AT: DEBUG
> >                        if(bereq.http.x-at-debug == "1") {
> >                          std.log("AT DEBUG - bereq URL: " + bereq.url +
> >                 ". Cookies: '" + bereq.http.Cookie + "'");
> >                          std.log("AT DEBUG - beresp set-cookie: '" +
> >                  beresp.http.set-cookie + "'");
> >                        }
> >
> >                 I set x-at-debug header in vcl_recv based on my IP, so I
> >                 can track only my requests. In the server side I see the
> >                 cookie "__IKUB_AL_Myikub_PopUp" being set, but in
> >                 vcl_backend_response is not.
> >
> >                 Is there anything in between that can strip the cookie?
> >
> >                 Thank you,
> >                 Albert
> >
> >
> >
> >                 --
> >                 Web: http://www.tollkuci.com <http://www.tollkuci.com/>
> >                 Follow me on: LinkedIn
> >                 <http://www.linkedin.com/in/alberttollkuci> Google+
> >                 <https://plus.google.com/+AlbertTollku%C3%A7i/posts>
> Facebook
> >                 <https://www.facebook.com/albert.tollkuci> Twitter
> >                 <https://twitter.com/AlbertTollkuci> Career 2.0
> >                 <http://careers.stackoverflow.com/atollkuci>
> >
> >
> >                 _______________________________________________
> >                 varnish-misc mailing list
> >                 varnish-misc at varnish-cache.org
> >                 <mailto:varnish-misc at varnish-cache.org>
> >                 https://www.varnish-cache.org/lists/mailman/listinfo/
> varnish-misc
> >                 <https://www.varnish-cache.org/lists/mailman/listinfo/
> varnish-misc>
> >
> >
> >
> >
> >         --
> >         Web: http://www.tollkuci.com <http://www.tollkuci.com/>
> >         Follow me on: LinkedIn
> >         <http://www.linkedin.com/in/alberttollkuci> Google+
> >         <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
> >         <https://www.facebook.com/albert.tollkuci> Twitter
> >         <https://twitter.com/AlbertTollkuci> Career 2.0
> >         <http://careers.stackoverflow.com/atollkuci>
> >         ------------------------------------------------------------
> ------------
> >
> >
> >
> >
> >     --
> >     Web: http://www.tollkuci.com <http://www.tollkuci.com/>
> >     Follow me on: LinkedIn
> >     <http://www.linkedin.com/in/alberttollkuci> Google+
> >     <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
> >     <https://www.facebook.com/albert.tollkuci> Twitter
> >     <https://twitter.com/AlbertTollkuci> Career 2.0
> >     <http://careers.stackoverflow.com/atollkuci>
> >     ------------------------------------------------------------
> ------------
> >
> >
> >
> >
> > --
> > Web: http://www.tollkuci.com <http://www.tollkuci.com/>
> > Follow me on: LinkedIn
> > <http://www.linkedin.com/in/alberttollkuci> Google+
> > <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
> > <https://www.facebook.com/albert.tollkuci> Twitter
> > <https://twitter.com/AlbertTollkuci> Career 2.0
> > <http://careers.stackoverflow.com/atollkuci>
> >
> >
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >
>
>


-- 
Web: http://www.tollkuci.com
Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
<https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
<https://www.facebook.com/albert.tollkuci> Twitter
<https://twitter.com/AlbertTollkuci> Career 2.0
<http://careers.stackoverflow.com/atollkuci>
------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160903/c007f945/attachment.html>

From albert.tollkuci at gmail.com  Sun Sep  4 19:45:02 2016
From: albert.tollkuci at gmail.com (=?UTF-8?Q?Albert_Tollku=C3=A7i?=)
Date: Sun, 04 Sep 2016 19:45:02 +0000
Subject: Caching for some cookies?
In-Reply-To: <CACqaJ6g1AMuBUSd8QGve7BBoiTA7zZF1Oi36tGcnhRn6e4pOrA@mail.gmail.com>
References: <CACqaJ6hqbnCUxeijhpU-zuH0iw8kkEr=TM21g9uMryv6=LM4gg@mail.gmail.com>
 <CACqaJ6gHEQNYbdOf=Bt7SLYDdTmxKH0nRa6dFO++VwH7rgQisg@mail.gmail.com>
 <33062d91-f14b-2df6-a039-2f5ad33bc268@yahoo.es>
 <CACqaJ6g1AMuBUSd8QGve7BBoiTA7zZF1Oi36tGcnhRn6e4pOrA@mail.gmail.com>
Message-ID: <CACqaJ6iYXC5==2pErOeZid1oxn9MWVAok2V05qKSJex9=PisHw@mail.gmail.com>

Finally everything is resolved. I should have read more carefully the
documentation:

- Official varnish modules (from https://github.com/varnish/varnish-modules)
work only with varnish 4.1

- There's another repository (from
https://github.com/nigoroll/varnish-modules/) that works with the latest
version of varnish (from https://github.com/varnishcache/varnish-cache)

The original problem was because beresp.http.set-cookie contains *only the
first set-cookie header*. I believe this should be stressed in
documentation and maybe implement something like an array because it's a
bit confusing. However for anyone stuck with something similar, you'll need
the header module form varnish modules.

Albert

On Sat, Sep 3, 2016 at 8:29 PM Albert Tollku?i <albert.tollkuci at gmail.com>
wrote:

> Update: The modules require at least varnish 4.1 so I did upgrade to
> latest version from github (varnish-trunk revision c99382d). However I'm
> still getting an error in make:
>
> In file included from /usr/local/include/varnish/common/common.h:39:0,
>                  from /usr/local/include/varnish/cache/cache.h:39,
>                  from vmod_softpurge.c:30:
> vmod_softpurge.c: In function 'vmod_softpurge':
> /usr/local/include/varnish/vqueue.h:540:39: error: 'struct objcore' has no
> member named 'list'
>  #define VTAILQ_NEXT(elm, field) ((elm)->field.vtqe_next)
>                                        ^
> /usr/local/include/varnish/vqueue.h:460:14: note: in expansion of macro
> 'VTAILQ_NEXT'
>       (var) = VTAILQ_NEXT((var), field))
>               ^
> vmod_softpurge.c:63:2: note: in expansion of macro 'VTAILQ_FOREACH'
>   VTAILQ_FOREACH(oc, &oh->objcs, list) {
>   ^
> vmod_softpurge.c:68:23: error: 'OC_EF_DYING' undeclared (first use in this
> function)
>    if (oc->exp_flags & OC_EF_DYING)
>                        ^
> vmod_softpurge.c:68:23: note: each undeclared identifier is reported only
> once for each function it appears in
> vmod_softpurge.c:85:27: error: 'struct objcore' has no member named 'exp'
>    EXP_Rearm(oc, now, 0, oc->exp.grace, oc->exp.keep);
>                            ^
> vmod_softpurge.c:85:42: error: 'struct objcore' has no member named 'exp'
>    EXP_Rearm(oc, now, 0, oc->exp.grace, oc->exp.keep);
>                                           ^
> Makefile:579: recipe for target 'vmod_softpurge.lo' failed
>
> Stuck completely now...
>
> On Sat, Sep 3, 2016 at 7:23 PM, Miguel Gonz?lez <
> miguel_3_gonzalez at yahoo.es> wrote:
>
>> Those vmods are for varnish open source version?
>>
>> ps: Why my messages are stored in the archives but are not getting
>> through by email?
>>
>>
>>
>> On 09/03/16 11:16 AM, Albert Tollku?i wrote:
>> > After digging more it seems I have to install varnish modules
>> > from https://github.com/varnish/varnish-modules. The instructions to
>> > install them are:
>> >
>> > sudo apt-get install libvarnishapi-dev || sudo yum install
>> varnish-libs-devel
>> > ./bootstrap   # If running from git.
>> > ./configure
>> > make
>> > make check   # optional
>> > sudo make install
>> >
>> > I was able to successfully perform the steps until make. When I try to
>> > make I get the message:
>> >
>> > /make  all-recursive/
>> > /make[1]: Entering directory
>> > '/home/atollkuci/varnish-4.0.3/varnish-modules'/
>> > /Making all in src/
>> > /make[2]: Entering directory
>> > '/home/atollkuci/varnish-4.0.3/varnish-modules/src'/
>> > //bin/bash: PYTHON@: command not found/
>> > /Makefile:803: recipe for target 'vcc_cookie_if.h' failed/
>> > /make[2]: *** [vcc_cookie_if.h] Error 127/
>> > /make[2]: Leaving directory
>> > '/home/atollkuci/varnish-4.0.3/varnish-modules/src'/
>> > /Makefile:485: recipe for target 'all-recursive' failed/
>> > /make[1]: *** [all-recursive] Error 1/
>> > /make[1]: Leaving directory
>> '/home/atollkuci/varnish-4.0.3/varnish-modules'/
>> > /Makefile:396: recipe for target 'all' failed/
>> > /make: *** [all] Error 2/
>> >
>> > It looks like it can't find python, but if I run from bash python or
>> > python3 I get the python interpreter.
>> >
>> > Any idea how to proceed?
>> >
>> > I'm running Ubuntu 15.10 and varnish 4.0.3.
>> >
>> > Thank you,
>> > Albert
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Sep 2, 2016 at 7:05 PM, Albert Tollku?i
>> > <albert.tollkuci at gmail.com <mailto:albert.tollkuci at gmail.com>> wrote:
>> >
>> >     Sorry for the long post, but I'm stuck here. Below is the full log
>> >     from varnishlog for a MISS request (so I can check what's happening
>> >     with the backend). In bold there are the important bits:
>> >
>> >     - In the original request I have "*-   Link           req 294922
>> >     restart*". Is the restart normal, because of the miss?
>> >
>> >     - In the backend response I can see the set cookie header: *-
>> >     BerespHeader   Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu,
>> >     24-Aug-2017 16:48:53 GMT; path=/*
>> >     *
>> >     *
>> >     - My log doesn't show the set-cookie header (*-   VCL_Log        AT
>> >     DEBUG - beresp set-cookie:
>> >     'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'*). Is
>> >     that because there are two set-cookie headers and it is showing only
>> >     the first?
>> >
>> >     - Immediately after the log there's a line "*-   BerespUnset
>> >      Set-Cookie: __IKUB_AL_Myikub_PopUp=1; expires=Thu, 24-Aug-2017
>> >     16:48:53 GMT; path=/"*. It looks like this is responsible for
>> >     un-setting the cookie, is that right?
>> >
>> >     - If so the cookie is striped between my first and second log and
>> >     the code in between is the following:
>> >
>> >     /    if(beresp.http.set-cookie)/
>> >     /    {/
>> >     /      std.log("AT Vanish - mobile.ikub.al <http://mobile.ikub.al>
>> >     original backend set cookie: '" + beresp.http.set-cookie + "'");/
>> >     /    }/
>> >     /
>> >     /
>> >     /    # AT: We have to keep all cookies starting with __IKUB/
>> >     /    set beresp.http.set-cookie = ";" + beresp.http.set-cookie;/
>> >     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
>> >     "; +", ";");/
>> >     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
>> >     ";(__IKUB.*)=", "; \1=");/
>> >     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
>> >     ";[^ ][^;]*", "");/
>> >     /    set beresp.http.set-cookie = regsuball(beresp.http.set-cookie,
>> >     "^[; ]+|[; ]+$", "");/
>> >     /
>> >     /
>> >     /    # Clean empty cookies/
>> >     /    if (beresp.http.set-cookie ~ "^\s*$") {/
>> >     /      set beresp.http.set-cookie = "";/
>> >     /    }/
>> >     /
>> >     /
>> >     /    if (beresp.http.set-cookie == ""){/
>> >     /      unset beresp.http.set-cookie;/
>> >     /    }/
>> >     /    else/
>> >     /    {/
>> >     /      std.log("AT Varnish - mobile.ikub.al <http://mobile.ikub.al>
>> >     processed backend cookie: '" + beresp.http.set-cookie + "'");/
>> >     /    }/
>> >
>> >     I have the same regex in the request processing and they're working,
>> >     but maybe for some reason not working here...
>> >
>> >
>> >     Full log from varnishlog:
>> >
>> >     *   << Request  >> 294921
>> >     -   Begin          req 294920 rxreq
>> >     -   Timestamp      Start: 1472834933.190575 0.000000 0.000000
>> >     -   Timestamp      Req: 1472834933.190575 0.000000 0.000000
>> >     -   ReqStart       185.158.1.35 5547
>> >     -   ReqMethod      GET
>> >     -   ReqURL         /Default2.aspx
>> >     -   ReqProtocol    HTTP/1.1
>> >     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
>> >     -   ReqHeader      Connection: keep-alive
>> >     -   ReqHeader      Pragma: no-cache
>> >     -   ReqHeader      Cache-Control: no-cache
>> >     -   ReqHeader      Upgrade-Insecure-Requests: 1
>> >     -   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64;
>> >     x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116
>> >     Safari/537.36
>> >     -   ReqHeader      Accept:
>> >
>>  text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
>> >     -   ReqHeader      Accept-Encoding: gzip, deflate, sdch
>> >     -   ReqHeader      Accept-Language: en-US,en;q=0.8,sq;q=0.6
>> >     -   ReqHeader      Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
>> >     __utma=1.12079592.1472832572.1
>> >     -   ReqHeader      X-Forwarded-For: 185.158.1.35
>> >     -   VCL_call       RECV
>> >     -   VCL_Log        AT DEBUG - recv URL: /Default2.aspx. Cookies:
>> >
>>  '__gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.147
>> >     -   ReqHeader      x-at-debug: 1
>> >     -   ReqUnset       Host: mobile.ikub.al <http://mobile.ikub.al>
>> >     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
>> >     -   ReqURL         /Default2.aspx
>> >     -   ReqUnset       Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
>> >     __utma=1.12079592.1472832572.1
>> >     -   ReqHeader      Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
>> >     __utma=1.12079592.1472832572.1
>> >     -   ReqUnset       Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     _gat=1; __utmt=1; OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
>> >     __utma=1.12079592.1472832572.1
>> >     -   ReqHeader      Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
>> >     -   ReqUnset       Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d; _ga=GA1.2.12079592.1472832572;
>> >     -   ReqHeader      Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     _gat=1; OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie:
>> >
>>  __gads=ID=c8c7e856e0927271:T=1472832575:S=ALNI_Mbqww5QA6W-HgsPe72zKoyugqygtg;
>> >     OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   VCL_return     purge
>> >     -   ReqUnset       Accept-Encoding: gzip, deflate, sdch
>> >     -   ReqHeader      Accept-Encoding: gzip
>> >     -   VCL_call       HASH
>> >     -   VCL_return     lookup
>> >     -   VCL_call       PURGE
>> >     -   ReqHeader      X-Purge: Yes
>> >     -   VCL_return     restart
>> >     -   Timestamp      Restart: 1472834933.190646 0.000071 0.000071
>> >     *-   Link           req 294922 restart*
>> >     -   End
>> >
>> >
>> >
>> >
>> >     *   << BeReq    >> 294923
>> >     -   Begin          bereq 294922 fetch
>> >     -   Timestamp      Start: 1472834933.190706 0.000000 0.000000
>> >     -   BereqMethod    GET
>> >     -   BereqURL       /Default2.aspx
>> >     -   BereqProtocol  HTTP/1.1
>> >     -   BereqHeader    Pragma: no-cache
>> >     -   BereqHeader    Upgrade-Insecure-Requests: 1
>> >     -   BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64;
>> >     x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116
>> >     Safari/537.36
>> >     -   BereqHeader    Accept:
>> >
>>  text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
>> >     -   BereqHeader    Accept-Language: en-US,en;q=0.8,sq;q=0.6
>> >     -   BereqHeader    X-Forwarded-For: 185.158.1.35
>> >     -   BereqHeader    Accept-Encoding: gzip
>> >     -   BereqHeader    X-Purge: Yes
>> >     -   BereqHeader    x-at-debug: 1
>> >     -   BereqHeader    Host: mobile.ikub.al <http://mobile.ikub.al>
>> >     -   BereqHeader    x-state: MISS
>> >     -   BereqHeader    X-Varnish: 294923
>> >     -   VCL_call       BACKEND_FETCH
>> >     -   VCL_Log        AT DEBUG - fetch bereq URL: /Default2.aspx.
>> >     Cookies: ''
>> >     -   VCL_return     fetch
>> >     -   Backend        43 ikubmobileserver
>> >     ikubmobileserver(13.94.212.211,,80)
>> >     -   Timestamp      Bereq: 1472834933.190737 0.000031 0.000031
>> >     -   Timestamp      Beresp: 1472834933.830661 0.639955 0.639924
>> >     -   BerespProtocol HTTP/1.1
>> >     -   BerespStatus   200
>> >     -   BerespReason   OK
>> >     -   BerespHeader   Cache-Control: private
>> >     -   BerespHeader   Content-Type: text/html; charset=utf-8
>> >     -   BerespHeader   Content-Encoding: gzip
>> >     -   BerespHeader   Vary: Accept-Encoding
>> >     -   BerespHeader   Server: Microsoft-IIS/8.5
>> >     -   BerespHeader   X-AspNet-Version: 2.0.50727
>> >     -   BerespHeader   Set-Cookie:
>> >     ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
>> >     *-   BerespHeader   Set-Cookie: __IKUB_AL_Myikub_PopUp=1;
>> >     expires=Thu, 24-Aug-2017 16:48:53 GMT; path=/*
>> >     -   BerespHeader   X-Powered-By: ASP.NET <http://ASP.NET>
>> >     -   BerespHeader   Date: Fri, 02 Sep 2016 16:48:53 GMT
>> >     -   BerespHeader   Content-Length: 52233
>> >     -   TTL            RFC 120 -1 -1 1472834934 1472834934 1472834933 0
>> 0
>> >     -   VCL_call       BACKEND_RESPONSE
>> >     -   VCL_Log        AT DEBUG - bereq URL: /Default2.aspx. Cookies: ''
>> >     *-   VCL_Log        AT DEBUG - beresp set-cookie:
>> >     'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'*
>> >     -   VCL_Log        AT Vanish - mobile.ikub.al
>> >     <http://mobile.ikub.al> original backend set cookie:
>> >     'ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly'
>> >     -   BerespUnset    Set-Cookie:
>> >     ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
>> >     *-   BerespUnset    Set-Cookie: __IKUB_AL_Myikub_PopUp=1;
>> >     expires=Thu, 24-Aug-2017 16:48:53 GMT; path=/*
>> >     -   BerespHeader   set-cookie:
>> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
>> >     -   BerespUnset    set-cookie:
>> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w; path=/; HttpOnly
>> >     -   BerespHeader   set-cookie:
>> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
>> >     -   BerespUnset    set-cookie:
>> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
>> >     -   BerespHeader   set-cookie:
>> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
>> >     -   BerespUnset    set-cookie:
>> >     ;ASP.NET_SessionId=pkkav5uflq02es45dktlo03w;path=/;HttpOnly
>> >     -   BerespHeader   set-cookie:
>> >     -   BerespUnset    set-cookie:
>> >     -   BerespHeader   set-cookie:
>> >     -   BerespUnset    set-cookie:
>> >     -   BerespHeader   set-cookie:
>> >     -   BerespUnset    set-cookie:
>> >     -   TTL            VCL 900 10 0 1472834934
>> >     -   VCL_Log        AT DEBUG - bereq2 URL: /Default2.aspx. Cookies:
>> ''
>> >     -   VCL_Log        AT DEBUG - beresp2 set-cookie: ''
>> >     -   TTL            VCL 900 21600 0 1472834934
>> >     -   VCL_Log        AT DEBUG - bereq3 URL: /Default2.aspx. Cookies:
>> ''
>> >     -   VCL_Log        AT DEBUG - beresp3 set-cookie: ''
>> >     -   VCL_return     deliver
>> >     -   Storage        malloc s0
>> >     -   ObjProtocol    HTTP/1.1
>> >     -   ObjStatus      200
>> >     -   ObjReason      OK
>> >     -   ObjHeader      Cache-Control: private
>> >     -   ObjHeader      Content-Type: text/html; charset=utf-8
>> >     -   ObjHeader      Content-Encoding: gzip
>> >     -   ObjHeader      Vary: Accept-Encoding
>> >     -   ObjHeader      Server: Microsoft-IIS/8.5
>> >     -   ObjHeader      X-AspNet-Version: 2.0.50727
>> >     -   ObjHeader      X-Powered-By: ASP.NET <http://ASP.NET>
>> >     -   ObjHeader      Date: Fri, 02 Sep 2016 16:48:53 GMT
>> >     -   ObjHeader      Content-Length: 52233
>> >     -   Fetch_Body     3 length stream
>> >     -   Gzip           u F - 52233 194069 80 80 417793
>> >     -   BackendReuse   43 ikubmobileserver(13.94.212.211,,80)
>> >     -   Timestamp      BerespBody: 1472834933.831743 0.641037 0.001082
>> >     -   Length         52233
>> >     -   BereqAcct      473 0 473 428 52233 52661
>> >     -   End
>> >
>> >     *   << Request  >> 294922
>> >     -   Begin          req 294921 restart
>> >     -   Timestamp      Start: 1472834933.190646 0.000071 0.000000
>> >     -   ReqStart       185.158.1.35 5547
>> >     -   ReqMethod      GET
>> >     -   ReqURL         /Default2.aspx
>> >     -   ReqProtocol    HTTP/1.1
>> >     -   ReqHeader      Connection: keep-alive
>> >     -   ReqHeader      Pragma: no-cache
>> >     -   ReqHeader      Cache-Control: no-cache
>> >     -   ReqHeader      Upgrade-Insecure-Requests: 1
>> >     -   ReqHeader      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64;
>> >     x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116
>> >     Safari/537.36
>> >     -   ReqHeader      Accept:
>> >
>>  text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
>> >     -   ReqHeader      Accept-Language: en-US,en;q=0.8,sq;q=0.6
>> >     -   ReqHeader      X-Forwarded-For: 185.158.1.35
>> >     -   ReqHeader      x-at-debug: 1
>> >     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Accept-Encoding: gzip
>> >     -   ReqHeader      X-Purge: Yes
>> >     -   VCL_call       RECV
>> >     -   VCL_Log        AT DEBUG - recv URL: /Default2.aspx. Cookies:
>> >     'OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d; '
>> >     -   ReqUnset       x-at-debug: 1
>> >     -   ReqHeader      x-at-debug: 1
>> >     -   ReqUnset       Host: mobile.ikub.al <http://mobile.ikub.al>
>> >     -   ReqHeader      Host: mobile.ikub.al <http://mobile.ikub.al>
>> >     -   ReqURL         /Default2.aspx
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   VCL_Log        AT DEBUG - recv2 URL: /Default2.aspx. Cookies:
>> >     'OX_plg=swf|shk|pm; __asc=c970f060156eba988423264f56d;
>> >     __auc=c970f060156eba988423264f56d; '
>> >     -   ReqUnset       Cookie: OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: ;OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie: ;OX_plg=swf|shk|pm;
>> >     __asc=c970f060156eba988423264f56d;
>> __auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie:
>> >
>>  ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie:
>> >
>>  ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie:
>> >
>>  ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
>> >     -   ReqUnset       Cookie:
>> >
>>  ;OX_plg=swf|shk|pm;__asc=c970f060156eba988423264f56d;__auc=c970f060156eba988423264f56d;
>> >     -   ReqHeader      Cookie: ;
>> >     -   ReqUnset       Cookie: ;
>> >     -   ReqHeader      Cookie:
>> >     -   ReqUnset       Cookie:
>> >     -   ReqHeader      Cookie:
>> >     -   ReqUnset       Cookie:
>> >     -   VCL_Log        AT DEBUG - recv3 URL: /Default2.aspx. Cookies: ''
>> >     -   VCL_return     hash
>> >     -   VCL_call       HASH
>> >     -   VCL_return     lookup
>> >     -   Debug          "XXXX MISS"
>> >     -   VCL_call       MISS
>> >     -   ReqHeader      x-state: MISS
>> >     -   VCL_return     fetch
>> >     -   Link           bereq 294923 fetch
>> >     -   Timestamp      Fetch: 1472834933.831768 0.641192 0.641122
>> >     -   RespProtocol   HTTP/1.1
>> >     -   RespStatus     200
>> >     -   RespReason     OK
>> >     -   RespHeader     Cache-Control: private
>> >     -   RespHeader     Content-Type: text/html; charset=utf-8
>> >     -   RespHeader     Content-Encoding: gzip
>> >     -   RespHeader     Vary: Accept-Encoding
>> >     -   RespHeader     Server: Microsoft-IIS/8.5
>> >     -   RespHeader     X-AspNet-Version: 2.0.50727
>> >     -   RespHeader     X-Powered-By: ASP.NET <http://ASP.NET>
>> >     -   RespHeader     Date: Fri, 02 Sep 2016 16:48:53 GMT
>> >     -   RespHeader     Content-Length: 52233
>> >     -   RespHeader     X-Varnish: 294922
>> >     -   RespHeader     Age: 0
>> >     -   RespHeader     Via: 1.1 varnish-v4
>> >     -   VCL_call       DELIVER
>> >     -   RespHeader     X-Cache: MISS
>> >     -   RespHeader     X-Cache-Hits: 0
>> >     -   RespUnset      X-Powered-By: ASP.NET <http://ASP.NET>
>> >     -   RespUnset      Server: Microsoft-IIS/8.5
>> >     -   RespUnset      X-Varnish: 294922
>> >     -   RespUnset      Via: 1.1 varnish-v4
>> >     -   RespUnset      X-AspNet-Version: 2.0.50727
>> >     -   VCL_Log        AT DEBUG - deliver URL: /Default2.aspx. Cookies:
>> ''
>> >     -   VCL_Log        AT DEBUG - deliver set-cookie: ''
>> >     -   VCL_return     deliver
>> >     -   Timestamp      Process: 1472834933.831794 0.641218 0.000026
>> >     -   Debug          "RES_MODE 2"
>> >     -   RespHeader     Connection: keep-alive
>> >     -   RespHeader     Accept-Ranges: bytes
>> >     -   Timestamp      Resp: 1472834933.831830 0.641255 0.000036
>> >     -   Debug          "XXX REF 2"
>> >     -   ReqAcct        828 0 828 276 52233 52509
>> >     -   End
>> >
>> >
>> >     On Fri, Sep 2, 2016 at 6:28 PM, Albert Tollku?i
>> >     <albert.tollkuci at gmail.com <mailto:albert.tollkuci at gmail.com>>
>> wrote:
>> >
>> >         Thank you Guillaume,
>> >         probably I'm missing something, but can't find any unset.
>> >
>> >         Is there anything running before vcl_backend_response? I see the
>> >         cookie in the backend side and is not there anymore in
>> >         vcl_backend_response?
>> >
>> >         Albert
>> >
>> >         On Fri, Sep 2, 2016 at 10:53 AM, Guillaume Quintard
>> >         <guillaume at varnish-software.com
>> >         <mailto:guillaume at varnish-software.com>> wrote:
>> >
>> >             Hi,
>> >
>> >             Look at the varnishlog, there should be a trace of that
>> >             unset, once confirmed, find it in your vcl.
>> >
>> >             Know that if you don't return from a subroutine, the
>> >             built-in vcl is executed ("locate builtin.vcl" to find a
>> >             commented version of it).
>> >
>> >
>> >             On Sep 1, 2016 12:59, "Albert Tollku?i"
>> >             <albert.tollkuci at gmail.com
>> >             <mailto:albert.tollkuci at gmail.com>> wrote:
>> >
>> >                 Hello,
>> >                 I've started using Varnish a few months ago and my vcl
>> >                 files has started to get complicated. I'm serving 5
>> >                 different sites with the same varnish instance (using
>> >                 different backends) and this part is working.
>> >
>> >                 Now I have some cases that for some sites I need to keep
>> >                 some cookies and cache different versions based on them.
>> >                 Basic scenario is to show the user a message only the
>> >                 first time he sees the page. Without varnish if the
>> >                 cookie is not there I set it and show the message. Next
>> >                 time the cookie is there and no message is shown.
>> >
>> >                 With varnish my logic is to include the cookie in hash
>> >                 and cache two different versions. The challenge is to
>> >                 make sure cookies are set correctly and here I'm kind of
>> >                 stack. I've added some debug messages in my vcl and for
>> >                 some reason I'm not seeing the cookies I set in my
>> >                 server in vcl_backend_response. Inside the routine, I
>> >                 have the following code in the beginning:
>> >
>> >                        # AT: DEBUG
>> >                        if(bereq.http.x-at-debug == "1") {
>> >                          std.log("AT DEBUG - bereq URL: " + bereq.url +
>> >                 ". Cookies: '" + bereq.http.Cookie + "'");
>> >                          std.log("AT DEBUG - beresp set-cookie: '" +
>> >                  beresp.http.set-cookie + "'");
>> >                        }
>> >
>> >                 I set x-at-debug header in vcl_recv based on my IP, so I
>> >                 can track only my requests. In the server side I see the
>> >                 cookie "__IKUB_AL_Myikub_PopUp" being set, but in
>> >                 vcl_backend_response is not.
>> >
>> >                 Is there anything in between that can strip the cookie?
>> >
>> >                 Thank you,
>> >                 Albert
>> >
>> >
>> >
>> >                 --
>> >                 Web: http://www.tollkuci.com <http://www.tollkuci.com/>
>> >                 Follow me on: LinkedIn
>> >                 <http://www.linkedin.com/in/alberttollkuci> Google+
>> >                 <https://plus.google.com/+AlbertTollku%C3%A7i/posts>
>> Facebook
>> >                 <https://www.facebook.com/albert.tollkuci> Twitter
>> >                 <https://twitter.com/AlbertTollkuci> Career 2.0
>> >                 <http://careers.stackoverflow.com/atollkuci>
>> >
>> >
>> >                 _______________________________________________
>> >                 varnish-misc mailing list
>> >                 varnish-misc at varnish-cache.org
>> >                 <mailto:varnish-misc at varnish-cache.org>
>> >
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>> >                 <
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>> >
>> >
>> >
>> >
>> >         --
>> >         Web: http://www.tollkuci.com <http://www.tollkuci.com/>
>> >         Follow me on: LinkedIn
>> >         <http://www.linkedin.com/in/alberttollkuci> Google+
>> >         <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
>> >         <https://www.facebook.com/albert.tollkuci> Twitter
>> >         <https://twitter.com/AlbertTollkuci> Career 2.0
>> >         <http://careers.stackoverflow.com/atollkuci>
>> >
>>  ------------------------------------------------------------------------
>> >
>> >
>> >
>> >
>> >     --
>> >     Web: http://www.tollkuci.com <http://www.tollkuci.com/>
>> >     Follow me on: LinkedIn
>> >     <http://www.linkedin.com/in/alberttollkuci> Google+
>> >     <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
>> >     <https://www.facebook.com/albert.tollkuci> Twitter
>> >     <https://twitter.com/AlbertTollkuci> Career 2.0
>> >     <http://careers.stackoverflow.com/atollkuci>
>> >
>>  ------------------------------------------------------------------------
>> >
>> >
>> >
>> >
>> > --
>> > Web: http://www.tollkuci.com <http://www.tollkuci.com/>
>> > Follow me on: LinkedIn
>> > <http://www.linkedin.com/in/alberttollkuci> Google+
>> > <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
>> > <https://www.facebook.com/albert.tollkuci> Twitter
>> > <https://twitter.com/AlbertTollkuci> Career 2.0
>> > <http://careers.stackoverflow.com/atollkuci>
>> >
>> >
>> >
>> > _______________________________________________
>> > varnish-misc mailing list
>> > varnish-misc at varnish-cache.org
>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>> >
>>
>>
>
>
> --
> Web: http://www.tollkuci.com
> Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
> <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
> <https://www.facebook.com/albert.tollkuci> Twitter
> <https://twitter.com/AlbertTollkuci> Career 2.0
> <http://careers.stackoverflow.com/atollkuci>
> ------------------------------
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160904/0839e73d/attachment.html>

From dridi at varni.sh  Mon Sep  5 08:15:23 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 5 Sep 2016 10:15:23 +0200
Subject: bitrate and memory allocated
In-Reply-To: <c4b627c6-67e0-eddc-cb97-e059addd45a9@yahoo.es>
References: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>
 <c4b627c6-67e0-eddc-cb97-e059addd45a9@yahoo.es>
Message-ID: <CABoVN9AJi8=Y3KSYjHZJgHSnGgeZA7n2Jp9Ly-PLjz5e5NWMpg@mail.gmail.com>

On Sat, Sep 3, 2016 at 12:14 PM, Miguel Gonz?lez
<miguel_3_gonzalez at yahoo.es> wrote:
> Why I don?t get a copy of my messages and I do get the rest of emails? I
> have just double checked my settings and I should get a copy of my messages

I don't know, but I found your emails in my spam folder.

> On 09/03/16 11:59 AM, Miguel Gonz?lez wrote:
>> Dear all,
>>
>>   I have Varnish 4.0.3 in front of a WHM/Cpanel hosting around 40
>> Wordpress sites. Only 4-5 sites are quite visited but the rest are blogs
>> that don?t have many visits.

<insert here usual advice to upgrade to 4.1 soon>

>>   I have assigned in malloc 6 Gb of RAM but I have been checking the
>> numbers provided by varnishstat and after 12 hours I don?t even get
>> close to 1 Gb of RAM (I assume c_bytes is what stands for) used by varnish:

Look at the varnish-counters manual for better understanding.
SMA.*.c_bytes is the counter for all bytes allocated by your malloc
storage while SMA.*.g_bytes is the gauge for the current amount of
allocated bytes.

>>  varnishstat -1 |grep SMA
>> SMA.s0.c_req                             33969         0.76 Allocator
>> requests
>> SMA.s0.c_fail                                0         0.00 Allocator
>> failures
>> SMA.s0.c_bytes                       984052835     21896.06 Bytes allocated
>> SMA.s0.c_freed                       395844358      8807.89 Bytes freed
>> SMA.s0.g_alloc                           19833          .   Allocations
>> outstanding
>> SMA.s0.g_bytes                       588208477          .   Bytes
>> outstanding
>> SMA.s0.g_space                      5854242467          .   Bytes available

Why is your Transient storage missing? By default it's an unbounded
malloc storage.

>>  I had a ttl of 2 hours and now I have changed it to 24 hours, maybe I
>> will increase it if I see the RAM allocated to Varnish is not entirely
>> used. The bitrate is erratic, reaching almost 1 and then dropping to
>> less than 0.5.
>>
>> Are my assumptions right?

I'm not sure to understand your assumptions, but basically what you
give for storage (eg 6GB) is what you get for storage and storage
only.

It doesn't include the rest of your Varnish memory footprint.

>> Another question, I know that running varnishtop I get the current most
>> visited site, but how do I aggregate that info over the hours? I would
>> like to get a picture of which are the most used sites (I?m considering
>> separating storages and dedicate only RAM for the most used and have
>> disk storage for the rest).

I don't know, shell scripting on top of varnishtop?



From schmidt at ze.tum.de  Mon Sep  5 10:35:17 2016
From: schmidt at ze.tum.de (Gerhard Schmidt)
Date: Mon, 5 Sep 2016 12:35:17 +0200
Subject: Varnish leaves old configs behind.
Message-ID: <df0b9ab9-da09-8942-bc42-06f0cb216e18@ze.tum.de>

Hi,

I'm in the process of upgrading from Varnish3 to varnish41.

I have a system running that creates the configs on the fly and sends
them to varnish via admin connection.

The system works after the needed modifications. The Problem is that
varnish is polluting my hard drive with old configs.

Every time my system generates a new config varnish creates a directory
with the config name and stores the compiled config there. When i delete
the config via vcl.discard the directory and the vgc.so remain in the
directory. In the admin connection the config disappears when usage
count dropped to 0.

After just one week it's about 400 MB already.

Did i miss something?

Regards
   Estartu

P.S. it's varnish 4.1.3 on FreeBSD 10.3

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 847 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160905/ce0eac06/attachment.bin>

From thomas.lecomte at virtual-expo.com  Mon Sep  5 12:05:40 2016
From: thomas.lecomte at virtual-expo.com (Thomas Lecomte)
Date: Mon, 5 Sep 2016 14:05:40 +0200
Subject: Varnish leaves old configs behind.
In-Reply-To: <df0b9ab9-da09-8942-bc42-06f0cb216e18@ze.tum.de>
References: <df0b9ab9-da09-8942-bc42-06f0cb216e18@ze.tum.de>
Message-ID: <CAJban9B9OvZe14WEqJS7Zr_fvtaknDgMXi_6VmqQ3QrVTG4BQA@mail.gmail.com>

On Mon, Sep 5, 2016 at 12:35 PM, Gerhard Schmidt <schmidt at ze.tum.de> wrote:
> Hi,
>
> I'm in the process of upgrading from Varnish3 to varnish41.
>
> I have a system running that creates the configs on the fly and sends
> them to varnish via admin connection.
>
> [...]
>
> After just one week it's about 400 MB already.
>
> Did i miss something?

Hi,

As part of your deploy script, you could use the vcl.discard command
to unload the previous configuration as soon as you've correctly
loaded the new one.

See https://www.varnish-cache.org/docs/trunk/reference/varnish-cli.html#vcl-discard-configname-label

-- 
Thomas Lecomte / Sysadmin / Virtual Expo



From schmidt at ze.tum.de  Mon Sep  5 12:09:54 2016
From: schmidt at ze.tum.de (Gerhard Schmidt)
Date: Mon, 5 Sep 2016 14:09:54 +0200
Subject: Varnish leaves old configs behind.
In-Reply-To: <CAJban9B9OvZe14WEqJS7Zr_fvtaknDgMXi_6VmqQ3QrVTG4BQA@mail.gmail.com>
References: <df0b9ab9-da09-8942-bc42-06f0cb216e18@ze.tum.de>
 <CAJban9B9OvZe14WEqJS7Zr_fvtaknDgMXi_6VmqQ3QrVTG4BQA@mail.gmail.com>
Message-ID: <e3ce0955-03c8-2c57-d534-06fc5584a043@ze.tum.de>

Am 05.09.2016 um 14:05 schrieb Thomas Lecomte:
> On Mon, Sep 5, 2016 at 12:35 PM, Gerhard Schmidt <schmidt at ze.tum.de> wrote:
>> Hi,
>>
>> I'm in the process of upgrading from Varnish3 to varnish41.
>>
>> I have a system running that creates the configs on the fly and sends
>> them to varnish via admin connection.
>>
>> [...]
>>
>> After just one week it's about 400 MB already.
>>
>> Did i miss something?
> 
> Hi,
> 
> As part of your deploy script, you could use the vcl.discard command
> to unload the previous configuration as soon as you've correctly
> loaded the new one.
> 
> See https://www.varnish-cache.org/docs/trunk/reference/varnish-cli.html#vcl-discard-configname-label

That was in the part you didn't quote. Every time I install a new config
the old one is discarded and disappears from vcl.list but not from the
directory.

Regards
   Estartu


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 847 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160905/5b59f05f/attachment.bin>

From thomas.lecomte at virtual-expo.com  Mon Sep  5 12:21:38 2016
From: thomas.lecomte at virtual-expo.com (Thomas Lecomte)
Date: Mon, 5 Sep 2016 14:21:38 +0200
Subject: Varnish leaves old configs behind.
In-Reply-To: <e3ce0955-03c8-2c57-d534-06fc5584a043@ze.tum.de>
References: <df0b9ab9-da09-8942-bc42-06f0cb216e18@ze.tum.de>
 <CAJban9B9OvZe14WEqJS7Zr_fvtaknDgMXi_6VmqQ3QrVTG4BQA@mail.gmail.com>
 <e3ce0955-03c8-2c57-d534-06fc5584a043@ze.tum.de>
Message-ID: <CAJban9DwuuhScwiGBcgdi6PNPgz9c7Pov9aNBrSYa5Td7Vr5zg@mail.gmail.com>

On Mon, Sep 5, 2016 at 2:09 PM, Gerhard Schmidt <schmidt at ze.tum.de> wrote:
> That was in the part you didn't quote. Every time I install a new config
> the old one is discarded and disappears from vcl.list but not from the
> directory.

Oh sorry - I don't know how I missed that.

I just tested on my varnish instance and vcl.discard effectively
removes the directory  /var/lib/varnish/$instance/$confname. The user
as which varnishadm is run doesn't seem to be important (root or my
own user).

The compiled VCL files are owned by the varnish user (as which
varnishd runs). Might it not be a permissions problem for you?

-- 
Thomas Lecomte / Sysadmin / Virtual Expo



From schmidt at ze.tum.de  Mon Sep  5 12:35:54 2016
From: schmidt at ze.tum.de (Gerhard Schmidt)
Date: Mon, 5 Sep 2016 14:35:54 +0200
Subject: Varnish leaves old configs behind.
In-Reply-To: <CAJban9DwuuhScwiGBcgdi6PNPgz9c7Pov9aNBrSYa5Td7Vr5zg@mail.gmail.com>
References: <df0b9ab9-da09-8942-bc42-06f0cb216e18@ze.tum.de>
 <CAJban9B9OvZe14WEqJS7Zr_fvtaknDgMXi_6VmqQ3QrVTG4BQA@mail.gmail.com>
 <e3ce0955-03c8-2c57-d534-06fc5584a043@ze.tum.de>
 <CAJban9DwuuhScwiGBcgdi6PNPgz9c7Pov9aNBrSYa5Td7Vr5zg@mail.gmail.com>
Message-ID: <540fa211-a028-8aec-1bb7-af7ed0075e51@ze.tum.de>

Am 05.09.2016 um 14:21 schrieb Thomas Lecomte:
> On Mon, Sep 5, 2016 at 2:09 PM, Gerhard Schmidt <schmidt at ze.tum.de> wrote:
>> That was in the part you didn't quote. Every time I install a new config
>> the old one is discarded and disappears from vcl.list but not from the
>> directory.
> 
> Oh sorry - I don't know how I missed that.
> 
> I just tested on my varnish instance and vcl.discard effectively
> removes the directory  /var/lib/varnish/$instance/$confname. The user
> as which varnishadm is run doesn't seem to be important (root or my
> own user).
> 
> The compiled VCL files are owned by the varnish user (as which
> varnishd runs). Might it not be a permissions problem for you?

OK i did some more digging. I seams that die directories left behind are
from configs that failed to compile. I don't get where the vcg.so comes
from if the compile failed. But since i fixed a bug in my system this
Morning there are no new directories since than but there are at least
50 config changes since than.

If this is intentional it might be a good idea to put the text of the
fail config and the error message there to help debugging.

Regards
   Estartu



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 847 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160905/c113e045/attachment.bin>

From miguel_3_gonzalez at yahoo.es  Mon Sep  5 17:19:41 2016
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Mon, 5 Sep 2016 19:19:41 +0200
Subject: bitrate and memory allocated
In-Reply-To: <CABoVN9AJi8=Y3KSYjHZJgHSnGgeZA7n2Jp9Ly-PLjz5e5NWMpg@mail.gmail.com>
References: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>
 <c4b627c6-67e0-eddc-cb97-e059addd45a9@yahoo.es>
 <CABoVN9AJi8=Y3KSYjHZJgHSnGgeZA7n2Jp9Ly-PLjz5e5NWMpg@mail.gmail.com>
Message-ID: <c5d198ef-63c1-4b13-5850-35b844dbb156@yahoo.es>


> 
> <insert here usual advice to upgrade to 4.1 soon>

I have just upgraded. Didn?t know that there was a yum/rpm installation
already





> Look at the varnish-counters manual for better understanding.
> SMA.*.c_bytes is the counter for all bytes allocated by your malloc
> storage while SMA.*.g_bytes is the gauge for the current amount of
> allocated bytes.

Any URL? I have googled for a few hours but I didn?t get a clear
understanding from the official manuals.




>>>  varnishstat -1 |grep SMA
>>> SMA.s0.c_req                             33969         0.76 Allocator
>>> requests
>>> SMA.s0.c_fail                                0         0.00 Allocator
>>> failures
>>> SMA.s0.c_bytes                       984052835     21896.06 Bytes allocated
>>> SMA.s0.c_freed                       395844358      8807.89 Bytes freed
>>> SMA.s0.g_alloc                           19833          .   Allocations
>>> outstanding
>>> SMA.s0.g_bytes                       588208477          .   Bytes
>>> outstanding
>>> SMA.s0.g_space                      5854242467          .   Bytes available
> 
> Why is your Transient storage missing? By default it's an unbounded
> malloc storage.

I took away that part:

SMA.s0.c_req                             73478         0.72 Allocator
requests
SMA.s0.c_fail                                0         0.00 Allocator
failures
SMA.s0.c_bytes                      2277130716     22386.92 Bytes allocated
SMA.s0.c_freed                       452652599      4450.12 Bytes freed
SMA.s0.g_alloc                           52489          .   Allocations
outstanding
SMA.s0.g_bytes                      1824478117          .   Bytes
outstanding
SMA.s0.g_space                      4617972827          .   Bytes available
SMA.Transient.c_req                     174076         1.71 Allocator
requests
SMA.Transient.c_fail                         0         0.00 Allocator
failures
SMA.Transient.c_bytes               1611603457     15843.99 Bytes allocated
SMA.Transient.c_freed               1611595873     15843.92 Bytes freed
SMA.Transient.g_alloc                       10          .   Allocations
outstanding
SMA.Transient.g_bytes                     7584          .   Bytes
outstanding
SMA.Transient.g_space                        0          .   Bytes available


Thanks!

Miguel




From dridi at varni.sh  Mon Sep  5 20:44:48 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 5 Sep 2016 22:44:48 +0200
Subject: bitrate and memory allocated
In-Reply-To: <c5d198ef-63c1-4b13-5850-35b844dbb156@yahoo.es>
References: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>
 <c4b627c6-67e0-eddc-cb97-e059addd45a9@yahoo.es>
 <CABoVN9AJi8=Y3KSYjHZJgHSnGgeZA7n2Jp9Ly-PLjz5e5NWMpg@mail.gmail.com>
 <c5d198ef-63c1-4b13-5850-35b844dbb156@yahoo.es>
Message-ID: <CABoVN9C=aXS1FFRBJjxSqeHfSFQ7wGJdo5w_3APsZza=qFihsw@mail.gmail.com>

> Any URL? I have googled for a few hours but I didn?t get a clear
> understanding from the official manuals.

https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma

You get the same contents by running `man varnish-counters` on your
system. But I'm very "surprised" that you're having a hard time
understanding the docs.

> I took away that part:

Unfortunately even with that part I can't help you. It would require
more effort than I can afford.

Good luck



From sujithnss at gmail.com  Tue Sep  6 06:07:48 2016
From: sujithnss at gmail.com (sujith pv)
Date: Tue, 6 Sep 2016 11:37:48 +0530
Subject: Varnish malloc issue
In-Reply-To: <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <20160901231832.GA3267@optusnet.com.au>
 <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>
Message-ID: <CAFzBPcUiAPkZtf-LO=r+7_FD9RV3pKNAcOzBEMQpRLFubpjkCg@mail.gmail.com>

Thanks Guillaume. I will try the transient settings first and verify the
same. Also could you please elaborate on the short lived objects.

Best Regards
Sujith P V

On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
guillaume at varnish-software.com> wrote:

> Hi Viktor,
>
> Have a look here https://www.varnish-cache.org/docs/trunk/users-
> guide/storage-backends.html#transient-storage
>
> Sujith, please try to use a packaged version from your distribution.
>
> --
> Guillaume Quintard
>
> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
> viktor.villafuerte at optusnet.com.au> wrote:
>
>> Hi Guillaume,
>>
>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>> > Is that virtual or real memory?
>> > Try to cap your Transient storage to 1GB, you may be overloaded with
>> short
>> > lived objects.
>>
>> Could you elaborate on this bit more? I've got very similar problem
>> here. Eg: Total memory 384g
>>
>> VIRT: 487g
>> RES:  311g
>>
>> free says: 46g free
>>
>> but SWAP is 91.3% used :(
>>
>>
>> How can this be?
>>
>>
>> thanks
>>
>> v
>>
>>
>> >
>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com> wrote:
>> >
>> > > Hi All
>> > >
>> > > I had already mailed this query long back but this time putting in a
>> > > different manner.
>> > >
>> > >
>> > >    - We are using Varnish 4.0 in our end. We have a machine with
>> memory
>> > >    of 16G with 8G being malloc for Varnish. We have a TTL for 2 hrs
>> as well.
>> > >    - During our peak traffic, when we see the total memory of the
>> machine
>> > >    reaching 90% and like varnishd process is taking some 89% .
>> > >    - So I'm just confused even though we had allocated just 8G malloc
>> > >    which is like 50% of the total memory, how the process is eating
>> up 89% of
>> > >    memory and the memory is not releasing even after TTL.
>> > >
>> > >
>> > > Any help please...
>> > >
>> > > Best Regards
>> > > Sujith P V
>> > >
>> > > _______________________________________________
>> > > varnish-misc mailing list
>> > > varnish-misc at varnish-cache.org
>> > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>> > >
>>
>> > _______________________________________________
>> > varnish-misc mailing list
>> > varnish-misc at varnish-cache.org
>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>> --
>> Regards
>>
>> Viktor Villafuerte
>> Optus Internet Engineering
>> t: +61 2 80825265
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160906/c2820b34/attachment.html>

From guillaume at varnish-software.com  Tue Sep  6 06:24:23 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Tue, 6 Sep 2016 08:24:23 +0200
Subject: Varnish malloc issue
In-Reply-To: <CAFzBPcUiAPkZtf-LO=r+7_FD9RV3pKNAcOzBEMQpRLFubpjkCg@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <20160901231832.GA3267@optusnet.com.au>
 <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>
 <CAFzBPcUiAPkZtf-LO=r+7_FD9RV3pKNAcOzBEMQpRLFubpjkCg@mail.gmail.com>
Message-ID: <CAJ6ZYQz0PV9exEzcc21sZt=Bca7CT322xhHq2OJKvYwfAKwifQ@mail.gmail.com>

https://www.varnish-cache.org/docs/4.1/reference/varnishd.html#shortlived

On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:

> Thanks Guillaume. I will try the transient settings first and verify the
> same. Also could you please elaborate on the short lived objects.
>
> Best Regards
> Sujith P V
>
> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
> guillaume at varnish-software.com> wrote:
>
>> Hi Viktor,
>>
>> Have a look here https://www.varnish-cache.org/docs/trunk/users-guide/
>> storage-backends.html#transient-storage
>>
>> Sujith, please try to use a packaged version from your distribution.
>>
>> --
>> Guillaume Quintard
>>
>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>> viktor.villafuerte at optusnet.com.au> wrote:
>>
>>> Hi Guillaume,
>>>
>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>> > Is that virtual or real memory?
>>> > Try to cap your Transient storage to 1GB, you may be overloaded with
>>> short
>>> > lived objects.
>>>
>>> Could you elaborate on this bit more? I've got very similar problem
>>> here. Eg: Total memory 384g
>>>
>>> VIRT: 487g
>>> RES:  311g
>>>
>>> free says: 46g free
>>>
>>> but SWAP is 91.3% used :(
>>>
>>>
>>> How can this be?
>>>
>>>
>>> thanks
>>>
>>> v
>>>
>>>
>>> >
>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com> wrote:
>>> >
>>> > > Hi All
>>> > >
>>> > > I had already mailed this query long back but this time putting in a
>>> > > different manner.
>>> > >
>>> > >
>>> > >    - We are using Varnish 4.0 in our end. We have a machine with
>>> memory
>>> > >    of 16G with 8G being malloc for Varnish. We have a TTL for 2 hrs
>>> as well.
>>> > >    - During our peak traffic, when we see the total memory of the
>>> machine
>>> > >    reaching 90% and like varnishd process is taking some 89% .
>>> > >    - So I'm just confused even though we had allocated just 8G malloc
>>> > >    which is like 50% of the total memory, how the process is eating
>>> up 89% of
>>> > >    memory and the memory is not releasing even after TTL.
>>> > >
>>> > >
>>> > > Any help please...
>>> > >
>>> > > Best Regards
>>> > > Sujith P V
>>> > >
>>> > > _______________________________________________
>>> > > varnish-misc mailing list
>>> > > varnish-misc at varnish-cache.org
>>> > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>> > >
>>>
>>> > _______________________________________________
>>> > varnish-misc mailing list
>>> > varnish-misc at varnish-cache.org
>>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>>
>>> --
>>> Regards
>>>
>>> Viktor Villafuerte
>>> Optus Internet Engineering
>>> t: +61 2 80825265
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160906/4b64cbd8/attachment.html>

From sujithnss at gmail.com  Tue Sep  6 06:31:47 2016
From: sujithnss at gmail.com (sujith pv)
Date: Tue, 6 Sep 2016 12:01:47 +0530
Subject: Varnish malloc issue
In-Reply-To: <CAJ6ZYQz0PV9exEzcc21sZt=Bca7CT322xhHq2OJKvYwfAKwifQ@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <20160901231832.GA3267@optusnet.com.au>
 <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>
 <CAFzBPcUiAPkZtf-LO=r+7_FD9RV3pKNAcOzBEMQpRLFubpjkCg@mail.gmail.com>
 <CAJ6ZYQz0PV9exEzcc21sZt=Bca7CT322xhHq2OJKvYwfAKwifQ@mail.gmail.com>
Message-ID: <CAFzBPcVffR6UWzCRwC4BBFWu7pGFb7QezYY4e2_D2YvxgyVN6Q@mail.gmail.com>

Thanks Guillaume. May be Im asking a very basic question , but still how
this short lived objects are created and I'm not giving any settings for
the same as well.

On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
guillaume at varnish-software.com> wrote:

> https://www.varnish-cache.org/docs/4.1/reference/varnishd.html#shortlived
>
> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:
>
>> Thanks Guillaume. I will try the transient settings first and verify the
>> same. Also could you please elaborate on the short lived objects.
>>
>> Best Regards
>> Sujith P V
>>
>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>> guillaume at varnish-software.com> wrote:
>>
>>> Hi Viktor,
>>>
>>> Have a look here https://www.varnish-cache.org/docs/trunk/users-guide/st
>>> orage-backends.html#transient-storage
>>>
>>> Sujith, please try to use a packaged version from your distribution.
>>>
>>> --
>>> Guillaume Quintard
>>>
>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>
>>>> Hi Guillaume,
>>>>
>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>> > Is that virtual or real memory?
>>>> > Try to cap your Transient storage to 1GB, you may be overloaded with
>>>> short
>>>> > lived objects.
>>>>
>>>> Could you elaborate on this bit more? I've got very similar problem
>>>> here. Eg: Total memory 384g
>>>>
>>>> VIRT: 487g
>>>> RES:  311g
>>>>
>>>> free says: 46g free
>>>>
>>>> but SWAP is 91.3% used :(
>>>>
>>>>
>>>> How can this be?
>>>>
>>>>
>>>> thanks
>>>>
>>>> v
>>>>
>>>>
>>>> >
>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com> wrote:
>>>> >
>>>> > > Hi All
>>>> > >
>>>> > > I had already mailed this query long back but this time putting in a
>>>> > > different manner.
>>>> > >
>>>> > >
>>>> > >    - We are using Varnish 4.0 in our end. We have a machine with
>>>> memory
>>>> > >    of 16G with 8G being malloc for Varnish. We have a TTL for 2 hrs
>>>> as well.
>>>> > >    - During our peak traffic, when we see the total memory of the
>>>> machine
>>>> > >    reaching 90% and like varnishd process is taking some 89% .
>>>> > >    - So I'm just confused even though we had allocated just 8G
>>>> malloc
>>>> > >    which is like 50% of the total memory, how the process is eating
>>>> up 89% of
>>>> > >    memory and the memory is not releasing even after TTL.
>>>> > >
>>>> > >
>>>> > > Any help please...
>>>> > >
>>>> > > Best Regards
>>>> > > Sujith P V
>>>> > >
>>>> > > _______________________________________________
>>>> > > varnish-misc mailing list
>>>> > > varnish-misc at varnish-cache.org
>>>> > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>> > >
>>>>
>>>> > _______________________________________________
>>>> > varnish-misc mailing list
>>>> > varnish-misc at varnish-cache.org
>>>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>
>>>>
>>>> --
>>>> Regards
>>>>
>>>> Viktor Villafuerte
>>>> Optus Internet Engineering
>>>> t: +61 2 80825265
>>>>
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160906/4c05504c/attachment.html>

From guillaume at varnish-software.com  Tue Sep  6 08:27:24 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Tue, 6 Sep 2016 10:27:24 +0200
Subject: Varnish malloc issue
In-Reply-To: <CAFzBPcVffR6UWzCRwC4BBFWu7pGFb7QezYY4e2_D2YvxgyVN6Q@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <20160901231832.GA3267@optusnet.com.au>
 <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>
 <CAFzBPcUiAPkZtf-LO=r+7_FD9RV3pKNAcOzBEMQpRLFubpjkCg@mail.gmail.com>
 <CAJ6ZYQz0PV9exEzcc21sZt=Bca7CT322xhHq2OJKvYwfAKwifQ@mail.gmail.com>
 <CAFzBPcVffR6UWzCRwC4BBFWu7pGFb7QezYY4e2_D2YvxgyVN6Q@mail.gmail.com>
Message-ID: <CAJ6ZYQwhLU-ckADgfTTrdN-W-GcR3VzXYjf3BPEC4G9a__96UA@mail.gmail.com>

Sorry, I didn't get your question.

-- 
Guillaume Quintard

On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com> wrote:

> Thanks Guillaume. May be Im asking a very basic question , but still how
> this short lived objects are created and I'm not giving any settings for
> the same as well.
>
> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
> guillaume at varnish-software.com> wrote:
>
>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.html#shortlived
>>
>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:
>>
>>> Thanks Guillaume. I will try the transient settings first and verify
>>> the same. Also could you please elaborate on the short lived objects.
>>>
>>> Best Regards
>>> Sujith P V
>>>
>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> Hi Viktor,
>>>>
>>>> Have a look here https://www.varnish-cache
>>>> .org/docs/trunk/users-guide/storage-backends.html#transient-storage
>>>>
>>>> Sujith, please try to use a packaged version from your distribution.
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>
>>>>> Hi Guillaume,
>>>>>
>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>> > Is that virtual or real memory?
>>>>> > Try to cap your Transient storage to 1GB, you may be overloaded with
>>>>> short
>>>>> > lived objects.
>>>>>
>>>>> Could you elaborate on this bit more? I've got very similar problem
>>>>> here. Eg: Total memory 384g
>>>>>
>>>>> VIRT: 487g
>>>>> RES:  311g
>>>>>
>>>>> free says: 46g free
>>>>>
>>>>> but SWAP is 91.3% used :(
>>>>>
>>>>>
>>>>> How can this be?
>>>>>
>>>>>
>>>>> thanks
>>>>>
>>>>> v
>>>>>
>>>>>
>>>>> >
>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>> >
>>>>> > > Hi All
>>>>> > >
>>>>> > > I had already mailed this query long back but this time putting in
>>>>> a
>>>>> > > different manner.
>>>>> > >
>>>>> > >
>>>>> > >    - We are using Varnish 4.0 in our end. We have a machine with
>>>>> memory
>>>>> > >    of 16G with 8G being malloc for Varnish. We have a TTL for 2
>>>>> hrs as well.
>>>>> > >    - During our peak traffic, when we see the total memory of the
>>>>> machine
>>>>> > >    reaching 90% and like varnishd process is taking some 89% .
>>>>> > >    - So I'm just confused even though we had allocated just 8G
>>>>> malloc
>>>>> > >    which is like 50% of the total memory, how the process is
>>>>> eating up 89% of
>>>>> > >    memory and the memory is not releasing even after TTL.
>>>>> > >
>>>>> > >
>>>>> > > Any help please...
>>>>> > >
>>>>> > > Best Regards
>>>>> > > Sujith P V
>>>>> > >
>>>>> > > _______________________________________________
>>>>> > > varnish-misc mailing list
>>>>> > > varnish-misc at varnish-cache.org
>>>>> > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>> > >
>>>>>
>>>>> > _______________________________________________
>>>>> > varnish-misc mailing list
>>>>> > varnish-misc at varnish-cache.org
>>>>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>
>>>>>
>>>>> --
>>>>> Regards
>>>>>
>>>>> Viktor Villafuerte
>>>>> Optus Internet Engineering
>>>>> t: +61 2 80825265
>>>>>
>>>>
>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160906/223c33a2/attachment.html>

From guillaume at varnish-software.com  Tue Sep  6 08:28:23 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Tue, 6 Sep 2016 10:28:23 +0200
Subject: varnish statistics
In-Reply-To: <cdd0d1b2-e442-4c9f-ac0a-4d7ad03ed29a@yahoo.es>
References: <cdd0d1b2-e442-4c9f-ac0a-4d7ad03ed29a@yahoo.es>
Message-ID: <CAJ6ZYQySPz6V56du-T0Y6uZK=K8pRMO-4_eYiHeR+JZ=ULOrHQ@mail.gmail.com>

Have a look at collectd, I'd say. I also so a few people using zabbix

-- 
Guillaume Quintard

On Thu, Aug 18, 2016 at 12:22 PM, Miguel Gonz?lez <
miguel_3_gonzalez at yahoo.es> wrote:

> Hi!
>
>  I have a WHM/Cpanel server behind Varnish. I would like to know which
> are the websites that use up the most of the Varnish cache (I?m thinking
> of even creating different storages for the most used websites).
>
>  I took a look into Munin and Pandora, but maybe you guys know simpler
> or better tools? I don?t mind if they are GUIs or command line tools.
>
>  Regards,
>
>  Miguel
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160906/0625fa78/attachment.html>

From miguel_3_gonzalez at yahoo.es  Tue Sep  6 10:08:22 2016
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Tue, 6 Sep 2016 12:08:22 +0200
Subject: bitrate and memory allocated
In-Reply-To: <CABoVN9C=aXS1FFRBJjxSqeHfSFQ7wGJdo5w_3APsZza=qFihsw@mail.gmail.com>
References: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>
 <c4b627c6-67e0-eddc-cb97-e059addd45a9@yahoo.es>
 <CABoVN9AJi8=Y3KSYjHZJgHSnGgeZA7n2Jp9Ly-PLjz5e5NWMpg@mail.gmail.com>
 <c5d198ef-63c1-4b13-5850-35b844dbb156@yahoo.es>
 <CABoVN9C=aXS1FFRBJjxSqeHfSFQ7wGJdo5w_3APsZza=qFihsw@mail.gmail.com>
Message-ID: <f108f8cc-482d-04cd-0cf7-815a10d57b0d@yahoo.es>

On 09/05/16 10:44 PM, Dridi Boukelmoune wrote:
>> Any URL? I have googled for a few hours but I didn?t get a clear
>> understanding from the official manuals.
> 
> https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma


I had read the 4.0 docs but I find them quite short and not very
explanatory. What are the counters? The current use of memory? The
amount allocated since Varnish was restarted (historical)? At least for
me, I would like an example to better understand what those counters mean.

Thanks

Miguel



From viktor.villafuerte at optusnet.com.au  Sun Sep 11 23:41:54 2016
From: viktor.villafuerte at optusnet.com.au (Viktor Villafuerte)
Date: Mon, 12 Sep 2016 09:41:54 +1000
Subject: Varnish malloc + swap
Message-ID: <20160911234154.GB3267@optusnet.com.au>

Hi all,

This is related to varnish 3.0.2 (yes I know upgrade is due) where I
have 48G of memory and storage is 'malloc,29G'. This is all good and
from graphs I can see the system having about 15G of unused memory.
Varnish (from smem) is using pretty much all the allocated memory. There
is more content expected then the memory available and Varnish is
expected to 'recycle' memory, which it does. I've changed nuke_limit to
500 (from 50). I don't want to raise it further as there's not that much
traffic on those servers and I absolutely believe that 500 should be
more than enough.

But then ('then' is sometimes 12hrs, sometimes 4days) it starts eating
swap but there is still 15G of unused memory. Further, the swap is
sometimes released but sometimes it's kept by Varnish. Again, from smem,
I can see that Varnish is holding the swap.

There was a suggestiong to limit transient storage but from varnishstat
I can see no allocator failures in transient storage and every byte that
is allocated there is also freed.

I have changed kernel parameters for memory/swap usage but this is still
occuring.


How/when does Varnish decide to use swap and for what reason if there's
still plenty of memory available?


Any pointers to links, or wisdom learnt from experience is welcomed :)


thanks


v


-- 
Regards

Viktor Villafuerte
Optus Internet Engineering
t: +61 2 80825265



From r at roze.lv  Mon Sep 12 00:22:36 2016
From: r at roze.lv (Reinis Rozitis)
Date: Mon, 12 Sep 2016 03:22:36 +0300
Subject: Varnish malloc + swap
In-Reply-To: <20160911234154.GB3267@optusnet.com.au>
References: <20160911234154.GB3267@optusnet.com.au>
Message-ID: <B58CF923C4CE4D9DBAFF814AD39C69B1@MezhRoze>

> I have changed kernel parameters for memory/swap usage but this is still 
> occuring.

Just to check (if it's linux even), does this also include vm.swappiness = 0 
?
Though my own experience with older kernels is that once swap is taken it 
might never be released unless a manual swapoff is performed.


> Any pointers to links, or wisdom learnt from experience is welcomed :)

Is there a reason you need swap at all?
In my mind it makes no sense having a swap on a 48G system. Even more if it 
(as it seems) is dedicated for a single service.

It takes just a bit (start with less and see at which point the ram usage 
stops growing) to tune the malloc setting so the varnish uses all the 
available ram optimally and the OOM killer doesn't kick in.

p.s. also when making bigger caches than physical memory I have had much 
better results (on 3.0.x at least) instead of 'malloc' having a 'file' 
backend (on swapless systems) and relying on linux page cache rather than 
swap.

rr 




From viktor.villafuerte at optusnet.com.au  Mon Sep 12 00:40:59 2016
From: viktor.villafuerte at optusnet.com.au (Viktor Villafuerte)
Date: Mon, 12 Sep 2016 10:40:59 +1000
Subject: Varnish malloc + swap
In-Reply-To: <B58CF923C4CE4D9DBAFF814AD39C69B1@MezhRoze>
References: <20160911234154.GB3267@optusnet.com.au>
 <B58CF923C4CE4D9DBAFF814AD39C69B1@MezhRoze>
Message-ID: <20160912004059.GC3267@optusnet.com.au>

Hi Reinis,



On Mon 12 Sep 2016 03:22:36, Reinis Rozitis wrote:
> > I have changed kernel parameters for memory/swap usage but this is still
> > occuring.
> 
> Just to check (if it's linux even), does this also include vm.swappiness = 0

Yes Linux RHEL6 and swappiness = 0. From my reading though varnish does
not follow this and it clearly appears so in my case.


> ?
> Though my own experience with older kernels is that once swap is taken it
> might never be released unless a manual swapoff is performed.
> 
> 
> > Any pointers to links, or wisdom learnt from experience is welcomed :)
> 
> Is there a reason you need swap at all?
> In my mind it makes no sense having a swap on a 48G system. Even more if it
> (as it seems) is dedicated for a single service.

Yes, that was my first thought too, to get rid of swap. But currently
it's there and there's not much I can do about it now. Possibly when
Varnish upgrade happens this could be changed but for now I'm stuck with
swap.


> 
> It takes just a bit (start with less and see at which point the ram usage
> stops growing) to tune the malloc setting so the varnish uses all the
> available ram optimally and the OOM killer doesn't kick in.
> 
> p.s. also when making bigger caches than physical memory I have had much
> better results (on 3.0.x at least) instead of 'malloc' having a 'file'
> backend (on swapless systems) and relying on linux page cache rather than
> swap.

As speed is an important factor.. I'm assuming the file storage is
mapped to memory?

> 
> rr


thanks


v

-- 
Regards

Viktor Villafuerte
Optus Internet Engineering
t: +61 2 80825265



From r at roze.lv  Mon Sep 12 01:08:04 2016
From: r at roze.lv (Reinis Rozitis)
Date: Mon, 12 Sep 2016 04:08:04 +0300
Subject: Varnish malloc + swap
In-Reply-To: <20160912004059.GC3267@optusnet.com.au>
References: <20160911234154.GB3267@optusnet.com.au>
 <B58CF923C4CE4D9DBAFF814AD39C69B1@MezhRoze>
 <20160912004059.GC3267@optusnet.com.au>
Message-ID: <6F9A16337D0447EB86EF024CEFC1FA33@MezhRoze>

> Yes Linux RHEL6 and swappiness = 0. From my reading though varnish does 
> not follow this and it clearly appears so in my case.

It's not a setting for varnish rather than kernel (via sysctl).


> Yes, that was my first thought too, to get rid of swap. But currently it's 
> there and there's not much I can do about it now. Possibly when Varnish 
> upgrade happens this could be changed but for now I'm stuck with swap.

How so?
You can dynamically turn off all the swaps with 'swapoff -a'  ( 
http://linux.die.net/man/8/swapoff ).
If there is a need later reenable with swapon.

rr





From viktor.villafuerte at optusnet.com.au  Mon Sep 12 01:58:58 2016
From: viktor.villafuerte at optusnet.com.au (Viktor Villafuerte)
Date: Mon, 12 Sep 2016 11:58:58 +1000
Subject: Varnish malloc + swap
In-Reply-To: <6F9A16337D0447EB86EF024CEFC1FA33@MezhRoze>
References: <20160911234154.GB3267@optusnet.com.au>
 <B58CF923C4CE4D9DBAFF814AD39C69B1@MezhRoze>
 <20160912004059.GC3267@optusnet.com.au>
 <6F9A16337D0447EB86EF024CEFC1FA33@MezhRoze>
Message-ID: <20160912015858.GD3267@optusnet.com.au>

On Mon 12 Sep 2016 04:08:04, Reinis Rozitis wrote:
> > Yes Linux RHEL6 and swappiness = 0. From my reading though varnish does
> > not follow this and it clearly appears so in my case.
> 
> It's not a setting for varnish rather than kernel (via sysctl).

Sure, what I wanted to say is that it makes no difference :)


> 
> 
> > Yes, that was my first thought too, to get rid of swap. But currently
> > it's there and there's not much I can do about it now. Possibly when
> > Varnish upgrade happens this could be changed but for now I'm stuck with
> > swap.
> 
> How so?
> You can dynamically turn off all the swaps with 'swapoff -a'  (
> http://linux.die.net/man/8/swapoff ).
> If there is a need later reenable with swapon.

That would be considered an architectural change and I cannot just do
it.. (corporate rules)


> 
> rr
> 
> 
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


v

-- 
Regards

Viktor Villafuerte
Optus Internet Engineering
t: +61 2 80825265



From mahesho at outlook.com  Mon Sep 12 09:07:29 2016
From: mahesho at outlook.com (Mahesh Ollalwar)
Date: Mon, 12 Sep 2016 09:07:29 +0000
Subject: varnish core dump
In-Reply-To: <CAEh05VYq-Nm6sH70CCVtAJtT02oob1nk9f_dUppDzyVtAZZjhA@mail.gmail.com>
References: <PN1PR01MB0575920E7E549C176B150A3DA92E0@PN1PR01MB0575.INDPRD01.PROD.OUTLOOK.COM>,
 <CAEh05VYq-Nm6sH70CCVtAJtT02oob1nk9f_dUppDzyVtAZZjhA@mail.gmail.com>
Message-ID: <PN1PR01MB0575331DC326679609EDF21CA9FF0@PN1PR01MB0575.INDPRD01.PROD.OUTLOOK.COM>


Hi,


Thanks for the response.

We have upgraded to latest stable version 4.1.3 but still see panics. below are the logs.


Sep 12 12:45:18 Prod-SHARED-CACHE6 abrt[32688]: Saved core dump of pid 24551 (/product/varnish/sbin/varnishd) to /var/spool/abrt/ccpp-2016-09-12-12:45:15-24551 (585650176 bytes)
Sep 12 12:45:18 Prod-SHARED-CACHE6 abrtd: Directory 'ccpp-2016-09-12-12:45:15-24551' creation detected
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Executable '/product/varnish/sbin/varnishd' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551' exited with 1
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551'
Sep 12 12:45:19 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) not responding to CLI, killing it.
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) died signal=6 (core dumped)
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) Last panic at: Mon, 12 Sep 2016 07:15:20 GMT#012"Assert error in http_EstimateWS(), cache/cache_http.c line 793:#012  Condition((fm->hd[u].b) != 0) not true.#012thread = (cache-worker)#012version = varnish-4.1.3 revision 5e3b6d2#012ident = Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012  0x432f13: pan_ic+0x163#012  0x42dec5: http_EstimateWS+0x435#012  0x421874: vbf_beresp2obj+0x84#012  0x422cb5: vbf_stp_error+0x235#012  0x42366b: vbf_fetch_thread+0x1db#012  0x449825: WRK_Thread+0x4a5#012  0x449c5b: pool_thread+0x2b#012  0x3c25e079d1: libpthread.so.0() [0x3c25e079d1]#012  0x3c25ae8b6d: libc.so.6(clone+0x6d) [0x3c25ae8b6d]#012busyobj = 0x7f7f505208a0 {#012  ws = 0x7f7f50520960 {#012    OVERFLOWED id = \"Bo\",#012    {s,f,r,e} = {0x7f7f50522818,+57440,(nil),+57440},#012  },#012  refcnt = 2,#012  retries = 0, failed = 1, state = 1,#012  flags = {do_gzip, do_stream, is_gunzip},#012  director_req = 0x1a16130 {#012    vcl_name = farm_anon,#012    type = round-robin {#012    },#012  },#012  director_resp = 0x1a08e98 {#012    vcl_name = PROD_LB_BE_ANON_1,#012    type = backend {#012      display_name = boot.PROD_LB_BE_ANON_1,#012      ipv4 = 10.254.4.254,#012      port = 8080,#012      hosthdr = 10.254.4.254,#012      health=healthy, admin_health=probe, changed=1473664247.5,#012      n_conn = 2,#012    },#012  },#012  http[bereq] = 0x7f7f50520f28 {#012    ws[Bo] = 0x7f7f50520960,#012    hdrs {#012      \"GET\",#012      \"/AVS/besc?action=GetArrayContentList&channel=ANDROID&categoryId=7022\",#012      \"HTTP/1.1\",#012      \"host: www.example.com<http://www.example.com> \",#012      \"Accept-Encoding: gzip\",#012      \"Akamai-Origin-Hop: 4\",#012      \"True-Client-IP: 183.83.249.187\",#012      \"User-Agent: Dalvik/2.1.0 (Linux; U; Android 6.0.1; Le X526 Build/IIXOSOP5801607291S)\",#012      \"Via: 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 aka
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (386) said Child starts


Thanks in advance,

Mahesh.

________________________________
From: Dag Haavi Finstad <daghf at varnish-software.com>
Sent: Monday, June 27, 2016 2:57 PM
To: Mahesh Ollalwar
Cc: varnish-misc at varnish-cache.org
Subject: Re: varnish core dump

Hi

I believe this might be #1539 [1], which is fixed in 4.0.3 and newer.

Please upgrade to the latest release and file a bug if you still run into this.

[1]: https://www.varnish-cache.org/trac/ticket/1539

On Fri, Jun 24, 2016 at 9:06 PM, Mahesh Ollalwar <mahesho at outlook.com> wrote:
> Can anyone help me on below error. looks like core dump is happening and
> child is getting restarted. This is happening recently without any change to
> the VCL and any system configuration.
>
> During this time the requests are getting terminated abruptly.
>
> Thanks in advance.
>
>
> Varnish version - varnish-4.0.2
>
>
> Jun 24 23:49:30 Prod-CACHE2 abrt[32341]: Saved core dump of pid 26526
> (/product/varnish/sbin/varnishd) to
> /var/spool/abrt/ccpp-2016-06-24-23:48:19-26526 (9034817536 bytes)
> Jun 24 23:49:30 Prod-CACHE2 abrtd: Directory
> 'ccpp-2016-06-24-23:48:19-26526' creation detected
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Executable
> '/product/varnish/sbin/varnishd' doesn't belong to any package and
> ProcessUnpackaged is set to 'no'
> Jun 24 23:49:31 Prod-CACHE2 abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526' exited with 1
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Deleting problem directory
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526'
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) died signal=6
> (core dumped)
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) Panic
> message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line
> 430:#012  Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread =
> (cache-worker)#012ident =
> Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012
> 0x4310e5: pan_ic+0xc5#012  0x434c30: cnt_purge+0x0#012  0x436461:
> CNT_Request+0x241#012  0x42b73b: HTTP1_Session+0x53b#012  0x439f98:
> ses_req_pool_task+0x68#012  0x43a6d4: SES_pool_accept_task+0x2a4#012
> 0x433b87: Pool_Work_Thread+0x357#012  0x4467d9: wrk_thread_real+0xb9#012
> 0x3c25e079d1: /lib64/libpthread.so.0() [0x3c25e079d1]#012  0x3c25ae8b6d:
> /lib64/libc.so.6(clone+0x6d) [0x3c25ae8b6d]#012req = 0x7f83baaf2cb0 {#012
> sp = 0x7f84dd71c430, vxid = 1108394165,  step = R_STP_LOOKUP,#012  req_body
> = R_BODY_NONE,#012  restarts = 0, esi_level = 0#012  sp = 0x7f84dd71c430
> {#012    fd = 64, vxid = 34652201,#012    client = 10.254.9.78 61217,#012
> step = S_STP_WORKING,#012  },#012  worker = 0x7f8574c0cc40 {#012    ws =
> 0x7f8574c0ce58 {#012      id = "wrk",#012      {s,f,r,e} =
> {0x7f8574c0ac30,0x7f8574c0ac30,(nil),+8192},#012    },#012  VCL::method =
> 0x0,#012  VCL::return = deliver,#012  },#012  ws = 0x7f83baaf2e48 {#012
> id = "req",#012    {s,f,r,e} = {0x7f83baaf4ca0,+1320,(nil),+57360},#012
> },#012  http[req] = {#012    ws = 0x7f83baaf2e48[req]#012      "GET",#012
> "/search-engine-web/SearchContents?action=SearchContents&moreFilters=series%3A3197&appVersion=5.2.2&query=*&startIndex=0&searchOrder=counter_week%20desc&maxResult=8&channel=ANDROID&type=CLIPS",#012
> "HTTP/1.1",#012      "host: xxx.xxx.com",#012      "Accept-Encoding:
> gzip",#012      "Akamai-Origin-Hop: 4",#012      "True-Client-IP:
> 49.15.204.238",#012      "User-Agent: Dalvik/2.1.0 (Linux; U; Android 5.1;
> 1201 Build/LMY47I)",#012      "Via: 1.1 v1-akamaitech.net(ghost)
> (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1
> v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost)
> (AkamaiGHost)",#012      "X-Aka
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: child (1277) Started
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: Child (1277) said Child
> starts
>
>
> Thanks,
>
> Mahesh.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



--
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160912/9c2d30d3/attachment.html>

From mic at strg.at  Mon Sep 12 10:10:25 2016
From: mic at strg.at (Michael Dosser)
Date: Mon, 12 Sep 2016 12:10:25 +0200
Subject: Varnish threads growing until no response
In-Reply-To: <EBAA8878-9A65-4CFC-B4FD-1A4B4D330E34@strg.at>
References: <F8E36B7D-604D-4B01-9045-FB8D71BE91E6@strg.at>
 <CAJ6ZYQwdFcgqL+BNP_VqFnfXbb6bVE28aZ6+pmfRZ1rHmi7BQA@mail.gmail.com>
 <90B82C21-4A94-4AAE-95BB-CF9623DD78F8@strg.at>
 <441BC562-AF7B-4103-81C3-82325B1B8E49@strg.at>
 <CAJ6ZYQzebBDXOKFK4pUGfmp6AXYXXCcSrnF8X0MqySLOx1c18g@mail.gmail.com>
 <7C164087-1CB2-438A-9A7F-1FAF0995C22E@strg.at>
 <EBAA8878-9A65-4CFC-B4FD-1A4B4D330E34@strg.at>
Message-ID: <CC0F02BA-0593-4D08-AD41-10071548A805@strg.at>

Hi,

Varnish is now 45 days up and running. Since that is by far the longest uptime in this configuration I think we can close this issue.

Thanks again for your help!

Best regards,
Michael Dosser

> Am 10.08.2016 um 16:23 schrieb Michael Dosser <mic at strg.at>:
> 
> Hi,
> 
> just an update - since we made those changes our Varnish instance is still up and running. I did some other changes:
> 
> - Removed ?set beresp.ttl = 0s;? in vcl_backend_response and added default_ttl=0 in Varnish startup
> - Removed the whole inline C-Code - replaced with Varnish own s-maxage handling
> 
> I?ll keep you informed if this solved our problems ?
> 
> Michael Dosser
> 
>> Am 29.07.2016 um 14:44 schrieb Michael Dosser <mic at strg.at>:
>> 
>> Hi,
>> 
>>> Am 29.07.2016 um 14:20 schrieb Guillaume Quintard <guillaume at varnish-software.com>:
>>> 
>>> Silly question, but have you tried having more threads? And at the same time, thread_pool should be set to 2, unless you have a very valid reason to do otherwise.
>> 
>> thanks for your help. I?ll try these new settings! I didn?t think about changing the startup logic and always thought our problem is coming from our quite complex VCL.
>> 
>> Michael Dosser
>> 
>> PS: I changed the startup to:
>> 
>> varnishd -f /etc/varnish/fuf.vcl \
>>      -h critbit \
>>      -a 127.0.0.1:6081 \
>>      -T 127.0.0.1:6082 \
>>      -t 120 \
>>      -S /etc/varnish/secret \
>>      -s malloc,12G \
>>      -p thread_pool_min=500 \
>>      -p thread_pool_max=3000 \
>>      -p thread_pools=2 \
>>      -p thread_pool_add_delay=0 \
>>      -p vcc_allow_inline_c=on \
>>      -p feature=+esi_disable_xml_check \
>>      -p feature=+esi_ignore_other_elements \
>>      -p feature=+esi_ignore_https"
>> 
>> -- 
>> strg.at gmbh  michael.dosser at strg.at
>>   gumpendorferstrasse 132, top 9, 1060 wien
>>   tel +43 (1) 526 56 29  mobile +43 699 1 7777 164
>> 
>> 
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> 
> -- 
> strg.at gmbh  michael.dosser at strg.at
>    gumpendorferstrasse 132, top 9, 1060 wien
>    tel +43 (1) 526 56 29  mobile +43 699 1 7777 164
> 
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

-- 
strg.at gmbh  michael.dosser at strg.at
    gumpendorferstrasse 132, top 9, 1060 wien
    tel +43 (1) 526 56 29  mobile +43 699 1 7777 164




From guillaume at varnish-software.com  Mon Sep 12 11:31:32 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Mon, 12 Sep 2016 13:31:32 +0200
Subject: Varnish threads growing until no response
In-Reply-To: <CC0F02BA-0593-4D08-AD41-10071548A805@strg.at>
References: <F8E36B7D-604D-4B01-9045-FB8D71BE91E6@strg.at>
 <CAJ6ZYQwdFcgqL+BNP_VqFnfXbb6bVE28aZ6+pmfRZ1rHmi7BQA@mail.gmail.com>
 <90B82C21-4A94-4AAE-95BB-CF9623DD78F8@strg.at>
 <441BC562-AF7B-4103-81C3-82325B1B8E49@strg.at>
 <CAJ6ZYQzebBDXOKFK4pUGfmp6AXYXXCcSrnF8X0MqySLOx1c18g@mail.gmail.com>
 <7C164087-1CB2-438A-9A7F-1FAF0995C22E@strg.at>
 <EBAA8878-9A65-4CFC-B4FD-1A4B4D330E34@strg.at>
 <CC0F02BA-0593-4D08-AD41-10071548A805@strg.at>
Message-ID: <CAJ6ZYQzFkHkS2MB0LwSTdAkV_rUgq_K-woXV7M4gm9C+ZqM=TQ@mail.gmail.com>

Glad to hear that, thanks a lot for coming back to us!

-- 
Guillaume Quintard

On Mon, Sep 12, 2016 at 12:10 PM, Michael Dosser <mic at strg.at> wrote:

> Hi,
>
> Varnish is now 45 days up and running. Since that is by far the longest
> uptime in this configuration I think we can close this issue.
>
> Thanks again for your help!
>
> Best regards,
> Michael Dosser
>
> > Am 10.08.2016 um 16:23 schrieb Michael Dosser <mic at strg.at>:
> >
> > Hi,
> >
> > just an update - since we made those changes our Varnish instance is
> still up and running. I did some other changes:
> >
> > - Removed ?set beresp.ttl = 0s;? in vcl_backend_response and added
> default_ttl=0 in Varnish startup
> > - Removed the whole inline C-Code - replaced with Varnish own s-maxage
> handling
> >
> > I?ll keep you informed if this solved our problems ?
> >
> > Michael Dosser
> >
> >> Am 29.07.2016 um 14:44 schrieb Michael Dosser <mic at strg.at>:
> >>
> >> Hi,
> >>
> >>> Am 29.07.2016 um 14:20 schrieb Guillaume Quintard <
> guillaume at varnish-software.com>:
> >>>
> >>> Silly question, but have you tried having more threads? And at the
> same time, thread_pool should be set to 2, unless you have a very valid
> reason to do otherwise.
> >>
> >> thanks for your help. I?ll try these new settings! I didn?t think about
> changing the startup logic and always thought our problem is coming from
> our quite complex VCL.
> >>
> >> Michael Dosser
> >>
> >> PS: I changed the startup to:
> >>
> >> varnishd -f /etc/varnish/fuf.vcl \
> >>      -h critbit \
> >>      -a 127.0.0.1:6081 \
> >>      -T 127.0.0.1:6082 \
> >>      -t 120 \
> >>      -S /etc/varnish/secret \
> >>      -s malloc,12G \
> >>      -p thread_pool_min=500 \
> >>      -p thread_pool_max=3000 \
> >>      -p thread_pools=2 \
> >>      -p thread_pool_add_delay=0 \
> >>      -p vcc_allow_inline_c=on \
> >>      -p feature=+esi_disable_xml_check \
> >>      -p feature=+esi_ignore_other_elements \
> >>      -p feature=+esi_ignore_https"
> >>
> >> --
> >> strg.at gmbh  michael.dosser at strg.at
> >>   gumpendorferstrasse 132, top 9, 1060 wien
> >>   tel +43 (1) 526 56 29  mobile +43 699 1 7777 164
> >>
> >>
> >> _______________________________________________
> >> varnish-misc mailing list
> >> varnish-misc at varnish-cache.org
> >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >
> > --
> > strg.at gmbh  michael.dosser at strg.at
> >    gumpendorferstrasse 132, top 9, 1060 wien
> >    tel +43 (1) 526 56 29  mobile +43 699 1 7777 164
> >
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> --
> strg.at gmbh  michael.dosser at strg.at
>     gumpendorferstrasse 132, top 9, 1060 wien
>     tel +43 (1) 526 56 29  mobile +43 699 1 7777 164
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160912/867693a6/attachment.html>

From mahesho at outlook.com  Wed Sep 14 05:14:53 2016
From: mahesho at outlook.com (Mahesh Ollalwar)
Date: Wed, 14 Sep 2016 05:14:53 +0000
Subject: varnish core dump
In-Reply-To: <1473673612.S.26929.8859.pro-236-99.1473676748.9606@webmail.rediffmail.com>
References: <PN1PR01MB0575331DC326679609EDF21CA9FF0@PN1PR01MB0575.INDPRD01.PROD.OUTLOOK.COM>,
 <1473673612.S.26929.8859.pro-236-99.1473676748.9606@webmail.rediffmail.com>
Message-ID: <PN1PR01MB0575151F3ECAEAA0A372081AA9F10@PN1PR01MB0575.INDPRD01.PROD.OUTLOOK.COM>

Thank you Anand.

we have set workspace_backend=262144 and the issue is resolved.


Thanks,

Mahesh.

________________________________
From: anand at rediff-inc.com <anand at rediff-inc.com> on behalf of Anand Shah <anand at rediff-inc.com>
Sent: Monday, September 12, 2016 4:09 PM
To: Mahesh Ollalwar
Cc: Dag Haavi Finstad ; varnish-misc at varnish-cache.org
Subject: Re: varnish core dump

In order to reduce contention between all the threads, Varnish uses workspaces which allow each thread to use or modify memory independently of other threads, which avoids locking. Try increasing session workspace and shm_workspace. Disable any VMOD's that you use.



Regards,
Anand Shah



From: Mahesh Ollalwar <mahesho at outlook.com>
Sent: Mon, 12 Sep 2016 15:16:52
To: Dag Haavi Finstad <daghf at varnish-software.com>, "varnish-misc at varnish-cache.org" <varnish-misc at varnish-cache.org>
Subject: Re: varnish core dump




Hi,



Thanks for the response.

We have upgraded to latest stable version 4.1.3 but still see panics. below are the logs.





Sep 12 12:45:18 Prod-SHARED-CACHE6 abrt[32688]: Saved core dump of pid 24551 (/product/varnish/sbin/varnishd) to /var/spool/abrt/ccpp-2016-09-12-12:45:15-24551 (585650176 bytes)
Sep 12 12:45:18 Prod-SHARED-CACHE6 abrtd: Directory 'ccpp-2016-09-12-12:45:15-24551' creation detected
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Executable '/product/varnish/sbin/varnishd' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551' exited with 1
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551'
Sep 12 12:45:19 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) not responding to CLI, killing it.
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) died signal=6 (core dumped)
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) Last panic at: Mon, 12 Sep 2016 07:15:20 GMT#012"Assert error in http_EstimateWS(), cache/cache_http.c line 793:#012  Condition((fm->hd[u].b) != 0) not true.#012thread = (cache-worker)#012version = varnish-4.1.3 revision 5e3b6d2#012ident = Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012  0x432f13: pan_ic+0x163#012  0x42dec5: http_EstimateWS+0x435#012  0x421874: vbf_beresp2obj+0x84#012  0x422cb5: vbf_stp_error+0x235#012  0x42366b: vbf_fetch_thread+0x1db#012  0x449825: WRK_Thread+0x4a5#012  0x449c5b: pool_thread+0x2b#012  0x3c25e079d1: libpthread.so<http://libpthread.so>.0() [0x3c25e079d1]#012  0x3c25ae8b6d: libc.so<http://libc.so>.6(clone+0x6d) [0x3c25ae8b6d]#012busyobj = 0x7f7f505208a0 {#012  ws = 0x7f7f50520960 {#012    OVERFLOWED id = \"Bo\",#012    {s,f,r,e} = {0x7f7f50522818,+57440,(nil),+57440},#012  },#012  refcnt = 2,#012  retries = 0, failed = 1, state = 1,#012  flags = {do_gzip, do_stream, is_gunzip},#012  director_req = 0x1a16130 {#012    vcl_name = farm_anon,#012    type = round-robin {#012    },#012  },#012  director_resp = 0x1a08e98 {#012    vcl_name = PROD_LB_BE_ANON_1,#012    type = backend {#012      display_name = boot.PROD_LB_BE_ANON_1,#012      ipv4 = 10.254.4.254<http://10.254.4.254>,#012      port = 8080,#012      hosthdr = 10.254.4.254<http://10.254.4.254>,#012      health=healthy, admin_health=probe, changed=1473664247.5,#012      n_conn = 2,#012    },#012  },#012  http[bereq] = 0x7f7f50520f28 {#012    ws[Bo] = 0x7f7f50520960,#012    hdrs {#012      \"GET\",#012      \"/AVS/besc?action=GetArrayContentList&channel=ANDROID&categoryId=7022\",#012      \"HTTP/1.1\",#012      \"host: www.example.com<http://www.example.com> \",#012      \"Accept-Encoding: gzip\",#012      \"Akamai-Origin-Hop: 4\",#012      \"True-Client-IP: 183.83.249.187<http://183.83.249.187>\",#012      \"User-Agent: Dalvik/2.1.0 (Linux; U; Android 6.0.1; Le X526 Build/IIXOSOP5801607291S)\",#012      \"Via: 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 aka
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (386) said Child starts





Thanks in advance,

Mahesh.


________________________________
From: Dag Haavi Finstad <daghf at varnish-software.com>
Sent: Monday, June 27, 2016 2:57 PM
To: Mahesh Ollalwar
Cc: varnish-misc at varnish-cache.org
Subject: Re: varnish core dump

Hi

I believe this might be #1539 [1], which is fixed in 4.0.3 and newer.

Please upgrade to the latest release and file a bug if you still run into this.

[1]: https://www.varnish-cache.org/trac/ticket/1539

On Fri, Jun 24, 2016 at 9:06 PM, Mahesh Ollalwar <mahesho at outlook.com> wrote:
> Can anyone help me on below error. looks like core dump is happening and
> child is getting restarted. This is happening recently without any change to
> the VCL and any system configuration.
>
> During this time the requests are getting terminated abruptly.
>
> Thanks in advance.
>
>
> Varnish version - varnish-4.0.2
>
>
> Jun 24 23:49:30 Prod-CACHE2 abrt[32341]: Saved core dump of pid 26526
> (/product/varnish/sbin/varnishd) to
> /var/spool/abrt/ccpp-2016-06-24-23:48:19-26526 (9034817536 bytes)
> Jun 24 23:49:30 Prod-CACHE2 abrtd: Directory
> 'ccpp-2016-06-24-23:48:19-26526' creation detected
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Executable
> '/product/varnish/sbin/varnishd' doesn't belong to any package and
> ProcessUnpackaged is set to 'no'
> Jun 24 23:49:31 Prod-CACHE2 abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526' exited with 1
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Deleting problem directory
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526'
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) died signal=6
> (core dumped)
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) Panic
> message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line
> 430:#012  Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread =
> (cache-worker)#012ident =
> Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012
> 0x4310e5: pan_ic+0xc5#012  0x434c30: cnt_purge+0x0#012  0x436461:
> CNT_Request+0x241#012  0x42b73b: HTTP1_Session+0x53b#012  0x439f98:
> ses_req_pool_task+0x68#012  0x43a6d4: SES_pool_accept_task+0x2a4#012
> 0x433b87: Pool_Work_Thread+0x357#012  0x4467d9: wrk_thread_real+0xb9#012
> 0x3c25e079d1: /lib64/libpthread.so<http://libpthread.so>.0() [0x3c25e079d1]#012  0x3c25ae8b6d:
> /lib64/libc.so<http://libc.so>.6(clone+0x6d) [0x3c25ae8b6d]#012req = 0x7f83baaf2cb0 {#012
> sp = 0x7f84dd71c430, vxid = 1108394165,  step = R_STP_LOOKUP,#012  req_body
> = R_BODY_NONE,#012  restarts = 0, esi_level = 0#012  sp = 0x7f84dd71c430
> {#012    fd = 64, vxid = 34652201,#012    client = 10.254.9.78<http://10.254.9.78> 61217,#012
> step = S_STP_WORKING,#012  },#012  worker = 0x7f8574c0cc40 {#012    ws =
> 0x7f8574c0ce58 {#012      id = "wrk",#012      {s,f,r,e} =
> {0x7f8574c0ac30,0x7f8574c0ac30,(nil),+8192},#012    },#012  VCL::method =
> 0x0,#012  VCL::return = deliver,#012  },#012  ws = 0x7f83baaf2e48 {#012
> id = "req",#012    {s,f,r,e} = {0x7f83baaf4ca0,+1320,(nil),+57360},#012
> },#012  http[req] = {#012    ws = 0x7f83baaf2e48[req]#012      "GET",#012
> "/search-engine-web/SearchContents?action=SearchContents&moreFilters=series%3A3197&appVersion=5.2.2&query=*&startIndex=0&searchOrder=counter_week%20desc&maxResult=8&channel=ANDROID&type=CLIPS",#012
> "HTTP/1.1",#012      "host: xxx.xxx.com<http://xxx.xxx.com>",#012      "Accept-Encoding:
> gzip",#012      "Akamai-Origin-Hop: 4",#012      "True-Client-IP:
> 49.15.204.238<http://49.15.204.238>",#012      "User-Agent: Dalvik/2.1.0 (Linux; U; Android 5.1;
> 1201 Build/LMY47I)",#012      "Via: 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost)
> (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1
> v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 akamai.net<http://akamai.net>(ghost)
> (AkamaiGHost)",#012      "X-Aka
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: child (1277) Started
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: Child (1277) said Child
> starts
>
>
> Thanks,
>
> Mahesh.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



--
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!
_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc<//prolinks.rediffmailpro.com/cgi-bin/prored.cgi?red=https%3A%2F%2Fwww%2Evarnish%2Dcache%2Eorg%2Flists%2Fmailman%2Flistinfo%2Fvarnish%2Dmisc&rediffng=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160914/1babd81d/attachment.html>

From sujithnss at gmail.com  Wed Sep 14 16:33:39 2016
From: sujithnss at gmail.com (sujith pv)
Date: Wed, 14 Sep 2016 22:03:39 +0530
Subject: Varnish malloc issue
In-Reply-To: <CAJ6ZYQwzkp=SznRr9jZ0kpLSwPT4+52QWX7CEwnFd1nJHu8RrQ@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <20160901231832.GA3267@optusnet.com.au>
 <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>
 <CAFzBPcUiAPkZtf-LO=r+7_FD9RV3pKNAcOzBEMQpRLFubpjkCg@mail.gmail.com>
 <CAJ6ZYQz0PV9exEzcc21sZt=Bca7CT322xhHq2OJKvYwfAKwifQ@mail.gmail.com>
 <CAFzBPcVffR6UWzCRwC4BBFWu7pGFb7QezYY4e2_D2YvxgyVN6Q@mail.gmail.com>
 <CAJ6ZYQwhLU-ckADgfTTrdN-W-GcR3VzXYjf3BPEC4G9a__96UA@mail.gmail.com>
 <CAFzBPcWo1+4iMBxzpKupx_E75hGrj4C3pbJxFqm9FfnHZh-bvQ@mail.gmail.com>
 <CAJ6ZYQziWZe6cRXOMnhRXJApYsKG53dn3oAVf258O3BQvn-3Vg@mail.gmail.com>
 <CAFzBPcUf8r5bj6681jzymkEwJcev8fcVM+RyV46Q+8m-tqpSwA@mail.gmail.com>
 <CAJ6ZYQw2RpYvtvzrMDZZc60yg2TuC4JLSx4tSkE0DMREO9LgQw@mail.gmail.com>
 <CAFzBPcV1YqGDTJYQYBv_YEa9S+dvRMKJyrpPFYTXgvzErr8WHg@mail.gmail.com>
 <CAJ6ZYQwzkp=SznRr9jZ0kpLSwPT4+52QWX7CEwnFd1nJHu8RrQ@mail.gmail.com>
Message-ID: <CAFzBPcUk2RisO0Rb+X08hHMhzF03Um2shnr5HaYQ2+7LeMhoow@mail.gmail.com>

It was showing as 0 for Transient_g_bytes and 1Gb for g_space as I
mentioned in my mail.

On Wed, Sep 14, 2016 at 7:45 PM, Guillaume Quintard <
guillaume at varnish-software.com> wrote:

> please keep using the mailing list, some other people may have ideas.
>
> what's the output of "varnishstat -1 | grep g_bytes" ?
>
> --
> Guillaume Quintard
>
> On Wed, Sep 14, 2016 at 3:50 PM, sujith pv <sujithnss at gmail.com> wrote:
>
>> Hi Guillaume
>>
>> I have tried capping Transient and I could see the respective values
>> under SMA.transient.g_bytes/g_space. But still on load I didnt see this
>> g_bytes being used and was showing 0 only.
>>
>> Best Regards
>> Sujith P V
>>
>> On Tue, Sep 6, 2016 at 7:39 PM, Guillaume Quintard <
>> guillaume at varnish-software.com> wrote:
>>
>>> Maybe, not sure. Unless you have a **good** reason to do so, use the
>>> packages.
>>>
>>> --
>>> Guillaume Quintard
>>>
>>> On Tue, Sep 6, 2016 at 3:19 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>
>>>> Thanks Guillaume. Actually in my end Varnish was compiled here and
>>>> installed (I was not part of that initially :-)). Will this have any impact
>>>> compared to as you have mentioned like try from a packaged version.
>>>>
>>>> On Tue, Sep 6, 2016 at 6:46 PM, Guillaume Quintard <
>>>> guillaume at varnish-software.com> wrote:
>>>>
>>>>> It's just a question of storage, shortlive objects got into transient
>>>>> storage instead of the one(s) you defined in the command line. But it's
>>>>> still a cache storage, based on malloc, that is by default unlimited in
>>>>> size.
>>>>>
>>>>> The rationale is that for object with such short ttl+grace+keep, a
>>>>> simple malloc is good enough.
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>> On Tue, Sep 6, 2016 at 3:04 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>>>
>>>>>> Actually I'm confused around the short lived objects. Is these
>>>>>> objects are something like cached objects or who is creating these objects.
>>>>>>
>>>>>> On Tue, Sep 6, 2016 at 1:57 PM, Guillaume Quintard <
>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>
>>>>>>> Sorry, I didn't get your question.
>>>>>>>
>>>>>>> --
>>>>>>> Guillaume Quintard
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Guillaume. May be Im asking a very basic question , but
>>>>>>>> still how this short lived objects are created and I'm not giving any
>>>>>>>> settings for the same as well.
>>>>>>>>
>>>>>>>> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>
>>>>>>>>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.ht
>>>>>>>>> ml#shortlived
>>>>>>>>>
>>>>>>>>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Guillaume. I will try the transient settings first and
>>>>>>>>>> verify the same. Also could you please elaborate on the short lived objects.
>>>>>>>>>>
>>>>>>>>>> Best Regards
>>>>>>>>>> Sujith P V
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Viktor,
>>>>>>>>>>>
>>>>>>>>>>> Have a look here https://www.varnish-cache
>>>>>>>>>>> .org/docs/trunk/users-guide/storage-backends.html#transient-
>>>>>>>>>>> storage
>>>>>>>>>>>
>>>>>>>>>>> Sujith, please try to use a packaged version from your
>>>>>>>>>>> distribution.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>>>>>>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>>>>>>>>> > Is that virtual or real memory?
>>>>>>>>>>>> > Try to cap your Transient storage to 1GB, you may be
>>>>>>>>>>>> overloaded with short
>>>>>>>>>>>> > lived objects.
>>>>>>>>>>>>
>>>>>>>>>>>> Could you elaborate on this bit more? I've got very similar
>>>>>>>>>>>> problem
>>>>>>>>>>>> here. Eg: Total memory 384g
>>>>>>>>>>>>
>>>>>>>>>>>> VIRT: 487g
>>>>>>>>>>>> RES:  311g
>>>>>>>>>>>>
>>>>>>>>>>>> free says: 46g free
>>>>>>>>>>>>
>>>>>>>>>>>> but SWAP is 91.3% used :(
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> How can this be?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> thanks
>>>>>>>>>>>>
>>>>>>>>>>>> v
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> >
>>>>>>>>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> >
>>>>>>>>>>>> > > Hi All
>>>>>>>>>>>> > >
>>>>>>>>>>>> > > I had already mailed this query long back but this time
>>>>>>>>>>>> putting in a
>>>>>>>>>>>> > > different manner.
>>>>>>>>>>>> > >
>>>>>>>>>>>> > >
>>>>>>>>>>>> > >    - We are using Varnish 4.0 in our end. We have a machine
>>>>>>>>>>>> with memory
>>>>>>>>>>>> > >    of 16G with 8G being malloc for Varnish. We have a TTL
>>>>>>>>>>>> for 2 hrs as well.
>>>>>>>>>>>> > >    - During our peak traffic, when we see the total memory
>>>>>>>>>>>> of the machine
>>>>>>>>>>>> > >    reaching 90% and like varnishd process is taking some
>>>>>>>>>>>> 89% .
>>>>>>>>>>>> > >    - So I'm just confused even though we had allocated just
>>>>>>>>>>>> 8G malloc
>>>>>>>>>>>> > >    which is like 50% of the total memory, how the process
>>>>>>>>>>>> is eating up 89% of
>>>>>>>>>>>> > >    memory and the memory is not releasing even after TTL.
>>>>>>>>>>>> > >
>>>>>>>>>>>> > >
>>>>>>>>>>>> > > Any help please...
>>>>>>>>>>>> > >
>>>>>>>>>>>> > > Best Regards
>>>>>>>>>>>> > > Sujith P V
>>>>>>>>>>>> > >
>>>>>>>>>>>> > > _______________________________________________
>>>>>>>>>>>> > > varnish-misc mailing list
>>>>>>>>>>>> > > varnish-misc at varnish-cache.org
>>>>>>>>>>>> > > https://www.varnish-cache.org/
>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>> > >
>>>>>>>>>>>>
>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>> > varnish-misc mailing list
>>>>>>>>>>>> > varnish-misc at varnish-cache.org
>>>>>>>>>>>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish
>>>>>>>>>>>> -misc
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards
>>>>>>>>>>>>
>>>>>>>>>>>> Viktor Villafuerte
>>>>>>>>>>>> Optus Internet Engineering
>>>>>>>>>>>> t: +61 2 80825265
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160914/05089516/attachment.html>

From guillaume at varnish-software.com  Wed Sep 14 17:12:08 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 14 Sep 2016 19:12:08 +0200
Subject: Varnish malloc issue
In-Reply-To: <CAFzBPcUk2RisO0Rb+X08hHMhzF03Um2shnr5HaYQ2+7LeMhoow@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <20160901231832.GA3267@optusnet.com.au>
 <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>
 <CAFzBPcUiAPkZtf-LO=r+7_FD9RV3pKNAcOzBEMQpRLFubpjkCg@mail.gmail.com>
 <CAJ6ZYQz0PV9exEzcc21sZt=Bca7CT322xhHq2OJKvYwfAKwifQ@mail.gmail.com>
 <CAFzBPcVffR6UWzCRwC4BBFWu7pGFb7QezYY4e2_D2YvxgyVN6Q@mail.gmail.com>
 <CAJ6ZYQwhLU-ckADgfTTrdN-W-GcR3VzXYjf3BPEC4G9a__96UA@mail.gmail.com>
 <CAFzBPcWo1+4iMBxzpKupx_E75hGrj4C3pbJxFqm9FfnHZh-bvQ@mail.gmail.com>
 <CAJ6ZYQziWZe6cRXOMnhRXJApYsKG53dn3oAVf258O3BQvn-3Vg@mail.gmail.com>
 <CAFzBPcUf8r5bj6681jzymkEwJcev8fcVM+RyV46Q+8m-tqpSwA@mail.gmail.com>
 <CAJ6ZYQw2RpYvtvzrMDZZc60yg2TuC4JLSx4tSkE0DMREO9LgQw@mail.gmail.com>
 <CAFzBPcV1YqGDTJYQYBv_YEa9S+dvRMKJyrpPFYTXgvzErr8WHg@mail.gmail.com>
 <CAJ6ZYQwzkp=SznRr9jZ0kpLSwPT4+52QWX7CEwnFd1nJHu8RrQ@mail.gmail.com>
 <CAFzBPcUk2RisO0Rb+X08hHMhzF03Um2shnr5HaYQ2+7LeMhoow@mail.gmail.com>
Message-ID: <CAJ6ZYQw-0fjjGrKkN+Y8O-_Yx6keXkEj2sHsyfoWVavoQyJoBw@mail.gmail.com>

What about the other g_space/g_bytes?

On Sep 14, 2016 18:33, "sujith pv" <sujithnss at gmail.com> wrote:

> It was showing as 0 for Transient_g_bytes and 1Gb for g_space as I
> mentioned in my mail.
>
> On Wed, Sep 14, 2016 at 7:45 PM, Guillaume Quintard <
> guillaume at varnish-software.com> wrote:
>
>> please keep using the mailing list, some other people may have ideas.
>>
>> what's the output of "varnishstat -1 | grep g_bytes" ?
>>
>> --
>> Guillaume Quintard
>>
>> On Wed, Sep 14, 2016 at 3:50 PM, sujith pv <sujithnss at gmail.com> wrote:
>>
>>> Hi Guillaume
>>>
>>> I have tried capping Transient and I could see the respective values
>>> under SMA.transient.g_bytes/g_space. But still on load I didnt see this
>>> g_bytes being used and was showing 0 only.
>>>
>>> Best Regards
>>> Sujith P V
>>>
>>> On Tue, Sep 6, 2016 at 7:39 PM, Guillaume Quintard <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> Maybe, not sure. Unless you have a **good** reason to do so, use the
>>>> packages.
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Tue, Sep 6, 2016 at 3:19 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>>
>>>>> Thanks Guillaume. Actually in my end Varnish was compiled here and
>>>>> installed (I was not part of that initially :-)). Will this have any impact
>>>>> compared to as you have mentioned like try from a packaged version.
>>>>>
>>>>> On Tue, Sep 6, 2016 at 6:46 PM, Guillaume Quintard <
>>>>> guillaume at varnish-software.com> wrote:
>>>>>
>>>>>> It's just a question of storage, shortlive objects got into transient
>>>>>> storage instead of the one(s) you defined in the command line. But it's
>>>>>> still a cache storage, based on malloc, that is by default unlimited in
>>>>>> size.
>>>>>>
>>>>>> The rationale is that for object with such short ttl+grace+keep, a
>>>>>> simple malloc is good enough.
>>>>>>
>>>>>> --
>>>>>> Guillaume Quintard
>>>>>>
>>>>>> On Tue, Sep 6, 2016 at 3:04 PM, sujith pv <sujithnss at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Actually I'm confused around the short lived objects. Is these
>>>>>>> objects are something like cached objects or who is creating these objects.
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 1:57 PM, Guillaume Quintard <
>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>
>>>>>>>> Sorry, I didn't get your question.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Guillaume Quintard
>>>>>>>>
>>>>>>>> On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Guillaume. May be Im asking a very basic question , but
>>>>>>>>> still how this short lived objects are created and I'm not giving any
>>>>>>>>> settings for the same as well.
>>>>>>>>>
>>>>>>>>> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>
>>>>>>>>>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.ht
>>>>>>>>>> ml#shortlived
>>>>>>>>>>
>>>>>>>>>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks Guillaume. I will try the transient settings first and
>>>>>>>>>>> verify the same. Also could you please elaborate on the short lived objects.
>>>>>>>>>>>
>>>>>>>>>>> Best Regards
>>>>>>>>>>> Sujith P V
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Viktor,
>>>>>>>>>>>>
>>>>>>>>>>>> Have a look here https://www.varnish-cache
>>>>>>>>>>>> .org/docs/trunk/users-guide/storage-backends.html#transient-
>>>>>>>>>>>> storage
>>>>>>>>>>>>
>>>>>>>>>>>> Sujith, please try to use a packaged version from your
>>>>>>>>>>>> distribution.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>>>>>>>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>>>>>>>>>> > Is that virtual or real memory?
>>>>>>>>>>>>> > Try to cap your Transient storage to 1GB, you may be
>>>>>>>>>>>>> overloaded with short
>>>>>>>>>>>>> > lived objects.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could you elaborate on this bit more? I've got very similar
>>>>>>>>>>>>> problem
>>>>>>>>>>>>> here. Eg: Total memory 384g
>>>>>>>>>>>>>
>>>>>>>>>>>>> VIRT: 487g
>>>>>>>>>>>>> RES:  311g
>>>>>>>>>>>>>
>>>>>>>>>>>>> free says: 46g free
>>>>>>>>>>>>>
>>>>>>>>>>>>> but SWAP is 91.3% used :(
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can this be?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> v
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > > Hi All
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > I had already mailed this query long back but this time
>>>>>>>>>>>>> putting in a
>>>>>>>>>>>>> > > different manner.
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > >    - We are using Varnish 4.0 in our end. We have a
>>>>>>>>>>>>> machine with memory
>>>>>>>>>>>>> > >    of 16G with 8G being malloc for Varnish. We have a TTL
>>>>>>>>>>>>> for 2 hrs as well.
>>>>>>>>>>>>> > >    - During our peak traffic, when we see the total memory
>>>>>>>>>>>>> of the machine
>>>>>>>>>>>>> > >    reaching 90% and like varnishd process is taking some
>>>>>>>>>>>>> 89% .
>>>>>>>>>>>>> > >    - So I'm just confused even though we had allocated
>>>>>>>>>>>>> just 8G malloc
>>>>>>>>>>>>> > >    which is like 50% of the total memory, how the process
>>>>>>>>>>>>> is eating up 89% of
>>>>>>>>>>>>> > >    memory and the memory is not releasing even after TTL.
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > Any help please...
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > Best Regards
>>>>>>>>>>>>> > > Sujith P V
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > _______________________________________________
>>>>>>>>>>>>> > > varnish-misc mailing list
>>>>>>>>>>>>> > > varnish-misc at varnish-cache.org
>>>>>>>>>>>>> > > https://www.varnish-cache.org/
>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>> > >
>>>>>>>>>>>>>
>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>> > varnish-misc mailing list
>>>>>>>>>>>>> > varnish-misc at varnish-cache.org
>>>>>>>>>>>>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish
>>>>>>>>>>>>> -misc
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>
>>>>>>>>>>>>> Viktor Villafuerte
>>>>>>>>>>>>> Optus Internet Engineering
>>>>>>>>>>>>> t: +61 2 80825265
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160914/53f54cc8/attachment.html>

From sujithnss at gmail.com  Wed Sep 14 18:09:25 2016
From: sujithnss at gmail.com (sujith pv)
Date: Wed, 14 Sep 2016 23:39:25 +0530
Subject: Varnish malloc issue
In-Reply-To: <CAJ6ZYQw-0fjjGrKkN+Y8O-_Yx6keXkEj2sHsyfoWVavoQyJoBw@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <20160901231832.GA3267@optusnet.com.au>
 <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>
 <CAFzBPcUiAPkZtf-LO=r+7_FD9RV3pKNAcOzBEMQpRLFubpjkCg@mail.gmail.com>
 <CAJ6ZYQz0PV9exEzcc21sZt=Bca7CT322xhHq2OJKvYwfAKwifQ@mail.gmail.com>
 <CAFzBPcVffR6UWzCRwC4BBFWu7pGFb7QezYY4e2_D2YvxgyVN6Q@mail.gmail.com>
 <CAJ6ZYQwhLU-ckADgfTTrdN-W-GcR3VzXYjf3BPEC4G9a__96UA@mail.gmail.com>
 <CAFzBPcWo1+4iMBxzpKupx_E75hGrj4C3pbJxFqm9FfnHZh-bvQ@mail.gmail.com>
 <CAJ6ZYQziWZe6cRXOMnhRXJApYsKG53dn3oAVf258O3BQvn-3Vg@mail.gmail.com>
 <CAFzBPcUf8r5bj6681jzymkEwJcev8fcVM+RyV46Q+8m-tqpSwA@mail.gmail.com>
 <CAJ6ZYQw2RpYvtvzrMDZZc60yg2TuC4JLSx4tSkE0DMREO9LgQw@mail.gmail.com>
 <CAFzBPcV1YqGDTJYQYBv_YEa9S+dvRMKJyrpPFYTXgvzErr8WHg@mail.gmail.com>
 <CAJ6ZYQwzkp=SznRr9jZ0kpLSwPT4+52QWX7CEwnFd1nJHu8RrQ@mail.gmail.com>
 <CAFzBPcUk2RisO0Rb+X08hHMhzF03Um2shnr5HaYQ2+7LeMhoow@mail.gmail.com>
 <CAJ6ZYQw-0fjjGrKkN+Y8O-_Yx6keXkEj2sHsyfoWVavoQyJoBw@mail.gmail.com>
Message-ID: <CAFzBPcUrFcu3Dv+UZpgfCBrvgUsY5Y_Pcpz+qU5CjF9vAM98YA@mail.gmail.com>

After the full load of requests..g_space became minimal and g_bytes became
like 8 Gb..I had assigned malloc as 8 gb

On 14-Sep-2016 22:42, "Guillaume Quintard" <guillaume at varnish-software.com>
wrote:

> What about the other g_space/g_bytes?
>
> On Sep 14, 2016 18:33, "sujith pv" <sujithnss at gmail.com> wrote:
>
>> It was showing as 0 for Transient_g_bytes and 1Gb for g_space as I
>> mentioned in my mail.
>>
>> On Wed, Sep 14, 2016 at 7:45 PM, Guillaume Quintard <
>> guillaume at varnish-software.com> wrote:
>>
>>> please keep using the mailing list, some other people may have ideas.
>>>
>>> what's the output of "varnishstat -1 | grep g_bytes" ?
>>>
>>> --
>>> Guillaume Quintard
>>>
>>> On Wed, Sep 14, 2016 at 3:50 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>
>>>> Hi Guillaume
>>>>
>>>> I have tried capping Transient and I could see the respective values
>>>> under SMA.transient.g_bytes/g_space. But still on load I didnt see this
>>>> g_bytes being used and was showing 0 only.
>>>>
>>>> Best Regards
>>>> Sujith P V
>>>>
>>>> On Tue, Sep 6, 2016 at 7:39 PM, Guillaume Quintard <
>>>> guillaume at varnish-software.com> wrote:
>>>>
>>>>> Maybe, not sure. Unless you have a **good** reason to do so, use the
>>>>> packages.
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>> On Tue, Sep 6, 2016 at 3:19 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>>>
>>>>>> Thanks Guillaume. Actually in my end Varnish was compiled here and
>>>>>> installed (I was not part of that initially :-)). Will this have any impact
>>>>>> compared to as you have mentioned like try from a packaged version.
>>>>>>
>>>>>> On Tue, Sep 6, 2016 at 6:46 PM, Guillaume Quintard <
>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>
>>>>>>> It's just a question of storage, shortlive objects got into
>>>>>>> transient storage instead of the one(s) you defined in the command line.
>>>>>>> But it's still a cache storage, based on malloc, that is by default
>>>>>>> unlimited in size.
>>>>>>>
>>>>>>> The rationale is that for object with such short ttl+grace+keep, a
>>>>>>> simple malloc is good enough.
>>>>>>>
>>>>>>> --
>>>>>>> Guillaume Quintard
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 3:04 PM, sujith pv <sujithnss at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Actually I'm confused around the short lived objects. Is these
>>>>>>>> objects are something like cached objects or who is creating these objects.
>>>>>>>>
>>>>>>>> On Tue, Sep 6, 2016 at 1:57 PM, Guillaume Quintard <
>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>
>>>>>>>>> Sorry, I didn't get your question.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Guillaume Quintard
>>>>>>>>>
>>>>>>>>> On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Guillaume. May be Im asking a very basic question , but
>>>>>>>>>> still how this short lived objects are created and I'm not giving any
>>>>>>>>>> settings for the same as well.
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.ht
>>>>>>>>>>> ml#shortlived
>>>>>>>>>>>
>>>>>>>>>>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks Guillaume. I will try the transient settings first and
>>>>>>>>>>>> verify the same. Also could you please elaborate on the short lived objects.
>>>>>>>>>>>>
>>>>>>>>>>>> Best Regards
>>>>>>>>>>>> Sujith P V
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Viktor,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Have a look here https://www.varnish-cache
>>>>>>>>>>>>> .org/docs/trunk/users-guide/storage-backends.html#transient-
>>>>>>>>>>>>> storage
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sujith, please try to use a packaged version from your
>>>>>>>>>>>>> distribution.
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>>>>>>>>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>>>>>>>>>>> > Is that virtual or real memory?
>>>>>>>>>>>>>> > Try to cap your Transient storage to 1GB, you may be
>>>>>>>>>>>>>> overloaded with short
>>>>>>>>>>>>>> > lived objects.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Could you elaborate on this bit more? I've got very similar
>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>> here. Eg: Total memory 384g
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> VIRT: 487g
>>>>>>>>>>>>>> RES:  311g
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> free says: 46g free
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> but SWAP is 91.3% used :(
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How can this be?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > > Hi All
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > I had already mailed this query long back but this time
>>>>>>>>>>>>>> putting in a
>>>>>>>>>>>>>> > > different manner.
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >    - We are using Varnish 4.0 in our end. We have a
>>>>>>>>>>>>>> machine with memory
>>>>>>>>>>>>>> > >    of 16G with 8G being malloc for Varnish. We have a TTL
>>>>>>>>>>>>>> for 2 hrs as well.
>>>>>>>>>>>>>> > >    - During our peak traffic, when we see the total
>>>>>>>>>>>>>> memory of the machine
>>>>>>>>>>>>>> > >    reaching 90% and like varnishd process is taking some
>>>>>>>>>>>>>> 89% .
>>>>>>>>>>>>>> > >    - So I'm just confused even though we had allocated
>>>>>>>>>>>>>> just 8G malloc
>>>>>>>>>>>>>> > >    which is like 50% of the total memory, how the process
>>>>>>>>>>>>>> is eating up 89% of
>>>>>>>>>>>>>> > >    memory and the memory is not releasing even after TTL.
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > Any help please...
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > Best Regards
>>>>>>>>>>>>>> > > Sujith P V
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > _______________________________________________
>>>>>>>>>>>>>> > > varnish-misc mailing list
>>>>>>>>>>>>>> > > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>> > > https://www.varnish-cache.org/
>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>>> > varnish-misc mailing list
>>>>>>>>>>>>>> > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>> > https://www.varnish-cache.org/
>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Viktor Villafuerte
>>>>>>>>>>>>>> Optus Internet Engineering
>>>>>>>>>>>>>> t: +61 2 80825265
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160914/4718e5a3/attachment.html>

From guillaume at varnish-software.com  Wed Sep 14 19:48:19 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 14 Sep 2016 21:48:19 +0200
Subject: Varnish malloc issue
In-Reply-To: <CAFzBPcUrFcu3Dv+UZpgfCBrvgUsY5Y_Pcpz+qU5CjF9vAM98YA@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <20160901231832.GA3267@optusnet.com.au>
 <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>
 <CAFzBPcUiAPkZtf-LO=r+7_FD9RV3pKNAcOzBEMQpRLFubpjkCg@mail.gmail.com>
 <CAJ6ZYQz0PV9exEzcc21sZt=Bca7CT322xhHq2OJKvYwfAKwifQ@mail.gmail.com>
 <CAFzBPcVffR6UWzCRwC4BBFWu7pGFb7QezYY4e2_D2YvxgyVN6Q@mail.gmail.com>
 <CAJ6ZYQwhLU-ckADgfTTrdN-W-GcR3VzXYjf3BPEC4G9a__96UA@mail.gmail.com>
 <CAFzBPcWo1+4iMBxzpKupx_E75hGrj4C3pbJxFqm9FfnHZh-bvQ@mail.gmail.com>
 <CAJ6ZYQziWZe6cRXOMnhRXJApYsKG53dn3oAVf258O3BQvn-3Vg@mail.gmail.com>
 <CAFzBPcUf8r5bj6681jzymkEwJcev8fcVM+RyV46Q+8m-tqpSwA@mail.gmail.com>
 <CAJ6ZYQw2RpYvtvzrMDZZc60yg2TuC4JLSx4tSkE0DMREO9LgQw@mail.gmail.com>
 <CAFzBPcV1YqGDTJYQYBv_YEa9S+dvRMKJyrpPFYTXgvzErr8WHg@mail.gmail.com>
 <CAJ6ZYQwzkp=SznRr9jZ0kpLSwPT4+52QWX7CEwnFd1nJHu8RrQ@mail.gmail.com>
 <CAFzBPcUk2RisO0Rb+X08hHMhzF03Um2shnr5HaYQ2+7LeMhoow@mail.gmail.com>
 <CAJ6ZYQw-0fjjGrKkN+Y8O-_Yx6keXkEj2sHsyfoWVavoQyJoBw@mail.gmail.com>
 <CAFzBPcUrFcu3Dv+UZpgfCBrvgUsY5Y_Pcpz+qU5CjF9vAM98YA@mail.gmail.com>
Message-ID: <CAJ6ZYQxtCX0y=p4KZwoOkEmQLt-3optjNuB69cuHpRop6Bwsqg@mail.gmail.com>

Have you tried the packages from your distribution?

On Sep 14, 2016 20:09, "sujith pv" <sujithnss at gmail.com> wrote:

> After the full load of requests..g_space became minimal and g_bytes became
> like 8 Gb..I had assigned malloc as 8 gb
>
> On 14-Sep-2016 22:42, "Guillaume Quintard" <guillaume at varnish-software.com>
> wrote:
>
>> What about the other g_space/g_bytes?
>>
>> On Sep 14, 2016 18:33, "sujith pv" <sujithnss at gmail.com> wrote:
>>
>>> It was showing as 0 for Transient_g_bytes and 1Gb for g_space as I
>>> mentioned in my mail.
>>>
>>> On Wed, Sep 14, 2016 at 7:45 PM, Guillaume Quintard <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> please keep using the mailing list, some other people may have ideas.
>>>>
>>>> what's the output of "varnishstat -1 | grep g_bytes" ?
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Wed, Sep 14, 2016 at 3:50 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>>
>>>>> Hi Guillaume
>>>>>
>>>>> I have tried capping Transient and I could see the respective values
>>>>> under SMA.transient.g_bytes/g_space. But still on load I didnt see this
>>>>> g_bytes being used and was showing 0 only.
>>>>>
>>>>> Best Regards
>>>>> Sujith P V
>>>>>
>>>>> On Tue, Sep 6, 2016 at 7:39 PM, Guillaume Quintard <
>>>>> guillaume at varnish-software.com> wrote:
>>>>>
>>>>>> Maybe, not sure. Unless you have a **good** reason to do so, use the
>>>>>> packages.
>>>>>>
>>>>>> --
>>>>>> Guillaume Quintard
>>>>>>
>>>>>> On Tue, Sep 6, 2016 at 3:19 PM, sujith pv <sujithnss at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Guillaume. Actually in my end Varnish was compiled here and
>>>>>>> installed (I was not part of that initially :-)). Will this have any impact
>>>>>>> compared to as you have mentioned like try from a packaged version.
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 6:46 PM, Guillaume Quintard <
>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>
>>>>>>>> It's just a question of storage, shortlive objects got into
>>>>>>>> transient storage instead of the one(s) you defined in the command line.
>>>>>>>> But it's still a cache storage, based on malloc, that is by default
>>>>>>>> unlimited in size.
>>>>>>>>
>>>>>>>> The rationale is that for object with such short ttl+grace+keep, a
>>>>>>>> simple malloc is good enough.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Guillaume Quintard
>>>>>>>>
>>>>>>>> On Tue, Sep 6, 2016 at 3:04 PM, sujith pv <sujithnss at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Actually I'm confused around the short lived objects. Is these
>>>>>>>>> objects are something like cached objects or who is creating these objects.
>>>>>>>>>
>>>>>>>>> On Tue, Sep 6, 2016 at 1:57 PM, Guillaume Quintard <
>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>
>>>>>>>>>> Sorry, I didn't get your question.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks Guillaume. May be Im asking a very basic question , but
>>>>>>>>>>> still how this short lived objects are created and I'm not giving any
>>>>>>>>>>> settings for the same as well.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.ht
>>>>>>>>>>>> ml#shortlived
>>>>>>>>>>>>
>>>>>>>>>>>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Guillaume. I will try the transient settings first and
>>>>>>>>>>>>> verify the same. Also could you please elaborate on the short lived objects.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>> Sujith P V
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Viktor,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have a look here https://www.varnish-cache
>>>>>>>>>>>>>> .org/docs/trunk/users-guide/storage-backends.html#transient-
>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sujith, please try to use a packaged version from your
>>>>>>>>>>>>>> distribution.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>>>>>>>>>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>>>>>>>>>>>> > Is that virtual or real memory?
>>>>>>>>>>>>>>> > Try to cap your Transient storage to 1GB, you may be
>>>>>>>>>>>>>>> overloaded with short
>>>>>>>>>>>>>>> > lived objects.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could you elaborate on this bit more? I've got very similar
>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>> here. Eg: Total memory 384g
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> VIRT: 487g
>>>>>>>>>>>>>>> RES:  311g
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> free says: 46g free
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> but SWAP is 91.3% used :(
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How can this be?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > > Hi All
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > I had already mailed this query long back but this time
>>>>>>>>>>>>>>> putting in a
>>>>>>>>>>>>>>> > > different manner.
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >    - We are using Varnish 4.0 in our end. We have a
>>>>>>>>>>>>>>> machine with memory
>>>>>>>>>>>>>>> > >    of 16G with 8G being malloc for Varnish. We have a
>>>>>>>>>>>>>>> TTL for 2 hrs as well.
>>>>>>>>>>>>>>> > >    - During our peak traffic, when we see the total
>>>>>>>>>>>>>>> memory of the machine
>>>>>>>>>>>>>>> > >    reaching 90% and like varnishd process is taking some
>>>>>>>>>>>>>>> 89% .
>>>>>>>>>>>>>>> > >    - So I'm just confused even though we had allocated
>>>>>>>>>>>>>>> just 8G malloc
>>>>>>>>>>>>>>> > >    which is like 50% of the total memory, how the
>>>>>>>>>>>>>>> process is eating up 89% of
>>>>>>>>>>>>>>> > >    memory and the memory is not releasing even after TTL.
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > Any help please...
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > Best Regards
>>>>>>>>>>>>>>> > > Sujith P V
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > _______________________________________________
>>>>>>>>>>>>>>> > > varnish-misc mailing list
>>>>>>>>>>>>>>> > > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>>> > > https://www.varnish-cache.org/
>>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>>>> > varnish-misc mailing list
>>>>>>>>>>>>>>> > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>>> > https://www.varnish-cache.org/
>>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Viktor Villafuerte
>>>>>>>>>>>>>>> Optus Internet Engineering
>>>>>>>>>>>>>>> t: +61 2 80825265
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160914/09e1b909/attachment.html>

From carlalexander at globalvoices.org  Thu Sep 15 00:45:22 2016
From: carlalexander at globalvoices.org (Carl Alexander)
Date: Wed, 14 Sep 2016 17:45:22 -0700
Subject: Incomplete responses for some requests
Message-ID: <CADOiiWpAwVP3bpVXoKVWcvv=FcF34+=DU2kL9Yg6wfTHpsnqNQ@mail.gmail.com>

Hi everyone,

I'm having a weird issue that I have no idea how to solve. I have a setup
where I have varnish sitting between two vhosts. An SSL termination proxy
which forwards requests to varnish who forwards them to a backend.

The setup works perfectly for 99% of pages. But, for certain pages, varnish
seems to choke and only returns part of the html response. This can happen
consistently or at random for the same page. It also varies between
browsers sometimes.

When digging into varnishlog, I can find this FetchError for the backend:

-   Fetch_Body     4 eof stream
-   FetchError     Resource temporarily unavailable
-   FetchError     eof socket fail
-   BackendClose   24 boot.webserver
-   BereqAcct      1274 0 1274 333 32554 32887
-   End


So I figured that maybe it's a nginx issue. When I turn on debug logs,
nginx tells me that it's varnish that closed the connection. It was never
able to send the complete html response back (that's probably why it's
truncated). Here's the nginx logs:


2016/09/14 23:49:08 [info] 24670#24670: *566 epoll_wait() reported that
client prematurely closed connection, so upstream connection is closed too
while reading upstream


So on one side, I have varnish saying that nginx became unavailable. And on
the other, I have nginx telling me that varnish closed the connection. I
don't know which one is right.

But if I remove varnish and send the proxy request to the backend directly,
everything works. So the signs seem to point to varnish, but I can't seem
to find what's causing this. It started on Varnish 4.1.1, but upgrading to
4.1.3 didn't change anything.

I'm also not sure how to troubleshoot this further. I tried to look at the
varnish source code. The FetchError is just a generic C error so I don't
have much else to go on.

I've also attached the output of "param.show" below:

accept_filter              off [bool] (default)
acceptor_sleep_decay       0.9 (default)
acceptor_sleep_incr        0.000 [seconds] (default)
acceptor_sleep_max         0.050 [seconds] (default)
auto_restart               on [bool] (default)
backend_idle_timeout       60.000 [seconds] (default)
ban_dups                   on [bool] (default)
ban_lurker_age             60.000 [seconds] (default)
ban_lurker_batch           1000 (default)
ban_lurker_sleep           0.010 [seconds] (default)
between_bytes_timeout      60.000 [seconds] (default)
cc_command                 "exec gcc -std=gnu99 -g -O2 -fstack-protector
--param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -Werror
-Wno-error=unused-result -pthread -fpic -shared -Wl,-x -o %o %s" (default)
cli_buffer                 8k [bytes] (default)
cli_limit                  48k [bytes] (default)
cli_timeout                60.000 [seconds] (default)
clock_skew                 10 [seconds] (default)
connect_timeout            3.500 [seconds] (default)
critbit_cooloff            180.000 [seconds] (default)
debug                      none (default)
default_grace              10.000 [seconds] (default)
default_keep               0.000 [seconds] (default)
default_ttl                3600.000 [seconds]
feature                    none (default)
fetch_chunksize            16k [bytes] (default)
fetch_maxchunksize         0.25G [bytes] (default)
first_byte_timeout         60.000 [seconds] (default)
gzip_buffer                32k [bytes] (default)
gzip_level                 6 (default)
gzip_memlevel              8 (default)
http_gzip_support          on [bool] (default)
http_max_hdr               64 [header lines] (default)
http_range_support         on [bool] (default)
http_req_hdr_len           8k [bytes] (default)
http_req_size              32k [bytes] (default)
http_resp_hdr_len          8k [bytes] (default)
http_resp_size             32k [bytes] (default)
idle_send_timeout          60.000 [seconds] (default)
listen_depth               1024 [connections] (default)
lru_interval               2.000 [seconds] (default)
max_esi_depth              5 [levels] (default)
max_restarts               4 [restarts] (default)
max_retries                4 [retries] (default)
nuke_limit                 50 [allocations] (default)
pcre_match_limit           10000 (default)
pcre_match_limit_recursion 20 (default)
ping_interval              3 [seconds] (default)
pipe_timeout               60.000 [seconds] (default)
pool_req                   10,100,10 (default)
pool_sess                  10,100,10 (default)
pool_vbo                   10,100,10 (default)
prefer_ipv6                off [bool] (default)
rush_exponent              3 [requests per request] (default)
send_timeout               600.000 [seconds] (default)
session_max                100000 [sessions] (default)
shm_reclen                 255b [bytes] (default)
shortlived                 10.000 [seconds] (default)
sigsegv_handler            on [bool] (default)
syslog_cli_traffic         on [bool] (default)
tcp_fastopen               off [bool] (default)
tcp_keepalive_intvl        75.000 [seconds] (default)
tcp_keepalive_probes       9 [probes] (default)
tcp_keepalive_time         7200.000 [seconds] (default)
thread_pool_add_delay      0.000 [seconds] (default)
thread_pool_destroy_delay  1.000 [seconds] (default)
thread_pool_fail_delay     0.200 [seconds] (default)
thread_pool_max            5000 [threads] (default)
thread_pool_min            100 [threads] (default)
thread_pool_stack          48k [bytes] (default)
thread_pool_timeout        300.000 [seconds] (default)
thread_pools               2 [pools] (default)
thread_queue_limit         20 (default)
thread_stats_rate          10 [requests] (default)
timeout_idle               5.000 [seconds] (default)
timeout_linger             0.050 [seconds] (default)
vcc_allow_inline_c         off [bool] (default)
vcc_err_unref              on [bool] (default)
vcc_unsafe_path            on [bool] (default)
vcl_cooldown               600.000 [seconds] (default)
vcl_dir                    /etc/varnish (default)
vmod_dir                   /usr/lib/varnish/vmods (default)
vsl_buffer                 4k [bytes] (default)
vsl_mask                   -VCL_trace,-WorkThread,-Hash,-VfpAcct (default)
vsl_reclen                 255b [bytes] (default)
vsl_space                  80M [bytes] (default)
vsm_free_cooldown          60.000 [seconds] (default)
vsm_space                  1M [bytes] (default)
workspace_backend          64k [bytes] (default)
workspace_client           64k [bytes] (default)
workspace_session          0.50k [bytes] (default)
workspace_thread           2k [bytes] (default)



Thanks in advance for any help,
Carl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160914/990987a5/attachment.html>

From sujithnss at gmail.com  Thu Sep 15 01:26:35 2016
From: sujithnss at gmail.com (sujith pv)
Date: Thu, 15 Sep 2016 06:56:35 +0530
Subject: Varnish malloc issue
In-Reply-To: <CAJ6ZYQxtCX0y=p4KZwoOkEmQLt-3optjNuB69cuHpRop6Bwsqg@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <20160901231832.GA3267@optusnet.com.au>
 <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>
 <CAFzBPcUiAPkZtf-LO=r+7_FD9RV3pKNAcOzBEMQpRLFubpjkCg@mail.gmail.com>
 <CAJ6ZYQz0PV9exEzcc21sZt=Bca7CT322xhHq2OJKvYwfAKwifQ@mail.gmail.com>
 <CAFzBPcVffR6UWzCRwC4BBFWu7pGFb7QezYY4e2_D2YvxgyVN6Q@mail.gmail.com>
 <CAJ6ZYQwhLU-ckADgfTTrdN-W-GcR3VzXYjf3BPEC4G9a__96UA@mail.gmail.com>
 <CAFzBPcWo1+4iMBxzpKupx_E75hGrj4C3pbJxFqm9FfnHZh-bvQ@mail.gmail.com>
 <CAJ6ZYQziWZe6cRXOMnhRXJApYsKG53dn3oAVf258O3BQvn-3Vg@mail.gmail.com>
 <CAFzBPcUf8r5bj6681jzymkEwJcev8fcVM+RyV46Q+8m-tqpSwA@mail.gmail.com>
 <CAJ6ZYQw2RpYvtvzrMDZZc60yg2TuC4JLSx4tSkE0DMREO9LgQw@mail.gmail.com>
 <CAFzBPcV1YqGDTJYQYBv_YEa9S+dvRMKJyrpPFYTXgvzErr8WHg@mail.gmail.com>
 <CAJ6ZYQwzkp=SznRr9jZ0kpLSwPT4+52QWX7CEwnFd1nJHu8RrQ@mail.gmail.com>
 <CAFzBPcUk2RisO0Rb+X08hHMhzF03Um2shnr5HaYQ2+7LeMhoow@mail.gmail.com>
 <CAJ6ZYQw-0fjjGrKkN+Y8O-_Yx6keXkEj2sHsyfoWVavoQyJoBw@mail.gmail.com>
 <CAFzBPcUrFcu3Dv+UZpgfCBrvgUsY5Y_Pcpz+qU5CjF9vAM98YA@mail.gmail.com>
 <CAJ6ZYQxtCX0y=p4KZwoOkEmQLt-3optjNuB69cuHpRop6Bwsqg@mail.gmail.com>
Message-ID: <CAFzBPcV4C3BwvUxQPMrV1Kv9G-C3ZgJa5X95ET-EJLODLnv03w@mail.gmail.com>

No..I'm planning to try that next. So during that time also so do we need
to cap transient storage ??

On 15-Sep-2016 01:18, "Guillaume Quintard" <guillaume at varnish-software.com>
wrote:

> Have you tried the packages from your distribution?
>
> On Sep 14, 2016 20:09, "sujith pv" <sujithnss at gmail.com> wrote:
>
>> After the full load of requests..g_space became minimal and g_bytes
>> became like 8 Gb..I had assigned malloc as 8 gb
>>
>> On 14-Sep-2016 22:42, "Guillaume Quintard" <guillaume at varnish-software.co
>> m> wrote:
>>
>>> What about the other g_space/g_bytes?
>>>
>>> On Sep 14, 2016 18:33, "sujith pv" <sujithnss at gmail.com> wrote:
>>>
>>>> It was showing as 0 for Transient_g_bytes and 1Gb for g_space as I
>>>> mentioned in my mail.
>>>>
>>>> On Wed, Sep 14, 2016 at 7:45 PM, Guillaume Quintard <
>>>> guillaume at varnish-software.com> wrote:
>>>>
>>>>> please keep using the mailing list, some other people may have ideas.
>>>>>
>>>>> what's the output of "varnishstat -1 | grep g_bytes" ?
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>> On Wed, Sep 14, 2016 at 3:50 PM, sujith pv <sujithnss at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Guillaume
>>>>>>
>>>>>> I have tried capping Transient and I could see the respective values
>>>>>> under SMA.transient.g_bytes/g_space. But still on load I didnt see this
>>>>>> g_bytes being used and was showing 0 only.
>>>>>>
>>>>>> Best Regards
>>>>>> Sujith P V
>>>>>>
>>>>>> On Tue, Sep 6, 2016 at 7:39 PM, Guillaume Quintard <
>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>
>>>>>>> Maybe, not sure. Unless you have a **good** reason to do so, use the
>>>>>>> packages.
>>>>>>>
>>>>>>> --
>>>>>>> Guillaume Quintard
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 3:19 PM, sujith pv <sujithnss at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Guillaume. Actually in my end Varnish was compiled here and
>>>>>>>> installed (I was not part of that initially :-)). Will this have any impact
>>>>>>>> compared to as you have mentioned like try from a packaged version.
>>>>>>>>
>>>>>>>> On Tue, Sep 6, 2016 at 6:46 PM, Guillaume Quintard <
>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>
>>>>>>>>> It's just a question of storage, shortlive objects got into
>>>>>>>>> transient storage instead of the one(s) you defined in the command line.
>>>>>>>>> But it's still a cache storage, based on malloc, that is by default
>>>>>>>>> unlimited in size.
>>>>>>>>>
>>>>>>>>> The rationale is that for object with such short ttl+grace+keep, a
>>>>>>>>> simple malloc is good enough.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Guillaume Quintard
>>>>>>>>>
>>>>>>>>> On Tue, Sep 6, 2016 at 3:04 PM, sujith pv <sujithnss at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Actually I'm confused around the short lived objects. Is these
>>>>>>>>>> objects are something like cached objects or who is creating these objects.
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 6, 2016 at 1:57 PM, Guillaume Quintard <
>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sorry, I didn't get your question.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks Guillaume. May be Im asking a very basic question , but
>>>>>>>>>>>> still how this short lived objects are created and I'm not giving any
>>>>>>>>>>>> settings for the same as well.
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.ht
>>>>>>>>>>>>> ml#shortlived
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Guillaume. I will try the transient settings first
>>>>>>>>>>>>>> and verify the same. Also could you please elaborate on the short lived
>>>>>>>>>>>>>> objects.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>>> Sujith P V
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Viktor,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Have a look here https://www.varnish-cache
>>>>>>>>>>>>>>> .org/docs/trunk/users-guide/storage-backends.html#transient-
>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sujith, please try to use a packaged version from your
>>>>>>>>>>>>>>> distribution.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>>>>>>>>>>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>>>>>>>>>>>>> > Is that virtual or real memory?
>>>>>>>>>>>>>>>> > Try to cap your Transient storage to 1GB, you may be
>>>>>>>>>>>>>>>> overloaded with short
>>>>>>>>>>>>>>>> > lived objects.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Could you elaborate on this bit more? I've got very similar
>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>> here. Eg: Total memory 384g
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> VIRT: 487g
>>>>>>>>>>>>>>>> RES:  311g
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> free says: 46g free
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> but SWAP is 91.3% used :(
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> How can this be?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > > Hi All
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > > I had already mailed this query long back but this time
>>>>>>>>>>>>>>>> putting in a
>>>>>>>>>>>>>>>> > > different manner.
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > >    - We are using Varnish 4.0 in our end. We have a
>>>>>>>>>>>>>>>> machine with memory
>>>>>>>>>>>>>>>> > >    of 16G with 8G being malloc for Varnish. We have a
>>>>>>>>>>>>>>>> TTL for 2 hrs as well.
>>>>>>>>>>>>>>>> > >    - During our peak traffic, when we see the total
>>>>>>>>>>>>>>>> memory of the machine
>>>>>>>>>>>>>>>> > >    reaching 90% and like varnishd process is taking
>>>>>>>>>>>>>>>> some 89% .
>>>>>>>>>>>>>>>> > >    - So I'm just confused even though we had allocated
>>>>>>>>>>>>>>>> just 8G malloc
>>>>>>>>>>>>>>>> > >    which is like 50% of the total memory, how the
>>>>>>>>>>>>>>>> process is eating up 89% of
>>>>>>>>>>>>>>>> > >    memory and the memory is not releasing even after
>>>>>>>>>>>>>>>> TTL.
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > > Any help please...
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > > Best Regards
>>>>>>>>>>>>>>>> > > Sujith P V
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > > _______________________________________________
>>>>>>>>>>>>>>>> > > varnish-misc mailing list
>>>>>>>>>>>>>>>> > > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>>>> > > https://www.varnish-cache.org/
>>>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>>>>> > varnish-misc mailing list
>>>>>>>>>>>>>>>> > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>>>> > https://www.varnish-cache.org/
>>>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Viktor Villafuerte
>>>>>>>>>>>>>>>> Optus Internet Engineering
>>>>>>>>>>>>>>>> t: +61 2 80825265
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160915/c1984966/attachment.html>

From guillaume at varnish-software.com  Thu Sep 15 06:48:44 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Thu, 15 Sep 2016 08:48:44 +0200
Subject: Varnish malloc issue
In-Reply-To: <CAFzBPcV4C3BwvUxQPMrV1Kv9G-C3ZgJa5X95ET-EJLODLnv03w@mail.gmail.com>
References: <CAFzBPcVyGGtfWqc0zH34m0Cp24tZ71=0ew9af=gzFNPwTHZFmw@mail.gmail.com>
 <CAJ6ZYQxmwkfy26ebQ5v2J5n5gayJVrtR_0VPjbJdj8_G8ZmkdA@mail.gmail.com>
 <20160901231832.GA3267@optusnet.com.au>
 <CAJ6ZYQy5F_E1i0u0Dqy6EeHOPJQP5S6sBKmfRt+pAQC4ZUStjw@mail.gmail.com>
 <CAFzBPcUiAPkZtf-LO=r+7_FD9RV3pKNAcOzBEMQpRLFubpjkCg@mail.gmail.com>
 <CAJ6ZYQz0PV9exEzcc21sZt=Bca7CT322xhHq2OJKvYwfAKwifQ@mail.gmail.com>
 <CAFzBPcVffR6UWzCRwC4BBFWu7pGFb7QezYY4e2_D2YvxgyVN6Q@mail.gmail.com>
 <CAJ6ZYQwhLU-ckADgfTTrdN-W-GcR3VzXYjf3BPEC4G9a__96UA@mail.gmail.com>
 <CAFzBPcWo1+4iMBxzpKupx_E75hGrj4C3pbJxFqm9FfnHZh-bvQ@mail.gmail.com>
 <CAJ6ZYQziWZe6cRXOMnhRXJApYsKG53dn3oAVf258O3BQvn-3Vg@mail.gmail.com>
 <CAFzBPcUf8r5bj6681jzymkEwJcev8fcVM+RyV46Q+8m-tqpSwA@mail.gmail.com>
 <CAJ6ZYQw2RpYvtvzrMDZZc60yg2TuC4JLSx4tSkE0DMREO9LgQw@mail.gmail.com>
 <CAFzBPcV1YqGDTJYQYBv_YEa9S+dvRMKJyrpPFYTXgvzErr8WHg@mail.gmail.com>
 <CAJ6ZYQwzkp=SznRr9jZ0kpLSwPT4+52QWX7CEwnFd1nJHu8RrQ@mail.gmail.com>
 <CAFzBPcUk2RisO0Rb+X08hHMhzF03Um2shnr5HaYQ2+7LeMhoow@mail.gmail.com>
 <CAJ6ZYQw-0fjjGrKkN+Y8O-_Yx6keXkEj2sHsyfoWVavoQyJoBw@mail.gmail.com>
 <CAFzBPcUrFcu3Dv+UZpgfCBrvgUsY5Y_Pcpz+qU5CjF9vAM98YA@mail.gmail.com>
 <CAJ6ZYQxtCX0y=p4KZwoOkEmQLt-3optjNuB69cuHpRop6Bwsqg@mail.gmail.com>
 <CAFzBPcV4C3BwvUxQPMrV1Kv9G-C3ZgJa5X95ET-EJLODLnv03w@mail.gmail.com>
Message-ID: <CAJ6ZYQxU7W2oPc_Z2qJHz2asomNNP9Q=+08wLHGqVZXi419Jqg@mail.gmail.com>

That can't, hurt.

On Sep 15, 2016 03:31, "sujith pv" <sujithnss at gmail.com> wrote:

> No..I'm planning to try that next. So during that time also so do we need
> to cap transient storage ??
>
> On 15-Sep-2016 01:18, "Guillaume Quintard" <guillaume at varnish-software.com>
> wrote:
>
>> Have you tried the packages from your distribution?
>>
>> On Sep 14, 2016 20:09, "sujith pv" <sujithnss at gmail.com> wrote:
>>
>>> After the full load of requests..g_space became minimal and g_bytes
>>> became like 8 Gb..I had assigned malloc as 8 gb
>>>
>>> On 14-Sep-2016 22:42, "Guillaume Quintard" <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> What about the other g_space/g_bytes?
>>>>
>>>> On Sep 14, 2016 18:33, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>
>>>>> It was showing as 0 for Transient_g_bytes and 1Gb for g_space as I
>>>>> mentioned in my mail.
>>>>>
>>>>> On Wed, Sep 14, 2016 at 7:45 PM, Guillaume Quintard <
>>>>> guillaume at varnish-software.com> wrote:
>>>>>
>>>>>> please keep using the mailing list, some other people may have ideas.
>>>>>>
>>>>>> what's the output of "varnishstat -1 | grep g_bytes" ?
>>>>>>
>>>>>> --
>>>>>> Guillaume Quintard
>>>>>>
>>>>>> On Wed, Sep 14, 2016 at 3:50 PM, sujith pv <sujithnss at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Guillaume
>>>>>>>
>>>>>>> I have tried capping Transient and I could see the respective values
>>>>>>> under SMA.transient.g_bytes/g_space. But still on load I didnt see this
>>>>>>> g_bytes being used and was showing 0 only.
>>>>>>>
>>>>>>> Best Regards
>>>>>>> Sujith P V
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 7:39 PM, Guillaume Quintard <
>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>
>>>>>>>> Maybe, not sure. Unless you have a **good** reason to do so, use
>>>>>>>> the packages.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Guillaume Quintard
>>>>>>>>
>>>>>>>> On Tue, Sep 6, 2016 at 3:19 PM, sujith pv <sujithnss at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Guillaume. Actually in my end Varnish was compiled here and
>>>>>>>>> installed (I was not part of that initially :-)). Will this have any impact
>>>>>>>>> compared to as you have mentioned like try from a packaged version.
>>>>>>>>>
>>>>>>>>> On Tue, Sep 6, 2016 at 6:46 PM, Guillaume Quintard <
>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>
>>>>>>>>>> It's just a question of storage, shortlive objects got into
>>>>>>>>>> transient storage instead of the one(s) you defined in the command line.
>>>>>>>>>> But it's still a cache storage, based on malloc, that is by default
>>>>>>>>>> unlimited in size.
>>>>>>>>>>
>>>>>>>>>> The rationale is that for object with such short ttl+grace+keep,
>>>>>>>>>> a simple malloc is good enough.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 6, 2016 at 3:04 PM, sujith pv <sujithnss at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Actually I'm confused around the short lived objects. Is these
>>>>>>>>>>> objects are something like cached objects or who is creating these objects.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 6, 2016 at 1:57 PM, Guillaume Quintard <
>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sorry, I didn't get your question.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Guillaume. May be Im asking a very basic question , but
>>>>>>>>>>>>> still how this short lived objects are created and I'm not giving any
>>>>>>>>>>>>> settings for the same as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
>>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.ht
>>>>>>>>>>>>>> ml#shortlived
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks Guillaume. I will try the transient settings first
>>>>>>>>>>>>>>> and verify the same. Also could you please elaborate on the short lived
>>>>>>>>>>>>>>> objects.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>>>> Sujith P V
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>>>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Viktor,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Have a look here https://www.varnish-cache
>>>>>>>>>>>>>>>> .org/docs/trunk/users-guide/st
>>>>>>>>>>>>>>>> orage-backends.html#transient-storage
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sujith, please try to use a packaged version from your
>>>>>>>>>>>>>>>> distribution.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>>>>>>>>>>>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>>>>>>>>>>>>>> > Is that virtual or real memory?
>>>>>>>>>>>>>>>>> > Try to cap your Transient storage to 1GB, you may be
>>>>>>>>>>>>>>>>> overloaded with short
>>>>>>>>>>>>>>>>> > lived objects.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Could you elaborate on this bit more? I've got very
>>>>>>>>>>>>>>>>> similar problem
>>>>>>>>>>>>>>>>> here. Eg: Total memory 384g
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> VIRT: 487g
>>>>>>>>>>>>>>>>> RES:  311g
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> free says: 46g free
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> but SWAP is 91.3% used :(
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How can this be?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > > Hi All
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > I had already mailed this query long back but this
>>>>>>>>>>>>>>>>> time putting in a
>>>>>>>>>>>>>>>>> > > different manner.
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >    - We are using Varnish 4.0 in our end. We have a
>>>>>>>>>>>>>>>>> machine with memory
>>>>>>>>>>>>>>>>> > >    of 16G with 8G being malloc for Varnish. We have a
>>>>>>>>>>>>>>>>> TTL for 2 hrs as well.
>>>>>>>>>>>>>>>>> > >    - During our peak traffic, when we see the total
>>>>>>>>>>>>>>>>> memory of the machine
>>>>>>>>>>>>>>>>> > >    reaching 90% and like varnishd process is taking
>>>>>>>>>>>>>>>>> some 89% .
>>>>>>>>>>>>>>>>> > >    - So I'm just confused even though we had allocated
>>>>>>>>>>>>>>>>> just 8G malloc
>>>>>>>>>>>>>>>>> > >    which is like 50% of the total memory, how the
>>>>>>>>>>>>>>>>> process is eating up 89% of
>>>>>>>>>>>>>>>>> > >    memory and the memory is not releasing even after
>>>>>>>>>>>>>>>>> TTL.
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > Any help please...
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > Best Regards
>>>>>>>>>>>>>>>>> > > Sujith P V
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > _______________________________________________
>>>>>>>>>>>>>>>>> > > varnish-misc mailing list
>>>>>>>>>>>>>>>>> > > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>>>>> > > https://www.varnish-cache.org/
>>>>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>>>>>> > varnish-misc mailing list
>>>>>>>>>>>>>>>>> > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>>>>> > https://www.varnish-cache.org/
>>>>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Viktor Villafuerte
>>>>>>>>>>>>>>>>> Optus Internet Engineering
>>>>>>>>>>>>>>>>> t: +61 2 80825265
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160915/799ff909/attachment.html>

From carlalexander at globalvoices.org  Fri Sep 16 17:54:38 2016
From: carlalexander at globalvoices.org (Carl Alexander)
Date: Fri, 16 Sep 2016 10:54:38 -0700
Subject: Incomplete responses for some requests
In-Reply-To: <CADOiiWpAwVP3bpVXoKVWcvv=FcF34+=DU2kL9Yg6wfTHpsnqNQ@mail.gmail.com>
References: <CADOiiWpAwVP3bpVXoKVWcvv=FcF34+=DU2kL9Yg6wfTHpsnqNQ@mail.gmail.com>
Message-ID: <CADOiiWrq2ie__agmUBCpgjY_=hin1POy8LNBYTvpPj9Xc0FfJA@mail.gmail.com>

I've managed to find a workaround to the issue. If I use "return(pipe)"
instead of "return(pass)" in my vcl, the problem disappears. That said,
this isn't ideal at all. I shouldn't have to use pipe for HTML content.

So this gives credence that something is going on inside Varnish. Should I
create a ticket for this? I've hit the limit of my ability troubleshooting
this at the moment.

On Wed, Sep 14, 2016 at 5:45 PM, Carl Alexander <
carlalexander at globalvoices.org> wrote:

> Hi everyone,
>
> I'm having a weird issue that I have no idea how to solve. I have a setup
> where I have varnish sitting between two vhosts. An SSL termination proxy
> which forwards requests to varnish who forwards them to a backend.
>
> The setup works perfectly for 99% of pages. But, for certain pages,
> varnish seems to choke and only returns part of the html response. This can
> happen consistently or at random for the same page. It also varies between
> browsers sometimes.
>
> When digging into varnishlog, I can find this FetchError for the backend:
>
> -   Fetch_Body     4 eof stream
> -   FetchError     Resource temporarily unavailable
> -   FetchError     eof socket fail
> -   BackendClose   24 boot.webserver
> -   BereqAcct      1274 0 1274 333 32554 32887
> -   End
>
>
> So I figured that maybe it's a nginx issue. When I turn on debug logs,
> nginx tells me that it's varnish that closed the connection. It was never
> able to send the complete html response back (that's probably why it's
> truncated). Here's the nginx logs:
>
>
> 2016/09/14 23:49:08 [info] 24670#24670: *566 epoll_wait() reported that
> client prematurely closed connection, so upstream connection is closed too
> while reading upstream
>
>
> So on one side, I have varnish saying that nginx became unavailable. And
> on the other, I have nginx telling me that varnish closed the connection. I
> don't know which one is right.
>
> But if I remove varnish and send the proxy request to the backend
> directly, everything works. So the signs seem to point to varnish, but I
> can't seem to find what's causing this. It started on Varnish 4.1.1, but
> upgrading to 4.1.3 didn't change anything.
>
> I'm also not sure how to troubleshoot this further. I tried to look at the
> varnish source code. The FetchError is just a generic C error so I don't
> have much else to go on.
>
> I've also attached the output of "param.show" below:
>
> accept_filter              off [bool] (default)
> acceptor_sleep_decay       0.9 (default)
> acceptor_sleep_incr        0.000 [seconds] (default)
> acceptor_sleep_max         0.050 [seconds] (default)
> auto_restart               on [bool] (default)
> backend_idle_timeout       60.000 [seconds] (default)
> ban_dups                   on [bool] (default)
> ban_lurker_age             60.000 [seconds] (default)
> ban_lurker_batch           1000 (default)
> ban_lurker_sleep           0.010 [seconds] (default)
> between_bytes_timeout      60.000 [seconds] (default)
> cc_command                 "exec gcc -std=gnu99 -g -O2 -fstack-protector
> --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -Werror
> -Wno-error=unused-result -pthread -fpic -shared -Wl,-x -o %o %s" (default)
> cli_buffer                 8k [bytes] (default)
> cli_limit                  48k [bytes] (default)
> cli_timeout                60.000 [seconds] (default)
> clock_skew                 10 [seconds] (default)
> connect_timeout            3.500 [seconds] (default)
> critbit_cooloff            180.000 [seconds] (default)
> debug                      none (default)
> default_grace              10.000 [seconds] (default)
> default_keep               0.000 [seconds] (default)
> default_ttl                3600.000 [seconds]
> feature                    none (default)
> fetch_chunksize            16k [bytes] (default)
> fetch_maxchunksize         0.25G [bytes] (default)
> first_byte_timeout         60.000 [seconds] (default)
> gzip_buffer                32k [bytes] (default)
> gzip_level                 6 (default)
> gzip_memlevel              8 (default)
> http_gzip_support          on [bool] (default)
> http_max_hdr               64 [header lines] (default)
> http_range_support         on [bool] (default)
> http_req_hdr_len           8k [bytes] (default)
> http_req_size              32k [bytes] (default)
> http_resp_hdr_len          8k [bytes] (default)
> http_resp_size             32k [bytes] (default)
> idle_send_timeout          60.000 [seconds] (default)
> listen_depth               1024 [connections] (default)
> lru_interval               2.000 [seconds] (default)
> max_esi_depth              5 [levels] (default)
> max_restarts               4 [restarts] (default)
> max_retries                4 [retries] (default)
> nuke_limit                 50 [allocations] (default)
> pcre_match_limit           10000 (default)
> pcre_match_limit_recursion 20 (default)
> ping_interval              3 [seconds] (default)
> pipe_timeout               60.000 [seconds] (default)
> pool_req                   10,100,10 (default)
> pool_sess                  10,100,10 (default)
> pool_vbo                   10,100,10 (default)
> prefer_ipv6                off [bool] (default)
> rush_exponent              3 [requests per request] (default)
> send_timeout               600.000 [seconds] (default)
> session_max                100000 [sessions] (default)
> shm_reclen                 255b [bytes] (default)
> shortlived                 10.000 [seconds] (default)
> sigsegv_handler            on [bool] (default)
> syslog_cli_traffic         on [bool] (default)
> tcp_fastopen               off [bool] (default)
> tcp_keepalive_intvl        75.000 [seconds] (default)
> tcp_keepalive_probes       9 [probes] (default)
> tcp_keepalive_time         7200.000 [seconds] (default)
> thread_pool_add_delay      0.000 [seconds] (default)
> thread_pool_destroy_delay  1.000 [seconds] (default)
> thread_pool_fail_delay     0.200 [seconds] (default)
> thread_pool_max            5000 [threads] (default)
> thread_pool_min            100 [threads] (default)
> thread_pool_stack          48k [bytes] (default)
> thread_pool_timeout        300.000 [seconds] (default)
> thread_pools               2 [pools] (default)
> thread_queue_limit         20 (default)
> thread_stats_rate          10 [requests] (default)
> timeout_idle               5.000 [seconds] (default)
> timeout_linger             0.050 [seconds] (default)
> vcc_allow_inline_c         off [bool] (default)
> vcc_err_unref              on [bool] (default)
> vcc_unsafe_path            on [bool] (default)
> vcl_cooldown               600.000 [seconds] (default)
> vcl_dir                    /etc/varnish (default)
> vmod_dir                   /usr/lib/varnish/vmods (default)
> vsl_buffer                 4k [bytes] (default)
> vsl_mask                   -VCL_trace,-WorkThread,-Hash,-VfpAcct (default)
> vsl_reclen                 255b [bytes] (default)
> vsl_space                  80M [bytes] (default)
> vsm_free_cooldown          60.000 [seconds] (default)
> vsm_space                  1M [bytes] (default)
> workspace_backend          64k [bytes] (default)
> workspace_client           64k [bytes] (default)
> workspace_session          0.50k [bytes] (default)
> workspace_thread           2k [bytes] (default)
>
>
>
> Thanks in advance for any help,
> Carl
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160916/9e192569/attachment.html>

From carlalexander at globalvoices.org  Sat Sep 17 00:21:47 2016
From: carlalexander at globalvoices.org (Carl Alexander)
Date: Fri, 16 Sep 2016 17:21:47 -0700
Subject: Incomplete responses for some requests
In-Reply-To: <CABoVN9CU1cLA6i_6J=xQ1R+9nk4UeSeZncjd_sjDsBFR7Admzg@mail.gmail.com>
References: <CADOiiWpAwVP3bpVXoKVWcvv=FcF34+=DU2kL9Yg6wfTHpsnqNQ@mail.gmail.com>
 <CADOiiWrq2ie__agmUBCpgjY_=hin1POy8LNBYTvpPj9Xc0FfJA@mail.gmail.com>
 <CABoVN9CU1cLA6i_6J=xQ1R+9nk4UeSeZncjd_sjDsBFR7Admzg@mail.gmail.com>
Message-ID: <CADOiiWqOMrW5rDNhQsm=a8uUZCXMGVdCu2YZgxi=h9dFNxPT0A@mail.gmail.com>

> Are you able to reproduce this in a minimal environment? That would help
a lot but in any case I think it's worth opening a ticket.

Do you have an example of a minimal environment that I could do this on? Or
do you mean using the default vcl file?

On Fri, Sep 16, 2016 at 2:47 PM, Dridi Boukelmoune <
dridi at varnish-software.com> wrote:

> > So this gives credence that something is going on inside Varnish. Should
> I create a ticket for this? I've hit the limit of my ability
> troubleshooting this at the moment.
>
> Are you able to reproduce this in a minimal environment? That would help a
> lot but in any case I think it's worth opening a ticket.
>
> Cheers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160916/859ef146/attachment.html>

From dridi at varni.sh  Mon Sep 19 11:30:54 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 19 Sep 2016 13:30:54 +0200
Subject: Incomplete responses for some requests
In-Reply-To: <CADOiiWqOMrW5rDNhQsm=a8uUZCXMGVdCu2YZgxi=h9dFNxPT0A@mail.gmail.com>
References: <CADOiiWpAwVP3bpVXoKVWcvv=FcF34+=DU2kL9Yg6wfTHpsnqNQ@mail.gmail.com>
 <CADOiiWrq2ie__agmUBCpgjY_=hin1POy8LNBYTvpPj9Xc0FfJA@mail.gmail.com>
 <CABoVN9CU1cLA6i_6J=xQ1R+9nk4UeSeZncjd_sjDsBFR7Admzg@mail.gmail.com>
 <CADOiiWqOMrW5rDNhQsm=a8uUZCXMGVdCu2YZgxi=h9dFNxPT0A@mail.gmail.com>
Message-ID: <CABoVN9Az8H7SxHhdsHe6xHGC568kuyBBowe2pyv9Dxpbm3OpTA@mail.gmail.com>

On Sat, Sep 17, 2016 at 2:21 AM, Carl Alexander
<carlalexander at globalvoices.org> wrote:
>> Are you able to reproduce this in a minimal environment? That would help a
>> lot but in any case I think it's worth opening a ticket.
>
> Do you have an example of a minimal environment that I could do this on? Or
> do you mean using the default vcl file?

No, I meant reproducing the bug with as little VCL as possible on a
simple application. Basically not dragging your whole backend to
reproduce the bug.

For non-regression tests (or tests in general) we use `varnishtest`
where we can mock clients and backends to run varnish in a controlled
environment.

Cheers



From albert.tollkuci at gmail.com  Mon Sep 19 12:37:50 2016
From: albert.tollkuci at gmail.com (=?UTF-8?Q?Albert_Tollku=C3=A7i?=)
Date: Mon, 19 Sep 2016 14:37:50 +0200
Subject: Recommendation for varnishlog in production
Message-ID: <CACqaJ6j64+t4w1fOO0TY-68GwbKrBqcDYO17YgxHifCafWEBCg@mail.gmail.com>

I've setup varnish as a proxy for a few websites and it's running smoothly.
However I did notice that the log file is huge (daily size of around 100
GB). I guess that so much logging will hurt the performance and I'm looking
at different options.

I tried to remove -w filename option but this was required. Probably
another option is to disable the service. However before making the changes
I would like to ask if there's any recommendation of how to setup
varnishlog in production server.

Thank you,
Albert

-- 
Web: http://www.tollkuci.com
Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
<https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
<https://www.facebook.com/albert.tollkuci> Twitter
<https://twitter.com/AlbertTollkuci> Career 2.0
<http://careers.stackoverflow.com/atollkuci>
------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160919/952116e6/attachment.html>

From guillaume at varnish-software.com  Mon Sep 19 14:38:51 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Mon, 19 Sep 2016 16:38:51 +0200
Subject: Recommendation for varnishlog in production
In-Reply-To: <CACqaJ6j64+t4w1fOO0TY-68GwbKrBqcDYO17YgxHifCafWEBCg@mail.gmail.com>
References: <CACqaJ6j64+t4w1fOO0TY-68GwbKrBqcDYO17YgxHifCafWEBCg@mail.gmail.com>
Message-ID: <CAJ6ZYQw44-=t+xdRcFJmvGL3KuOSsin=GSZCsBru8_9XTLStNg@mail.gmail.com>

1. use logrotate, or something like that
2. varnishlog is probably overkill for what you need, turn your gaze
towards varnishncsa
3. if you really want to use varnishlog, you should use '-q' to filter only
the "important" transactions (slow fetches/deliveries, errors, etc.)

-- 
Guillaume Quintard

On Mon, Sep 19, 2016 at 2:37 PM, Albert Tollku?i <albert.tollkuci at gmail.com>
wrote:

> I've setup varnish as a proxy for a few websites and it's running
> smoothly. However I did notice that the log file is huge (daily size of
> around 100 GB). I guess that so much logging will hurt the performance and
> I'm looking at different options.
>
> I tried to remove -w filename option but this was required. Probably
> another option is to disable the service. However before making the changes
> I would like to ask if there's any recommendation of how to setup
> varnishlog in production server.
>
> Thank you,
> Albert
>
> --
> Web: http://www.tollkuci.com
> Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
> <https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
> <https://www.facebook.com/albert.tollkuci> Twitter
> <https://twitter.com/AlbertTollkuci> Career 2.0
> <http://careers.stackoverflow.com/atollkuci>
> ------------------------------
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160919/b4a7efdc/attachment.html>

From albert.tollkuci at gmail.com  Mon Sep 19 16:15:42 2016
From: albert.tollkuci at gmail.com (=?UTF-8?Q?Albert_Tollku=C3=A7i?=)
Date: Mon, 19 Sep 2016 18:15:42 +0200
Subject: Recommendation for varnishlog in production
In-Reply-To: <600e8959f4ff1c9ee60af3f3d8609a5b@www.cromagnon.com>
References: <CACqaJ6j64+t4w1fOO0TY-68GwbKrBqcDYO17YgxHifCafWEBCg@mail.gmail.com>
 <600e8959f4ff1c9ee60af3f3d8609a5b@www.cromagnon.com>
Message-ID: <CACqaJ6jHhkO_EpaMzwDO=9YwPHY=rGbKy+ZFTZ76pp-Nc9RBQQ@mail.gmail.com>

Thank you Craig & Guillame. I did disable varnishlog and left only
varnishncsa.

Albert


On Mon, Sep 19, 2016 at 5:44 PM, Craig Servin <cservin at cromagnon.com> wrote:

> Hi Albert,
>
> What I do is run varnishncsa for the day to day logging, and only run
> varnishlog for debugging purposes.  You can also filter the output from
> varnishlog, but that would be specific to what you are trying to log.
>
> Cheers,
>
> Craig
>
>
>
> On 2016-09-19 07:37, Albert Tollku?i wrote:
>
>> I've setup varnish as a proxy for a few websites and it's running
>> smoothly.
>> However I did notice that the log file is huge (daily size of around 100
>> GB). I guess that so much logging will hurt the performance and I'm
>> looking
>> at different options.
>>
>> I tried to remove -w filename option but this was required. Probably
>> another option is to disable the service. However before making the
>> changes
>> I would like to ask if there's any recommendation of how to setup
>> varnishlog in production server.
>>
>> Thank you,
>> Albert
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>


-- 
Web: http://www.tollkuci.com
Follow me on: LinkedIn <http://www.linkedin.com/in/alberttollkuci> Google+
<https://plus.google.com/+AlbertTollku%C3%A7i/posts> Facebook
<https://www.facebook.com/albert.tollkuci> Twitter
<https://twitter.com/AlbertTollkuci> Career 2.0
<http://careers.stackoverflow.com/atollkuci>
------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160919/39245ee2/attachment.html>

From t.honacker at googlemail.com  Wed Sep 21 08:53:45 2016
From: t.honacker at googlemail.com (Tobias Honacker)
Date: Wed, 21 Sep 2016 10:53:45 +0200
Subject: varnish + gatling http first read error: EOF
Message-ID: <CADepKeCdwGGm6+y2_BFwsQeftqZiHxgXUmfbBQRzCh_dgPePsw@mail.gmail.com>

Hi all,

we are facing some issues trying to stresstest our oAuth App using Varnish
4.0.3 and Gatling

Infra:

LB -> Apache (SSL Offloading) + Varnish -> Appserver (Apache + Tomcat)

Varnishlog:
http://pastebin.com/tCWgi01H

VCL:
http://pastebin.com/cEdzAwxs

tcpdump on our app server with 1 req/s (varnishlog 200):
http://pastebin.com/XFVsANiN

tcpdump on our app server with >= 2 req/s (varnishlog 503er):
http://pastebin.com/f2EKADpx

If varnish throws HTTP 503 the connection: close header is missing

Running tests using pipe works fine, because pipe sets connection: close
Running tests without varnish works fine, too.

The Apaches using KeepAlive:

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 10


Any suggestions?

Thanks in advance and best regards
Tobias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160921/e8ef817c/attachment.html>

From guillaume at varnish-software.com  Wed Sep 21 10:08:04 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 21 Sep 2016 12:08:04 +0200
Subject: varnish + gatling http first read error: EOF
In-Reply-To: <CADepKeCdwGGm6+y2_BFwsQeftqZiHxgXUmfbBQRzCh_dgPePsw@mail.gmail.com>
References: <CADepKeCdwGGm6+y2_BFwsQeftqZiHxgXUmfbBQRzCh_dgPePsw@mail.gmail.com>
Message-ID: <CAJ6ZYQwVbfKY2GyxOq4YehiB1E4VhHYsLfoBPCBMHt3viq-F1w@mail.gmail.com>

Look at varnishlog, and find the error in the log. And possibly look at a
few BeReq before that.

I've seen apache wrongly replying 304 with body. As Varnish expect no body,
it bytes are read as the next request's headers, and as you can guess, that
doesn't work well. You may be affected my a variant of that.

On Sep 21, 2016 11:14, "Tobias Honacker" <t.honacker at googlemail.com> wrote:

> Hi all,
>
> we are facing some issues trying to stresstest our oAuth App using Varnish
> 4.0.3 and Gatling
>
> Infra:
>
> LB -> Apache (SSL Offloading) + Varnish -> Appserver (Apache + Tomcat)
>
> Varnishlog:
> http://pastebin.com/tCWgi01H
>
> VCL:
> http://pastebin.com/cEdzAwxs
>
> tcpdump on our app server with 1 req/s (varnishlog 200):
> http://pastebin.com/XFVsANiN
>
> tcpdump on our app server with >= 2 req/s (varnishlog 503er):
> http://pastebin.com/f2EKADpx
>
> If varnish throws HTTP 503 the connection: close header is missing
>
> Running tests using pipe works fine, because pipe sets connection: close
> Running tests without varnish works fine, too.
>
> The Apaches using KeepAlive:
>
> KeepAlive On
> MaxKeepAliveRequests 100
> KeepAliveTimeout 10
>
>
> Any suggestions?
>
> Thanks in advance and best regards
> Tobias
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160921/f93ad834/attachment.html>

From akshayaccha03 at gmail.com  Wed Sep 21 10:31:09 2016
From: akshayaccha03 at gmail.com (akshay accha)
Date: Wed, 21 Sep 2016 16:01:09 +0530
Subject: Is varnish agent is an open source what are the dependency so required
Message-ID: <CADxYJXyKcGDqn3PE-WtF7wL89H_5XDY0q8-LyCsR00QPT4q8ZQ@mail.gmail.com>

I am not able to install varnish-agent what dependencies are required and
what is the procedure to install varnish-agent such that i already have
varnish 4.1 installed and have ubuntu as os. I have the vagent2-master file
in /etc/vagent2-master how to install varnish agent from here ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160921/efbd0532/attachment.html>

From guillaume at varnish-software.com  Wed Sep 21 13:58:30 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 21 Sep 2016 15:58:30 +0200
Subject: Is varnish agent is an open source what are the dependency so
 required
In-Reply-To: <CADxYJXyKcGDqn3PE-WtF7wL89H_5XDY0q8-LyCsR00QPT4q8ZQ@mail.gmail.com>
References: <CADxYJXyKcGDqn3PE-WtF7wL89H_5XDY0q8-LyCsR00QPT4q8ZQ@mail.gmail.com>
Message-ID: <CAJ6ZYQwT=hTouAU4uyq95WH0uCztZzxEAZfS6J8_aAxpi7Vh0g@mail.gmail.com>

Are you using the ubuntu packages or compiling from source?

-- 
Guillaume Quintard

On Wed, Sep 21, 2016 at 12:31 PM, akshay accha <akshayaccha03 at gmail.com>
wrote:

> I am not able to install varnish-agent what dependencies are required and
> what is the procedure to install varnish-agent such that i already have
> varnish 4.1 installed and have ubuntu as os. I have the vagent2-master file
> in /etc/vagent2-master how to install varnish agent from here ?
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160921/a1a3dbe2/attachment.html>

From guillaume at varnish-software.com  Wed Sep 21 14:55:52 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 21 Sep 2016 16:55:52 +0200
Subject: Is varnish agent is an open source what are the dependency so
 required
In-Reply-To: <CADxYJXxmKJuaP4PZxuuDBcwFV0K=QP=QvdifH9rVCo8NnuqzDw@mail.gmail.com>
References: <CADxYJXyKcGDqn3PE-WtF7wL89H_5XDY0q8-LyCsR00QPT4q8ZQ@mail.gmail.com>
 <CAJ6ZYQwT=hTouAU4uyq95WH0uCztZzxEAZfS6J8_aAxpi7Vh0g@mail.gmail.com>
 <CADxYJXxmKJuaP4PZxuuDBcwFV0K=QP=QvdifH9rVCo8NnuqzDw@mail.gmail.com>
Message-ID: <CAJ6ZYQzPwgAv0bvOH9XNndfAkdZ06mD9ofgQhraiMbv3Dkkgsw@mail.gmail.com>

I don't know what you are talking about, what was the URL you downloaded it
from?

-- 
Guillaume Quintard

On Wed, Sep 21, 2016 at 4:44 PM, akshay accha <akshayaccha03 at gmail.com>
wrote:

> I have downloaded the master file and moved it to the destination folder
> as mentioned ,how do I proceed ?
>
>
> On Wednesday, 21 September 2016, Guillaume Quintard <
> guillaume at varnish-software.com> wrote:
>
>> Are you using the ubuntu packages or compiling from source?
>>
>> --
>> Guillaume Quintard
>>
>> On Wed, Sep 21, 2016 at 12:31 PM, akshay accha <akshayaccha03 at gmail.com>
>> wrote:
>>
>>> I am not able to install varnish-agent what dependencies are required
>>> and what is the procedure to install varnish-agent such that i already have
>>> varnish 4.1 installed and have ubuntu as os. I have the vagent2-master file
>>> in /etc/vagent2-master how to install varnish agent from here ?
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160921/50b187c7/attachment.html>

From guillaume at varnish-software.com  Wed Sep 21 16:15:52 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 21 Sep 2016 18:15:52 +0200
Subject: Is varnish agent is an open source what are the dependency so
 required
In-Reply-To: <CADxYJXyFma_Orx_7xM7a1=2-6e+Hvmjgqpk8ed9U1Lye=ryzhg@mail.gmail.com>
References: <CADxYJXyKcGDqn3PE-WtF7wL89H_5XDY0q8-LyCsR00QPT4q8ZQ@mail.gmail.com>
 <CAJ6ZYQwT=hTouAU4uyq95WH0uCztZzxEAZfS6J8_aAxpi7Vh0g@mail.gmail.com>
 <CADxYJXxmKJuaP4PZxuuDBcwFV0K=QP=QvdifH9rVCo8NnuqzDw@mail.gmail.com>
 <CAJ6ZYQzPwgAv0bvOH9XNndfAkdZ06mD9ofgQhraiMbv3Dkkgsw@mail.gmail.com>
 <CADxYJXyFma_Orx_7xM7a1=2-6e+Hvmjgqpk8ed9U1Lye=ryzhg@mail.gmail.com>
Message-ID: <CAJ6ZYQxqqofzVMVjng17_03mHs93j-XRwAzNj0GzGZQLPXv1WQ@mail.gmail.com>

Try installing the varnish-agent package, it'll be much easier.

Please, keep varnish-misc address in the CC when you reply.

-- 
Guillaume Quintard

On Wed, Sep 21, 2016 at 5:08 PM, akshay accha <akshayaccha03 at gmail.com>
wrote:

> Url for varnish agent I downloaded the master file from
> https://github.com/varnish/vagent2 .
>
> Varnish 4.1 I installed using Ubuntu package and I tried installing
> varnishagent but it could not develope dependency tree so I downloaded
> master file and tried installing it.
>
> Could you please tell me how to install varnish agent given that I already
> have varnish 4.1 installed using Ubuntu package
>
> On Wednesday, 21 September 2016, Guillaume Quintard <
> guillaume at varnish-software.com> wrote:
>
>> I don't know what you are talking about, what was the URL you downloaded
>> it from?
>>
>> --
>> Guillaume Quintard
>>
>> On Wed, Sep 21, 2016 at 4:44 PM, akshay accha <akshayaccha03 at gmail.com>
>> wrote:
>>
>>> I have downloaded the master file and moved it to the destination folder
>>> as mentioned ,how do I proceed ?
>>>
>>>
>>> On Wednesday, 21 September 2016, Guillaume Quintard <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> Are you using the ubuntu packages or compiling from source?
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Wed, Sep 21, 2016 at 12:31 PM, akshay accha <akshayaccha03 at gmail.com
>>>> > wrote:
>>>>
>>>>> I am not able to install varnish-agent what dependencies are required
>>>>> and what is the procedure to install varnish-agent such that i already have
>>>>> varnish 4.1 installed and have ubuntu as os. I have the vagent2-master file
>>>>> in /etc/vagent2-master how to install varnish agent from here ?
>>>>> _______________________________________________
>>>>> varnish-misc mailing list
>>>>> varnish-misc at varnish-cache.org
>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>
>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160921/3bc4f716/attachment.html>

From akshayaccha03 at gmail.com  Thu Sep 22 04:51:52 2016
From: akshayaccha03 at gmail.com (akshay accha)
Date: Thu, 22 Sep 2016 10:21:52 +0530
Subject: I have installed varnish-agent but it has some issues.
Message-ID: <CADxYJXx99=X+XByd7Un_P93Uz_UM5jWrEVSBwaBYjpa5ZEYCMA@mail.gmail.com>

varnish-agent -H option is not working and when i type man varnish-agent
the -H option is not there what might be the solution?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160922/d8190404/attachment.html>

From guillaume at varnish-software.com  Thu Sep 22 07:55:00 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Thu, 22 Sep 2016 09:55:00 +0200
Subject: I have installed varnish-agent but it has some issues.
In-Reply-To: <CADxYJXx99=X+XByd7Un_P93Uz_UM5jWrEVSBwaBYjpa5ZEYCMA@mail.gmail.com>
References: <CADxYJXx99=X+XByd7Un_P93Uz_UM5jWrEVSBwaBYjpa5ZEYCMA@mail.gmail.com>
Message-ID: <CAJ6ZYQy9N7mqoKXPs4kGY453uRPYyKZXR8x-jQ7PgNzJ9Vie0Q@mail.gmail.com>

I do see it, look again?

-- 
Guillaume Quintard

On Thu, Sep 22, 2016 at 6:51 AM, akshay accha <akshayaccha03 at gmail.com>
wrote:

> varnish-agent -H option is not working and when i type man varnish-agent
> the -H option is not there what might be the solution?
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160922/3965b99f/attachment.html>

From akshayaccha03 at gmail.com  Thu Sep 22 08:24:34 2016
From: akshayaccha03 at gmail.com (akshay accha)
Date: Thu, 22 Sep 2016 13:54:34 +0530
Subject: i have sent the screen shot when i type varnish-agent -H
Message-ID: <CADxYJXyaW-y7j+W+6r=m5YUi2DAUwz5RG7_XgOBd56oH88HUWA@mail.gmail.com>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160922/fcfeea96/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot from 2016-09-22 13:53:06.png
Type: image/png
Size: 56386 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160922/fcfeea96/attachment.png>

From akshayaccha03 at gmail.com  Thu Sep 22 08:53:27 2016
From: akshayaccha03 at gmail.com (akshay accha)
Date: Thu, 22 Sep 2016 14:23:27 +0530
Subject: Does varnish 3.09 support varnish-agent -H and varnishlog -g option?
Message-ID: <CADxYJXwENpP60Kz_WW5RNtF9PF92aKgNdnk2NqsF78S_3YE=Vg@mail.gmail.com>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160922/a0fdee07/attachment.html>

From guillaume at varnish-software.com  Thu Sep 22 09:10:46 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Thu, 22 Sep 2016 11:10:46 +0200
Subject: Does varnish 3.09 support varnish-agent -H and varnishlog -g
 option?
In-Reply-To: <CADxYJXwENpP60Kz_WW5RNtF9PF92aKgNdnk2NqsF78S_3YE=Vg@mail.gmail.com>
References: <CADxYJXwENpP60Kz_WW5RNtF9PF92aKgNdnk2NqsF78S_3YE=Vg@mail.gmail.com>
Message-ID: <CAJ6ZYQx3WGFi6QqGavnQcN_LTVPpXxVeyUp2kEgvbPf7LwDvaA@mail.gmail.com>

Sir, you got to stop spamming this mailing list. We also have an IRC
channel, consider using it.

And no, varnishlog -g is only supported stating v4.0.

the varnish-agent shouldn't be affected by the varnish version.

-- 
Guillaume Quintard

On Thu, Sep 22, 2016 at 10:53 AM, akshay accha <akshayaccha03 at gmail.com>
wrote:

>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160922/7fb82c3f/attachment.html>

From guillaume at varnish-software.com  Fri Sep 23 09:16:54 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 23 Sep 2016 11:16:54 +0200
Subject: bitrate and memory allocated
In-Reply-To: <f108f8cc-482d-04cd-0cf7-815a10d57b0d@yahoo.es>
References: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>
 <c4b627c6-67e0-eddc-cb97-e059addd45a9@yahoo.es>
 <CABoVN9AJi8=Y3KSYjHZJgHSnGgeZA7n2Jp9Ly-PLjz5e5NWMpg@mail.gmail.com>
 <c5d198ef-63c1-4b13-5850-35b844dbb156@yahoo.es>
 <CABoVN9C=aXS1FFRBJjxSqeHfSFQ7wGJdo5w_3APsZza=qFihsw@mail.gmail.com>
 <f108f8cc-482d-04cd-0cf7-815a10d57b0d@yahoo.es>
Message-ID: <CAJ6ZYQzAO8s4C-PE1ROTWG_-U_S=9vn1JWhi8RqsN-NPDL_1vg@mail.gmail.com>

Focus on g_space (free space in the storage) and g_bytes (used bytes in the
storage).

-- 
Guillaume Quintard

On Tue, Sep 6, 2016 at 12:08 PM, Miguel Gonz?lez <miguel_3_gonzalez at yahoo.es
> wrote:

> On 09/05/16 10:44 PM, Dridi Boukelmoune wrote:
> >> Any URL? I have googled for a few hours but I didn?t get a clear
> >> understanding from the official manuals.
> >
> > https://www.varnish-cache.org/docs/4.1/reference/varnish-
> counters.html#per-malloc-storage-counters-sma
>
>
> I had read the 4.0 docs but I find them quite short and not very
> explanatory. What are the counters? The current use of memory? The
> amount allocated since Varnish was restarted (historical)? At least for
> me, I would like an example to better understand what those counters mean.
>
> Thanks
>
> Miguel
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160923/83ae28df/attachment.html>

From Drew.AJ at principal.com  Fri Sep 23 11:52:14 2016
From: Drew.AJ at principal.com (Drew, AJ)
Date: Fri, 23 Sep 2016 11:52:14 +0000
Subject: Problem Upgrade to Varnish 4.1 - Multiple Backends Not Working
Message-ID: <caf70417c008432baff20a6570cb83b2@PFGDSMMSG101.principalusa.corp.principal.com>

Hello,

We are trying to upgrade to Varnish 4.1 from Varnish 3.0.5.

Most things are working fine, but a big problem we are running into is that we have multiple backends.  We switch backends based on URLs.  This was working fine in Varnish 3, but when we switched over to Varnish 4, it stopped working.

I have tried many variations of the URLs, and even simplified the directors so that they only contain one backend (in my tests) so that I could switch from trying the backend directly to trying the director.

Using either of the options does not seem to matter.

Has anyone else run into this problem?

Here are the sections of my VCL that contain the backends / director definitions and the if statements to switch.  I have also left in the different options (lines) I have used to come up with different combinations for testing.

probe B_probe {
        .request =
        "GET  /zzzzzz/status.php HTTP/1.1"
        "HOST: xxxxxx.example.com"
        "Connection: close";
        .timeout   = 0.3 s;
        .window    = 8;
        .threshold = 7;
}

backend B_backend1 {
        .host = "162.131.196.220";
        .port = "8080";
        .probe = B_probe;
}

backend syscheck {
#        .host = "localhost";
        .host = "127.0.0.1";
        .port = "8081";
}

backend searchBackend {
        .host = "rp.www.example.com";
        .port = "80";
}

backend providerBackend {
        .host = "172.23.82.28";
        .port = "80";
}

sub vcl_init {
        new examplehosts = directors.round_robin();
        examplehosts.add_backend(B_backend1);

        new examplesyscheck = directors.round_robin();
        examplesyscheck.add_backend(syscheck);
}




sub vcl_recv {

#       set req.backend_hint = examplehosts.backend();
        set req.backend_hint = B_backend1;

        if (req.url ~ "^/syscheck/.*$") {
#       if (req.url ~ "^/syscheck/*") {
#       if (req.url ~ "^/syscheck/") {
                set req.backend_hint = syscheck;
#               set req.backend_hint = examplesyscheck.backend();
                return(pass);
        }

        # Add simple health check for upstream load balancers
        if (req.url ~ "^/health-check") {
                return( synth(200, "UP"));
        }
...


Thanks!

A J Drew


-----Message Disclaimer-----

This e-mail message is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the intended recipient, any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply email to Connect at principal.com and delete or destroy all copies of the original message and attachments thereto. Email sent to or from the Principal Financial Group or any of its member companies may be retained as required by law or regulation.

Nothing in this message is intended to constitute an Electronic signature for purposes of the Uniform Electronic Transactions Act (UETA) or the Electronic Signatures in Global and National Commerce Act ("E-Sign") unless a specific statement to the contrary is included in this message.

If you no longer wish to receive any further solicitation from the Principal Financial Group you may unsubscribe at https://www.principal.com/do-not-contact-form any time.

If you are a Canadian resident and no longer wish to receive commercial electronic messages you may unsubscribe at https://www.principal.com/do-not-email-request-canadian-residents any time.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160923/43bf3583/attachment.html>

From guillaume at varnish-software.com  Fri Sep 23 13:11:49 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 23 Sep 2016 15:11:49 +0200
Subject: Problem Upgrade to Varnish 4.1 - Multiple Backends Not Working
In-Reply-To: <caf70417c008432baff20a6570cb83b2@PFGDSMMSG101.principalusa.corp.principal.com>
References: <caf70417c008432baff20a6570cb83b2@PFGDSMMSG101.principalusa.corp.principal.com>
Message-ID: <CAJ6ZYQxj=pgiCZnCSWqmYLPS4CTmZVyNnmyvqfPreRFLn1ifzA@mail.gmail.com>

What does "stop working" mean? Does the vcl fail to compile? do you receive
503s? does it open a portal to an ancient dimension filled with unspoken
nightmares? Help us help you.

-- 
Guillaume Quintard

On Fri, Sep 23, 2016 at 1:52 PM, Drew, AJ <Drew.AJ at principal.com> wrote:

> Hello,
>
> We are trying to upgrade to Varnish 4.1 from Varnish 3.0.5.
>
> Most things are working fine, but a big problem we are running into is
> that we have multiple backends.  We switch backends based on URLs.  This
> was working fine in Varnish 3, but when we switched over to Varnish 4, it
> stopped working.
>
> I have tried many variations of the URLs, and even simplified the
> directors so that they only contain one backend (in my tests) so that I
> could switch from trying the backend directly to trying the director.
>
> Using either of the options does not seem to matter.
>
> Has anyone else run into this problem?
>
> Here are the sections of my VCL that contain the backends / director
> definitions and the if statements to switch.  I have also left in the
> different options (lines) I have used to come up with different
> combinations for testing.
>
> probe B_probe {
>         .request =
>         "GET  /zzzzzz/status.php HTTP/1.1"
>         "HOST: xxxxxx.example.com"
>         "Connection: close";
>         .timeout   = 0.3 s;
>         .window    = 8;
>         .threshold = 7;
> }
>
> backend B_backend1 {
>         .host = "162.131.196.220";
>         .port = "8080";
>         .probe = B_probe;
> }
>
> backend syscheck {
> #        .host = "localhost";
>         .host = "127.0.0.1";
>         .port = "8081";
> }
>
> backend searchBackend {
>         .host = "rp.www.example.com";
>         .port = "80";
> }
>
> backend providerBackend {
>         .host = "172.23.82.28";
>         .port = "80";
> }
>
> sub vcl_init {
>         new examplehosts = directors.round_robin();
>         examplehosts.add_backend(B_backend1);
>
>         new examplesyscheck = directors.round_robin();
>         examplesyscheck.add_backend(syscheck);
> }
>
>
>
>
> sub vcl_recv {
>
> #       set req.backend_hint = examplehosts.backend();
>         set req.backend_hint = B_backend1;
>
>         if (req.url ~ "^/syscheck/.*$") {
> #       if (req.url ~ "^/syscheck/*") {
> #       if (req.url ~ "^/syscheck/") {
>                 set req.backend_hint = syscheck;
> #               set req.backend_hint = examplesyscheck.backend();
>                 return(pass);
>         }
>
>         # Add simple health check for upstream load balancers
>         if (req.url ~ "^/health-check") {
>                 return( synth(200, "UP"));
>         }
> ...
>
>
> Thanks!
>
> A J Drew
>
>
>
> -----Message Disclaimer-----
>
> This e-mail message is intended only for the use of the individual or
> entity to which it is addressed, and may contain information that is
> privileged, confidential and exempt from disclosure under applicable law.
> If you are not the intended recipient, any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this communication in error, please notify us immediately by reply email to
> Connect at principal.com and delete or destroy all copies of the original
> message and attachments thereto. Email sent to or from the Principal
> Financial Group or any of its member companies may be retained as required
> by law or regulation.
>
> Nothing in this message is intended to constitute an Electronic signature
> for purposes of the Uniform Electronic Transactions Act (UETA) or the
> Electronic Signatures in Global and National Commerce Act ("E-Sign") unless
> a specific statement to the contrary is included in this message.
>
> If you no longer wish to receive any further solicitation from the
> Principal Financial Group you may unsubscribe at
> https://www.principal.com/do-not-contact-form any time.
>
> If you are a Canadian resident and no longer wish to receive commercial
> electronic messages you may unsubscribe at https://www.principal.com/do-
> not-email-request-canadian-residents any time.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160923/4dc1f752/attachment.html>

From Drew.AJ at principal.com  Fri Sep 23 13:20:46 2016
From: Drew.AJ at principal.com (Drew, AJ)
Date: Fri, 23 Sep 2016 13:20:46 +0000
Subject: Problem Upgrade to Varnish 4.1 - Multiple Backends Not Working
In-Reply-To: <CAJ6ZYQxj=pgiCZnCSWqmYLPS4CTmZVyNnmyvqfPreRFLn1ifzA@mail.gmail.com>
References: <caf70417c008432baff20a6570cb83b2@PFGDSMMSG101.principalusa.corp.principal.com>
 <CAJ6ZYQxj=pgiCZnCSWqmYLPS4CTmZVyNnmyvqfPreRFLn1ifzA@mail.gmail.com>
Message-ID: <f950b614dcdd4740a0e2b829936f8410@PFGDSMMSG101.principalusa.corp.principal.com>

Sorry for not being for specific.

The initial backend being set is the one that is always used no matter what the URL.

So, when I come in with a URL of /syscheck/blah.htm, I would expect the alternative backend to be used.  It does not get select, and instead the original one is the one used, and therefore I end up with either a HTTP 404 or a HTTP 503, depending on the mood of Drupal the destination of the initial backend.

Thanks!

A J Drew


Here is some output from varnishlog of a call:

         2 Begin          c req 1 rxreq
         2 Timestamp      c Start: 1474629889.600227 0.000000 0.000000
         2 Timestamp      c Req: 1474629889.600227 0.000000 0.000000
         2 ReqStart       c 127.0.0.1 34608
         2 ReqMethod      c GET
         2 ReqURL         c /syscheck/wwwul2370.htm
         2 ReqProtocol    c HTTP/1.1
         2 ReqHeader      c Host: xxxxxx.example.com
         2 ReqHeader      c Accept: text/html, application/xhtml+xml, */*
         2 ReqHeader      c Accept-Language: en-US
         2 ReqHeader      c User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
         2 ReqHeader      c Accept-Encoding: gzip, deflate
         2 ReqHeader      c Cookie: ObSSOCookie=loggedout; TLTSID=A5D86DBA4DA97DFA943BD59DA9EFF5B8; dtCookie=3A7E953F591088FA2F6DB23E157B3A54|Q1NTX19FQV9fYXV0aG9yaXphdGlvbi1oYW5kbGVyLWVhcnwxfENTU19fRUFfX2FjY291bnQtYWdncmVnYXRpb24tZWFyfDE; dtLatC=36971; dtPC=-; RANDOM_ID=6529725af06
         2 ReqHeader      c Via: 1.1 xxxxxx.example.com
         2 ReqHeader      c X-Forwarded-For: 172.22.159.116
         2 ReqHeader      c X-Forwarded-Host: xxxxxx.example.com
         2 ReqHeader      c X-Forwarded-Server: xxxxxx.example.com
         2 ReqHeader      c Connection: Keep-Alive
         2 ReqUnset       c X-Forwarded-For: 172.22.159.116
         2 ReqHeader      c X-Forwarded-For: 172.22.159.116, 127.0.0.1
         2 VCL_call       c RECV
         2 VCL_return     c pass
         2 VCL_call       c HASH
         2 VCL_return     c lookup
         2 VCL_call       c PASS
         2 VCL_return     c fetch
         2 Link           c bereq 3 pass
         2 Timestamp      c Fetch: 1474629891.152683 1.552456 1.552456
         2 RespProtocol   c HTTP/1.1
         2 RespStatus     c 404
         2 RespReason     c Not Found
         2 RespHeader     c Date: Fri, 23 Sep 2016 11:24:49 GMT
         2 RespHeader     c Server: Apache
         2 RespHeader     c X-Drupal-Cache: MISS
         2 RespHeader     c Expires: Sun, 19 Nov 1978 05:00:00 GMT
         2 RespHeader     c Cache-Control: public, max-age=3600
         2 RespHeader     c Content-Language: en
         2 RespHeader     c X-Generator: Drupal 7 (http://drupal.org)
         2 RespHeader     c Etag: "1474629889-0"
         2 RespHeader     c Last-Modified: Fri, 23 Sep 2016 11:24:49 GMT
         2 RespHeader     c Vary: Cookie,Accept-Encoding
         2 RespHeader     c Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
         2 RespHeader     c X-XSS-Protection: 1;
         2 RespHeader     c Content-Type: text/html; charset=utf-8
         2 RespHeader     c X-Varnish: 2
         2 RespHeader     c Age: 0
         2 RespHeader     c Via: 1.1 varnish-v4
         2 VCL_call       c DELIVER
         2 RespHeader     c X-Cache: MISS
         2 RespUnset      c Via: 1.1 varnish-v4
         2 RespUnset      c X-Generator: Drupal 7 (http://drupal.org)
         2 RespUnset      c Server: Apache
         2 VCL_return     c deliver
         2 Timestamp      c Process: 1474629891.152737 1.552510 0.000054
         2 RespHeader     c Content-Length: 7685
         2 Debug          c "RES_MODE 2"
         2 RespHeader     c Connection: keep-alive
         2 Timestamp      c Resp: 1474629891.152783 1.552556 0.000046
         2 ReqAcct        c 791 0 791 500 7685 8185
         2 End            c


From: Guillaume Quintard [mailto:guillaume at varnish-software.com]
Sent: Friday, September 23, 2016 8:12 AM
To: Drew, AJ
Cc: varnish-misc at varnish-cache.org
Subject: Re: Problem Upgrade to Varnish 4.1 - Multiple Backends Not Working

What does "stop working" mean? Does the vcl fail to compile? do you receive 503s? does it open a portal to an ancient dimension filled with unspoken nightmares? Help us help you.

--
Guillaume Quintard

On Fri, Sep 23, 2016 at 1:52 PM, Drew, AJ <Drew.AJ at principal.com<mailto:Drew.AJ at principal.com>> wrote:
Hello,

We are trying to upgrade to Varnish 4.1 from Varnish 3.0.5.

Most things are working fine, but a big problem we are running into is that we have multiple backends.  We switch backends based on URLs.  This was working fine in Varnish 3, but when we switched over to Varnish 4, it stopped working.

I have tried many variations of the URLs, and even simplified the directors so that they only contain one backend (in my tests) so that I could switch from trying the backend directly to trying the director.

Using either of the options does not seem to matter.

Has anyone else run into this problem?

Here are the sections of my VCL that contain the backends / director definitions and the if statements to switch.  I have also left in the different options (lines) I have used to come up with different combinations for testing.

probe B_probe {
        .request =
        "GET  /zzzzzz/status.php HTTP/1.1"
        "HOST: xxxxxx.example.com<http://xxxxxx.example.com>"
        "Connection: close";
        .timeout   = 0.3 s;
        .window    = 8;
        .threshold = 7;
}

backend B_backend1 {
        .host = "162.131.196.220";
        .port = "8080";
        .probe = B_probe;
}

backend syscheck {
#        .host = "localhost";
        .host = "127.0.0.1";
        .port = "8081";
}

backend searchBackend {
        .host = "rp.www.example.com<http://rp.www.example.com>";
        .port = "80";
}

backend providerBackend {
        .host = "172.23.82.28";
        .port = "80";
}

sub vcl_init {
        new examplehosts = directors.round_robin();
        examplehosts.add_backend(B_backend1);

        new examplesyscheck = directors.round_robin();
        examplesyscheck.add_backend(syscheck);
}




sub vcl_recv {

#       set req.backend_hint = examplehosts.backend();
        set req.backend_hint = B_backend1;

        if (req.url ~ "^/syscheck/.*$") {
#       if (req.url ~ "^/syscheck/*") {
#       if (req.url ~ "^/syscheck/") {
                set req.backend_hint = syscheck;
#               set req.backend_hint = examplesyscheck.backend();
                return(pass);
        }

        # Add simple health check for upstream load balancers
        if (req.url ~ "^/health-check") {
                return( synth(200, "UP"));
        }
...


Thanks!

A J Drew


-----Message Disclaimer-----
This e-mail message is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the intended recipient, any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply email to Connect at principal.com<mailto:Connect at principal.com> and delete or destroy all copies of the original message and attachments thereto. Email sent to or from the Principal Financial Group or any of its member companies may be retained as required by law or regulation.
Nothing in this message is intended to constitute an Electronic signature for purposes of the Uniform Electronic Transactions Act (UETA) or the Electronic Signatures in Global and National Commerce Act ("E-Sign") unless a specific statement to the contrary is included in this message.
If you no longer wish to receive any further solicitation from the Principal Financial Group you may unsubscribe at https://www.principal.com/do-not-contact-form any time.
If you are a Canadian resident and no longer wish to receive commercial electronic messages you may unsubscribe at https://www.principal.com/do-not-email-request-canadian-residents any time.


_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160923/aabb3edf/attachment.html>

From guillaume at varnish-software.com  Fri Sep 23 13:38:27 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 23 Sep 2016 15:38:27 +0200
Subject: Problem Upgrade to Varnish 4.1 - Multiple Backends Not Working
In-Reply-To: <f950b614dcdd4740a0e2b829936f8410@PFGDSMMSG101.principalusa.corp.principal.com>
References: <caf70417c008432baff20a6570cb83b2@PFGDSMMSG101.principalusa.corp.principal.com>
 <CAJ6ZYQxj=pgiCZnCSWqmYLPS4CTmZVyNnmyvqfPreRFLn1ifzA@mail.gmail.com>
 <f950b614dcdd4740a0e2b829936f8410@PFGDSMMSG101.principalusa.corp.principal.com>
Message-ID: <CAJ6ZYQz1gu1kuViVok63XREn5gFMnECfGeS7H1VkP=anBNAk+g@mail.gmail.com>

They look good, but make sure your regex are doing the way you want, and to
debug, put some "std.log()" statement to make sure the request is
triggering the right ifs.

Also, if that doesn't help, please provide use with the Request section (as
you did) as well as the BeReq section associated (in this case "2
Link           c bereq 3 pass" -> BeReq 3).

Having the whole VCL (on pastebin) would also help.



-- 
Guillaume Quintard

On Fri, Sep 23, 2016 at 3:20 PM, Drew, AJ <Drew.AJ at principal.com> wrote:

> Sorry for not being for specific.
>
>
>
> The initial backend being set is the one that is always used no matter
> what the URL.
>
>
>
> So, when I come in with a URL of /syscheck/blah.htm, I would expect the
> alternative backend to be used.  It does not get select, and instead the
> original one is the one used, and therefore I end up with either a HTTP 404
> or a HTTP 503, depending on the mood of Drupal the destination of the
> initial backend.
>
>
>
> Thanks!
>
>
>
> A J Drew
>
>
>
>
>
> Here is some output from varnishlog of a call:
>
>
>
>          2 Begin          c req 1 rxreq
>
>          2 Timestamp      c Start: 1474629889.600227 0.000000 0.000000
>
>          2 Timestamp      c Req: 1474629889.600227 0.000000 0.000000
>
>          2 ReqStart       c 127.0.0.1 34608
>
>          2 ReqMethod      c GET
>
>          2 ReqURL         c /syscheck/wwwul2370.htm
>
>          2 ReqProtocol    c HTTP/1.1
>
>          2 ReqHeader      c Host: xxxxxx.example.com
>
>          2 ReqHeader      c Accept: text/html, application/xhtml+xml, */*
>
>          2 ReqHeader      c Accept-Language: en-US
>
>          2 ReqHeader      c User-Agent: Mozilla/5.0 (Windows NT 6.1;
> WOW64; Trident/7.0; rv:11.0) like Gecko
>
>          2 ReqHeader      c Accept-Encoding: gzip, deflate
>
>          2 ReqHeader      c Cookie: ObSSOCookie=loggedout; TLTSID=
> A5D86DBA4DA97DFA943BD59DA9EFF5B8; dtCookie=3A7E953F591088FA2F6DB23E157B3A
> 54|Q1NTX19FQV9fYXV0aG9yaXphdGlvbi1oYW5kbGVyLWVhcnwxfENTU19fRUFf
> X2FjY291bnQtYWdncmVnYXRpb24tZWFyfDE; dtLatC=36971; dtPC=-;
> RANDOM_ID=6529725af06
>
>          2 ReqHeader      c Via: 1.1 xxxxxx.example.com
>
>          2 ReqHeader      c X-Forwarded-For: 172.22.159.116
>
>          2 ReqHeader      c X-Forwarded-Host: xxxxxx.example.com
>
>          2 ReqHeader      c X-Forwarded-Server: xxxxxx.example.com
>
>          2 ReqHeader      c Connection: Keep-Alive
>
>          2 ReqUnset       c X-Forwarded-For: 172.22.159.116
>
>          2 ReqHeader      c X-Forwarded-For: 172.22.159.116, 127.0.0.1
>
>          2 VCL_call       c RECV
>
>          2 VCL_return     c pass
>
>          2 VCL_call       c HASH
>
>          2 VCL_return     c lookup
>
>          2 VCL_call       c PASS
>
>          2 VCL_return     c fetch
>
>          2 Link           c bereq 3 pass
>
>          2 Timestamp      c Fetch: 1474629891.152683 1.552456 1.552456
>
>          2 RespProtocol   c HTTP/1.1
>
>          2 RespStatus     c 404
>
>          2 RespReason     c Not Found
>
>          2 RespHeader     c Date: Fri, 23 Sep 2016 11:24:49 GMT
>
>          2 RespHeader     c Server: Apache
>
>          2 RespHeader     c X-Drupal-Cache: MISS
>
>          2 RespHeader     c Expires: Sun, 19 Nov 1978 05:00:00 GMT
>
>          2 RespHeader     c Cache-Control: public, max-age=3600
>
>          2 RespHeader     c Content-Language: en
>
>          2 RespHeader     c X-Generator: Drupal 7 (http://drupal.org)
>
>          2 RespHeader     c Etag: "1474629889-0"
>
>          2 RespHeader     c Last-Modified: Fri, 23 Sep 2016 11:24:49 GMT
>
>          2 RespHeader     c Vary: Cookie,Accept-Encoding
>
>          2 RespHeader     c Strict-Transport-Security: max-age=31536000;
> includeSubDomains; preload
>
>          2 RespHeader     c X-XSS-Protection: 1;
>
>          2 RespHeader     c Content-Type: text/html; charset=utf-8
>
>          2 RespHeader     c X-Varnish: 2
>
>          2 RespHeader     c Age: 0
>
>          2 RespHeader     c Via: 1.1 varnish-v4
>
>          2 VCL_call       c DELIVER
>
>          2 RespHeader     c X-Cache: MISS
>
>          2 RespUnset      c Via: 1.1 varnish-v4
>
>          2 RespUnset      c X-Generator: Drupal 7 (http://drupal.org)
>
>          2 RespUnset      c Server: Apache
>
>          2 VCL_return     c deliver
>
>          2 Timestamp      c Process: 1474629891.152737 1.552510 0.000054
>
>          2 RespHeader     c Content-Length: 7685
>
>          2 Debug          c "RES_MODE 2"
>
>          2 RespHeader     c Connection: keep-alive
>
>          2 Timestamp      c Resp: 1474629891.152783 1.552556 0.000046
>
>          2 ReqAcct        c 791 0 791 500 7685 8185
>
>          2 End            c
>
>
>
>
>
> *From:* Guillaume Quintard [mailto:guillaume at varnish-software.com]
> *Sent:* Friday, September 23, 2016 8:12 AM
> *To:* Drew, AJ
> *Cc:* varnish-misc at varnish-cache.org
> *Subject:* Re: Problem Upgrade to Varnish 4.1 - Multiple Backends Not
> Working
>
>
>
> What does "stop working" mean? Does the vcl fail to compile? do you
> receive 503s? does it open a portal to an ancient dimension filled with
> unspoken nightmares? Help us help you.
>
>
> --
>
> Guillaume Quintard
>
>
>
> On Fri, Sep 23, 2016 at 1:52 PM, Drew, AJ <Drew.AJ at principal.com> wrote:
>
> Hello,
>
>
>
> We are trying to upgrade to Varnish 4.1 from Varnish 3.0.5.
>
>
>
> Most things are working fine, but a big problem we are running into is
> that we have multiple backends.  We switch backends based on URLs.  This
> was working fine in Varnish 3, but when we switched over to Varnish 4, it
> stopped working.
>
>
>
> I have tried many variations of the URLs, and even simplified the
> directors so that they only contain one backend (in my tests) so that I
> could switch from trying the backend directly to trying the director.
>
>
>
> Using either of the options does not seem to matter.
>
>
>
> Has anyone else run into this problem?
>
>
>
> Here are the sections of my VCL that contain the backends / director
> definitions and the if statements to switch.  I have also left in the
> different options (lines) I have used to come up with different
> combinations for testing.
>
>
>
> probe B_probe {
>
>         .request =
>
>         "GET  /zzzzzz/status.php HTTP/1.1"
>
>         "HOST: xxxxxx.example.com"
>
>         "Connection: close";
>
>         .timeout   = 0.3 s;
>
>         .window    = 8;
>
>         .threshold = 7;
>
> }
>
>
>
> backend B_backend1 {
>
>         .host = "162.131.196.220";
>
>         .port = "8080";
>
>         .probe = B_probe;
>
> }
>
>
>
> backend syscheck {
>
> #        .host = "localhost";
>
>         .host = "127.0.0.1";
>
>         .port = "8081";
>
> }
>
>
>
> backend searchBackend {
>
>         .host = "rp.www.example.com";
>
>         .port = "80";
>
> }
>
>
>
> backend providerBackend {
>
>         .host = "172.23.82.28";
>
>         .port = "80";
>
> }
>
>
>
> sub vcl_init {
>
>         new examplehosts = directors.round_robin();
>
>         examplehosts.add_backend(B_backend1);
>
>
>
>         new examplesyscheck = directors.round_robin();
>
>         examplesyscheck.add_backend(syscheck);
>
> }
>
>
>
>
>
>
>
>
>
> sub vcl_recv {
>
>
>
> #       set req.backend_hint = examplehosts.backend();
>
>         set req.backend_hint = B_backend1;
>
>
>
>         if (req.url ~ "^/syscheck/.*$") {
>
> #       if (req.url ~ "^/syscheck/*") {
>
> #       if (req.url ~ "^/syscheck/") {
>
>                 set req.backend_hint = syscheck;
>
> #               set req.backend_hint = examplesyscheck.backend();
>
>                 return(pass);
>
>         }
>
>
>
>         # Add simple health check for upstream load balancers
>
>         if (req.url ~ "^/health-check") {
>
>                 return( synth(200, "UP"));
>
>         }
>
> ...
>
>
>
>
>
> Thanks!
>
> A J Drew
>
>
>
>
>
> -----Message Disclaimer-----
>
> This e-mail message is intended only for the use of the individual or
> entity to which it is addressed, and may contain information that is
> privileged, confidential and exempt from disclosure under applicable law.
> If you are not the intended recipient, any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this communication in error, please notify us immediately by reply email to
> Connect at principal.com and delete or destroy all copies of the original
> message and attachments thereto. Email sent to or from the Principal
> Financial Group or any of its member companies may be retained as required
> by law or regulation.
>
> Nothing in this message is intended to constitute an Electronic signature
> for purposes of the Uniform Electronic Transactions Act (UETA) or the
> Electronic Signatures in Global and National Commerce Act ("E-Sign") unless
> a specific statement to the contrary is included in this message.
>
> If you no longer wish to receive any further solicitation from the
> Principal Financial Group you may unsubscribe at
> https://www.principal.com/do-not-contact-form any time.
>
> If you are a Canadian resident and no longer wish to receive commercial
> electronic messages you may unsubscribe at https://www.principal.com/do-
> not-email-request-canadian-residents any time.
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160923/da2aac55/attachment.html>

From infos at opendoc.net  Sun Sep 25 18:48:49 2016
From: infos at opendoc.net (Alexandre)
Date: Sun, 25 Sep 2016 20:48:49 +0200
Subject: Connections "CLOSE_WAIT" that never close
Message-ID: <20160925204849.0cfbeaea@R2D2>

Hello everyone,

for some time, we have connections "CLOSE_WAIT" that never close. We
use Akamai front of Varnish for many applications.

I adjusted my sysctl.conf but it does not work for this application.
---
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 20
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
---

Do you have an idea to fix this problem?

Regards,
Alexandre.

---
tcp        1      0 my_server:7042         88.221.82.37:52443
CLOSE_WAIT  18187/varnishd tcp        1      0 my_server:7042
2.16.117.87:36012       CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         95.101.182.125:43760    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
92.123.226.166:58796    CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.159:52244      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
88.221.113.45:59244     CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.159:53067      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
88.221.113.45:54092     CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         95.101.182.150:62044    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
88.221.113.45:59123     CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         92.123.226.166:40948    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
173.223.10.118:51051    CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         95.101.182.125:53566    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.16.117.103:39918      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         173.223.10.118:54579    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.181:45599      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.159:45879      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.159:61824      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.159:59291      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.159:37456      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.119:50724      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.119:52487      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         95.101.182.143:42309    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
88.221.112.246:41614    CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.119:41737      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.181:46179      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.159:62914      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.119:52647      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         88.221.112.246:37128    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.159:58640      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         92.123.226.174:56061    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.119:63367      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.159:34843      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.119:35159      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         173.223.10.118:46862    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
88.221.112.246:34340    CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.119:46095      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.159:46841      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         92.123.226.174:45993    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.119:45478      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.119:43897      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.119:40259      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         92.123.226.174:39482    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.119:52310      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         88.221.112.246:62295    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.22.112.159:60489      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.119:37733      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
88.221.82.14:35025      CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.119:42471      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.16.117.92:62335       CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         88.221.82.14:59443      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
92.123.226.166:55261    CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         92.123.226.166:48501    CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
2.18.244.28:64960       CLOSE_WAIT  18187/varnishd tcp        1      0
my_server:7042         2.22.112.119:59856      CLOSE_WAIT
18187/varnishd tcp        1      0 my_server:7042
88.221.82.37:64593 CLOSE_WAIT  18187/varnishd  
---



From miguel_3_gonzalez at yahoo.es  Sun Sep 25 20:14:16 2016
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Sun, 25 Sep 2016 22:14:16 +0200
Subject: bitrate and memory allocated
In-Reply-To: <CAJ6ZYQzAO8s4C-PE1ROTWG_-U_S=9vn1JWhi8RqsN-NPDL_1vg@mail.gmail.com>
References: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>
 <c4b627c6-67e0-eddc-cb97-e059addd45a9@yahoo.es>
 <CABoVN9AJi8=Y3KSYjHZJgHSnGgeZA7n2Jp9Ly-PLjz5e5NWMpg@mail.gmail.com>
 <c5d198ef-63c1-4b13-5850-35b844dbb156@yahoo.es>
 <CABoVN9C=aXS1FFRBJjxSqeHfSFQ7wGJdo5w_3APsZza=qFihsw@mail.gmail.com>
 <f108f8cc-482d-04cd-0cf7-815a10d57b0d@yahoo.es>
 <CAJ6ZYQzAO8s4C-PE1ROTWG_-U_S=9vn1JWhi8RqsN-NPDL_1vg@mail.gmail.com>
Message-ID: <3e2d7441-14ed-501e-a39d-d3f1760288cc@yahoo.es>

Still I don?t understand the numbers. I have set the Varnish storage in
memory of 6 Gb:

SMA.s0.c_req                           1040458         3.53 Allocator
requests
SMA.s0.c_fail                           715580         2.43 Allocator
failures
SMA.s0.c_bytes                      8027903959     27242.60 Bytes allocated
SMA.s0.c_freed                      7923307928     26887.65 Bytes freed
SMA.s0.g_alloc                            5217          .   Allocations
outstanding
SMA.s0.g_bytes                       104596031          .   Bytes
outstanding
SMA.s0.g_space                          261569          .   Bytes available
SMA.Transient.c_req                     236994         0.80 Allocator
requests
SMA.Transient.c_fail                         0         0.00 Allocator
failures
SMA.Transient.c_bytes               2047296885      6947.48 Bytes allocated
SMA.Transient.c_freed               2047294773      6947.47 Bytes freed
SMA.Transient.g_alloc                        6          .   Allocations
outstanding
SMA.Transient.g_bytes                     2112          .   Bytes
outstanding
SMA.Transient.g_space                        0          .   Bytes available



On 09/23/16 11:16 AM, Guillaume Quintard wrote:
> Focus on g_space (free space in the storage) and g_bytes (used bytes in
> the storage).
> 
> -- 
> Guillaume Quintard
> 
> On Tue, Sep 6, 2016 at 12:08 PM, Miguel Gonz?lez
> <miguel_3_gonzalez at yahoo.es <mailto:miguel_3_gonzalez at yahoo.es>> wrote:
> 
>     On 09/05/16 10:44 PM, Dridi Boukelmoune wrote:
>     >> Any URL? I have googled for a few hours but I didn?t get a clear
>     >> understanding from the official manuals.
>     >
>     > https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma
>     <https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma>
> 
> 
>     I had read the 4.0 docs but I find them quite short and not very
>     explanatory. What are the counters? The current use of memory? The
>     amount allocated since Varnish was restarted (historical)? At least for
>     me, I would like an example to better understand what those counters
>     mean.
> 
>     Thanks
> 
>     Miguel
> 
>     _______________________________________________
>     varnish-misc mailing list
>     varnish-misc at varnish-cache.org <mailto:varnish-misc at varnish-cache.org>
>     https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>     <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
> 
> 




From guillaume at varnish-software.com  Mon Sep 26 07:40:35 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Mon, 26 Sep 2016 09:40:35 +0200
Subject: bitrate and memory allocated
In-Reply-To: <3e2d7441-14ed-501e-a39d-d3f1760288cc@yahoo.es>
References: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>
 <c4b627c6-67e0-eddc-cb97-e059addd45a9@yahoo.es>
 <CABoVN9AJi8=Y3KSYjHZJgHSnGgeZA7n2Jp9Ly-PLjz5e5NWMpg@mail.gmail.com>
 <c5d198ef-63c1-4b13-5850-35b844dbb156@yahoo.es>
 <CABoVN9C=aXS1FFRBJjxSqeHfSFQ7wGJdo5w_3APsZza=qFihsw@mail.gmail.com>
 <f108f8cc-482d-04cd-0cf7-815a10d57b0d@yahoo.es>
 <CAJ6ZYQzAO8s4C-PE1ROTWG_-U_S=9vn1JWhi8RqsN-NPDL_1vg@mail.gmail.com>
 <3e2d7441-14ed-501e-a39d-d3f1760288cc@yahoo.es>
Message-ID: <CAJ6ZYQznKOYcRZ3KpiG5iD0k7zVtb0TLn3veXqHw_PBjJw=iGw@mail.gmail.com>

Your varnishstat doesn't seem to indicate that (s0.g_bytes + s0.g_space =
100M). Can you give us the output of "ps aux | grep varnish" ?

-- 
Guillaume Quintard

On Sun, Sep 25, 2016 at 10:14 PM, Miguel Gonz?lez <
miguel_3_gonzalez at yahoo.es> wrote:

> Still I don?t understand the numbers. I have set the Varnish storage in
> memory of 6 Gb:
>
> SMA.s0.c_req                           1040458         3.53 Allocator
> requests
> SMA.s0.c_fail                           715580         2.43 Allocator
> failures
> SMA.s0.c_bytes                      8027903959     27242.60 Bytes allocated
> SMA.s0.c_freed                      7923307928     26887.65 Bytes freed
> SMA.s0.g_alloc                            5217          .   Allocations
> outstanding
> SMA.s0.g_bytes                       104596031          .   Bytes
> outstanding
> SMA.s0.g_space                          261569          .   Bytes available
> SMA.Transient.c_req                     236994         0.80 Allocator
> requests
> SMA.Transient.c_fail                         0         0.00 Allocator
> failures
> SMA.Transient.c_bytes               2047296885      6947.48 Bytes
> allocated
> SMA.Transient.c_freed               2047294773      6947.47 Bytes freed
> SMA.Transient.g_alloc                        6          .   Allocations
> outstanding
> SMA.Transient.g_bytes                     2112          .   Bytes
> outstanding
> SMA.Transient.g_space                        0          .   Bytes available
>
>
>
> On 09/23/16 11:16 AM, Guillaume Quintard wrote:
> > Focus on g_space (free space in the storage) and g_bytes (used bytes in
> > the storage).
> >
> > --
> > Guillaume Quintard
> >
> > On Tue, Sep 6, 2016 at 12:08 PM, Miguel Gonz?lez
> > <miguel_3_gonzalez at yahoo.es <mailto:miguel_3_gonzalez at yahoo.es>> wrote:
> >
> >     On 09/05/16 10:44 PM, Dridi Boukelmoune wrote:
> >     >> Any URL? I have googled for a few hours but I didn?t get a clear
> >     >> understanding from the official manuals.
> >     >
> >     > https://www.varnish-cache.org/docs/4.1/reference/varnish-
> counters.html#per-malloc-storage-counters-sma
> >     <https://www.varnish-cache.org/docs/4.1/reference/
> varnish-counters.html#per-malloc-storage-counters-sma>
> >
> >
> >     I had read the 4.0 docs but I find them quite short and not very
> >     explanatory. What are the counters? The current use of memory? The
> >     amount allocated since Varnish was restarted (historical)? At least
> for
> >     me, I would like an example to better understand what those counters
> >     mean.
> >
> >     Thanks
> >
> >     Miguel
> >
> >     _______________________________________________
> >     varnish-misc mailing list
> >     varnish-misc at varnish-cache.org <mailto:varnish-misc at varnish-
> cache.org>
> >     https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >     <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160926/1b3807a3/attachment.html>

From miguel_3_gonzalez at yahoo.es  Mon Sep 26 21:48:21 2016
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Mon, 26 Sep 2016 23:48:21 +0200
Subject: bitrate and memory allocated
In-Reply-To: <CAJ6ZYQznKOYcRZ3KpiG5iD0k7zVtb0TLn3veXqHw_PBjJw=iGw@mail.gmail.com>
References: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>
 <c4b627c6-67e0-eddc-cb97-e059addd45a9@yahoo.es>
 <CABoVN9AJi8=Y3KSYjHZJgHSnGgeZA7n2Jp9Ly-PLjz5e5NWMpg@mail.gmail.com>
 <c5d198ef-63c1-4b13-5850-35b844dbb156@yahoo.es>
 <CABoVN9C=aXS1FFRBJjxSqeHfSFQ7wGJdo5w_3APsZza=qFihsw@mail.gmail.com>
 <f108f8cc-482d-04cd-0cf7-815a10d57b0d@yahoo.es>
 <CAJ6ZYQzAO8s4C-PE1ROTWG_-U_S=9vn1JWhi8RqsN-NPDL_1vg@mail.gmail.com>
 <3e2d7441-14ed-501e-a39d-d3f1760288cc@yahoo.es>
 <CAJ6ZYQznKOYcRZ3KpiG5iD0k7zVtb0TLn3veXqHw_PBjJw=iGw@mail.gmail.com>
Message-ID: <1e56943a-dfbf-bf41-1f10-7401e390cbc3@yahoo.es>

ps -aux |grep varnish
Warning: bad syntax, perhaps a bogus '-'? See
/usr/share/doc/procps-3.2.8/FAQ
root      4487  0.0  0.0 103312   820 pts/0    S+   23:47   0:00 grep
varnish
varnish  18206  0.0  0.0 119904  4932 ?        Ss   Sep22   1:28
/usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082
-t 120 -p thread_pools=1 -p thread_pool_add_delay=2 -p
thread_pool_min=500 -p thread_pool_max=2000 -p thread_pool_timeout=120
-p http_gzip_support=true
varnish  18210  1.6  3.5 996192 756816 ?       Sl   Sep22 103:53
/usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082
-t 120 -p thread_pools=1 -p thread_pool_add_delay=2 -p
thread_pool_min=500 -p thread_pool_max=2000 -p thread_pool_timeout=120
-p http_gzip_support=true



On 09/26/16 9:40 AM, Guillaume Quintard wrote:
> Your varnishstat doesn't seem to indicate that (s0.g_bytes + s0.g_space
> = 100M). Can you give us the output of "ps aux | grep varnish" ?
> 
> -- 
> Guillaume Quintard
> 
> On Sun, Sep 25, 2016 at 10:14 PM, Miguel Gonz?lez
> <miguel_3_gonzalez at yahoo.es <mailto:miguel_3_gonzalez at yahoo.es>> wrote:
> 
>     Still I don?t understand the numbers. I have set the Varnish storage in
>     memory of 6 Gb:
> 
>     SMA.s0.c_req                           1040458         3.53 Allocator
>     requests
>     SMA.s0.c_fail                           715580         2.43 Allocator
>     failures
>     SMA.s0.c_bytes                      8027903959     27242.60 Bytes
>     allocated
>     SMA.s0.c_freed                      7923307928     26887.65 Bytes freed
>     SMA.s0.g_alloc                            5217          .   Allocations
>     outstanding
>     SMA.s0.g_bytes                       104596031          .   Bytes
>     outstanding
>     SMA.s0.g_space                          261569          .   Bytes
>     available
>     SMA.Transient.c_req                     236994         0.80 Allocator
>     requests
>     SMA.Transient.c_fail                         0         0.00 Allocator
>     failures
>     SMA.Transient.c_bytes               2047296885 <tel:2047296885>     
>     6947.48 Bytes allocated
>     SMA.Transient.c_freed               2047294773 <tel:2047294773>     
>     6947.47 Bytes freed
>     SMA.Transient.g_alloc                        6          .   Allocations
>     outstanding
>     SMA.Transient.g_bytes                     2112          .   Bytes
>     outstanding
>     SMA.Transient.g_space                        0          .   Bytes
>     available
> 
> 
> 
>     On 09/23/16 11:16 AM, Guillaume Quintard wrote:
>     > Focus on g_space (free space in the storage) and g_bytes (used bytes in
>     > the storage).
>     >
>     > --
>     > Guillaume Quintard
>     >
>     > On Tue, Sep 6, 2016 at 12:08 PM, Miguel Gonz?lez
>     > <miguel_3_gonzalez at yahoo.es <mailto:miguel_3_gonzalez at yahoo.es>
>     <mailto:miguel_3_gonzalez at yahoo.es
>     <mailto:miguel_3_gonzalez at yahoo.es>>> wrote:
>     >
>     >     On 09/05/16 10:44 PM, Dridi Boukelmoune wrote:
>     >     >> Any URL? I have googled for a few hours but I didn?t get a clear
>     >     >> understanding from the official manuals.
>     >     >
>     >     > https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma
>     <https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma>
>     >     <https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma
>     <https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma>>
>     >
>     >
>     >     I had read the 4.0 docs but I find them quite short and not very
>     >     explanatory. What are the counters? The current use of memory? The
>     >     amount allocated since Varnish was restarted (historical)? At least for
>     >     me, I would like an example to better understand what those counters
>     >     mean.
>     >
>     >     Thanks
>     >
>     >     Miguel
>     >
>     >     _______________________________________________
>     >     varnish-misc mailing list
>     >     varnish-misc at varnish-cache.org
>     <mailto:varnish-misc at varnish-cache.org>
>     <mailto:varnish-misc at varnish-cache.org
>     <mailto:varnish-misc at varnish-cache.org>>
>     >   
>      https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>     <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>     >   
>      <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>     <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>>
>     >
>     >
> 
> 




From slink at schokola.de  Tue Sep 27 08:31:34 2016
From: slink at schokola.de (Nils Goroll)
Date: Tue, 27 Sep 2016 10:31:34 +0200
Subject: Connections "CLOSE_WAIT" that never close
In-Reply-To: <20160925204849.0cfbeaea@R2D2>
References: <20160925204849.0cfbeaea@R2D2>
Message-ID: <2a54323b-c79b-8629-faa9-71002d6ba7c4@schokola.de>

On 25/09/16 20:48, Alexandre wrote:
> for some time, we have connections "CLOSE_WAIT" that never close. We
> use Akamai front of Varnish for many applications.

I've actually seen a similar issue last week. Would you do us a favor and file a
bug under https://github.com/varnishcache/varnish-cache/issues

In particular, please provide info on the exact varnish version you are using.
Also, are you absolutely certain that the most CLOSE_WAITs are on the client
side? In other words, when you generated your netstat excerpt, where you
filtering for client/backend connections?

Thanks, Nils



From miguel_3_gonzalez at yahoo.es  Fri Sep 30 09:48:29 2016
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Fri, 30 Sep 2016 11:48:29 +0200
Subject: bitrate and memory allocated
In-Reply-To: <1e56943a-dfbf-bf41-1f10-7401e390cbc3@yahoo.es>
References: <945d4f9d-717c-f458-f8f0-6e75ddd34026@yahoo.es>
 <c4b627c6-67e0-eddc-cb97-e059addd45a9@yahoo.es>
 <CABoVN9AJi8=Y3KSYjHZJgHSnGgeZA7n2Jp9Ly-PLjz5e5NWMpg@mail.gmail.com>
 <c5d198ef-63c1-4b13-5850-35b844dbb156@yahoo.es>
 <CABoVN9C=aXS1FFRBJjxSqeHfSFQ7wGJdo5w_3APsZza=qFihsw@mail.gmail.com>
 <f108f8cc-482d-04cd-0cf7-815a10d57b0d@yahoo.es>
 <CAJ6ZYQzAO8s4C-PE1ROTWG_-U_S=9vn1JWhi8RqsN-NPDL_1vg@mail.gmail.com>
 <3e2d7441-14ed-501e-a39d-d3f1760288cc@yahoo.es>
 <CAJ6ZYQznKOYcRZ3KpiG5iD0k7zVtb0TLn3veXqHw_PBjJw=iGw@mail.gmail.com>
 <1e56943a-dfbf-bf41-1f10-7401e390cbc3@yahoo.es>
Message-ID: <e57d4299-41fa-2315-1190-38ee3bcfb33b@yahoo.es>

Hi,

 Sorry, I have just realized that I commented out an old line from
Varnish 4.0 and the storage size was not defined after I migrated to 4.1

 Thanks

 Miguel

On 09/26/16 11:48 PM, Miguel Gonz?lez wrote:
> ps -aux |grep varnish
> Warning: bad syntax, perhaps a bogus '-'? See
> /usr/share/doc/procps-3.2.8/FAQ
> root      4487  0.0  0.0 103312   820 pts/0    S+   23:47   0:00 grep
> varnish
> varnish  18206  0.0  0.0 119904  4932 ?        Ss   Sep22   1:28
> /usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082
> -t 120 -p thread_pools=1 -p thread_pool_add_delay=2 -p
> thread_pool_min=500 -p thread_pool_max=2000 -p thread_pool_timeout=120
> -p http_gzip_support=true
> varnish  18210  1.6  3.5 996192 756816 ?       Sl   Sep22 103:53
> /usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082
> -t 120 -p thread_pools=1 -p thread_pool_add_delay=2 -p
> thread_pool_min=500 -p thread_pool_max=2000 -p thread_pool_timeout=120
> -p http_gzip_support=true
> 
> 
> 
> On 09/26/16 9:40 AM, Guillaume Quintard wrote:
>> Your varnishstat doesn't seem to indicate that (s0.g_bytes + s0.g_space
>> = 100M). Can you give us the output of "ps aux | grep varnish" ?
>>
>> -- 
>> Guillaume Quintard
>>
>> On Sun, Sep 25, 2016 at 10:14 PM, Miguel Gonz?lez
>> <miguel_3_gonzalez at yahoo.es <mailto:miguel_3_gonzalez at yahoo.es>> wrote:
>>
>>     Still I don?t understand the numbers. I have set the Varnish storage in
>>     memory of 6 Gb:
>>
>>     SMA.s0.c_req                           1040458         3.53 Allocator
>>     requests
>>     SMA.s0.c_fail                           715580         2.43 Allocator
>>     failures
>>     SMA.s0.c_bytes                      8027903959     27242.60 Bytes
>>     allocated
>>     SMA.s0.c_freed                      7923307928     26887.65 Bytes freed
>>     SMA.s0.g_alloc                            5217          .   Allocations
>>     outstanding
>>     SMA.s0.g_bytes                       104596031          .   Bytes
>>     outstanding
>>     SMA.s0.g_space                          261569          .   Bytes
>>     available
>>     SMA.Transient.c_req                     236994         0.80 Allocator
>>     requests
>>     SMA.Transient.c_fail                         0         0.00 Allocator
>>     failures
>>     SMA.Transient.c_bytes               2047296885 <tel:2047296885>     
>>     6947.48 Bytes allocated
>>     SMA.Transient.c_freed               2047294773 <tel:2047294773>     
>>     6947.47 Bytes freed
>>     SMA.Transient.g_alloc                        6          .   Allocations
>>     outstanding
>>     SMA.Transient.g_bytes                     2112          .   Bytes
>>     outstanding
>>     SMA.Transient.g_space                        0          .   Bytes
>>     available
>>
>>
>>
>>     On 09/23/16 11:16 AM, Guillaume Quintard wrote:
>>     > Focus on g_space (free space in the storage) and g_bytes (used bytes in
>>     > the storage).
>>     >
>>     > --
>>     > Guillaume Quintard
>>     >
>>     > On Tue, Sep 6, 2016 at 12:08 PM, Miguel Gonz?lez
>>     > <miguel_3_gonzalez at yahoo.es <mailto:miguel_3_gonzalez at yahoo.es>
>>     <mailto:miguel_3_gonzalez at yahoo.es
>>     <mailto:miguel_3_gonzalez at yahoo.es>>> wrote:
>>     >
>>     >     On 09/05/16 10:44 PM, Dridi Boukelmoune wrote:
>>     >     >> Any URL? I have googled for a few hours but I didn?t get a clear
>>     >     >> understanding from the official manuals.
>>     >     >
>>     >     > https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma
>>     <https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma>
>>     >     <https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma
>>     <https://www.varnish-cache.org/docs/4.1/reference/varnish-counters.html#per-malloc-storage-counters-sma>>
>>     >
>>     >
>>     >     I had read the 4.0 docs but I find them quite short and not very
>>     >     explanatory. What are the counters? The current use of memory? The
>>     >     amount allocated since Varnish was restarted (historical)? At least for
>>     >     me, I would like an example to better understand what those counters
>>     >     mean.
>>     >
>>     >     Thanks
>>     >
>>     >     Miguel
>>     >
>>     >     _______________________________________________
>>     >     varnish-misc mailing list
>>     >     varnish-misc at varnish-cache.org
>>     <mailto:varnish-misc at varnish-cache.org>
>>     <mailto:varnish-misc at varnish-cache.org
>>     <mailto:varnish-misc at varnish-cache.org>>
>>     >   
>>      https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>     <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>>     >   
>>      <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>     <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>>
>>     >
>>     >
>>
>>
> 




From subharaj.manna at gmail.com  Fri Sep 30 15:58:00 2016
From: subharaj.manna at gmail.com (Debraj Manna)
Date: Fri, 30 Sep 2016 21:28:00 +0530
Subject: Round Robin Load Balancing in Varnish
Message-ID: <CAF6DVKN7peZHo3r-B9qHbi2Zge3HtxdqLxwJ1Ls-QQrkCH22hQ@mail.gmail.com>

Hi

Let's say I am having four backends b1, b2, b3 & b4 added in round robin.
If one backend(say b1 ) is not healthy, will varnish take out b1 from
backend pool & send the incoming requests only to b2, b3 or b4 or it will
keep on doing round robin among all the four servers with the requests
going to b1 failing?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160930/f9754fd0/attachment.html>

From subharaj.manna at gmail.com  Fri Sep 30 16:04:06 2016
From: subharaj.manna at gmail.com (Debraj Manna)
Date: Fri, 30 Sep 2016 21:34:06 +0530
Subject: Round Robin Load Balancing in Varnish
In-Reply-To: <CAF6DVKN7peZHo3r-B9qHbi2Zge3HtxdqLxwJ1Ls-QQrkCH22hQ@mail.gmail.com>
References: <CAF6DVKN7peZHo3r-B9qHbi2Zge3HtxdqLxwJ1Ls-QQrkCH22hQ@mail.gmail.com>
Message-ID: <CAF6DVKOb-a0xBG1LSmh418OThTdYDDG0DzVfTfa8cV8icmyh2Q@mail.gmail.com>

I am on varnish 4.1

On Fri, Sep 30, 2016 at 9:28 PM, Debraj Manna <subharaj.manna at gmail.com>
wrote:

> Hi
>
> Let's say I am having four backends b1, b2, b3 & b4 added in round robin.
> If one backend(say b1 ) is not healthy, will varnish take out b1 from
> backend pool & send the incoming requests only to b2, b3 or b4 or it will
> keep on doing round robin among all the four servers with the requests
> going to b1 failing?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160930/4988352b/attachment.html>