From perbu at varnish-software.com  Sat Sep  1 15:45:07 2012
From: perbu at varnish-software.com (Per Buer)
Date: Sat, 1 Sep 2012 17:45:07 +0200
Subject: Output Buffering with Varnish
In-Reply-To: <1643260013.216804.1346439471645.JavaMail.root@thomas-krenn.com>
References: <20120831145833.GA12138@nat.myhome>
 <1643260013.216804.1346439471645.JavaMail.root@thomas-krenn.com>
Message-ID: <CAOXZevDFR19Z8CaoaG9K12n7O-W_YbovFABbJS7Er1dbPK8pQQ@mail.gmail.com>

Use the 3.0.3plus release. It has proper streaming (and working persistent
storage).

Per.

(Note: It is not as well supported as the 3.0.3 release)

On Fri, Aug 31, 2012 at 8:57 PM, Christoph Mitasch <
cmitasch at thomas-krenn.com> wrote:

> great, that works!
>
> One thing is open now, the same request can only be delivered to one
> client at a time. Any ideas?
>
> Christoph
>
> ----- Urspr?ngliche Mail -----
> Von: "Paul A. Procacci" <pprocacci at datapipe.com>
> An: "Christoph Mitasch" <cmitasch at thomas-krenn.com>
> CC: varnish-misc at varnish-cache.org
> Gesendet: Freitag, 31. August 2012 16:58:33
> Betreff: Re: Output Buffering with Varnish
>
> `man vcl`
>
> ############
> beresp.do_stream
>           Deliver  the  object to the client directly without fetching the
>           whole object into varnish. If this request is  pass'ed  it  will
>           not be stored in memory. As of Varnish Cache 3.0 the object will
>           marked as busy as it is delivered so only client can access  the
>           object.
> ############
>
> This probably does what you want.
>
> ~Paul
>
> On Fri, Aug 31, 2012 at 04:39:11PM +0200, Christoph Mitasch wrote:
> > Hello,
> >
> > I'm using Varnish 3.0.2-2~bpo60+1 on Debian Squeeze.
> >
> > The following PHP code behaves differently when Varnish is used. The
> output is returned after 10 seconds at once. When Varnish is not used, an
> output is returned every second.
> > <?
> > echo "started";
> > for ($i=0;$i<10;$i++) {
> >  echo "+++";
> >  echo str_repeat(" ", 4096);
> >  sleep(1);
> >  flush();
> >  ob_flush();
> > }
> > echo "stopped";
> > ?>
> >
> > How can Varnish be configured that scripts making use of output
> buffering are working as expected?
> >
> > Thank you,
> > Christoph
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> ________________________________
>
> This message may contain confidential or privileged information. If you
> are not the intended recipient, please advise us immediately and delete
> this message. See http://www.datapipe.com/legal/email_disclaimer/ for
> further information on confidentiality and the risks of non-secure
> electronic communication. If you cannot access these links, please notify
> us by reply message and we will send the contents to you.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



-- 
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120901/3668100f/attachment.html>

From zeeshan at infoshield.info  Sun Sep  2 11:30:13 2012
From: zeeshan at infoshield.info (Zeeshan Ali Shah)
Date: Sun, 2 Sep 2012 16:30:13 +0500
Subject: Mp3 settings ?
Message-ID: <CAJSm=nsP9xNHpN6aCvrC=k88wTe6FuTFqnp64MdKKCjhjY7z-w@mail.gmail.com>

Hi,
For images this settings work
  if (req.url ~
"(?i)\.(pdf|mp3|png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$") {
    # beresp == Back-end response from the web server.
    unset beresp.http.set-cookie;
# set the ttl to 10  minutes
 set beresp.ttl = 10m;

  }


but for mp3 it first cache all and than stream ..


any hint such that varnish start streaming with out waiting for whole file
in cache ?


varnish version : 3.0.2-1ubuntu0.1

Zeeshan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120902/ef2e8d92/attachment.html>

From jocelyn.delarosa at smartjog.com  Mon Sep  3 08:34:46 2012
From: jocelyn.delarosa at smartjog.com (Jocelyn De La Rosa)
Date: Mon, 03 Sep 2012 10:34:46 +0200
Subject: Mp3 settings ?
In-Reply-To: <CAJSm=nsP9xNHpN6aCvrC=k88wTe6FuTFqnp64MdKKCjhjY7z-w@mail.gmail.com>
References: <CAJSm=nsP9xNHpN6aCvrC=k88wTe6FuTFqnp64MdKKCjhjY7z-w@mail.gmail.com>
Message-ID: <50446BA6.40701@smartjog.com>

Hi,

You should use the 3.0.3-plus release that include streaming support 
(and persistent work)

https://github.com/mbgrydeland/varnish-cache

Jocelyn.

On 09/02/2012 01:30 PM, Zeeshan Ali Shah wrote:
> Hi,
> For images this settings work
>   if (req.url ~ 
> "(?i)\.(pdf|mp3|png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$") 
> {
>     # beresp == Back-end response from the web server.
>     unset beresp.http.set-cookie;
> # set the ttl to 10  minutes
>  set beresp.ttl = 10m;
>
>   }
>
>
> but for mp3 it first cache all and than stream ..
>
>
> any hint such that varnish start streaming with out waiting for whole 
> file in cache ?
>
>
> varnish version : 3.0.2-1ubuntu0.1
>
> Zeeshan
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


-- 
Jocelyn De La Rosa - Developer, Research&  Engineering
SmartJog | www.smartjog.com
27 Blvd Hippolyte Marqu?s, 94200 Ivry-sur-Seine, France

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120903/46a84e42/attachment.html>

From sahmed1020 at gmail.com  Mon Sep  3 20:44:08 2012
From: sahmed1020 at gmail.com (S Ahmed)
Date: Mon, 3 Sep 2012 16:44:08 -0400
Subject: how fast can you expire a cache key?
Message-ID: <CAG2rwuMVEbByvYEVpXhTDgh=JHF5mEGG2Bnydk66f2gDw-2H9w@mail.gmail.com>

If I have hundreds of thousands to millions of pages in a web application,
and the pages are being updated at a rate of a few hundred per second (and
thus require the cache to be invalidated), is this a reasonable level of
cache expiration that I hit varnish with?

Would love to hear the high end of cache expiration that others are doing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120903/abac1ffe/attachment.html>

From perbu at varnish-software.com  Mon Sep  3 20:50:56 2012
From: perbu at varnish-software.com (Per Buer)
Date: Mon, 3 Sep 2012 22:50:56 +0200
Subject: how fast can you expire a cache key?
In-Reply-To: <CAG2rwuMVEbByvYEVpXhTDgh=JHF5mEGG2Bnydk66f2gDw-2H9w@mail.gmail.com>
References: <CAG2rwuMVEbByvYEVpXhTDgh=JHF5mEGG2Bnydk66f2gDw-2H9w@mail.gmail.com>
Message-ID: <CAOXZevAbiJ3NZLFaSuEuo5g=3Uj8zq2R5R+sxQ5WeMLLb-misA@mail.gmail.com>

On Mon, Sep 3, 2012 at 10:44 PM, S Ahmed <sahmed1020 at gmail.com> wrote:

> If I have hundreds of thousands to millions of pages in a web application,
> and the pages are being updated at a rate of a few hundred per second (and
> thus require the cache to be invalidated), is this a reasonable level of
> cache expiration that I hit varnish with?


Yes.


> Would love to hear the high end of cache expiration that others are doing.
>

You're fine as long as you PURGE the objects. The bans probably won't work
so well at those speeds.



-- 
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120903/50cfb627/attachment.html>

From rosco at hostinglab.nl  Tue Sep  4 15:58:49 2012
From: rosco at hostinglab.nl (Rosco Nap)
Date: Tue, 4 Sep 2012 17:58:49 +0200
Subject: First hit always cached.
Message-ID: <F016B57F-05FF-4274-B3C1-37BD604929F2@hostinglab.nl>

Hi, 

Is it possible to have the first hit on a frontpage index (eq: www.example.com/index.php)
be served from Varnish, and every hit after that  redirected to the backend?

With kind regards,

Rosco 




From millennium.bug at gmail.com  Tue Sep  4 16:04:30 2012
From: millennium.bug at gmail.com (=?ISO-8859-1?B?W0RpZ2l0YWxeRHVkZV0grg==?=)
Date: Tue, 4 Sep 2012 21:04:30 +0500
Subject: First hit always cached.
In-Reply-To: <F016B57F-05FF-4274-B3C1-37BD604929F2@hostinglab.nl>
References: <F016B57F-05FF-4274-B3C1-37BD604929F2@hostinglab.nl>
Message-ID: <CAArAGK=uCJ3a2UtpC0t1ECghzrUoY6PtTAJHe5uwipmSFXtdMw@mail.gmail.com>

I would like to know that too. Varnish seems to do the opposite by default.

On Tue, Sep 4, 2012 at 8:58 PM, Rosco Nap <rosco at hostinglab.nl> wrote:

> Hi,
>
> Is it possible to have the first hit on a frontpage index (eq:
> www.example.com/index.php)
> be served from Varnish, and every hit after that  redirected to the
> backend?
>
> With kind regards,
>
> Rosco
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120904/71f3a193/attachment.html>

From tobias.eichelbroenner at lamp-solutions.de  Tue Sep  4 16:10:15 2012
From: tobias.eichelbroenner at lamp-solutions.de (=?ISO-8859-1?Q?Tobias_Eichelbr=F6nner?=)
Date: Tue, 04 Sep 2012 18:10:15 +0200
Subject: First hit always cached.
In-Reply-To: <F016B57F-05FF-4274-B3C1-37BD604929F2@hostinglab.nl>
References: <F016B57F-05FF-4274-B3C1-37BD604929F2@hostinglab.nl>
Message-ID: <504627E7.7090708@lamp-solutions.de>

Hallo Rosco,

> Is it possible to have the first hit on a frontpage index (eq: www.example.com/index.php)
> be served from Varnish, and every hit after that  redirected to the backend?

first hit from a single user or first hit at all?

Sincerely,

Tobias

-- 
LAMP solutions GmbH
Gostenhofer Hauptstrasse 35
90443 Nuernberg

Amtsgericht Nuernberg: HRB 22366
Geschaeftsfuehrer: Heiko Schubert

Es gelten unsere allgemeinen Geschaeftsbedingungen.
http://www.lamp-solutions.de/agbs/

Telefon      : 0911 / 376 516 0
Fax          : 0911 / 376 516 11
E-Mail       : support at lamp-solutions.de
Web          : www.lamp-solutions.de
Facebook     : http://www.facebook.com/LAMPsolutions
Twitter      : http://twitter.com/#!/lampsolutions



From drais at icantclick.org  Tue Sep  4 16:11:31 2012
From: drais at icantclick.org (david raistrick)
Date: Tue, 4 Sep 2012 12:11:31 -0400 (EDT)
Subject: First hit always cached.
In-Reply-To: <F016B57F-05FF-4274-B3C1-37BD604929F2@hostinglab.nl>
References: <F016B57F-05FF-4274-B3C1-37BD604929F2@hostinglab.nl>
Message-ID: <alpine.BSF.2.00.1209041211130.38542@murf.icantclick.org>

On Tue, 4 Sep 2012, Rosco Nap wrote:

> Is it possible to have the first hit on a frontpage index (eq: 
> www.example.com/index.php) be served from Varnish, and every hit after 
> that redirected to the backend?


A "first" hit, by definition, couldn't already be in the cache - and so 
couldn't be served from the cache. (you could preload your cache, but 
really that would be the "first" hit...)


The whole point of a cache is to reduce hits (decreasing load or latency) 
to the backend - what possible problem would your solution solve?


--
david raistrick        http://www.netmeister.org/news/learn2quote.html
drais at icantclick.org       ascii ribbon campaign - stop html mail
                                 http://www.asciiribbon.org/






From millennium.bug at gmail.com  Tue Sep  4 16:29:14 2012
From: millennium.bug at gmail.com (=?ISO-8859-1?B?W0RpZ2l0YWxeRHVkZV0grg==?=)
Date: Tue, 4 Sep 2012 21:29:14 +0500
Subject: First hit always cached.
In-Reply-To: <504627E7.7090708@lamp-solutions.de>
References: <F016B57F-05FF-4274-B3C1-37BD604929F2@hostinglab.nl>
 <504627E7.7090708@lamp-solutions.de>
Message-ID: <CAArAGKmHWaNiqY_KyeJvqjq3V15GTYjRAH9C4LXTjXXczBQPCQ@mail.gmail.com>

The first hit from all users.

If its serverside cache, each page should load from the cache present on
the server until cache TTL expires.

On Tue, Sep 4, 2012 at 9:10 PM, Tobias Eichelbr?nner <
tobias.eichelbroenner at lamp-solutions.de> wrote:

> Hallo Rosco,
>
> > Is it possible to have the first hit on a frontpage index (eq:
> www.example.com/index.php)
> > be served from Varnish, and every hit after that  redirected to the
> backend?
>
> first hit from a single user or first hit at all?
>
> Sincerely,
>
> Tobias
>
> --
> LAMP solutions GmbH
> Gostenhofer Hauptstrasse 35
> 90443 Nuernberg
>
> Amtsgericht Nuernberg: HRB 22366
> Geschaeftsfuehrer: Heiko Schubert
>
> Es gelten unsere allgemeinen Geschaeftsbedingungen.
> http://www.lamp-solutions.de/agbs/
>
> Telefon      : 0911 / 376 516 0
> Fax          : 0911 / 376 516 11
> E-Mail       : support at lamp-solutions.de
> Web          : www.lamp-solutions.de
> Facebook     : http://www.facebook.com/LAMPsolutions
> Twitter      : http://twitter.com/#!/lampsolutions
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120904/ac7e01f0/attachment.html>

From tobias.eichelbroenner at lamp-solutions.de  Tue Sep  4 17:27:06 2012
From: tobias.eichelbroenner at lamp-solutions.de (=?ISO-8859-1?Q?Tobias_Eichelbr=F6nner?=)
Date: Tue, 04 Sep 2012 19:27:06 +0200
Subject: First hit always cached.
In-Reply-To: <CAArAGKmHWaNiqY_KyeJvqjq3V15GTYjRAH9C4LXTjXXczBQPCQ@mail.gmail.com>
References: <F016B57F-05FF-4274-B3C1-37BD604929F2@hostinglab.nl>
 <504627E7.7090708@lamp-solutions.de>
 <CAArAGKmHWaNiqY_KyeJvqjq3V15GTYjRAH9C4LXTjXXczBQPCQ@mail.gmail.com>
Message-ID: <504639EA.105@lamp-solutions.de>

> The first hit from all users.

as david said, the first hit by definition can not be in any cache.

If you do not count a preloading request, you could make a rule that for
example the first request from outside 127.0.0.1 stores a trigger to a
variable
(https://github.com/varnish/libvmod-var)
so that the url will no longer be cached.

But i really would like to read about any use case for that.


-- 
LAMP solutions GmbH
Gostenhofer Hauptstrasse 35
90443 Nuernberg

Amtsgericht Nuernberg: HRB 22366
Geschaeftsfuehrer: Heiko Schubert

Es gelten unsere allgemeinen Geschaeftsbedingungen.
http://www.lamp-solutions.de/agbs/

Telefon      : 0911 / 376 516 0
Fax          : 0911 / 376 516 11
E-Mail       : support at lamp-solutions.de
Web          : www.lamp-solutions.de
Facebook     : http://www.facebook.com/LAMPsolutions
Twitter      : http://twitter.com/#!/lampsolutions



From millennium.bug at gmail.com  Tue Sep  4 18:25:46 2012
From: millennium.bug at gmail.com (=?ISO-8859-1?B?W0RpZ2l0YWxeRHVkZV0grg==?=)
Date: Tue, 4 Sep 2012 23:25:46 +0500
Subject: First hit always cached.
In-Reply-To: <504639EA.105@lamp-solutions.de>
References: <F016B57F-05FF-4274-B3C1-37BD604929F2@hostinglab.nl>
 <504627E7.7090708@lamp-solutions.de>
 <CAArAGKmHWaNiqY_KyeJvqjq3V15GTYjRAH9C4LXTjXXczBQPCQ@mail.gmail.com>
 <504639EA.105@lamp-solutions.de>
Message-ID: <CAArAGK=FPmJKWMC5E_ZDTgPR5fT+JnSnHPRA8pOhm9+pirJDHg@mail.gmail.com>

I see. How about varnishreplay as a workaround for this?
How successful is it? anyone with a working example?

On Tue, Sep 4, 2012 at 10:27 PM, Tobias Eichelbr?nner <
tobias.eichelbroenner at lamp-solutions.de> wrote:

> > The first hit from all users.
>
> as david said, the first hit by definition can not be in any cache.
>
> If you do not count a preloading request, you could make a rule that for
> example the first request from outside 127.0.0.1 stores a trigger to a
> variable
> (https://github.com/varnish/libvmod-var)
> so that the url will no longer be cached.
>
> But i really would like to read about any use case for that.
>
>
> --
> LAMP solutions GmbH
> Gostenhofer Hauptstrasse 35
> 90443 Nuernberg
>
> Amtsgericht Nuernberg: HRB 22366
> Geschaeftsfuehrer: Heiko Schubert
>
> Es gelten unsere allgemeinen Geschaeftsbedingungen.
> http://www.lamp-solutions.de/agbs/
>
> Telefon      : 0911 / 376 516 0
> Fax          : 0911 / 376 516 11
> E-Mail       : support at lamp-solutions.de
> Web          : www.lamp-solutions.de
> Facebook     : http://www.facebook.com/LAMPsolutions
> Twitter      : http://twitter.com/#!/lampsolutions
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120904/94f8e632/attachment.html>

From smsmail at roadrunner.com  Tue Sep  4 19:08:09 2012
From: smsmail at roadrunner.com (smsmail at roadrunner.com)
Date: Tue, 4 Sep 2012 15:08:09 -0400
Subject: Should I Use Varnish for RESTful API
Message-ID: <20120904190809.PJD75.84127.root@hrndva-web11-z02>

I have a server running Ubuntu, Apache2, Drupal 7, Varnish.  I have  a PHP RESTful API that I run on this computer that takes search requests from the API request and then returns the data from the drupal database in a JSON format and shortly will be returning XML and HTML if requested.  Right now I have varnish setup to pass on the API requests, should I let varnish cache the data?  Currently the search terms are pretty small and the returned data is in the hundreds of records, hopefully soon the number of records in the database will jump alot and the search terms and returned data will jump to large number of records.



From tobias.eichelbroenner at lamp-solutions.de  Tue Sep  4 19:11:22 2012
From: tobias.eichelbroenner at lamp-solutions.de (=?ISO-8859-1?Q?Tobias_Eichelbr=F6nner?=)
Date: Tue, 04 Sep 2012 21:11:22 +0200
Subject: First hit always cached.
In-Reply-To: <CAArAGK=FPmJKWMC5E_ZDTgPR5fT+JnSnHPRA8pOhm9+pirJDHg@mail.gmail.com>
References: <F016B57F-05FF-4274-B3C1-37BD604929F2@hostinglab.nl>
 <504627E7.7090708@lamp-solutions.de>
 <CAArAGKmHWaNiqY_KyeJvqjq3V15GTYjRAH9C4LXTjXXczBQPCQ@mail.gmail.com>
 <504639EA.105@lamp-solutions.de>
 <CAArAGK=FPmJKWMC5E_ZDTgPR5fT+JnSnHPRA8pOhm9+pirJDHg@mail.gmail.com>
Message-ID: <5046525A.4040502@lamp-solutions.de>

> I see. How about varnishreplay as a workaround for this?

this would just be an easy way make your first request in fill the
cache. As far as i see, you would have to switch some trigger between
the first real request and the second one.

Sincerely


-- 
LAMP solutions GmbH
Gostenhofer Hauptstrasse 35
90443 Nuernberg

Amtsgericht Nuernberg: HRB 22366
Geschaeftsfuehrer: Heiko Schubert

Es gelten unsere allgemeinen Geschaeftsbedingungen.
http://www.lamp-solutions.de/agbs/

Telefon      : 0911 / 376 516 0
Fax          : 0911 / 376 516 11
E-Mail       : support at lamp-solutions.de
Web          : www.lamp-solutions.de
Facebook     : http://www.facebook.com/LAMPsolutions
Twitter      : http://twitter.com/#!/lampsolutions



From perbu at varnish-software.com  Wed Sep  5 07:12:41 2012
From: perbu at varnish-software.com (Per Buer)
Date: Wed, 5 Sep 2012 09:12:41 +0200
Subject: Should I Use Varnish for RESTful API
In-Reply-To: <20120904190809.PJD75.84127.root@hrndva-web11-z02>
References: <20120904190809.PJD75.84127.root@hrndva-web11-z02>
Message-ID: <CAOXZevBU406HGWjEckTswFJPaNLvtut2bW2qC086PcXQJYf34g@mail.gmail.com>

Hi Mark,

On Tue, Sep 4, 2012 at 9:08 PM, <smsmail at roadrunner.com> wrote:

> I have a server running Ubuntu, Apache2, Drupal 7, Varnish.  I have  a PHP
> RESTful API that I run on this computer that takes search requests from the
> API request and then returns the data from the drupal database in a JSON
> format and shortly will be returning XML and HTML if requested.  Right now
> I have varnish setup to pass on the API requests, should I let varnish
> cache the data?


Yes.

You need to make sure that they are GET requests, though.

-- 
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120905/ce891c5c/attachment.html>

From cmitasch at thomas-krenn.com  Wed Sep  5 09:04:17 2012
From: cmitasch at thomas-krenn.com (Christoph Mitasch)
Date: Wed, 5 Sep 2012 11:04:17 +0200 (CEST)
Subject: Using pipe for large files
In-Reply-To: <781261408.294028.1346835580689.JavaMail.root@thomas-krenn.com>
Message-ID: <1037409978.294241.1346835857606.JavaMail.root@thomas-krenn.com>

Hello,

some of our users experienced problems when downloading larger files (e.g an iso with 700MB). This happened especially when the clients had a slow Internet connection.

I figured out that this is related to the send_timeout parameter that has changed from 600 to 60 seconds with 3.0.

Since I don't want to have a time limit for slow clients I solved it with the following code:

added to vcl_recv:
  /* Bypass cache for large files.  The x-pipe header is
     set in vcl_fetch when a too large file is detected. */
  if (req.http.x-pipe && req.restarts > 0) {
    remove req.http.x-pipe;
    return (pipe);
  }

added to vcl_fetch:
  # don't cache files larger than 10MB
  /* Don't try to cache too large files.  It appears
     Varnish just crashes if we don't filter them. */
  if (beresp.http.Content-Length ~ "[0-9]{8,}" ) {
    set req.http.x-pipe = "1";
    return (restart);
  }

Is there any disadvantage by using a pipe instead of a pass?

Christoph



From gonzalo.paniagua at acquia.com  Wed Sep  5 15:11:50 2012
From: gonzalo.paniagua at acquia.com (Gonzalo Paniagua)
Date: Wed, 5 Sep 2012 11:11:50 -0400
Subject: Using pipe for large files
In-Reply-To: <1037409978.294241.1346835857606.JavaMail.root@thomas-krenn.com>
References: <781261408.294028.1346835580689.JavaMail.root@thomas-krenn.com>
 <1037409978.294241.1346835857606.JavaMail.root@thomas-krenn.com>
Message-ID: <CAOvFWRX3iabVBZAae+C_Yw7M2TStUhJn6BuHxyL6qo1jkwqP1w@mail.gmail.com>

On Wed, Sep 5, 2012 at 5:04 AM, Christoph Mitasch
<cmitasch at thomas-krenn.com> wrote:
[...]
>
> Is there any disadvantage by using a pipe instead of a pass?

Since you are not caching the response, it is the right thing to do.
(btw, see http://phk.freebsd.dk/misc/varnish.gif)
Make sure you close the backend connection after the first request
(https://www.varnish-cache.org/trac/wiki/VCLExamplePipe)

-Gonzalo



From zeeshan at infoshield.info  Wed Sep  5 16:04:39 2012
From: zeeshan at infoshield.info (Zeeshan Ali Shah)
Date: Wed, 5 Sep 2012 21:04:39 +0500
Subject: Using pipe for large files
In-Reply-To: <CAOvFWRX3iabVBZAae+C_Yw7M2TStUhJn6BuHxyL6qo1jkwqP1w@mail.gmail.com>
References: <781261408.294028.1346835580689.JavaMail.root@thomas-krenn.com>
 <1037409978.294241.1346835857606.JavaMail.root@thomas-krenn.com>
 <CAOvFWRX3iabVBZAae+C_Yw7M2TStUhJn6BuHxyL6qo1jkwqP1w@mail.gmail.com>
Message-ID: <CAJSm=nvxo2AJV9hMzeVWTEnOv0ahutQK_q=BStn3-WQyXik=aw@mail.gmail.com>

But what if Some one want to cache even the larger files ..  for example in
Geo CDN cases for Videos.

I face the same issue when cache images (<1MB) it works when MP3 (>10MB) it
time out . No i  am not using pipe since i want to even cache media files


any hint please ?

Zeeshan

On Wed, Sep 5, 2012 at 8:11 PM, Gonzalo Paniagua <
gonzalo.paniagua at acquia.com> wrote:

> On Wed, Sep 5, 2012 at 5:04 AM, Christoph Mitasch
> <cmitasch at thomas-krenn.com> wrote:
> [...]
> >
> > Is there any disadvantage by using a pipe instead of a pass?
>
> Since you are not caching the response, it is the right thing to do.
> (btw, see http://phk.freebsd.dk/misc/varnish.gif)
> Make sure you close the backend connection after the first request
> (https://www.varnish-cache.org/trac/wiki/VCLExamplePipe)
>
> -Gonzalo
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120905/344032b3/attachment.html>

From member at linkedin.com  Wed Sep  5 17:52:42 2012
From: member at linkedin.com (Javier Casares via LinkedIn)
Date: Wed, 5 Sep 2012 17:52:42 +0000 (UTC)
Subject: Join my network on LinkedIn
Message-ID: <1068707861.9545649.1346867562313.JavaMail.app@ela4-app2317.prod>

LinkedIn
------------




    Javier Casares requested to add you as a connection on LinkedIn:
  

------------------------------------------

I'd like to add you to my professional network on LinkedIn.

Accept invitation from Javier Casares
http://www.linkedin.com/e/-9jv3q4-h6qq669t-3z/uUwiAqzf418olGWw58mjSTvkm1HgTG3IrG82TCkxhF79SC/blk/I315074386_13/0UcDpKqiRzolZKqiRybmRSrCBvrmRLoORIrmkZt5YCpnlOt3RApnhMpmdzgmhxrSNBszYPclYSe3cQdP0Rcjd9bTtLknlTlTtCbPwPdzoNc3sMczwLrCBxbOYWrSlI/eml-comm_invm-b-in_ac-inv28/?hs=false&tok=2I8HURGVKMxBo1

View profile of Javier Casares
http://www.linkedin.com/e/-9jv3q4-h6qq669t-3z/rso/47733828/jgTm/name/95734402_I315074386_13/?hs=false&tok=1jygXd18aMxBo1
------------------------------------------
You are receiving Invitation emails.


This email was intended for eris rachmatturisman.
Learn why this is included: http://www.linkedin.com/e/-9jv3q4-h6qq669t-3z/plh/http%3A%2F%2Fhelp%2Elinkedin%2Ecom%2Fapp%2Fanswers%2Fdetail%2Fa_id%2F4788/-GXI/?hs=false&tok=1JikOtf4WMxBo1

(c) 2012, LinkedIn Corporation. 2029 Stierlin Ct, Mountain View, CA 94043, USA.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120905/50053155/attachment.html>

From gonzalo.paniagua at acquia.com  Wed Sep  5 17:57:26 2012
From: gonzalo.paniagua at acquia.com (Gonzalo Paniagua)
Date: Wed, 5 Sep 2012 13:57:26 -0400
Subject: Using pipe for large files
In-Reply-To: <CAJSm=nvxo2AJV9hMzeVWTEnOv0ahutQK_q=BStn3-WQyXik=aw@mail.gmail.com>
References: <781261408.294028.1346835580689.JavaMail.root@thomas-krenn.com>
 <1037409978.294241.1346835857606.JavaMail.root@thomas-krenn.com>
 <CAOvFWRX3iabVBZAae+C_Yw7M2TStUhJn6BuHxyL6qo1jkwqP1w@mail.gmail.com>
 <CAJSm=nvxo2AJV9hMzeVWTEnOv0ahutQK_q=BStn3-WQyXik=aw@mail.gmail.com>
Message-ID: <CAOvFWRUhBrwOe3BNeL0AQNz4cGHq_7DDzbKf3UR99L92TdQTdw@mail.gmail.com>

On Wed, Sep 5, 2012 at 12:04 PM, Zeeshan Ali Shah
<zeeshan at infoshield.info> wrote:
> But what if Some one want to cache even the larger files ..  for example in
> Geo CDN cases for Videos.
>
> I face the same issue when cache images (<1MB) it works when MP3 (>10MB) it
> time out . No i  am not using pipe since i want to even cache media files

I think that the answer to that question is
https://www.varnish-cache.org/lists/pipermail/varnish-misc/2012-September/022454.html
or https://www.varnish-cache.org/lists/pipermail/varnish-misc/2012-September/022456.html

-Gonzalo



From zeeshan at infoshield.info  Wed Sep  5 18:00:44 2012
From: zeeshan at infoshield.info (Zeeshan Ali Shah)
Date: Wed, 5 Sep 2012 23:00:44 +0500
Subject: Using pipe for large files
In-Reply-To: <CAOvFWRUhBrwOe3BNeL0AQNz4cGHq_7DDzbKf3UR99L92TdQTdw@mail.gmail.com>
References: <781261408.294028.1346835580689.JavaMail.root@thomas-krenn.com>
 <1037409978.294241.1346835857606.JavaMail.root@thomas-krenn.com>
 <CAOvFWRX3iabVBZAae+C_Yw7M2TStUhJn6BuHxyL6qo1jkwqP1w@mail.gmail.com>
 <CAJSm=nvxo2AJV9hMzeVWTEnOv0ahutQK_q=BStn3-WQyXik=aw@mail.gmail.com>
 <CAOvFWRUhBrwOe3BNeL0AQNz4cGHq_7DDzbKf3UR99L92TdQTdw@mail.gmail.com>
Message-ID: <CAJSm=ntHweznR_N+gV-tNiR0=S3=2ybFmAqHztawPoN7pstijw@mail.gmail.com>

Right i also see that the only way forward..

Zeeshan

On Wed, Sep 5, 2012 at 10:57 PM, Gonzalo Paniagua <
gonzalo.paniagua at acquia.com> wrote:

> On Wed, Sep 5, 2012 at 12:04 PM, Zeeshan Ali Shah
> <zeeshan at infoshield.info> wrote:
> > But what if Some one want to cache even the larger files ..  for example
> in
> > Geo CDN cases for Videos.
> >
> > I face the same issue when cache images (<1MB) it works when MP3 (>10MB)
> it
> > time out . No i  am not using pipe since i want to even cache media files
>
> I think that the answer to that question is
>
> https://www.varnish-cache.org/lists/pipermail/varnish-misc/2012-September/022454.html
> or
> https://www.varnish-cache.org/lists/pipermail/varnish-misc/2012-September/022456.html
>
> -Gonzalo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120905/dbf94460/attachment.html>

From perbu at varnish-software.com  Thu Sep  6 07:15:19 2012
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 6 Sep 2012 09:15:19 +0200
Subject: Using pipe for large files
In-Reply-To: <CAJSm=nvxo2AJV9hMzeVWTEnOv0ahutQK_q=BStn3-WQyXik=aw@mail.gmail.com>
References: <781261408.294028.1346835580689.JavaMail.root@thomas-krenn.com>
 <1037409978.294241.1346835857606.JavaMail.root@thomas-krenn.com>
 <CAOvFWRX3iabVBZAae+C_Yw7M2TStUhJn6BuHxyL6qo1jkwqP1w@mail.gmail.com>
 <CAJSm=nvxo2AJV9hMzeVWTEnOv0ahutQK_q=BStn3-WQyXik=aw@mail.gmail.com>
Message-ID: <CAOXZevAe52=HHe-QwCEPfaFhBbP-ooJTc2qU+exqvftnfANm-w@mail.gmail.com>

On Wed, Sep 5, 2012 at 6:04 PM, Zeeshan Ali Shah <zeeshan at infoshield.info>wrote:

> But what if Some one want to cache even the larger files ..  for example
> in Geo CDN cases for Videos.
>
> I face the same issue when cache images (<1MB) it works when MP3 (>10MB)
> it time out . No i  am not using pipe since i want to even cache media files
>

Use 3.0.3-plus and enable streaming.




-- 
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120906/1ea42fef/attachment.html>

From Henry.Qian at datasphere.com  Fri Sep  7 00:13:25 2012
From: Henry.Qian at datasphere.com (Henry Qian)
Date: Fri, 7 Sep 2012 00:13:25 +0000
Subject: req.hash_always_miss doesn't work sometimes
Message-ID: <D50FC5EC2F9E4F499F14588BC98521D2027E02F8@HQEXCH01.secondspace.local>

Environment is Varnish 3.03 + CentOS 6.2.

I use req.hash_always_miss=true in vcl_recv to refresh cached pages. However I found the req.hash_always_miss=true doesn?t always works. Sometimes Varnish serves newly refreshed cache content, sometimes Varnish servers old cache content, even through the varnish log reveals the new cache content has been looked up and correctly delivered.

Here is the VCL.

sub vcl_recv {
    set req.backend=hyperlocal;

    if (req.backend.healthy) {
      set req.grace = 30s;
    } else {
      set req.grace = 24h;
    }

    if ((req.http.user-agent == "refresh") || (req.url ~ "\?refreshcache")) {
          set req.hash_always_miss = true;
    }

    if (req.restarts == 0) {
      if (req.http.x-forwarded-for) {
          set req.http.X-Forwarded-For =
            req.http.X-Forwarded-For + ", " + client.ip;
      } else {
          set req.http.X-Forwarded-For = client.ip;
      }
    }
    if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }
    if (req.request != "GET" && req.request != "HEAD") {
        /* We only deal with GET and HEAD by default */
        return (pass);
}

    return (lookup);
}

# sub vcl_pipe {
#     # Note that only the first request to the backend will have
#     # X-Forwarded-For set.  If you use X-Forwarded-For and want to
#     # have it set for all requests, make sure to have:
#     # set bereq.http.connection = "close";
#     # here.  It is not set by default as it might break some broken web
#     # applications, like IIS with NTLM authentication.
#     return (pipe);
# }
#
# sub vcl_pass {
#     return (pass);
# }

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # Add something to differential mobile, tablet, desktop visit.
    if (req.http.X-UA-Family) {
        hash_data(req.http.X-UA-Family);
    }

    return (hash);
}
#sub vcl_hit {
#    return (deliver);
#}
# sub vcl_miss {
#     return (fetch);
# }

sub vcl_fetch {
    # Grace to allow varnish to serve content if backend is lagged
set beresp.grace = 24h;

if (req.url == "/") {
      set beresp.ttl = 720m;        // 12 hours
    } else {
      set beresp.ttl = 14400m;               // 10 days
    }

    if (req.http.Cookie ~ "(VARNISH|DRUPAL_UID=[^-])") {
      set beresp.http.cache-control = "no-cache";
      set beresp.http.X-Cacheable = "NO";
      set beresp.http.ETag = "ADMIN";
    } else {
      set beresp.http.X-Cacheable = "YES";
    }

    if (beresp.ttl <= 0s ||
        beresp.http.Set-Cookie ||
        beresp.http.Vary == "*") {
            /*
            * Mark as "Hit-For-Pass" for the next 2 minutes
            */
            set beresp.ttl = 120 s;
            return (hit_for_pass);
    }

    return (deliver);
}

sub vcl_deliver {
   set resp.http.age = "0";

   #add cache hit data
   if (obj.hits > 0) {
     #if hit add hit count
     set resp.http.X-Cache = "HIT";
     set resp.http.X-Cache-Hits = obj.hits;
   } else {
     set resp.http.X-Cache = "MISS";
   }

    return (deliver);
}


The below Varnishlog reveals the content of http://riverdale.11alive.com/ has been refreshed, triggered by a special request http://riverdale.11alive.com/?refreshcache. The log reveals the refresh is successful.

   60 SessionOpen  c 127.0.0.1 56116 :6081
   60 ReqStart     c 127.0.0.1 56116 1570899703
   60 RxRequest    c GET
   60 RxURL        c /?refreshcache
   60 RxProtocol   c HTTP/1.0
   60 RxHeader     c Host: riverdale.11alive.com
   60 RxHeader     c X-Real-IP: 10.112.0.221
   60 RxHeader     c X-Forwarded-For: 10.113.0.134, 10.112.0.221
   60 RxHeader     c X-Client-Verify: SUCCESS
   60 RxHeader     c X-UA-Family: desktop
   60 RxHeader     c Connection: close
   60 RxHeader     c User-Agent: Java/1.6.0_20
   60 RxHeader     c Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
   60 VCL_call     c recv lookup
   60 VCL_call     c hash
   60 Hash         c /
   60 Hash         c riverdale.11alive.com
   60 Hash         c desktop
   60 VCL_return   c hash
   60 VCL_call     c miss fetch
   60 Backend      c 61 hyperlocal hyperlocal
   60 TTL          c 1570899703 RFC 120 -1 -1 1346968996 0 1346968992 0 0
   60 VCL_call     c fetch
   60 TTL          c 1570899703 VCL 123 86400 -1 1346968993 -3
   60 TTL          c 1570899703 VCL 43203 86400 -1 1346968993 -3
   60 VCL_return   c deliver
   60 ObjProtocol  c HTTP/1.1
   60 ObjResponse  c OK
   60 ObjHeader    c Date: Thu, 06 Sep 2012 22:03:12 GMT
   60 ObjHeader    c Server: Apache/2.2.15 (CentOS)
   60 ObjHeader    c Vary: User-Agent,Accept-Encoding
   60 ObjHeader    c X-Powered-By: PHP/5.3.3
   60 ObjHeader    c X-Drupal-Cache: MISS
   60 ObjHeader    c Last-Modified: Thu, 06 Sep 2012 22:03:12 +0000
   60 ObjHeader    c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
   60 ObjHeader    c ETag: "1346968992"
   60 ObjHeader    c Content-Encoding: gzip
   60 ObjHeader    c Content-Type: text/html; charset=utf-8
   60 ObjHeader    c X-Cacheable: YES
   60 Gzip         c u F - 27026 155024 80 80 216139
   60 VCL_call     c deliver deliver
   60 TxProtocol   c HTTP/1.1
   60 TxStatus     c 200
   60 TxResponse   c OK
   60 TxHeader     c Server: Apache/2.2.15 (CentOS)
   60 TxHeader     c Vary: User-Agent,Accept-Encoding
   60 TxHeader     c X-Powered-By: PHP/5.3.3
   60 TxHeader     c X-Drupal-Cache: MISS
   60 TxHeader     c Last-Modified: Thu, 06 Sep 2012 22:03:12 +0000
   60 TxHeader     c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
   60 TxHeader     c ETag: "1346968992"
   60 TxHeader     c Content-Type: text/html; charset=utf-8
   60 TxHeader     c X-Cacheable: YES
   60 TxHeader     c Date: Thu, 06 Sep 2012 22:03:16 GMT
   60 TxHeader     c X-Varnish: 1570899703
   60 TxHeader     c Via: 1.1 varnish
   60 TxHeader     c Connection: close
   60 TxHeader     c age: 0
   60 TxHeader     c X-Cache: MISS
   60 Gzip         c U D - 27026 155024 80 80 216139
   60 Length       c 155024
   60 ReqEnd       c 1570899703 1346968992.932512045 1346968996.214715719 0.000031471 3.280443430 0.001760244?

?However later when I visited the http://riverdale.11alive.com/, I got old content with the following header information. From the header information below, you can find the content is last modified on Thu, 06 Sep 2012 21:31:16 +0000, instead of the Last-Modified: Thu, 06 Sep 2012 22:03:12 +0000 as indicated in the varnish log.

Response Headersview source
Age 0
Cache-Control no-cache, must-revalidate, post-check=0, pre-check=0
Connection close
Content-Encoding gzip
Content-Type text/html; charset=utf-8
Date Thu, 06 Sep 2012 22:10:57 GMT
Etag "1346967076"
Last-Modified Thu, 06 Sep 2012 21:31:16 +0000
Server nginx/1.0.14
Transfer-Encoding chunked
Vary Accept-Encoding, Accept-Encoding
Via 1.1 varnish
X-Cache HIT
X-Cache-Hits 4
X-Cacheable YES
X-Drupal-Cache MISS
X-Powered-By PHP/5.2.10
X-Varnish 1570906726 1570869110

Request Headersview source
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-us,en;q=0.5
Cache-Control no-cache
Connection keep-alive
Cookie s_cc=true; s_sq=%5B%5BB%5D%5D; hl-email-avoid=true; __utma=196336364.508874118.1346959523.1346959523.1346969008.2; __utmc=196336364; __utmz=196336364.1346959523.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SiteLifeHost=l3vm105l3pluckcom; anonId=0227849c-521b-4054-a1cc-864ad8a54073; __utmb=196336364.34.0.1346969440935; BC_BANDWIDTH=1346969016992X1808
Host riverdale.11alive.com
Pragma no-cache
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0

Did I miss anything or Varnish req.hash_always_miss doesn't work as expected sometimes and thus is a bug?

Here I also post a good one, which is actually the request just before the bad one. You can tell it by checking the ObjHeader ETag. The good one is ?1346968991?, while the bad one is ?1346968992?.  Varnish as expected miss fetched the content from backend.

   79 SessionOpen  c 127.0.0.1 55822 :6081

   79 ReqStart     c 127.0.0.1 55822 1570899684

   79 RxRequest    c GET

   79 RxURL        c /?refreshcache

   79 RxProtocol   c HTTP/1.0

   79 RxHeader     c Host: northslocounty.kcoy.com

   79 RxHeader     c X-Real-IP: 10.112.0.221

   79 RxHeader     c X-Forwarded-For: 10.113.0.134, 10.112.0.221

   79 RxHeader     c X-Client-Verify: SUCCESS

   79 RxHeader     c X-UA-Family: desktop

   79 RxHeader     c Connection: close

   79 RxHeader     c User-Agent: Java/1.6.0_20

   79 RxHeader     c Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

   79 VCL_call     c recv lookup

   79 VCL_call     c hash

   79 Hash         c /

   79 Hash         c northslocounty.kcoy.com

   79 Hash         c desktop

   79 VCL_return   c hash

   79 VCL_call     c miss fetch

   79 Backend      c 80 hyperlocal hyperlocal

   79 TTL          c 1570899684 RFC 120 -1 -1 1346968993 0 1346968991 0 0

   79 VCL_call     c fetch

   79 TTL          c 1570899684 VCL 122 86400 -1 1346968991 -2

   79 TTL          c 1570899684 VCL 43202 86400 -1 1346968991 -2

   79 VCL_return   c deliver

   79 ObjProtocol  c HTTP/1.1

   79 ObjResponse  c OK

   79 ObjHeader    c Date: Thu, 06 Sep 2012 22:03:11 GMT

   79 ObjHeader    c Server: Apache/2.2.3 (CentOS)

   79 ObjHeader    c X-Powered-By: PHP/5.2.10

   79 ObjHeader    c X-Drupal-Cache: MISS

   79 ObjHeader    c Last-Modified: Thu, 06 Sep 2012 22:03:11 +0000

   79 ObjHeader    c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0

   79 ObjHeader    c ETag: "1346968991"

   79 ObjHeader    c Vary: Accept-Encoding

   79 ObjHeader    c Content-Encoding: gzip

   79 ObjHeader    c Content-Type: text/html; charset=utf-8

   79 ObjHeader    c X-Cacheable: YES

   79 Gzip         c u F - 19388 110022 80 80 155035

   79 VCL_call     c deliver deliver

   79 TxProtocol   c HTTP/1.1

   79 TxStatus     c 200

   79 TxResponse   c OK

   79 TxHeader     c Server: Apache/2.2.3 (CentOS)

   79 TxHeader     c X-Powered-By: PHP/5.2.10

   79 TxHeader     c X-Drupal-Cache: MISS

   79 TxHeader     c Last-Modified: Thu, 06 Sep 2012 22:03:11 +0000

   79 TxHeader     c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0

   79 TxHeader     c ETag: "1346968991"

   79 TxHeader     c Vary: Accept-Encoding

   79 TxHeader     c Content-Type: text/html; charset=utf-8

   79 TxHeader     c X-Cacheable: YES

   79 TxHeader     c Date: Thu, 06 Sep 2012 22:03:12 GMT

   79 TxHeader     c X-Varnish: 1570899684

   79 TxHeader     c Via: 1.1 varnish

   79 TxHeader     c Connection: close

   79 TxHeader     c age: 0

   79 TxHeader     c X-Cache: MISS

   79 Gzip         c U D - 19388 110022 80 80 155035

   79 Length       c 110022

   79 ReqEnd       c 1570899684 1346968991.255424976 1346968992.866729498 0.000033379 1.610149622 0.001154900



And then later when I visit the site, http://northslocounty.kcoy.com, the latest cache content was served. Please see header information below.

Response Headersview source

Age     0

Cache-Control no-cache, must-revalidate, post-check=0, pre-check=0

Connection      close

Content-Encoding      gzip

Content-Type  text/html; charset=utf-8

Date    Thu, 06 Sep 2012 22:05:42 GMT

Etag     "1346968991"

Last-Modified Thu, 06 Sep 2012 22:03:11 +0000

Server  nginx/1.0.14

Transfer-Encoding      chunked

Vary    Accept-Encoding, Accept-Encoding

Via      1.1 varnish

X-Cache          HIT

X-Cache-Hits  2

X-Cacheable   YES

X-Drupal-Cache          MISS

X-Powered-By            PHP/5.2.10

X-Varnish       1570901954 1570899684



Request Headersview source

Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Encoding        gzip, deflate

Accept-Language        en-us,en;q=0.5

Connection      keep-alive

Cookie hl-email-avoid=true

Host    northslocounty.kcoy.com

User-Agent     Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0
Thanks,
-Henry


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120907/07adc121/attachment.html>

From Henry.Qian at datasphere.com  Fri Sep  7 00:59:05 2012
From: Henry.Qian at datasphere.com (Henry Qian)
Date: Fri, 7 Sep 2012 00:59:05 +0000
Subject: req.hash_always_miss doesn't work sometimes
In-Reply-To: <D50FC5EC2F9E4F499F14588BC98521D2027E02F8@HQEXCH01.secondspace.local>
References: <D50FC5EC2F9E4F499F14588BC98521D2027E02F8@HQEXCH01.secondspace.local>
Message-ID: <D50FC5EC2F9E4F499F14588BC98521D2027E0337@HQEXCH01.secondspace.local>

One additional information is that if I use browser to trigger the refresh http://riverdale.11alive.com/?refreshcache, and later use browser to visit http://riverdale.11alive.com, the newly refreshed content is always served.

The randomness of serving refreshed content occurs when I use a java program to programmatically trigger the refresh, and later use browser to visit the site.  In vcl_hash you can find only req.url, req.http.host, server.ip, X-UA-Family are used for hash. I can guarantee these 4 attributes are exactly same when I trigger the refresh with browser and java program. So the hashed object should be same. Could Varnish 3.x change something and use some additional attributes not defined in vcl_hash to identify hash objects?

I tested on Varnish 3.02 + CentOS 6.2 with same VCL, same problem still exists.

While under old Varnish version 2.0.6, hash_always_miss was not available, so I used the following VCL to refresh the cache with the same java program, it always succeed with no issues.
sub vcl_recv {
set req.grace = 5m; if ((req.http.user-agent == "refresh")||(req.url ~ ?\?refreshcache?)) {
set req.grace = 0s;
}
}
sub vcl_hit {
if (((req.http.user-agent == "refresh")||(req.url ~ ?\?refreshcache?)) && req.restarts == 0) {
set obj.ttl = 0s; restart;
}
}
From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Henry Qian
Sent: Thursday, September 06, 2012 5:13 PM
To: varnish-misc at varnish-cache.org
Subject: req.hash_always_miss doesn't work sometimes

Environment is Varnish 3.03 + CentOS 6.2.

I use req.hash_always_miss=true in vcl_recv to refresh cached pages. However I found the req.hash_always_miss=true doesn?t always works. Sometimes Varnish serves newly refreshed cache content, sometimes Varnish servers old cache content, even through the varnish log reveals the new cache content has been looked up and correctly delivered.

Here is the VCL.

sub vcl_recv {
    set req.backend=hyperlocal;

    if (req.backend.healthy) {
      set req.grace = 30s;
    } else {
      set req.grace = 24h;
    }

    if ((req.http.user-agent == "refresh") || (req.url ~ "\?refreshcache")) {
          set req.hash_always_miss = true;
    }

    if (req.restarts == 0) {
      if (req.http.x-forwarded-for) {
          set req.http.X-Forwarded-For =
            req.http.X-Forwarded-For + ", " + client.ip;
      } else {
          set req.http.X-Forwarded-For = client.ip;
      }
    }
    if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }
    if (req.request != "GET" && req.request != "HEAD") {
        /* We only deal with GET and HEAD by default */
        return (pass);
}

    return (lookup);
}

# sub vcl_pipe {
#     # Note that only the first request to the backend will have
#     # X-Forwarded-For set.  If you use X-Forwarded-For and want to
#     # have it set for all requests, make sure to have:
#     # set bereq.http.connection = "close";
#     # here.  It is not set by default as it might break some broken web
#     # applications, like IIS with NTLM authentication.
#     return (pipe);
# }
#
# sub vcl_pass {
#     return (pass);
# }

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # Add something to differential mobile, tablet, desktop visit.
    if (req.http.X-UA-Family) {
        hash_data(req.http.X-UA-Family);
    }

    return (hash);
}

#sub vcl_hit {
#    return (deliver);
#}

# sub vcl_miss {
#     return (fetch);
# }

sub vcl_fetch {
    # Grace to allow varnish to serve content if backend is lagged
set beresp.grace = 24h;

if (req.url == "/") {
      set beresp.ttl = 720m;        // 12 hours
    } else {
      set beresp.ttl = 14400m;               // 10 days
    }

    if (req.http.Cookie ~ "(VARNISH|DRUPAL_UID=[^-])") {
      set beresp.http.cache-control = "no-cache";
      set beresp.http.X-Cacheable = "NO";
      set beresp.http.ETag = "ADMIN";
    } else {
      set beresp.http.X-Cacheable = "YES";
    }

    if (beresp.ttl <= 0s ||
        beresp.http.Set-Cookie ||
        beresp.http.Vary == "*") {
            /*
            * Mark as "Hit-For-Pass" for the next 2 minutes
            */
            set beresp.ttl = 120 s;
            return (hit_for_pass);
    }

    return (deliver);
}

sub vcl_deliver {
   set resp.http.age = "0";

   #add cache hit data
   if (obj.hits > 0) {
     #if hit add hit count
     set resp.http.X-Cache = "HIT";
     set resp.http.X-Cache-Hits = obj.hits;
   } else {
     set resp.http.X-Cache = "MISS";
   }

    return (deliver);
}


The below Varnishlog reveals the content of http://riverdale.11alive.com/ has been refreshed, triggered by a special request http://riverdale.11alive.com/?refreshcache. The log reveals the refresh is successful.

   60 SessionOpen  c 127.0.0.1 56116 :6081
   60 ReqStart     c 127.0.0.1 56116 1570899703
   60 RxRequest    c GET
   60 RxURL        c /?refreshcache
   60 RxProtocol   c HTTP/1.0
   60 RxHeader     c Host: riverdale.11alive.com
   60 RxHeader     c X-Real-IP: 10.112.0.221
   60 RxHeader     c X-Forwarded-For: 10.113.0.134, 10.112.0.221
   60 RxHeader     c X-Client-Verify: SUCCESS
   60 RxHeader     c X-UA-Family: desktop
   60 RxHeader     c Connection: close
   60 RxHeader     c User-Agent: Java/1.6.0_20
   60 RxHeader     c Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
   60 VCL_call     c recv lookup
   60 VCL_call     c hash
   60 Hash         c /
   60 Hash         c riverdale.11alive.com
   60 Hash         c desktop
   60 VCL_return   c hash
   60 VCL_call     c miss fetch
   60 Backend      c 61 hyperlocal hyperlocal
   60 TTL          c 1570899703 RFC 120 -1 -1 1346968996 0 1346968992 0 0
   60 VCL_call     c fetch
   60 TTL          c 1570899703 VCL 123 86400 -1 1346968993 -3
   60 TTL          c 1570899703 VCL 43203 86400 -1 1346968993 -3
   60 VCL_return   c deliver
   60 ObjProtocol  c HTTP/1.1
   60 ObjResponse  c OK
   60 ObjHeader    c Date: Thu, 06 Sep 2012 22:03:12 GMT
   60 ObjHeader    c Server: Apache/2.2.15 (CentOS)
   60 ObjHeader    c Vary: User-Agent,Accept-Encoding
   60 ObjHeader    c X-Powered-By: PHP/5.3.3
   60 ObjHeader    c X-Drupal-Cache: MISS
   60 ObjHeader    c Last-Modified: Thu, 06 Sep 2012 22:03:12 +0000
   60 ObjHeader    c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
   60 ObjHeader    c ETag: "1346968992"
   60 ObjHeader    c Content-Encoding: gzip
   60 ObjHeader    c Content-Type: text/html; charset=utf-8
   60 ObjHeader    c X-Cacheable: YES
   60 Gzip         c u F - 27026 155024 80 80 216139
   60 VCL_call     c deliver deliver
   60 TxProtocol   c HTTP/1.1
   60 TxStatus     c 200
   60 TxResponse   c OK
   60 TxHeader     c Server: Apache/2.2.15 (CentOS)
   60 TxHeader     c Vary: User-Agent,Accept-Encoding
   60 TxHeader     c X-Powered-By: PHP/5.3.3
   60 TxHeader     c X-Drupal-Cache: MISS
   60 TxHeader     c Last-Modified: Thu, 06 Sep 2012 22:03:12 +0000
   60 TxHeader     c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
   60 TxHeader     c ETag: "1346968992"
   60 TxHeader     c Content-Type: text/html; charset=utf-8
   60 TxHeader     c X-Cacheable: YES
   60 TxHeader     c Date: Thu, 06 Sep 2012 22:03:16 GMT
   60 TxHeader     c X-Varnish: 1570899703
   60 TxHeader     c Via: 1.1 varnish
   60 TxHeader     c Connection: close
   60 TxHeader     c age: 0
   60 TxHeader     c X-Cache: MISS
   60 Gzip         c U D - 27026 155024 80 80 216139
   60 Length       c 155024
   60 ReqEnd       c 1570899703 1346968992.932512045 1346968996.214715719 0.000031471 3.280443430 0.001760244?

?However later when I visited the http://riverdale.11alive.com/, I got old content with the following header information. From the header information below, you can find the content is last modified on Thu, 06 Sep 2012 21:31:16 +0000, instead of the Last-Modified: Thu, 06 Sep 2012 22:03:12 +0000 as indicated in the varnish log.

Response Headersview source
Age 0
Cache-Control no-cache, must-revalidate, post-check=0, pre-check=0
Connection close
Content-Encoding gzip
Content-Type text/html; charset=utf-8
Date Thu, 06 Sep 2012 22:10:57 GMT
Etag "1346967076"
Last-Modified Thu, 06 Sep 2012 21:31:16 +0000
Server nginx/1.0.14
Transfer-Encoding chunked
Vary Accept-Encoding, Accept-Encoding
Via 1.1 varnish
X-Cache HIT
X-Cache-Hits 4
X-Cacheable YES
X-Drupal-Cache MISS
X-Powered-By PHP/5.2.10
X-Varnish 1570906726 1570869110

Request Headersview source
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-us,en;q=0.5
Cache-Control no-cache
Connection keep-alive
Cookie s_cc=true; s_sq=%5B%5BB%5D%5D; hl-email-avoid=true; __utma=196336364.508874118.1346959523.1346959523.1346969008.2; __utmc=196336364; __utmz=196336364.1346959523.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SiteLifeHost=l3vm105l3pluckcom; anonId=0227849c-521b-4054-a1cc-864ad8a54073; __utmb=196336364.34.0.1346969440935; BC_BANDWIDTH=1346969016992X1808
Host riverdale.11alive.com
Pragma no-cache
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0

Did I miss anything or Varnish req.hash_always_miss doesn't work as expected sometimes and thus is a bug?

Here I also post a good one, which is actually the request just before the bad one. You can tell it by checking the ObjHeader ETag. The good one is ?1346968991?, while the bad one is ?1346968992?.  Varnish as expected miss fetched the content from backend.

   79 SessionOpen  c 127.0.0.1 55822 :6081

   79 ReqStart     c 127.0.0.1 55822 1570899684

   79 RxRequest    c GET

   79 RxURL        c /?refreshcache

   79 RxProtocol   c HTTP/1.0

   79 RxHeader     c Host: northslocounty.kcoy.com

   79 RxHeader     c X-Real-IP: 10.112.0.221

   79 RxHeader     c X-Forwarded-For: 10.113.0.134, 10.112.0.221

   79 RxHeader     c X-Client-Verify: SUCCESS

   79 RxHeader     c X-UA-Family: desktop

   79 RxHeader     c Connection: close

   79 RxHeader     c User-Agent: Java/1.6.0_20

   79 RxHeader     c Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

   79 VCL_call     c recv lookup

   79 VCL_call     c hash

   79 Hash         c /

   79 Hash         c northslocounty.kcoy.com

   79 Hash         c desktop

   79 VCL_return   c hash

   79 VCL_call     c miss fetch

   79 Backend      c 80 hyperlocal hyperlocal

   79 TTL          c 1570899684 RFC 120 -1 -1 1346968993 0 1346968991 0 0

   79 VCL_call     c fetch

   79 TTL          c 1570899684 VCL 122 86400 -1 1346968991 -2

   79 TTL          c 1570899684 VCL 43202 86400 -1 1346968991 -2

   79 VCL_return   c deliver

   79 ObjProtocol  c HTTP/1.1

   79 ObjResponse  c OK

   79 ObjHeader    c Date: Thu, 06 Sep 2012 22:03:11 GMT

   79 ObjHeader    c Server: Apache/2.2.3 (CentOS)

   79 ObjHeader    c X-Powered-By: PHP/5.2.10

   79 ObjHeader    c X-Drupal-Cache: MISS

   79 ObjHeader    c Last-Modified: Thu, 06 Sep 2012 22:03:11 +0000

   79 ObjHeader    c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0

   79 ObjHeader    c ETag: "1346968991"

   79 ObjHeader    c Vary: Accept-Encoding

   79 ObjHeader    c Content-Encoding: gzip

   79 ObjHeader    c Content-Type: text/html; charset=utf-8

   79 ObjHeader    c X-Cacheable: YES

   79 Gzip         c u F - 19388 110022 80 80 155035

   79 VCL_call     c deliver deliver

   79 TxProtocol   c HTTP/1.1

   79 TxStatus     c 200

   79 TxResponse   c OK

   79 TxHeader     c Server: Apache/2.2.3 (CentOS)

   79 TxHeader     c X-Powered-By: PHP/5.2.10

   79 TxHeader     c X-Drupal-Cache: MISS

   79 TxHeader     c Last-Modified: Thu, 06 Sep 2012 22:03:11 +0000

   79 TxHeader     c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0

   79 TxHeader     c ETag: "1346968991"

   79 TxHeader     c Vary: Accept-Encoding

   79 TxHeader     c Content-Type: text/html; charset=utf-8

   79 TxHeader     c X-Cacheable: YES

   79 TxHeader     c Date: Thu, 06 Sep 2012 22:03:12 GMT

   79 TxHeader     c X-Varnish: 1570899684

   79 TxHeader     c Via: 1.1 varnish

   79 TxHeader     c Connection: close

   79 TxHeader     c age: 0

   79 TxHeader     c X-Cache: MISS

   79 Gzip         c U D - 19388 110022 80 80 155035

   79 Length       c 110022

   79 ReqEnd       c 1570899684 1346968991.255424976 1346968992.866729498 0.000033379 1.610149622 0.001154900



And then later when I visit the site, http://northslocounty.kcoy.com, the latest cache content was served. Please see header information below.

Response Headersview source

Age     0

Cache-Control no-cache, must-revalidate, post-check=0, pre-check=0

Connection      close

Content-Encoding      gzip

Content-Type  text/html; charset=utf-8

Date    Thu, 06 Sep 2012 22:05:42 GMT

Etag     "1346968991"

Last-Modified Thu, 06 Sep 2012 22:03:11 +0000

Server  nginx/1.0.14

Transfer-Encoding      chunked

Vary    Accept-Encoding, Accept-Encoding

Via      1.1 varnish

X-Cache          HIT

X-Cache-Hits  2

X-Cacheable   YES

X-Drupal-Cache          MISS

X-Powered-By            PHP/5.2.10

X-Varnish       1570901954 1570899684



Request Headersview source

Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Encoding        gzip, deflate

Accept-Language        en-us,en;q=0.5

Connection      keep-alive

Cookie hl-email-avoid=true

Host    northslocounty.kcoy.com

User-Agent     Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0
Thanks,
-Henry


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120907/fbfbbce2/attachment.html>

From Henry.Qian at datasphere.com  Fri Sep  7 17:09:41 2012
From: Henry.Qian at datasphere.com (Henry Qian)
Date: Fri, 7 Sep 2012 17:09:41 +0000
Subject: req.hash_always_miss doesn't work sometimes
In-Reply-To: <D50FC5EC2F9E4F499F14588BC98521D2027E0337@HQEXCH01.secondspace.local>
References: <D50FC5EC2F9E4F499F14588BC98521D2027E02F8@HQEXCH01.secondspace.local>
 <D50FC5EC2F9E4F499F14588BC98521D2027E0337@HQEXCH01.secondspace.local>
Message-ID: <D50FC5EC2F9E4F499F14588BC98521D2027E0851@HQEXCH01.secondspace.local>

Find the problem. It?s the Vary: User Agent added by the backend web application caused the break. After normalizing req.http.user-agent, the problem went away.

From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Henry Qian
Sent: Thursday, September 06, 2012 5:59 PM
To: varnish-misc at varnish-cache.org
Subject: RE: req.hash_always_miss doesn't work sometimes

One additional information is that if I use browser to trigger the refresh http://riverdale.11alive.com/?refreshcache, and later use browser to visit http://riverdale.11alive.com, the newly refreshed content is always served.

The randomness of serving refreshed content occurs when I use a java program to programmatically trigger the refresh, and later use browser to visit the site.  In vcl_hash you can find only req.url, req.http.host, server.ip, X-UA-Family are used for hash. I can guarantee these 4 attributes are exactly same when I trigger the refresh with browser and java program. So the hashed object should be same. Could Varnish 3.x change something and use some additional attributes not defined in vcl_hash to identify hash objects?

I tested on Varnish 3.02 + CentOS 6.2 with same VCL, same problem still exists.

While under old Varnish version 2.0.6, hash_always_miss was not available, so I used the following VCL to refresh the cache with the same java program, it always succeed with no issues.
sub vcl_recv {
set req.grace = 5m; if ((req.http.user-agent == "refresh")||(req.url ~ ?\?refreshcache?)) {
set req.grace = 0s;
}
}
sub vcl_hit {
if (((req.http.user-agent == "refresh")||(req.url ~ ?\?refreshcache?)) && req.restarts == 0) {
set obj.ttl = 0s; restart;
}
}
From: varnish-misc-bounces at varnish-cache.org<mailto:varnish-misc-bounces at varnish-cache.org> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Henry Qian
Sent: Thursday, September 06, 2012 5:13 PM
To: varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
Subject: req.hash_always_miss doesn't work sometimes

Environment is Varnish 3.03 + CentOS 6.2.

I use req.hash_always_miss=true in vcl_recv to refresh cached pages. However I found the req.hash_always_miss=true doesn?t always works. Sometimes Varnish serves newly refreshed cache content, sometimes Varnish servers old cache content, even through the varnish log reveals the new cache content has been looked up and correctly delivered.

Here is the VCL.

sub vcl_recv {
    set req.backend=hyperlocal;

    if (req.backend.healthy) {
      set req.grace = 30s;
    } else {
      set req.grace = 24h;
    }

    if ((req.http.user-agent == "refresh") || (req.url ~ "\?refreshcache")) {
          set req.hash_always_miss = true;
    }

    if (req.restarts == 0) {
      if (req.http.x-forwarded-for) {
          set req.http.X-Forwarded-For =
            req.http.X-Forwarded-For + ", " + client.ip;
      } else {
          set req.http.X-Forwarded-For = client.ip;
      }
    }
    if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }
    if (req.request != "GET" && req.request != "HEAD") {
        /* We only deal with GET and HEAD by default */
        return (pass);
}

    return (lookup);
}

# sub vcl_pipe {
#     # Note that only the first request to the backend will have
#     # X-Forwarded-For set.  If you use X-Forwarded-For and want to
#     # have it set for all requests, make sure to have:
#     # set bereq.http.connection = "close";
#     # here.  It is not set by default as it might break some broken web
#     # applications, like IIS with NTLM authentication.
#     return (pipe);
# }
#
# sub vcl_pass {
#     return (pass);
# }

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # Add something to differential mobile, tablet, desktop visit.
    if (req.http.X-UA-Family) {
        hash_data(req.http.X-UA-Family);
    }

    return (hash);
}

#sub vcl_hit {
#    return (deliver);
#}

# sub vcl_miss {
#     return (fetch);
# }

sub vcl_fetch {
    # Grace to allow varnish to serve content if backend is lagged
set beresp.grace = 24h;

if (req.url == "/") {
      set beresp.ttl = 720m;        // 12 hours
    } else {
      set beresp.ttl = 14400m;               // 10 days
    }

    if (req.http.Cookie ~ "(VARNISH|DRUPAL_UID=[^-])") {
      set beresp.http.cache-control = "no-cache";
      set beresp.http.X-Cacheable = "NO";
      set beresp.http.ETag = "ADMIN";
    } else {
      set beresp.http.X-Cacheable = "YES";
    }

    if (beresp.ttl <= 0s ||
        beresp.http.Set-Cookie ||
        beresp.http.Vary == "*") {
            /*
            * Mark as "Hit-For-Pass" for the next 2 minutes
            */
            set beresp.ttl = 120 s;
            return (hit_for_pass);
    }

    return (deliver);
}

sub vcl_deliver {
   set resp.http.age = "0";

   #add cache hit data
   if (obj.hits > 0) {
     #if hit add hit count
     set resp.http.X-Cache = "HIT";
     set resp.http.X-Cache-Hits = obj.hits;
   } else {
     set resp.http.X-Cache = "MISS";
   }

    return (deliver);
}


The below Varnishlog reveals the content of http://riverdale.11alive.com/ has been refreshed, triggered by a special request http://riverdale.11alive.com/?refreshcache. The log reveals the refresh is successful.

   60 SessionOpen  c 127.0.0.1 56116 :6081
   60 ReqStart     c 127.0.0.1 56116 1570899703
   60 RxRequest    c GET
   60 RxURL        c /?refreshcache
   60 RxProtocol   c HTTP/1.0
   60 RxHeader     c Host: riverdale.11alive.com
   60 RxHeader     c X-Real-IP: 10.112.0.221
   60 RxHeader     c X-Forwarded-For: 10.113.0.134, 10.112.0.221
   60 RxHeader     c X-Client-Verify: SUCCESS
   60 RxHeader     c X-UA-Family: desktop
   60 RxHeader     c Connection: close
   60 RxHeader     c User-Agent: Java/1.6.0_20
   60 RxHeader     c Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
   60 VCL_call     c recv lookup
   60 VCL_call     c hash
   60 Hash         c /
   60 Hash         c riverdale.11alive.com
   60 Hash         c desktop
   60 VCL_return   c hash
   60 VCL_call     c miss fetch
   60 Backend      c 61 hyperlocal hyperlocal
   60 TTL          c 1570899703 RFC 120 -1 -1 1346968996 0 1346968992 0 0
   60 VCL_call     c fetch
   60 TTL          c 1570899703 VCL 123 86400 -1 1346968993 -3
   60 TTL          c 1570899703 VCL 43203 86400 -1 1346968993 -3
   60 VCL_return   c deliver
   60 ObjProtocol  c HTTP/1.1
   60 ObjResponse  c OK
   60 ObjHeader    c Date: Thu, 06 Sep 2012 22:03:12 GMT
   60 ObjHeader    c Server: Apache/2.2.15 (CentOS)
   60 ObjHeader    c Vary: User-Agent,Accept-Encoding
   60 ObjHeader    c X-Powered-By: PHP/5.3.3
   60 ObjHeader    c X-Drupal-Cache: MISS
   60 ObjHeader    c Last-Modified: Thu, 06 Sep 2012 22:03:12 +0000
   60 ObjHeader    c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
   60 ObjHeader    c ETag: "1346968992"
   60 ObjHeader    c Content-Encoding: gzip
   60 ObjHeader    c Content-Type: text/html; charset=utf-8
   60 ObjHeader    c X-Cacheable: YES
   60 Gzip         c u F - 27026 155024 80 80 216139
   60 VCL_call     c deliver deliver
   60 TxProtocol   c HTTP/1.1
   60 TxStatus     c 200
   60 TxResponse   c OK
   60 TxHeader     c Server: Apache/2.2.15 (CentOS)
   60 TxHeader     c Vary: User-Agent,Accept-Encoding
   60 TxHeader     c X-Powered-By: PHP/5.3.3
   60 TxHeader     c X-Drupal-Cache: MISS
   60 TxHeader     c Last-Modified: Thu, 06 Sep 2012 22:03:12 +0000
   60 TxHeader     c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
   60 TxHeader     c ETag: "1346968992"
   60 TxHeader     c Content-Type: text/html; charset=utf-8
   60 TxHeader     c X-Cacheable: YES
   60 TxHeader     c Date: Thu, 06 Sep 2012 22:03:16 GMT
   60 TxHeader     c X-Varnish: 1570899703
   60 TxHeader     c Via: 1.1 varnish
   60 TxHeader     c Connection: close
   60 TxHeader     c age: 0
   60 TxHeader     c X-Cache: MISS
   60 Gzip         c U D - 27026 155024 80 80 216139
   60 Length       c 155024
   60 ReqEnd       c 1570899703 1346968992.932512045 1346968996.214715719 0.000031471 3.280443430 0.001760244?

?However later when I visited the http://riverdale.11alive.com/, I got old content with the following header information. From the header information below, you can find the content is last modified on Thu, 06 Sep 2012 21:31:16 +0000, instead of the Last-Modified: Thu, 06 Sep 2012 22:03:12 +0000 as indicated in the varnish log.

Response Headersview source
Age 0
Cache-Control no-cache, must-revalidate, post-check=0, pre-check=0
Connection close
Content-Encoding gzip
Content-Type text/html; charset=utf-8
Date Thu, 06 Sep 2012 22:10:57 GMT
Etag "1346967076"
Last-Modified Thu, 06 Sep 2012 21:31:16 +0000
Server nginx/1.0.14
Transfer-Encoding chunked
Vary Accept-Encoding, Accept-Encoding
Via 1.1 varnish
X-Cache HIT
X-Cache-Hits 4
X-Cacheable YES
X-Drupal-Cache MISS
X-Powered-By PHP/5.2.10
X-Varnish 1570906726 1570869110

Request Headersview source
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-us,en;q=0.5
Cache-Control no-cache
Connection keep-alive
Cookie s_cc=true; s_sq=%5B%5BB%5D%5D; hl-email-avoid=true; __utma=196336364.508874118.1346959523.1346959523.1346969008.2; __utmc=196336364; __utmz=196336364.1346959523.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SiteLifeHost=l3vm105l3pluckcom; anonId=0227849c-521b-4054-a1cc-864ad8a54073; __utmb=196336364.34.0.1346969440935; BC_BANDWIDTH=1346969016992X1808
Host riverdale.11alive.com
Pragma no-cache
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0

Did I miss anything or Varnish req.hash_always_miss doesn't work as expected sometimes and thus is a bug?

Here I also post a good one, which is actually the request just before the bad one. You can tell it by checking the ObjHeader ETag. The good one is ?1346968991?, while the bad one is ?1346968992?.  Varnish as expected miss fetched the content from backend.

   79 SessionOpen  c 127.0.0.1 55822 :6081

   79 ReqStart     c 127.0.0.1 55822 1570899684

   79 RxRequest    c GET

   79 RxURL        c /?refreshcache

   79 RxProtocol   c HTTP/1.0

   79 RxHeader     c Host: northslocounty.kcoy.com

   79 RxHeader     c X-Real-IP: 10.112.0.221

   79 RxHeader     c X-Forwarded-For: 10.113.0.134, 10.112.0.221

   79 RxHeader     c X-Client-Verify: SUCCESS

   79 RxHeader     c X-UA-Family: desktop

   79 RxHeader     c Connection: close

   79 RxHeader     c User-Agent: Java/1.6.0_20

   79 RxHeader     c Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

   79 VCL_call     c recv lookup

   79 VCL_call     c hash

   79 Hash         c /

   79 Hash         c northslocounty.kcoy.com

   79 Hash         c desktop

   79 VCL_return   c hash

   79 VCL_call     c miss fetch

   79 Backend      c 80 hyperlocal hyperlocal

   79 TTL          c 1570899684 RFC 120 -1 -1 1346968993 0 1346968991 0 0

   79 VCL_call     c fetch

   79 TTL          c 1570899684 VCL 122 86400 -1 1346968991 -2

   79 TTL          c 1570899684 VCL 43202 86400 -1 1346968991 -2

   79 VCL_return   c deliver

   79 ObjProtocol  c HTTP/1.1

   79 ObjResponse  c OK

   79 ObjHeader    c Date: Thu, 06 Sep 2012 22:03:11 GMT

   79 ObjHeader    c Server: Apache/2.2.3 (CentOS)

   79 ObjHeader    c X-Powered-By: PHP/5.2.10

   79 ObjHeader    c X-Drupal-Cache: MISS

   79 ObjHeader    c Last-Modified: Thu, 06 Sep 2012 22:03:11 +0000

   79 ObjHeader    c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0

   79 ObjHeader    c ETag: "1346968991"

   79 ObjHeader    c Vary: Accept-Encoding

   79 ObjHeader    c Content-Encoding: gzip

   79 ObjHeader    c Content-Type: text/html; charset=utf-8

   79 ObjHeader    c X-Cacheable: YES

   79 Gzip         c u F - 19388 110022 80 80 155035

   79 VCL_call     c deliver deliver

   79 TxProtocol   c HTTP/1.1

   79 TxStatus     c 200

   79 TxResponse   c OK

   79 TxHeader     c Server: Apache/2.2.3 (CentOS)

   79 TxHeader     c X-Powered-By: PHP/5.2.10

   79 TxHeader     c X-Drupal-Cache: MISS

   79 TxHeader     c Last-Modified: Thu, 06 Sep 2012 22:03:11 +0000

   79 TxHeader     c Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0

   79 TxHeader     c ETag: "1346968991"

   79 TxHeader     c Vary: Accept-Encoding

   79 TxHeader     c Content-Type: text/html; charset=utf-8

   79 TxHeader     c X-Cacheable: YES

   79 TxHeader     c Date: Thu, 06 Sep 2012 22:03:12 GMT

   79 TxHeader     c X-Varnish: 1570899684

   79 TxHeader     c Via: 1.1 varnish

   79 TxHeader     c Connection: close

   79 TxHeader     c age: 0

   79 TxHeader     c X-Cache: MISS

   79 Gzip         c U D - 19388 110022 80 80 155035

   79 Length       c 110022

   79 ReqEnd       c 1570899684 1346968991.255424976 1346968992.866729498 0.000033379 1.610149622 0.001154900



And then later when I visit the site, http://northslocounty.kcoy.com, the latest cache content was served. Please see header information below.

Response Headersview source

Age     0

Cache-Control no-cache, must-revalidate, post-check=0, pre-check=0

Connection      close

Content-Encoding      gzip

Content-Type  text/html; charset=utf-8

Date    Thu, 06 Sep 2012 22:05:42 GMT

Etag     "1346968991"

Last-Modified Thu, 06 Sep 2012 22:03:11 +0000

Server  nginx/1.0.14

Transfer-Encoding      chunked

Vary    Accept-Encoding, Accept-Encoding

Via      1.1 varnish

X-Cache          HIT

X-Cache-Hits  2

X-Cacheable   YES

X-Drupal-Cache          MISS

X-Powered-By            PHP/5.2.10

X-Varnish       1570901954 1570899684



Request Headersview source

Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Encoding        gzip, deflate

Accept-Language        en-us,en;q=0.5

Connection      keep-alive

Cookie hl-email-avoid=true

Host    northslocounty.kcoy.com

User-Agent     Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0
Thanks,
-Henry


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120907/78226e24/attachment.html>

From huozhe at gmail.com  Sat Sep  8 00:43:14 2012
From: huozhe at gmail.com (Zheng Liu)
Date: Fri, 7 Sep 2012 17:43:14 -0700
Subject: can not load more VCL with error "cannot allocate memory", but with
 plenty memory left
Message-ID: <CAOCtP1f-EvYQwAaeAnHV8iU_WS1RDigDAEWNyHXvpOgEWyE+rw@mail.gmail.com>

Hi,

I have seen people report exactly the same error, but there is no solution
to it. I hope some one here can shed some light on the issue. I saw a very
high SMA.s0.c_fail "allocator failures". I think it is related. Is there a
way to free some memory to just fit in the VCL config? I hate to restart
the instance, since filling it up (160G) takes 1 day.

thanks,
Zheng

vcl.load newconfig /etc/varnish/default.vcl
106
VCL compiled.dlopen(./vcl.lA3o8iw_.so): ./vcl.lA3o8iw_.so: cannot apply
additional memory protection after relocation: Cannot allocate memory


# /usr/sbin/varnishd -V
varnishd (varnish-3.0.3 revision 9e6a70f)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2011 Varnish Software AS


# free -g
             total       used       free     shared    buffers     cached
Mem:           189        176         12          0          0          0
-/+ buffers/cache:        175         13
Swap:            3          0          3


# uname -a
Linux hcache6.houzz.com 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux


# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1550517
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1550517
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


# varnishstat -1
client_conn          520291099 520291099.00 Client connections accepted
client_drop                 52        52.00 Connection dropped, no sess/wrk
client_req           520311099 520311099.00 Client requests received
cache_hit            479753215 479753215.00 Cache hits
cache_hitpass             5396      5396.00 Cache hits for pass
cache_miss            40571613  40571613.00 Cache misses
backend_conn          18647472  18647472.00 Backend conn. success
backend_unhealthy            0         0.00 Backend conn. not attempted
backend_busy                 0         0.00 Backend conn. too many
backend_fail             84912     84912.00 Backend conn. failures
backend_reuse         21937169  21937169.00 Backend conn. reuses
backend_toolate          35368     35368.00 Backend conn. was closed
backend_recycle       21973014  21973014.00 Backend conn. recycles
backend_retry             1529      1529.00 Backend conn. retry
fetch_head                   0         0.00 Fetch head
fetch_length          40567091  40567091.00 Fetch with Length
fetch_chunked                0         0.00 Fetch chunked
fetch_eof                    0         0.00 Fetch EOF
fetch_bad                    0         0.00 Fetch had bad headers
fetch_close                  0         0.00 Fetch wanted close
fetch_oldhttp                0         0.00 Fetch pre HTTP/1.1 closed
fetch_zero                   0         0.00 Fetch zero len
fetch_failed              1924      1924.00 Fetch failed
fetch_1xx                    0         0.00 Fetch no body (1xx)
fetch_204                    0         0.00 Fetch no body (204)
fetch_304                    0         0.00 Fetch no body (304)
n_sess_mem                1119          .   N struct sess_mem
n_sess                      18          .   N struct sess
n_object               4173835          .   N struct object
n_vampireobject              0          .   N unresurrected objects
n_objectcore           4174219          .   N struct objectcore
n_objecthead           4190726          .   N struct objecthead
n_waitinglist            12437          .   N struct waitinglist
n_vbc                        7          .   N struct vbc
n_wrk                      400          .   N worker threads
n_wrk_create               465       465.00 N worker threads created
n_wrk_failed               256       256.00 N worker threads not created
n_wrk_max                    0         0.00 N worker threads limited
n_wrk_lqueue                 0         0.00 work request queue length
n_wrk_queued             16153     16153.00 N queued work requests
n_wrk_drop                  53        53.00 N dropped work requests
n_backend                    2          .   N backends
n_expired               191407          .   N expired objects
n_lru_nuked           36119853          .   N LRU nuked objects
n_lru_moved          424530660          .   N LRU moved objects
losthdr                      0         0.00 HTTP header overflows
n_objsendfile                0         0.00 Objects sent with sendfile
n_objwrite           529243867 529243867.00 Objects sent with write
n_objoverflow                0         0.00 Objects overflowing workspace
s_sess               520314370 520314370.00 Total Sessions
s_req                520311099 520311099.00 Total Requests
s_pipe                    2677      2677.00 Total pipe
s_pass                   37582     37582.00 Total pass
s_fetch               40511177  40511177.00 Total fetch
s_hdrbytes        203065278649 203065278649.00 Total header bytes
s_bodybytes       23783551235305 23783551235305.00 Total body bytes
sess_closed          520303774 520303774.00 Session Closed
sess_pipeline                0         0.00 Session Pipeline
sess_readahead               0         0.00 Session Read Ahead
sess_linger              72892     72892.00 Session Linger
sess_herd                16001     16001.00 Session herd
shm_records        24046733247 24046733247.00 SHM records
shm_writes          2162622737 2162622737.00 SHM writes
shm_flushes                  0         0.00 SHM flushes due to overflow
shm_cont              13971844  13971844.00 SHM MTX contention
shm_cycles               10341     10341.00 SHM cycles through buffer
sms_nreq                 44028     44028.00 SMS allocator requests
sms_nobj                     0          .   SMS outstanding allocations
sms_nbytes                   0          .   SMS outstanding bytes
sms_balloc            18403704          .   SMS bytes allocated
sms_bfree             18403704          .   SMS bytes freed
backend_req           40582821  40582821.00 Backend requests made
n_vcl                        1         1.00 N vcl total
n_vcl_avail                  1         1.00 N vcl available
n_vcl_discard                0         0.00 N vcl discarded
n_ban                        1          .   N total active bans
n_ban_gone                   1          .   N total gone bans
n_ban_add                    1         1.00 N new bans added
n_ban_retire                 0         0.00 N old bans deleted
n_ban_obj_test               0         0.00 N objects tested
n_ban_re_test                0         0.00 N regexps tested against
n_ban_dups                   0         0.00 N duplicate bans removed
hcb_nolock           519442684 519442684.00 HCB Lookups without lock
hcb_lock              40524771  40524771.00 HCB Lookups with lock
hcb_insert            40524740  40524740.00 HCB Inserts
esi_errors                   0         0.00 ESI parse errors (unlock)
esi_warnings                 0         0.00 ESI parse warnings (unlock)
accept_fail                  0         0.00 Accept failures
client_drop_late             1         1.00 Connection dropped late
uptime                  755101    755101.00 Client uptime
dir_dns_lookups              0         0.00 DNS director lookups
dir_dns_failed               0         0.00 DNS director failed lookups
dir_dns_hit                  0         0.00 DNS director cached lookups hit
dir_dns_cache_full           0         0.00 DNS director full dnscache
vmods                        0          .   Loaded VMODs
n_gzip                       0         0.00 Gzip operations
n_gunzip                 77505     77505.00 Gunzip operations
LCK.sms.creat                1         1.00 Created locks
LCK.sms.destroy              0         0.00 Destroyed locks
LCK.sms.locks           132084    132084.00 Lock Operations
LCK.sms.colls                0         0.00 Collisions
LCK.smp.creat                0         0.00 Created locks
LCK.smp.destroy              0         0.00 Destroyed locks
LCK.smp.locks                0         0.00 Lock Operations
LCK.smp.colls                0         0.00 Collisions
LCK.sma.creat                2         2.00 Created locks
LCK.sma.destroy              0         0.00 Destroyed locks
LCK.sma.locks        200409429 200409429.00 Lock Operations
LCK.sma.colls                0         0.00 Collisions
LCK.smf.creat                0         0.00 Created locks
LCK.smf.destroy              0         0.00 Destroyed locks
LCK.smf.locks                0         0.00 Lock Operations
LCK.smf.colls                0         0.00 Collisions
LCK.hsl.creat                0         0.00 Created locks
LCK.hsl.destroy              0         0.00 Destroyed locks
LCK.hsl.locks                0         0.00 Lock Operations
LCK.hsl.colls                0         0.00 Collisions
LCK.hcb.creat                1         1.00 Created locks
LCK.hcb.destroy              0         0.00 Destroyed locks
LCK.hcb.locks         76879907  76879907.00 Lock Operations
LCK.hcb.colls                0         0.00 Collisions
LCK.hcl.creat                0         0.00 Created locks
LCK.hcl.destroy              0         0.00 Destroyed locks
LCK.hcl.locks                0         0.00 Lock Operations
LCK.hcl.colls                0         0.00 Collisions
LCK.vcl.creat                1         1.00 Created locks
LCK.vcl.destroy              0         0.00 Destroyed locks
LCK.vcl.locks            26598     26598.00 Lock Operations
LCK.vcl.colls                0         0.00 Collisions
LCK.stat.creat               1         1.00 Created locks
LCK.stat.destroy             0         0.00 Destroyed locks
LCK.stat.locks       520315541 520315541.00 Lock Operations
LCK.stat.colls               0         0.00 Collisions
LCK.sessmem.creat            1         1.00 Created locks
LCK.sessmem.destroy            0         0.00 Destroyed locks
LCK.sessmem.locks      521596676 521596676.00 Lock Operations
LCK.sessmem.colls              0         0.00 Collisions
LCK.wstat.creat                1         1.00 Created locks
LCK.wstat.destroy              0         0.00 Destroyed locks
LCK.wstat.locks          1611720   1611720.00 Lock Operations
LCK.wstat.colls                0         0.00 Collisions
LCK.herder.creat               1         1.00 Created locks
LCK.herder.destroy             0         0.00 Destroyed locks
LCK.herder.locks            2884      2884.00 Lock Operations
LCK.herder.colls               0         0.00 Collisions
LCK.wq.creat                   8         8.00 Created locks
LCK.wq.destroy                 0         0.00 Destroyed locks
LCK.wq.locks          1046638906 1046638906.00 Lock Operations
LCK.wq.colls                   0         0.00 Collisions
LCK.objhdr.creat        40523726  40523726.00 Created locks
LCK.objhdr.destroy      36334480  36334480.00 Destroyed locks
LCK.objhdr.locks      2153405517 2153405517.00 Lock Operations
LCK.objhdr.colls               0         0.00 Collisions
LCK.exp.creat                  1         1.00 Created locks
LCK.exp.destroy                0         0.00 Destroyed locks
LCK.exp.locks           77551306  77551306.00 Lock Operations
LCK.exp.colls                  0         0.00 Collisions
LCK.lru.creat                  2         2.00 Created locks
LCK.lru.destroy                0         0.00 Destroyed locks
LCK.lru.locks           76604938  76604938.00 Lock Operations
LCK.lru.colls                  0         0.00 Collisions
LCK.cli.creat                  1         1.00 Created locks
LCK.cli.destroy                0         0.00 Destroyed locks
LCK.cli.locks                 65        65.00 Lock Operations
LCK.cli.colls                  0         0.00 Collisions
LCK.ban.creat                  1         1.00 Created locks
LCK.ban.destroy                0         0.00 Destroyed locks
LCK.ban.locks           77555176  77555176.00 Lock Operations
LCK.ban.colls                  0         0.00 Collisions
LCK.vbp.creat                  1         1.00 Created locks
LCK.vbp.destroy                0         0.00 Destroyed locks
LCK.vbp.locks                  0         0.00 Lock Operations
LCK.vbp.colls                  0         0.00 Collisions
LCK.vbe.creat                  1         1.00 Created locks
LCK.vbe.destroy                0         0.00 Destroyed locks
LCK.vbe.locks           37466927  37466927.00 Lock Operations
LCK.vbe.colls                  0         0.00 Collisions
LCK.backend.creat              3         3.00 Created locks
LCK.backend.destroy            0         0.00 Destroyed locks
LCK.backend.locks      100145698 100145698.00 Lock Operations
LCK.backend.colls              0         0.00 Collisions
SMA.s0.c_req           123055964 123055964.00 Allocator requests
SMA.s0.c_fail       3551561563273 3551561563273.00 Allocator failures
SMA.s0.c_bytes      1615116324709 1615116324709.00 Bytes allocated
SMA.s0.c_freed      1447612608403 1447612608403.00 Bytes freed
SMA.s0.g_alloc           8654138          .   Allocations outstanding
SMA.s0.g_bytes      167503716306          .   Bytes outstanding
SMA.s0.g_space              8238          .   Bytes available
SMA.Transient.c_req        43934     43934.00 Allocator requests
SMA.Transient.c_fail           0         0.00 Allocator failures
SMA.Transient.c_bytes    646003217 646003217.00 Bytes allocated
SMA.Transient.c_freed    646003217 646003217.00 Bytes freed
SMA.Transient.g_alloc            0          .   Allocations outstanding
SMA.Transient.g_bytes            0          .   Bytes outstanding
SMA.Transient.g_space            0          .   Bytes available
VBE.node_master(192.168.0.21,,80).vcls            1          .   VCL
references
VBE.node_master(192.168.0.21,,80).happy           0          .   Happy
health probes
VBE.node_slave(192.168.0.39,,80).vcls             1          .   VCL
references
VBE.node_slave(192.168.0.39,,80).happy            0          .   Happy
health probes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120907/3b98cb48/attachment.html>

From sahmed1020 at gmail.com  Sun Sep  9 20:31:30 2012
From: sahmed1020 at gmail.com (S Ahmed)
Date: Sun, 9 Sep 2012 16:31:30 -0400
Subject: how fast can you expire a cache key?
In-Reply-To: <CAOXZevAbiJ3NZLFaSuEuo5g=3Uj8zq2R5R+sxQ5WeMLLb-misA@mail.gmail.com>
References: <CAG2rwuMVEbByvYEVpXhTDgh=JHF5mEGG2Bnydk66f2gDw-2H9w@mail.gmail.com>
 <CAOXZevAbiJ3NZLFaSuEuo5g=3Uj8zq2R5R+sxQ5WeMLLb-misA@mail.gmail.com>
Message-ID: <CAG2rwuNFzit1-CZH5XiDMNJ5NVWaurswS5Kkx7CqGd4hfwPL9g@mail.gmail.com>

So I'll have some sort of a process that will send 100's of commands to
varnish to purge a given url?

On Mon, Sep 3, 2012 at 4:50 PM, Per Buer <perbu at varnish-software.com> wrote:

> On Mon, Sep 3, 2012 at 10:44 PM, S Ahmed <sahmed1020 at gmail.com> wrote:
>
>> If I have hundreds of thousands to millions of pages in a web
>> application, and the pages are being updated at a rate of a few hundred per
>> second (and thus require the cache to be invalidated), is this a reasonable
>> level of cache expiration that I hit varnish with?
>
>
> Yes.
>
>
>> Would love to hear the high end of cache expiration that others are doing.
>>
>
> You're fine as long as you PURGE the objects. The bans probably won't work
> so well at those speeds.
>
>
>
> --
> Per Buer
> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
> *Varnish makes websites fly!*
> Whitepapers <http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
> Twitter <https://twitter.com/varnishsoftware>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120909/44eefd39/attachment.html>

From smsmail at roadrunner.com  Wed Sep 12 04:14:44 2012
From: smsmail at roadrunner.com (Mark Strickland)
Date: Wed, 12 Sep 2012 00:14:44 -0400
Subject: prevent varnish from caching Iframe
Message-ID: <001e01cd909d$244e5130$6ceaf390$@roadrunner.com>

How do I prevent varnish from caching a iframe?

 

 

Also my website is loading in about 2=3 seconds, is that normal with
varnish?  How can I verify that varnish is working correctly.  I am using
drupal 7.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120912/77152117/attachment.html>

From z999666999 at gmail.com  Thu Sep 13 03:14:57 2012
From: z999666999 at gmail.com (z999666999 at gmail.com)
Date: Thu, 13 Sep 2012 03:14:57 +0000
Subject: How to put varnishlog to one line
Message-ID: <1641038052-1347506097-cardhu_decombobulator_blackberry.rim.net-639759824-@b4.c16.bise6.blackberry>

Hi,
I am pretty new to varnish. Sorry for newbie question.
I use varnishlog -c -m 'TxStatus:50(2|3)' to get 502,503 errors. My question is:
If I want this save to a file but all info should be on one line, how to do that?

Thanks,
Z9
Sent from my BlackBerry device on the Rogers Wireless Network

From shib4u at gmail.com  Mon Sep 17 09:08:37 2012
From: shib4u at gmail.com (Shibashish)
Date: Mon, 17 Sep 2012 14:38:37 +0530
Subject: prevent varnish from caching Iframe
In-Reply-To: <001e01cd909d$244e5130$6ceaf390$@roadrunner.com>
References: <001e01cd909d$244e5130$6ceaf390$@roadrunner.com>
Message-ID: <CAOV_DHihjS-REm0TGYMVF7_acov0mV1P2=9kAWf3wZMf8Vwx7g@mail.gmail.com>

On Wed, Sep 12, 2012 at 9:44 AM, Mark Strickland <smsmail at roadrunner.com>wrote:

> How do I prevent varnish from caching a iframe?
>
For this, one has to understand your iframe, is it setting any cookie, or
any url pattern that helps in identifying it is an iframe... see this for
more details
http://serverfault.com/questions/42164/varnish-configuration-to-only-cache-for-non-logged-in-users


> ****
>
> ** **
>
> ** **
>
> Also my website is loading in about 2=3 seconds, is that normal with
> varnish?
>
How much time does it take without Varnish? Also check if the pages are
being served out of cache or not? See the link below on how to check that.



> How can I verify that varnish is working correctly.  I am using drupal 7.*
> ***
>

Verifying Varnish is working... see
https://www.varnish-cache.org/trac/wiki/VCLExampleHitMissHeader
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120917/3acd3817/attachment.html>

From perbu at varnish-software.com  Tue Sep 18 07:12:11 2012
From: perbu at varnish-software.com (Per Buer)
Date: Tue, 18 Sep 2012 09:12:11 +0200
Subject: How to put varnishlog to one line
In-Reply-To: <1641038052-1347506097-cardhu_decombobulator_blackberry.rim.net-639759824-@b4.c16.bise6.blackberry>
References: <1641038052-1347506097-cardhu_decombobulator_blackberry.rim.net-639759824-@b4.c16.bise6.blackberry>
Message-ID: <CAOXZevAw4am+y9Y9TbjB4Fd1g-c8iJT8DVS2EKaOwgJP-ko5yw@mail.gmail.com>

Hi,

On Thu, Sep 13, 2012 at 5:14 AM, <z999666999 at gmail.com> wrote:

> Hi,
> I am pretty new to varnish. Sorry for newbie question.
> I use varnishlog -c -m 'TxStatus:50(2|3)' to get 502,503 errors. My
> question is:
> If I want this save to a file but all info should be on one line, how to
> do that?
>

You would have to come up with some perl or similar code to do that.
varnishlog  won't do that out of the box. varnishncsa might, however, if
you can live with the limited data there.


-- 
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120918/80db5a67/attachment.html>

From james at ifixit.com  Wed Sep 19 00:36:04 2012
From: james at ifixit.com (James Pearson)
Date: Tue, 18 Sep 2012 17:36:04 -0700
Subject: Varnish sometimes delays requests for minutes
Message-ID: <1348013499-sup-8011@geror.local>

I've only recently begun using Varnish, so let me first and foremost apologize
for anything obvious that I've overlooked or misunderstood.

We've just started using Varnish on a small part of our infrastructure, and for
the most part, it's actually beautifully.  However, periodically a page request
will take around two minutes to return.

When a request acts like this, all other requests to the same URL act the same;
that is, they all hang and get back responses at the same time.  My guess is
that the hang is exactly 2 minutes, but I never make the first request wrapped
in a time(1) call.  During this time, other requests passing through the same
VCL logic and going to the same backend execute successfully - both those
served from the cache and those not.

The hung requests only happen on pages that have expired caches, but not every
request to that page while it is expired will trigger a hang.  That is, I have
a difficult time reproducing it, and kinda have to just watch requests in the
corner of my eye while going about other duties.

I am fairly certain that these hangs are not the fault of the backend, but not
completely sure - mark me down at 95% certainty.

Here are the parts of my configuration that seem potentially relevant:

========

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 30s;       # Default: 0.7s
    .first_byte_timeout = 3m;     # Default: 60s
    .between_bytes_timeout = 2m;  # Default: 60s
    # Always feed requests to the backend, even if Varnish thinks it's down.
    .saintmode_threshold = 0;
}

sub vcl_recv {
    # Set X-Forwarded-For so we get the client IP on the backend.
    if (req.restarts == 0) {
        if (req.http.X-Forwarded-For) {
            set req.http.X-Forwarded-For =
             req.http.X-Forwarded-For + ", " + client.ip;
        } else {
            set req.http.X-Forwarded-For = client.ip;
        }
    }
    
    # Ignore non-RFC2616 or CONNECT requests entirely.
    if (req.request != "GET"
     && req.request != "HEAD"
     && req.request != "PUT"
     && req.request != "POST"
     && req.request != "TRACE"
     && req.request != "OPTIONS"
     && req.request != "DELETE") {
        return (pipe);
    }
    # And don't cache anything not GET or HEAD, since POST and friends usually
    # modify state.
    if (req.request != "GET" && req.request != "HEAD") {
        return (pass);
    }
    
    # Turn on rules a little bit at a time.
    if (req.http.host ~ "^my.example.com$") {
        # [snip] force non-caching on a few pages

        set req.grace = 30s;
        
        return (lookup);
    }
    
    # Don't cache anything we haven't explicitly told Varnish to cache.
    return (pass);
}

sub vcl_fetch {
    if (req.http.host ~ "^my.example.com$") {
        # Cache permanent redirects for a while, since... they're permanent.
        if (beresp.status == 301) {
            set beresp.ttl = 30m;
        }
        
        # Sometimes 404s are the result of us not getting assets built quite in
        # time, or similar things.  So, let us recover quickly.
        if (beresp.status == 404) {
            set beresp.ttl = 0s;
            return (hit_for_pass);
        }

        # Hopefully refreshing an error page will cause it to not error.
        if (beresp.status == 500
         || beresp.status == 502
         || beresp.status == 503
         || beresp.status == 504) {
            set beresp.ttl = 0s;
            return (hit_for_pass);
        }
        
        # Allow objects to live a bit longer than their expiry time, in case we
        # want to serve up expired objects while waiting for the backend to
        # respond.  This should never be shorter than req.grace, as set in
        # vcl_recv.
        set beresp.grace = 1m;
        
        return (deliver);
    }
}

========

While I would appreciate any solutions, what I'm really looking for are
pointers on how to go about debugging this sort of problem - does varnishlog
provide information that's useful?  It spits out so much information that I'm
not quite sure what I need to filter out to see what's going on here.

Any help appreciated.  Thanks.
 - P




From nick.tailor at gmail.com  Thu Sep 20 07:31:07 2012
From: nick.tailor at gmail.com (nick tailor)
Date: Thu, 20 Sep 2012 00:31:07 -0700
Subject: syntax varnishreplay
Message-ID: <CA+8iBq8AuPGGACt2H29p3p34xhf37N5Q+DqUQVMPRbQsK-FBPw@mail.gmail.com>

Hello,

Could someone be kind of enough to provide me with the proper syntax of how
to use varnishreplay and what it's suppose to look like if its working
correctly. I can not seem to find it on google anywhere.

Cheers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120920/7fd4d103/attachment.html>

From nick.tailor at gmail.com  Thu Sep 20 07:59:06 2012
From: nick.tailor at gmail.com (nick tailor)
Date: Thu, 20 Sep 2012 00:59:06 -0700
Subject: prevent varnish from caching Iframe
In-Reply-To: <001e01cd909d$244e5130$6ceaf390$@roadrunner.com>
References: <001e01cd909d$244e5130$6ceaf390$@roadrunner.com>
Message-ID: <CA+8iBq_6ircn--de6WgCScD=OZCi4VQTe0jJiq3HBaODfw7Obg@mail.gmail.com>

If your site is taking 2-3 seconds that pretty slow. With Drupal 7 you need
to do a couple things, such as disable the cron that runs form anonymous
visits for maintenance and updates on each site and setup your own script
that runs the cron at a specified time. You will need to get the unique ID
from the inside the D7 GUI for each site to do this.

The reason why, is because varnish is checking the webserver, depending on
how you setup your probe could be inadvertently be triggering the cron. The
second piece to this how is you are handling your session cookie ID's. As
in what is apache handling when sessionsID come in login creds., and what
is the cache handling for anonymous visits. This is all defined in your
VCL.

Cheers

Nick Tailor

On Tue, Sep 11, 2012 at 9:14 PM, Mark Strickland <smsmail at roadrunner.com>wrote:

> How do I prevent varnish from caching a iframe?****
>
> ** **
>
> ** **
>
> Also my website is loading in about 2=3 seconds, is that normal with
> varnish?  How can I verify that varnish is working correctly.  I am using
> drupal 7.****
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120920/2660105f/attachment.html>

From nick.tailor at gmail.com  Thu Sep 20 08:04:13 2012
From: nick.tailor at gmail.com (nick tailor)
Date: Thu, 20 Sep 2012 01:04:13 -0700
Subject: prevent varnish from caching Iframe
In-Reply-To: <CA+8iBq_6ircn--de6WgCScD=OZCi4VQTe0jJiq3HBaODfw7Obg@mail.gmail.com>
References: <001e01cd909d$244e5130$6ceaf390$@roadrunner.com>
 <CA+8iBq_6ircn--de6WgCScD=OZCi4VQTe0jJiq3HBaODfw7Obg@mail.gmail.com>
Message-ID: <CA+8iBq-e2agXab1m1++OCdES=CLK0gJyN0U-iTF4Da+b7NWT9g@mail.gmail.com>

Oh forgot one thing. You can use a tool called Varnishhist, which will show
you a graphic representation on if your hitting the cache or not.

Cheers

Nick Tailor

On Thu, Sep 20, 2012 at 12:59 AM, nick tailor <nick.tailor at gmail.com> wrote:

> If your site is taking 2-3 seconds that pretty slow. With Drupal 7 you
> need to do a couple things, such as disable the cron that runs form
> anonymous visits for maintenance and updates on each site and setup your
> own script that runs the cron at a specified time. You will need to get the
> unique ID from the inside the D7 GUI for each site to do this.
>
> The reason why, is because varnish is checking the webserver, depending on
> how you setup your probe could be inadvertently be triggering the cron. The
> second piece to this how is you are handling your session cookie ID's. As
> in what is apache handling when sessionsID come in login creds., and what
> is the cache handling for anonymous visits. This is all defined in your
> VCL.
>
> Cheers
>
> Nick Tailor
>
> On Tue, Sep 11, 2012 at 9:14 PM, Mark Strickland <smsmail at roadrunner.com>wrote:
>
>> How do I prevent varnish from caching a iframe?****
>>
>> ** **
>>
>> ** **
>>
>> Also my website is loading in about 2=3 seconds, is that normal with
>> varnish?  How can I verify that varnish is working correctly.  I am using
>> drupal 7.****
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120920/7921ef16/attachment.html>

From Gauthier.Delacroix at coreye.fr  Thu Sep 20 09:40:35 2012
From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier)
Date: Thu, 20 Sep 2012 09:40:35 +0000
Subject: OOM caused by infinite loop in response
Message-ID: <5F530A9242E7F84F999DB40E0E268FBD0C0C4F@mercalli.lild01.pictime.fr>

Hi,

I have a varnish instance going out of memory because of an infinite loop in a chunked backend response.

The app is sending the same string over and over, resulting in more than 4GB to varnish in 10 minutes...until it crashes.

The dev team is fixing the bug in the backend app but I'd like to avoid varnish crashes for this kind of problem.

I have seen the "http_resp_size" option in the doc but it seems to only handle headers size and I haven't seen anything about response body limit.

Is there a way to drop connection when response body exceeds a maximum size ?

Thanks !

Gauthier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120920/bed46569/attachment.html>

From Gauthier.Delacroix at coreye.fr  Thu Sep 20 11:52:51 2012
From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier)
Date: Thu, 20 Sep 2012 11:52:51 +0000
Subject: OOM caused by infinite loop in response
In-Reply-To: <5FBBFB10-850E-43A4-BE9E-B9028A91E30F@gmail.com>
References: <5F530A9242E7F84F999DB40E0E268FBD0C0C4F@mercalli.lild01.pictime.fr>
 <5FBBFB10-850E-43A4-BE9E-B9028A91E30F@gmail.com>
Message-ID: <5F530A9242E7F84F999DB40E0E268FBD0C0F75@mercalli.lild01.pictime.fr>

It?s coming from the backend.

Varnish cannot cache the response since it never ends.

De : Nick Tailor [mailto:nick.tailor at gmail.com]
Envoy? : jeudi 20 septembre 2012 12:35
? : Delacroix, Gauthier
Objet : Re: OOM caused by infinite loop in response

If it's coming from the cache set the ttl higher for objects cached so it serves cached content that you know is working until u fix the issue

Sent from my iPhone

On Sep 20, 2012, at 2:40 AM, "Delacroix, Gauthier" <Gauthier.Delacroix at coreye.fr<mailto:Gauthier.Delacroix at coreye.fr>> wrote:
Hi,

I have a varnish instance going out of memory because of an infinite loop in a chunked backend response.

The app is sending the same string over and over, resulting in more than 4GB to varnish in 10 minutes?until it crashes.

The dev team is fixing the bug in the backend app but I?d like to avoid varnish crashes for this kind of problem.

I have seen the ?http_resp_size" option in the doc but it seems to only handle headers size and I haven?t seen anything about response body limit.

Is there a way to drop connection when response body exceeds a maximum size ?

Thanks !

Gauthier
_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

--
Ce message a ete verifie par MailScanner.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120920/4d08f148/attachment.html>

From perbu at varnish-software.com  Thu Sep 20 12:48:54 2012
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 20 Sep 2012 14:48:54 +0200
Subject: syntax varnishreplay
In-Reply-To: <CA+8iBq8AuPGGACt2H29p3p34xhf37N5Q+DqUQVMPRbQsK-FBPw@mail.gmail.com>
References: <CA+8iBq8AuPGGACt2H29p3p34xhf37N5Q+DqUQVMPRbQsK-FBPw@mail.gmail.com>
Message-ID: <CAOXZevCdqHkX8sy7uvVd6=WkZLqyc7w_Hq6mU42iEA=o65aPLw@mail.gmail.com>

Hi Nick,

On Thu, Sep 20, 2012 at 9:31 AM, nick tailor <nick.tailor at gmail.com> wrote:

> Hello,
>
> Could someone be kind of enough to provide me with the proper syntax of
> how to use varnishreplay and what it's suppose to look like if its working
> correctly. I can not seem to find it on google anywhere.
>


>From the number two result when I google "varnishreplay": "Warming up
Varnish with varnishreplay" I quote:

$  ssh www.varnish-cache.org varnishlog -w -|varnishreplay -a
localhost:6081 -r -
0x7f19b8c85700 thread 0x7f19b8caa700:14 started
0x7f19b8c85700 thread 0x7f19b8ca2700:15 started
0x7f19b8c85700 thread 0x7f19b8c9a700:17 started
0x7f19b8c85700 thread 0x7f19b8c92700:22 started
...

Cheers,

Per.


-- 
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120920/f61cc36c/attachment.html>

From bedis9 at gmail.com  Fri Sep 21 12:16:53 2012
From: bedis9 at gmail.com (Baptiste)
Date: Fri, 21 Sep 2012 14:16:53 +0200
Subject: OOM caused by infinite loop in response
In-Reply-To: <5F530A9242E7F84F999DB40E0E268FBD0C0F75@mercalli.lild01.pictime.fr>
References: <5F530A9242E7F84F999DB40E0E268FBD0C0C4F@mercalli.lild01.pictime.fr>
 <5FBBFB10-850E-43A4-BE9E-B9028A91E30F@gmail.com>
 <5F530A9242E7F84F999DB40E0E268FBD0C0F75@mercalli.lild01.pictime.fr>
Message-ID: <CAODHi7qOycb1JEZdT2maa=eBFzgnJ1qGg1HwOYT+Sm2F_Gez2w@mail.gmail.com>

Hi,

Can't you simply tells Varnish to return a void response when users
hit this URL?
Hopefully you know in advance which URLs are buggy....

cheers

On Thu, Sep 20, 2012 at 1:52 PM, Delacroix, Gauthier
<Gauthier.Delacroix at coreye.fr> wrote:
> It?s coming from the backend.
>
>
>
> Varnish cannot cache the response since it never ends.
>
>
>
> De : Nick Tailor [mailto:nick.tailor at gmail.com]
> Envoy? : jeudi 20 septembre 2012 12:35
> ? : Delacroix, Gauthier
> Objet : Re: OOM caused by infinite loop in response
>
>
>
> If it's coming from the cache set the ttl higher for objects cached so it
> serves cached content that you know is working until u fix the issue
>
> Sent from my iPhone
>
>
> On Sep 20, 2012, at 2:40 AM, "Delacroix, Gauthier"
> <Gauthier.Delacroix at coreye.fr> wrote:
>
> Hi,
>
>
>
> I have a varnish instance going out of memory because of an infinite loop in
> a chunked backend response.
>
>
>
> The app is sending the same string over and over, resulting in more than 4GB
> to varnish in 10 minutes?until it crashes.
>
>
>
> The dev team is fixing the bug in the backend app but I?d like to avoid
> varnish crashes for this kind of problem.
>
>
>
> I have seen the ?http_resp_size" option in the doc but it seems to only
> handle headers size and I haven?t seen anything about response body limit.
>
>
>
> Is there a way to drop connection when response body exceeds a maximum size
> ?
>
>
>
> Thanks !
>
>
>
> Gauthier
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
> --
> Ce message a ete verifie par MailScanner.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From Gauthier.Delacroix at coreye.fr  Fri Sep 21 12:48:56 2012
From: Gauthier.Delacroix at coreye.fr (Delacroix, Gauthier)
Date: Fri, 21 Sep 2012 12:48:56 +0000
Subject: OOM caused by infinite loop in response
In-Reply-To: <CAODHi7qOycb1JEZdT2maa=eBFzgnJ1qGg1HwOYT+Sm2F_Gez2w@mail.gmail.com>
References: <5F530A9242E7F84F999DB40E0E268FBD0C0C4F@mercalli.lild01.pictime.fr>
 <5FBBFB10-850E-43A4-BE9E-B9028A91E30F@gmail.com>
 <5F530A9242E7F84F999DB40E0E268FBD0C0F75@mercalli.lild01.pictime.fr>
 <CAODHi7qOycb1JEZdT2maa=eBFzgnJ1qGg1HwOYT+Sm2F_Gez2w@mail.gmail.com>
Message-ID: <5F530A9242E7F84F999DB40E0E268FBD0C180D@mercalli.lild01.pictime.fr>

This would be a curative fix.

This bug is now fixed, but as I don't always have control over backend applications, I'd like a preventive fix to avoid varnish to crash if it happens again in the future, since when it happens on a single URL, all the service is down.

(I actually would like to avoid varnish to crash in any case ;-) )

I think varnish stability should not depends on backend content, and it would be good to control maximum response body size...

I insist on the fact that backend sends chunked responses. It would be easy to limit size based on content-length header, but I don't have this header in chunked mode...

Gauthier

-----Message d'origine-----
De?: Baptiste [mailto:bedis9 at gmail.com] 
Envoy??: vendredi 21 septembre 2012 14:17
??: Delacroix, Gauthier
Cc?: Nick Tailor; varnish-misc at varnish-cache.org
Objet?: Re: OOM caused by infinite loop in response

Hi,

Can't you simply tells Varnish to return a void response when users hit this URL?
Hopefully you know in advance which URLs are buggy....

cheers

On Thu, Sep 20, 2012 at 1:52 PM, Delacroix, Gauthier <Gauthier.Delacroix at coreye.fr> wrote:
> It's coming from the backend.
>
>
>
> Varnish cannot cache the response since it never ends.
>
>
>
> De : Nick Tailor [mailto:nick.tailor at gmail.com] Envoy? : jeudi 20 
> septembre 2012 12:35 ? : Delacroix, Gauthier Objet : Re: OOM caused by 
> infinite loop in response
>
>
>
> If it's coming from the cache set the ttl higher for objects cached so 
> it serves cached content that you know is working until u fix the 
> issue
>
> Sent from my iPhone
>
>
> On Sep 20, 2012, at 2:40 AM, "Delacroix, Gauthier"
> <Gauthier.Delacroix at coreye.fr> wrote:
>
> Hi,
>
>
>
> I have a varnish instance going out of memory because of an infinite 
> loop in a chunked backend response.
>
>
>
> The app is sending the same string over and over, resulting in more 
> than 4GB to varnish in 10 minutes.until it crashes.
>
>
>
> The dev team is fixing the bug in the backend app but I'd like to 
> avoid varnish crashes for this kind of problem.
>
>
>
> I have seen the "http_resp_size" option in the doc but it seems to 
> only handle headers size and I haven't seen anything about response body limit.
>
>
>
> Is there a way to drop connection when response body exceeds a maximum 
> size ?
>
>
>
> Thanks !
>
>
>
> Gauthier
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
> --
> Ce message a ete verifie par MailScanner.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

--
Ce message a ete verifie par MailScanner.




From parakrama1282 at gmail.com  Mon Sep 24 14:11:13 2012
From: parakrama1282 at gmail.com (dhanushka ranasinghe)
Date: Mon, 24 Sep 2012 19:41:13 +0530
Subject: Multiple virtual host with different IP's
Message-ID: <CANq=HMjxReVQq9yuYw1eBLL3Q55pPLDNF_+kcvZpG9SqGr34vQ@mail.gmail.com>

Hi  guys ,

Im using nginx with varnish,  and i have following  two virtual-host *[
content.com, content.org ]  *configuration in nginx

With follwing configuration   varnish only cache the content of the *
content.org * , its not caching the *content.com* , Do i need to define
multiple back-ends for each virtual-hosts, can any one giude me how to do
that .Or do i need to configure  separate Varnish instance for each VHOST.


Im not try to cache the https contents , only http need to cache.


This is my configuration...


Configuration
-------------------

*content.com*
===========

server {
    listen      10.100.1.1:8080;
    server_name   content.com;
    access_log /var/log/nginx/content.com.log;
        root /var/www/content.com;
        expires       14d;
        add_header Cache-Control  public;
  }


server {
    listen      10.100.1.1:443;
    server_name   content.com;
    access_log /var/log/nginx/content.com.log;
        root /var/www/content.com;
        expires       14d;
        add_header Cache-Control  public;

        ssl                 on;
        ssl_certificate     /etc/nginx/ssl/content.pem;
        ssl_certificate_key /etc/nginx/ssl/content.key;


  }


*content.org
============*

 server {
                listen      10.100.1.2:8080 default;
                server_name  cache.org;
                {
                        root   /home/httpd/content.org;
                        autoindex on;
                }
        }

     server {
                listen  10.100.1.2:443;
                ssl    on;
                ssl_certificate    /etc/nginx/ssl/content-org.pem;
                ssl_certificate_key    /etc/nginx/ssl/content-org.key;

                server_name content.org;

                {
                root   /home/httpd/content.org;
                autoindex on;
                }



*This is my  varnish default.vcl
======================*

backend default {
    .host = "10.100.1.2";
    .port = "8080";
}

sub vcl_fetch {
     if (!beresp.cacheable) {
                        unset beresp.http.expires;
                        set beresp.http.cache-control = "max-age=900";
                        set beresp.ttl = 1w;
                        set beresp.http.magicmarker = "1";
         return (pass);
     }
     if (beresp.http.Set-Cookie) {
         return (pass);
     }
     return (deliver);
 }
 sub vcl_deliver {
     return (deliver);
 }





Thanks in advance


Thank You
Dhanushka
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120924/fe735f79/attachment.html>

From socha at socha.it  Mon Sep 24 16:04:29 2012
From: socha at socha.it (Robert Socha)
Date: Mon, 24 Sep 2012 18:04:29 +0200
Subject: Multiple virtual host with different IP's
In-Reply-To: <CANq=HMjxReVQq9yuYw1eBLL3Q55pPLDNF_+kcvZpG9SqGr34vQ@mail.gmail.com>
References: <CANq=HMjxReVQq9yuYw1eBLL3Q55pPLDNF_+kcvZpG9SqGr34vQ@mail.gmail.com>
Message-ID: <5060848D.50100@socha.it>

Hi


W dniu 2012-09-24 16:11, dhanushka ranasinghe pisze:
> Hi  guys ,
> With follwing configuration   varnish only cache the content of the
> *content.org <http://content.org> * , its not caching the *content.com
> <http://content.com>* , Do i need to define multiple back-ends for each
> virtual-hosts, can any one giude me how to do that .Or do i need to
> configure  separate Varnish instance for each VHOST.
> 

You need define multiple backends. No need for separate instances.




> server {
>     listen      10.100.1.1:8080 <http://10.100.1.1:8080>;
>     server_name   content.com <http://content.com>;
...

backend bkcontentcom {
     .host = "10.100.1.1";
     .port = "8080";
}


>  server {
>                 listen      10.100.1.2:8080 <http://10.100.1.2:8080>
>                 server_name  cache.org <http://cache.org>;
...

backend bkcacheorg {
     .host = "10.100.1.2";
     .port = "8080";
}

And in the begining of vcl_recv()

if (req.http.host ~ "^(www\.)?content\.com$") {
	set req.backend = bkcontentcom;
} else {
	// for cache.org
	set req.backend = bkcacheorg;
}



And of course your backends need to provide correct Cache-Control
headers (ie. max-age or s-max-age).

Greetings
-- 
Robert Socha





From parakrama1282 at gmail.com  Tue Sep 25 07:31:07 2012
From: parakrama1282 at gmail.com (dhanushka ranasinghe)
Date: Tue, 25 Sep 2012 13:01:07 +0530
Subject: Multiple virtual host with different IP's
In-Reply-To: <5060848D.50100@socha.it>
References: <CANq=HMjxReVQq9yuYw1eBLL3Q55pPLDNF_+kcvZpG9SqGr34vQ@mail.gmail.com>
 <5060848D.50100@socha.it>
Message-ID: <CANq=HMjoDLx=7mGbZ2Fj1DAKw-S5btcR6kMK_9FfAq4wO6kmsQ@mail.gmail.com>

Hi..guys

Thanks for the help, i creared the default.vcl with multiple back-ends ,
now my varnish is working fine can access both content.com and
content.orgsites without any problem , This is my configuration , is
there any tips
[configuration changes ] to get maxim performance  or is there any draw
backs in this configuration...

configuration
--------------------

backend bkcachecom {
    .host = "10.100.1.1";
    .port = "8080";
}


backend bkcacheorg {
    .host = "10.100.1.2";
    .port = "8080";
}


sub vcl_recv {

if (req.http.host ~ "^a.content.org$") {
        set req.backend = bkcacheorg;
        return (lookup);
}

else if (req.http.host ~ "^b.content.org$") {
        set req.backend = bkcacheorg;
        return (lookup);
}

else if (req.http.host ~ "^a.content.com$") {
        set req.backend = bkcachecom;
        return (lookup);
}

else if (req.http.host ~ "^b.content.com$") {
        set req.backend = bkcachecom;
        return (lookup);
}

sub vcl_fetch {
     if (!beresp.cacheable) {
                        unset beresp.http.expires;
                        set beresp.http.cache-control = "max-age=900";
                        set beresp.ttl = 1w;
                        set beresp.http.magicmarker = "1";
         return (pass);
     }
     if (beresp.http.Set-Cookie) {
         return (pass);
     }
     return (deliver);
 }
 sub vcl_deliver {
     return (deliver);
 }


Thanks in Advance

Thank You


On 24 September 2012 21:34, Robert Socha <socha at socha.it> wrote:

> Hi
>
>
> W dniu 2012-09-24 16:11, dhanushka ranasinghe pisze:
> > Hi  guys ,
> > With follwing configuration   varnish only cache the content of the
> > *content.org <http://content.org> * , its not caching the *content.com
> > <http://content.com>* , Do i need to define multiple back-ends for each
> > virtual-hosts, can any one giude me how to do that .Or do i need to
> > configure  separate Varnish instance for each VHOST.
> >
>
> You need define multiple backends. No need for separate instances.
>
>
>
>
> > server {
> >     listen      10.100.1.1:8080 <http://10.100.1.1:8080>;
> >     server_name   content.com <http://content.com>;
> ...
>
> backend bkcontentcom {
>      .host = "10.100.1.1";
>      .port = "8080";
> }
>
>
> >  server {
> >                 listen      10.100.1.2:8080 <http://10.100.1.2:8080>
> >                 server_name  cache.org <http://cache.org>;
> ...
>
> backend bkcacheorg {
>      .host = "10.100.1.2";
>      .port = "8080";
> }
>
> And in the begining of vcl_recv()
>
> if (req.http.host ~ "^(www\.)?content\.com$") {
>         set req.backend = bkcontentcom;
> } else {
>         // for cache.org
>         set req.backend = bkcacheorg;
> }
>
>
>
> And of course your backends need to provide correct Cache-Control
> headers (ie. max-age or s-max-age).
>
> Greetings
> --
> Robert Socha
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120925/c8e157d5/attachment.html>

From parakrama1282 at gmail.com  Tue Sep 25 13:55:15 2012
From: parakrama1282 at gmail.com (dhanushka ranasinghe)
Date: Tue, 25 Sep 2012 19:25:15 +0530
Subject: return (lookup) means
Message-ID: <CANq=HMhvgaUntXveNE9f5MOcAMg03WJDHbQQ6sEVYWbYt-R3hg@mail.gmail.com>

Hi.. guys ,

In my varnish code im using  "return (lookup)" ,


sub vcl_recv {

if (req.http.host ~ "^a.content.org$") {
        set req.backend = bkcacheorg;
        return (lookup);
}

}



Please correct me if im wrong ..in varnish doc says   *return (lookup)
means *  "Varnish to deliver content from cache even if the request
othervise indicates that the request should be passed."

i just want to know im using "return (lookup)"  , if the content does not
found in varnish cache , its it going to talk to  backend and get the
correct content...

Thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120925/2ee3c3e7/attachment.html>

From parakrama1282 at gmail.com  Tue Sep 25 14:26:17 2012
From: parakrama1282 at gmail.com (dhanushka ranasinghe)
Date: Tue, 25 Sep 2012 19:56:17 +0530
Subject: return (lookup) means
In-Reply-To: <CANq=HMhvgaUntXveNE9f5MOcAMg03WJDHbQQ6sEVYWbYt-R3hg@mail.gmail.com>
References: <CANq=HMhvgaUntXveNE9f5MOcAMg03WJDHbQQ6sEVYWbYt-R3hg@mail.gmail.com>
Message-ID: <CANq=HMhROiTw0nrdPoxtEi3Akc9enUAKoA89hGGvrEN1-2-Ugw@mail.gmail.com>

Hi..

when i removed  *return (lookup);*  from  "*IF*" statement  content request
are going to the backed all the time.Not serving from the cache.

What should i follow ? ,IS there any recommended settings ? , or is it ok
to use *return (lookup); *in VCL code ? as mention in the previous mail.


Thanks in Advance

Thank You






On 25 September 2012 19:25, dhanushka ranasinghe <parakrama1282 at gmail.com>wrote:

> Hi.. guys ,
>
> In my varnish code im using  "return (lookup)" ,
>
>
> sub vcl_recv {
>
> if (req.http.host ~ "^a.content.org$") {
>         set req.backend = bkcacheorg;
>         return (lookup);
> }
>
> }
>
>
>
> Please correct me if im wrong ..in varnish doc says   *return (lookup)
> means *  "Varnish to deliver content from cache even if the request
> othervise indicates that the request should be passed."
>
> i just want to know im using "return (lookup)"  , if the content does not
> found in varnish cache , its it going to talk to  backend and get the
> correct content...
>
> Thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120925/7d28d8c8/attachment.html>

From enno at tty.nl  Tue Sep 25 15:10:50 2012
From: enno at tty.nl (Enno van Amerongen)
Date: Tue, 25 Sep 2012 17:10:50 +0200
Subject: return (lookup) means
In-Reply-To: <CANq=HMhROiTw0nrdPoxtEi3Akc9enUAKoA89hGGvrEN1-2-Ugw@mail.gmail.com>
Message-ID: <20120925151050.81073ff9@groupware.tty.nl>

that is correct.
  _____  

From: dhanushka ranasinghe [mailto:parakrama1282 at gmail.com]
Subject: Re: return (lookup) means


i just want to know im using "return (lookup)"  , if the content does   not found in varnish cache , its it going to talk to  backend and get the correct content...

    
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120925/f19bf7ae/attachment.html>

From sahmed1020 at gmail.com  Tue Sep 25 19:58:18 2012
From: sahmed1020 at gmail.com (S Ahmed)
Date: Tue, 25 Sep 2012 15:58:18 -0400
Subject: varnish use case, something like an ebay listing page
Message-ID: <CAG2rwuPf6uKWbgCQjhM33RVg0mBE48AJGmVsM4Dk=UfJtvigJA@mail.gmail.com>

An ebay listing page is a page that will have many reads, and at the same
time it will experience a high load of updates also.

So someone lists an item on ebay, and people will keep checking the page
constantly, but near the end of the listing people will start bidding on
the page (invalidating the cache).

So whenever someone bids, I would write that to a queue, and then I would
somehow send a call to varnish to expire the page correct?

This could result it many calls to varnish per second, is there a
reasonable # of requests per second that varnish can handle? 10 per second,
100, 1000?

Is there a better way to do this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120925/1b93e61f/attachment.html>

From pprocacci at datapipe.com  Tue Sep 25 20:15:44 2012
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Tue, 25 Sep 2012 15:15:44 -0500
Subject: varnish use case, something like an ebay listing page
In-Reply-To: <CAG2rwuPf6uKWbgCQjhM33RVg0mBE48AJGmVsM4Dk=UfJtvigJA@mail.gmail.com>
References: <CAG2rwuPf6uKWbgCQjhM33RVg0mBE48AJGmVsM4Dk=UfJtvigJA@mail.gmail.com>
Message-ID: <20120925201544.GG97380@nat.myhome>

`Bidding` has to retain realtime updates in order to provide realtime
data to the participants of said bid.  IMHO, you don't want varnish
caching this value at all.

What I would consider is caching the page as per normal, but consider
using esi or javascript to pull in the values of what the current bid is.

This allows you to retain caching most of the page, while controlling
the truly dynamic parts of your page that must be shown in realtime.

~Paul

On Tue, Sep 25, 2012 at 03:58:18PM -0400, S Ahmed wrote:
>    An ebay listing page is a page that will have many reads, and at the
>    same time it will experience a high load of updates also.
>
>    So someone lists an item on ebay, and people will keep checking the
>    page constantly, but near the end of the listing people will start
>    bidding on the page (invalidating the cache).
>
>    So whenever someone bids, I would write that to a queue, and then I
>    would somehow send a call to varnish to expire the page correct?
>
>    This could result it many calls to varnish per second, is there a
>    reasonable # of requests per second that varnish can handle? 10 per
>    second, 100, 1000?
>
>    Is there a better way to do this?

> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.



From parakrama1282 at gmail.com  Wed Sep 26 07:11:02 2012
From: parakrama1282 at gmail.com (dhanushka ranasinghe)
Date: Wed, 26 Sep 2012 12:41:02 +0530
Subject: return (lookup) means
In-Reply-To: <20120925151050.81073ff9@groupware.tty.nl>
References: <CANq=HMhROiTw0nrdPoxtEi3Akc9enUAKoA89hGGvrEN1-2-Ugw@mail.gmail.com>
 <20120925151050.81073ff9@groupware.tty.nl>
Message-ID: <CANq=HMiUjMhKX6rVAg-SPtFA4LoJCVDOBdYDQEpCVy_Mmo08+Q@mail.gmail.com>

Hi... guys

just one more question  ,  lets say

i have  BLACK colored image with following URL
http://a.content.com/wp-content/themes/image.gif [1]

then i change the image color to RED  , and URL [1] is same as above . i
just want to know  since im using  *return (lookup) , *is varnish going to
serve old [black] image from cache  or new  image [red] form back-end .
when users accessing the URL [1]

Thank in Advance..

Thank You
*

*
On 25 September 2012 20:40, Enno van Amerongen <enno at tty.nl> wrote:

> **
> that is correct.
>
> ------------------------------
> *From:* dhanushka ranasinghe [mailto:parakrama1282 at gmail.com]
> *Subject:* Re: return (lookup) means
>
>
> i just want to know im using "return (lookup)"  , if the content does not
>> found in varnish cache , its it going to talk to  backend and get the
>> correct content...
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120926/8ac921ee/attachment.html>

From zeeshan at infoshield.info  Wed Sep 26 07:26:59 2012
From: zeeshan at infoshield.info (Zeeshan Ali Shah)
Date: Wed, 26 Sep 2012 09:26:59 +0200
Subject: return (lookup) means
In-Reply-To: <CANq=HMiUjMhKX6rVAg-SPtFA4LoJCVDOBdYDQEpCVy_Mmo08+Q@mail.gmail.com>
References: <CANq=HMhROiTw0nrdPoxtEi3Akc9enUAKoA89hGGvrEN1-2-Ugw@mail.gmail.com>
 <20120925151050.81073ff9@groupware.tty.nl>
 <CANq=HMiUjMhKX6rVAg-SPtFA4LoJCVDOBdYDQEpCVy_Mmo08+Q@mail.gmail.com>
Message-ID: <CAJSm=nuAWTXLbSecDj0so7mNHHdejLQA6aN44ka1K2Rx=f-H+A@mail.gmail.com>

I think lookup will pick the image from cache until TTL is expired.. so you
will get the black image. if you want red  than Purge the cache

BR

Zeeshan

On Wed, Sep 26, 2012 at 9:11 AM, dhanushka ranasinghe <
parakrama1282 at gmail.com> wrote:

> Hi... guys
>
> just one more question  ,  lets say
>
> i have  BLACK colored image with following URL
> http://a.content.com/wp-content/themes/image.gif [1]
>
> then i change the image color to RED  , and URL [1] is same as above . i
> just want to know  since im using  *return (lookup) , *is varnish going
> to serve old [black] image from cache  or new  image [red] form back-end .
> when users accessing the URL [1]
>
> Thank in Advance..
>
> Thank You
>
> *
>
> *
> On 25 September 2012 20:40, Enno van Amerongen <enno at tty.nl> wrote:
>
>> **
>> that is correct.
>>
>> ------------------------------
>> *From:* dhanushka ranasinghe [mailto:parakrama1282 at gmail.com]
>> *Subject:* Re: return (lookup) means
>>
>>
>> i just want to know im using "return (lookup)"  , if the content does not
>>> found in varnish cache , its it going to talk to  backend and get the
>>> correct content...
>>>
>>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120926/41d36c43/attachment.html>

From nick.tailor at gmail.com  Wed Sep 26 15:06:17 2012
From: nick.tailor at gmail.com (nick tailor)
Date: Wed, 26 Sep 2012 08:06:17 -0700
Subject: return (lookup) means
In-Reply-To: <CANq=HMiUjMhKX6rVAg-SPtFA4LoJCVDOBdYDQEpCVy_Mmo08+Q@mail.gmail.com>
References: <CANq=HMhROiTw0nrdPoxtEi3Akc9enUAKoA89hGGvrEN1-2-Ugw@mail.gmail.com>
 <20120925151050.81073ff9@groupware.tty.nl>
 <CANq=HMiUjMhKX6rVAg-SPtFA4LoJCVDOBdYDQEpCVy_Mmo08+Q@mail.gmail.com>
Message-ID: <CA+8iBq_=3Y3jko1OFQ55Ss2ir+z4TMTgd=GehCR4K1SBTwCGmw@mail.gmail.com>

Hi  dhanushka,

Usually, all image files are static and you can just have varnish always
cache them. The objects cached. you can have it set so that the cache is
updated say every 30 seconds as long as your webservers are up, and you can
also set, should your webservers go down the ttl for the objects cached is
now longer making everything appear that its all still up, you fix your
issues with your backend. The better you know your application and how it
works, the more fine tuned you can make your VCL

Cheers

Nick Tailor

On Wed, Sep 26, 2012 at 12:11 AM, dhanushka ranasinghe <
parakrama1282 at gmail.com> wrote:

> Hi... guys
>
> just one more question  ,  lets say
>
> i have  BLACK colored image with following URL
> http://a.content.com/wp-content/themes/image.gif [1]
>
> then i change the image color to RED  , and URL [1] is same as above . i
> just want to know  since im using  *return (lookup) , *is varnish going
> to serve old [black] image from cache  or new  image [red] form back-end .
> when users accessing the URL [1]
>
> Thank in Advance..
>
> Thank You
>
> *
>
> *
> On 25 September 2012 20:40, Enno van Amerongen <enno at tty.nl> wrote:
>
>> **
>> that is correct.
>>
>> ------------------------------
>> *From:* dhanushka ranasinghe [mailto:parakrama1282 at gmail.com]
>> *Subject:* Re: return (lookup) means
>>
>>
>> i just want to know im using "return (lookup)"  , if the content does not
>>> found in varnish cache , its it going to talk to  backend and get the
>>> correct content...
>>>
>>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120926/e76fada6/attachment.html>

From z999666999 at gmail.com  Thu Sep 27 14:56:22 2012
From: z999666999 at gmail.com (z999666999 at gmail.com)
Date: Thu, 27 Sep 2012 14:56:22 +0000
Subject: =?gb18030?B?tPC4tKO6IFJFOiBIb3cgdG8gcHV0IHZhcm5pc2hsb2cgdG8gb25lIGxpbmU=?=
In-Reply-To: <4E4FA8CEA7D2CA46B10FAAE3F58E84225A75AB52DE@HOLANDAMBX.elmercurio.org>
References: <1641038052-1347506097-cardhu_decombobulator_blackberry.rim.net-639759824-@b4.c16.bise6.blackberry>
 <4E4FA8CEA7D2CA46B10FAAE3F58E84225A75AB52DE@HOLANDAMBX.elmercurio.org>
Message-ID: <347148787-1348757783-cardhu_decombobulator_blackberry.rim.net-1983195027-@b14.c16.bise6.blackberry>

Thank you Juan.
Varnishncsa only provide very limit information in good one line format, but varnishlog actually provide very detail information on multi lines, like retry times, error reason.
The thing I want is with all the details in one line, so I can process it with my tools that only deal with logs one line at a time.

Thanks,
Z9
Sent from my BlackBerry device on the Rogers Wireless Network

-----Original Message-----
From: <juan.salinas at mercurio.cl>
Date: Thu, 27 Sep 2012 11:14:07 
To: <z999666999 at gmail.com>; <varnish-misc at varnish-cache.org>
Subject: RE: How to put varnishlog to one line

I don't know if this is what you want to do, try with varnishncsa instead varnishlog...
Example:
varnishncsa | grep -P "\"GET .+\" 50(2|3) "

Juan Salinas Contreras
Correo: juan.salinas at mercurio.cl
Telefono: +56-2-3301897



-----Mensaje original-----
De: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] En nombre de z999666999 at gmail.com
Enviado el: jueves, 13 de septiembre de 2012 0:15
Para: varnish-misc at varnish-cache.org
Asunto: How to put varnishlog to one line

Hi,
I am pretty new to varnish. Sorry for newbie question.
I use varnishlog -c -m 'TxStatus:50(2|3)' to get 502,503 errors. My question is:
If I want this save to a file but all info should be on one line, how to do that?

Thanks,
Z9
Sent from my BlackBerry device on the Rogers Wireless Network _______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

From huozhe at gmail.com  Thu Sep 27 15:47:09 2012
From: huozhe at gmail.com (Zheng Liu)
Date: Thu, 27 Sep 2012 08:47:09 -0700
Subject: memory issue when running varnishncsa
Message-ID: <CAOCtP1d82eFiUXSksvZVMcTiDRANMhNd7hzL_q3n2DHu9=60zg@mail.gmail.com>

Hi,

I tried to use varnishncsa to generate log on disk, so I can have another
daemon tail and analyze it. However, when I enable it, I see varnishd
starts to use more and more memory. I am using varnish-3.0.3 on Ubuntu
11.10. The box has 190G and I configured varnish to use 160G. Its footprint
has been stable at 180G for months. Then after I started varnishncsa,
within 5 hours, all the memory was gone and it starts to use swap. I wonder
if this is expected behavior, and we are suppose to allow more head room
during logging with varnishncsa?

thanks,
Zheng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120927/89e8ad6c/attachment.html>

From Johan.Olsson at bonnierdigital.se  Thu Sep 27 22:02:08 2012
From: Johan.Olsson at bonnierdigital.se (Johan Olsson)
Date: Thu, 27 Sep 2012 22:02:08 +0000
Subject: Monitor per site in Varnish
Message-ID: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>

Hi
I've been looking on how to monitor varnish. I've found that there exists a snmp for varnish which gives some info that is good to have. I've found it and looked at it (http://sourceforge.net/projects/varnishsnmp/), but it dosen't give all that I need (I think). What I'm missing is to be able to monitor how much traffic one site is using. So if I have two sites like www.example1.com<http://www.example1.com> and www.example2.com<http://www.example2.com>, I would like to be able to get how many connections each one gets and how much Mbps each one is using.
Is this possible to do?

Regards
Johan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120927/7eb14c4e/attachment.html>

From nick.tailor at gmail.com  Thu Sep 27 22:09:13 2012
From: nick.tailor at gmail.com (nick tailor)
Date: Thu, 27 Sep 2012 15:09:13 -0700
Subject: Monitor per site in Varnish
In-Reply-To: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
References: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
Message-ID: <CA+8iBq8aDyaqVF4sjSydJxS8Y4Og3B40oiDTGd9ypTeObknDBQ@mail.gmail.com>

Why not use google analytics?

Cheers

Nick Tailor

On Thu, Sep 27, 2012 at 3:02 PM, Johan Olsson <
Johan.Olsson at bonnierdigital.se> wrote:

>  Hi****
>
> I?ve been looking on how to monitor varnish. I?ve found that there exists
> a snmp for varnish which gives some info that is good to have. I?ve found
> it and looked at it (http://sourceforge.net/projects/varnishsnmp/), but
> it dosen?t give all that I need (I think). What I?m missing is to be able
> to monitor how much traffic one site is using. So if I have two sites like
> www.example1.com and www.example2.com, I would like to be able to get how
> many connections each one gets and how much Mbps each one is using.****
>
> Is this possible to do? ****
>
> ** **
>
> Regards****
>
> Johan ****
>
> ** **
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120927/40481ba3/attachment.html>

From Johan.Olsson at bonnierdigital.se  Thu Sep 27 22:25:02 2012
From: Johan.Olsson at bonnierdigital.se (Johan Olsson)
Date: Thu, 27 Sep 2012 22:25:02 +0000
Subject: Monitor per site in Varnish
In-Reply-To: <CA+8iBq8aDyaqVF4sjSydJxS8Y4Og3B40oiDTGd9ypTeObknDBQ@mail.gmail.com>
References: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
 <CA+8iBq8aDyaqVF4sjSydJxS8Y4Og3B40oiDTGd9ypTeObknDBQ@mail.gmail.com>
Message-ID: <C1C0937CADC64B49A0B1F5C83E284FE50A381703@EXCH2.bds.internal>

Hi Nick,

Thanks for the answer.
With Google Analytics I will be able to see how many pageviews we have. But I want to be able to see how many http connections and how much Mbps is used. By using pageviews, we won't be able to calculate the actual amount of Mbps that for example www.example1.com<http://www.example1.com> is using.

Thanks,
Johan Olsson

From: nick tailor [mailto:nick.tailor at gmail.com]
Sent: den 28 september 2012 00:09
To: Johan Olsson
Cc: varnish-misc at varnish-cache.org
Subject: Re: Monitor per site in Varnish

Why not use google analytics?

Cheers

Nick Tailor
On Thu, Sep 27, 2012 at 3:02 PM, Johan Olsson <Johan.Olsson at bonnierdigital.se<mailto:Johan.Olsson at bonnierdigital.se>> wrote:
Hi
I've been looking on how to monitor varnish. I've found that there exists a snmp for varnish which gives some info that is good to have. I've found it and looked at it (http://sourceforge.net/projects/varnishsnmp/), but it dosen't give all that I need (I think). What I'm missing is to be able to monitor how much traffic one site is using. So if I have two sites like www.example1.com<http://www.example1.com> and www.example2.com<http://www.example2.com>, I would like to be able to get how many connections each one gets and how much Mbps each one is using.
Is this possible to do?

Regards
Johan


_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120927/9e65a2bc/attachment.html>

From nick.tailor at gmail.com  Thu Sep 27 22:29:22 2012
From: nick.tailor at gmail.com (nick tailor)
Date: Thu, 27 Sep 2012 15:29:22 -0700
Subject: Monitor per site in Varnish
In-Reply-To: <C1C0937CADC64B49A0B1F5C83E284FE50A381703@EXCH2.bds.internal>
References: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
 <CA+8iBq8aDyaqVF4sjSydJxS8Y4Og3B40oiDTGd9ypTeObknDBQ@mail.gmail.com>
 <C1C0937CADC64B49A0B1F5C83E284FE50A381703@EXCH2.bds.internal>
Message-ID: <CA+8iBq-F2eKUAg-aYYrSnbDKy5yKCiOFdUh7AWF3oUyjtMgqDg@mail.gmail.com>

Are you sure about that? I have a colleague that has a high traffic site,
does something like 3 terabytes a month, and they use google analyitcs to
see all that? I will message him to confirm

Cheers

Nick Tailor

On Thu, Sep 27, 2012 at 3:25 PM, Johan Olsson <
Johan.Olsson at bonnierdigital.se> wrote:

>  Hi Nick,****
>
> ** **
>
> Thanks for the answer.****
>
> With Google Analytics I will be able to see how many pageviews we have.
> But I want to be able to see how many http connections and how much Mbps is
> used. By using pageviews, we won?t be able to calculate the actual amount
> of Mbps that for example www.example1.com is using.****
>
> ** **
>
> Thanks,****
>
> Johan Olsson****
>
> ** **
>
> *From:* nick tailor [mailto:nick.tailor at gmail.com]
> *Sent:* den 28 september 2012 00:09
> *To:* Johan Olsson
> *C**c:* varnish-misc at varnish-cache.org
> *Subject:* Re: Monitor per site in Varnish****
>
> ** **
>
> Why not use google analytics?****
>
> ** **
>
> Cheers****
>
> ** **
>
> Nick Tailor****
>
> On Thu, Sep 27, 2012 at 3:02 PM, Johan Olsson <
> Johan.Olsson at bonnierdigital.se> wrote:****
>
> Hi****
>
> I?ve been looking on how to monitor varnish. I?ve found that there exists
> a snmp for varnish which gives some info that is good to have. I?ve found
> it and looked at it (http://sourceforge.net/projects/varnishsnmp/), but
> it dosen?t give all that I need (I think). What I?m missing is to be able
> to monitor how much traffic one site is using. So if I have two sites like
> www.example1.com and www.example2.com, I would like to be able to get how
> many connections each one gets and how much Mbps each one is using.****
>
> Is this possible to do? ****
>
>  ****
>
> Regards****
>
> Johan ****
>
>  ****
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc****
>
> ** **
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120927/66801c06/attachment.html>

From drais at icantclick.org  Thu Sep 27 22:42:46 2012
From: drais at icantclick.org (david raistrick)
Date: Thu, 27 Sep 2012 18:42:46 -0400 (EDT)
Subject: Monitor per site in Varnish
In-Reply-To: <CA+8iBq-F2eKUAg-aYYrSnbDKy5yKCiOFdUh7AWF3oUyjtMgqDg@mail.gmail.com>
References: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
 <CA+8iBq8aDyaqVF4sjSydJxS8Y4Og3B40oiDTGd9ypTeObknDBQ@mail.gmail.com>
 <C1C0937CADC64B49A0B1F5C83E284FE50A381703@EXCH2.bds.internal>
 <CA+8iBq-F2eKUAg-aYYrSnbDKy5yKCiOFdUh7AWF3oUyjtMgqDg@mail.gmail.com>
Message-ID: <alpine.BSF.2.00.1209271839250.34457@murf.icantclick.org>

On Thu, 27 Sep 2012, nick tailor wrote:

> Are you sure about that? I have a colleague that has a high traffic site,
> does something like 3 terabytes a month, and they use google analyitcs to
> see all that? I will message him to confirm

Google analytics isn't actually free, btw....

And it only works for eyeball sites since it depends on javascript running 
in the browser.  So for those of us that run webservices your answer is a 
nonstarter.

GA has no way to give you real network or connection statistics since it 
doesnt ever talk to your server.


For the OP, afaik short of building your own stats collection/reporting 
system inside varnish (vmod or inline c), your best bet is to gather this 
data at your webserver/application server(s).  Admittedly this leaves out 
cached responses.

Alternately, use varnishncsa to logfiles and process them like any 
webserver logs.....



--
david raistrick        http://www.netmeister.org/news/learn2quote.html
drais at icantclick.org       ascii ribbon campaign - stop html mail
                                 http://www.asciiribbon.org/






From magnus at hagander.net  Thu Sep 27 22:46:36 2012
From: magnus at hagander.net (Magnus Hagander)
Date: Fri, 28 Sep 2012 00:46:36 +0200
Subject: Monitor per site in Varnish
In-Reply-To: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
References: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
Message-ID: <CABUevEwgsbXxFQMy5+p4pjOZChBeY=mY1PJhiKrbi79Ykfwoug@mail.gmail.com>

On Fri, Sep 28, 2012 at 12:02 AM, Johan Olsson
<Johan.Olsson at bonnierdigital.se> wrote:
> Hi
>
> I?ve been looking on how to monitor varnish. I?ve found that there exists a
> snmp for varnish which gives some info that is good to have. I?ve found it
> and looked at it (http://sourceforge.net/projects/varnishsnmp/), but it
> dosen?t give all that I need (I think). What I?m missing is to be able to
> monitor how much traffic one site is using. So if I have two sites like
> www.example1.com and www.example2.com, I would like to be able to get how
> many connections each one gets and how much Mbps each one is using.
>
> Is this possible to do?

You can always hook yourself up directly to the logging buffer and
read the data from there, or feed off varnishncsa in a pipe for
example. IIRC the internal counters don't really give you things
broken down per domain - because a domain is just on of a bazillion
different things you might want to do things off...


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/



From Johan.Olsson at bonnierdigital.se  Thu Sep 27 23:18:17 2012
From: Johan.Olsson at bonnierdigital.se (Johan Olsson)
Date: Thu, 27 Sep 2012 23:18:17 +0000
Subject: Monitor per site in Varnish
In-Reply-To: <CABUevEwgsbXxFQMy5+p4pjOZChBeY=mY1PJhiKrbi79Ykfwoug@mail.gmail.com>
References: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
 <CABUevEwgsbXxFQMy5+p4pjOZChBeY=mY1PJhiKrbi79Ykfwoug@mail.gmail.com>
Message-ID: <C1C0937CADC64B49A0B1F5C83E284FE50A38179B@EXCH2.bds.internal>

Hi Magnus,

Will have a look at the logging buffer and see what I can do there. Is there any different way to gather this data if I instead would look at a vhost in varnish?

Regards
Johan Olsson

-----Original Message-----
From: Magnus Hagander [mailto:magnus at hagander.net] 
Sent: den 28 september 2012 00:47
To: Johan Olsson
Cc: varnish-misc at varnish-cache.org
Subject: Re: Monitor per site in Varnish

On Fri, Sep 28, 2012 at 12:02 AM, Johan Olsson <Johan.Olsson at bonnierdigital.se> wrote:
> Hi
>
> I've been looking on how to monitor varnish. I've found that there 
> exists a snmp for varnish which gives some info that is good to have. 
> I've found it and looked at it 
> (http://sourceforge.net/projects/varnishsnmp/), but it dosen't give 
> all that I need (I think). What I'm missing is to be able to monitor 
> how much traffic one site is using. So if I have two sites like 
> www.example1.com and www.example2.com, I would like to be able to get how many connections each one gets and how much Mbps each one is using.
>
> Is this possible to do?

You can always hook yourself up directly to the logging buffer and read the data from there, or feed off varnishncsa in a pipe for example. IIRC the internal counters don't really give you things broken down per domain - because a domain is just on of a bazillion different things you might want to do things off...


--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/



From magnus at hagander.net  Thu Sep 27 23:21:16 2012
From: magnus at hagander.net (Magnus Hagander)
Date: Fri, 28 Sep 2012 01:21:16 +0200
Subject: Monitor per site in Varnish
In-Reply-To: <C1C0937CADC64B49A0B1F5C83E284FE50A38179B@EXCH2.bds.internal>
References: <C1C0937CADC64B49A0B1F5C83E284FE50A3816AF@EXCH2.bds.internal>
 <CABUevEwgsbXxFQMy5+p4pjOZChBeY=mY1PJhiKrbi79Ykfwoug@mail.gmail.com>
 <C1C0937CADC64B49A0B1F5C83E284FE50A38179B@EXCH2.bds.internal>
Message-ID: <CABUevExz8ADJFm3cZ4dOY+5vLmJRHRU2MaZsp=T8joZ_5LMTPQ@mail.gmail.com>

On Fri, Sep 28, 2012 at 1:18 AM, Johan Olsson
<Johan.Olsson at bonnierdigital.se> wrote:
> Hi Magnus,
>
> Will have a look at the logging buffer and see what I can do there. Is there any different way to gather this data if I instead would look at a vhost in varnish?

What would you define as a vhost in varnish? There is really no such
thing, is there? These things are defined in VCL, so it's just code..
You could add a tag of some kind (e.g. a custom http header) in vcl
and then read that out, but I'm not sure that actually makes it
easier.

The easiest thing is probably to work off varnishncsa if it has enough
info for you.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/



From smsmail at roadrunner.com  Thu Sep 27 23:25:44 2012
From: smsmail at roadrunner.com (Mark Strickland)
Date: Thu, 27 Sep 2012 19:25:44 -0400
Subject: not sure if Varnish is working correctly
Message-ID: <000f01cd9d07$6b8c2150$42a463f0$@roadrunner.com>

I have varnish installed on my Linode VPS, I am getting good hit rates, but
site still seems to me to be slow loading.  I think I have it setup to use
128M of  memory instead of using a file.  I am sorry if I gave to much
information or the wrong information.   Thanks in advance for any help or
guidance.

 

Here is my configuration file:

 

# Configuration file for varnish

#

# /etc/init.d/varnish expects the variables $DAEMON_OPTS, $NFILES and
$MEMLOCK

# to be set from this shell script fragment.

#

 

# Should we start varnishd at boot?  Set to "no" to disable.

START=yes

 

# Maximum number of open files (for ulimit -n)

NFILES=131072

 

# Maximum locked memory size (for ulimit -l)

# Used for locking the shared memory log in memory.  If you increase log
size,

# you need to increase this number as well

MEMLOCK=82000

 

# Default varnish instance name is the local nodename.  Can be overridden
with

# the -n switch, to have more instances on a single server.

# INSTANCE=$(uname -n)

 

# This file contains 4 alternatives, please use only one.

 

## Alternative 1, Minimal configuration, no VCL

#

# Listen on port 6081, administration on localhost:6082, and forward to

# content server on localhost:8080.  Use a 1GB fixed-size cache file.

#

# DAEMON_OPTS="-a :6081 \

#              -T localhost:6082 \

#           -b localhost:8080 \

#           -u varnish -g varnish \

#            -S /etc/varnish/secret \

#           -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

 

 

## Alternative 2, Configuration with VCL

#

# Listen on port 6081, administration on localhost:6082, and forward to

# one content server selected by the vcl file, based on the request.  Use a
1GB

# fixed-size cache file.

#

DAEMON_OPTS="-a :80 \

             -T localhost:6082 \

             -f /etc/varnish/default.vcl \

             -S /etc/varnish/secret \

              -s malloc,128M"

#             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,128M"

 

 

## Alternative 3, Advanced configuration

#

# See varnishd(1) for more information.

#

# # Main configuration file. You probably want to change it :)

# VARNISH_VCL_CONF=/etc/varnish/default.vcl

#

# # Default address and port to bind to

# # Blank address means all IPv4 and IPv6 interfaces, otherwise specify

# # a host name, an IPv4 dotted quad, or an IPv6 address in brackets.

# VARNISH_LISTEN_ADDRESS=

# VARNISH_LISTEN_PORT=6081

#

# # Telnet admin interface listen address and port

# VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1

# VARNISH_ADMIN_LISTEN_PORT=6082

#

# # The minimum number of worker threads to start

# VARNISH_MIN_THREADS=1

#

# # The Maximum number of worker threads to start

# VARNISH_MAX_THREADS=1000

#

# # Idle timeout for worker threads

# VARNISH_THREAD_TIMEOUT=120

#

# # Cache file location

# VARNISH_STORAGE_FILE=/var/lib/varnish/$INSTANCE/varnish_storage.bin

#

# # Cache file size: in bytes, optionally using k / M / G / T suffix,

# # or in percentage of available disk space using the % suffix.

# VARNISH_STORAGE_SIZE=1G

#

# # File containing administration secret

# VARNISH_SECRET_FILE=/etc/varnish/secret

# 

# # Backend storage specification

# VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"

#

# # Default TTL used when the backend does not specify one

# VARNISH_TTL=120

#

# # DAEMON_OPTS is used by the init script.  If you add or remove options,
make

# # sure you update this section, too.

# DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \

#              -f ${VARNISH_VCL_CONF} \

#              -T
${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \

#              -t ${VARNISH_TTL} \

#              -w
${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \

#             -S ${VARNISH_SECRET_FILE} \

#              -s ${VARNISH_STORAGE}"

#

 

 

## Alternative 4, Do It Yourself

#

# DAEMON_OPTS=""

 

 

here is my default vcl

 

 

# This is a basic VCL configuration file for varnish.  See the vcl(7)

# man page for details on VCL syntax and semantics.

#

# TODO: Update internal subnet ACL and security.

# Define the internal network subnet.

# These are used below to allow internal access to certain files while not

# allowing access from the public internet.

# acl internal {

#  "192.10.0.0"/24;

# }

# Default backend definition.  Set this to point to your content

# server.

#

backend default {

  .host = "127.0.0.1";

  .port = "8080";

}

# Respond to incoming requests.

sub vcl_recv {

  # Use anonymous, cached pages if all backends are down.

  if (!req.backend.healthy) {

    unset req.http.Cookie;

  }

 

   

 

  # Allow the backend to serve up stale content if it is responding slowly.

  set req.grace = 6h;

  # Pipe these paths directly to Apache for streaming.

  #if (req.url ~ "^/admin/content/backup_migrate/export") {

  #  return (pipe);

  #}

  # Do not cache these paths.

  if (req.url ~ "^/status\.php$" ||

      req.url ~ "^/update\.php$" ||

      req.url ~ "^/admin$" ||

      req.url ~ "^/admin/.*$" ||

      req.url ~ "^/flag/.*$" ||

      req.url ~ "^.*/munin/.*$" ||

      req.url ~ "^.*/ajax/.*$" ||

      req.url ~ "^.*/api/.*$" ||

      req.url ~ "^.*/apidev/.*$" ||

      req.url ~ "^.*/serviioweb/.*$" ||

      req.url ~ "^.*/mediabrowser/.*$" ||

      req.url ~ "^.*/phpmyadmin/.*$" ||

      req.url ~ "^.*/ahah/.*$") {

       return (pass);

  }

  # Do not allow outside access to cron.php or install.php.

  #if (req.url ~ "^/(cron|install)\.php$" && !client.ip ~ internal) {

    # Have Varnish throw the error directly.

  #  error 404 "Page not found.";

    # Use a custom error page that you've defined in Drupal at the path
"404".

    # set req.url = "/404";

  #}

  # Always cache the following file types for all users. This list of
extensions

  # appears twice, once here and again in vcl_fetch so make sure you edit
both

  # and keep them equal.

  if (req.url ~
"(?i)\.(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)
(\?.*)?$") {

    unset req.http.Cookie;

  }

  # Remove all cookies that Drupal doesn't need to know about. We explicitly


  # list the ones that Drupal does need, the SESS and NO_CACHE. If, after 

  # running this code we find that either of these two cookies remains, we 

  # will pass as the page cannot be cached.

  if (req.http.Cookie) {

    # 1. Append a semi-colon to the front of the cookie string.

    # 2. Remove all spaces that appear after semi-colons.

    # 3. Match the cookies we want to keep, adding the space we removed 

    #    previously back. (\1) is first matching group in the regsuball.

    # 4. Remove all other cookies, identifying them by the fact that they
have

    #    no space after the preceding semi-colon.

    # 5. Remove all spaces and semi-colons from the beginning and end of the


    #    cookie string. 

    set req.http.Cookie = ";" + req.http.Cookie;

    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");    

    set req.http.Cookie = regsuball(req.http.Cookie,
";(SESS[a-z0-9]+|NO_CACHE)=", "; \1=");

    set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");

    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

    if (req.http.Cookie == "") {

      # If there are no remaining cookies, remove the cookie header. If
there

      # aren't any cookie headers, Varnish's default behavior will be to
cache

      # the page.

      unset req.http.Cookie;

    }

    else {

      # If there is any cookies left (a session or NO_CACHE cookie), do not

      # cache the page. Pass it on to Apache directly.

      return (pass);

    }

  }

}

# Set a header to track a cache HIT/MISS.

sub vcl_deliver {

  if (obj.hits > 0) {

    set resp.http.X-Varnish-Cache = "HIT";

  }

  else {

    set resp.http.X-Varnish-Cache = "MISS";

  }

}

# Code determining what to do when serving items from the Apache servers.

# beresp == Back-end response from the web server.

sub vcl_fetch {

  # We need this to cache 404s, 301s, 500s. Otherwise, depending on backend
but 

  # definitely in Drupal's case these responses are not cacheable by
default.

  if (beresp.status == 404 || beresp.status == 301 || beresp.status == 500)
{

    set beresp.ttl = 10m;

  }

    if(beresp.status == 302 && !beresp.http.Location){

    set beresp.status = 200;

    set beresp.response = "OK";

  }

 

  # Don't allow static files to set cookies. 

  # (?i) denotes case insensitive in PCRE (perl compatible regular
expressions).

  # This list of extensions appears twice, once here and again in vcl_recv
so 

  # make sure you edit both and keep them equal.

  if (req.url ~
"(?i)\.(pdf|asc|dat|txt|doc|xls|ppt|tgz|csv|png|gif|jpeg|jpg|ico|swf|css|js)
(\?.*)?$") {

    unset beresp.http.set-cookie;

  }

  # Allow items to be stale if needed.

  set beresp.grace = 6h;

}

# In the event of an error, show friendlier messages.

sub vcl_error {

  # Redirect to some other URL in the case of a homepage failure.

  #if (req.url ~ "^/?$") {

  #  set obj.status = 302;

  #  set obj.http.Location = "http://backup.example.com/";

  #}

  # Otherwise redirect to the homepage, which will likely be in the cache.

  set obj.http.Content-Type = "text/html; charset=utf-8";

  synthetic {"

<html>

<head>

  <title>Page Unavailable</title>

  <style>

    body { background: #303030; text-align: center; color: white; }

    #page { border: 1px solid #CCC; width: 500px; margin: 100px auto 0;
padding: 30px; background: #323232; }

    a, a:link, a:visited { color: #CCC; }

    .error { color: #222; }

  </style>

</head>

<body onload="setTimeout(function() { window.location = '/' }, 5000)">

  <div id="page">

    <h1 class="title">Page Unavailable</h1>

    <p>The page you requested is temporarily unavailable.</p>

    <p>We're redirecting you to the <a href="/">homepage</a> in 5
seconds.</p>

    <div class="error">(Error "} + obj.status + " " + obj.response +
{")</div>

  </div>

</body>

</html>

"};

  return (deliver);

}

#

# Below is a commented-out copy of the default VCL logic.  If you

# redefine any of these subroutines, the built-in logic will be

# appended to your code.

# sub vcl_recv {

#     if (req.restarts == 0) {

#   if (req.http.x-forwarded-for) {

 

 

Here is a copy of the waterfall chart from webpagetest

 



 

Here is the results from varnishstat -1

 

client_conn               7179         0.42 Client connections accepted

client_drop                  0         0.00 Connection dropped, no sess/wrk

client_req                7674         0.45 Client requests received

cache_hit                 5474         0.32 Cache hits

cache_hitpass                1         0.00 Cache hits for pass

cache_miss                1682         0.10 Cache misses

backend_conn              1150         0.07 Backend conn. success

backend_unhealthy            0         0.00 Backend conn. not attempted

backend_busy                 0         0.00 Backend conn. too many

backend_fail                 0         0.00 Backend conn. failures

backend_reuse             1049         0.06 Backend conn. reuses

backend_toolate            922         0.05 Backend conn. was closed

backend_recycle           1973         0.12 Backend conn. recycles

backend_retry                0         0.00 Backend conn. retry

fetch_head                   0         0.00 Fetch head

fetch_length              1880         0.11 Fetch with Length

fetch_chunked               85         0.00 Fetch chunked

fetch_eof                    0         0.00 Fetch EOF

fetch_bad                    0         0.00 Fetch had bad headers

fetch_close                227         0.01 Fetch wanted close

fetch_oldhttp                0         0.00 Fetch pre HTTP/1.1 closed

fetch_zero                   0         0.00 Fetch zero len

fetch_failed                 0         0.00 Fetch failed

fetch_1xx                    0         0.00 Fetch no body (1xx)

fetch_204                    0         0.00 Fetch no body (204)

fetch_304                    8         0.00 Fetch no body (304)

n_sess_mem                  26          .   N struct sess_mem

n_sess                       1          .   N struct sess

n_object                  1682          .   N struct object

n_vampireobject              0          .   N unresurrected objects

n_objectcore              1691          .   N struct objectcore

n_objecthead              1445          .   N struct objecthead

n_waitinglist               10          .   N struct waitinglist

n_vbc                        1          .   N struct vbc

n_wrk                       10          .   N worker threads

n_wrk_create                10         0.00 N worker threads created

n_wrk_failed                 0         0.00 N worker threads not created

n_wrk_max                    0         0.00 N worker threads limited

n_wrk_lqueue                 0         0.00 work request queue length

n_wrk_queued                 0         0.00 N queued work requests

n_wrk_drop                   0         0.00 N dropped work requests

n_backend                    1          .   N backends

n_expired                    0          .   N expired objects

n_lru_nuked                  0          .   N LRU nuked objects

n_lru_moved               5090          .   N LRU moved objects

losthdr                      0         0.00 HTTP header overflows

n_objsendfile                0         0.00 Objects sent with sendfile

n_objwrite                7411         0.43 Objects sent with write

n_objoverflow                0         0.00 Objects overflowing workspace

s_sess                    7179         0.42 Total Sessions

s_req                     7674         0.45 Total Requests

s_pipe                       0         0.00 Total pipe

s_pass                     518         0.03 Total pass

s_fetch                   2200         0.13 Total fetch

s_hdrbytes             4394386       257.09 Total header bytes

s_bodybytes          369487008     21616.28 Total body bytes

sess_closed                602         0.04 Session Closed

sess_pipeline                0         0.00 Session Pipeline

sess_readahead               0         0.00 Session Read Ahead

sess_linger               7117         0.42 Session Linger

sess_herd                 7562         0.44 Session herd

shm_records             488273        28.57 SHM records

shm_writes               53496         3.13 SHM writes

shm_flushes                  0         0.00 SHM flushes due to overflow

shm_cont                    19         0.00 SHM MTX contention

shm_cycles                   0         0.00 SHM cycles through buffer

sms_nreq                     0         0.00 SMS allocator requests

sms_nobj                     0          .   SMS outstanding allocations

sms_nbytes                   0          .   SMS outstanding bytes

sms_balloc                   0          .   SMS bytes allocated

sms_bfree                    0          .   SMS bytes freed

backend_req               2200         0.13 Backend requests made

n_vcl                        1         0.00 N vcl total

n_vcl_avail                  1         0.00 N vcl available

n_vcl_discard                0         0.00 N vcl discarded

n_ban                        1          .   N total active bans

n_ban_gone                   1          .   N total gone bans

n_ban_add                    1         0.00 N new bans added

n_ban_retire                 0         0.00 N old bans deleted

n_ban_obj_test               0         0.00 N objects tested

n_ban_re_test                0         0.00 N regexps tested against

n_ban_dups                   0         0.00 N duplicate bans removed

hcb_nolock                7155         0.42 HCB Lookups without lock

hcb_lock                  1436         0.08 HCB Lookups with lock

hcb_insert                1436         0.08 HCB Inserts

esi_errors                   0         0.00 ESI parse errors (unlock)

esi_warnings                 0         0.00 ESI parse warnings (unlock)

accept_fail                  0         0.00 Accept failures

client_drop_late             0         0.00 Connection dropped late

uptime                   17093         1.00 Client uptime

dir_dns_lookups              0         0.00 DNS director lookups

dir_dns_failed               0         0.00 DNS director failed lookups

dir_dns_hit                  0         0.00 DNS director cached lookups hit

dir_dns_cache_full           0         0.00 DNS director full dnscache

vmods                        0          .   Loaded VMODs

n_gzip                       0         0.00 Gzip operations

n_gunzip                  6757         0.40 Gunzip operations

LCK.sms.creat                1         0.00 Created locks

LCK.sms.destroy              0         0.00 Destroyed locks

LCK.sms.locks                0         0.00 Lock Operations

LCK.sms.colls                0         0.00 Collisions

LCK.smp.creat                0         0.00 Created locks

LCK.smp.destroy              0         0.00 Destroyed locks

LCK.smp.locks                0         0.00 Lock Operations

LCK.smp.colls                0         0.00 Collisions

LCK.sma.creat                2         0.00 Created locks

LCK.sma.destroy              0         0.00 Destroyed locks

LCK.sma.locks             7470         0.44 Lock Operations

LCK.sma.colls                0         0.00 Collisions

LCK.smf.creat                0         0.00 Created locks

LCK.smf.destroy              0         0.00 Destroyed locks

LCK.smf.locks                0         0.00 Lock Operations

LCK.smf.colls                0         0.00 Collisions

LCK.hsl.creat                0         0.00 Created locks

LCK.hsl.destroy              0         0.00 Destroyed locks

LCK.hsl.locks                0         0.00 Lock Operations

LCK.hsl.colls                0         0.00 Collisions

LCK.hcb.creat                1         0.00 Created locks

LCK.hcb.destroy              0         0.00 Destroyed locks

LCK.hcb.locks             1531         0.09 Lock Operations

LCK.hcb.colls                0         0.00 Collisions

LCK.hcl.creat                0         0.00 Created locks

LCK.hcl.destroy              0         0.00 Destroyed locks

LCK.hcl.locks                0         0.00 Lock Operations

LCK.hcl.colls                0         0.00 Collisions

LCK.vcl.creat                1         0.00 Created locks

LCK.vcl.destroy              0         0.00 Destroyed locks

LCK.vcl.locks               12         0.00 Lock Operations

LCK.vcl.colls                0         0.00 Collisions

LCK.stat.creat               1         0.00 Created locks

LCK.stat.destroy             0         0.00 Destroyed locks

LCK.stat.locks            7204         0.42 Lock Operations

LCK.stat.colls               0         0.00 Collisions

LCK.sessmem.creat            1         0.00 Created locks

LCK.sessmem.destroy            0         0.00 Destroyed locks

LCK.sessmem.locks           7547         0.44 Lock Operations

LCK.sessmem.colls              0         0.00 Collisions

LCK.wstat.creat                1         0.00 Created locks

LCK.wstat.destroy              0         0.00 Destroyed locks

LCK.wstat.locks            34286         2.01 Lock Operations

LCK.wstat.colls                0         0.00 Collisions

LCK.herder.creat               1         0.00 Created locks

LCK.herder.destroy             0         0.00 Destroyed locks

LCK.herder.locks               1         0.00 Lock Operations

LCK.herder.colls               0         0.00 Collisions

LCK.wq.creat                   2         0.00 Created locks

LCK.wq.destroy                 0         0.00 Destroyed locks

LCK.wq.locks               50522         2.96 Lock Operations

LCK.wq.colls                   0         0.00 Collisions

LCK.objhdr.creat            1445         0.08 Created locks

LCK.objhdr.destroy             0         0.00 Destroyed locks

LCK.objhdr.locks           28872         1.69 Lock Operations

LCK.objhdr.colls               0         0.00 Collisions

LCK.exp.creat                  1         0.00 Created locks

LCK.exp.destroy                0         0.00 Destroyed locks

LCK.exp.locks              18773         1.10 Lock Operations

LCK.exp.colls                  0         0.00 Collisions

LCK.lru.creat                  2         0.00 Created locks

LCK.lru.destroy                0         0.00 Destroyed locks

LCK.lru.locks               1682         0.10 Lock Operations

LCK.lru.colls                  0         0.00 Collisions

LCK.cli.creat                  1         0.00 Created locks

LCK.cli.destroy                0         0.00 Destroyed locks

LCK.cli.locks               5707         0.33 Lock Operations

LCK.cli.colls                  0         0.00 Collisions

LCK.ban.creat                  1         0.00 Created locks

LCK.ban.destroy                0         0.00 Destroyed locks

LCK.ban.locks              18775         1.10 Lock Operations

LCK.ban.colls                  0         0.00 Collisions

LCK.vbp.creat                  1         0.00 Created locks

LCK.vbp.destroy                0         0.00 Destroyed locks

LCK.vbp.locks                  0         0.00 Lock Operations

LCK.vbp.colls                  0         0.00 Collisions

LCK.vbe.creat                  1         0.00 Created locks

LCK.vbe.destroy                0         0.00 Destroyed locks

LCK.vbe.locks               2299         0.13 Lock Operations

LCK.vbe.colls                  0         0.00 Collisions

LCK.backend.creat              1         0.00 Created locks

LCK.backend.destroy            0         0.00 Destroyed locks

LCK.backend.locks           7394         0.43 Lock Operations

LCK.backend.colls              0         0.00 Collisions

SMA.s0.c_req                3364         0.20 Allocator requests

SMA.s0.c_fail                  0         0.00 Allocator failures

SMA.s0.c_bytes         215415613     12602.56 Bytes allocated

SMA.s0.c_freed         204713824     11976.47 Bytes freed

SMA.s0.g_alloc              3233          .   Allocations outstanding

SMA.s0.g_bytes          10701789          .   Bytes outstanding

SMA.s0.g_space         123515939          .   Bytes available

SMA.Transient.c_req         1028         0.06 Allocator requests

SMA.Transient.c_fail           0         0.00 Allocator failures

SMA.Transient.c_bytes     42146510      2465.72 Bytes allocated

SMA.Transient.c_freed     42146510      2465.72 Bytes freed

SMA.Transient.g_alloc            0          .   Allocations outstanding

SMA.Transient.g_bytes            0          .   Bytes outstanding

SMA.Transient.g_space            0          .   Bytes available

VBE.default(127.0.0.1,,8080).vcls            1          .   VCL references

VBE.default(127.0.0.1,,8080).happy           0          .   Happy health
probes

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120927/d14b3964/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 47022 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120927/d14b3964/attachment.png>

From perbu at varnish-software.com  Fri Sep 28 13:46:23 2012
From: perbu at varnish-software.com (Per Buer)
Date: Fri, 28 Sep 2012 15:46:23 +0200
Subject: memory issue when running varnishncsa
In-Reply-To: <CAOCtP1d82eFiUXSksvZVMcTiDRANMhNd7hzL_q3n2DHu9=60zg@mail.gmail.com>
References: <CAOCtP1d82eFiUXSksvZVMcTiDRANMhNd7hzL_q3n2DHu9=60zg@mail.gmail.com>
Message-ID: <CAOXZevDtzxUyxKAouAy2bur5TvbMGcq6xctzM5eU9pT6aov5Vw@mail.gmail.com>

Hi,

On Thu, Sep 27, 2012 at 5:47 PM, Zheng Liu <huozhe at gmail.com> wrote:

> Hi,
>
> I tried to use varnishncsa to generate log on disk, so I can have another
> daemon tail and analyze it. However, when I enable it, I see varnishd
> starts to use more and more memory. I am using varnish-3.0.3 on Ubuntu
> 11.10. The box has 190G and I configured varnish to use 160G. Its footprint
> has been stable at 180G for months. Then after I started varnishncsa,
> within 5 hours, all the memory was gone and it starts to use swap. I wonder
> if this is expected behavior, and we are suppose to allow more head room
> during logging with varnishncsa?
>

There have been some reports of a varnishleak in varnishncsa - but in the
cases I've seen it's varnishncsa  itself which balloons up. Would that be
the case here aswell?

-- 
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120928/6efaaf95/attachment.html>

From huozhe at gmail.com  Fri Sep 28 15:31:26 2012
From: huozhe at gmail.com (Zheng Liu)
Date: Fri, 28 Sep 2012 08:31:26 -0700
Subject: memory issue when running varnishncsa
In-Reply-To: <CAOXZevDtzxUyxKAouAy2bur5TvbMGcq6xctzM5eU9pT6aov5Vw@mail.gmail.com>
References: <CAOCtP1d82eFiUXSksvZVMcTiDRANMhNd7hzL_q3n2DHu9=60zg@mail.gmail.com>
 <CAOXZevDtzxUyxKAouAy2bur5TvbMGcq6xctzM5eU9pT6aov5Vw@mail.gmail.com>
Message-ID: <CAOCtP1fT6MvuTBKQvHUOi44UNzzoHoVNp688PYpUx4=FaGn_Hw@mail.gmail.com>

Per,

Thanks for the info. I remember I was seeing the ram usage by varnishncsa
did not change a lot. I didn't take any screen shot, so I may remember
wrong. Let me try it one more time and see what happened. Could it be the
log dumping to disk can't keep up with the insertion rate, so varnish has
to allocate more for the in-memory logging? Just a guess.

thanks,
Zheng

On Fri, Sep 28, 2012 at 6:46 AM, Per Buer <perbu at varnish-software.com>wrote:

> Hi,
>
> On Thu, Sep 27, 2012 at 5:47 PM, Zheng Liu <huozhe at gmail.com> wrote:
>
>> Hi,
>>
>> I tried to use varnishncsa to generate log on disk, so I can have another
>> daemon tail and analyze it. However, when I enable it, I see varnishd
>> starts to use more and more memory. I am using varnish-3.0.3 on Ubuntu
>> 11.10. The box has 190G and I configured varnish to use 160G. Its footprint
>> has been stable at 180G for months. Then after I started varnishncsa,
>> within 5 hours, all the memory was gone and it starts to use swap. I wonder
>> if this is expected behavior, and we are suppose to allow more head room
>> during logging with varnishncsa?
>>
>
> There have been some reports of a varnishleak in varnishncsa - but in the
> cases I've seen it's varnishncsa  itself which balloons up. Would that be
> the case here aswell?
>
> --
> Per Buer
> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
> *Varnish makes websites fly!*
> Whitepapers <http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
> Twitter <https://twitter.com/varnishsoftware>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120928/9e6f9988/attachment.html>

From marco at marcocorte.it  Fri Sep 28 17:00:28 2012
From: marco at marcocorte.it (Marco Corte)
Date: Fri, 28 Sep 2012 19:00:28 +0200
Subject: not sure if Varnish is working correctly
In-Reply-To: <000f01cd9d07$6b8c2150$42a463f0$@roadrunner.com>
References: <000f01cd9d07$6b8c2150$42a463f0$@roadrunner.com>
Message-ID: <5065D7AC.1010908@marcocorte.it>

Hi, Mark

I am not an experienced varnish user, but I can tell you anyway how I 
would dig into the issue.

- Varnishhist
I would check what "varnishhist" reports in real time during normal site 
access to see if varnish is serving cached and non cached objects in 
reasonable times: I like it because I can "feel" if everything is 
working properly or not.

- Regexp
IMHO this does not affects performances, but I would anyway rewrite some 
regexp, for example:

    req.url ~ "^.*/ajax/.*$" ||

should match the same as

    req.url ~ "/ajax/" ||

which I find easier to read and to mantain.

- varnishstat
I started from "Some values to care about" in 
http://kly.no/posts/2009_12_08__Varnishstat_for_dummies__.html

No other ideas at this point.

.marcoc



From newnewxu at gmail.com  Wed Sep 12 16:47:29 2012
From: newnewxu at gmail.com (xu jiajun)
Date: Wed, 12 Sep 2012 16:47:29 -0000
Subject: no benefits with varnish?
Message-ID: <CAC4Snytv4YMzZh8YdrCdh5TJZ9Tg+36Q9i6vqO-X6So54xGC1A@mail.gmail.com>

Hi all,
I am using varnish with Drupal 7. I find that when I use ab to access
some small files, around 10KB, varnish could bring benefits by
increasing the [Requests per second] from 200 to 600. But when I use
ab to access som files bigger than that, it shows no benefits. Is
there any special option I should set for bigger files?

Thanks a lot in advance.

My varnish config file as below:
######acl internal {
  "192.168.98.0"/24;
}

backend web1 { .host = "192.168.98.24"; .port = "80"; .connect_timeout
= 600s; .first_byte_timeout = 600s; .between_bytes_timeout = 600s;}

director default_director round-robin {
  { .backend = web1; }
}


sub vcl_recv {
   set req.backend = default_director;

  # Use anonymous, cached pages if all backends are down.
  if (!req.backend.healthy) {
    unset req.http.Cookie;
  }

  # Allow the backend to serve up stale content if it is responding slowly.
  set req.grace = 6h;

  # Do not cache these paths.
  if (req.url ~ "^/status\.php$" ||
      req.url ~ "^/update\.php$" ||
      req.url ~ "^/ooyala/ping$" ||
      req.url ~ "^/admin/build/features" ||
      req.url ~ "^/info/.*$" ||
      req.url ~ "^/flag/.*$" ||
      req.url ~ "^.*/ajax/.*$" ||
      req.url ~ "^.*/ahah/.*$") {
       return (pass);
  }

  # Pipe these paths directly to Apache for streaming.
  if (req.url ~ "^/admin/content/backup_migrate/export") {
    return (pipe);
  }

  # Do not allow outside access to cron.php or install.php.
  if (req.url ~ "^/(cron|install)\.php$" && !client.ip ~ internal) {
    # Have Varnish throw the error directly.
    error 404 "Page not found.";
    # Use a custom error page that you've defined in Drupal at the path "404".
    # set req.url = "/404";
  }

  # Handle compression correctly. Different browsers send different
  # "Accept-Encoding" headers, even though they mostly all support the same
  # compression mechanisms. By consolidating these compression headers into
  # a consistent format, we can reduce the size of the cache and get more hits.=
  # @see: http:// varnish.projects.linpro.no/wiki/FAQ/Compression
  if (req.http.Accept-Encoding) {
    if (req.http.Accept-Encoding ~ "gzip") {
      # If the browser supports it, we'll use gzip.
      set req.http.Accept-Encoding = "gzip";
    }
    else if (req.http.Accept-Encoding ~ "deflate") {
      # Next, try deflate if it is supported.
      set req.http.Accept-Encoding = "deflate";
    }
    else {
      # Unknown algorithm. Remove it and send unencoded.
      unset req.http.Accept-Encoding;
    }
  }

  # Always cache the following file types for all users.
  if (req.url ~
"(?i)\.(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$") {
    unset req.http.Cookie;
  }

  # Remove all cookies that Drupal doesn't need to know about. ANY remaining
  # cookie will cause the request to pass-through to Apache. For the most part
  # we always set the NO_CACHE cookie after any POST request, disabling the
  # Varnish cache temporarily. The session cookie allows all authenticated users
  # to pass through as long as they're logged in.
  if (req.http.Cookie) {
    set req.http.Cookie = ";" + req.http.Cookie;
    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
    set req.http.Cookie = regsuball(req.http.Cookie,
";(SESS[a-z0-9]+|NO_CACHE)=", "; \1=");
    set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

    if (req.http.Cookie == "") {
      # If there are no remaining cookies, remove the cookie header. If there
      # aren't any cookie headers, Varnish's default behavior will be to cache
      # the page.
      unset req.http.Cookie;
    }
    else {
      # If there is any cookies left (a session or NO_CACHE cookie), do not
      # cache the page. Pass it on to Apache directly.
      return (pass);
    }
  }
}

sub vcl_hash {
  # Include cookie in cache hash.
  # This check is unnecessary because we already pass on all cookies.
  # if (req.http.Cookie) {
  #   set req.hash += req.http.Cookie;
  # }
}

sub vcl_fetch {
  # Don't allow static files to set cookies.
  if (req.url ~
"(?i)\.(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$") {
    # beresp == Back-end response from the web server.
    unset beresp.http.set-cookie;
  }

  # Allow items to be stale if needed.
  set beresp.grace = 6h;
}

sub vcl_error {
  # Redirect to some other URL in the case of a homepage failure.
  #if (req.url ~ "^/?$") {
  #  set obj.status = 302;
  #  set obj.http.Location = "http://backup.example.com/";
  #}

  # Otherwise redirect to the homepage, which will likely be in the cache.
  set obj.http.Content-Type = "text/html; charset=utf-8";
  synthetic {"
<html>
<head>
  <title>Page Unavailable</title>
  <style>
    body { background: #303030; text-align: center; color: white; }
    #page { border: 1px solid #CCC; width: 500px; margin: 100px auto
0; padding: 30px; background: #323232; }
    a, a:link, a:visited { color: #CCC; }
    .error { color: #222; }
  </style>
</head>
<body onload="setTimeout(function() { window.location = '/' }, 5000)">
  <div id="page">
    <h1 class="title">Page Unavailable</h1>
    <p>The page you requested is temporarily unavailable.</p>
    <p>We're redirecting you to the <a href="/">homepage</a> in 5 seconds.</p>
    <div class="error">(Error "} + obj.status + " " + obj.response + {")</div>
  </div>
</body>
</html>
"};
  return (deliver);
}

sub vcl_deliver {
  if (obj.hits > 0) {
    set resp.http.X-Varnish-Cache = "HIT" + obj.hits;
  }
  else {
    set resp.http.X-Varnish-Cache = "MISS";
  }
}
#######


-- 
Best Regards,
Jiajun



From kylem at cavecreek.net  Thu Sep 20 06:59:30 2012
From: kylem at cavecreek.net (Kyle Morgan)
Date: Thu, 20 Sep 2012 06:59:30 +0000
Subject: Cookie Normalization
Message-ID: <5FCA0B4D0B956A43BC91305B07A90DEDD73029@exch10-mb2.ccbill-hq.local>

I'm writing in reference to the blog post: https://www.varnish-software.com/blog/validating-cookies-varnish. (Originally posted to varnish-dev, and was redirected here, apologies in advance)

We've been attempting to configure content caching for a paid members area on a client's site. The content does not change per member, but our implementation is creating separate caches for each authenticated user.

Is there a way to normalize the cookie for authenticated users so that only 1 cache is created for all members? So far, our trial/error allows anyone (authenticated or not) to view the cached contents once any content from the members area is cached. Below is the relevant contents of .vcl:

   # Piped Directly to the webserver
   if (req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$") {
        # return (pipe);
        unset req.http.cookie;
   }

   # if labeled fja or auth pass through to backend
   if (req.http.Cookie ~ "(auth|fja)") { return (pipe); }

   # if labeled userdata or concept pass to cache
   if (req.http.Cookie ~ "(userdata|concept)") { return (lookup); }

   #KyleM bypass cookies from #######
   if (req.http.host ~ "^messages\.#########\.com$") { return (pass); }

   set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *__utm.=[^;]+;? *", "");
   set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *__atu.=[^;]+;? *", "");
   set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *OA.*=[^;]+;? *", "");
   #set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *fja=[^;]+;? *", "fja=member");

   if (req.http.cookie ~ "^ *$") {
        remove req.http.cookie;
   }

   return(lookup);

}

sub vcl_hash {    hash_data(req.http.cookie);   }

sub vcl_pass {

    if (req.http.Authorization) {
        return(pass);
    }
}

sub vcl_fetch {
    unset beresp.http.Server;
    set beresp.http.Server = "#####";
    set beresp.grace = 30s;

Any help is greatly appreciated.

Thanks!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20120920/587994a7/attachment.html>