From dridi.boukelmoune at zenika.com  Tue Jul  1 09:10:24 2014
From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune)
Date: Tue, 1 Jul 2014 11:10:24 +0200
Subject: RX > TX, not storing files in cache as long as it should
In-Reply-To: <53B11DAA.6070906@imer.cc>
References: <53B11DAA.6070906@imer.cc>
Message-ID: <CABtDKm6eVnub4B73nq3JrAjV2FPROQqvgzoH_+pCqb5ViAS9VA@mail.gmail.com>

Hi,

On Mon, Jun 30, 2014 at 10:19 AM, Robin <imer at imer.cc> wrote:
> Hello,
> I recently set up a varnish cache to serve as an edge server to take some
> traffic load off our main server.
> What it's supposed to do is cache ~3GB (for now, will expand later) of
> static files which never change (new files will get a new url).
> This seems to work mostly ok, looking at the interface statistics after 2
> days of runtime it turns out it's actually recieving more data than its
> sending.
> One of our nodes shows (for example): RX bytes:119388151300 (111.1 GiB)  TX
> bytes:60809550709 (56.6 GiB)

This one will definitely not fit in your cache. Output from varnishlog
for those transactions might help figure what's going on. Varnish
might send it gzipped, which could explain the size difference but
that wouldn't make sense if this file is already compressed.

> The static files range from a few MB to (the biggest) 1.2GB, usually are
> around max. 100MB though.
> They are originally served using nginx with the "expires max" directive set,
> this is reflected by the varnish cache too:
> http://puu.sh/9Qq3A/6c934ad71c.png

According to this image, you have a cache miss.

> As you can see though, the Age is 0 - which means (from what I understand)
> the file didn't exist yet and is just being downloaded by varnish - which is
> not true.
> The file has been downloaded by that node multiple times, each time with a
> varying content length: http://puu.sh/9Qqds/506e73e589.png

Could it be because you generate the rar files on the fly in a
non-deterministic fashion? Although even with that size variations are
disturbing.

> My .vcl looks like this:
>>
>> backend default {
>>     .host = "backend ip";
>>     .port = "80";
>> }
>> sub vcl_fetch {
>>     if (beresp.ttl <= 0s ||
>>         beresp.http.Set-Cookie ||
>>         beresp.http.Vary == "*") {
>>                 /*
>>                  * Mark as "Hit-For-Pass" for the next 2 minutes
>>                  */
>>                 set beresp.ttl = 120 s;
>>                 return (hit_for_pass);
>>     }
>>     set beresp.do_stream = true;
>>     return (deliver);
>> }
>
> And the options passed to varnish are:
>>
>> DAEMON_OPTS="-a :80 \
>>              -T localhost:6082 \
>>              -f /etc/varnish/default.vcl \
>>              -S /etc/varnish/secret \
>>              -p send_timeout=86400 \
>>              -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,7G"

Which version of varnish are you using?

> Any idea why this is happening?
>>
>> n_lru_nuked                  0          . N LRU nuked objects
>
> Means the cache isnt filled up, which it shouldnt be anyways - so why is it
> downloading it over and over again and why is it aborting prematurely and
> not downloading the full file to cache?

Do you receive cookies? If those files are static (or semi-static) and
need to be cached, you need to either fix your backend to make sure
you set cookies only on relevant paths, or prune/ignore the irrelevant
cookies in vcl_recv.

> Trying to download the file always results in working files, so does it
> cancel the download to varnish if the user aborts? If so this is really
> disadvantageous for us, cause it may cause a large file to be downloaded
> multiple times before finally ending up in the cache
>
> Hope someone can help me out
>
> - Robin
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From dridi.boukelmoune at zenika.com  Tue Jul  1 09:15:21 2014
From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune)
Date: Tue, 1 Jul 2014 11:15:21 +0200
Subject: backend availability
In-Reply-To: <BLU436-SMTP155DF80A3F01ACA4CD7D1C7A0050@phx.gbl>
References: <CAOZy0emhWptDrHDG_Rk5J1P6XmgCJYN7-pQ61hKX7o0SAwMKTA@mail.gmail.com>
 <BLU436-SMTP155DF80A3F01ACA4CD7D1C7A0050@phx.gbl>
Message-ID: <CABtDKm6miSt4RaAFAmAy=Vcq4fHXENYdtYafdvhmTXiPQpDHBQ@mail.gmail.com>

I'd go for varnish 4, it can nest your round-robin directors inside a
fallback director.

You can maybe do that with varnish 3 by checking the health of your
activepool director.

Dridi

On Sun, Jun 29, 2014 at 1:04 PM, Ashish Nepal <aashisn at hotmail.com> wrote:
> Hi all,
>
> Is there any work around so that i could say -> minimum set of backendserver
> has to be x if not available add from reserve pool.
>
> e.g. -> i have 5 web servers -> web1,2,3,4,5
>             i have reserve pool of 5 web servers -> web6,7,8,9,10
>
>
> director activepool round-robin {
>     { .backend = web1; }
>     { .backend = web2; }
>     { .backend = web3; }
>     { .backend = web4; }
>     { .backend = web5; }
> }
>
> director reservepool round-robin {
>     { .backend = web6; }
>     { .backend = web7; }
>     { .backend = web8; }
>     { .backend = web9; }
>     { .backend = web10; }
> }
>
> and if web1 is down or web1/2 is down depends upon setting of what is
> minimum set of boxes required. i would like director to pull one of the box
> from reservepool.
>
> or, should this be worth having such feature on backendpolling?
>
>
>
> Regards
> Ashish
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From phk at phk.freebsd.dk  Thu Jul  3 06:09:26 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 03 Jul 2014 06:09:26 +0000
Subject: [oss-security] Varnish - no CVE == bug regression
In-Reply-To: <20140702234503.GA11203@openwall.com>
References: <20140702165701.GA2493@dft-labs.eu>
 <20140702234503.GA11203@openwall.com>
Message-ID: <79047.1404367766@critter.freebsd.dk>

>> Latest version of Varnish cache (4.0.1 https://www.varnish-cache.org/ ) has 
>> the same DoS vulnerability that 3.x had (which was subsequently fixed in
>> that branch). 

Official response of the Varnish Project:
-----------------------------------------

It is of course a mistake to have such a regressions and we'll fix
that (and any other relevant bugs we become aware of).

But this is not a DoS vulnerability, and a CVE is not warranted.

Explanation:
------------

Varnish is a server side cache, which speeds up delivery of HTTP
objects coming from one or more backend HTTP servers (typically
apache, ngnix etc.)

By definition Varnish must explicitly and implicitly trust the
backend HTTP server -- it is the only source of authority it has over
the HTTP content.

Therefore, if your backend is compromised, your Varnish will
faithfully serve whatever bogus contents the attackers put on your
homepage, so I really don't think that the attackers being able to
force an automatic restart of the varnish in front of it, is what
you should be worried about.

Once you fix your backend, Varnish will work the same as always.


Even deeper explanation:
------------------------

Notice that all the reported issues causes assert failures ?

About 10% of Varnish source code are asserts in one form or another,
to make sure that Varnish does not operate on bogus data or violate
invariants.

There are many error situations so pathlogical that an assert is
the only relevant error handling.  Adding a lot of code to handle
an obscure error condition gracefully, means that you have a lot
of code which never gets run and therefore may contain *real* bugs
or vulnerabilities.

Our goal is that you should never be able to cause an assert from
the client side -- that we would consider a DoS attack -- and
eventually we may set the same goal for the backend side.

But given that we trust the backend ultimately, and that varnish
and the backend are both controlled by the same HTTP content
owner, that is not a particular high priority for us:  If your
backend is that screwed, it doesn't really matter what Varnish does
anyway.

Also notice, that Varnish is designed to automatically restart after
an assert, so very little, if any service disruption takes place
as a result of these asserts.  These automatic restarts can be
disabled if you prefer.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From phk at phk.freebsd.dk  Thu Jul  3 07:42:37 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 03 Jul 2014 07:42:37 +0000
Subject: [oss-security] Varnish - no CVE == bug regression
In-Reply-To: <CAOurorZCjmrrw0MPhca=8+qjLKofrhdHsJuee5_=rCBv87SPbg@mail.gmail.com>
References: <20140702165701.GA2493@dft-labs.eu>
 <20140702234503.GA11203@openwall.com> <79047.1404367766@critter.freebsd.dk>
 <CAOurorZCjmrrw0MPhca=8+qjLKofrhdHsJuee5_=rCBv87SPbg@mail.gmail.com>
Message-ID: <79520.1404373357@critter.freebsd.dk>

In message <CAOurorZCjmrrw0MPhca=8+qjLKofrhdHsJuee5_=rCBv87SPbg at mail.gmail.com>, Marek Kroemeke writes:

>I'm not entirely convinced that there is a trust relationship between the
>cache and the backend in every single use case. 

It may not be total trust, but trust there is:  On party delivers
the other partys web-property.

But as I said:  We will fix bugs, but we don't consider them DoS vulns.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From cdgraff at gmail.com  Thu Jul  3 17:55:06 2014
From: cdgraff at gmail.com (Alejandro)
Date: Thu, 3 Jul 2014 14:55:06 -0300
Subject: How handle high number of redirects urls
Message-ID: <CALJtzxBF1OR3WwVCmnLV_B9Ph+41kciLtfRXGASwz_=7gRPj5A@mail.gmail.com>

Hi Guys,

We have varnish in front to all our sites... we are moving from one CMS to
other, and all our urls will change...

>From bussiness need, we need provide some way to mapping old urls to new...
the issue is will be near to 300 urls.

How is the best way to handle with this request from varnish perspective?

Have some good vmod for redirect we are using varnish 3.

thanks
Alejandro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140703/834c49e3/attachment.html>

From phk at phk.freebsd.dk  Thu Jul  3 18:23:33 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 03 Jul 2014 18:23:33 +0000
Subject: How handle high number of redirects urls
In-Reply-To: <CALJtzxBF1OR3WwVCmnLV_B9Ph+41kciLtfRXGASwz_=7gRPj5A@mail.gmail.com>
References: <CALJtzxBF1OR3WwVCmnLV_B9Ph+41kciLtfRXGASwz_=7gRPj5A@mail.gmail.com>
Message-ID: <18348.1404411813@critter.freebsd.dk>

In message <CALJtzxBF1OR3WwVCmnLV_B9Ph+41kciLtfRXGASwz_=7gRPj5A at mail.gmail.com>, Alejandro writes:

>We have varnish in front to all our sites... we are moving from one CMS to
>other, and all our urls will change...
>
>>From bussiness need, we need provide some way to mapping old urls to new...
>the issue is will be near to 300 urls.

300 entirely different urls or are there any system to the mapping ?

VCL is compiled to C and on to machine-code, so even if you took
your list of old->new mappings and machine-generated a long VCL
function like;

	if (req.url == "/foo/bar") {
		set req.url = "/newfoo/barnew";
	} else if (req.url == "snafu") {
		set req.url = "/fusnanew") {
	} ...

you will probably not feel any slowdown at all for 300 check as long
as you use "==" comparisons.

If you use regular expresions it might hurt your performance a little
bit, but probably less than you think.

In both cases you can use varnishtop to find the most common urls
and put those near the top, to maximize speed on average.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From tobias.eichelbroenner at lamp-solutions.de  Thu Jul  3 18:35:01 2014
From: tobias.eichelbroenner at lamp-solutions.de (=?ISO-8859-1?Q?Tobias_Eichelbr=F6nner?=)
Date: Thu, 03 Jul 2014 20:35:01 +0200
Subject: How handle high number of redirects urls
In-Reply-To: <18348.1404411813@critter.freebsd.dk>
References: <CALJtzxBF1OR3WwVCmnLV_B9Ph+41kciLtfRXGASwz_=7gRPj5A@mail.gmail.com>
 <18348.1404411813@critter.freebsd.dk>
Message-ID: <53B5A255.7030202@lamp-solutions.de>

Hi,

> 	if (req.url == "/foo/bar") {
> 		set req.url = "/newfoo/barnew";
> 	} else if (req.url == "snafu") {
> 		set req.url = "/fusnanew") {
> 	} ...

https://www.varnish-cache.org/vmod/database-driven-rewrites

Is marked as under develop. I havn?t tested it.
You can use

https://github.com/brandonwamboldt/libvmod-redis

Then you could store the rules in redis in modify the rule above as
set req.url = redis.call("GET " + req.url);

or something similar.

Sincerely,


-- 
LAMP solutions GmbH
Gostenhofer Hauptstrasse 35
90443 Nuernberg

Amtsgericht Nuernberg: HRB 22366
Geschaeftsfuehrer: Heiko Schubert

Es gelten unsere allgemeinen Geschaeftsbedingungen.
http://www.lamp-solutions.de/agbs/

Telefon      : 0911 / 376 516 0
Fax          : 0911 / 376 516 11
E-Mail       : support at lamp-solutions.de
Web          : www.lamp-solutions.de
Facebook     : http://www.facebook.com/LAMPsolutions
Twitter      : http://twitter.com/#!/lampsolutions



From machuidel at gmail.com  Fri Jul  4 14:12:22 2014
From: machuidel at gmail.com (Mike Machuidel)
Date: Fri, 4 Jul 2014 16:12:22 +0200
Subject: Support for streaming objects included with ESI
Message-ID: <CAKAE+jmCWkTWBrkRiFpicDtDJpNkxvvOkE0A+VuL-3Wbbg_g2g@mail.gmail.com>

Hello Everyone,

My web-framework has support for streaming all content to the client as
soon as it gets available. Also I make heavy use of ESI. Today I upgraded
to Varnish 4 to get the "Full support for streaming objects through from
the backend on a cache miss.". I assumed that meant it is was able to
support streaming ESI as well (processing ESI in series *without* buffering
or processing ESI in parallel *with* buffering). Now I noticed as soon as I
enable ESI it disables streaming.

It is possible to enable streaming for ESI objects as well? Should I maybe
use another branch?

Thanks,
Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140704/0460923f/attachment.html>

From phk at phk.freebsd.dk  Fri Jul  4 15:59:58 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 04 Jul 2014 15:59:58 +0000
Subject: Support for streaming objects included with ESI
In-Reply-To: <CAKAE+jmCWkTWBrkRiFpicDtDJpNkxvvOkE0A+VuL-3Wbbg_g2g@mail.gmail.com>
References: <CAKAE+jmCWkTWBrkRiFpicDtDJpNkxvvOkE0A+VuL-3Wbbg_g2g@mail.gmail.com>
Message-ID: <2889.1404489598@critter.freebsd.dk>

In message <CAKAE+jmCWkTWBrkRiFpicDtDJpNkxvvOkE0A+VuL-3Wbbg_g2g at mail.gmail.com>
, Mike Machuidel writes:

At this point streaming only pertains to esi:included elements not
subject to ESI processing themselves.

We're working on a full streaming version, but that probably wont
be until 4.1

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From phk at phk.freebsd.dk  Sat Jul  5 14:36:52 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 05 Jul 2014 14:36:52 +0000
Subject: [oss-security] Varnish - no CVE == bug regression
In-Reply-To: <79520.1404373357@critter.freebsd.dk>
References: <20140702165701.GA2493@dft-labs.eu>
 <20140702234503.GA11203@openwall.com> <79047.1404367766@critter.freebsd.dk>
 <CAOurorZCjmrrw0MPhca=8+qjLKofrhdHsJuee5_=rCBv87SPbg@mail.gmail.com>
 <79520.1404373357@critter.freebsd.dk>
Message-ID: <44356.1404571012@critter.freebsd.dk>

I have just read the followup discussion and will add these comments:

First of all, if you want an overview of the security design of
Varnish, it is here:

	https://www.varnish-cache.org/docs/trunk/phk/barriers.html

Second, since Varnish serves HTTP, a DoS is not something out of
the ordinary.  It happens all the time to our users, we consider
DoS attacks a fact of life.  The better we handle them, the better we
handle them, but we will never be able to cope with them all,
not in a world where Evil botnets or Good authors can point millions
of browsers at the same web property in an instant.

Third, with respect to "never trusting input":  Varnish doesn't.
But in some cases we distust with an assert.  Either because it is
an utterly pathological situation where no sane handling or recovery
is possible or because the condition is so rare that our time is
better spent improving quality and error handling elsewhere.

Fourth, comparisons to root-shells and OpenSSL ciphers ?  Really ?
Has nobody told you that a bad analogy is like a wet screwdriver ?

Fifth, some of you have a really weird definition of "DoS", and
since the same people seem very fond of analogies, I'll answer with
one:  When I say "Varnish trust the backend backend", I mean that
it does so because that is its job.  The backend is the guitar,
Varnish is the PA system.  If you plug the guitar cable into 110VAC,
you don't expect the PA to generate an 60Hz earthquake, you expect
it to blow a fuse.  In Varnish that "fuse" is an assert.  If, like
most PA systems, such abuse left Varnish as an irepairable smoking
environmental hazard, *then* I would agree that it constituted a
DoS, but the "fuse" in Varnish self-repairs in a fraction of a
second, and as soon as you plug a working microphone back in again,
Varnish will keep on rocking with you.  Would it be better not to
blow the fuse ? Sure.  Does it matter ?  Not really.

Sixth, people building CDNs for third party traffic with Varnish
had better know what they're doing, since that is (slightly) outside
Varnish security and authority design.  The CDNs I know about do
know what they're doing.  (There may be others.)

(If there are any questions, please keep me in the CC: I'm not
on the oss-sec list).

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From mumersidd at gmail.com  Thu Jul 10 14:01:33 2014
From: mumersidd at gmail.com (Muhammed Umer Siddiqui)
Date: Thu, 10 Jul 2014 10:01:33 -0400
Subject: Large Video upload issue
Message-ID: <CAAHZe0aZHqEa6GQ4kch5HTSLjq0HLNyZU4hYP10GN3Bf1AROrQ@mail.gmail.com>

Dear team,

We are trying to upload the videos through varnish but videos more than 90
mb it shows, 503 error.

Kindly help me out what setting i use to avoid 503 on varnish layer.

-- 
Regards,
Muhammed Umer Siddiqui
Cell:+923009284182
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140710/8fd172b6/attachment.html>

From pprocacci at datapipe.com  Thu Jul 10 17:46:10 2014
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Thu, 10 Jul 2014 12:46:10 -0500
Subject: Large Video upload issue
In-Reply-To: <CAAHZe0aZHqEa6GQ4kch5HTSLjq0HLNyZU4hYP10GN3Bf1AROrQ@mail.gmail.com>
References: <CAAHZe0aZHqEa6GQ4kch5HTSLjq0HLNyZU4hYP10GN3Bf1AROrQ@mail.gmail.com>
Message-ID: <20140710174610.GA1619@workmachine>

On Thu, Jul 10, 2014 at 10:01:33AM -0400, Muhammed Umer Siddiqui wrote:
> Dear team,
> 
> We are trying to upload the videos through varnish but videos more than 90
> mb it shows, 503 error.
> 
> Kindly help me out what setting i use to avoid 503 on varnish layer.


What does your web server logs say?  POST's are pipe'd directly to the
backend.

~Paul



From thiagop1985 at gmail.com  Thu Jul 10 18:25:03 2014
From: thiagop1985 at gmail.com (Thiago Paulino)
Date: Thu, 10 Jul 2014 15:25:03 -0300
Subject: Large Video upload issue
In-Reply-To: <20140710174610.GA1619@workmachine>
References: <CAAHZe0aZHqEa6GQ4kch5HTSLjq0HLNyZU4hYP10GN3Bf1AROrQ@mail.gmail.com>
 <20140710174610.GA1619@workmachine>
Message-ID: <CAJ+apX9Nd+WejjRFHhkxt8-5a=6J8UyjOw9LyXugsoMhaCRTOg@mail.gmail.com>

Hello,


Correct Paul.

Maybe you need check te tcp (Kernel) timeout conections and varnish timeout
conections with backend's.


Regards,

Thiago Paulino


Atenciosamente,

Thiago Paulino

11-97670-9779

<http://br.linkedin.com/in/tpaulino>


2014-07-10 14:46 GMT-03:00 Paul A. Procacci <pprocacci at datapipe.com>:

> On Thu, Jul 10, 2014 at 10:01:33AM -0400, Muhammed Umer Siddiqui wrote:
> > Dear team,
> >
> > We are trying to upload the videos through varnish but videos more than
> 90
> > mb it shows, 503 error.
> >
> > Kindly help me out what setting i use to avoid 503 on varnish layer.
>
>
> What does your web server logs say?  POST's are pipe'd directly to the
> backend.
>
> ~Paul
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140710/9408d0f4/attachment.html>

From mumersidd at gmail.com  Fri Jul 11 10:29:29 2014
From: mumersidd at gmail.com (Muhammed Umer Siddiqui)
Date: Fri, 11 Jul 2014 06:29:29 -0400
Subject: Large Video upload issue
In-Reply-To: <CAJ+apX9Nd+WejjRFHhkxt8-5a=6J8UyjOw9LyXugsoMhaCRTOg@mail.gmail.com>
References: <CAAHZe0aZHqEa6GQ4kch5HTSLjq0HLNyZU4hYP10GN3Bf1AROrQ@mail.gmail.com>
 <20140710174610.GA1619@workmachine>
 <CAJ+apX9Nd+WejjRFHhkxt8-5a=6J8UyjOw9LyXugsoMhaCRTOg@mail.gmail.com>
Message-ID: <CAAHZe0Yt-LP6qAiof7z+Sm=9ho4Hi7TVZgt-W8vcAhxT_+vsOA@mail.gmail.com>

Hi,

Thanks for quick response i am using following setting  and error below ,
would you please suggest me what setting i should use to avoid this issue.

backend back_1 {
    .host = "10.30.30.55";
    .probe = {
        .request =
            "GET /HealthCheck.cfm HTTP/1.1"
            "Host: app04.web"
            "Connection: close";
            .interval = 30s; .timeout = 60s; .window = 5;.threshold = 3;
        }

    .connect_timeout = 6000s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
}


Here is what i found in the logs
[root at varnish01 backends]# varnishlog -c -m TxStatus:503
   17 SessionOpen  c 127.0.0.1 49015 :80
   17 ReqStart     c 127.0.0.1 49015 1847994289
   17 RxRequest    c POST
   17 RxURL        c /app/index.cfm
   17 RxProtocol   c HTTP/1.1
   17 RxHeader     c Host: website
   17 RxHeader     c User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
Gecko/20100101 Firefox/24.0
   17 RxHeader     c Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
   17 RxHeader     c Accept-Language: en-US,en;q=0.5
   17 RxHeader     c Accept-Encoding: gzip, deflate
   17 RxHeader     c Referer:
http://website/app/index.cfm?act=shared.videos.edit
   17 RxHeader     c Cookie:
SITEID=9FE80D8B%2DCD6F%2DFB33%2D11C1B078F2E3FFBD; CFID=875364;
CFTOKEN=aa8100f3e554d080-467F90FA-9CF0-D45D-6681EB084AC8C22F;
JSESSIONID=8430985e49dca2de7d8f91c2cd227111a2a3
   17 RxHeader     c Connection: keep-alive
   17 RxHeader     c Content-Type: multipart/form-data;
boundary=---------------------------55708277233208979222334388
   17 RxHeader     c Content-Length: 99210838
   17 VCL_call     c recv pass
   17 VCL_call     c hash
   17 Hash         c /app/index.cfm
   17 Hash         c website
   17 VCL_return   c hash
   17 VCL_call     c pass pass
   17 Backend      c 18 cm_admin cm_admin_1
*   17 FetchError   c http first read error: -1 11 (Resource temporarily
unavailable)*
   17 VCL_call     c error deliver
   17 VCL_call     c deliver deliver
   17 TxProtocol   c HTTP/1.1
   17 TxStatus     c 503
   17 TxResponse   c Service Unavailable
   17 TxHeader     c Server: Varnish
   17 TxHeader     c Content-Type: text/html; charset=utf-8
   17 TxHeader     c Retry-After: 5
   17 TxHeader     c Content-Length: 419
   17 TxHeader     c Accept-Ranges: bytes
   17 TxHeader     c Date: Fri, 11 Jul 2014 10:23:57 GMT
   17 TxHeader     c X-Varnish: 1847994289
   17 TxHeader     c Age: 607
   17 TxHeader     c Via: 1.1 varnish
   17 TxHeader     c Connection: close
   17 Length       c 419
   17 ReqEnd       c 1847994289 1405073630.438976765 1405074237.772819042
0.000091076 607.333770990 0.0000712


On Thu, Jul 10, 2014 at 2:25 PM, Thiago Paulino <thiagop1985 at gmail.com>
wrote:

> Hello,
>
>
> Correct Paul.
>
> Maybe you need check te tcp (Kernel) timeout conections and varnish
> timeout conections with backend's.
>
>
> Regards,
>
> Thiago Paulino
>
>
> Atenciosamente,
>
> Thiago Paulino
>
> 11-97670-9779
>
> <http://br.linkedin.com/in/tpaulino>
>
>
> 2014-07-10 14:46 GMT-03:00 Paul A. Procacci <pprocacci at datapipe.com>:
>
>> On Thu, Jul 10, 2014 at 10:01:33AM -0400, Muhammed Umer Siddiqui wrote:
>> > Dear team,
>> >
>> > We are trying to upload the videos through varnish but videos more than
>> 90
>> > mb it shows, 503 error.
>> >
>> > Kindly help me out what setting i use to avoid 503 on varnish layer.
>>
>>
>> What does your web server logs say?  POST's are pipe'd directly to the
>> backend.
>>
>> ~Paul
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>


-- 
Regards,
Muhammed Umer Siddiqui
Cell:+923009284182
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140711/119c97fa/attachment.html>

From lkarsten at varnish-software.com  Fri Jul 11 11:10:21 2014
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Fri, 11 Jul 2014 13:10:21 +0200
Subject: Large Video upload issue
In-Reply-To: <CAAHZe0Yt-LP6qAiof7z+Sm=9ho4Hi7TVZgt-W8vcAhxT_+vsOA@mail.gmail.com>
References: <CAAHZe0aZHqEa6GQ4kch5HTSLjq0HLNyZU4hYP10GN3Bf1AROrQ@mail.gmail.com>
 <20140710174610.GA1619@workmachine>
 <CAJ+apX9Nd+WejjRFHhkxt8-5a=6J8UyjOw9LyXugsoMhaCRTOg@mail.gmail.com>
 <CAAHZe0Yt-LP6qAiof7z+Sm=9ho4Hi7TVZgt-W8vcAhxT_+vsOA@mail.gmail.com>
Message-ID: <20140711111020.GG5080@immer.varnish-software.com>

On Fri, Jul 11, 2014 at 06:29:29AM -0400, Muhammed Umer Siddiqui wrote:
[..]
>     .connect_timeout = 6000s;
>     .first_byte_timeout = 600s;
>     .between_bytes_timeout = 600s;

This is a very bad idea.

-- 
Lasse Karstensen



From mumersidd at gmail.com  Fri Jul 11 11:27:02 2014
From: mumersidd at gmail.com (Muhammed Umer Siddiqui)
Date: Fri, 11 Jul 2014 07:27:02 -0400
Subject: Large Video upload issue
In-Reply-To: <20140711111020.GG5080@immer.varnish-software.com>
References: <CAAHZe0aZHqEa6GQ4kch5HTSLjq0HLNyZU4hYP10GN3Bf1AROrQ@mail.gmail.com>
 <20140710174610.GA1619@workmachine>
 <CAJ+apX9Nd+WejjRFHhkxt8-5a=6J8UyjOw9LyXugsoMhaCRTOg@mail.gmail.com>
 <CAAHZe0Yt-LP6qAiof7z+Sm=9ho4Hi7TVZgt-W8vcAhxT_+vsOA@mail.gmail.com>
 <20140711111020.GG5080@immer.varnish-software.com>
Message-ID: <CAAHZe0bV=3MLrMH9pqjh5GZc9ka5jU804Y+tnzUY9az4WW_VUA@mail.gmail.com>

Apologize i am using following
>     .connect_timeout = 600s;
>     .first_byte_timeout = 600s;
>     .between_bytes_timeout = 600s;

Is there any better setting please let me know ,


On Fri, Jul 11, 2014 at 7:10 AM, Lasse Karstensen <
lkarsten at varnish-software.com> wrote:

> On Fri, Jul 11, 2014 at 06:29:29AM -0400, Muhammed Umer Siddiqui wrote:
> [..]
> >     .connect_timeout = 6000s;
> >     .first_byte_timeout = 600s;
> >     .between_bytes_timeout = 600s;
>
> This is a very bad idea.
>
> --
> Lasse Karstensen
>



-- 
Regards,
Muhammed Umer Siddiqui
Cell:+923009284182
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140711/4ba939ca/attachment.html>

From jim.robinson at gmail.com  Fri Jul 11 20:29:12 2014
From: jim.robinson at gmail.com (James A. Robinson)
Date: Fri, 11 Jul 2014 13:29:12 -0700
Subject: varnish-2.1.4 (SVN 5447M) w/ backend Connection: close
Message-ID: <CAPd04b7-yUDQ1o=nzSEyKOs=Pi4T3m9H1OVeWczNbZR2AVECmQ@mail.gmail.com>

Folks,

I'm trying to track down a problem where varnish is sometimes serving
up empty responses, though the backend claims it sent the full body
back.  I think I've found the source of the problem, and wanted to ask
some advice from you folks as to possible causes and remedies.

With a packet capture I see the following sequence:

1. varnish sends a GET request for one URL

2. backend responds with 200 OK, and headers, including
a Connection:close header

3. varnish immediately sends a new GET request,
for a different URL

4. backend finishes sending the body of the first URL

5. the connection is closed.

If I understand HTTP/1.1 correctly, varnish should be reading the
body from the 1st request to the end, and closing the socket, not
sending a new request on the same socket.

Here's some data from varnishlog, it's the sequence corresponding
to the 1st request and the 2nd request.  It ends up caching the 1st
response as having a 0 length body and returns a failure on the 2nd
request.

The VCL that is being used was something we inherited from a person
who has since left the company, so I'm not very familiar with how
it is supposed to be working.

I was hoping the description of this problem might sound enough
like a well known problem, either with a bad VCL configuration or
possibly a bug with Varnish's handling of Connection: close responses
from the backend.


 1249 ReqStart     c 10.67.117.19 46543 1920805735
 1249 RxRequest    c GET
...
 1249 RxHeader     c Connection: keep-alive
 1249 VCL_call     c recv
 1249 VCL_acl      c MATCH highwire_acl 10.0.0.0/8
 1249 VCL_return   c lookup
 1249 VCL_call     c hash
 1249 VCL_return   c hash
 1249 VCL_call     c miss
 1249 VCL_return   c fetch

 6178 BackendOpen  b sass[3] 10.67.121.53 59375 110.67.117.88080

 1249 Backend      c 6178 sass sass[3]

 6178 TxRequest    b GET
...
 6178 RxProtocol   b HTTP/1.1
 6178 RxStatus     b 200
 6178 RxResponse   b OK
 ...
 6178 RxHeader     b Connection: close

 1249 TTL          c 1920805735 RFC 3600 1405091160 0 0 3600 0
 1249 VCL_call     c fetch
 1249 TTL          c 1920805735 VCL 86400 1405091160
 1249 VCL_return   c deliver
 1249 ObjProtocol  c HTTP/1.1
 1249 ObjStatus    c 200
 1249 ObjResponse  c OK
...

 6178 Length       b 0
 6178 BackendReuse b sass[3]

 1249 VCL_call     c deliver
 1249 VCL_return   c deliver
 1249 TxProtocol   c HTTP/1.1
 1249 TxStatus     c 200
 1249 TxResponse   c OK
...
 1249 Length       c 0

Right after 1249 is finished I see the data for that 2nd GET:

 6065 ReqStart     c 10.67.127.100 36016 1920805889
 6065 RxRequest    c GET
...
 6065 RxHeader     c Connection: keep-alive
 6065 VCL_call     c recv
 6065 VCL_return   c lookup
 6065 VCL_call     c hash
 6065 VCL_return   c hash
 6065 VCL_call     c miss
 6065 VCL_return   c fetch
...

 6065 Backend      c 6178 sass sass[3]
 6178 TxRequest    b GET
...
 6065 FetchError   c http first read error: -1 0 (Success)
 6178 BackendClose b sass[3]
 6065 VCL_call     c error
 6065 VCL_return   c deliver
 6065 VCL_call     c deliver
 6065 VCL_return   c deliver
 6065 TxProtocol   c HTTP/1.1
 6065 TxStatus     c 503
 6065 TxResponse   c Service Unavailable
 6065 TxHeader     c Server: Varnish
 6065 TxHeader     c Retry-After: 0
 6065 TxHeader     c Content-Type: text/html; charset=utf-8
 6065 TxHeader     c Content-Length: 854
 6065 TxHeader     c Accept-Ranges: bytes
 6065 TxHeader     c Date: Fri, 11 Jul 2014 15:06:00 GMT
 6065 TxHeader     c X-Varnish: 1920805889
 6065 TxHeader     c Age: 0
 6065 TxHeader     c Via: 1.1 varnish
 6065 TxHeader     c Connection: close
 6065 TxHeader     c Cache-Control: max-age=120
 6065 TxHeader     c X-Varnish-Cache: miss
 6065 Length       c 854
 6065 ReqEnd       c 1920805889 1405091160.433222055 1405091160.437772036
0.293696165 0.004407883 0.000142097
 6065 SessionClose c error
 6065 StatSess     c 10.67.127.100 36016 5 1 6 0 3 5 3010 39346
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140711/4b699f89/attachment.html>

From jim.robinson at gmail.com  Fri Jul 11 21:17:23 2014
From: jim.robinson at gmail.com (James A. Robinson)
Date: Fri, 11 Jul 2014 14:17:23 -0700
Subject: varnish-2.1.4 (SVN 5447M) w/ backend Connection: close
In-Reply-To: <CAPd04b7-yUDQ1o=nzSEyKOs=Pi4T3m9H1OVeWczNbZR2AVECmQ@mail.gmail.com>
References: <CAPd04b7-yUDQ1o=nzSEyKOs=Pi4T3m9H1OVeWczNbZR2AVECmQ@mail.gmail.com>
Message-ID: <CAPd04b5ug6K-TwOGvsTf3pOc=ZDUhgwtN-w=H5PcmZOXYcii7A@mail.gmail.com>

On Fri, Jul 11, 2014 at 1:29 PM, James A. Robinson <jim.robinson at gmail.com>
wrote:

> With a packet capture I see the following sequence:
>
> 1. varnish sends a GET request for one URL
>
> 2. backend responds with 200 OK, and headers, including
> a Connection:close header
>
> 3. varnish immediately sends a new GET request,
> for a different URL
>
> 4. backend finishes sending the body of the first URL
>
> 5. the connection is closed.
>
> If I understand HTTP/1.1 correctly, varnish should be reading the
> body from the 1st request to the end, and closing the socket, not
> sending a new request on the same socket.
>

I have another case where I can see this in the packet capture:

1. varnish sends a GET request

2. backend responds with 200 OK, and headers, including
a Connection: close header

3. backend sends the body

4. connection closes

and that all sounds great, but varnishlog tells me it did
the same thing as my other case -- it marked the body as
zero length and returned an empty page to the client.

In both cases I see that the backend is not provided a
Content-Length header.

Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140711/7862423f/attachment.html>

From ruben at varnish-software.com  Tue Jul 15 10:44:58 2014
From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=)
Date: Tue, 15 Jul 2014 12:44:58 +0200
Subject: Varnish Meetup in NYC on July 23rd, 6:30PM @ 1stDibs Office
Message-ID: <CAEfnkDZqNum0eqxCbFt378ZqVTWVOXGcvacx6n6npC3YoEYr-g@mail.gmail.com>

Hello everyone,

After a while with no activity in the NYC Meetup group we're excited to
announce that there will be a Varnish Meetup in New York next week!

The group has been resurrected by Fastly and 1stDibs. They collaborated to
organize and host, bringing in Fastly CEO Artur Bergman in to talk about
how he got started with Varnish and why he chose to build a high
performance content delivery network on top of it. It's definitely worth
attending, so join and bring all your webperf and speed-fanatic friends
with you!

There will be food and refreshments, along with a great group of NYC
Varnish users, so be sure to RSVP and stop by! Group's page >
http://www.meetup.com/NYC-Varnish-Meetup/

If you have any questions you can contact Elaine (on Cc) directly and she
will be happy to help you out.


Best regards,
-- 
<http://www.varnish-software.com/>*Rub?n Romero*
Community & Sales | Varnish Software AS
Cell: +47 95964088 / Office: +47 21989260
Skype, Twitter & IRC: ruben_varnish
We Make Websites Fly!Winner of the 2013 Red Herring Top 100 Global Awards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140715/13566c6e/attachment.html>

From Umesh.Telang at bbc.co.uk  Wed Jul 16 09:11:55 2014
From: Umesh.Telang at bbc.co.uk (Umesh Telang)
Date: Wed, 16 Jul 2014 09:11:55 +0000
Subject: Varnish 4 EL7 source RPM
Message-ID: <B28E1934F72F05418ED42B5174609A1F178E2070@BGB01XUD1005.national.core.bbc.co.uk>

Hello,

We've come across the source RPMs for Varnish 4 on EL6 here:
https://repo.varnish-cache.org/redhat/varnish-4.0/el6/src/varnish-release/

Just wondering whether there are any plans in the near future to create a source RPM for EL7?

Thanks,
Umesh

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140716/2ad7a1fa/attachment.html>

From spaceman at antispaceman.com  Wed Jul 16 09:17:32 2014
From: spaceman at antispaceman.com (spaceman)
Date: Wed, 16 Jul 2014 10:17:32 +0100
Subject: Purge object transparently
Message-ID: <20140716091732.GC6880@work.asset.local>

Hi,

Since the purge keyword was removed in Varnish 4.1 I have been looking for a way
to purge an object transparently. The VCL I need to be able to replace is:

sub vcl_hit {
    # purge cache when no-cache header present
    if ( req.backend.healthy && !req.http.User-Agent ~ "(B|b)ot" && req.http.Cache-Control ~ "no-cache" ) {
        purge;
    }
}

return(purge) is useless as I don't want a page telling me my object has been
purged just the refreshed object. Obviously req.backend.healthy will be replaced
with std.healthy(req.backend_hint).

Any ideas?

Regards,
spaceman

!DSPAM:53c6432d7583086614493!





From apj at mutt.dk  Wed Jul 16 09:48:00 2014
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Wed, 16 Jul 2014 11:48:00 +0200
Subject: Purge object transparently
In-Reply-To: <20140716091732.GC6880@work.asset.local>
References: <20140716091732.GC6880@work.asset.local>
Message-ID: <20140716094800.GD19870@nerd.dk>

On Wed, Jul 16, 2014 at 10:17:32AM +0100, spaceman wrote:
> 
> return(purge) is useless as I don't want a page telling me my object has been
> purged just the refreshed object. Obviously req.backend.healthy will be replaced
> with std.healthy(req.backend_hint).
> 
> Any ideas?

You can restart in vcl_purge.

-- 
Andreas



From lkarsten at varnish-software.com  Wed Jul 16 10:33:38 2014
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 16 Jul 2014 12:33:38 +0200
Subject: Varnish 4 EL7 source RPM
In-Reply-To: <B28E1934F72F05418ED42B5174609A1F178E2070@BGB01XUD1005.national.core.bbc.co.uk>
References: <B28E1934F72F05418ED42B5174609A1F178E2070@BGB01XUD1005.national.core.bbc.co.uk>
Message-ID: <20140716103337.GA6169@immer.varnish-software.com>

On Wed, Jul 16, 2014 at 09:11:55AM +0000, Umesh Telang wrote:
> We've come across the source RPMs for Varnish 4 on EL6 here:
> https://repo.varnish-cache.org/redhat/varnish-4.0/el6/src/varnish-release/
> Just wondering whether there are any plans in the near future to create a source RPM for EL7?

We're currently working on getting the build infrastructure set up for
EL7. I suppose we'll need to do some work for systemd adoption as well.

Timeframe is late july/early august.

-- 
Lasse Karstensen
Varnish Software AS



From maillists0 at gmail.com  Wed Jul 16 12:05:15 2014
From: maillists0 at gmail.com (maillists0 at gmail.com)
Date: Wed, 16 Jul 2014 08:05:15 -0400
Subject: Handling 400's without saint mode
Message-ID: <CAB+OxSD3vMSaNx2NxqMfpMfv-hT1Ns4wpTmZh15vhiPrLxCzRQ@mail.gmail.com>

Using Varnish 3.x to retrieve objects from s3. Due to a release process
that is beyond my control, some items are occasionally unavailable for 3 to
5 minutes when they're being recreated. Saint mode doesn't seem to do what
I need here.

What I really want is to check the backend first and then serve from cache
on a 400 for each request to this backend. Is that possible with Saint
mode, or is there a better way to do this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140716/2d124444/attachment.html>

From lkarsten at varnish-software.com  Wed Jul 16 13:12:11 2014
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 16 Jul 2014 15:12:11 +0200
Subject: Handling 400's without saint mode
In-Reply-To: <CAB+OxSD3vMSaNx2NxqMfpMfv-hT1Ns4wpTmZh15vhiPrLxCzRQ@mail.gmail.com>
References: <CAB+OxSD3vMSaNx2NxqMfpMfv-hT1Ns4wpTmZh15vhiPrLxCzRQ@mail.gmail.com>
Message-ID: <20140716131211.GC6169@immer.varnish-software.com>

On Wed, Jul 16, 2014 at 08:05:15AM -0400, maillists0 at gmail.com wrote:
> Using Varnish 3.x to retrieve objects from s3. Due to a release process
> that is beyond my control, some items are occasionally unavailable for 3 to
> 5 minutes when they're being recreated. Saint mode doesn't seem to do what
> I need here.
> What I really want is to check the backend first and then serve from cache
> on a 400 for each request to this backend. Is that possible with Saint
> mode, or is there a better way to do this?

Saint mode and grace should help you out here.

Something like this could perhaps work:

	backend s3server {
	    .host = ..
	    .saintmode_threshold = 999999999;
	}

	sub vcl_recv {
		set req.grace = 10m;
	}

	sub vcl_fetch {
		set beresp.grace = 10m;
		if (req.backend == s3server && beresp.status == 400) {
			set beresp.saintmode = 300s;
			return(restart);
		}
	}

Any URLs giving a 400s from s3server is then marked as sick for 300s.

As long as there is an old version of the object still in cache, give
that to this and any clients asking for this URL for the next 300s.


-- 
Lasse Karstensen
Varnish Software AS



From spaceman at antispaceman.com  Wed Jul 16 16:27:35 2014
From: spaceman at antispaceman.com (spaceman)
Date: Wed, 16 Jul 2014 17:27:35 +0100
Subject: Purge object transparently
In-Reply-To: <20140716094800.GD19870@nerd.dk>
References: <20140716091732.GC6880@work.asset.local>
 <20140716094800.GD19870@nerd.dk>
Message-ID: <20140716162734.GB1997@work.asset.local>

Thanks Andreas,

For anyone else interested as it wasn't that obvious:
sub vcl_purge {
    # restart request
    set req.http.X-Purge = "Yes";
    return(restart);
}
This X-Purge header is used to prevent purge from being called again because the
Cache-Control header is still set to no-cache on restart. If you fail to do that
you will get 503. I did it that way so the headers would be preserved for the
backend.

Regards,
spaceman

On 2014/07/16 11:07:00 , Andreas Plesner Jacobsen wrote:
> On Wed, Jul 16, 2014 at 10:17:32AM +0100, spaceman wrote:
> > 
> > return(purge) is useless as I don't want a page telling me my object has been
> > purged just the refreshed object. Obviously req.backend.healthy will be replaced
> > with std.healthy(req.backend_hint).
> > 
> > Any ideas?
> 
> You can restart in vcl_purge.
> 
> -- 
> Andreas
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> 
> 
> 
> 

!DSPAM:53c6a7f87581428214722!





From maillists0 at gmail.com  Wed Jul 16 21:54:47 2014
From: maillists0 at gmail.com (maillists0 at gmail.com)
Date: Wed, 16 Jul 2014 17:54:47 -0400
Subject: Handling 400's without saint mode
In-Reply-To: <20140716131211.GC6169@immer.varnish-software.com>
References: <CAB+OxSD3vMSaNx2NxqMfpMfv-hT1Ns4wpTmZh15vhiPrLxCzRQ@mail.gmail.com>
 <20140716131211.GC6169@immer.varnish-software.com>
Message-ID: <CAB+OxSCOU09Yzs9UwiR3Y2CZA7iD-rTt1ie92ZCES1bNrGWJRg@mail.gmail.com>

Thanks for the response. Setting saintmode_threshold made it behave more
like what I expected.

I have saintmode set to 300 seconds. To test, I'm making a single request
in a loop, then moving the file. While Varnish is in Saint mode, I see
these sorts of responses. Note that the age goes from 75 to 0 to 78, and
the X-Varnish fields are different. It really looks like two different
things are being cached for the same request, but I don't know how that's
possible. Suggestions on where to look?

HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 75
Content-length: 146369
Content-Type: text/css
Date: Wed, 16 Jul 2014 21:27:24 GMT
Last-Modified: Wed, 16 Jul 2014 21:23:09 GMT
Server: AmazonS3
Via: 1.1 varnish
X-Cache: HIT
X-Cache-Hits: 59
X-Cacheable: YES
X-Varnish: 1189098768 1189098616
Connection: keep-alive

HTTP/1.1 404 Not Found
Accept-Ranges: bytes
Age: 0
Content-length: 349
Content-Type: text/html; charset=utf-8
Date: Wed, 16 Jul 2014 21:27:25 GMT
Server: AmazonS3
Via: 1.1 varnish
X-Cache: MISS
X-Cacheable: NO:Bad status
X-Varnish: 1189098774
Connection: keep-alive

HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 78
Content-length: 146369
Content-Type: text/css
Date: Wed, 16 Jul 2014 21:27:26 GMT
Last-Modified: Wed, 16 Jul 2014 21:23:09 GMT
Server: AmazonS3
Via: 1.1 varnish
X-Cache: HIT
X-Cache-Hits: 60
X-Cacheable: YES
X-Varnish: 1189098775 1189098616
Connection: keep-alive



On Wed, Jul 16, 2014 at 9:12 AM, Lasse Karstensen <
lkarsten at varnish-software.com> wrote:

> On Wed, Jul 16, 2014 at 08:05:15AM -0400, maillists0 at gmail.com wrote:
> > Using Varnish 3.x to retrieve objects from s3. Due to a release process
> > that is beyond my control, some items are occasionally unavailable for 3
> to
> > 5 minutes when they're being recreated. Saint mode doesn't seem to do
> what
> > I need here.
> > What I really want is to check the backend first and then serve from
> cache
> > on a 400 for each request to this backend. Is that possible with Saint
> > mode, or is there a better way to do this?
>
> Saint mode and grace should help you out here.
>
> Something like this could perhaps work:
>
>         backend s3server {
>             .host = ..
>             .saintmode_threshold = 999999999;
>         }
>
>         sub vcl_recv {
>                 set req.grace = 10m;
>         }
>
>         sub vcl_fetch {
>                 set beresp.grace = 10m;
>                 if (req.backend == s3server && beresp.status == 400) {
>                         set beresp.saintmode = 300s;
>                         return(restart);
>                 }
>         }
>
> Any URLs giving a 400s from s3server is then marked as sick for 300s.
>
> As long as there is an old version of the object still in cache, give
> that to this and any clients asking for this URL for the next 300s.
>
>
> --
> Lasse Karstensen
> Varnish Software AS
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140716/9a593e3c/attachment.html>

From lkarsten at varnish-software.com  Thu Jul 17 08:55:38 2014
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Thu, 17 Jul 2014 10:55:38 +0200
Subject: Handling 400's without saint mode
In-Reply-To: <CAB+OxSCOU09Yzs9UwiR3Y2CZA7iD-rTt1ie92ZCES1bNrGWJRg@mail.gmail.com>
References: <CAB+OxSD3vMSaNx2NxqMfpMfv-hT1Ns4wpTmZh15vhiPrLxCzRQ@mail.gmail.com>
 <20140716131211.GC6169@immer.varnish-software.com>
 <CAB+OxSCOU09Yzs9UwiR3Y2CZA7iD-rTt1ie92ZCES1bNrGWJRg@mail.gmail.com>
Message-ID: <20140717085537.GC24694@immer.varnish-software.com>

On Wed, Jul 16, 2014 at 05:54:47PM -0400, maillists0 at gmail.com wrote:
[..]
> I have saintmode set to 300 seconds. To test, I'm making a single request
> in a loop, then moving the file. While Varnish is in Saint mode, I see
> these sorts of responses. Note that the age goes from 75 to 0 to 78, and
> the X-Varnish fields are different. It really looks like two different
> things are being cached for the same request, but I don't know how that's
> possible. Suggestions on where to look?
[..]

Do you have a load balancer in front? :)

Remember that Age is additive, Varnish will add to any incoming Age header value it gets from the backend.

varnishlog will tell you what is happening in detail.

-- 
Lasse Karstensen
Varnish Software AS



From bluethundr at gmail.com  Sun Jul 20 14:36:34 2014
From: bluethundr at gmail.com (Tim Dunphy)
Date: Sun, 20 Jul 2014 10:36:34 -0400
Subject: serve healthcheck file locally
Message-ID: <CAOZy0e=gm+MUoNce1M1yF7FRMrXKxsrzH8Tti+aicnDOzuyK-w@mail.gmail.com>

Hey all,

 I have a situation that I need to try and solve with varnish. As of now
I'm serving up a website with two varnish nodes being balanced (leastconn)
by an F5. The two varnish nodes then do a client balancing scheme across 3
web servers.

But as of now I'm only running TCP port checks against varnish on 8080. But
the F5 team is unhappy with that solution and say it's not up to their
standards of doing content checks. So what I'd like to be able to do is
have the varnish cache serve a static file locally.

Such that you can do a :

GET http://$(hostname -f):8080/healtcheck.php

And return the contents of that file (which is simply the word 'good').

I've been googling a while and not found any advice on how to server a text
file locally on the varnish server itself.

Is there any way I can do this easily in my VCL?

Thanks!
Tim

-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140720/681d6241/attachment.html>

From varnish-cache at otoh.org  Sun Jul 20 14:35:58 2014
From: varnish-cache at otoh.org (Paul Armstrong)
Date: Sun, 20 Jul 2014 14:35:58 +0000
Subject: serve healthcheck file locally
In-Reply-To: <CAOZy0e=gm+MUoNce1M1yF7FRMrXKxsrzH8Tti+aicnDOzuyK-w@mail.gmail.com>
References: <CAOZy0e=gm+MUoNce1M1yF7FRMrXKxsrzH8Tti+aicnDOzuyK-w@mail.gmail.com>
Message-ID: <20140720143558.GN17750@suricate.otoh.org>

At 2014-07-20T10:36-0400, Tim Dunphy wrote:
>    Such that you can do a :
>    GET http://$(hostname -f):8080/healtcheck.php
>    And return the contents of that file (which is simply the word 'good').
>    I've been googling a while and not found any advice on how to server a
>    text file locally on the varnish server itself.
>    Is there any way I can do this easily in my VCL?

You can serve content directly using a synthetic response. Other than
that, you'll need to run a local webserver and redirect requests for
that file to the appropriate backend.

Having said that, if you want proper end to end testing, you're better
off either serving the healthcheck file from the backends with a
passthrough (so you can ensure that varnish is able to get to at least
one backend) or with a very short TTL (tests caching as well as backend
availability, but has a longer recovery time if a varnish host is unable
to reach any backends).

Paul



From bluethundr at gmail.com  Sun Jul 20 15:07:35 2014
From: bluethundr at gmail.com (Tim Dunphy)
Date: Sun, 20 Jul 2014 11:07:35 -0400
Subject: serve healthcheck file locally
In-Reply-To: <20140720143558.GN17750@suricate.otoh.org>
References: <CAOZy0e=gm+MUoNce1M1yF7FRMrXKxsrzH8Tti+aicnDOzuyK-w@mail.gmail.com>
 <20140720143558.GN17750@suricate.otoh.org>
Message-ID: <CAOZy0emw1tUF4kB7uUQ8og4-1HvNZMirqbQRkwzSX905dRfx6g@mail.gmail.com>

>
> You can serve content directly using a synthetic response. Other than
> that, you'll need to run a local webserver and redirect requests for
> that file to the appropriate backend.


Having said that, if you want proper end to end testing, you're better
> off either serving the healthcheck file from the backends with a
> passthrough (so you can ensure that varnish is able to get to at least
> one backend) or with a very short TTL (tests caching as well as backend
> availability, but has a longer recovery time if a varnish host is unable
> to reach any backends).


Thanks for your input! So the idea here is to get the varnish servers
themselves check in with the F5. It's a form of health check polling that
they do to determine that the varnish hosts are online.  I could do a pass
through to the web servers to serve the health check file. That should
still let the F5's know that the varnish hosts are alive.

But for now I've just fired up a copy of apache locally on port 80. So as
of now I can do a GET on that healthcheck file:

[root at varn001 varnish]# GET http://$(hostname -f)/healthcheck.php
good

And I tried popping the following into my sub recv_vcl in the hopes that it
could return the same result on 8080:

  # Let all health check requests from F5 pass through completely.
  if (req.url ~ "^/healthcheck\.php") {
    return (pipe);
  }

But after I bounce varnish with the change I get this result:

[root at varn001 varnish]# GET http://$(hostname -f):8080/healthcheck.php
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /healthcheck.php was not found on this server.</p>
<hr>
<address>Apache Server at varn001.dmz.tfayd.com Port 8080</address>
</body></html>

Any thoughts on how to solve this?

Thanks,
Tim






On Sun, Jul 20, 2014 at 10:35 AM, Paul Armstrong <varnish-cache at otoh.org>
wrote:

> At 2014-07-20T10:36-0400, Tim Dunphy wrote:
> >    Such that you can do a :
> >    GET http://$(hostname -f):8080/healtcheck.php
> >    And return the contents of that file (which is simply the word
> 'good').
> >    I've been googling a while and not found any advice on how to server a
> >    text file locally on the varnish server itself.
> >    Is there any way I can do this easily in my VCL?
>
> You can serve content directly using a synthetic response. Other than
> that, you'll need to run a local webserver and redirect requests for
> that file to the appropriate backend.
>
> Having said that, if you want proper end to end testing, you're better
> off either serving the healthcheck file from the backends with a
> passthrough (so you can ensure that varnish is able to get to at least
> one backend) or with a very short TTL (tests caching as well as backend
> availability, but has a longer recovery time if a varnish host is unable
> to reach any backends).
>
> Paul
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140720/8e312275/attachment.html>

From bluethundr at gmail.com  Mon Jul 21 20:41:10 2014
From: bluethundr at gmail.com (Tim Dunphy)
Date: Mon, 21 Jul 2014 16:41:10 -0400
Subject: least connections director in varnish 3.5
Message-ID: <CAOZy0em7_f5cUvaqJnD8dNa5eGyjp3kCXv4m2YJwNwW1Ak=itw@mail.gmail.com>

Hey all,

 Is there any way to implement a least conn load balancing algorithm inside
a varnish 3.5 VCL config?  How can I go about doing that, if possible? I
googled around for a bit and saw that there was a patch, but it doesn't
allow for .weight to be added. Not sure if I'm missing something.

Thanks!
Tim

-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140721/26a48b97/attachment.html>

From toasty at dragondata.com  Mon Jul 21 21:35:25 2014
From: toasty at dragondata.com (Kevin Day)
Date: Mon, 21 Jul 2014 16:35:25 -0500
Subject: Determine if grace object is available
Message-ID: <7C954D35-328D-49B1-B854-302A26CC1981@dragondata.com>

Hi, Varnish people!

I?d like to get fancy with grace stored objects, but I?m not sure how to do this. Can I determine if there?s a grace object I could deliver? Basically I want my logic to be:

1) If the backend returns a 200, cache and deliver as normal.
2) If the backend returns anything other than a 200, and I have a stale object I could serve anyway, serve it.
3) If the backend returns anything other than a 200, restart to redirect to a final fallback.


I know how to do #1 and #3, but I?m not sure where/how I can do #2. Is there a way of determining if I could fallback to a stale object? If so, any pointers?

? Kevin




From t.honacker at googlemail.com  Tue Jul 22 05:13:51 2014
From: t.honacker at googlemail.com (Tobias Honacker)
Date: Tue, 22 Jul 2014 07:13:51 +0200
Subject: 503 remove backends response
Message-ID: <CADepKeD9O4ZBZW_aDafajT5ggi7dcLAG84UO6EWc9reK4LbtoA@mail.gmail.com>

Hi all,

first some details of my varnish:

varnish-3.0.5-1.el6.x86_64
varnish-libs-3.0.5-1.el6.x86_64


We are trying to remove the response of 503 status the backend delivers to
our varnish.

--snip--

 37 TxHeader     b X-Backend-Healthy: yes
   37 TxHeader     b X-Varnish: 1092996393
   37 RxProtocol   b HTTP/1.1
   37 RxStatus     b 503
   37 RxResponse   b Service Unavailable
   37 RxHeader     b Server: cloudflare-nginx
   37 RxHeader     b Date: Mon, 21 Jul 2014 14:18:51 GMT
   37 RxHeader     b Content-Type: text/html
   37 RxHeader     b Transfer-Encoding: chunked
   37 RxHeader     b Connection: keep-alive
   37 RxHeader     b Set-Cookie: __
   37 RxHeader     b Cache-Control: no-cache
   37 RxHeader     b CF-Cache-Status: MISS
   37 RxHeader     b Vary: Accept-Encoding
   37 RxHeader     b CF-RAY:
   37 Fetch_Body   b 3(chunked) cls 0 mklen 1
   37 Length       b 459
   37 BackendReuse b host_1

--snip--

These Backend includes ESI to our site and if the backend (not sick at this
time) throw a 503 to our varnish we see the nginx error message at our page:

Error 503 .. Service Unavailable ...and so on

Now we tried to fix this using in vcl_error these method without success:

if ( (obj.status >= 500 || obj.status <= 505) && (req.url ~ "^/appserver/")
) {
      set obj.http.Cache-Control = "public, max-age=10";
      set obj.status = 200;
      set obj.response = "OK";
      return (deliver);
  }

we are aiming to serve a blank 200 status so that the error disappear on
our site where the ESI is included.

how could we fix that issue? is there any way not using synthetic while
serving blank html code? i dont like this "workaround".

thanks a lot for any help.

best regards
tobias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140722/3f4887b7/attachment.html>

From thomas.lecomte at virtual-expo.com  Tue Jul 22 05:24:29 2014
From: thomas.lecomte at virtual-expo.com (Thomas Lecomte)
Date: Tue, 22 Jul 2014 07:24:29 +0200
Subject: 503 remove backends response
In-Reply-To: <CADepKeD9O4ZBZW_aDafajT5ggi7dcLAG84UO6EWc9reK4LbtoA@mail.gmail.com>
References: <CADepKeD9O4ZBZW_aDafajT5ggi7dcLAG84UO6EWc9reK4LbtoA@mail.gmail.com>
Message-ID: <CAJban9CufVzOFdUMcsJFt-uakc1KbkoBM3UY5u5oz4pJLy+x3w@mail.gmail.com>

On Tue, Jul 22, 2014 at 7:13 AM, Tobias Honacker
<t.honacker at googlemail.com> wrote:
> if ( (obj.status >= 500 || obj.status <= 505) && (req.url ~ "^/appserver/")
> ) {
>       set obj.http.Cache-Control = "public, max-age=10";
>       set obj.status = 200;
>       set obj.response = "OK";
>       return (deliver);
>   }
>
> we are aiming to serve a blank 200 status so that the error disappear on our
> site where the ESI is included.
>
> how could we fix that issue? is there any way not using synthetic while
> serving blank html code? i dont like this "workaround".

Hello,

You can throw a custom error in the VCL, and catch it in vcl_error to
serve blank synthetic content:

 if ( (obj.status >= 500 || obj.status <= 505) && (req.url ~ "^/appserver/") ) {
     error 942 "Backend down";

     return (deliver);
 }

sub vcl_error {
  if (obj.status == 942) {
       set obj.http.Cache-Control = "public, max-age=10";
       set obj.status = 200;
       set obj.response = "OK";
       synthetic "";
       return (deliver);
   }
}

Please note however than when varnish delivers content from vcl_error,
it sends a Connection: close header, which can make things slower
since the client will have to re-establish the connection for further
requests.

-- 
Thomas Lecomte



From imer at imer.cc  Wed Jul 23 00:35:22 2014
From: imer at imer.cc (Robin)
Date: Wed, 23 Jul 2014 02:35:22 +0200
Subject: Content-Length on first fetch, Varnish 4
Message-ID: <53CF034A.2010001@imer.cc>

Like this question asks 
http://stackoverflow.com/questions/23643233/how-do-i-disable-transfer-encoding-chunked-encoding-in-varnish

We have some code in place that relies on having a content-length header 
to display a progress bar, we have patched this code, but to deliver 
it.. well, we have to let it download the new binary.
I have tried to just set streaming to false for the binary
> sub vcl_backend_response {
>     if (bereq.url ~ "\.exe$") {
>         set beresp.do_stream = false;
>     }
>     set beresp.do_gzip = false;
> }
And while it seems to set streaming to false (requests loads for 10-20s 
to retrieve to binary completely) it doesnt have a content-length header 
the first time.
The suggested esi-processing doesnt work either (streaming true and false)
Is there any workaround I could use?

I have also seen there are some patches in the works to make use of the 
content-length header from the backend, does anyone know what the 
current state on that is?



From lkarsten at varnish-software.com  Wed Jul 23 13:03:44 2014
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 23 Jul 2014 15:03:44 +0200
Subject: Determine if grace object is available
In-Reply-To: <7C954D35-328D-49B1-B854-302A26CC1981@dragondata.com>
References: <7C954D35-328D-49B1-B854-302A26CC1981@dragondata.com>
Message-ID: <20140723130343.GB11654@immer.varnish-software.com>

On Mon, Jul 21, 2014 at 04:35:25PM -0500, Kevin Day wrote:
> I?d like to get fancy with grace stored objects, but I?m not sure how to do this. Can I determine if there?s a grace object I could deliver? Basically I want my logic to be:
> 1) If the backend returns a 200, cache and deliver as normal.
> 2) If the backend returns anything other than a 200, and I have a stale object I could serve anyway, serve it.
> 3) If the backend returns anything other than a 200, restart to redirect to a final fallback.
> I know how to do #1 and #3, but I?m not sure where/how I can do #2. Is there a way of determining if I could fallback to a stale object? If so, any pointers?

You can do (2) with saint mode and a restart:

    https://www.varnish-cache.org/docs/3.0/tutorial/handling_misbehaving_servers.html#saint-mode


If this fails, because you didn't have a stale object available, you can
do (3) via vcl_error automatically.

-- 
Lasse Karstensen
Varnish Software AS



From jon at magnetic.com  Fri Jul 25 02:56:02 2014
From: jon at magnetic.com (Jon Bogaty)
Date: Thu, 24 Jul 2014 22:56:02 -0400
Subject: Wildly different cache object count, cache size, and cache hit rate
 on 1 out of 3 servers (Exact same config)
Message-ID: <CAC0HZrOVeb4mPLeZMXkQ8Q2aRQ8gCX-xr5-p2Q5AOoGQPTYAvg@mail.gmail.com>

I'm an engineer on the eng-ops side of my company and I cannot figure out
recent numbers from our monitor for Varnish. I've got three servers that
run production Solr, each with Varnish on the same node in front. These are
physical servers, plenty of resources.

For the past few days because of JAR incompatibilities I've been
downgrading them from Solr 4.8 to Solr 4.7.2. This I do not think would
impact Varnish (Since once downgraded they will serve the exact same data
again) but ever since work began the numbers for cache object count, cache
size, and cache hit rate have been... Weird.

I think maybe I've used the word 'weird' to quantify a problem maybe twice
in my career. Because usually there's some kinda reason and I can go well,
that's because of the thingymagummer going outta wack.

These machines are all provisioned via Chef. Same CentOS OS, same physical
specs, same versions down to patch level of all services, Varnish, et al.
Configs have been diffed, no discrepancies that might explain this. They
are behind a Stingray load balance and have been verified as getting pretty
much the same division of load consistently.

For the past few days with no real pattern one machine has been way above
the others. I mean variances of a million cached objects, variances of 5GB
on the cache size, glaring 50% variances on hit rate. If there's one
pattern it's that one machine arbitrarily did twice as much as one of the
others, which did about equal.

I'm really stumped here. We run jobs that query Solr every hour plus so it
stays used outside of normal business hours. But this is the other thing. I
just checked the monitoring before posting to get the variances and found
that at about 22:00 EST there was a convergence and all three are now at
the same low rate.

Could somehow they have been stabilizing caches and one just shot ahead for
no reason? If the variances were small I'd say fine... But first one
machine had massive variances, then another, and they have no pattern that
I can see. They didn't even correspond to the order they were brought out
of active service and downgraded.

I can post config if it helps. What could possibly cause one of three
identical Varnish machines to so drastically outperform the others?

Thanks! Hope this is the right place to ask. Really appreciate any thoughts
you folks can lend!

Jon Bogaty,
Magnetic, DevOps Engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140724/39220b40/attachment.html>

From yobabu at yahoo.com  Sat Jul 26 11:04:34 2014
From: yobabu at yahoo.com (Lakshmi Mahesh Babu)
Date: Sat, 26 Jul 2014 04:04:34 -0700
Subject: Varnish for multiple domains in cpanel
Message-ID: <1406372674.86643.YahooMailNeo@web121101.mail.ne1.yahoo.com>

Hi, I would like to serve multiple domains with their own VCLs. I have 128GB RAM server and I have around 50 CMS sites and I want to serve all dynamic content via varnish except SSL content. I am able to do this for just one site as usual, but all sites have different CMS and all dont fit into one VCL. I want the ability to purge cache for all sites individually. I have seen forums and mailing lists that are available in public, but nowhere I could find the default.vcl that can be configured with mutliple VCLs. I am not a typical regular programmer. Please help with this. This is not commercial project but my content sites needs to be given some speed boost.

Please help with this.?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140726/800ae4a7/attachment.html>

From yobabu at yahoo.com  Sat Jul 26 11:30:42 2014
From: yobabu at yahoo.com (Lakshmi Mahesh Babu)
Date: Sat, 26 Jul 2014 04:30:42 -0700
Subject: Varnish for multiple domains in cpanel
In-Reply-To: <C30D6D67AB4E744DA1E9793E194536C1107B11263D@THHS2E12BE3X.hostedservice2.net>
References: <C30D6D67AB4E744DA1E9793E194536C1107B11263D@THHS2E12BE3X.hostedservice2.net>
Message-ID: <1406374242.66803.YahooMailNeo@web121103.mail.ne1.yahoo.com>

Thank you for taking time to reply. Yes, I am aware of the plugin, but this plugin is encoded and does not allow me to change individual vcls, in fact does not allow me to see if each site has different VCLs or not. I need freedom to change the VCL for each site.



________________________________
 From: Sam Pegler <sam.pegler at firstclarity.com>
To: Lakshmi Mahesh Babu <yobabu at yahoo.com> 
Sent: Saturday, July 26, 2014 4:54 PM
Subject: Re: Varnish for multiple domains in cpanel
 


There's a Varnish plugin for WHM. 


On 26 Jul 2014 12:06, Lakshmi Mahesh Babu <yobabu at yahoo.com> wrote:

Hi, I would like to serve multiple domains with their own VCLs. I have 128GB RAM server and I have around 50 CMS sites and I want to serve all dynamic content via varnish except SSL content. I am able to do this for just one site as usual, but all sites have different CMS and all dont fit into one VCL. I want the ability to purge cache for all sites individually. I have seen forums and mailing lists that are available in public, but nowhere I could find the default.vcl that can be configured with mutliple VCLs. I am not a typical regular programmer. Please help with this. This is not commercial project but my content sites needs to be given some speed boost.

Please help with this.?
______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140726/056cc5e6/attachment.html>

From stef at scaleengine.com  Sat Jul 26 11:58:59 2014
From: stef at scaleengine.com (Stefan Caunter)
Date: Sat, 26 Jul 2014 07:58:59 -0400
Subject: Varnish for multiple domains in cpanel
In-Reply-To: <1406372674.86643.YahooMailNeo@web121101.mail.ne1.yahoo.com>
References: <1406372674.86643.YahooMailNeo@web121101.mail.ne1.yahoo.com>
Message-ID: <CALdpnry+AK-kcJKEA-FGjMRHi80uErXR=66991NeaVqhv7gvyQ@mail.gmail.com>

On Sat, Jul 26, 2014 at 7:04 AM, Lakshmi Mahesh Babu <yobabu at yahoo.com> wrote:
> Hi, I would like to serve multiple domains with their own VCLs. I have 128GB
> RAM server and I have around 50 CMS sites and I want to serve all dynamic
> content via varnish except SSL content. I am able to do this for just one
> site as usual, but all sites have different CMS and all dont fit into one
> VCL. I want the ability to purge cache for all sites individually. I have
> seen forums and mailing lists that are available in public, but nowhere I
> could find the default.vcl that can be configured with mutliple VCLs. I am
> not a typical regular programmer. Please help with this. This is not
> commercial project but my content sites needs to be given some speed boost.
>
> Please help with this.
>

50 sites? Are you under the impression that there is a way to
automagically detect how each webapp and cms works, and somehow
reverse engineer appropriate caching rules that don't completely break
how the sites work?
You write specific vcl for each one and test each one until it works
properly. That's what everyone does.



From yobabu at yahoo.com  Sat Jul 26 12:01:14 2014
From: yobabu at yahoo.com (Lakshmi Mahesh Babu)
Date: Sat, 26 Jul 2014 05:01:14 -0700
Subject: Varnish for multiple domains in cpanel
In-Reply-To: <CALdpnry+AK-kcJKEA-FGjMRHi80uErXR=66991NeaVqhv7gvyQ@mail.gmail.com>
References: <1406372674.86643.YahooMailNeo@web121101.mail.ne1.yahoo.com>
 <CALdpnry+AK-kcJKEA-FGjMRHi80uErXR=66991NeaVqhv7gvyQ@mail.gmail.com>
Message-ID: <1406376074.69021.YahooMailNeo@web121104.mail.ne1.yahoo.com>

Ya, I agree with you, I will of course test each one and see how far I can go with number of sites, but before that I need to make it work to accept more than one VCL,i.e, multiple sites and then I can test the limits.


________________________________
 From: Stefan Caunter <stef at scaleengine.com>
To: Lakshmi Mahesh Babu <yobabu at yahoo.com> 
Cc: "varnish-misc at varnish-cache.org" <varnish-misc at varnish-cache.org> 
Sent: Saturday, July 26, 2014 5:28 PM
Subject: Re: Varnish for multiple domains in cpanel
 

On Sat, Jul 26, 2014 at 7:04 AM, Lakshmi Mahesh Babu <yobabu at yahoo.com> wrote:



> Hi, I would like to serve multiple domains with their own VCLs. I have 128GB
> RAM server and I have around 50 CMS sites and I want to serve all dynamic
> content via varnish except SSL content. I am able to do this for just one
> site as usual, but all sites have different CMS and all dont fit into one
> VCL. I want the ability to purge cache for all sites individually. I have
> seen forums and mailing lists that are available in public, but nowhere I
> could find the default.vcl that can be configured with mutliple VCLs. I am
> not a typical regular programmer. Please help with this. This is not
> commercial project but my content sites needs to be given some speed boost.
>
> Please help with this.
>

50 sites? Are you under the impression that there is a way to
automagically detect how each webapp and cms works, and somehow
reverse engineer appropriate caching rules that don't completely break
how the sites work?
You write specific vcl for each one and test each one until it works
properly. That's what everyone does.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140726/9a8925e6/attachment.html>

From dan at magnetic.com  Sat Jul 26 12:57:45 2014
From: dan at magnetic.com (Dan Crosta)
Date: Sat, 26 Jul 2014 08:57:45 -0400
Subject: High LRU nuked with plenty of cache space available
Message-ID: <CAMwwTFx92fatckRqz=caCdpo6yTwN=opUM_cahJ4XMb4sxw5tQ@mail.gmail.com>

We're running varnish (3.0.4) like:

/usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f
/etc/varnish/default.vcl -T 127.0.0.1:8080 -t 2628000 -w 50,1000,120
-u varnish -g varnish -S /etc/varnish/secret -s
file,/srv/varnish/varnish_storage.bin,12G -p thread_pools=4 -p
http_gzip_support=off -p saintmode_threshold=0 -p
http_range_support=off

varnishstat shows the following for SMF:

SMF.s0.c_req            45213612       317.34 Allocator requests
SMF.s0.c_fail           14781854       103.75 Allocator failures
SMF.s0.c_bytes      2056164147200  14431753.97 Bytes allocated
SMF.s0.c_freed      2052654858240  14407123.06 Bytes freed
SMF.s0.g_alloc            856760          .   Allocations outstanding
SMF.s0.g_bytes        3509288960          .   Bytes outstanding
SMF.s0.g_space        9375612928          .   Bytes available
SMF.s0.g_smf              950080          .   N struct smf
SMF.s0.g_smf_frag          93319          .   N small free smf
SMF.s0.g_smf_large             1          .   N large free smf

which I interpret to mean that ~9G of the 12G are available ("bytes
available") for use by the cache.

However, when our applications service the API through varnish at a
rate of about 150 QPS, we see that n_lru_nuked increases at about the
same rate, the result of which is that the n_object counter stays more
or less constant. The cache has only been running for a few days, so I
don't believe TTL is to blame here, since we use -t with about 30 days
(assuming the value there is in seconds, which the docs seem to
imply).

I have tried setting a granularity on the storage, which did not seem
to have any impact. I'm looking for other suggestions or things to
try, as intuition and the stats seem to suggest we should be able to
store a lot more objects before things start to get nuked.

Thanks,
- Dan



From carlos.abalde at gmail.com  Mon Jul 28 10:18:59 2014
From: carlos.abalde at gmail.com (Carlos Abalde)
Date: Mon, 28 Jul 2014 12:18:59 +0200
Subject: Updating VMODs with zero downtime
Message-ID: <BA130183-72D3-4D52-B594-8E4293FDA430@gmail.com>

Hi all,

Lately I've been wondering about the possibility of updating Varnish VMODs with zero downtime. Being able of reloading VCL with zero downtime is a great Varnish Feature; it reminds me a lot to hot code replacement in Erlang, which is one of the killer features of the language. But what about VMODs? Is this an existing / planned feature? Any workarounds (apart from embedding all the VMOD logic in VCL using inline C code)?

Thanks!

?
Carlos Abalde.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140728/8e545023/attachment.bin>

From phk at phk.freebsd.dk  Mon Jul 28 10:21:20 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 28 Jul 2014 10:21:20 +0000
Subject: Updating VMODs with zero downtime
In-Reply-To: <BA130183-72D3-4D52-B594-8E4293FDA430@gmail.com>
References: <BA130183-72D3-4D52-B594-8E4293FDA430@gmail.com>
Message-ID: <27950.1406542880@critter.freebsd.dk>

--------
In message <BA130183-72D3-4D52-B594-8E4293FDA430 at gmail.com>, Carlos Abalde writes:

>Lately I've been wondering about the possibility of updating Varnish 
>VMODs with zero downtime. Being able of reloading VCL with zero downtime 
>is a great Varnish Feature; it reminds me a lot to hot code replacement 
>in Erlang, which is one of the killer features of the language. But what 
>about VMODs? Is this an existing / planned feature? Any workarounds 
>(apart from embedding all the VMOD logic in VCL using inline C code)?

Just reload your VCL and the VMODs should be updated too.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From apj at mutt.dk  Mon Jul 28 10:24:26 2014
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Mon, 28 Jul 2014 12:24:26 +0200
Subject: Updating VMODs with zero downtime
In-Reply-To: <BA130183-72D3-4D52-B594-8E4293FDA430@gmail.com>
References: <BA130183-72D3-4D52-B594-8E4293FDA430@gmail.com>
Message-ID: <20140728102426.GE19870@nerd.dk>

On Mon, Jul 28, 2014 at 12:18:59PM +0200, Carlos Abalde wrote:
> 
> Lately I've been wondering about the possibility of updating Varnish VMODs
> with zero downtime. Being able of reloading VCL with zero downtime is a great
> Varnish Feature; it reminds me a lot to hot code replacement in Erlang, which
> is one of the killer features of the language. But what about VMODs? Is this
> an existing / planned feature? Any workarounds (apart from embedding all the
> VMOD logic in VCL using inline C code)?

If you reload VCL, it should reload VMODs too.

-- 
Andreas



From suyogs at rediff-inc.com  Mon Jul 28 10:24:27 2014
From: suyogs at rediff-inc.com (Suyog Shirgaonkar)
Date: 28 Jul 2014 10:24:27 -0000
Subject: =?utf-8?B?dmNsX21pc3MgZm9yIFBVUkdFIGluIHZhcm5pc2ggNC4w?=
Message-ID: <20140728102427.4196.qmail@pro236-134.mxout.rediffmailpro.com>

Hello,Have we obsolete use of vcl_miss with version 4.0.1 for purge calls. Objects not available in cache also responds with 200 Purged as the calls to purge is always successful. Is there any remedy available to distinguish if the objects are present in the cache or not after i send a PURGE call. Below is a sample purge call which should have missed. I am calling synthentic response after it lands upto vcl_synth.MY VCL:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (req.method == &quot;PURGE&quot;) {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (!client.ip ~ purge) {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return (synth(000, &quot;Not Allowed&quot;));&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return (purge);&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }Varnishlog:-&nbsp;&nbsp; ReqMethod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PURGE-&nbsp;&nbsp; ReqURL&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; XXXXXXXXXXXXXXXXXXXXXXXXXXXX-&nbsp;&nbsp; ReqProtocol&nbsp;&nbsp;&nbsp; HTTP/1.1-&nbsp;&nbsp; ReqHeader&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; User-Agent: XXXXXXXXXXXXXXXXXXXXXXXXXX-&nbsp;&nbsp; ReqHeader&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Accept: */*-&nbsp;&nbsp; ReqHeader&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Host:XXXXXXXXXXXXXXXXXXXXXXXXX-&nbsp;&nbsp; ReqHeader&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; X-Forwarded-For: ::1-&nbsp;&nbsp; VCL_call&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RECV-&nbsp;&nbsp; ReqUnset&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Host:XXXXXXXXXXXXXXXX-&nbsp;&nbsp; ReqHeader&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; host: XXXXXXXXXXXXXX-&nbsp;&nbsp; VCL_acl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; MATCH purge &quot;localhost&quot;-&nbsp;&nbsp; VCL_acl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; MATCH purge &quot;localhost&quot;-&nbsp;&nbsp; VCL_return&nbsp;&nbsp;&nbsp;&nbsp; purge-&nbsp;&nbsp; VCL_call&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; HASH-&nbsp;&nbsp; VCL_return&nbsp;&nbsp;&nbsp;&nbsp; lookup-&nbsp;&nbsp; VCL_call&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PURGE-&nbsp;&nbsp; Debug&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;VCL_error(200, Purged)&quot;-&nbsp;&nbsp; VCL_return&nbsp;&nbsp;&nbsp;&nbsp; synth-&nbsp;&nbsp; Timestamp&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Process: 1406541624.863881 0.000110 0.000110-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Date: Mon, 28 Jul 2014 10:00:24 GMT-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Server: Varnish-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; X-Varnish: 3607188-&nbsp;&nbsp; RespProtocol&nbsp;&nbsp; HTTP/1.1-&nbsp;&nbsp; RespStatus&nbsp;&nbsp;&nbsp;&nbsp; 200-&nbsp;&nbsp; RespReason&nbsp;&nbsp;&nbsp;&nbsp; OK-&nbsp;&nbsp; RespReason&nbsp;&nbsp;&nbsp;&nbsp; Purged-&nbsp;&nbsp; VCL_call&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SYNTH-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Content-Type: text/html; charset=utf-8-&nbsp;&nbsp; VCL_return&nbsp;&nbsp;&nbsp;&nbsp; deliver-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Content-Length: 602-&nbsp;&nbsp; Debug&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;RES_MODE 2&quot;-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Connection: keep-alive-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Accept-Ranges: bytes-&nbsp;&nbsp; Timestamp&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Resp: 1406541624.863990 0.000220 0.000110-&nbsp;&nbsp; ReqAcct&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 212 0 212 204 602 806-&nbsp;&nbsp; EndRegards,Suyog Shirgaonkar&nbsp;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140728/564d6bff/attachment.html>

From carlos.abalde at gmail.com  Mon Jul 28 11:38:49 2014
From: carlos.abalde at gmail.com (Carlos Abalde)
Date: Mon, 28 Jul 2014 13:38:49 +0200
Subject: Updating VMODs with zero downtime
In-Reply-To: <27950.1406542880@critter.freebsd.dk>
References: <BA130183-72D3-4D52-B594-8E4293FDA430@gmail.com>
 <27950.1406542880@critter.freebsd.dk>
Message-ID: <A1B68F58-DFD7-4E03-945D-6AF583B545D1@gmail.com>

On 28 Jul 2014, at 12:21, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:

>> Lately I've been wondering about the possibility of updating Varnish 
>> VMODs with zero downtime. Being able of reloading VCL with zero downtime 
>> is a great Varnish Feature; it reminds me a lot to hot code replacement 
>> ...
> 
> Just reload your VCL and the VMODs should be updated too.

Thanks.

That?s great. However, I?ve been testing with Varnish 3.0.3 running in CentOS 6.5, using a minimal VCL importing the example VMOD (https://github.com/varnish/libvmod-example) to generate a synthetic response using the example.hello() method. After changing the ?Hello? string included in the VMOD by something else, recompiling and reinstalling the VMOD, the behaviour does not change when reloading the service. Only a restart of the service seems to work to update the VMOD.

Maybe I?m missing something here. I?ll do some more testing using the CLI?

Thanks,

?
Carlos Abalde.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140728/c7205d52/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140728/c7205d52/attachment.bin>

From carlos.abalde at gmail.com  Mon Jul 28 15:11:21 2014
From: carlos.abalde at gmail.com (Carlos Abalde)
Date: Mon, 28 Jul 2014 17:11:21 +0200
Subject: Updating VMODs with zero downtime
In-Reply-To: <27950.1406542880@critter.freebsd.dk>
References: <BA130183-72D3-4D52-B594-8E4293FDA430@gmail.com>
 <27950.1406542880@critter.freebsd.dk>
Message-ID: <2EB395BA-A8A4-4458-AD5F-C5F7EF0E3960@gmail.com>

On 28 Jul 2014, at 12:21, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:

>> Lately I've been wondering about the possibility of updating Varnish 
>> VMODs with zero downtime. Being able of reloading VCL with zero downtime 
>> is a great Varnish Feature; it reminds me a lot to hot code replacement 
>> in Erlang, which is one of the killer features of the language. But what 
>> about VMODs? Is this an existing / planned feature? Any workarounds 
>> (apart from embedding all the VMOD logic in VCL using inline C code)?
> 
> Just reload your VCL and the VMODs should be updated too.

Hi,

I've been testing this today, both with Varnish Plus 3.0.3-6b280ef and Varnish Cache 3.0.5-1a89b1f, and using two newly installed VMs (CentOS 6.5 and Ubuntu Precise). I used the example VMOD as PoC (https://github.com/varnish/libvmod-example), and the VCL below:

    backend default {
      .host = "127.0.0.1";
      .port = "9000";
    }

    import example;

    sub vcl_recv {
      error 404;
    }

    sub vcl_deliver {
      set resp.http.Test = example.hello("world");
    }

    sub vcl_error {
      synthetic example.hello("world");
      return (deliver);
    }

In all cases, any change in the VCL (i.e. replacing "world" by something else) was immediately reflected in Varnish once the service was reloaded. However, any change in the VMOD (i.e. replacing "Hello" by something else, followed by a compilation and installation of the VMOD), was not reflected when reloading the service. It only worked as expected after a restart of the service. 

Sounds this right to you? Am I missing something here?

Thanks!

?
Carlos Abalde.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140728/ed929829/attachment.bin>

From dan at magnetic.com  Mon Jul 28 15:18:11 2014
From: dan at magnetic.com (Dan Crosta)
Date: Mon, 28 Jul 2014 11:18:11 -0400
Subject: High LRU nuked with plenty of cache space available
In-Reply-To: <5D103CE839D50E4CBC62C9FD7B83287CC9C04FF1@EXCN015.encara.local.ads>
References: <CAMwwTFx92fatckRqz=caCdpo6yTwN=opUM_cahJ4XMb4sxw5tQ@mail.gmail.com>
 <5D103CE839D50E4CBC62C9FD7B83287CC9C04FF1@EXCN015.encara.local.ads>
Message-ID: <CAMwwTFwsBuHEJqrgo=Mq6zJ_rxy03+TCAXGB9uM3vg9KQSJzhQ@mail.gmail.com>

On Mon, Jul 28, 2014 at 9:35 AM, MAGNIEN, Thierry
<thierry.magnien at sfr.com> wrote:
> Hi Dan,
>
> Things you should check first are :
> - varnishlog for "could not get storage" errors

OK -- I'm grepping varnishlog, will let you know if I see anything there.


> - disk activity/performance: maybe varnish cannot get storage because your disks are responding too slowly

I thought the "file" storage class was mmap'd storage, and so writes
should basically hit memory and be periodically background flushed to
disk, is that not true? I'll keep an eye on await in iostat and let
you know how it looks. Is there a particular threshold I should be
watching out for? Is the threshold tunable?


Thanks,
- Dan
MAGNE+IC
Dan Crosta | Director, Engineering


On Mon, Jul 28, 2014 at 9:35 AM, MAGNIEN, Thierry
<thierry.magnien at sfr.com> wrote:
> Hi Dan,
>
> Things you should check first are :
> - varnishlog for "could not get storage" errors
> - disk activity/performance: maybe varnish cannot get storage because your disks are responding too slowly
>
> Regards,
> Thierry
>
> -----Message d'origine-----
> De : varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Dan Crosta
> Envoy? : samedi 26 juillet 2014 14:58
> ? : varnish-misc at varnish-cache.org
> Objet : High LRU nuked with plenty of cache space available
>
> We're running varnish (3.0.4) like:
>
> /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f
> /etc/varnish/default.vcl -T 127.0.0.1:8080 -t 2628000 -w 50,1000,120
> -u varnish -g varnish -S /etc/varnish/secret -s
> file,/srv/varnish/varnish_storage.bin,12G -p thread_pools=4 -p
> http_gzip_support=off -p saintmode_threshold=0 -p
> http_range_support=off
>
> varnishstat shows the following for SMF:
>
> SMF.s0.c_req            45213612       317.34 Allocator requests
> SMF.s0.c_fail           14781854       103.75 Allocator failures
> SMF.s0.c_bytes      2056164147200  14431753.97 Bytes allocated
> SMF.s0.c_freed      2052654858240  14407123.06 Bytes freed
> SMF.s0.g_alloc            856760          .   Allocations outstanding
> SMF.s0.g_bytes        3509288960          .   Bytes outstanding
> SMF.s0.g_space        9375612928          .   Bytes available
> SMF.s0.g_smf              950080          .   N struct smf
> SMF.s0.g_smf_frag          93319          .   N small free smf
> SMF.s0.g_smf_large             1          .   N large free smf
>
> which I interpret to mean that ~9G of the 12G are available ("bytes
> available") for use by the cache.
>
> However, when our applications service the API through varnish at a
> rate of about 150 QPS, we see that n_lru_nuked increases at about the
> same rate, the result of which is that the n_object counter stays more
> or less constant. The cache has only been running for a few days, so I
> don't believe TTL is to blame here, since we use -t with about 30 days
> (assuming the value there is in seconds, which the docs seem to
> imply).
>
> I have tried setting a granularity on the storage, which did not seem
> to have any impact. I'm looking for other suggestions or things to
> try, as intuition and the stats seem to suggest we should be able to
> store a lot more objects before things start to get nuked.
>
> Thanks,
> - Dan
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From thierry.magnien at sfr.com  Mon Jul 28 15:22:57 2014
From: thierry.magnien at sfr.com (MAGNIEN, Thierry)
Date: Mon, 28 Jul 2014 15:22:57 +0000
Subject: High LRU nuked with plenty of cache space available
In-Reply-To: <CAMwwTFwsBuHEJqrgo=Mq6zJ_rxy03+TCAXGB9uM3vg9KQSJzhQ@mail.gmail.com>
References: <CAMwwTFx92fatckRqz=caCdpo6yTwN=opUM_cahJ4XMb4sxw5tQ@mail.gmail.com>
 <5D103CE839D50E4CBC62C9FD7B83287CC9C04FF1@EXCN015.encara.local.ads>
 <CAMwwTFwsBuHEJqrgo=Mq6zJ_rxy03+TCAXGB9uM3vg9KQSJzhQ@mail.gmail.com>
Message-ID: <5D103CE839D50E4CBC62C9FD7B83287CC9C05202@EXCN015.encara.local.ads>

You're right, file storage is a mmap, but once varnish decides to write to disk, this can result in a very large performance penalty, depending on your disks.

With 12G, I would basically suggest to switch to RAM storage, unless you don't have enough.

Regards,
Thierry

-----Message d'origine-----
De?: varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Dan Crosta
Envoy??: lundi 28 juillet 2014 17:18
??: varnish-misc at varnish-cache.org
Objet?: Re: High LRU nuked with plenty of cache space available

On Mon, Jul 28, 2014 at 9:35 AM, MAGNIEN, Thierry
<thierry.magnien at sfr.com> wrote:
> Hi Dan,
>
> Things you should check first are :
> - varnishlog for "could not get storage" errors

OK -- I'm grepping varnishlog, will let you know if I see anything there.


> - disk activity/performance: maybe varnish cannot get storage because your disks are responding too slowly

I thought the "file" storage class was mmap'd storage, and so writes
should basically hit memory and be periodically background flushed to
disk, is that not true? I'll keep an eye on await in iostat and let
you know how it looks. Is there a particular threshold I should be
watching out for? Is the threshold tunable?


Thanks,
- Dan
MAGNE+IC
Dan Crosta | Director, Engineering


On Mon, Jul 28, 2014 at 9:35 AM, MAGNIEN, Thierry
<thierry.magnien at sfr.com> wrote:
> Hi Dan,
>
> Things you should check first are :
> - varnishlog for "could not get storage" errors
> - disk activity/performance: maybe varnish cannot get storage because your disks are responding too slowly
>
> Regards,
> Thierry
>
> -----Message d'origine-----
> De : varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Dan Crosta
> Envoy? : samedi 26 juillet 2014 14:58
> ? : varnish-misc at varnish-cache.org
> Objet : High LRU nuked with plenty of cache space available
>
> We're running varnish (3.0.4) like:
>
> /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f
> /etc/varnish/default.vcl -T 127.0.0.1:8080 -t 2628000 -w 50,1000,120
> -u varnish -g varnish -S /etc/varnish/secret -s
> file,/srv/varnish/varnish_storage.bin,12G -p thread_pools=4 -p
> http_gzip_support=off -p saintmode_threshold=0 -p
> http_range_support=off
>
> varnishstat shows the following for SMF:
>
> SMF.s0.c_req            45213612       317.34 Allocator requests
> SMF.s0.c_fail           14781854       103.75 Allocator failures
> SMF.s0.c_bytes      2056164147200  14431753.97 Bytes allocated
> SMF.s0.c_freed      2052654858240  14407123.06 Bytes freed
> SMF.s0.g_alloc            856760          .   Allocations outstanding
> SMF.s0.g_bytes        3509288960          .   Bytes outstanding
> SMF.s0.g_space        9375612928          .   Bytes available
> SMF.s0.g_smf              950080          .   N struct smf
> SMF.s0.g_smf_frag          93319          .   N small free smf
> SMF.s0.g_smf_large             1          .   N large free smf
>
> which I interpret to mean that ~9G of the 12G are available ("bytes
> available") for use by the cache.
>
> However, when our applications service the API through varnish at a
> rate of about 150 QPS, we see that n_lru_nuked increases at about the
> same rate, the result of which is that the n_object counter stays more
> or less constant. The cache has only been running for a few days, so I
> don't believe TTL is to blame here, since we use -t with about 30 days
> (assuming the value there is in seconds, which the docs seem to
> imply).
>
> I have tried setting a granularity on the storage, which did not seem
> to have any impact. I'm looking for other suggestions or things to
> try, as intuition and the stats seem to suggest we should be able to
> store a lot more objects before things start to get nuked.
>
> Thanks,
> - Dan
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

From dan at magnetic.com  Mon Jul 28 16:55:39 2014
From: dan at magnetic.com (Dan Crosta)
Date: Mon, 28 Jul 2014 12:55:39 -0400
Subject: High LRU nuked with plenty of cache space available
In-Reply-To: <5D103CE839D50E4CBC62C9FD7B83287CC9C05202@EXCN015.encara.local.ads>
References: <CAMwwTFx92fatckRqz=caCdpo6yTwN=opUM_cahJ4XMb4sxw5tQ@mail.gmail.com>
 <5D103CE839D50E4CBC62C9FD7B83287CC9C04FF1@EXCN015.encara.local.ads>
 <CAMwwTFwsBuHEJqrgo=Mq6zJ_rxy03+TCAXGB9uM3vg9KQSJzhQ@mail.gmail.com>
 <5D103CE839D50E4CBC62C9FD7B83287CC9C05202@EXCN015.encara.local.ads>
Message-ID: <CAMwwTFyTb45=SEq22u=7Hd=6LTa0j6j6U5CAtnVhX=jhG8TwRw@mail.gmail.com>

OK, after our service was hitting varnish for about 30 minutes, I see
a total of 2 "could not get storage" errors in the logs, and disk
await and util are both low (well, await is around 15ms, but
utilization is about 5-7%).

We can try switching to malloc storage. I'll update here when we have
a chance to do that.
MAGNE+IC
Dan Crosta | Director, Engineering


On Mon, Jul 28, 2014 at 11:22 AM, MAGNIEN, Thierry
<thierry.magnien at sfr.com> wrote:
> You're right, file storage is a mmap, but once varnish decides to write to disk, this can result in a very large performance penalty, depending on your disks.
>
> With 12G, I would basically suggest to switch to RAM storage, unless you don't have enough.
>
> Regards,
> Thierry
>
> -----Message d'origine-----
> De : varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Dan Crosta
> Envoy? : lundi 28 juillet 2014 17:18
> ? : varnish-misc at varnish-cache.org
> Objet : Re: High LRU nuked with plenty of cache space available
>
> On Mon, Jul 28, 2014 at 9:35 AM, MAGNIEN, Thierry
> <thierry.magnien at sfr.com> wrote:
>> Hi Dan,
>>
>> Things you should check first are :
>> - varnishlog for "could not get storage" errors
>
> OK -- I'm grepping varnishlog, will let you know if I see anything there.
>
>
>> - disk activity/performance: maybe varnish cannot get storage because your disks are responding too slowly
>
> I thought the "file" storage class was mmap'd storage, and so writes
> should basically hit memory and be periodically background flushed to
> disk, is that not true? I'll keep an eye on await in iostat and let
> you know how it looks. Is there a particular threshold I should be
> watching out for? Is the threshold tunable?
>
>
> Thanks,
> - Dan
> MAGNE+IC
> Dan Crosta | Director, Engineering
>
>
> On Mon, Jul 28, 2014 at 9:35 AM, MAGNIEN, Thierry
> <thierry.magnien at sfr.com> wrote:
>> Hi Dan,
>>
>> Things you should check first are :
>> - varnishlog for "could not get storage" errors
>> - disk activity/performance: maybe varnish cannot get storage because your disks are responding too slowly
>>
>> Regards,
>> Thierry
>>
>> -----Message d'origine-----
>> De : varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Dan Crosta
>> Envoy? : samedi 26 juillet 2014 14:58
>> ? : varnish-misc at varnish-cache.org
>> Objet : High LRU nuked with plenty of cache space available
>>
>> We're running varnish (3.0.4) like:
>>
>> /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f
>> /etc/varnish/default.vcl -T 127.0.0.1:8080 -t 2628000 -w 50,1000,120
>> -u varnish -g varnish -S /etc/varnish/secret -s
>> file,/srv/varnish/varnish_storage.bin,12G -p thread_pools=4 -p
>> http_gzip_support=off -p saintmode_threshold=0 -p
>> http_range_support=off
>>
>> varnishstat shows the following for SMF:
>>
>> SMF.s0.c_req            45213612       317.34 Allocator requests
>> SMF.s0.c_fail           14781854       103.75 Allocator failures
>> SMF.s0.c_bytes      2056164147200  14431753.97 Bytes allocated
>> SMF.s0.c_freed      2052654858240  14407123.06 Bytes freed
>> SMF.s0.g_alloc            856760          .   Allocations outstanding
>> SMF.s0.g_bytes        3509288960          .   Bytes outstanding
>> SMF.s0.g_space        9375612928          .   Bytes available
>> SMF.s0.g_smf              950080          .   N struct smf
>> SMF.s0.g_smf_frag          93319          .   N small free smf
>> SMF.s0.g_smf_large             1          .   N large free smf
>>
>> which I interpret to mean that ~9G of the 12G are available ("bytes
>> available") for use by the cache.
>>
>> However, when our applications service the API through varnish at a
>> rate of about 150 QPS, we see that n_lru_nuked increases at about the
>> same rate, the result of which is that the n_object counter stays more
>> or less constant. The cache has only been running for a few days, so I
>> don't believe TTL is to blame here, since we use -t with about 30 days
>> (assuming the value there is in seconds, which the docs seem to
>> imply).
>>
>> I have tried setting a granularity on the storage, which did not seem
>> to have any impact. I'm looking for other suggestions or things to
>> try, as intuition and the stats seem to suggest we should be able to
>> store a lot more objects before things start to get nuked.
>>
>> Thanks,
>> - Dan
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From suyogs at rediff-inc.com  Wed Jul 30 05:48:01 2014
From: suyogs at rediff-inc.com (Suyog Shirgaonkar)
Date: 30 Jul 2014 05:48:01 -0000
Subject: =?utf-8?B?UmU6IHZjbF9taXNzIGZvciBQVVJHRSBpbiB2YXJuaXNoIDQuMA==?=
Message-ID: <1406543071.S.8405.RU.psfs5, psfs5, 203,
 565.5797.pro236-134.rediffmailpro.com.old.1406699281.19826@webmail.rediffmail.com>

Hello,Guys please help with this issue.Regards,SuyogFrom: Suyog Shirgaonkar&lt;suyogs at rediff-inc.com&gt;Sent: Mon, 28 Jul 2014 15:54:31To: &lt;varnish-misc at varnish-cache.org&gt;Subject: vcl_miss for PURGE in varnish 4.0Hello,Have we obsolete use of vcl_miss with version 4.0.1 for purge calls. Objects not available in cache also responds with 200 Purged as the calls to purge is always successful. Is there any remedy available to distinguish if the objects are present in the cache or not after i send a PURGE call. Below is a sample purge call which should have missed. I am calling synthentic response after it lands upto vcl_synth.MY VCL:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (req.method == &quot;PURGE&quot;) {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (!client.ip ~ purge) {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return (synth(000, &quot;Not Allowed&quot;));&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return (purge);&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }Varnishlog:-&nbsp;&nbsp; ReqMethod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PURGE-&nbsp;&nbsp; ReqURL&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; XXXXXXXXXXXXXXXXXXXXXXXXXXXX-&nbsp;&nbsp; ReqProtocol&nbsp;&nbsp;&nbsp; HTTP/1.1-&nbsp;&nbsp; ReqHeader&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; User-Agent: XXXXXXXXXXXXXXXXXXXXXXXXXX-&nbsp;&nbsp; ReqHeader&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Accept: */*-&nbsp;&nbsp; ReqHeader&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Host:XXXXXXXXXXXXXXXXXXXXXXXXX-&nbsp;&nbsp; ReqHeader&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; X-Forwarded-For: ::1-&nbsp;&nbsp; VCL_call&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RECV-&nbsp;&nbsp; ReqUnset&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Host:XXXXXXXXXXXXXXXX-&nbsp;&nbsp; ReqHeader&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; host: XXXXXXXXXXXXXX-&nbsp;&nbsp; VCL_acl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; MATCH purge &quot;localhost&quot;-&nbsp;&nbsp; VCL_acl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; MATCH purge &quot;localhost&quot;-&nbsp;&nbsp; VCL_return&nbsp;&nbsp;&nbsp;&nbsp; purge-&nbsp;&nbsp; VCL_call&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; HASH-&nbsp;&nbsp; VCL_return&nbsp;&nbsp;&nbsp;&nbsp; lookup-&nbsp;&nbsp; VCL_call&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PURGE-&nbsp;&nbsp; Debug&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;VCL_error(200, Purged)&quot;-&nbsp;&nbsp; VCL_return&nbsp;&nbsp;&nbsp;&nbsp; synth-&nbsp;&nbsp; Timestamp&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Process: 1406541624.863881 0.000110 0.000110-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Date: Mon, 28 Jul 2014 10:00:24 GMT-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Server: Varnish-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; X-Varnish: 3607188-&nbsp;&nbsp; RespProtocol&nbsp;&nbsp; HTTP/1.1-&nbsp;&nbsp; RespStatus&nbsp;&nbsp;&nbsp;&nbsp; 200-&nbsp;&nbsp; RespReason&nbsp;&nbsp;&nbsp;&nbsp; OK-&nbsp;&nbsp; RespReason&nbsp;&nbsp;&nbsp;&nbsp; Purged-&nbsp;&nbsp; VCL_call&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SYNTH-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Content-Type: text/html; charset=utf-8-&nbsp;&nbsp; VCL_return&nbsp;&nbsp;&nbsp;&nbsp; deliver-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Content-Length: 602-&nbsp;&nbsp; Debug&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &quot;RES_MODE 2&quot;-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Connection: keep-alive-&nbsp;&nbsp; RespHeader&nbsp;&nbsp;&nbsp;&nbsp; Accept-Ranges: bytes-&nbsp;&nbsp; Timestamp&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Resp: 1406541624.863990 0.000220 0.000110-&nbsp;&nbsp; ReqAcct&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 212 0 212 204 602 806-&nbsp;&nbsp; EndRegards,Suyog Shirgaonkar&nbsp;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140730/e9fdc510/attachment.html>

From apj at mutt.dk  Wed Jul 30 07:28:36 2014
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Wed, 30 Jul 2014 09:28:36 +0200
Subject: vcl_miss for PURGE in varnish 4.0
Message-ID: <20140730072836.GF19870@nerd.dk>

On Wed, Jul 30, 2014 at 05:48:01AM -0000, Suyog Shirgaonkar wrote:

> Hello,Guys please help with this issue.

Please send your question in plain text. I can't read your mail, and I'm sure
others will have a problem too.

-- 
Andreas



From mattias at nucleus.be  Wed Jul 30 07:38:12 2014
From: mattias at nucleus.be (Mattias Geniar)
Date: Wed, 30 Jul 2014 07:38:12 +0000
Subject: Determine if grace object is available
In-Reply-To: <20140723130343.GB11654@immer.varnish-software.com>
References: <7C954D35-328D-49B1-B854-302A26CC1981@dragondata.com>
 <20140723130343.GB11654@immer.varnish-software.com>
Message-ID: <CFFE6C37.3FB2D%mattias@nucleus.be>


>> I?d like to get fancy with grace stored objects, but I?m not sure how
>>to do this. Can I determine if there?s a grace object I could deliver?
>>Basically I want my logic to be:

As a follow-up to this thread, I'm wondering if the following is possible,
given that there is a director present with 2 servers, having health
probes configured.

If the director has no healthy backends;
1. See if a grace object is available, if so, deliver it
2. If no grace object is available, change the backend to a "maintenance"
one to serve a static HTML page for maintenance purposes

The struggle is in vcl_recv {}, how would this be able to work? If I use
req.backend.healthy to determine the backend health to set a new backend,
I lose the grace ability (as it'll be passed to the new, available,
backend?). Or I'm missing something here.

# Use grace, when available and when needed
if (! req.backend.healthy) {
  # Backends are sick, so fall back to a stale object, if possible
  set req.grace = 5m;

  # If no stale object is available, how should we switch to a new backend
here?
  set req.backend = maintenance_backend;  # This could serve static pages
with maintenance info
}


I'm thing something like this, but it's not possible?
if (req.grace.available) {
  set req.grace = 5m;
} else {
  # No grace object available, set new backend
  set req.backend = maintenance_backend;
}

Thanks,

Mattias




From phk at phk.freebsd.dk  Wed Jul 30 07:58:45 2014
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 30 Jul 2014 07:58:45 +0000
Subject: Updating VMODs with zero downtime
In-Reply-To: <2EB395BA-A8A4-4458-AD5F-C5F7EF0E3960@gmail.com>
References: <BA130183-72D3-4D52-B594-8E4293FDA430@gmail.com>
 <27950.1406542880@critter.freebsd.dk>
 <2EB395BA-A8A4-4458-AD5F-C5F7EF0E3960@gmail.com>
Message-ID: <16375.1406707125@critter.freebsd.dk>

--------
In message <2EB395BA-A8A4-4458-AD5F-C5F7EF0E3960 at gmail.com>, Carlos Abalde writ
es:

>In all cases, any change in the VCL (i.e. replacing "world" by something 
>else) was immediately reflected in Varnish once the service was 
>reloaded. However, any change in the VMOD (i.e. replacing "Hello" by 
>something else, followed by a compilation and installation of the VMOD), 
>was not reflected when reloading the service. It only worked as expected 
>after a restart of the service. 
>
>Sounds this right to you? Am I missing something here?

So one murky bit here is if your OS's dlopen(3) implementation discovers
that the file has changed.  It sounds like it doesn't, and just reuses
the old already loaded copy :-(

The surefire way to fix that is to append a version number to the
vmod filename, but it's kind of ugly...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From suyogs at rediff-inc.com  Wed Jul 30 08:40:16 2014
From: suyogs at rediff-inc.com (Suyog Shirgaonkar)
Date: 30 Jul 2014 08:40:16 -0000
Subject: =?utf-8?B?UmU6IHZjbF9taXNzIGZvciBQVVJHRSBpbiB2YXJuaXNoIDQuMA==?=
Message-ID: <1406709920.S.37691.autosave.drafts.1406709616.21793@webmail.rediffmail.com>

Hello,Have we obsolete use of vcl_miss with version 4.0.1 for purge calls. Objects not available in cache also responds with 200 Purged as the calls to purge is always successful. Is there any remedy available to distinguish if the objects are present in the cache or not after i send a PURGE call. Below is a sample purge call which should have missed. I am calling synthentic response after it lands upto vcl_synth.PFARegards,Suyog ShirgaonkarFrom: Andreas Plesner Jacobsen &lt;apj at mutt.dk&gt;Sent: Wed, 30 Jul 2014 12:59:20To: varnish-misc at varnish-cache.orgSubject: Re: vcl_miss for PURGE in varnish 4.0On Wed, Jul 30, 2014 at 05:48:01AM -0000, Suyog Shirgaonkar wrote:&gt; Hello,Guys please help with this issue.Please send your question in plain text. I can&#39;t read your mail, and I&#39;m sureothers will have a problem too.--Andreas_______________________________________________varnish-misc mailing listvarnish-misc at varnish-cache.orghttps://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140730/022de471/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: New_Text_Document.txt
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140730/022de471/attachment.txt>

From apj at mutt.dk  Wed Jul 30 08:52:34 2014
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Wed, 30 Jul 2014 10:52:34 +0200
Subject: vcl_miss for PURGE in varnish 4.0
In-Reply-To: <1406709920.S.37691.autosave.drafts.1406709616.21793@webmail.rediffmail.com>
References: <1406709920.S.37691.autosave.drafts.1406709616.21793@webmail.rediffmail.com>
Message-ID: <20140730085234.GH19870@nerd.dk>

On Wed, Jul 30, 2014 at 08:40:16AM -0000, Suyog Shirgaonkar wrote:
> Hello,Have we obsolete use of vcl_miss with version 4.0.1 for purge calls.
> Objects not available in cache also responds with 200 Purged as the calls to
> purge is always successful. Is there any remedy available to distinguish if
> the objects are present in the cache or not after i send a PURGE call. Below

The object is by definition not present after a purge, so I'm not sure I
understand the question.

You can fetch the object with a new request and see if you hit vcl_miss. But be
aware that other clients could have put it in the cache between your purge and
your fetch.

-- 
Andreas



From suyogs at rediff-inc.com  Thu Jul 31 11:15:58 2014
From: suyogs at rediff-inc.com (Suyog Shirgaonkar)
Date: 31 Jul 2014 11:15:58 -0000
Subject: =?utf-8?B?SmF2YXNjcmlwdCBQT1NUIHJlcXVlc3Qgc3VwcG9ydA==?=
Message-ID: <20140731111558.15910.qmail@pro236-138.mxout.rediffmailpro.com>

Hi,

While using javascript file upload script, we are getting session closed error in varnishlog.
We are using Varnish 4.0.1. Earlier in varnish 3.0X we were using below code which is working fine with 3.0 version. 
But After upgrading we are unable to process the request.:


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (req.http.host ~ "^XXXX.com") {
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return (pass);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }




*&nbsp;&nbsp; &lt;&lt; Session&nbsp; &gt;&gt; 131106
-&nbsp;&nbsp; Begin&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sess 0 HTTP/1
-&nbsp;&nbsp; SessOpen&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 119.252.159.240 11622 :80 202.137.239.228 80 1406805322.102382 16
-&nbsp;&nbsp; Link&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; req 131107 rxreq
-&nbsp;&nbsp; Link&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; req 131108 rxreq
-&nbsp;&nbsp; SessClose&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; REM_CLOSE 0.716
-&nbsp;&nbsp; End
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140731/9d951e55/attachment.html>

From carlos.abalde at gmail.com  Thu Jul 31 11:19:36 2014
From: carlos.abalde at gmail.com (Carlos Abalde)
Date: Thu, 31 Jul 2014 13:19:36 +0200
Subject: Updating VMODs with zero downtime
In-Reply-To: <16375.1406707125@critter.freebsd.dk>
References: <BA130183-72D3-4D52-B594-8E4293FDA430@gmail.com>
 <27950.1406542880@critter.freebsd.dk>
 <2EB395BA-A8A4-4458-AD5F-C5F7EF0E3960@gmail.com>
 <16375.1406707125@critter.freebsd.dk>
Message-ID: <C3F5B55B-D63A-4D60-BF17-5B5F7A00D720@gmail.com>

On 30 Jul 2014, at 09:58, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:

> So one murky bit here is if your OS's dlopen(3) implementation discovers
> that the file has changed.  It sounds like it doesn't, and just reuses
> the old already loaded copy :-(
> 
> The surefire way to fix that is to append a version number to the
> vmod filename, but it's kind of ugly...

Hi,

I've been checking the sources of Varnish and doing some more testing, and, as you suggest, this definitely seems related to dlopen(3) / dlclose(3) and how they are used when reloading Varnish. 

When reloading Varnish, the current VCL is deactivated and the new one loaded and activated. Assuming both the old and the new VCL import the VMOD to be updated with zero downtime, reference counting for that VMOD never drops to 0 (in fact the number of references is going to be 2 after reloading the service).This implies that dlclose(3) will never be called for the VMOD (VRT_Vmod_Fini), and further calls to dlopen(3) will always return a handle to the original VMOD, even if the VMOD / library changed on disk. This sounds consistent with the explanation in the man page of dlopen(3):

       If  the  same library is loaded again with dlopen(), the same file handle is returned.  The dl library maintains reference counts for library handles, so a dynamic library is not deallocated until dlclose(3) has been called on it as many times as dlopen() has succeeded on it.


I've also checked that if the original VCL is replaced by a new intermediate VCL not loading the VMOD, then the original VCL is discarded using the CLI (which generates a call to dlclose(3)), and finally the intermediate VCL is again replaced with the original one, then the VMOD is correctly reloaded from disk.


I considered copying the new VMOD to a different file and using something like "import example from "/usr/lib64/varnish/vmods/libvmod_example_new.so";", but it does not work. When reloading Varnish it checks that a VMOD with that name is already loaded and does not execute the dlopen(3) (VRT_Vmod_Init). 


In summary, I'm afraid that your suggestion of versioning module names that may require upgrades with zero downtime is the only option here... :/


Thanks!

?
Carlos Abalde.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20140731/93a981d0/attachment.bin>