From des at linpro.no  Sun Jul  1 08:53:03 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sun, 01 Jul 2007 10:53:03 +0200
Subject: Problem with varnish and caching
In-Reply-To: <1183248616.3955.7.camel@gabriela> (Manuel Amador's message of
	"Sat\, 30 Jun 2007 19\:10\:16 -0500")
References: <1183146070.32019.20.camel@gabriela>
	<87ejjt15nd.fsf@des.linpro.no> <1183248616.3955.7.camel@gabriela>
Message-ID: <871wfs1p5c.fsf@des.linpro.no>

"Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> In effect, the default Varnish policy of not caching Cookied requests
> causes Varnish not to cache anything at all for most sites (you know,
> there are tons of people out there using Google Analytics).  Think about
> it: why would people want the overhead of a non-caching accelerating
> proxy?

Perhaps your assumptions are flawed?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Sun Jul  1 08:54:28 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sun, 01 Jul 2007 10:54:28 +0200
Subject: utma utmz
In-Reply-To: <1183248418.3955.3.camel@gabriela> (Manuel Amador's message of
	"Sat\, 30 Jun 2007 19\:06\:58 -0500")
References: <1183147489.32019.23.camel@gabriela>
	<87ir9515x1.fsf@des.linpro.no> <1183248418.3955.3.camel@gabriela>
Message-ID: <87wsxkzepn.fsf@des.linpro.no>

"Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> Then another question (which will surely help me MUCH MORE than this
> conundrum).  I want to alter the Varnish caching policy to cache objects
> whose response contains an ETag or a Last-Modified header, irrespective
> of whether the user sent cookies along.  That'll allow me to make
> Varnish cache static objects while ignoring dynamic (non-Last-Modified,
> non-ETag) objects.

This still boils down to the same issue as your previous questions, and
the answer is still that we can't do that yet, but we will in a few
weeks.

> Bonus question: I would like the Last-Modified header to be used by
> Varnish as the basis for an adaptive caching algorith just like Squid's,
> instead of caching files for 120 seconds.

See above.

> Third question: does Varnish obey Pragma: no-cache in response headers?

See above.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rudd-o at rudd-o.com  Sun Jul  1 00:06:58 2007
From: rudd-o at rudd-o.com (Manuel Amador (Rudd-O))
Date: Sat, 30 Jun 2007 19:06:58 -0500
Subject: utma utmz
In-Reply-To: <87ir9515x1.fsf@des.linpro.no>
References: <1183147489.32019.23.camel@gabriela> <87ir9515x1.fsf@des.linpro.no>
Message-ID: <1183248418.3955.3.camel@gabriela>

Then another question (which will surely help me MUCH MORE than this
conundrum).  I want to alter the Varnish caching policy to cache objects
whose response contains an ETag or a Last-Modified header, irrespective
of whether the user sent cookies along.  That'll allow me to make
Varnish cache static objects while ignoring dynamic (non-Last-Modified,
non-ETag) objects.

Bonus question: I would like the Last-Modified header to be used by
Varnish as the basis for an adaptive caching algorith just like Squid's,
instead of caching files for 120 seconds.

Third question: does Varnish obey Pragma: no-cache in response headers?

On s?b, 2007-06-30 at 23:36 +0200, Dag-Erling Sm?rgrav wrote:
> [moved from -dev to -misc]
> 
> "Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> > My site always feeds the utma and utmz (google analytics) cookies to
> > requests.  How can I specify in the varnish configuration file
> > default.vcl that if *only* those two cookies are present, it should
> > supply files from cache, but if any other cookies are present, it should
> > contact the backend server (or use the appropriate caching policy)?
> 
> There's no easy way to do that with the current code base, but we expect
> to have the required functionality in place within a couple of weeks.
> 
> DES
	Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
	The R Zone - http://rudd-o.com/
	GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: 
Your mode of life will be changed for the better because of good news soon.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070630/04105df5/attachment.bin>

From rudd-o at rudd-o.com  Sun Jul  1 00:10:16 2007
From: rudd-o at rudd-o.com (Manuel Amador (Rudd-O))
Date: Sat, 30 Jun 2007 19:10:16 -0500
Subject: Problem with varnish and caching
In-Reply-To: <87ejjt15nd.fsf@des.linpro.no>
References: <1183146070.32019.20.camel@gabriela> <87ejjt15nd.fsf@des.linpro.no>
Message-ID: <1183248616.3955.7.camel@gabriela>

Whether this is part of a different specification or not, it's
unreasonable for Varnish to NOT cache static, cacheable objects by
default, solely because these requests were sent along with Cookies.

In effect, the default Varnish policy of not caching Cookied requests
causes Varnish not to cache anything at all for most sites (you know,
there are tons of people out there using Google Analytics).  Think about
it: why would people want the overhead of a non-caching accelerating
proxy?

On s?b, 2007-06-30 at 23:41 +0200, Dag-Erling Sm?rgrav wrote:
> [moved from -dev to -misc]
> 
> "Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> > As you might already have noted, I reported a bug on varnish caching
> > files indiscriminately.
> 
> This is not a bug, it is a misunderstanding.  It appears you expect
> Varnish to act like an RFC 2616 "shared cache" whereas it is in fact a
> "surrogate" (see the "Edge Architecture Specification" by Oracle and
> Akamai, although Varnish does not yet fully implement that specification
> either)
> 
> > My page sets a few cookies.  That'd be okay and it should produce
> > dynamic pages, which Varnish is furnishing through my backend.  The
> > thing is, these cookies are sent along requests for CSS and PNG and JPG
> > and JS files, which causes varnish to contact the backend.  I don't want
> > that to happen (I'm happy with them being cached by Varnish 120
> > seconds).
> >
> > How can I tell Varnish that requests with a response that includes ETag
> > (a discriminant for static files) should be forcibly cached?
> 
> This is basically the same issue as in your previous email, and the
> answer is the same.
> 
> DES
	Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
	The R Zone - http://rudd-o.com/
	GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: 
Small things make base men proud.
		-- William Shakespeare, "Henry VI"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070630/10af24f2/attachment.bin>

From phk at phk.freebsd.dk  Sun Jul  1 09:05:41 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 01 Jul 2007 09:05:41 +0000
Subject: Problem with varnish and caching 
In-Reply-To: Your message of "Sat, 30 Jun 2007 19:10:16 EST."
	<1183248616.3955.7.camel@gabriela> 
Message-ID: <24569.1183280741@critter.freebsd.dk>


Dear Manuel,

The Varnish *DEFAULT* VCL code implements what we have decided is a
sensible default policy, which will Do The Right Thing for most people,
at least to get them going with Varnish.

Your complaints all seem to center on the fact that the default
VCL code doesn't work for you.

You seem to have overlooked the fact that you are not forced to
run with the default VCL code.

Varnish is on track to offer the most comprehensive and flexible
configuration mechanism yet to be seen on any web server or cache.

Now, instead of whining about the default, tailor Varnish to your
situation.

Thankyou!

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From andersb at vgnett.no  Sun Jul  1 14:07:04 2007
From: andersb at vgnett.no (Anders Berg)
Date: Sun, 01 Jul 2007 16:07:04 +0200
Subject: Problem with varnish and caching
In-Reply-To: <mailman.3.1183284002.1965.varnish-misc@projects.linpro.no>
References: <mailman.3.1183284002.1965.varnish-misc@projects.linpro.no>
Message-ID: <4687B508.4040605@vgnett.no>

Hi,

> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Sat, 30 Jun 2007 19:10:16 -0500
> From: "Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com>
> Subject: Re: Problem with varnish and caching
> To: Dag-Erling Sm?rgrav <des at linpro.no>
> Cc: varnish-misc at projects.linpro.no
> Message-ID: <1183248616.3955.7.camel at gabriela>
> Content-Type: text/plain; charset="utf-8"
> 
> Whether this is part of a different specification or not, it's
> unreasonable for Varnish to NOT cache static, cacheable objects by
> default, solely because these requests were sent along with Cookies.

I am not aware of any HTTP accelerator that would caching this by 
default. Since I believe it is a direct violation of all standards. 
Think of it this way: Why cache pages that are said to be dynamic by the 
webpage.

Squid and Bluecoats will not cache these pages, and the mechanism(s) to 
tune that are far from fine grained.

> In effect, the default Varnish policy of not caching Cookied requests
> causes Varnish not to cache anything at all for most sites (you know,
> there are tons of people out there using Google Analytics).  Think about
> it: why would people want the overhead of a non-caching accelerating
> proxy?

Here you assume "most sites" use Google Analytics. I am not so sure 
about that. Either way, while Varnish will suite almost any site you can 
imagine, it's strengths will be most appreciated by "larger" sites. With 
larger sites I mean sites that would not dream of using Google Analytics.

YS
Anders Berg


> On s?b, 2007-06-30 at 23:41 +0200, Dag-Erling Sm?rgrav wrote:
>> [moved from -dev to -misc]
>>
>> "Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
>>> As you might already have noted, I reported a bug on varnish caching
>>> files indiscriminately.
>> This is not a bug, it is a misunderstanding.  It appears you expect
>> Varnish to act like an RFC 2616 "shared cache" whereas it is in fact a
>> "surrogate" (see the "Edge Architecture Specification" by Oracle and
>> Akamai, although Varnish does not yet fully implement that specification
>> either)
>>
>>> My page sets a few cookies.  That'd be okay and it should produce
>>> dynamic pages, which Varnish is furnishing through my backend.  The
>>> thing is, these cookies are sent along requests for CSS and PNG and JPG
>>> and JS files, which causes varnish to contact the backend.  I don't want
>>> that to happen (I'm happy with them being cached by Varnish 120
>>> seconds).
>>>
>>> How can I tell Varnish that requests with a response that includes ETag
>>> (a discriminant for static files) should be forcibly cached?
>> This is basically the same issue as in your previous email, and the
>> answer is the same.
>>
>> DES
> 	Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
> 	The R Zone - http://rudd-o.com/
> 	GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/
> 
> Now playing, courtesy of Amarok: 
> Small things make base men proud.
> 		-- William Shakespeare, "Henry VI"


From denis at zeno.org  Sun Jul  1 16:01:09 2007
From: denis at zeno.org (Denis Ahrens)
Date: Sun, 1 Jul 2007 18:01:09 +0200
Subject: missing Content-Lenght header
In-Reply-To: <87abuh15h2.fsf@des.linpro.no>
References: <13906.1183229822@critter.freebsd.dk>
	<A0D94030-38D9-4737-90CA-02FE4B6CB475@zeno.org>
	<87abuh15h2.fsf@des.linpro.no>
Message-ID: <3C33EDA7-EE30-4017-B5FC-B9E87A22AA38@zeno.org>


On 30.06.2007, at 23:45, Dag-Erling Sm?rgrav wrote:

> Denis Ahrens <denis at zeno.org> writes:
>> svnversion says 1589M. you made no changes after that.
>
> Are you sure your tree is up-to-date and you rebuilt everything?   
> r1584
> should have fixed that bug.

Iam now running r1603 and it started to happen again

13 ObjLostHeader c Content-Length:

Denis



From tiamo at tfr.org  Sun Jul  1 22:36:04 2007
From: tiamo at tfr.org (TiAMO)
Date: Mon, 2 Jul 2007 00:36:04 +0200
Subject: Static only caching
Message-ID: <200707020036.04891.tiamo@tfr.org>

I just signed up to the list so sorry if my questions have been asked a 
million times already, if so i please point me to the responses.

This is what i want to do:

I want to setup anycasted proxys for static content that will have 48gig of 
RAM, 6TB (6x1TB disks in a raid0 setup) of storage and 2x Opteron 2214 CPU's. 
I want to have the 48gig most requested data in RAM and the rest on disk. The 
proxy will have backend content that's much larger then the storage size of 
each node. I have'nt had a chance to test how many mbit i can get out of each 
box, im hoping for gigabit+ speeds, maybe anyone have some numbers to share? 
I plan to uplink each box with a 2x1gbit etherchannel.

What fileystem is recomended to have the storage file on, i favor reiserfs for 
all my DB and http aplications, but then again i have never had 5.5TB+ files 
on them.

I think i got most of it figured out, but it's kinda hard to get it all right 
with the lack of docs and all =). 
What i want to do i limit the method to only HEAD and GET in HTTP/1.1 or else 
just return a service unavailable status (or just close the connection or 
whatever). Also i want to ignore all headers like cookies, no-cache, auth and 
so on, and always use the cache if it's in there. 

Any performance tips on how to tune whats in the cache when having more 
backend data then proxy storage, is fragmentation an issue? Also turnoff 
aging altogether so that data in the cache is only dropped when it's replaced 
by something more used.

Is it possible to have diffrent backends based on the http/1.1 host header?

Configuration examples would be much appreciated =)

If anyone is curious what it's for, it's for all the static content (like 
images, videos, torrents, audio files) of http://thepiratebay.org 
http://bayimg.com and the upcoming http://thevideobay.org http://playble.com/ 
and our other projects.

// Fredrik Neij



From denis at zeno.org  Sun Jul  1 23:28:00 2007
From: denis at zeno.org (Denis Ahrens)
Date: Mon, 2 Jul 2007 01:28:00 +0200
Subject: Static only caching
In-Reply-To: <200707020036.04891.tiamo@tfr.org>
References: <200707020036.04891.tiamo@tfr.org>
Message-ID: <A3E9E4E0-867D-48F2-A7FE-5E4548297667@zeno.org>


On 02.07.2007, at 00:36, TiAMO wrote:

> I just signed up to the list so sorry if my questions have been  
> asked a
> million times already, if so i please point me to the responses.
>
> This is what i want to do:
>
> I want to setup anycasted proxys for static content that will have  
> 48gig of
> RAM, 6TB (6x1TB disks in a raid0 setup) of storage and 2x Opteron  
> 2214 CPU's.
> I want to have the 48gig most requested data in RAM and the rest on  
> disk. The
> proxy will have backend content that's much larger then the storage  
> size of
> each node. I have'nt had a chance to test how many mbit i can get  
> out of each
> box, im hoping for gigabit+ speeds, maybe anyone have some numbers  
> to share?
> I plan to uplink each box with a 2x1gbit etherchannel.
>
> What fileystem is recomended to have the storage file on, i favor  
> reiserfs for
> all my DB and http aplications, but then again i have never had 5.5TB 
> + files
> on them.
>
> I think i got most of it figured out, but it's kinda hard to get it  
> all right
> with the lack of docs and all =).
> What i want to do i limit the method to only HEAD and GET in HTTP/ 
> 1.1 or else
> just return a service unavailable status (or just close the  
> connection or
> whatever). Also i want to ignore all headers like cookies, no-cache,  
> auth and
> so on, and always use the cache if it's in there.
>
> Any performance tips on how to tune whats in the cache when having  
> more
> backend data then proxy storage, is fragmentation an issue? Also  
> turnoff
> aging altogether so that data in the cache is only dropped when it's  
> replaced
> by something more used.
>
> Is it possible to have diffrent backends based on the http/1.1 host  
> header?
>
> Configuration examples would be much appreciated =)
>
> If anyone is curious what it's for, it's for all the static content  
> (like
> images, videos, torrents, audio files) of http://thepiratebay.org
> http://bayimg.com and the upcoming http://thevideobay.org http://playble.com/
> and our other projects.

In my tests I could easily saturate a 1GBit link with varnish on 10%  
load
on a dual quad core 2 xeons with 2.66GHz.

example for your backends would be:

at the start define two backends:

backend server1 {
         set backend.host = "backend1.domain.com";
         set backend.port = "80";
}

backend server2 {
         set backend.host = "backend2.domain.com";
         set backend.port = "80";
}

in sub vcl_recv the following:

     if (req.http.Host == "server1.piratebay.org")
     {
         set req.backend = server1;
     }

     if (req.http.Host == "server2.piratebay.org")
     {
         set req.backend = server2;
     }

I think you will figure out the rest easily!

Denis Ahrens



From phk at phk.freebsd.dk  Mon Jul  2 04:59:44 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 02 Jul 2007 04:59:44 +0000
Subject: Problem with varnish and caching 
In-Reply-To: Your message of "Sun, 01 Jul 2007 14:39:00 EST."
	<1183318740.14999.38.camel@gabriela> 
Message-ID: <54327.1183352384@critter.freebsd.dk>

In message <1183318740.14999.38.camel at gabriela>, "Manuel Amador (Rudd-O)" write
s:

>I moved from Squid to Varnish and got stumped by your default policy.
>Squid accelerated static objects by default, cookies or no cookies.
>Varnish doesn't.  Period.
>
>It took me 15 minutes to install Squid, learn how to set it up as an
>accelerator, and set it up.  It took me over two hours just to answer
>the question "why isn't Varnish accelerating my site?", and over 15
>minutes to configure it to act sensibly.  Doesn't look too well for your
>product now, does it?

Nobody forces you to use Varnish, and given your unnecssarily snotty
attitude, I suggest you stick with Squid.

This is of course pretty harsh on the squid project, but you will
almost certainly not be able to contribute positively to the Varnish
project with that attitude.

Killfile, meet Manuel, Manuel, bye!

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Mon Jul  2 05:29:39 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 02 Jul 2007 07:29:39 +0200
Subject: Problem with varnish and caching
In-Reply-To: <1183317164.14999.12.camel@gabriela> (Manuel Amador's message of
	"Sun\, 01 Jul 2007 14\:12\:44 -0500")
References: <1183146070.32019.20.camel@gabriela>
	<87ejjt15nd.fsf@des.linpro.no> <1183248616.3955.7.camel@gabriela>
	<871wfs1p5c.fsf@des.linpro.no> <1183317164.14999.12.camel@gabriela>
Message-ID: <87sl87z83g.fsf@des.linpro.no>

"Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> But my data seems to contradict your assumption that my assumptions are
> flawed, since for each request on my Varnish log, there's a matching
> request on my Apache log.

You assume that your site is typical of those that use Varnish.  You
assume that we developed Varnish without any reference to real-life
sites, applications and users.  You assume that you are infinintely
smarter and wiser than us, and that because of this, insulting us and
swearing at us will make us realize the error of our ways and embrace
your interpretation of reality.  All of these assumptions are incorrect.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rudd-o at rudd-o.com  Sun Jul  1 19:12:44 2007
From: rudd-o at rudd-o.com (Manuel Amador (Rudd-O))
Date: Sun, 01 Jul 2007 14:12:44 -0500
Subject: Problem with varnish and caching
In-Reply-To: <871wfs1p5c.fsf@des.linpro.no>
References: <1183146070.32019.20.camel@gabriela>
	<87ejjt15nd.fsf@des.linpro.no> <1183248616.3955.7.camel@gabriela>
	<871wfs1p5c.fsf@des.linpro.no>
Message-ID: <1183317164.14999.12.camel@gabriela>

Perhaps.

But my data seems to contradict your assumption that my assumptions are
flawed, since for each request on my Varnish log, there's a matching
request on my Apache log.

If Varnish isn't accelerating anything (not even static content) due to
Cookies, then... hmmmm.... all that fancy new-world engineering
architecture  is useless.  By the way, I find the Varnish cache
architecture absolutely sensible and I commend you guys for it in
earnest, Squid should work like that.

At the moment I have found a temporary hackish solution where:


                if (req.http.Cookie ~ "wordpresspass") {
                        pipe;
                }

                if (req.http.Authenticate) {
                        pipe;
                }

                if (req.http.Expect) {
                        pipe;
                }

                if (req.http.Pragma ~ "no-cache") {
                        pipe;
                }

                lookup;

And it seems to be working fine (see the "wordpresspass" regexp match
for the cookie).

By the way, the manual page doesn't explain the difference between pipe
and pass.  Yes, technically it explains the difference, but it doesn't
discuss the implications of the difference for the caching subsystem,
and the implications aren't obvious.  I initially used "pass" but that
made my pages malfunction, so I reverted to pipe (which was
counterintuitive).  I was expecting "pass" to feed data straight from
the Web server to the client without caching the data, and for some
reason my site malfunctioned (redirects didn't work, logins didn't work,
etcetera).  I verified these statements using Wireshark.


On dom, 2007-07-01 at 10:53 +0200, Dag-Erling Sm?rgrav wrote:
> "Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> > In effect, the default Varnish policy of not caching Cookied requests
> > causes Varnish not to cache anything at all for most sites (you know,
> > there are tons of people out there using Google Analytics).  Think about
> > it: why would people want the overhead of a non-caching accelerating
> > proxy?
> 
> Perhaps your assumptions are flawed?
> 
> DES
	Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
	The R Zone - http://rudd-o.com/
	GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: UB40 - Reggae music
O, what a tangled web we weave, When first we practice to deceive.
		-- Sir Walter Scott, "Marmion"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070701/67fe9cf3/attachment.bin>

From rudd-o at rudd-o.com  Sun Jul  1 19:39:00 2007
From: rudd-o at rudd-o.com (Manuel Amador (Rudd-O))
Date: Sun, 01 Jul 2007 14:39:00 -0500
Subject: Problem with varnish and caching
In-Reply-To: <24569.1183280741@critter.freebsd.dk>
References: <24569.1183280741@critter.freebsd.dk>
Message-ID: <1183318740.14999.38.camel@gabriela>

See my last e-mail to find out the hack I had to implement in order for
Varnish to sort of work on my site.

I moved from Squid to Varnish and got stumped by your default policy.
Squid accelerated static objects by default, cookies or no cookies.
Varnish doesn't.  Period.

It took me 15 minutes to install Squid, learn how to set it up as an
accelerator, and set it up.  It took me over two hours just to answer
the question "why isn't Varnish accelerating my site?", and over 15
minutes to configure it to act sensibly.  Doesn't look too well for your
product now, does it?

Now I'm going to state what up until now I've only been implying: your
default policy is braindead and stupid.  For 99% of your potential user
base (who tend to use these annoying little things called cookies),
Varnish by default won't accelerate squat -- it will, however, introduce
additional overhead.  The other 1% serving all-static content with no
cookies most probably doesn't need an accelerator anyway.

Let me say it again: the default policy is braindead and stupid.  It is
braindead and stupid because no large site (you know, the type of site
which actually needs acceleration) will run without a cookie-based
visitor tracking system, and as a consequence, Varnish won't accelerate
anything.

I suggest you alter your default VCL policy to roughly match Squid's:

- Cache everything cacheable with an adaptive TTL based on the
Last-Modified header.  Or 120s.  I don't care, but I'd prefer an
adaptive TTL -- old objects have low probability of being modified.
- Do not use a client's Cookie headers as a sign that an object should
be re-fetched from the backend.  Web applications that need to issue
fresh objects already implement either varying URLs or
Cache-Control/Pragma headers to instruct caches not to cache content.
- Anytime a client sends Cache-Control: no-cache headers, re-fetch the
content from the backend.
- Obey the backend's Cache-Control and Pragma no-cache headers.  Also
obey the Expires header.

You see, Squid at least *accelerates something* out-of-the-box.  In the
meantime, try not to advertise Varnish as an accelerator, when it
effectively isn't without heavy user intervention.

I'm tired of arguing with you guys over something that should have been
*obvious* for expert engineers who, in fact, have written a solid and
revolutionary piece of software.  It seems you're overlooking the
details, and regrettably in this case the devil *is* in the details.

Thanks for your time.

On dom, 2007-07-01 at 09:05 +0000, Poul-Henning Kamp wrote:
> Dear Manuel,
> 
> The Varnish *DEFAULT* VCL code implements what we have decided is a
> sensible default policy, which will Do The Right Thing for most people,
> at least to get them going with Varnish.
> 
> Your complaints all seem to center on the fact that the default
> VCL code doesn't work for you.
> 
> You seem to have overlooked the fact that you are not forced to
> run with the default VCL code.
> 
> Varnish is on track to offer the most comprehensive and flexible
> configuration mechanism yet to be seen on any web server or cache.
> 
> Now, instead of whining about the default, tailor Varnish to your
> situation.
> 
> Thankyou!
> 
> Poul-Henning
> 
	Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
	The R Zone - http://rudd-o.com/
	GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: UB40 - Reggae music
Don't Worry, Be Happy.
		-- Meher Baba
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070701/970a9ab6/attachment.bin>

From phk at phk.freebsd.dk  Mon Jul  2 07:34:36 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 02 Jul 2007 07:34:36 +0000
Subject: missing Content-Lenght header 
In-Reply-To: Your message of "Sun, 01 Jul 2007 18:01:09 +0200."
	<3C33EDA7-EE30-4017-B5FC-B9E87A22AA38@zeno.org> 
Message-ID: <19320.1183361676@critter.freebsd.dk>

In message <3C33EDA7-EE30-4017-B5FC-B9E87A22AA38 at zeno.org>, Denis Ahrens writes
:

>> Are you sure your tree is up-to-date and you rebuilt everything?   =
>
>> r1584
>> should have fixed that bug.
>
>Iam now running r1603 and it started to happen again
>
>13 ObjLostHeader c Content-Length:

Can you try this patch ?


Index: cache_backend.c
===================================================================
--- cache_backend.c	(revision 1604)
+++ cache_backend.c	(working copy)
@@ -337,6 +337,8 @@
 		AZ(close(vc->fd));
 	vc->fd = -1;
 	vc->backend = NULL;
+	WS_Reset(vc->http->ws);
+	WS_Reset(vc->http2->ws);
 	LOCK(&vbemtx);
 	TAILQ_INSERT_HEAD(&vbe_head, vc, list);
 	VSL_stats->backend_unused++;
-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From andre.cruz at segula.pt  Mon Jul  2 11:18:16 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Mon, 2 Jul 2007 12:18:16 +0100
Subject: Varnish and Perlbal
Message-ID: <B86E3614-E2B5-4FC2-ACD5-C80224D6D2A8@segula.pt>

Hello all.

I'm a newbie to Varnish and I'm evaluating it in order to know if/how  
there is an advantage in adding it to our application flow. (well,  
I'm pretty much convinced there IS an advantage so now I'm just  
looking for the best way to use it).

Right now I have 2 Perlbals balancing requests to 4 apache backends.  
I'm very happy with Perlbal's load balancing capabilities so I'm  
looking for the best way to integrate varnish with this Perlbal 
+Apache configuration. Which should come first in the flow; Perbal or  
Varnish?

I read here in the list someone saying that they had Varnish  
alongside apache, but this way I would have 4 separate caches. In my  
case it seems a better idea to have Varnish in the 2 Perlbal machines  
since Perlbal uses only one processor and very little RAM. What do  
you guys think?

Thanks and keep up the good work!

Andr? Cruz




From des at linpro.no  Mon Jul  2 11:43:03 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 02 Jul 2007 13:43:03 +0200
Subject: Varnish and Perlbal
In-Reply-To: <B86E3614-E2B5-4FC2-ACD5-C80224D6D2A8@segula.pt>
	(=?utf-8?Q?=22Andr=C3=A9?= Cruz"'s
	message of "Mon\, 2 Jul 2007 12\:18\:16 +0100")
References: <B86E3614-E2B5-4FC2-ACD5-C80224D6D2A8@segula.pt>
Message-ID: <873b07doag.fsf@des.linpro.no>

Andr? Cruz <andre.cruz at segula.pt> writes:
> Right now I have 2 Perlbals balancing requests to 4 apache backends.
> I'm very happy with Perlbal's load balancing capabilities so I'm
> looking for the best way to integrate varnish with this Perlbal +
> Apache configuration. Which should come first in the flow; Perbal or
> Varnish?

Probably the easiest way to integrate Varnish, at least to begin with,
is to run it on the same servers as Apache.  This way, cache misses are
processed without network overhead, and you can easily run Varnish on
just one server in each pair to compare performance.

Running Varnish on the Perlbal servers will halve the load on each
Apache server, but increase the response time for cache misses.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From andre.cruz at segula.pt  Mon Jul  2 11:06:52 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Mon, 2 Jul 2007 12:06:52 +0100
Subject: Varnish and Perlbal
Message-ID: <D73100AB-F311-4B44-A23B-D705DE83C954@segula.pt>

Hello all.

I'm a newbie to Varnish and I'm evaluating it in order to know if/how  
there is an advantage in adding it to our application flow. (well,  
I'm pretty much convinced there IS an advantage so now I'm just  
looking for the best way to use it).

Right now I have 2 Perlbals balancing requests to 4 apache backends.  
I'm very happy with Perlbal's load balancing capabilities so I'm  
looking for the best way to integrate varnish with this Perlbal 
+Apache configuration. Which should come first in the flow; Perbal or  
Varnish?

I read here in the list someone saying that they had Varnish  
alongside apache, but this way I would have 4 separate caches. In my  
case it seems a better idea to have Varnish in the 2 Perlbal machines  
since Perlbal uses only one processor and very little RAM. What do  
you guys think?

Thanks and keep up the good work!

Andr? Cruz




From jean-marc.pouchoulon at ac-montpellier.fr  Mon Jul  2 12:18:26 2007
From: jean-marc.pouchoulon at ac-montpellier.fr (jean-marc pouchoulon)
Date: Mon, 02 Jul 2007 14:18:26 +0200
Subject: setting varnish storage size
In-Reply-To: <4685D33C.4060103@iamcool.net>
References: <4685D33C.4060103@iamcool.net>
Message-ID: <4688ED12.7050308@ac-montpellier.fr>

Bonjour Anup,

thanks for your answer but I am able to create lfs with dd and I haven't 
got yet any 64 bits machine.

dd if=/dev/zero bs=1048576 count=5000 of=test.file
5000+0 enregistrements lus
5000+0 enregistrements ?crits
5242880000 octets (5,2 GB) copi?s, 57,1358 seconde, 91,8 MB/s

 ls -alh test.file
-rw-r--r-- 1 root root 4,9G jui  2 13:25 test.file

the problem seems to be elsewhere.
I try to compile varnish with
gcc -D_FILE_OFFSET_BITS=64
but the limits of 2Gb still remains

Any ideas ?
thanks
jean-marc


>
> Are you using 32-bit hardware?
> If yes, there lies the problem.
> I had similar issues when testing on my local 32-bit PC.
> However once i installed on our actual servers (64-bit), the 1G limit 
> went away.
> Have never tried setting the storage file to sizes in thousands of G 
> though.
>
> Also, as far as i know, Linux will not support files bigger than 4G 
> unless the support is added to the kernel.
> Not sure if Fedora Core 6 has the patch applied.
>
> I cannot be very sure of this though.
>
> Regards
> A.S
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>   



From phk at phk.freebsd.dk  Mon Jul  2 12:32:12 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 02 Jul 2007 12:32:12 +0000
Subject: setting varnish storage size 
In-Reply-To: Your message of "Mon, 02 Jul 2007 14:18:26 +0200."
	<4688ED12.7050308@ac-montpellier.fr> 
Message-ID: <5483.1183379532@critter.freebsd.dk>


The limit is in the CPU hardware:  with 32 bit address bits you
can only see 4GB of RAM at any one moment.

Usually operating systems and (PCI) devices eat up 1-2 GB of the
4GB, and you also need space for the program, shared librararies,
stack and other data.

Get a 64bit machine.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From andre.cruz at segula.pt  Mon Jul  2 13:40:22 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Mon, 2 Jul 2007 14:40:22 +0100
Subject: Varnish and Perlbal
In-Reply-To: <873b07doag.fsf@des.linpro.no>
References: <B86E3614-E2B5-4FC2-ACD5-C80224D6D2A8@segula.pt>
	<873b07doag.fsf@des.linpro.no>
Message-ID: <62F9CEE6-C5BB-4F5D-9C40-E8240020AF5E@segula.pt>

Ok, I'll try it both ways to test.

And regarding my other question... Should Perlbal handle the request  
first, and pass it to some varnish process or should varnish process  
the request first and send only the misses to PerlBal+Apache?

Perlbal is probably better at load-balancing since it is it's core  
function, no?

Thanks for your help,
Andr?

On 2007/07/02, at 12:43, Dag-Erling Sm?rgrav wrote:

> Andr? Cruz <andre.cruz at segula.pt> writes:
>> Right now I have 2 Perlbals balancing requests to 4 apache backends.
>> I'm very happy with Perlbal's load balancing capabilities so I'm
>> looking for the best way to integrate varnish with this Perlbal +
>> Apache configuration. Which should come first in the flow; Perbal or
>> Varnish?
>
> Probably the easiest way to integrate Varnish, at least to begin with,
> is to run it on the same servers as Apache.  This way, cache misses  
> are
> processed without network overhead, and you can easily run Varnish on
> just one server in each pair to compare performance.
>
> Running Varnish on the Perlbal servers will halve the load on each
> Apache server, but increase the response time for cache misses.
>
> DES
> -- 
> Dag-Erling Sm?rgrav
> Senior Software Developer
> Linpro AS - www.linpro.no



From outbackdingo at gmail.com  Mon Jul  2 13:57:12 2007
From: outbackdingo at gmail.com (Dingo)
Date: Mon, 02 Jul 2007 16:57:12 +0300
Subject: varnish and Nginx 
Message-ID: <46890438.1050202@gmail.com>

How would one compare say varnich with nginx 
http://wiki.codemongers.com/Nginx

where i see Nginx is not only a web server but a balancer for 
http/pop/imap/smtp

and do you have any further designs for other protocols or straight http ?


From des at linpro.no  Mon Jul  2 13:59:10 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 02 Jul 2007 15:59:10 +0200
Subject: Varnish and Perlbal
In-Reply-To: <62F9CEE6-C5BB-4F5D-9C40-E8240020AF5E@segula.pt>
	(=?utf-8?Q?=22Andr=C3=A9?= Cruz"'s
	message of "Mon\, 2 Jul 2007 14\:40\:22 +0100")
References: <B86E3614-E2B5-4FC2-ACD5-C80224D6D2A8@segula.pt>
	<873b07doag.fsf@des.linpro.no>
	<62F9CEE6-C5BB-4F5D-9C40-E8240020AF5E@segula.pt>
Message-ID: <87tzsmdhzl.fsf@des.linpro.no>

Andr? Cruz <andre.cruz at segula.pt> writes:
> And regarding my other question... Should Perlbal handle the request
> first, and pass it to some varnish process or should varnish process
> the request first and send only the misses to PerlBal+Apache?

Isn't that really the same question?  Either you run Varnish in front of
Perlbal on the Perlbal servers, or you run it in front of Apache on the
Apache servers.

> Perlbal is probably better at load-balancing since it is it's core
> function, no?

Considering that Varnish doesn't do load balancing at all (yet), I would
concur that Perlbal is probably better at it :)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From denis at startsiden.no  Mon Jul  2 14:00:53 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Mon, 2 Jul 2007 16:00:53 +0200 (CEST)
Subject: Varnish and Perlbal
In-Reply-To: <62F9CEE6-C5BB-4F5D-9C40-E8240020AF5E@segula.pt>
Message-ID: <31099547.18091183384853112.JavaMail.root@ms1.startsiden.no>

----- Andr? Cruz <andre.cruz at segula.pt> wrote:
> Ok, I'll try it both ways to test.
> 
> And regarding my other question... Should Perlbal handle the request 
> 
> first, and pass it to some varnish process or should varnish process 
> 
> the request first and send only the misses to PerlBal+Apache?
> 
> Perlbal is probably better at load-balancing since it is it's core  
> function, no?
> 
> Thanks for your help,
> Andr?

Andr?,

If we can assume one of the reasons you want to use Perlbal is to achieve some sort of failover capability, I would say place Perlbal in front of Varnish. If you have another provision to handle that and you only want to improve performance I would say it depends on your application really. I completely agree with DES though that implementing Varnish locally on the same box as apache is indeed the path of least configuration and fewest changes :P

>From what I have read on Perlbal it should be suited for placement in front of a cache such as varnish. 

Could I ask what your experience with Perlbal is? Is it a nice loadbalancer? How does your setup with it look like?  What kind of traffice do you see?


I gather the Varnish project is looking to implement some sort of basic load balancing capabilities into Varnish at some point in time.

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From des at linpro.no  Mon Jul  2 14:02:30 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 02 Jul 2007 16:02:30 +0200
Subject: varnish and Nginx
In-Reply-To: <46890438.1050202@gmail.com> (outbackdingo@gmail.com's message of
	"Mon\, 02 Jul 2007 16\:57\:12 +0300")
References: <46890438.1050202@gmail.com>
Message-ID: <87ps3adhu1.fsf@des.linpro.no>

Dingo <outbackdingo at gmail.com> writes:
> How would one compare say varnich with nginx [...]  where i see Nginx
> is not only a web server but a balancer for http/pop/imap/smtp

Varnish is neither of these.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From denis at startsiden.no  Mon Jul  2 14:05:50 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Mon, 2 Jul 2007 16:05:50 +0200 (CEST)
Subject: varnish and Nginx
In-Reply-To: <46890438.1050202@gmail.com>
Message-ID: <21171599.18191183385150208.JavaMail.root@ms1.startsiden.no>

----- Dingo <outbackdingo at gmail.com> wrote:
> How would one compare say varnich with nginx 
> http://wiki.codemongers.com/Nginx
> where i see Nginx is not only a web server but a balancer for 
> http/pop/imap/smtp

Interesting question. I have myself been looking at Nginx, mainly for its capabilities as a web loadbalancer / front-end.

>From what I've read Nginx is not a cahce per se. It does serve static content (like any other webserver), can be coupled with fastcgi backends to serve dynamic content, and can also be used as an accelerating reverse proxy, in that it handles requests on behalf of slower backend servers (including loadbalancing). It does not (AFAIK) function as a web cache. 

Varnish is a web cache, and to serve any content at all you need some sort of webserver behind it. 

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From andre.cruz at segula.pt  Mon Jul  2 14:21:12 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Mon, 2 Jul 2007 15:21:12 +0100
Subject: Varnish and Perlbal
In-Reply-To: <87tzsmdhzl.fsf@des.linpro.no>
References: <B86E3614-E2B5-4FC2-ACD5-C80224D6D2A8@segula.pt>
	<873b07doag.fsf@des.linpro.no>
	<62F9CEE6-C5BB-4F5D-9C40-E8240020AF5E@segula.pt>
	<87tzsmdhzl.fsf@des.linpro.no>
Message-ID: <80DA9B94-6C8F-484C-BB24-1E489D9E7B49@segula.pt>

On 2007/07/02, at 14:59, Dag-Erling Sm?rgrav wrote:
> Andr? Cruz <andre.cruz at segula.pt> writes:
>> And regarding my other question... Should Perlbal handle the request
>> first, and pass it to some varnish process or should varnish process
>> the request first and send only the misses to PerlBal+Apache?
>
> Isn't that really the same question?  Either you run Varnish in  
> front of
> Perlbal on the Perlbal servers, or you run it in front of Apache on  
> the
> Apache servers.
>

Well... You can run varnish on the Perlbal servers and Perbal can  
still be in front of Varnish. But since varnish doesn't do load  
balancing it seems that it's better to let Perlbal handle the request  
first.

I would prefer to run Varnish on the Perlbal machines because they  
have 4 processors and lots of RAM, largely unused by perlbal. The  
apache machines on the other hand.... :) Again, if Varnish can't  
select a backend from a pool to satisfy cache misses then I may just  
as well have one Varnish for each apache and run them on the same  
machine... We'll see.

>> Perlbal is probably better at load-balancing since it is it's core
>> function, no?
>
> Considering that Varnish doesn't do load balancing at all (yet), I  
> would
> concur that Perlbal is probably better at it :)
>

Ok, got it. :)

Thanks again,
Andr?

From outbackdingo at gmail.com  Mon Jul  2 14:30:21 2007
From: outbackdingo at gmail.com (Dingo)
Date: Mon, 02 Jul 2007 17:30:21 +0300
Subject: varnish and Nginx
In-Reply-To: <21171599.18191183385150208.JavaMail.root@ms1.startsiden.no>
References: <21171599.18191183385150208.JavaMail.root@ms1.startsiden.no>
Message-ID: <46890BFD.3030003@gmail.com>

So its feasible to run Nginx as the server/load balancer and varnish as 
the front end cache giving potentially a decent high speed/high capacity 
design.

Denis Br?khus wrote:
> ----- Dingo <outbackdingo at gmail.com> wrote:
>   
>> How would one compare say varnich with nginx 
>> http://wiki.codemongers.com/Nginx
>> where i see Nginx is not only a web server but a balancer for 
>> http/pop/imap/smtp
>>     
>
> Interesting question. I have myself been looking at Nginx, mainly for its capabilities as a web loadbalancer / front-end.
>
> >From what I've read Nginx is not a cahce per se. It does serve static content (like any other webserver), can be coupled with fastcgi backends to serve dynamic content, and can also be used as an accelerating reverse proxy, in that it handles requests on behalf of slower backend servers (including loadbalancing). It does not (AFAIK) function as a web cache. 
>
> Varnish is a web cache, and to serve any content at all you need some sort of webserver behind it. 
>
> Regards
>   



From andre.cruz at segula.pt  Mon Jul  2 14:30:36 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Mon, 2 Jul 2007 15:30:36 +0100
Subject: Varnish and Perlbal
In-Reply-To: <31099547.18091183384853112.JavaMail.root@ms1.startsiden.no>
References: <31099547.18091183384853112.JavaMail.root@ms1.startsiden.no>
Message-ID: <6061B961-1166-4556-B937-3D3396CF9294@segula.pt>


On 2007/07/02, at 15:00, Denis Br?khus wrote:
>
> Andr?,
>
> If we can assume one of the reasons you want to use Perlbal is to  
> achieve some sort of failover capability, I would say place Perlbal  
> in front of Varnish. If you have another provision to handle that  
> and you only want to improve performance I would say it depends on  
> your application really. I completely agree with DES though that  
> implementing Varnish locally on the same box as apache is indeed  
> the path of least configuration and fewest changes :P
>

I'll start with that scenario then.

> From what I have read on Perlbal it should be suited for placement  
> in front of a cache such as varnish.
>
> Could I ask what your experience with Perlbal is? Is it a nice  
> loadbalancer? How does your setup with it look like?  What kind of  
> traffice do you see?
>

I use Perlbal for more than a year now and it has been a very good  
experience. Besides the great performance we have developed some  
custom plugins for it which I don't think would be possible with  
other solutions. We have numerous applications that use it... As an  
example, in one of them we have 2 perlbal servers in front of a pool  
of 4 apache servers and a traffic of about 40 Mbit/s. Without Perlbal  
all the apache workers would get used up quickly.

Hope it helps,
Andr? Cruz

From rudd-o at rudd-o.com  Mon Jul  2 14:26:38 2007
From: rudd-o at rudd-o.com (Manuel Amador (Rudd-O))
Date: Mon, 02 Jul 2007 09:26:38 -0500
Subject: Problem with varnish and caching
In-Reply-To: <87sl87z83g.fsf@des.linpro.no>
References: <1183146070.32019.20.camel@gabriela>
	<87ejjt15nd.fsf@des.linpro.no> <1183248616.3955.7.camel@gabriela>
	<871wfs1p5c.fsf@des.linpro.no> <1183317164.14999.12.camel@gabriela>
	<87sl87z83g.fsf@des.linpro.no>
Message-ID: <1183386398.27561.20.camel@gabriela>

Sigh, I keep going back to the engineering, and you keep personalizing
the matter and returning to your lies.

I never insulted any of you.  You're personalizing the matter, and
attempting to somehow equate my bug reports with personal criticism
towards the developers.

You're wrong about the majority of sites being accelerated by Varnish.
Almost no modern site works without cookies -- hence, almost no modern
site is cached according to Varnish default policies.  You have not
provided a single counterexample or a single snippet of VCL that could
solve the problems I have, or a single snippet of VCL that you guys are
actually using on production servers.

I can admit that I'm considerably less capable than you, after all I
didn't write Varnish whereas you *did*.  However, I'm considerably less
stupid than you would have me be.  Here I am, finding a real problem in
the real-life world -- and here you are, dismissing the problem like
you're the owner of the universe and the truth, just because YOU chose
to interpret my e-mails as direct criticism of YOU.

Want direct criticism and harsh words to match your interpretations?
Well, here they go, try and eat it with a side dish:

YOU're a rude thick-headed and thin-skinned individual who hasn't taken
a single minute out of his blame-me routine to actually HELP.  You're
carrying your heart on your sleeve.  Grow up, stop acting like a prima
donna, and understand that careless engineering and glib remarks aren't
gonna be tolerated by your users, no matter how cool you think you are
or how open-source-you're-free-not-to-use-us your project may be.

This discussion has been absolutely useless.

(Suggestion: perhaps you want to add me to your killfile like your
partner did; otherwise you could conceivably receive more "insults and
swears" may get to your inbox.)

On lun, 2007-07-02 at 07:29 +0200, Dag-Erling Sm?rgrav wrote:
> "Manuel Amador (Rudd-O)" <rudd-o at rudd-o.com> writes:
> > But my data seems to contradict your assumption that my assumptions are
> > flawed, since for each request on my Varnish log, there's a matching
> > request on my Apache log.
> 
> You assume that your site is typical of those that use Varnish.  You
> assume that we developed Varnish without any reference to real-life
> sites, applications and users.  You assume that you are infinintely
> smarter and wiser than us, and that because of this, insulting us and
> swearing at us will make us realize the error of our ways and embrace
> your interpretation of reality.  All of these assumptions are incorrect.
> 
> DES
	Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
	The R Zone - http://rudd-o.com/
	GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: Q:	How many IBM CPU's does it take to do a logical right shift?
A:	33.  1 to hold the bits and 32 to push the register.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070702/77a1f813/attachment.bin>

From jean-marc.pouchoulon at ac-montpellier.fr  Mon Jul  2 15:17:45 2007
From: jean-marc.pouchoulon at ac-montpellier.fr (jean-marc pouchoulon)
Date: Mon, 02 Jul 2007 17:17:45 +0200
Subject: truncated page problem 
Message-ID: <46891719.1060008@ac-montpellier.fr>

helo

Using varnish with this url:

http://publinet.ac-montpellier.fr/publinet/Resultats?Unom_ecr=5&Ubase=publi_13&Utyp_aff=Resultats&Unum_cri=2&Uval_cri=B&Uval_cri1=32405

the page is truncated beginning by

IGN=LEFT><font face='arial' class="libelleDecision">ADMIS </font></TD>
</TR>
<TR  class="ligneImpaire">
<TD><font face='arial' class="nomCandidat">BRIGATI 
Anne-Marie</font></TD><TD ALIGN=LEFT><font face='arial' 
class="libelleDecision">ADMIS </font></TD>
</TR>

( "line break"  in html page )

without varnish , direct to the http server there are no problems

I've got this log and a 104 error:

RxRequest    c GET
   69 RxURL        c 
/publinet/Resultats?Unom_ecr=5&Ubase=publi_13&Utyp_aff=Resultats&Unum_cri=2&Uval_cri=B&Uval_cri1=32405
   69 RxProtocol   c HTTP/1.1
   69 RxHeader     c Accept: image/gif, image/x-xbitmap, image/jpeg, 
image/pjpeg, application/msword, application/vnd.ms-powerpoint, 
application/vnd.ms-excel, application/x-shockwave-flash, */*
   69 RxHeader     c Referer: 
http://publinet.ac-montpellier.fr/publinet/Resultats?Unom_ecr=5&Ubase=publi_13&Utyp_aff=Resultats&Unum_cri=2&Uval_cri=A&Uval_cri1=32405
   69 RxHeader     c Accept-Language: fr
   69 RxHeader     c Accept-Encoding:
   69 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; 
Windows NT 5.1; SV1; .NET CLR 1.1.4322)
   69 RxHeader     c Host: publinet.ac-montpellier.fr
   69 RxHeader     c Connection: Keep-Alive
   69 VCL_call     c recv
   69 VCL_return   c lookup
   69 VCL_call     c hash
   69 VCL_return   c hash
   69 Hit          c 187404565
   69 VCL_call     c hit
   69 VCL_return   c deliver
   69 Length       c 16143
   69 TxProtocol   c HTTP/1.1
   69 TxStatus     c 200
   69 TxResponse   c OK
   69 TxHeader     c Date: Mon, 02 Jul 2007 14:13:09 GMT
   69 TxHeader     c Server: Apache/2.0.52 (Unix) DAV/2
   69 TxHeader     c Content-Type: text/html; charset=ISO-8859-1
   69 TxHeader     c Content-Length: 16143
   69 TxHeader     c X-Varnish: 187476688 187404565
   69 TxHeader     c Age: 536
   69 TxHeader     c Via: 1.1 varnish
   69 ReqEnd       c 187476688 1183386112.545343552 1183386112.545831603 
0.000027936 0.000024305 0.000463746
    0 StatAddr       90.14.20.225 0 59 20 29 0 0 0 5026 120142
   44 HttpError    c Received errno 104
   44 SessionClose c no request
   44 StatSess     c 86.206.235.88 4737 0 1 1 0 0 0 152 0
   26 ReqStart     c 90.28.32.85 1940 187476689
 
I use varnish 1.0.0.4.

thanks for your help



From des at linpro.no  Mon Jul  2 15:42:02 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 02 Jul 2007 17:42:02 +0200
Subject: truncated page problem
In-Reply-To: <46891719.1060008@ac-montpellier.fr> (jean-marc pouchoulon's
	message of "Mon\, 02 Jul 2007 17\:17\:45 +0200")
References: <46891719.1060008@ac-montpellier.fr>
Message-ID: <87ir92dd85.fsf@des.linpro.no>

jean-marc pouchoulon <jean-marc.pouchoulon at ac-montpellier.fr> writes:
> Using varnish with this url:
>
> http://publinet.ac-montpellier.fr/publinet/Resultats?Unom_ecr=5&Ubase=publi_13&Utyp_aff=Resultats&Unum_cri=2&Uval_cri=B&Uval_cri1=32405
>
> the page is truncated beginning by
>
> IGN=LEFT><font face='arial' class="libelleDecision">ADMIS </font></TD>
> </TR>
> <TR  class="ligneImpaire">
> <TD><font face='arial' class="nomCandidat">BRIGATI 
> Anne-Marie</font></TD><TD ALIGN=LEFT><font face='arial' 
> class="libelleDecision">ADMIS </font></TD>
> </TR>
>
> ( "line break"  in html page )
>
> without varnish , direct to the http server there are no problems
>
> I've got this log and a 104 error:
>
>    69 RxRequest    c GET
>    69 RxURL        c /publinet/Resultats?Unom_ecr=5&Ubase=publi_13&Utyp_aff=Resultats&Unum_cri=2&Uval_cri=B&Uval_cri1=32405
>    69 RxProtocol   c HTTP/1.1
>    69 RxHeader     c Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/msword, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/x-shockwave-flash, */*
>    69 RxHeader     c Referer: http://publinet.ac-montpellier.fr/publinet/Resultats?Unom_ecr=5&Ubase=publi_13&Utyp_aff=Resultats&Unum_cri=2&Uval_cri=A&Uval_cri1=32405
>    69 RxHeader     c Accept-Language: fr
>    69 RxHeader     c Accept-Encoding:
>    69 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
>    69 RxHeader     c Host: publinet.ac-montpellier.fr
>    69 RxHeader     c Connection: Keep-Alive
>    69 VCL_call     c recv
>    69 VCL_return   c lookup
>    69 VCL_call     c hash
>    69 VCL_return   c hash
>    69 Hit          c 187404565
>    69 VCL_call     c hit
>    69 VCL_return   c deliver
>    69 Length       c 16143
>    69 TxProtocol   c HTTP/1.1
>    69 TxStatus     c 200
>    69 TxResponse   c OK
>    69 TxHeader     c Date: Mon, 02 Jul 2007 14:13:09 GMT
>    69 TxHeader     c Server: Apache/2.0.52 (Unix) DAV/2
>    69 TxHeader     c Content-Type: text/html; charset=ISO-8859-1
>    69 TxHeader     c Content-Length: 16143
>    69 TxHeader     c X-Varnish: 187476688 187404565
>    69 TxHeader     c Age: 536
>    69 TxHeader     c Via: 1.1 varnish
>    69 ReqEnd       c 187476688 1183386112.545343552 1183386112.545831603 0.000027936 0.000024305 0.000463746

Looks fine to me...

>     0 StatAddr       90.14.20.225 0 59 20 29 0 0 0 5026 120142
>    44 HttpError    c Received errno 104
>    44 SessionClose c no request
>    44 StatSess     c 86.206.235.88 4737 0 1 1 0 0 0 152 0

#define ECONNRESET      104     /* Connection reset by peer */

but this is a different request, on a different socket.  Could you send
me a full log?  Start 'varnishlog -w full.log', start varnishd, request
the problem page a couple of times (so I can see both a cache miss and a
cache hit), then stop varnishd, stop varnishlog and send me the file.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From denis at zeno.org  Mon Jul  2 17:24:19 2007
From: denis at zeno.org (Denis Ahrens)
Date: Mon, 2 Jul 2007 19:24:19 +0200
Subject: missing Content-Lenght header 
In-Reply-To: <19320.1183361676@critter.freebsd.dk>
References: <19320.1183361676@critter.freebsd.dk>
Message-ID: <C9D1894C-3B61-441B-A752-2ED47EA6B56A@zeno.org>


On 02.07.2007, at 09:34, Poul-Henning Kamp wrote:

> In message <3C33EDA7-EE30-4017-B5FC-B9E87A22AA38 at zeno.org>, Denis  
> Ahrens writes
> :
>
>>> Are you sure your tree is up-to-date and you rebuilt everything?   =
>>
>>> r1584
>>> should have fixed that bug.
>>
>> Iam now running r1603 and it started to happen again
>>
>> 13 ObjLostHeader c Content-Length:
>
> Can you try this patch ?

Your patch is now running for 9h and I have not
seen the problem again. So I think it works.

Denis



From mike at boom.net  Mon Jul  2 18:22:01 2007
From: mike at boom.net (Mike Hedlund)
Date: Mon, 2 Jul 2007 11:22:01 -0700
Subject: Varnish bypassing cache when if-modified-since header present?
Message-ID: <20070702182201.GF5005@boom.net>

Hey guys,

	I've been playing with varnish for the last few days, trying to eval it as a squid replacement my image serving farm. I enabled it on one of the squids friday (replacing 2 seperate squid processes on a single server), and seen pretty awesome results:

http://boom.net/~mike/squid_vs_varnish.GIF

However, the load on my backend NFS server went up by a substantial amount. I let it run over the weekend thinking it would smooth out once the caches were primed, but to my dismay, this morning nfs load was still the same. After doing some basic debugging, it appears as if the (default) varnish config is always re-validating if-modified-since headers (when supplied by the client), thus bypassing the local cache and hitting the origin server again. My cache objects *never* change, so if a client supplies an if-modified-since header, I'd like the cache to always respond '304 not modified' immediately. I've configured squid to do this, but my vcl-fu is very weak. Before I spent more time investigating the VCL config options, I just wanted a sanity check to make sure it was possible to do this in varnish.  At the bottom of this message i've included a small sample of a log file i generated this morning, I could be misreading it completely. Thanks!


ps: the above load graph is from an image server in my production pool, that was spitting out around 150Mb/s of images at the time.

-mike




   11 SessionOpen  c 10.1.1.9 33194
   11 ReqStart     c 10.1.1.9 33194 249494004
   11 RxRequest    c GET
   11 RxURL        c /85/40/458/tn_4096253716.jpg
   11 RxProtocol   c HTTP/1.1
   11 RxHeader     c Accept: */*
   11 RxHeader     c Accept-Language: en-us
   11 RxHeader     c Accept-Encoding: gzip, deflate
   11 RxHeader     c If-Modified-Since: Thu, 23 Feb 2006 00:02:51 GMT
   11 RxHeader     c If-None-Match: "13c02df-b23-40d6af72600c0"
   11 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
   11 RxHeader     c Host: a.pc.xyz.com:87
   11 RxHeader     c Connection: Keep-Alive
   11 VCL_call     c recv
   11 VCL_return   c pass
   11 VCL_call     c pass
   11 VCL_return   c pass
   14 BackendClose   default
   14 BackendOpen  b default 127.0.0.1 33977 127.0.0.1 81
   14 BackendXID   b 249494004
   11 Backend      c 14 default
   14 TxRequest    b GET
   14 TxURL        b /85/40/458/tn_4096253716.jpg
   14 TxProtocol   b HTTP/1.1
   14 TxHeader     b Accept: */*
   14 TxHeader     b Accept-Language: en-us
   14 TxHeader     b Accept-Encoding: gzip, deflate
   14 TxHeader     b User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
   14 TxHeader     b Host: a.pc.xyz.com:87
   14 TxHeader     b X-Varnish: 249494004
   14 TxHeader     b X-Forwarded-for: 10.1.1.9
   14 RxProtocol   b HTTP/1.1
   14 RxStatus     b 200
   14 RxResponse   b OK
   14 RxHeader     b Date: Mon, 02 Jul 2007 17:59:03 GMT
   14 RxHeader     b Server: Apache
   14 RxHeader     b Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
   14 RxHeader     b ETag: "13c02df-b23-40d6af72600c0"
   14 RxHeader     b Accept-Ranges: bytes
   14 RxHeader     b Content-Length: 2851
   14 RxHeader     b Content-Type: image/jpeg
   11 ObjProtocol  c HTTP/1.1
   11 ObjStatus    c 200
   11 ObjResponse  c OK
   11 ObjHeader    c Date: Mon, 02 Jul 2007 17:59:03 GMT
   11 ObjHeader    c Server: Apache
   11 ObjHeader    c Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
   11 ObjHeader    c ETag: "13c02df-b23-40d6af72600c0"
   11 ObjHeader    c Content-Type: image/jpeg
   11 ObjHeader    c Content-Length: 2851
   14 BackendReuse b default
   11 TTL          c 249494004 RFC 3600 1183399143 1183399143 0 0 0
   11 VCL_call     c fetch
   11 VCL_return   c insert
   11 Length       c 0
   11 TxProtocol   c HTTP/1.1
   11 TxStatus     c 304
   11 TxResponse   c Not Modified
   11 TxHeader     c Date: Mon, 02 Jul 2007 17:59:03 GMT
   11 TxHeader     c Via: 1.1 varnish
   11 TxHeader     c X-Varnish: 249494004
   11 TxHeader     c Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
   11 ReqEnd       c 249494004 1183399143.654401000 1183399143.656763000 0.000043000 0.002308000 0.000054000
    0 StatAddr       10.1.1.9 0 579 116 68 0 0 13 18505 6270
   11 SessionOpen  c 10.1.1.9 61746
   11 ReqStart     c 10.1.1.9 61746 249494005
   11 RxRequest    c GET
   11 RxURL        c /85/40/458/tn_4096253716.jpg
   11 RxProtocol   c HTTP/1.1
   11 RxHeader     c Accept: */*
   11 RxHeader     c Accept-Language: en-us
   11 RxHeader     c Accept-Encoding: gzip, deflate
   11 RxHeader     c If-Modified-Since: Thu, 23 Feb 2006 00:02:51 GMT
   11 RxHeader     c If-None-Match: "13c02df-b23-40d6af72600c0"
   11 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
   11 RxHeader     c Host: a.pc.xyz.com:87
   11 RxHeader     c Connection: Keep-Alive
   11 VCL_call     c recv
   11 VCL_return   c pass
   11 VCL_call     c pass
   11 VCL_return   c pass
   14 BackendClose b default
   14 BackendOpen  b default 127.0.0.1 33978 127.0.0.1 81
   14 BackendXID   b 249494005
   11 Backend      c 14 default
   14 TxRequest    b GET
   14 TxURL        b /85/40/458/tn_4096253716.jpg
   14 TxProtocol   b HTTP/1.1
   14 TxHeader     b Accept: */*
   14 TxHeader     b Accept-Language: en-us
   14 TxHeader     b Accept-Encoding: gzip, deflate
   14 TxHeader     b User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
   14 TxHeader     b Host: a.pc.xyz.com:87
   14 TxHeader     b X-Varnish: 249494005
   14 TxHeader     b X-Forwarded-for: 10.1.1.9
   14 RxProtocol   b HTTP/1.1
   14 RxStatus     b 200
   14 RxResponse   b OK
   14 RxHeader     b Date: Mon, 02 Jul 2007 17:59:21 GMT
   14 RxHeader     b Server: Apache
   14 RxHeader     b Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
   14 RxHeader     b ETag: "13c02df-b23-40d6af72600c0"
   14 RxHeader     b Accept-Ranges: bytes
   14 RxHeader     b Content-Length: 2851
   14 RxHeader     b Content-Type: image/jpeg
   11 ObjProtocol  c HTTP/1.1
   11 ObjStatus    c 200
   11 ObjResponse  c OK
   11 ObjHeader    c Date: Mon, 02 Jul 2007 17:59:21 GMT
   11 ObjHeader    c Server: Apache
   11 ObjHeader    c Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
   11 ObjHeader    c ETag: "13c02df-b23-40d6af72600c0"
   11 ObjHeader    c Content-Type: image/jpeg
   11 ObjHeader    c Content-Length: 2851
   14 BackendReuse b default
   11 TTL          c 249494005 RFC 3600 1183399161 1183399161 0 0 0
   11 VCL_call     c fetch
   11 VCL_return   c insert
   11 Length       c 0
   11 TxProtocol   c HTTP/1.1
   11 TxStatus     c 304
   11 TxResponse   c Not Modified
   11 TxHeader     c Date: Mon, 02 Jul 2007 17:59:21 GMT
   11 TxHeader     c Via: 1.1 varnish
   11 TxHeader     c X-Varnish: 249494005
   11 TxHeader     c Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
   11 ReqEnd       c 249494005 1183399161.148634000 1183399161.149295000 0.000041000 0.000607000 0.000054000
    0 StatAddr       10.1.1.9 0 597 117 69 0 0 14 18657 6270






From mike at boom.net  Mon Jul  2 19:00:19 2007
From: mike at boom.net (Mike Hedlund)
Date: Mon, 2 Jul 2007 12:00:19 -0700
Subject: Varnish bypassing cache when if-modified-since header present?
In-Reply-To: <20070702182201.GF5005@boom.net>
References: <20070702182201.GF5005@boom.net>
Message-ID: <20070702190016.GG5005@boom.net>

sorry for the spam, but i just realized the cause of the behavior i was seeing-- the cookie being set. after i modified the default vcl to:

#       if (req.http.Authenticate || req.http.Cookie) {
        if (req.http.Authenticate) {
                pass;
        }

it works as expected... 

-mike



On Mon, Jul 02, 2007 at 11:22:01AM -0700, Mike Hedlund <mike at boom.net> wrote:

> Hey guys,
> 
> 	I've been playing with varnish for the last few days, trying to eval it as a squid replacement my image serving farm. I enabled it on one of the squids friday (replacing 2 seperate squid processes on a single server), and seen pretty awesome results:
> 
> http://boom.net/~mike/squid_vs_varnish.GIF
> 
> However, the load on my backend NFS server went up by a substantial amount. I let it run over the weekend thinking it would smooth out once the caches were primed, but to my dismay, this morning nfs load was still the same. After doing some basic debugging, it appears as if the (default) varnish config is always re-validating if-modified-since headers (when supplied by the client), thus bypassing the local cache and hitting the origin server again. My cache objects *never* change, so if a client supplies an if-modified-since header, I'd like the cache to always respond '304 not modified' immediately. I've configured squid to do this, but my vcl-fu is very weak. Before I spent more time investigating the VCL config options, I just wanted a sanity check to make sure it was possible to do this in varnish.  At the bottom of this message i've included a small sample of a log file i generated this morning, I could be misreading it completely. Thanks!
> 
> 
> ps: the above load graph is from an image server in my production pool, that was spitting out around 150Mb/s of images at the time.
> 
> -mike
> 
> 
> 
> 
>    11 SessionOpen  c 10.1.1.9 33194
>    11 ReqStart     c 10.1.1.9 33194 249494004
>    11 RxRequest    c GET
>    11 RxURL        c /85/40/458/tn_4096253716.jpg
>    11 RxProtocol   c HTTP/1.1
>    11 RxHeader     c Accept: */*
>    11 RxHeader     c Accept-Language: en-us
>    11 RxHeader     c Accept-Encoding: gzip, deflate
>    11 RxHeader     c If-Modified-Since: Thu, 23 Feb 2006 00:02:51 GMT
>    11 RxHeader     c If-None-Match: "13c02df-b23-40d6af72600c0"
>    11 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
>    11 RxHeader     c Host: a.pc.xyz.com:87
>    11 RxHeader     c Connection: Keep-Alive
>    11 VCL_call     c recv
>    11 VCL_return   c pass
>    11 VCL_call     c pass
>    11 VCL_return   c pass
>    14 BackendClose   default
>    14 BackendOpen  b default 127.0.0.1 33977 127.0.0.1 81
>    14 BackendXID   b 249494004
>    11 Backend      c 14 default
>    14 TxRequest    b GET
>    14 TxURL        b /85/40/458/tn_4096253716.jpg
>    14 TxProtocol   b HTTP/1.1
>    14 TxHeader     b Accept: */*
>    14 TxHeader     b Accept-Language: en-us
>    14 TxHeader     b Accept-Encoding: gzip, deflate
>    14 TxHeader     b User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
>    14 TxHeader     b Host: a.pc.xyz.com:87
>    14 TxHeader     b X-Varnish: 249494004
>    14 TxHeader     b X-Forwarded-for: 10.1.1.9
>    14 RxProtocol   b HTTP/1.1
>    14 RxStatus     b 200
>    14 RxResponse   b OK
>    14 RxHeader     b Date: Mon, 02 Jul 2007 17:59:03 GMT
>    14 RxHeader     b Server: Apache
>    14 RxHeader     b Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
>    14 RxHeader     b ETag: "13c02df-b23-40d6af72600c0"
>    14 RxHeader     b Accept-Ranges: bytes
>    14 RxHeader     b Content-Length: 2851
>    14 RxHeader     b Content-Type: image/jpeg
>    11 ObjProtocol  c HTTP/1.1
>    11 ObjStatus    c 200
>    11 ObjResponse  c OK
>    11 ObjHeader    c Date: Mon, 02 Jul 2007 17:59:03 GMT
>    11 ObjHeader    c Server: Apache
>    11 ObjHeader    c Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
>    11 ObjHeader    c ETag: "13c02df-b23-40d6af72600c0"
>    11 ObjHeader    c Content-Type: image/jpeg
>    11 ObjHeader    c Content-Length: 2851
>    14 BackendReuse b default
>    11 TTL          c 249494004 RFC 3600 1183399143 1183399143 0 0 0
>    11 VCL_call     c fetch
>    11 VCL_return   c insert
>    11 Length       c 0
>    11 TxProtocol   c HTTP/1.1
>    11 TxStatus     c 304
>    11 TxResponse   c Not Modified
>    11 TxHeader     c Date: Mon, 02 Jul 2007 17:59:03 GMT
>    11 TxHeader     c Via: 1.1 varnish
>    11 TxHeader     c X-Varnish: 249494004
>    11 TxHeader     c Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
>    11 ReqEnd       c 249494004 1183399143.654401000 1183399143.656763000 0.000043000 0.002308000 0.000054000
>     0 StatAddr       10.1.1.9 0 579 116 68 0 0 13 18505 6270
>    11 SessionOpen  c 10.1.1.9 61746
>    11 ReqStart     c 10.1.1.9 61746 249494005
>    11 RxRequest    c GET
>    11 RxURL        c /85/40/458/tn_4096253716.jpg
>    11 RxProtocol   c HTTP/1.1
>    11 RxHeader     c Accept: */*
>    11 RxHeader     c Accept-Language: en-us
>    11 RxHeader     c Accept-Encoding: gzip, deflate
>    11 RxHeader     c If-Modified-Since: Thu, 23 Feb 2006 00:02:51 GMT
>    11 RxHeader     c If-None-Match: "13c02df-b23-40d6af72600c0"
>    11 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
>    11 RxHeader     c Host: a.pc.xyz.com:87
>    11 RxHeader     c Connection: Keep-Alive
>    11 VCL_call     c recv
>    11 VCL_return   c pass
>    11 VCL_call     c pass
>    11 VCL_return   c pass
>    14 BackendClose b default
>    14 BackendOpen  b default 127.0.0.1 33978 127.0.0.1 81
>    14 BackendXID   b 249494005
>    11 Backend      c 14 default
>    14 TxRequest    b GET
>    14 TxURL        b /85/40/458/tn_4096253716.jpg
>    14 TxProtocol   b HTTP/1.1
>    14 TxHeader     b Accept: */*
>    14 TxHeader     b Accept-Language: en-us
>    14 TxHeader     b Accept-Encoding: gzip, deflate
>    14 TxHeader     b User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
>    14 TxHeader     b Host: a.pc.xyz.com:87
>    14 TxHeader     b X-Varnish: 249494005
>    14 TxHeader     b X-Forwarded-for: 10.1.1.9
>    14 RxProtocol   b HTTP/1.1
>    14 RxStatus     b 200
>    14 RxResponse   b OK
>    14 RxHeader     b Date: Mon, 02 Jul 2007 17:59:21 GMT
>    14 RxHeader     b Server: Apache
>    14 RxHeader     b Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
>    14 RxHeader     b ETag: "13c02df-b23-40d6af72600c0"
>    14 RxHeader     b Accept-Ranges: bytes
>    14 RxHeader     b Content-Length: 2851
>    14 RxHeader     b Content-Type: image/jpeg
>    11 ObjProtocol  c HTTP/1.1
>    11 ObjStatus    c 200
>    11 ObjResponse  c OK
>    11 ObjHeader    c Date: Mon, 02 Jul 2007 17:59:21 GMT
>    11 ObjHeader    c Server: Apache
>    11 ObjHeader    c Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
>    11 ObjHeader    c ETag: "13c02df-b23-40d6af72600c0"
>    11 ObjHeader    c Content-Type: image/jpeg
>    11 ObjHeader    c Content-Length: 2851
>    14 BackendReuse b default
>    11 TTL          c 249494005 RFC 3600 1183399161 1183399161 0 0 0
>    11 VCL_call     c fetch
>    11 VCL_return   c insert
>    11 Length       c 0
>    11 TxProtocol   c HTTP/1.1
>    11 TxStatus     c 304
>    11 TxResponse   c Not Modified
>    11 TxHeader     c Date: Mon, 02 Jul 2007 17:59:21 GMT
>    11 TxHeader     c Via: 1.1 varnish
>    11 TxHeader     c X-Varnish: 249494005
>    11 TxHeader     c Last-Modified: Thu, 23 Feb 2006 00:02:51 GMT
>    11 ReqEnd       c 249494005 1183399161.148634000 1183399161.149295000 0.000041000 0.000607000 0.000054000
>     0 StatAddr       10.1.1.9 0 597 117 69 0 0 14 18657 6270
> 
> 
> 
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc


From phk at phk.freebsd.dk  Mon Jul  2 19:07:24 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 02 Jul 2007 19:07:24 +0000
Subject: Varnish bypassing cache when if-modified-since header present? 
In-Reply-To: Your message of "Mon, 02 Jul 2007 11:22:01 MST."
	<20070702182201.GF5005@boom.net> 
Message-ID: <3205.1183403244@critter.freebsd.dk>

In message <20070702182201.GF5005 at boom.net>, Mike Hedlund writes:
>Hey guys,
>
>       I've been playing with varnish for the last few days, trying
>to eval it as a squid replacement my image serving farm. I enabled
>it on one of the squids friday (replacing 2 seperate squid processes
>on a single server), and seen pretty awesome results:
>
>http://boom.net/~mike/squid_vs_varnish.GIF

That's a pretty typical experience :-)

>[...] varnish config is always re-validating
>if-modified-since headers (when supplied by the client)[...]

This shouldn't happen, but there is on case where it will: if your
varnish and backend don't agree what time it is.

Could you check that ?

If you want to fill in some debugging look for "res_do_304" in
cache_response.c

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From denis at startsiden.no  Mon Jul  2 19:22:57 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Mon, 2 Jul 2007 21:22:57 +0200 (CEST)
Subject: varnish and Nginx
In-Reply-To: <46890BFD.3030003@gmail.com>
Message-ID: <31326250.19001183404177568.JavaMail.root@ms1.startsiden.no>

----- Dingo <outbackdingo at gmail.com> wrote:
> So its feasible to run Nginx as the server/load balancer and varnish
> as the front end cache giving potentially a decent high speed/high
> capacity design.

Well, I have no experience with Nginx whatsoever, but yes, both could be components in a high speed/high capacity design. 

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From varnish at held-im-ruhestand.de  Mon Jul  2 21:14:50 2007
From: varnish at held-im-ruhestand.de (Christoph)
Date: Mon, 2 Jul 2007 23:14:50 +0200
Subject: Varnish Dirty Caching
Message-ID: <20070702211450.GA16119@falcon>


hi,

i'd like to implement dirty-caching using varnish.

So what is dirty caching and why use it? Think of a very unreliable
backend. If varnish can't reach it's backend, it will simply return the
last content it has (even if the content is stale). That way i can cover
hickups.

Is this possible? What are the side effects?




As far as i understood varnish the "normal" configurations works like
this

request comes in

try to find it in cache

if obj.cacheable (missing documentation on the exact meaning of
cacheable, probably a check is Expires Header is still in the future) then 
return this object to client (will this update the age header?)

fetch object from backend. If cacheable insert into cache.

Return object to client.



Now after some time comes the reaper. If an object expired the reaper
will call vcl_timeout. vcl_timeout will either discard the object, or it
will fetch an update. 
If i discard it on timeout, this will keep my cache tidy. If i always
fetch an update it will constantly keep a complete copy of my backend
(at least the part that was hit once). Both options seem bad. Keeping it
tidy will force a complete retrieval of the object, even if it didn't
change on the backend (just new expire headers). Keeping a copy will
hammer my backend with requests for files that are normaly hit every ten
years. 

so i'm slightly confused and looking for some documentation...


Greetings
   Christoph


From anup at iamcool.net  Tue Jul  3 04:42:02 2007
From: anup at iamcool.net (Anup Shukla)
Date: Tue, 03 Jul 2007 10:12:02 +0530
Subject: Problem with varnish and caching
In-Reply-To: <1183386398.27561.20.camel@gabriela>
References: <1183146070.32019.20.camel@gabriela>	<87ejjt15nd.fsf@des.linpro.no>
	<1183248616.3955.7.camel@gabriela>	<871wfs1p5c.fsf@des.linpro.no>
	<1183317164.14999.12.camel@gabriela>	<87sl87z83g.fsf@des.linpro.no>
	<1183386398.27561.20.camel@gabriela>
Message-ID: <4689D39A.1030606@iamcool.net>

Manuel Amador (Rudd-O) wrote:
> site is cached according to Varnish default policies.  You have not
> provided a single counterexample or a single snippet of VCL that could
> solve the problems I have, or a single snippet of VCL that you guys are
> actually using on production servers.
>
>   
"man vcl" does provide an example about how to cache even if Cookies are 
present.
Did i miss something?

Regards
A.S


From ssm at linpro.no  Tue Jul  3 05:05:58 2007
From: ssm at linpro.no (Stig Sandbeck Mathisen)
Date: Tue, 03 Jul 2007 07:05:58 +0200
Subject: varnish and Nginx
In-Reply-To: <46890BFD.3030003@gmail.com> (outbackdingo@gmail.com's message of
	"Mon, 02 Jul 2007 17:30:21 +0300")
References: <21171599.18191183385150208.JavaMail.root@ms1.startsiden.no>
	<46890BFD.3030003@gmail.com>
Message-ID: <7xtzsm2i15.fsf@iostat.e.linpro.no>

Dingo <outbackdingo at gmail.com> writes:

> So its feasible to run Nginx as the server/load balancer and varnish
> as the front end cache giving potentially a decent high speed/high
> capacity design.

Yes, those two will complement eachother.

-- 
Stig Sandbeck Mathisen, Linpro


From ask at develooper.com  Tue Jul  3 05:42:41 2007
From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Mon, 2 Jul 2007 22:42:41 -0700
Subject: Problem with varnish and caching
In-Reply-To: <1183386398.27561.20.camel@gabriela>
References: <1183146070.32019.20.camel@gabriela> <87ejjt15nd.fsf@des.linpro.no>
	<1183248616.3955.7.camel@gabriela> <871wfs1p5c.fsf@des.linpro.no>
	<1183317164.14999.12.camel@gabriela> <87sl87z83g.fsf@des.linpro.no>
	<1183386398.27561.20.camel@gabriela>
Message-ID: <C00D25D9-634E-4066-807A-05FC9F472635@develooper.com>


On Jul 2, 2007, at 7:26, Manuel Amador (Rudd-O) wrote:

> I never insulted any of you.

You really should go see a therapist.   Just tell him or her: "People  
think I'm a rude idiot - please help me."

In your weblog post you called Dag-Erling and Poul-Henning "thin- 
skinned".  Rather than disagree (although I do) with that I'll  
contest the assumption that having "thick skin" should be a  
prerequisite for building or participating in an open source community.

To be on-topic: I agree with the current conservative, RFC compliant  
default behavior.   (Or to be more accurate: it was caching just fine  
for my use-case, too).


  - ask

-- 
http://develooper.com/ - http://askask.com/




From gaute at pht.no  Tue Jul  3 06:41:56 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Tue, 3 Jul 2007 08:41:56 +0200
Subject: Failure cenarios?
Message-ID: <200707030841.56626.gaute@pht.no>


I have come to understand that in some builds under some conditions varnish 
may hang or a crash. (we run 1.0.4-3el4.i386.rpm)

I have now routed all our ~180 sites troug varnish, pipe by default, cached 
for selected hostnames. Talk a bout all ones eggs in one basket :)

The way it is all set up, we have varnsih on port 80 on one IP and apache on 
80 on another whch is not in use for anything directly.
If anything should hapen to varnish, or we need to upgrade or anything, 3 
lines to netfilter wil reroute all the trafic directly to apache.

Now the question is, how do I best detect if varnsih should have a problem?
Would it be reasonably reliable to just chek if the pid 
from /var/run/varnish.pid is running, do I need to fetch a page, or is there 
some better way?

Gaute


From denis at startsiden.no  Tue Jul  3 07:47:38 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Tue, 3 Jul 2007 09:47:38 +0200 (CEST)
Subject: Failure cenarios?
In-Reply-To: <200707030841.56626.gaute@pht.no>
Message-ID: <825134.20021183448858290.JavaMail.root@ms1.startsiden.no>

----- Gaute Amundsen <gaute at pht.no> wrote:
> I have come to understand that in some builds under some conditions
> varnish 
> may hang or a crash. (we run 1.0.4-3el4.i386.rpm)

Hi Gaute,

I'll just say that in my experience Varnish has proven itself to be extremely stable. We actually run 1.0.3 across the board (yes I know there are known bugs, however we do not experience them at all) and Varnish currently serves up all requests at www.startsiden.no and www.abcnyheter.no. If anything breaks, it has not so far been varnish. 

However our scenario is pretty different from yours, we have very few vhosts but each has a very high amount of traffic. There is little or no advanced VCL configuration at all on our sites. We're pretty close to the default. The two sites have a different setup with regards to placement of Varnish. One site runs with dedicated varnish servers, the other has varnish and apache2 on the same box.

> Now the question is, how do I best detect if varnsih should have a
> problem?
> Would it be reasonably reliable to just chek if the pid 
> from /var/run/varnish.pid is running, do I need to fetch a page, or is
> there 
> some better way?

Well, we always monitor as high up as possible to make sure everything works on all levels. Lower level monitoring is useful too, but for pinpointing with more accuracy where the problem is. 

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From des at linpro.no  Tue Jul  3 09:27:00 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 03 Jul 2007 11:27:00 +0200
Subject: Failure cenarios?
In-Reply-To: <200707030841.56626.gaute@pht.no> (Gaute Amundsen's message of
	"Tue\, 3 Jul 2007 08\:41\:56 +0200")
References: <200707030841.56626.gaute@pht.no>
Message-ID: <87oditbzx7.fsf@des.linpro.no>

Gaute Amundsen <gaute at pht.no> writes:
> Now the question is, how do I best detect if varnsih should have a
> problem?  Would it be reasonably reliable to just chek if the pid from
> /var/run/varnish.pid is running, do I need to fetch a page, or is
> there some better way?

I would recommend retrieving a page (or a set of pages).  Simply
checking the pid won't help you if Varnish has gone off into la-la land,
or been SIGSTOPped or something.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jul  3 09:28:55 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 03 Jul 2007 11:28:55 +0200
Subject: Varnish Dirty Caching
In-Reply-To: <20070702211450.GA16119@falcon> (varnish@held-im-ruhestand.de's
	message of "Mon\, 2 Jul 2007 23\:14\:50 +0200")
References: <20070702211450.GA16119@falcon>
Message-ID: <87k5thbzu0.fsf@des.linpro.no>

Christoph <varnish at held-im-ruhestand.de> writes:
> So what is dirty caching and why use it? Think of a very unreliable
> backend. If varnish can't reach it's backend, it will simply return
> the last content it has (even if the content is stale). That way i can
> cover hickups.

It's on our list for 2.0, and will probably hit trunk in late July.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From denis at startsiden.no  Tue Jul  3 09:32:45 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Tue, 3 Jul 2007 11:32:45 +0200 (CEST)
Subject: Varnish Dirty Caching
In-Reply-To: <87k5thbzu0.fsf@des.linpro.no>
Message-ID: <8285907.21301183455165837.JavaMail.root@ms1.startsiden.no>

----- Dag-Erling Sm?rgrav <des at linpro.no> wrote:
> Christoph <varnish at held-im-ruhestand.de> writes:
> > So what is dirty caching and why use it? Think of a very unreliable
> > backend. If varnish can't reach it's backend, it will simply return
> > the last content it has (even if the content is stale). That way i
> > can cover hickups.
> It's on our list for 2.0, and will probably hit trunk in late July.

Way cool, really looking forward to that feature! As I stated in a different thread it's more often other parts of the system that break than varnish  so having such a failsafe enables us to do a lot more fault tolerant setups.

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From denis at startsiden.no  Tue Jul  3 09:36:51 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Tue, 3 Jul 2007 11:36:51 +0200 (CEST)
Subject: Problem with varnish and caching
In-Reply-To: <4689D39A.1030606@iamcool.net>
Message-ID: <1676231.21431183455411511.JavaMail.root@ms1.startsiden.no>

----- Anup Shukla <anup at iamcool.net> wrote:
> Manuel Amador (Rudd-O) wrote:
> > site is cached according to Varnish default policies.  You have not
> > provided a single counterexample or a single snippet of VCL that
> > could solve the problems I have, or a single snippet of VCL that you guys
> > are actually using on production servers.
> "man vcl" does provide an example about how to cache even if Cookies
> are present. 
> Did i miss something?

Yeah, I think you missed the fact that Manuel isn't really interested in anything else than spewing out insults and bulls*** on the list. Nothing to see / hear here people move on, just an angry troll.

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From phk at phk.freebsd.dk  Tue Jul  3 10:06:01 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 03 Jul 2007 10:06:01 +0000
Subject: Varnish Dirty Caching 
In-Reply-To: Your message of "Mon, 02 Jul 2007 23:14:50 +0200."
	<20070702211450.GA16119@falcon> 
Message-ID: <31256.1183457161@critter.freebsd.dk>

In message <20070702211450.GA16119 at falcon>, Christoph writes:

>i'd like to implement dirty-caching using varnish.

I'm busy twisting the variable visibility in VCL into proper shape
right now, and that will move us a bit closer to what your want
to do.

The critical question is how we define "backend is down" and how
fast and efficient we can detect it.

Ideas for how to express it in VCL are very welcome.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From dwetzel at nerim.net  Tue Jul  3 10:33:17 2007
From: dwetzel at nerim.net (Damien Wetzel)
Date: Tue, 3 Jul 2007 12:33:17 +0200
Subject: Varnish Dirty Caching 
In-Reply-To: <31256.1183457161@critter.freebsd.dk>
References: <20070702211450.GA16119@falcon>
	<31256.1183457161@critter.freebsd.dk>
Message-ID: <18058.9709.447160.884593@localhost.localdomain>

Hello all,


when working for speedera, we use to have the ability to define
two time-out timers in the config: timer1 and timer2

timer1 would define the time to receive the TCP ack of the http
request to the backend. (or something equivalent)

timer2 the time to receive the first chunk of http data

upon completion of one of these timers we could configure the caches
to say :
1) serve stale content
2) serve a user defined static html page or null gif image
3) redirect the requests to a backup backend.

with savvis we had a variable called Background Refresh Mode when
positionned on a URL would make the cache serve the content
immediatly when it becomes staled and then the cache would (try to) make
the refresh in the background.

Damien, 

Poul-Henning Kamp writes:
 > In message <20070702211450.GA16119 at falcon>, Christoph writes:
 > 
 > >i'd like to implement dirty-caching using varnish.
 > 
 > I'm busy twisting the variable visibility in VCL into proper shape
 > right now, and that will move us a bit closer to what your want
 > to do.
 > 
 > The critical question is how we define "backend is down" and how
 > fast and efficient we can detect it.
 > 
 > Ideas for how to express it in VCL are very welcome.
 > 
 > -- 
 > Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
 > phk at FreeBSD.ORG         | TCP/IP since RFC 956
 > FreeBSD committer       | BSD since 4.3-tahoe    
 > Never attribute to malice what can adequately be explained by incompetence.
 > _______________________________________________
 > varnish-misc mailing list
 > varnish-misc at projects.linpro.no
 > http://projects.linpro.no/mailman/listinfo/varnish-misc


From gaute at pht.no  Tue Jul  3 11:16:27 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Tue, 3 Jul 2007 13:16:27 +0200
Subject: Failure cenarios?
In-Reply-To: <87oditbzx7.fsf@des.linpro.no>
References: <200707030841.56626.gaute@pht.no> <87oditbzx7.fsf@des.linpro.no>
Message-ID: <200707031316.28279.gaute@pht.no>

On Tuesday 03 July 2007 11:27, Dag-Erling Sm?rgrav wrote:
> Gaute Amundsen <gaute at pht.no> writes:
> > Now the question is, how do I best detect if varnsih should have a
> > problem?  Would it be reasonably reliable to just chek if the pid from
> > /var/run/varnish.pid is running, do I need to fetch a page, or is
> > there some better way?
>
> I would recommend retrieving a page (or a set of pages).  Simply
> checking the pid won't help you if Varnish has gone off into la-la land,
> or been SIGSTOPped or something.
>
> DES

Not what I _wanted_ to hear, but what I expected i guess :)

Gaute


From stonorn at giraffen.dk  Tue Jul  3 11:25:05 2007
From: stonorn at giraffen.dk (Anton Stonor)
Date: Tue, 03 Jul 2007 13:25:05 +0200
Subject: Varnish Dirty Caching
In-Reply-To: <31256.1183457161@critter.freebsd.dk>
References: <31256.1183457161@critter.freebsd.dk>
Message-ID: <468A3211.80306@giraffen.dk>

Poul-Henning Kamp wrote:

> The critical question is how we define "backend is down" and how
> fast and efficient we can detect it.

Right. I tend to like the Perlbal approach: Issue a http OPTIONS and 
check if we get anything back from the backend. It is quite lightweight.

/Anton



From jean-marc.pouchoulon at ac-montpellier.fr  Tue Jul  3 12:23:34 2007
From: jean-marc.pouchoulon at ac-montpellier.fr (jean-marc pouchoulon)
Date: Tue, 03 Jul 2007 14:23:34 +0200
Subject: truncated page problem
In-Reply-To: <87ir92dd85.fsf@des.linpro.no>
References: <46891719.1060008@ac-montpellier.fr> <87ir92dd85.fsf@des.linpro.no>
Message-ID: <468A3FC6.1030305@ac-montpellier.fr>

Bonjour,
the problem is not related to varnish , disable compression and all is 
running.

Thanks for your time and work  et sorry for this false alarm

jean-marc.



Dag-Erling Sm?rgrav a ?crit :
> jean-marc pouchoulon <jean-marc.pouchoulon at ac-montpellier.fr> writes:
>   
>> Using varnish with this url:
>>
>> http://publinet.ac-montpellier.fr/publinet/Resultats?Unom_ecr=5&Ubase=publi_13&Utyp_aff=Resultats&Unum_cri=2&Uval_cri=B&Uval_cri1=32405
>>
>> the page is truncated beginning by
>>
>> IGN=LEFT><font face='arial' class="libelleDecision">ADMIS </font></TD>
>> </TR>
>> <TR  class="ligneImpaire">
>> <TD><font face='arial' class="nomCandidat">BRIGATI 
>> Anne-Marie</font></TD><TD ALIGN=LEFT><font face='arial' 
>> class="libelleDecision">ADMIS </font></TD>
>> </TR>
>>
>> ( "line break"  in html page )
>>
>> without varnish , direct to the http server there are no problems
>>
>> I've got this log and a 104 error:
>>
>>    69 RxRequest    c GET
>>    69 RxURL        c /publinet/Resultats?Unom_ecr=5&Ubase=publi_13&Utyp_aff=Resultats&Unum_cri=2&Uval_cri=B&Uval_cri1=32405
>>    69 RxProtocol   c HTTP/1.1
>>    69 RxHeader     c Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/msword, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/x-shockwave-flash, */*
>>    69 RxHeader     c Referer: http://publinet.ac-montpellier.fr/publinet/Resultats?Unom_ecr=5&Ubase=publi_13&Utyp_aff=Resultats&Unum_cri=2&Uval_cri=A&Uval_cri1=32405
>>    69 RxHeader     c Accept-Language: fr
>>    69 RxHeader     c Accept-Encoding:
>>    69 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
>>    69 RxHeader     c Host: publinet.ac-montpellier.fr
>>    69 RxHeader     c Connection: Keep-Alive
>>    69 VCL_call     c recv
>>    69 VCL_return   c lookup
>>    69 VCL_call     c hash
>>    69 VCL_return   c hash
>>    69 Hit          c 187404565
>>    69 VCL_call     c hit
>>    69 VCL_return   c deliver
>>    69 Length       c 16143
>>    69 TxProtocol   c HTTP/1.1
>>    69 TxStatus     c 200
>>    69 TxResponse   c OK
>>    69 TxHeader     c Date: Mon, 02 Jul 2007 14:13:09 GMT
>>    69 TxHeader     c Server: Apache/2.0.52 (Unix) DAV/2
>>    69 TxHeader     c Content-Type: text/html; charset=ISO-8859-1
>>    69 TxHeader     c Content-Length: 16143
>>    69 TxHeader     c X-Varnish: 187476688 187404565
>>    69 TxHeader     c Age: 536
>>    69 TxHeader     c Via: 1.1 varnish
>>    69 ReqEnd       c 187476688 1183386112.545343552 1183386112.545831603 0.000027936 0.000024305 0.000463746
>>     
>
> Looks fine to me...
>
>   
>>     0 StatAddr       90.14.20.225 0 59 20 29 0 0 0 5026 120142
>>    44 HttpError    c Received errno 104
>>    44 SessionClose c no request
>>    44 StatSess     c 86.206.235.88 4737 0 1 1 0 0 0 152 0
>>     
>
> #define ECONNRESET      104     /* Connection reset by peer */
>
> but this is a different request, on a different socket.  Could you send
> me a full log?  Start 'varnishlog -w full.log', start varnishd, request
> the problem page a couple of times (so I can see both a cache miss and a
> cache hit), then stop varnishd, stop varnishlog and send me the file.
>
> DES
>   



From des at linpro.no  Tue Jul  3 12:27:29 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 03 Jul 2007 14:27:29 +0200
Subject: truncated page problem
In-Reply-To: <468A3FC6.1030305@ac-montpellier.fr> (jean-marc pouchoulon's
	message of "Tue\, 03 Jul 2007 14\:23\:34 +0200")
References: <46891719.1060008@ac-montpellier.fr>
	<87ir92dd85.fsf@des.linpro.no> <468A3FC6.1030305@ac-montpellier.fr>
Message-ID: <878x9xbrke.fsf@des.linpro.no>

jean-marc pouchoulon <jean-marc.pouchoulon at ac-montpellier.fr> writes:
> the problem is not related to varnish , disable compression and all is
> running.

Ah, it *is* related to Varnish: 1.0.4 doesn't support Vary: headers.
Compression works fine with trunk, and with the upcoming 1.1.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From tiamo at tfr.org  Tue Jul  3 12:52:53 2007
From: tiamo at tfr.org (TiAMO)
Date: Tue, 3 Jul 2007 14:52:53 +0200
Subject: Config change.
Message-ID: <200707031452.53961.tiamo@tfr.org>

I need to add more backend servers and req.http.Host definitions quite often, 
is there a way to change the config without restarting varnish and loose the 
entire cache?

Or is it just simpler to have a backend on loopback and write a separate app 
to handle the backend distribution without having to restart varnish and 
loose the cache. This might not be a problem for most uses but i plan to 
store around 6TB in the cache on each box, so i would realy like not having 
to refill it each time i add a new site to the caches =)

also is it possible to make varnish listen on sevral adresses/ports, i tried 
with mutiple -a x.x.x.x:y but it did'nt seem to work.

// Fredrik Neij


From phk at phk.freebsd.dk  Tue Jul  3 12:55:00 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 03 Jul 2007 12:55:00 +0000
Subject: Config change. 
In-Reply-To: Your message of "Tue, 03 Jul 2007 14:52:53 +0200."
	<200707031452.53961.tiamo@tfr.org> 
Message-ID: <32074.1183467300@critter.freebsd.dk>

In message <200707031452.53961.tiamo at tfr.org>, TiAMO writes:

>I need to add more backend servers and req.http.Host definitions quite often, 
>is there a way to change the config without restarting varnish and loose the 
>entire cache?

Yes, you can load a new VCL code from the CLI interface with no disruption
to your active traffic.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Tue Jul  3 13:08:10 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 03 Jul 2007 15:08:10 +0200
Subject: Config change.
In-Reply-To: <200707031452.53961.tiamo@tfr.org> (tiamo@tfr.org's message of
	"Tue\, 3 Jul 2007 14\:52\:53 +0200")
References: <200707031452.53961.tiamo@tfr.org>
Message-ID: <874pklbpol.fsf@des.linpro.no>

TiAMO <tiamo at tfr.org> writes:
> I need to add more backend servers and req.http.Host definitions quite
> often, is there a way to change the config without restarting varnish
> and loose the entire cache?

vcl.load newconf /my/new/config
vcl.use newconf

> also is it possible to make varnish listen on sevral adresses/ports, i
> tried with mutiple -a x.x.x.x:y but it did'nt seem to work.

Not supported yet, but if you open a ticket I'll try to fix it for 1.1.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From gaute at pht.no  Tue Jul  3 13:14:31 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Tue, 3 Jul 2007 15:14:31 +0200
Subject: logging problems - "Pipe Shut"
Message-ID: <200707031514.33396.gaute@pht.no>


Hi again.

We have had logging running for about a week now with no apparent problems, 
but yesterday I routed all our traffic into varnish, and now bad things 
happen.

/var/log/varnish/varnish.log just stops growing after a while and a tail -n 1 
gives me pages after pages of:

Pipe Shut write(read)Pipe Shut read(read)Pipe Shut write(read)Pipe Shut 
read(read)Pipe Shut write(read)Pipe Shut read(read)Pipe Shut write(read)Pipe 

Over 4 megs of it just now actualy.

/etc/init.d/varnishlog status reports "varnishlog (pid 11936) running".

I thought it took a fairly long while at first, but now it happens quckly, 
whithin 5 mins.

On a possibly related note, varnishncsa segfaults when trying to process this 
log. both as 
cat /var/log/varnish/varnish.log |  varnishncsa -b -r /dev/stdin and
varnishncsa -c -r /var/log/varnish/varnish.log
Log size ~1G

varnishlog -c -r /var/log/varnish/varnish.log seems to work fine

Truncating and starting over does not seem to help on the first problem, but 
varnishncsa seems to be more comfortable with a smaller log.

What to do?

Gaute



From des at linpro.no  Tue Jul  3 13:18:02 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 03 Jul 2007 15:18:02 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <200707031514.33396.gaute@pht.no> (Gaute Amundsen's message of
	"Tue\, 3 Jul 2007 15\:14\:31 +0200")
References: <200707031514.33396.gaute@pht.no>
Message-ID: <87zm2daanp.fsf@des.linpro.no>

Gaute Amundsen <gaute at pht.no> writes:
> We have had logging running for about a week now with no apparent
> problems, but yesterday I routed all our traffic into varnish, and now
> bad things happen.
>
> /var/log/varnish/varnish.log just stops growing after a while and a
> tail -n 1 gives me pages after pages of:
>
> Pipe Shut write(read)Pipe Shut read(read)Pipe Shut write(read)Pipe
> Shut read(read)Pipe Shut write(read)Pipe Shut read(read)Pipe Shut
> write(read)Pipe
>
> Over 4 megs of it just now actualy.

It's not a text file, so tailing it is meaningless.

Have you verified that logrotate is correctly set up?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From gaute at pht.no  Tue Jul  3 13:31:35 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Tue, 3 Jul 2007 15:31:35 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <87zm2daanp.fsf@des.linpro.no>
References: <200707031514.33396.gaute@pht.no> <87zm2daanp.fsf@des.linpro.no>
Message-ID: <200707031531.36100.gaute@pht.no>

On Tuesday 03 July 2007 15:18, Dag-Erling Sm?rgrav wrote:
> Gaute Amundsen <gaute at pht.no> writes:
> > We have had logging running for about a week now with no apparent
> > problems, but yesterday I routed all our traffic into varnish, and now
> > bad things happen.
> >
> > /var/log/varnish/varnish.log just stops growing after a while and a
> > tail -n 1 gives me pages after pages of:
> >
> > Pipe Shut write(read)Pipe Shut read(read)Pipe Shut write(read)Pipe
> > Shut read(read)Pipe Shut write(read)Pipe Shut read(read)Pipe Shut
> > write(read)Pipe
> >
> > Over 4 megs of it just now actualy.
>
> It's not a text file, so tailing it is meaningless.

I know, in theory, except for tail -f into varnshncsa, but when all is ok I 
have not gotten more than half a screenfull yet.

> Have you verified that logrotate is correctly set up?
>
Not thorougly, but all I have changed is addning:
    prerotate
        /usr/bin/python /root/bin/awstats_from_varnishlog.py
    endscript

Rotation is weekly, and the previous logs have sane dates.

G.


From gaute at pht.no  Tue Jul  3 14:01:10 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Tue, 3 Jul 2007 16:01:10 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <87zm2daanp.fsf@des.linpro.no>
References: <200707031514.33396.gaute@pht.no> <87zm2daanp.fsf@des.linpro.no>
Message-ID: <200707031601.10772.gaute@pht.no>


Hm..

I was finding quite a bit of "Pipe Shut" just running varnishlog -o.
I's out of my buffer, so I cant paste it in right now, but could it bee that I 
was opening to many pipes?

I the default action in vcl_recv was pipe, and only a few hosts would get a 
lookup... Trying with pass now, and it has held upp longer than last time 
anyway.

Gaute


From des at linpro.no  Tue Jul  3 14:05:55 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 03 Jul 2007 16:05:55 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <200707031531.36100.gaute@pht.no> (Gaute Amundsen's message of
	"Tue\, 3 Jul 2007 15\:31\:35 +0200")
References: <200707031514.33396.gaute@pht.no> <87zm2daanp.fsf@des.linpro.no>
	<200707031531.36100.gaute@pht.no>
Message-ID: <87sl85a8fw.fsf@des.linpro.no>

Gaute Amundsen <gaute at pht.no> writes:
> Rotation is weekly, and the previous logs have sane dates.

Weekly rotation is probably far too seldom, Varnish can easily generate
several gigabytes of log data *per hour* under high load.

Have you checked the file size limit (ulimit -f) that varnishlog
operates under?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jul  3 14:14:12 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 03 Jul 2007 16:14:12 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <200707031601.10772.gaute@pht.no> (Gaute Amundsen's message of
	"Tue\, 3 Jul 2007 16\:01\:10 +0200")
References: <200707031514.33396.gaute@pht.no> <87zm2daanp.fsf@des.linpro.no>
	<200707031601.10772.gaute@pht.no>
Message-ID: <87odita823.fsf@des.linpro.no>

Gaute Amundsen <gaute at pht.no> writes:
> I was finding quite a bit of "Pipe Shut" just running varnishlog -o.
> I's out of my buffer, so I cant paste it in right now, but could it
> bee that I was opening to many pipes?

"pipe shut" happens when either the backend or the client closes the
connection, although there is a possibility that the code (rdf() in
cache_pipe.c) is too sensitive and shuts down the pipe as soon as the
client's TCP window fills up.  Could you try the attached patch?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

-------------- next part --------------
A non-text attachment was scrubbed...
Name: cache_pipe.diff
Type: text/x-diff
Size: 1105 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070703/d56146c2/attachment.bin>

From gaute at pht.no  Tue Jul  3 14:15:13 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Tue, 3 Jul 2007 16:15:13 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <87sl85a8fw.fsf@des.linpro.no>
References: <200707031514.33396.gaute@pht.no> <200707031531.36100.gaute@pht.no>
	<87sl85a8fw.fsf@des.linpro.no>
Message-ID: <200707031615.13705.gaute@pht.no>

On Tuesday 03 July 2007 16:05, Dag-Erling Sm?rgrav wrote:
> Gaute Amundsen <gaute at pht.no> writes:
> > Rotation is weekly, and the previous logs have sane dates.
>
> Weekly rotation is probably far too seldom, Varnish can easily generate
> several gigabytes of log data *per hour* under high load.

The tought have struck me yes. What would you suggest?
Daily ought to stay under 5G by guesstimate, or would it be better to rotate 
by size? What size is varnishncsa comfortable with?

> Have you checked the file size limit (ulimit -f) that varnishlog
> operates under?
>
unlimited

Gaute


From des at linpro.no  Tue Jul  3 14:30:47 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 03 Jul 2007 16:30:47 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <200707031615.13705.gaute@pht.no> (Gaute Amundsen's message of
	"Tue\, 3 Jul 2007 16\:15\:13 +0200")
References: <200707031514.33396.gaute@pht.no>
	<200707031531.36100.gaute@pht.no> <87sl85a8fw.fsf@des.linpro.no>
	<200707031615.13705.gaute@pht.no>
Message-ID: <87k5tha7ag.fsf@des.linpro.no>

Gaute Amundsen <gaute at pht.no> writes:
> On Tuesday 03 July 2007 16:05, Dag-Erling Sm?rgrav wrote:
> > Gaute Amundsen <gaute at pht.no> writes:
> > > Rotation is weekly, and the previous logs have sane dates.
> > Weekly rotation is probably far too seldom, Varnish can easily generate
> > several gigabytes of log data *per hour* under high load.
> The tought have struck me yes. What would you suggest?  Daily ought to
> stay under 5G by guesstimate, or would it be better to rotate by size?
> What size is varnishncsa comfortable with?

varnishncsa shouldn't care, as it processes the log file linearly, but I
generally prefer to rotate by size.

Is this a 32-bit machine, BTW?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From james at nyi.net  Tue Jul  3 15:00:25 2007
From: james at nyi.net (James Quacinella)
Date: Tue, 03 Jul 2007 11:00:25 -0400
Subject: Failure cenarios?
In-Reply-To: <200707031316.28279.gaute@pht.no>
References: <200707030841.56626.gaute@pht.no> <87oditbzx7.fsf@des.linpro.no>
	<200707031316.28279.gaute@pht.no>
Message-ID: <468A6489.4040209@nyi.net>

Gaute Amundsen wrote:
> On Tuesday 03 July 2007 11:27, Dag-Erling Sm?rgrav wrote:
>   
>>
>> I would recommend retrieving a page (or a set of pages).  Simply
>> checking the pid won't help you if Varnish has gone off into la-la land,
>> or been SIGSTOPped or something.
>>
>> DES
>>     
> Not what I _wanted_ to hear, but what I expected i guess :)
>   

I use monit for monitoring programs. Here is a snippet I had used when 
monitoring a varnish install (too bad it never went into production; 
change values to you liking / environment):

##
## Check Varnishd
check process varnishd with pidfile /var/run/varnishd.pid
   start program = "/etc/init.d/varnishd start"
   stop program  = "/etc/init.d/varnishd stop"
   if cpu > 60% for 2 cycles then alert
   if cpu > 80% for 5 cycles then alert
   if children > 50 then alert
   if loadavg(5min) greater than 5 for 2 cycles then alert
   if 3 restarts within 3 cycles then timeout
   if failed host ipaddy port 80 type tcp then restart
   if failed host ipaddy port 8080 type tcp send "ping\r\n" then restart

Monit also allows you to check the response using a regex, though I 
never got it to work. Check the manual at 
http://www.tildeslash.com/monit/doc/manual.php#connection_testing

Also, maybe you can use swatch to monitor its log file for nasty things 
and automatically restart it / email you when it happens?

-- james


From gaute at pht.no  Tue Jul  3 16:08:33 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Tue, 3 Jul 2007 18:08:33 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <87k5tha7ag.fsf@des.linpro.no>
References: <200707031514.33396.gaute@pht.no> <200707031615.13705.gaute@pht.no>
	<87k5tha7ag.fsf@des.linpro.no>
Message-ID: <200707031808.34094.gaute@pht.no>

On Tuesday 03 July 2007 16:30, Dag-Erling Sm?rgrav wrote:
<snip>
> varnishncsa shouldn't care, as it processes the log file linearly, but I
> generally prefer to rotate by size.

Hm.. ok. But that would have me running awstats at odd times..
Wil just have to try it out I guess

> Is this a 32-bit machine, BTW?

It is. Quad Xeon 3 GHz.

>Could you try the attached patch?

Having some problems with that. My fault I'm sure, but I installed from the 
RPM originaly, so I can't just re-run. Compilation goes fine, but the 
binaries ends up widely different sizes. (5567 vs 168684) 
before "make install" that is. After that it's 478984 vs 168684.
The source rpms are no different i presume?

Is there a way to get the binary made without doing an install?
I'd rather not do that on our production box..

Gaute


From anup at iamcool.net  Wed Jul  4 04:07:46 2007
From: anup at iamcool.net (Anup Shukla)
Date: Wed, 04 Jul 2007 09:37:46 +0530
Subject: Problem with varnish and caching
In-Reply-To: <1183480023.1553.36.camel@gabriela>
References: <1183146070.32019.20.camel@gabriela>	
	<87ejjt15nd.fsf@des.linpro.no> <1183248616.3955.7.camel@gabriela>	
	<871wfs1p5c.fsf@des.linpro.no>
	<1183317164.14999.12.camel@gabriela>	
	<87sl87z83g.fsf@des.linpro.no>
	<1183386398.27561.20.camel@gabriela>	
	<4689D39A.1030606@iamcool.net> <1183480023.1553.36.camel@gabriela>
Message-ID: <468B1D12.40803@iamcool.net>

Manuel Amador (Rudd-O) wrote:
> The optimum behaviour would be Varnish caching static files
> (img,css,jpg,png, based on response HTTP headers) and not caching
> dynamic ones.  But as it currently stands, Varnish configuration
> language doesn't allow caching to be controlled based on the response
> headers, only on the request ones.  Squid, however, does.
>
>   

I had a similar situation.
In my case the site is 100% php, so in effect anything that comes out 
with Content-Type text/html is php generated and given the nature of the 
site, its not cacheable.

I have also put below my VCL configuration which works fine for me.
Its not much of a high quality configuration, but works for me.

Suggestion, as always, are welcome :)

sub vcl_recv {
        if (req.request == "GET" && req.http.cookie) {
                lookup;
        }
}

sub vcl_fetch {
        if (!obj.valid) {
                error;
        }
        if (!obj.cacheable) {
                pass;
        }
        if (resp.http.Content-Type ~ "text/html") {
                pass;
        }
        if (resp.http.Set-Cookie) {
                insert;
        }
        insert;
}

Hope this helps.

Regards
A.S


From anup at iamcool.net  Wed Jul  4 04:11:49 2007
From: anup at iamcool.net (Anup Shukla)
Date: Wed, 04 Jul 2007 09:41:49 +0530
Subject: Failure cenarios?
In-Reply-To: <468A6489.4040209@nyi.net>
References: <200707030841.56626.gaute@pht.no>
	<87oditbzx7.fsf@des.linpro.no>	<200707031316.28279.gaute@pht.no>
	<468A6489.4040209@nyi.net>
Message-ID: <468B1E05.9010400@iamcool.net>

James Quacinella wrote:
> I use monit for monitoring programs. Here is a snippet I had used when 
> monitoring a varnish install (too bad it never went into production; 
> change values to you liking / environment):
>   

A Billion thanks to you :)
I have been looking for a tool that could do this for me.

Regards
A.S


From varnish at held-im-ruhestand.de  Wed Jul  4 07:23:32 2007
From: varnish at held-im-ruhestand.de (Christoph)
Date: Wed, 4 Jul 2007 09:23:32 +0200
Subject: Q: Caching Hierachy
Message-ID: <20070704072332.GA23899@supporter.megaspace.de>

I need to build a hierachical caching plattform. The Goal is, to have
multiple frontends that will expire in sync. 

Currently we cascade a pool of Cacher in front of a single Intermediate
Cache. The Front-Cacher will honor any "Age"-Header it will retrieve
from the Intermediate. 

So cacher-1 will be asked for a document at 10:00. It will ask the
intermediate, this one will ask the backend. 

Next cacher-2 will be asked for a document at 10:01. It will ask the
intermediate, intermediate will return the document with an age header
of 60 Seconds. 

After max-age = 300 Seconds, the document will be invalid on cacher-1
and cacher-2. 

Is this possible with varnish? 

Greetings
   Christoph


From des at linpro.no  Wed Jul  4 07:47:56 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Wed, 04 Jul 2007 09:47:56 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <200707031808.34094.gaute@pht.no> (Gaute Amundsen's message of
	"Tue\, 3 Jul 2007 18\:08\:33 +0200")
References: <200707031514.33396.gaute@pht.no>
	<200707031615.13705.gaute@pht.no> <87k5tha7ag.fsf@des.linpro.no>
	<200707031808.34094.gaute@pht.no>
Message-ID: <87ir90haoj.fsf@des.linpro.no>

Gaute Amundsen <gaute at pht.no> writes:
> On Tuesday 03 July 2007 16:30, Dag-Erling Sm?rgrav wrote:
> > Is this a 32-bit machine, BTW?
> It is. Quad Xeon 3 GHz.

Ah, OK, you won't be able to process log files larger than about 2 GB on
a 32-bit machine.  I should probably figure out a way to get around
that.

> > Could you try the attached patch?
> Having some problems with that. My fault I'm sure, but I installed
> from the RPM originaly, so I can't just re-run. Compilation goes fine,
> but the binaries ends up widely different sizes. (5567 vs 168684)
> before "make install" that is. After that it's 478984 vs 168684.  The
> source rpms are no different i presume?

What configure options did you use?  The difference in size is probably
due to debugging symbols.

> Is there a way to get the binary made without doing an install?

You can install into a staging directory (make install
DESTDIR=/tmp/varnish) and copy the files over manually if you prefer.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rudd-o at rudd-o.com  Tue Jul  3 16:27:03 2007
From: rudd-o at rudd-o.com (Manuel Amador (Rudd-O))
Date: Tue, 03 Jul 2007 11:27:03 -0500
Subject: Problem with varnish and caching
In-Reply-To: <4689D39A.1030606@iamcool.net>
References: <1183146070.32019.20.camel@gabriela>
	<87ejjt15nd.fsf@des.linpro.no> <1183248616.3955.7.camel@gabriela>
	<871wfs1p5c.fsf@des.linpro.no> <1183317164.14999.12.camel@gabriela>
	<87sl87z83g.fsf@des.linpro.no> <1183386398.27561.20.camel@gabriela>
	<4689D39A.1030606@iamcool.net>
Message-ID: <1183480023.1553.36.camel@gabriela>

Thanks, Anup.  I did read the vcl man page several times.  The technical
problem with Varnish is that, if I set it up to cache things even if
Cookies are present, my site malfunctions because pages for logged-in
users are cached.  I have to make the caching conditional to the login
Cookie specifically (I did use a conditional "hack" to get this in
default.vcl) otherwise the site completely malfunctions.

The optimum behaviour would be Varnish caching static files
(img,css,jpg,png, based on response HTTP headers) and not caching
dynamic ones.  But as it currently stands, Varnish configuration
language doesn't allow caching to be controlled based on the response
headers, only on the request ones.  Squid, however, does.

On mar, 2007-07-03 at 10:12 +0530, Anup Shukla wrote:
> Manuel Amador (Rudd-O) wrote:
> > site is cached according to Varnish default policies.  You have not
> > provided a single counterexample or a single snippet of VCL that could
> > solve the problems I have, or a single snippet of VCL that you guys are
> > actually using on production servers.
> >
> >   
> "man vcl" does provide an example about how to cache even if Cookies are 
> present.
> Did i miss something?
> 
> Regards
> A.S
	Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
	The R Zone - http://rudd-o.com/
	GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: 
Q:	What do you say to a New Yorker with a job?
A:	Big Mac, fries and a Coke, please!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070703/54f915b1/attachment.bin>

From rudd-o at rudd-o.com  Tue Jul  3 16:30:23 2007
From: rudd-o at rudd-o.com (Manuel Amador (Rudd-O))
Date: Tue, 03 Jul 2007 11:30:23 -0500
Subject: Problem with varnish and caching
In-Reply-To: <C00D25D9-634E-4066-807A-05FC9F472635@develooper.com>
References: <1183146070.32019.20.camel@gabriela>
	<87ejjt15nd.fsf@des.linpro.no> <1183248616.3955.7.camel@gabriela>
	<871wfs1p5c.fsf@des.linpro.no> <1183317164.14999.12.camel@gabriela>
	<87sl87z83g.fsf@des.linpro.no> <1183386398.27561.20.camel@gabriela>
	<C00D25D9-634E-4066-807A-05FC9F472635@develooper.com>
Message-ID: <1183480223.1553.41.camel@gabriela>

On lun, 2007-07-02 at 22:42 -0700, Ask Bj?rn Hansen wrote:
> On Jul 2, 2007, at 7:26, Manuel Amador (Rudd-O) wrote:
> 
> > I never insulted any of you.
> 
> You really should go see a therapist.   Just tell him or her: "People  
> think I'm a rude idiot - please help me."

I will convey your thoughts to my therapist.  We'll probably laugh about
it over a bottle (or two) of scotch.

> 
> In your weblog post you called Dag-Erling and Poul-Henning "thin- 
> skinned".  Rather than disagree (although I do) with that I'll  
> contest the assumption that having "thick skin" should be a  
> prerequisite for building or participating in an open source community.

Yes, having a thick skin is a prerequisite for participating in a
community where everyone can actually *talk* to you and *reach* you.
Being thick-skinned means:

- not taking criticism of your product PERSONALLY
- conceding that others may have a point, however wrong their arguments
appear to be
- empathy

Under that definition, neither Dag-Erling, nor Poul-Henning have a thick
skin.

> 
> To be on-topic: I agree with the current conservative, RFC compliant  
> default behavior.   (Or to be more accurate: it was caching just fine  
> for my use-case, too).

Well, then Varnish is a perfect fit for you.  It wasn't for me, and (I
spent some time reading Varnish's varnish-misc mailing list) I
discovered others had experienced the same problem.

> 
> 
>   - ask
> 
	Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
	The R Zone - http://rudd-o.com/
	GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: 
Never reveal your best argument.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070703/98b06b78/attachment.bin>

From rudd-o at rudd-o.com  Wed Jul  4 05:03:30 2007
From: rudd-o at rudd-o.com (Manuel Amador (Rudd-O))
Date: Wed, 04 Jul 2007 00:03:30 -0500
Subject: Problem with varnish and caching
In-Reply-To: <468B1D12.40803@iamcool.net>
References: <1183146070.32019.20.camel@gabriela>
	<87ejjt15nd.fsf@des.linpro.no> <1183248616.3955.7.camel@gabriela>
	<871wfs1p5c.fsf@des.linpro.no> <1183317164.14999.12.camel@gabriela>
	<87sl87z83g.fsf@des.linpro.no> <1183386398.27561.20.camel@gabriela>
	<4689D39A.1030606@iamcool.net> <1183480023.1553.36.camel@gabriela>
	<468B1D12.40803@iamcool.net>
Message-ID: <1183525411.12771.5.camel@gabriela>

Thanks a lot.  Will use this to tune Varnish better.  This functionality
is not mentioned in the manual page of vcl, nor was it conveyed to me by
Dag-Erling or Poul-Henning.

On mi?, 2007-07-04 at 09:37 +0530, Anup Shukla wrote:
> Manuel Amador (Rudd-O) wrote:
> > The optimum behaviour would be Varnish caching static files
> > (img,css,jpg,png, based on response HTTP headers) and not caching
> > dynamic ones.  But as it currently stands, Varnish configuration
> > language doesn't allow caching to be controlled based on the response
> > headers, only on the request ones.  Squid, however, does.
> >
> >   
> 
> I had a similar situation.
> In my case the site is 100% php, so in effect anything that comes out 
> with Content-Type text/html is php generated and given the nature of the 
> site, its not cacheable.
> 
> I have also put below my VCL configuration which works fine for me.
> Its not much of a high quality configuration, but works for me.
> 
> Suggestion, as always, are welcome :)
> 
> sub vcl_recv {
>         if (req.request == "GET" && req.http.cookie) {
>                 lookup;
>         }
> }
> 
> sub vcl_fetch {
>         if (!obj.valid) {
>                 error;
>         }
>         if (!obj.cacheable) {
>                 pass;
>         }
>         if (resp.http.Content-Type ~ "text/html") {
>                 pass;
>         }
>         if (resp.http.Set-Cookie) {
>                 insert;
>         }
>         insert;
> }
> 
> Hope this helps.
> 
> Regards
> A.S
	Manuel Amador (Rudd-O) <rudd-o at rudd-o.com>
	The R Zone - http://rudd-o.com/
	GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/

Now playing, courtesy of Amarok: VA - Ingen sommar utan reggae - Markoolio
As flies to wanton boys are we to the gods; they kill us for their sport.
		-- Shakespeare, "King Lear"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070704/a95d7642/attachment.bin>

From ask at develooper.com  Wed Jul  4 07:52:43 2007
From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Wed, 4 Jul 2007 00:52:43 -0700
Subject: Q: Caching Hierachy
In-Reply-To: <20070704072332.GA23899@supporter.megaspace.de>
References: <20070704072332.GA23899@supporter.megaspace.de>
Message-ID: <FF6F9CD4-39FA-48E9-9893-8F8A50E66015@develooper.com>


On Jul 4, 2007, at 12:23 AM, Christoph wrote:

> I need to build a hierachical caching plattform. The Goal is, to have
> multiple frontends that will expire in sync.
[...]
> Is this possible with varnish?

I haven't tried it with varnish, but I think it should work if you  
use "Expires: ..." rather than "Cache-Control: max-age=..." to  
specify the TTL.

I'm curious why you want to use that setup though.  It seems like you  
are just introducing an extra single point of failure and wasting a  
lot of memory.    There are at least a couple of different setups  
that seems more reasonable.    What are you trying to accomplish?

  - ask

-- 
http://develooper.com/ - http://askask.com/




From varnish at held-im-ruhestand.de  Wed Jul  4 08:33:52 2007
From: varnish at held-im-ruhestand.de (Christoph)
Date: Wed, 4 Jul 2007 10:33:52 +0200
Subject: Q: Caching Hierachy
In-Reply-To: <FF6F9CD4-39FA-48E9-9893-8F8A50E66015@develooper.com>
References: <20070704072332.GA23899@supporter.megaspace.de>
	<FF6F9CD4-39FA-48E9-9893-8F8A50E66015@develooper.com>
Message-ID: <20070704083352.GA24174@supporter.megaspace.de>

On Wed, Jul 04, 2007 at 12:52:43AM -0700, Ask Bjrn Hansen wrote:
> 
> On Jul 4, 2007, at 12:23 AM, Christoph wrote:
> 
> >I need to build a hierachical caching plattform. The Goal is, to have
> >multiple frontends that will expire in sync.
> [...]
> >Is this possible with varnish?
> 
> I haven't tried it with varnish, but I think it should work if you  
> use "Expires: ..." rather than "Cache-Control: max-age=..." to  
> specify the TTL.

The backend (apache) will always set a relative Expires Headers
(access+300seconds)

> 
> I'm curious why you want to use that setup though.  It seems like you  
> are just introducing an extra single point of failure and wasting a  
> lot of memory.    There are at least a couple of different setups  
> that seems more reasonable.    What are you trying to accomplish?

Goal is to synchronize all cachers. Have content expire at the same time
on all cachers. All cachers will be used in a round-robin-DNS entry. As
users are not bound to always hit the same Cacher. if a user hits reload
he might get jumping-content when switching between cachers. 

Could you hint me on a setup to reach this goal?

Greetings
   Christoph


From havard.futseter at met.no  Wed Jul  4 10:46:50 2007
From: havard.futseter at met.no (=?ISO-8859-1?Q?H=E5vard_Futs=E6ter?=)
Date: Wed, 04 Jul 2007 12:46:50 +0200
Subject: child process restarts frequently
Message-ID: <468B7A9A.3040706@met.no>

We have a problem with the child process regularly dying and restarting.

We use Varnish in front of three backends, each serving different types 
of content. One of these backends is quite slow, and initially we 
thought the problem with the child process was related to this slow 
backend. We hoped that upgrading from 1.0.3 to 1.0.4 would solve this, 
but unfortunately is has not.

We have a 64-bits machine with a 400G disk cache.

Below I have attached excerpts from what I think is relevant parts of 
the log( with some description in between) for one of these restart events:

-------------------------------------

First request with id 138012248 is fetched and inserted in the cache:

76 ReqStart     c xxx.xx.xxx.xx 4589 138012248
    76 RxRequest    c GET
    76 RxURL        c 
/weatherapi/locationforecast/1.1/?lat=59.2082531856325;lon=
9.60892344715165
    76 RxProtocol   c HTTP/1.1
    76 VCL_call     c recv
    76 VCL_return   c lookup
    76 VCL_call     c hash
    76 VCL_return   c hash
    76 VCL_call     c miss
    76 VCL_return   c fetch
    55 BackendXID   b 138012248
    76 Backend      c 55 weatherapi
    55 TxRequest    b GET
    55 TxURL        b 
/weatherapi/locationforecast/1.1/?lat=59.2082531856325;lon=
9.60892344715165
    55 TxProtocol   b HTTP/1.1
    55 TxHeader     b X-Varnish: 138012248
    55 TxHeader     b X-Forwarded-for:
    55 RxProtocol   b HTTP/1.1
    55 RxStatus     b 203
    55 RxResponse   b Non-Authoritative Information
    55 RxHeader     b Date: Tue, 03 Jul 2007 14:07:08 GMT
    55 RxHeader     b Server: Apache/1.3.34 (Debian) mod_perl/1.29
    55 RxHeader     b Expires: Tue, 03 Jul 2007 15:00:00 GMT
    55 RxHeader     b Content-length: 115322
    55 RxHeader     b Content-Type: text/xml; charset=utf-8
    76 ObjProtocol  c HTTP/1.1
    76 ObjStatus    c 203
    76 ObjResponse  c Non-Authoritative Information
    76 ObjHeader    c Date: Tue, 03 Jul 2007 14:07:08 GMT
    76 ObjHeader    c Server: Apache/1.3.34 (Debian) mod_perl/1.29
    76 ObjHeader    c Expires: Tue, 03 Jul 2007 15:00:00 GMT
    76 ObjHeader    c Content-Type: text/xml; charset=utf-8
    76 ObjHeader    c Content-Length: 115322
    55 BackendReuse b weatherapi
    76 TTL          c 138012248 RFC 3172 1183471628 1183471628 
1183474800 0 0
    76 VCL_call     c fetch
    76 VCL_return   c insert
    76 Length       c 115322
    76 TxProtocol   c HTTP/1.1
    76 TxStatus     c 203
    76 TxResponse   c Non-Authoritative Information
76 ObjHeader    c Date: Tue, 03 Jul 2007 14:07:08 GMT
    76 ObjHeader    c Server: Apache/1.3.34 (Debian) mod_perl/1.29
    76 ObjHeader    c Expires: Tue, 03 Jul 2007 15:00:00 GMT
    76 ObjHeader    c Content-Type: text/xml; charset=utf-8
    76 ObjHeader    c Content-Length: 115322
    55 BackendReuse b weatherapi
    76 TTL          c 138012248 RFC 3172 1183471628 1183471628 
1183474800 0 0
    76 VCL_call     c fetch
    76 VCL_return   c insert
    76 Length       c 115322
    76 TxProtocol   c HTTP/1.1
    76 TxStatus     c 203
    76 TxResponse   c Non-Authoritative Information
    76 TxHeader     c Date: Tue, 03 Jul 2007 14:07:08 GMT
    76 TxHeader     c Server: Apache/1.3.34 (Debian) mod_perl/1.29
    76 TxHeader     c Expires: Tue, 03 Jul 2007 15:00:00 GMT
    76 TxHeader     c Content-Type: text/xml; charset=utf-8
    76 TxHeader     c Content-Length: 115322
    76 TxHeader     c X-Varnish: 138012248
    76 TxHeader     c Age: 0
    76 TxHeader     c Via: 1.1 varnish
    76 ReqEnd       c 138012248 1183471628.619347000 
1183471628.781244000 0.05796
5000 0.161765000 0.000132000
     0 StatAddr       xxx.xx.xxx.xxx 0 22038 3349 127106 0 0 18810 
37429990 95687
09644


This object is then requested and delivered from cache several times. 
Then we see the following in the log, at a time immediately after 14:59:27:


0 ExpPick        138012248
0 VCL_call       timeout
0 VCL_return     discard

...immediately following this( ca.14:59:33 ) we get:

0 ExpBan         138012248 was banned
0 CLI            Rd vcl.load boot /tmp/vcl.XXfdES7y
0 CLI            Wr 0 200 Loaded "/tmp/vcl.XXfdES7y" as "boot"

0 CLI            Rd vcl.use boot
0 CLI            Wr 0 200
0 CLI            Rd start
0 CLI            Wr 0 200

This pattern repeats itself regularly every 5 - 15 hours. I am not sure 
  if the request and the ExpBan tag is related to the child process 
beeing restarted, but since a restart is always preceded by such a tag 
it does seem likely.

The weird thing is that it seems to happen at the exact same time every 
day. At 07:59 and 14:59 the child processs is restarted in the manner 
shown above. We have a lot of requests which expires at each whole hour, 
but I have no idea if this is related in any way.

---------------------------------------------

It would be great if anyone could explain this behaviour? What does the 
ExpBan tag really mean? It seems kind of counterintuitive that a request 
can be banned after beeing discarded....

Also, please let me know if you need any more information to be able to 
look into the problem.


Best regards,
H?vard Futs?ter


From gaute at pht.no  Wed Jul  4 12:19:59 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Wed, 4 Jul 2007 14:19:59 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <87ir90haoj.fsf@des.linpro.no>
References: <200707031514.33396.gaute@pht.no> <200707031808.34094.gaute@pht.no>
	<87ir90haoj.fsf@des.linpro.no>
Message-ID: <200707041420.00179.gaute@pht.no>

On Wednesday 04 July 2007 09:47, Dag-Erling Sm?rgrav wrote:
> Gaute Amundsen <gaute at pht.no> writes:
> > On Tuesday 03 July 2007 16:30, Dag-Erling Sm?rgrav wrote:
> > > Is this a 32-bit machine, BTW?
> >
> > It is. Quad Xeon 3 GHz.
>
> Ah, OK, you won't be able to process log files larger than about 2 GB on
> a 32-bit machine.  I should probably figure out a way to get around
> that.

Hm.. I had not tought of that. But we only have a 30G /var, so we wil need to 
keep the files fairly small anyway.
Perhaps we can just save the nsca-log and delete the varnishlog right in the 
postrotate-script, and then rotate every hour.

> > > Could you try the attached patch?
> >
> > Having some problems with that. My fault I'm sure, but I installed
> > from the RPM originaly, so I can't just re-run. Compilation goes fine,
> > but the binaries ends up widely different sizes. (5567 vs 168684)
> > before "make install" that is. After that it's 478984 vs 168684.  The
> > source rpms are no different i presume?
>
> What configure options did you use?  The difference in size is probably
> due to debugging symbols.

None at all. I'm not very knowledgeable about c development...

> > Is there a way to get the binary made without doing an install?
>
> You can install into a staging directory (make install
> DESTDIR=/tmp/varnish) and copy the files over manually if you prefer.

Ah, that's an improvement :)


But, anyway, I had much better results with the rpm, just a few K difference.
1.0.4-4el4.i386 running nicely now :)
( I cheked the cache_pipe.c that rpmbuild left behind to verify that the patch 
was applied. )

But.. the problem with "Pipe Shut" persists. 
Logging stopps within 5 minutes.

Doing 'tail /var/log/varnish/varnish.log | varnishlog -or /dev/stdin' 
displays:
<snip>
  276 VCL_call
26992 RxRequest      e Shut read(read)
30578 (null)         ite(read)
  277 VCL_call
26992 (null)         e Shut write(read)
 8306 (null)         ead(read)
25888 (null)         Shut write(read)
24932 (null)         )
  276 VCL_call
26992 RxRequest      e Shut read(read)
30578 (null)         ite(read)
  277 VCL_call
26992 (null)         e Shut write(read)
 8306 (null)         ead(read)
25888 (null)         Shut write(read)
24932 (null)         )
  276 VCL_call
26992 RxRequest      e Shut read(read)
 9269 BackendOpen  b


When I let the vcl_recv default be pass, it seems much more stable.

Gaute






From andre.cruz at segula.pt  Wed Jul  4 14:39:46 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Wed, 4 Jul 2007 15:39:46 +0100
Subject: Fetch called after pass
Message-ID: <D5515C23-24A5-40B8-B516-7DCA575EF39A@segula.pt>

Hello.

After reading the vcl man page I was under the impression that after  
a "pass" return from vcl_recv the request was simply forwarded to the  
backend and the response would NOT be cached. In other words:

pass - bypass cache and just forward response from backend.
pipe - bypass cache and possibly insert response into cache depending  
no vcl_fetch
lookup - look at cache and if not found contact backend, etc, etc.

But from the log bellow I see fetch being called after a pass (and in  
this case the response gets inserted into the cache). Are my  
assumptions wrong?

btw I'm using trunk from 3/07/2007.

Regards,
Andr? Cruz


      0 WorkThread     0xb4bd6134 start
    11 SessionOpen  c 10.134.145.2 51268
    11 ReqStart     c 10.134.145.2 51268 1192805805
    11 RxRequest    c GET
    11 RxURL        c /index.bml
    11 RxProtocol   c HTTP/1.1
    11 RxHeader     c Host: andre.localhost:8080
    11 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; U; Intel  
Mac OS X; en-US; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
    11 RxHeader     c Accept: text/xml,application/xml,application/ 
xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    11 RxHeader     c Accept-Language: en-us,en;q=0.5
    11 RxHeader     c Accept-Encoding: gzip,deflate
    11 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    11 RxHeader     c Keep-Alive: 300
    11 RxHeader     c Connection: keep-alive
    11 RxHeader     c Cookie: ljuniq=btZhgKFp2s9UaAi:1183474072;  
ljuniq=btZhgKFp2s9UaAi:1183474005; ljsession=ws:infernogelado: 
244:RZWHic6Puc; ljsession=ws:infernogelado:244:RZWHic6Puc
    11 RxHeader     c If-None-Match: "98eb61ba9c2e7b12d4fce78c559bf1fc"
    11 VCL_call     c recv
    11 VCL_return   c pass
    11 VCL_call     c pass
    11 VCL_return   c pass
    11 VCL_call     c recv
    11 VCL_return   c pass
    11 VCL_call     c pass
    11 VCL_return   c pass
    14 BackendOpen  b default 127.0.0.1 55997 127.0.0.1 80
    14 BackendXID   b 1192805805
    11 Backend      c 14 default
    14 TxRequest    b GET
    14 TxURL        b /index.bml
    14 TxProtocol   b HTTP/1.1
    14 TxHeader     b Host: andre.localhost:8080
    14 TxHeader     b User-Agent: Mozilla/5.0 (Macintosh; U; Intel  
Mac OS X; en-US; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
    14 TxHeader     b Accept: text/xml,application/xml,application/ 
xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    14 TxHeader     b Accept-Language: en-us,en;q=0.5
    14 TxHeader     b Accept-Encoding: gzip,deflate
    14 TxHeader     b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    14 TxHeader     b Cookie: ljuniq=btZhgKFp2s9UaAi:1183474072;  
ljuniq=btZhgKFp2s9UaAi:1183474005; ljsession=ws:infernogelado: 
244:RZWHic6Puc; ljsession=ws:infernogelado:244:RZWHic6Puc
    14 TxHeader     b X-Varnish: 1192805805
    14 TxHeader     b X-Forwarded-for: 10.134.145.2
    14 RxProtocol   b HTTP/1.1
    14 RxStatus     b 200
    14 RxResponse   b OK
    14 RxHeader     b Date: Wed, 04 Jul 2007 14:18:00 GMT
    14 RxHeader     b Server: Apache/1.3.34 (Ubuntu) mod_perl/1.29
    14 RxHeader     b Cache-Control: private, proxy-revalidate
    14 RxHeader     b ETag: "98eb61ba9c2e7b12d4fce78c559bf1fc"
    14 RxHeader     b Content-length: 7749
    14 RxHeader     b Content-Type: text/html; charset=utf-8
    14 RxHeader     b Content-Language: en
    11 ObjProtocol  c HTTP/1.1
    11 ObjStatus    c 200
    11 ObjResponse  c OK
    11 ObjHeader    c Date: Wed, 04 Jul 2007 14:18:00 GMT
    11 ObjHeader    c Server: Apache/1.3.34 (Ubuntu) mod_perl/1.29
    11 ObjHeader    c Cache-Control: private, proxy-revalidate
    11 ObjHeader    c ETag: "98eb61ba9c2e7b12d4fce78c559bf1fc"
    11 ObjHeader    c Content-Type: text/html; charset=utf-8
    11 ObjHeader    c Content-Language: en
    11 ObjHeader    c Content-Length: 7749
    14 BackendReuse b default
    11 TTL          c 1192805805 RFC 120 1183558680 1183558680 0 0 0
    11 VCL_call     c fetch
    11 VCL_return   c insert
    11 Length       c 7749
    11 TxProtocol   c HTTP/1.1
    11 TxStatus     c 200
    11 TxResponse   c OK
    11 TxHeader     c Date: Wed, 04 Jul 2007 14:18:00 GMT
    11 TxHeader     c Server: Apache/1.3.34 (Ubuntu) mod_perl/1.29
    11 TxHeader     c Cache-Control: private, proxy-revalidate
    11 TxHeader     c ETag: "98eb61ba9c2e7b12d4fce78c559bf1fc"
    11 TxHeader     c Content-Type: text/html; charset=utf-8
    11 TxHeader     c Content-Language: en
    11 TxHeader     c Content-Length: 7749
    11 TxHeader     c X-Varnish: 1192805805
    11 TxHeader     c Age: 0
    11 TxHeader     c Via: 1.1 varnish
    11 ReqEnd       c 1192805805 1183558680.398978585  
1183558680.420266415 0.000327788 0.021247632 0.000040198
     0 StatAddr       10.134.145.2 0 0 1 1 0 0 1 319 7749


From andre.cruz at segula.pt  Wed Jul  4 15:58:55 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Wed, 4 Jul 2007 16:58:55 +0100
Subject: Post requests
Message-ID: <55305F06-58BD-4B01-9C5F-923B70B9895A@segula.pt>

Hello again.

I'm having problems with POST requests. They reach varnish but  
nothing happens and the backend is not contacted.

I see this in the log:
    11 SessionOpen  c 10.134.145.2 51721

...
    11 Debug        c "Pipe Shut read(read)"
    11 Debug        c "Pipe Shut write(read)"

I found this ticket "http://varnish.projects.linpro.no/ticket/47"  
which seemed to refer to my problem but even switching to "pipe" does  
not solve it.

Is this a known problem?

Thanks,
Andr? Cruz


From andre.cruz at segula.pt  Wed Jul  4 16:21:41 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Wed, 4 Jul 2007 17:21:41 +0100
Subject: Post requests
In-Reply-To: <55305F06-58BD-4B01-9C5F-923B70B9895A@segula.pt>
References: <55305F06-58BD-4B01-9C5F-923B70B9895A@segula.pt>
Message-ID: <8168E1DE-25FD-457E-AB47-3193C35E5127@segula.pt>

Forgot to include a pic of the result in the browser after some time  
passes.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Skitch.jpg
Type: image/jpeg
Size: 54017 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070704/8f091fb4/attachment.jpg>
-------------- next part --------------



On 2007/07/04, at 16:58, Andr? Cruz wrote:

> Hello again.
>
> I'm having problems with POST requests. They reach varnish but
> nothing happens and the backend is not contacted.
>
> I see this in the log:
>     11 SessionOpen  c 10.134.145.2 51721
>
> ...
>     11 Debug        c "Pipe Shut read(read)"
>     11 Debug        c "Pipe Shut write(read)"
>
> I found this ticket "http://varnish.projects.linpro.no/ticket/47"
> which seemed to refer to my problem but even switching to "pipe" does
> not solve it.
>
> Is this a known problem?
>
> Thanks,
> Andr? Cruz
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc


From phk at phk.freebsd.dk  Wed Jul  4 16:30:20 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 04 Jul 2007 16:30:20 +0000
Subject: Fetch called after pass 
In-Reply-To: Your message of "Wed, 04 Jul 2007 15:39:46 +0100."
	<D5515C23-24A5-40B8-B516-7DCA575EF39A@segula.pt> 
Message-ID: <2344.1183566620@critter.freebsd.dk>


Hi Andre,

The state diagram has morphed a bit lately, DES is busy updating the
documentation and hopefully this will be covered RSN.

In the meantime, here is a snapshot of the current -trunks stateenginge:

	http://phk.freebsd.dk/misc/_.pdf

This is machine generated, see comment at the top of cache_center.c

We collapsed the pass and fetch code, after all, it does the same
basic thing, the distinction is only what happens to the object
afterwards: is it cached or thrown away.

That way we eliminated a lot of almost, but not entirely identical
code and got rid a number of problems in the pass version of the
code, that were already fixed in the fetch version.

So short version: yes, pass goes on to fetch (and now from there
to deliver), but the object is not cached.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From jean-marc.pouchoulon at ac-montpellier.fr  Wed Jul  4 16:31:45 2007
From: jean-marc.pouchoulon at ac-montpellier.fr (jean-marc pouchoulon)
Date: Wed, 04 Jul 2007 18:31:45 +0200
Subject: create worker thread failed 12 Cannot allocate memory
Message-ID: <468BCB71.2040205@ac-montpellier.fr>

Helo,

varnish was stopped with this error:

0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1183565922
    0 Debug          "Create worker thread failed 12 Cannot allocate memory"
    0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1183565925
    0 Debug          "Create worker thread failed 12 Cannot allocate memory"
    0 Debug          "Create worker thread failed 12 Cannot allocate memory"
    0 Debug          "Create worker thread failed 12 Cannot allocate memory"
    0 Debug          "Create worker thread failed 12 Cannot allocate memory"
    0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1183565928

any suggestions ?



From andre.cruz at segula.pt  Wed Jul  4 16:39:21 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Wed, 4 Jul 2007 17:39:21 +0100
Subject: Fetch called after pass 
In-Reply-To: <2344.1183566620@critter.freebsd.dk>
References: <2344.1183566620@critter.freebsd.dk>
Message-ID: <CE9C9DDB-B243-4572-BD4B-0EF419D358B2@segula.pt>

Hello Poul.

Thanks for the quick answer.

On 2007/07/04, at 17:30, Poul-Henning Kamp wrote:

>
> The state diagram has morphed a bit lately, DES is busy updating the
> documentation and hopefully this will be covered RSN.
>
> In the meantime, here is a snapshot of the current -trunks  
> stateenginge:
>
> 	http://phk.freebsd.dk/misc/_.pdf
>
> So short version: yes, pass goes on to fetch (and now from there
> to deliver), but the object is not cached.
>

 From what I can see, if vcl_recv() returns "pass" the request can  
still be cached by logic in vcl_fetch(), right?

Because that's what I think is happening in my case. The default  
vcl_fetch() handler is inserting the request in cache even though it  
got a "pass" earlier... Do we have to manual check the obj.pass flag  
in the handler?

Thanks,
Andr?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070704/6a44010c/attachment.html>

From phk at phk.freebsd.dk  Wed Jul  4 18:08:46 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 04 Jul 2007 18:08:46 +0000
Subject: Fetch called after pass 
In-Reply-To: Your message of "Wed, 04 Jul 2007 17:39:21 +0100."
	<CE9C9DDB-B243-4572-BD4B-0EF419D358B2@segula.pt> 
Message-ID: <2732.1183572526@critter.freebsd.dk>

In message <CE9C9DDB-B243-4572-BD4B-0EF419D358B2 at segula.pt>, =?ISO-8859-1?Q?And
r=E9_Cruz?= writes:

> From what I can see, if vcl_recv() returns "pass" the request can  
>still be cached by logic in vcl_fetch(), right?

No, this is one of the things that figure does not show: the object
created by vcl_pass() is not in the hash table at all, so no matter
what vcl_fetch() does, it will not be cached.

>Because that's what I think is happening in my case. The default  
>vcl_fetch() handler is inserting the request in cache even though it  
>got a "pass" earlier... Do we have to manual check the obj.pass flag  
>in the handler?

That would be a bug.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From anup at iamcool.net  Thu Jul  5 06:40:41 2007
From: anup at iamcool.net (Anup Shukla)
Date: Thu, 05 Jul 2007 12:10:41 +0530
Subject: Problems with revision 1643.
Message-ID: <468C9269.6030807@iamcool.net>

Hi All,

A few things are causing trouble in r1463

Varnish always uses uid nobody and gid nogroup.

i did this:
#useradd varnish
#varnishd -u varnish -g varnish ....

But, it still complained about group: nogroup not being present.

I added a group nogroup, just to keep varnish happy.

Varnish then complained about my VCL configuration in vcl_fetch()

<snip>
   if (resp.http.Content-Type ~ "text/html") {
      pass;
   }
</snip>

Is this something wrong?
The version that i am currently using (1.0.4 20 May 2007) never gave an 
error about resp.http.Content-Type.

Thank you.

Regards
A.S


From stonorn at giraffen.dk  Thu Jul  5 06:44:34 2007
From: stonorn at giraffen.dk (Anton Stonor)
Date: Thu, 05 Jul 2007 08:44:34 +0200
Subject: Varnish Dirty Caching
In-Reply-To: <31256.1183457161@critter.freebsd.dk>
References: <31256.1183457161@critter.freebsd.dk>
Message-ID: <468C9352.90106@giraffen.dk>

Poul-Henning Kamp wrote:

> The critical question is how we define "backend is down" and how
> fast and efficient we can detect it.
> 
> Ideas for how to express it in VCL are very welcome.

Maybe naive:

# First, we setup decide how to "sniff" that a backend is down
#
# options_ping: Send a HTTP OPTIONS (Perlbal does that)
# timeout: 	If the backend does not answer within x seconds, it is
#          	probably down
# icp: 		Abuse the protocol. (Squid + Zope does that)
backend.down_protocol = options_ping | timeout | icp

# What is the timeout limit?
backend.timeout = 30

# How long time should the backend be marked as "down" before we try
# again?
backend.retry_after = 300


# And then just use it

if(backend.down)....



/Anton Stonor


From phk at phk.freebsd.dk  Thu Jul  5 10:01:52 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 05 Jul 2007 10:01:52 +0000
Subject: New VCL toys, please test!
Message-ID: <29653.1183629712@critter.freebsd.dk>


I have just committed some new VCL toys to -trunk, please help me test
these if you can do things like:

	sub vcl_miss {
		set bereq.http.HeyYou = "This" req.url " is Urgent!" ;
		remove bereq.http.user-agent ;
	}

which would add this HTTP header to the request to the backend:

	HeyYou: This / is Urgent!

and remove any User-Agent: headers.

There are now a total of four HTTP protocol messages that can be accessed
from VCL:

	req.*		- The original request from the client
	bereq.*		- The request we send to the backend 
	obj.*		- The response stored in the object
	resp.*		- The response we send to the client

Here is a machine-generated plot of the VCL flow in -trunk:

	http://phk.freebsd.dk/misc/_.pdf

The right hand side of the boxes show which HTTP objects you have
access to in each of the VCL methods.

Feedback please!


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Thu Jul  5 11:33:27 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 13:33:27 +0200
Subject: Post requests
In-Reply-To: <55305F06-58BD-4B01-9C5F-923B70B9895A@segula.pt>
	(=?utf-8?Q?=22Andr=C3=A9?= Cruz"'s
	message of "Wed\, 4 Jul 2007 16\:58\:55 +0100")
References: <55305F06-58BD-4B01-9C5F-923B70B9895A@segula.pt>
Message-ID: <87zm2bdr08.fsf@des.linpro.no>

Andr? Cruz <andre.cruz at segula.pt> writes:
> I'm having problems with POST requests. They reach varnish but nothing
> happens and the backend is not contacted.
>
> I see this in the log:
>     11 SessionOpen  c 10.134.145.2 51721
>
> ...
>     11 Debug        c "Pipe Shut read(read)"
>     11 Debug        c "Pipe Shut write(read)"
>
> I found this ticket "http://varnish.projects.linpro.no/ticket/47"
> which seemed to refer to my problem but even switching to "pipe" does
> not solve it.

"Pipe Shut" comes from the pipe code, so switching *to* pipe won't make
any difference because it already *is* pipe.

This is not an error message, BTW, just an indication that a pipe
session ended.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Jul  5 11:33:57 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 13:33:57 +0200
Subject: create worker thread failed 12 Cannot allocate memory
In-Reply-To: <468BCB71.2040205@ac-montpellier.fr> (jean-marc pouchoulon's
	message of "Wed\, 04 Jul 2007 18\:31\:45 +0200")
References: <468BCB71.2040205@ac-montpellier.fr>
Message-ID: <87veczdqze.fsf@des.linpro.no>

jean-marc pouchoulon <jean-marc.pouchoulon at ac-montpellier.fr> writes:
> varnish was stopped with this error:
>
> 0 CLI            Rd ping
>     0 CLI            Wr 0 200 PONG 1183565922
>     0 Debug          "Create worker thread failed 12 Cannot allocate memory"
>     0 CLI            Rd ping
>     0 CLI            Wr 0 200 PONG 1183565925
>     0 Debug          "Create worker thread failed 12 Cannot allocate memory"
>     0 Debug          "Create worker thread failed 12 Cannot allocate memory"
>     0 Debug          "Create worker thread failed 12 Cannot allocate memory"
>     0 Debug          "Create worker thread failed 12 Cannot allocate memory"
>     0 CLI            Rd ping
>     0 CLI            Wr 0 200 PONG 1183565928
>
> any suggestions ?

Add RAM or reduce maxthreads.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Jul  5 11:36:54 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 13:36:54 +0200
Subject: Problems with revision 1643.
In-Reply-To: <468C9269.6030807@iamcool.net> (Anup Shukla's message of "Thu\, 05
	Jul 2007 12\:10\:41 +0530")
References: <468C9269.6030807@iamcool.net>
Message-ID: <87r6nndquh.fsf@des.linpro.no>

Anup Shukla <anup at iamcool.net> writes:
> Varnish always uses uid nobody and gid nogroup.
>
> i did this:
> #useradd varnish
> #varnishd -u varnish -g varnish ....
>
> But, it still complained about group: nogroup not being present.

All parameters are initialized to their default values before the
command-line arguments are processed.  This is not normally an issue,
but lately we've introduced a handful of parameters that have side
effects.

> Varnish then complained about my VCL configuration in vcl_fetch()
>
> <snip>
>    if (resp.http.Content-Type ~ "text/html") {
>       pass;
>    }
> </snip>
>
> Is this something wrong?

s/resp/obj/

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Jul  5 11:43:39 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 13:43:39 +0200
Subject: child process restarts frequently
In-Reply-To: <468B7A9A.3040706@met.no> (=?utf-8?Q?=22H=C3=A5vard_Futs?=
	=?utf-8?Q?=C3=A6ter=22's?= message of "Wed\,
	04 Jul 2007 12\:46\:50 +0200")
References: <468B7A9A.3040706@met.no>
Message-ID: <87myybdqj8.fsf@des.linpro.no>

H?vard Futs?ter <havard.futseter at met.no> writes:
> This object is then requested and delivered from cache several times.
> Then we see the following in the log, at a time immediately after
> 14:59:27:
>
> 0 ExpPick        138012248
> 0 VCL_call       timeout
> 0 VCL_return     discard

vcl_timeout() is called for every object in cache 30 seconds before it
expires.  The default action is "discard".  The alternative, "prefetch",
has not been implemented yet.

> ...immediately following this( ca.14:59:33 ) we get:
>
> 0 ExpBan         138012248 was banned
> 0 CLI            Rd vcl.load boot /tmp/vcl.XXfdES7y
> 0 CLI            Wr 0 200 Loaded "/tmp/vcl.XXfdES7y" as "boot"
> 0 CLI            Rd vcl.use boot
> 0 CLI            Wr 0 200
> 0 CLI            Rd start
> 0 CLI            Wr 0 200

The child process crashed while or immediately after recycling the
discarded object.

> The weird thing is that it seems to happen at the exact same time
> every day. At 07:59 and 14:59 the child processs is restarted in the
> manner shown above. We have a lot of requests which expires at each
> whole hour, but I have no idea if this is related in any way.

Does your backend deliver objects which always expire at either 08:00 or
15:00?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From optilude at gmx.net  Thu Jul  5 12:03:38 2007
From: optilude at gmx.net (Martin Aspeli)
Date: Thu, 5 Jul 2007 12:03:38 +0000 (UTC)
Subject: Mac and regular expressions
Message-ID: <loom.20070705T135943-423@post.gmane.org>

Hi,

I think I'm being bitten by a bug related to Mac OS X (10.4) and regular
expressions.

I've build Varnish using MacPorts, and also tried to build manually. To get it
to build manually, I've had to apply this patch:

http://projects.linpro.no/pipermail/varnish-misc/2007-June/000485.html

The result is the same, though.

The default Varnish config works fine, but as soon as I use a regular expression
(e.g. if(req.url ~ "\.(css|js)$") { ...} or whatever) in a custom VCL file, I
get this error:

$ /usr/local/sbin/varnishd -a localhost:8080  -f /etc/varnish.conf 
Assert error in mgt_CallCc(), mgt_vcc.c line 214:
  Condition((dlclose(p)) == 0) not true.
  errno = 2 (No such file or directory)
Abort trap

That error is described in http://varnish.projects.linpro.no/ticket/117 and I
added a note there, but I think the underlying issue may be different.

Is there something more I can do to debug this problem? Has anyone else seen it?

Thanks,
Martin



From des at linpro.no  Thu Jul  5 12:16:38 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 14:16:38 +0200
Subject: Mac and regular expressions
In-Reply-To: <loom.20070705T135943-423@post.gmane.org> (Martin Aspeli's
	message of "Thu\, 5 Jul 2007 12\:03\:38 +0000 \(UTC\)")
References: <loom.20070705T135943-423@post.gmane.org>
Message-ID: <87abubdp09.fsf@des.linpro.no>

Martin Aspeli <optilude at gmx.net> writes:
> I think I'm being bitten by a bug related to Mac OS X (10.4) and regular
> expressions.

Looks like MacOS's libc lacks the POSIX regexp API.  Unfortunately,
until we manage to get hold of a MacOS box to test on, there isn't much
we can do to fix the build.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Jul  5 12:21:27 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 14:21:27 +0200
Subject: branches/1.1
Message-ID: <87644zdos8.fsf@des.linpro.no>

In preparation for the release of Varnish 1.1 on July 20, I've created
branches/1.1 and bumped the version numbers.  1.1 will consist of trunk
as of today + any bug fixes that crop up before the release + possibly
additional header / URL rewriting code from Poul-Henning, time
permitting.  Plugins for Munin, Nagios and Webmin will be released
separately.

Please test and report any bugs you find, be they in the code or in the
documentation.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From gaute at pht.no  Thu Jul  5 12:21:56 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Thu, 5 Jul 2007 14:21:56 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <200707041420.00179.gaute@pht.no>
References: <200707031514.33396.gaute@pht.no> <87ir90haoj.fsf@des.linpro.no>
	<200707041420.00179.gaute@pht.no>
Message-ID: <200707051421.56526.gaute@pht.no>

> Ah, OK, you won't be able to process log files larger than about 2 GB on
> a 32-bit machine.  I should probably figure out a way to get around
> that.

Ok, now I have set up /var/log/varnish/varnish.log to rotate every hour, and a 
postrotate action on that to pipe through varnishncsa and append 
to /var/log/varnish/varnish.access_log which in turn gets rotated and 
processed by awstats dayly.

That should take care of the 2G limit in varnishncsa at least. 
I was hoping it would help with the logging it self but apparently not..

I have about 20 logfiles by now, all under 100M, and have made various 
observations:

Everything went well during the night. The accesstimes are about a minute past 
the hour, and the last date string in the log corresponds with that.

Then at 09:13 suddenly it's all "Pipe Shut read(read)" and "Pipe Shut 
write(read)" again. The situation when I noticed this, about 12:30, this was 
in 'log.1' whith modtime 09:13 and 'log' was still 0 bytes.
I guess that just means warnishlog had stopped responding to SIGHUP.

Wil try to do a restart insted in postrotate to see if runningtime has 
anything to do with it.

That's the logging problem, now for the processing problem:

I made a small script to run varnish(log|ncsa) on every file and get the 
returnstatus. log works fine, and a tail of that diplays sensible output and 
timestamps, but ncsa segfaults on about half of them. 
No pattern discernible..

I tried to make a small python script to feed a log chunk by chunk into ncsa 
on stdin, but I was not able to make that work reliably enough to get any 
hint what precisely is causing the problem. Don't even know if ncsa can work 
like that..
Tried to look for the last output lines from nscalogs in the varnishlog as 
well, but nothing jumps out so far...

Starting to run out of things to try here...
Ideas?

Gaute



From optilude at gmx.net  Thu Jul  5 12:35:38 2007
From: optilude at gmx.net (Martin Aspeli)
Date: Thu, 5 Jul 2007 13:35:38 +0100
Subject: Mac and regular expressions
In-Reply-To: <87abubdp09.fsf@des.linpro.no>
References: <loom.20070705T135943-423@post.gmane.org>
	<87abubdp09.fsf@des.linpro.no>
Message-ID: <5af21ed10707050535w6659efchede18214a1b4bef6@mail.gmail.com>

Buy a Mac Mini? :)

I can help you test if you can give me instructions. My C skills are
perhaps a bit rusty, but I can definitely make it build and try out
things.

Just from a quick Google,
http://developer.apple.com/documentation/Darwin/Reference/Manpages/man3/regex.3.html
seems to suggest that it does have POSIX regex.

Martin

On 7/5/07, Dag-Erling Sm?rgrav <des at linpro.no> wrote:
> Martin Aspeli <optilude at gmx.net> writes:
> > I think I'm being bitten by a bug related to Mac OS X (10.4) and regular
> > expressions.
>
> Looks like MacOS's libc lacks the POSIX regexp API.  Unfortunately,
> until we manage to get hold of a MacOS box to test on, there isn't much
> we can do to fix the build.
>
> DES
> --
> Dag-Erling Sm?rgrav
> Senior Software Developer
> Linpro AS - www.linpro.no
>
>


From des at linpro.no  Thu Jul  5 13:32:27 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 15:32:27 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <200707051421.56526.gaute@pht.no> (Gaute Amundsen's message of
	"Thu\, 5 Jul 2007 14\:21\:56 +0200")
References: <200707031514.33396.gaute@pht.no> <87ir90haoj.fsf@des.linpro.no>
	<200707041420.00179.gaute@pht.no> <200707051421.56526.gaute@pht.no>
Message-ID: <871wfndlhw.fsf@des.linpro.no>

Gaute Amundsen <gaute at pht.no> writes:
> I made a small script to run varnish(log|ncsa) on every file and get
> the returnstatus. log works fine, and a tail of that diplays sensible
> output and timestamps, but ncsa segfaults on about half of them.  No
> pattern discernible..

Can you send me one of those?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Jul  5 13:34:42 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 15:34:42 +0200
Subject: Mac and regular expressions
In-Reply-To: <5af21ed10707050535w6659efchede18214a1b4bef6@mail.gmail.com>
	(Martin Aspeli's message of "Thu\, 5 Jul 2007 13\:35\:38 +0100")
References: <loom.20070705T135943-423@post.gmane.org>
	<87abubdp09.fsf@des.linpro.no>
	<5af21ed10707050535w6659efchede18214a1b4bef6@mail.gmail.com>
Message-ID: <87wsxfc6tp.fsf@des.linpro.no>

"Martin Aspeli" <optilude at gmx.net> writes:
> Buy a Mac Mini? :)

Sure, what's your credit card number again?  :)

> I can help you test if you can give me instructions.

I don't want code in the repo that I can't test myself, or at least
build.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From optilude at gmx.net  Thu Jul  5 13:47:39 2007
From: optilude at gmx.net (Martin Aspeli)
Date: Thu, 5 Jul 2007 14:47:39 +0100
Subject: Mac and regular expressions
In-Reply-To: <87wsxfc6tp.fsf@des.linpro.no>
References: <loom.20070705T135943-423@post.gmane.org>
	<87abubdp09.fsf@des.linpro.no>
	<5af21ed10707050535w6659efchede18214a1b4bef6@mail.gmail.com>
	<87wsxfc6tp.fsf@des.linpro.no>
Message-ID: <5af21ed10707050647q48fece72md818443ab9ac5216@mail.gmail.com>

Hi Dag

On 7/5/07, Dag-Erling Sm?rgrav <des at linpro.no> wrote:
> "Martin Aspeli" <optilude at gmx.net> writes:
> > Buy a Mac Mini? :)
>
> Sure, what's your credit card number again?  :)

Find a customer with one, that's what I normally do. :)

> > I can help you test if you can give me instructions.
>
> I don't want code in the repo that I can't test myself, or at least
> build.

Neither do I, but we're basing this on an assumption (which may or may
not be correct) about the OS X regex implementation. If you can
identify any points in the code where I could apply some hypothetical
patch/change or add debug info that may at least help prove or
disprove that theory. It may well be that this is an issue that
affects other platforms as well.

Martin


From des at linpro.no  Thu Jul  5 14:02:54 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 16:02:54 +0200
Subject: Mac and regular expressions
In-Reply-To: <5af21ed10707050647q48fece72md818443ab9ac5216@mail.gmail.com>
	(Martin Aspeli's message of "Thu\, 5 Jul 2007 14\:47\:39 +0100")
References: <loom.20070705T135943-423@post.gmane.org>
	<87abubdp09.fsf@des.linpro.no>
	<5af21ed10707050535w6659efchede18214a1b4bef6@mail.gmail.com>
	<87wsxfc6tp.fsf@des.linpro.no>
	<5af21ed10707050647q48fece72md818443ab9ac5216@mail.gmail.com>
Message-ID: <87sl83c5ip.fsf@des.linpro.no>

"Martin Aspeli" <optilude at gmx.net> writes:
> Dag-Erling Sm?rgrav <des at linpro.no> writes:
> > I don't want code in the repo that I can't test myself, or at least
> > build.
> Neither do I, but we're basing this on an assumption (which may or may
> not be correct) about the OS X regex implementation. If you can
> identify any points in the code where I could apply some hypothetical
> patch/change or add debug info that may at least help prove or
> disprove that theory. It may well be that this is an issue that
> affects other platforms as well.

There are other issues as well.  Varnish likely runs the compiler with
the wrong flags (why on earth couldn't Apple keep gcc's command-line
syntax when they adopted it as their system compiler?), and for some
reason configure doesn't correctly detect the absence of strndup() and
enable the compat implementation.  Not to mention the clock_gettime()
issue, which will surely come back to bite us at some point...

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From optilude at gmx.net  Thu Jul  5 14:09:58 2007
From: optilude at gmx.net (Martin Aspeli)
Date: Thu, 5 Jul 2007 15:09:58 +0100
Subject: Mac and regular expressions
In-Reply-To: <87sl83c5ip.fsf@des.linpro.no>
References: <loom.20070705T135943-423@post.gmane.org>
	<87abubdp09.fsf@des.linpro.no>
	<5af21ed10707050535w6659efchede18214a1b4bef6@mail.gmail.com>
	<87wsxfc6tp.fsf@des.linpro.no>
	<5af21ed10707050647q48fece72md818443ab9ac5216@mail.gmail.com>
	<87sl83c5ip.fsf@des.linpro.no>
Message-ID: <5af21ed10707050709k6c3a71d6rbd679cd11513c933@mail.gmail.com>

> > Neither do I, but we're basing this on an assumption (which may or may
> > not be correct) about the OS X regex implementation. If you can
> > identify any points in the code where I could apply some hypothetical
> > patch/change or add debug info that may at least help prove or
> > disprove that theory. It may well be that this is an issue that
> > affects other platforms as well.
>
> There are other issues as well.  Varnish likely runs the compiler with
> the wrong flags (why on earth couldn't Apple keep gcc's command-line
> syntax when they adopted it as their system compiler?), and for some
> reason configure doesn't correctly detect the absence of strndup() and
> enable the compat implementation.  Not to mention the clock_gettime()
> issue, which will surely come back to bite us at some point...

That's a shame. So what we're saying is that Varnish at this point
doesn't really support Mac OS X, and won't unless someone donates a
Mac to the core developers?

Perhaps we could start some kind of pledge? The Plone community
basically bought the developer of poEdit a Mac via fundable.org so
that he'd port poEdit to OS X. There's a lot of interest in Varnish
from the Plone community (since Squid scares me senseless), and a lot
of Plone developers use Macs, so we may be able to achieve something
similar if there's a commitment from the core Varnish developers to
support OS X in return for a Mac.

Martin


From des at linpro.no  Thu Jul  5 14:17:58 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 16:17:58 +0200
Subject: Mac and regular expressions
In-Reply-To: <5af21ed10707050709k6c3a71d6rbd679cd11513c933@mail.gmail.com>
	(Martin Aspeli's message of "Thu\, 5 Jul 2007 15\:09\:58 +0100")
References: <loom.20070705T135943-423@post.gmane.org>
	<87abubdp09.fsf@des.linpro.no>
	<5af21ed10707050535w6659efchede18214a1b4bef6@mail.gmail.com>
	<87wsxfc6tp.fsf@des.linpro.no>
	<5af21ed10707050647q48fece72md818443ab9ac5216@mail.gmail.com>
	<87sl83c5ip.fsf@des.linpro.no>
	<5af21ed10707050709k6c3a71d6rbd679cd11513c933@mail.gmail.com>
Message-ID: <87odirc4tl.fsf@des.linpro.no>

"Martin Aspeli" <optilude at gmx.net> writes:
> That's a shame. So what we're saying is that Varnish at this point
> doesn't really support Mac OS X

correct

> and won't unless someone donates a Mac to the core developers?

I'd rather say "until the core developers get their hands on a Mac",
which may happen in a variety of ways.  Linpro is an Apple partner, so
we may be able to get a good price.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Jul  5 14:20:46 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 16:20:46 +0200
Subject: Mac and regular expressions
In-Reply-To: <87odirc4tl.fsf@des.linpro.no> ("Dag-Erling =?utf-8?Q?Sm?=
	=?utf-8?Q?=C3=B8rgrav=22's?= message
	of "Thu\, 05 Jul 2007 16\:17\:58 +0200")
References: <loom.20070705T135943-423@post.gmane.org>
	<87abubdp09.fsf@des.linpro.no>
	<5af21ed10707050535w6659efchede18214a1b4bef6@mail.gmail.com>
	<87wsxfc6tp.fsf@des.linpro.no>
	<5af21ed10707050647q48fece72md818443ab9ac5216@mail.gmail.com>
	<87sl83c5ip.fsf@des.linpro.no>
	<5af21ed10707050709k6c3a71d6rbd679cd11513c933@mail.gmail.com>
	<87odirc4tl.fsf@des.linpro.no>
Message-ID: <87k5tfc4ox.fsf@des.linpro.no>

Dag-Erling Sm?rgrav <des at linpro.no> writes:
> I'd rather say "until the core developers get their hands on a Mac",
> which may happen in a variety of ways.  Linpro is an Apple partner, so
> we may be able to get a good price.

(of course, I'd prefer an XServe or some other 64-bit model, but they
cost real money...)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Jul  5 16:13:35 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 05 Jul 2007 18:13:35 +0200
Subject: Away
Message-ID: <87wsxebzgw.fsf@des.linpro.no>

I'll be away tomorrow and all of next week, with only sporadic Internet
connectivity.  I trust Poul-Henning will take good care of you all while
I'm gone :)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From andre.cruz at segula.pt  Thu Jul  5 17:39:26 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Thu, 5 Jul 2007 18:39:26 +0100
Subject: Regarding url purging
Message-ID: <AEA1E82F-7632-4A03-846A-D488EE6DA602@segula.pt>

Hello.

I'm caching pages from different vhosts, say x.com and y.com.

If I want to purge all the data from one of the vhosts how do I do  
it? "url.purge x.com.*" does not work. I'm a bit confused with the  
syntax since the wiki only talks about paths and not complete URLs.

Thanks for the help,
Andr?




From gaute at pht.no  Thu Jul  5 18:13:24 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Thu, 5 Jul 2007 20:13:24 +0200
Subject: Regarding url purging
In-Reply-To: <AEA1E82F-7632-4A03-846A-D488EE6DA602@segula.pt>
References: <AEA1E82F-7632-4A03-846A-D488EE6DA602@segula.pt>
Message-ID: <200707052013.24760.gaute@pht.no>

On Thursday 05 July 2007 19:39, Andr? Cruz wrote:
> Hello.
>
> I'm caching pages from different vhosts, say x.com and y.com.
>
> If I want to purge all the data from one of the vhosts how do I do
> it? "url.purge x.com.*" does not work. I'm a bit confused with the
> syntax since the wiki only talks about paths and not complete URLs.

That's not easily done, as far as I have been able to determine.
I've submitted a feature request for it..

As I understand it, you can only use hostname as part of the url when you 
purge by http, but then no wildcards.
You can use full regexps when purging on the console, but then only the path 
is available to match against.

I guess a workaround would be parse the log to make a list of all urls served 
within the expiery limit, do your filtering on that list, and then purge all 
the matching urls one by one by http.

The nsca log contains the full url including host, which i think is somewhat 
unusual, so grepping for that should not be too hard.

Here is the pipe I use to get host as a separate column, that should be a 
start.

/usr/bin/varnishncsa -c -r /var/log/varnish/varnish.log | \
sed -e 's#\("\([A-Z]\+\) \(http://\([^/]*\)\)\(.*\)\)#\4 "\2 \5#g' \


Gaute


From varnish at held-im-ruhestand.de  Thu Jul  5 18:15:18 2007
From: varnish at held-im-ruhestand.de (Christoph)
Date: Thu, 5 Jul 2007 20:15:18 +0200
Subject: Q: multiple backends
Message-ID: <20070705181517.GA5791@falcon>

Hi,

are there any plans to include support for multiple backends in varnish?

It would be great to have either fault-tolerance and/or simple
load-balancing. 

It could be integrated to the proposed backend.down extension. 

Greetings
   Christoph


From andre.cruz at segula.pt  Thu Jul  5 18:30:23 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Thu, 5 Jul 2007 19:30:23 +0100
Subject: Regarding url purging
In-Reply-To: <200707052013.24760.gaute@pht.no>
References: <AEA1E82F-7632-4A03-846A-D488EE6DA602@segula.pt>
	<200707052013.24760.gaute@pht.no>
Message-ID: <8E38CEFD-06A1-465D-ADFC-D883AB3E8794@segula.pt>


On 2007/07/05, at 19:13, Gaute Amundsen wrote:

> That's not easily done, as far as I have been able to determine.
> I've submitted a feature request for it..
>

Where is it so that I can vote for it? :) Do you know if it's  
targeted for 1.1?

> As I understand it, you can only use hostname as part of the url  
> when you
> purge by http, but then no wildcards.
> You can use full regexps when purging on the console, but then only  
> the path
> is available to match against.
>

Eeck.

> I guess a workaround would be parse the log to make a list of all  
> urls served
> within the expiery limit, do your filtering on that list, and then  
> purge all
> the matching urls one by one by http.
>

Thanks for your help but I think I'll wait either for regexp on http  
purging or full urls on console purging, which seem like basic  
functions to have anyway.

Regards,
Andr? Cruz

From gaute at pht.no  Thu Jul  5 18:49:54 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Thu, 5 Jul 2007 20:49:54 +0200
Subject: Regarding url purging
In-Reply-To: <8E38CEFD-06A1-465D-ADFC-D883AB3E8794@segula.pt>
References: <AEA1E82F-7632-4A03-846A-D488EE6DA602@segula.pt>
	<200707052013.24760.gaute@pht.no>
	<8E38CEFD-06A1-465D-ADFC-D883AB3E8794@segula.pt>
Message-ID: <200707052049.56184.gaute@pht.no>

On Thursday 05 July 2007 20:30, Andr? Cruz wrote:
> On 2007/07/05, at 19:13, Gaute Amundsen wrote:
> > That's not easily done, as far as I have been able to determine.
> > I've submitted a feature request for it..
>
> Where is it so that I can vote for it? :) Do you know if it's
> targeted for 1.1?

http://varnish.projects.linpro.no/ticket/116
1.1? No idea. Don't think there is any voting system either..
Except perhaps bribing DES with a Mac ;-)

> > As I understand it, you can only use hostname as part of the url
> > when you
> > purge by http, but then no wildcards.
> > You can use full regexps when purging on the console, but then only
> > the path
> > is available to match against.
>
> Eeck.
>
> > I guess a workaround would be parse the log to make a list of all
> > urls served
> > within the expiery limit, do your filtering on that list, and then
> > purge all
> > the matching urls one by one by http.
>
> Thanks for your help but I think I'll wait either for regexp on http
> purging or full urls on console purging, which seem like basic
> functions to have anyway.

Hm, well, varnish is a pretty young project still you know :)
I think the majority of users so far are "one site shops", or can reliably 
predict what wil have to be purged when something is updated..

Myself I'd rather have better url rewriting to be honest,
http://varnish.projects.linpro.no/ticket/77
then I could phase out apache from our current varnish->apache->zope setup.

Gaute


From gaute at pht.no  Thu Jul  5 19:50:51 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Thu, 5 Jul 2007 21:50:51 +0200
Subject: logging problems - "Pipe Shut"
In-Reply-To: <200707052020.32374.gaute@pht.no>
References: <200707031514.33396.gaute@pht.no> <874pkidgkv.fsf@des.linpro.no>
	<200707052020.32374.gaute@pht.no>
Message-ID: <200707052150.52018.gaute@pht.no>

That did the trick for the processing :-D

Only the logging itself left then.
With som luck logger restarts on logrotate every hour will keep that stable 
til 1.1 is out :-)

Gaute

On Thursday 05 July 2007 20:20, Gaute Amundsen wrote:
> On Thursday 05 July 2007 17:18, Dag-Erling Sm?rgrav wrote:
> > Dag-Erling Sm?rgrav <des at linpro.no> writes:
> > > Gaute Amundsen <gaute at pht.no> writes:
> > > > log.1 works, and log.2 segfaults
> > >
> > > They both work fine here, with varnishncsa from trunk (although on a
> > > 64-bit machine)
> >
> > I just realized - maybe you ran across the bug that was fixed in r1531.
>
> You mean this one?
> Wil try that then.
>
> Gaute
>
> Index: trunk/varnish-cache/bin/varnishncsa/varnishncsa.c
> ===================================================================
> --- trunk/varnish-cache/bin/varnishncsa/varnishncsa.c (revision 1525)
> +++ trunk/varnish-cache/bin/varnishncsa/varnishncsa.c (revision 1531)
> @@ -156,5 +156,5 @@
>
>  	/* trim trailing space */
> -	while (str[len - 1] == ' ')
> +	while (len && str[len - 1] == ' ')
>  		--len;

-- 
Programmerer - Pixelhospitalet AS
T?rkoppveien 10, 1570 Dilling
Tlf. 24 12 97 81 - 9074 7344


From anup at iamcool.net  Fri Jul  6 04:18:54 2007
From: anup at iamcool.net (Anup Shukla)
Date: Fri, 06 Jul 2007 09:48:54 +0530
Subject: Varnish shutting down under heavy load
Message-ID: <468DC2AE.3050006@iamcool.net>

Hi All,

This is in continuation of the previous thread "Best Practices - 
Suggestion Request"

The problem of varnish shutting down every 7-10 days still continues.
This resulted in significant amount of downtime, and hence i shifted all 
but one server to "pound".
pound has been working fine.. but the one server that is running varnish 
still kept crashing.

This is what i found in the logs
......
......
......
  536 TxHeader     c X-Varnish: 1278080177
  536 TxHeader     c Age: 4
  536 TxHeader     c Via: 1.1 varnish
  536 ReqEnd       c 1278080177 1183663532.742941000 
1183663536.720536000 0.867369000 3.489705000 0.487890000
    0 StatAddr       66.249.67.59 0 2220 340 4388 0 0 4373 1729613 349338293
    0 WorkThread     0x2ccb851c60 start
    0 CLI            Rd vcl.load boot /tmp/vcl.XXBL9f1R
    0 Error          CLI write failed (errno=32)
[Log Ends Here]

Not that i never saw the logs, but i never make varnish save the log to 
a file.
Yesterday night, i finally decided to let varnish log everything to a 
file, and this is what i saw this morning.

Do i need to change something?

I am using the rpms provided for 1.0.4 on Centos 4.

/etc/sysconfig/varnish
---------------------------------------------------------------------
NFILES=131072
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_ADDRESS=XXX..XXX.XXX.XXX
VARNISH_LISTEN_PORT=80
# VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
# VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_MIN_THREADS=1
VARNISH_MAX_THREADS=1000
VARNISH_THREAD_TIMEOUT=120
VARNISH_STORAGE_FILE=/cache/varnish/varnish_storage.bin
VARNISH_STORAGE_SIZE=8G
VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
VARNISH_TTL=120

DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -t ${VARNISH_TTL} \
             -w 
${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
             -s ${VARNISH_STORAGE}"


/etc/varnish/default.vcl
---------------------------------------------------------------------
backend default {
        set backend.host = "127.0.0.1";
        set backend.port = "8000";
}

sub vcl_recv {
        if (req.request == "GET" && req.http.cookie) {
                lookup;
        }
}

sub vcl_fetch {
        if (!obj.valid) {
                error;
        }
        if (!obj.cacheable) {
                pass;
        }
        if (resp.http.Content-Type ~ "text/html") {
                pass;
        }
        if (resp.http.Set-Cookie) {
                insert;
        }
        insert;
}


Regards
A.S




From havard.futseter at met.no  Fri Jul  6 07:44:25 2007
From: havard.futseter at met.no (=?UTF-8?B?SMOldmFyZCBGdXRzw6Z0ZXI=?=)
Date: Fri, 06 Jul 2007 09:44:25 +0200
Subject: child process restarts frequently
In-Reply-To: <87myybdqj8.fsf@des.linpro.no>
References: <468B7A9A.3040706@met.no> <87myybdqj8.fsf@des.linpro.no>
Message-ID: <468DF2D9.3090607@met.no>

Thanks for the quick reply!

Dag-Erling Sm?rgrav wrote:
> H?vard Futs?ter <havard.futseter at met.no> writes:
>> This object is then requested and delivered from cache several times.
>> Then we see the following in the log, at a time immediately after
>> 14:59:27:
>>
>> 0 ExpPick        138012248
>> 0 VCL_call       timeout
>> 0 VCL_return     discard

>> ...immediately following this( ca.14:59:33 ) we get:
>>
>> 0 ExpBan         138012248 was banned
>> 0 CLI            Rd vcl.load boot /tmp/vcl.XXfdES7y
>> 0 CLI            Wr 0 200 Loaded "/tmp/vcl.XXfdES7y" as "boot"
>> 0 CLI            Rd vcl.use boot
>> 0 CLI            Wr 0 200
>> 0 CLI            Rd start
>> 0 CLI            Wr 0 200
> 
> The child process crashed while or immediately after recycling the
> discarded object.

>> The weird thing is that it seems to happen at the exact same time
>> every day. At 07:59 and 14:59 the child processs is restarted in the
>> manner shown above. We have a lot of requests which expires at each
>> whole hour, but I have no idea if this is related in any way.
> 
> Does your backend deliver objects which always expire at either 08:00 or
> 15:00?

Well, yes. The backend actually delivers objects that expire every hour
sharp. So, a bunch of objects( about 2000 - 2500) will be discarded
simultaneously, 30 seconds before each whole hour.

We have also found an interesting thing in the logs:

Immediately after object 138012248 was banned and the child was 
restarted, we see that a request was made for object 138012248 ( or 
rather, a request for the same url that this object represents).

Might this perhaps be an indication that Varnish gets into trouble when 
it receives a request for a resource while the cached object for this 
resource is beeing recycled?


-- 
Mvh,
H?vard Futs?ter
Telefon: +47 22 96 32 69
Mobil:   +47 99 69 61 84



From andre.cruz at segula.pt  Fri Jul  6 09:14:30 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Fri, 6 Jul 2007 10:14:30 +0100
Subject: Post requests
In-Reply-To: <87zm2bdr08.fsf@des.linpro.no>
References: <55305F06-58BD-4B01-9C5F-923B70B9895A@segula.pt>
	<87zm2bdr08.fsf@des.linpro.no>
Message-ID: <A3AE30CC-AADB-4C85-B430-8DCED8990E88@segula.pt>

I'm still having problems with POST requests. This time I recorded  
more information to try and make sense of this.

Version: 1.1 branch r1656
cmd line:  ./varnishd -a :8080 -b 127.0.0.1:80 -d -d -n /tmp/pretty
VCL: the default is used

The browser request as seen through wireshark:

POST /create.bml HTTP/1.1
Host: andre.localhost:8080
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv: 
1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
Accept: text/xml,application/xml,application/xhtml+xml,text/ 
html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://andre.localhost:8080/create.bml
Cookie: ljuniq=btZhgKFp2s9UaAi:1183650844; ljuniq=btZhgKFp2s9UaAi: 
1183650844
Content-Type: application/x-www-form-urlencoded
Content-Length: 164

mode=submit&code=&ssl=&email=&password1=&user=&answer=&captcha_chal=c0%3 
A1183708800%3A3053%3A900%3A7IS6nKjgirIie2gTdXVz% 
3Ae51274fe8457d7b25ef7c1f6e68fbbae&x=61&y=19


Varnish log content:

     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712178
     0 WorkThread     0x8679a134 start
    11 SessionOpen  c 10.134.145.2 49924
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712181
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712184
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712187
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712190
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712193
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712196
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712199
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712202
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712205
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712208
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712211
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712214
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712217
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712220
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712223
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712226
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712229
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712232
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712235
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712238
    11 SessionClose c pipe
    11 ReqStart     c 10.134.145.2 49924 2113606171
    11 RxRequest    c POST
    11 RxURL        c /create.bml
    11 RxProtocol   c HTTP/1.1
    11 RxHeader     c Host: andre.localhost:8080
    11 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; U; Intel  
Mac OS X; en-US; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
    11 RxHeader     c Accept: text/xml,application/xml,application/ 
xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    11 RxHeader     c Accept-Language: en-us,en;q=0.5
    11 RxHeader     c Accept-Encoding: gzip,deflate
    11 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    11 RxHeader     c Keep-Alive: 300
    11 RxHeader     c Connection: keep-alive
    11 RxHeader     c Referer: http://andre.localhost:8080/create.bml
    11 RxHeader     c Cookie: ljuniq=btZhgKFp2s9UaAi:1183650844;  
ljuniq=btZhgKFp2s9UaAi:1183650844
    11 RxHeader     c Content-Type: application/x-www-form-urlencoded
    11 RxHeader     c Content-Length: 164
    11 VCL_call     c recv
    11 VCL_return   c pipe
    11 VCL_call     c pipe
    11 VCL_return   c pipe
    14 BackendOpen  b default 127.0.0.1 38662 127.0.0.1 80
    14 BackendXID   b 2113606171
    11 Backend      c 14 default
    14 TxRequest    b GET
    14 TxURL        b /create.bml
    14 TxProtocol   b HTTP/1.1
    14 TxHeader     b Host: andre.localhost:8080
    14 TxHeader     b User-Agent: Mozilla/5.0 (Macintosh; U; Intel  
Mac OS X; en-US; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
    14 TxHeader     b Accept: text/xml,application/xml,application/ 
xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    14 TxHeader     b Accept-Language: en-us,en;q=0.5
    14 TxHeader     b Accept-Encoding: gzip,deflate
    14 TxHeader     b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    14 TxHeader     b Connection: keep-alive
    14 TxHeader     b Referer: http://andre.localhost:8080/create.bml
    14 TxHeader     b Cookie: ljuniq=btZhgKFp2s9UaAi:1183650844;  
ljuniq=btZhgKFp2s9UaAi:1183650844
    14 TxHeader     b Content-Type: application/x-www-form-urlencoded
    14 TxHeader     b Content-Length: 164
    14 TxHeader     b X-Varnish: 2113606171
    14 TxHeader     b X-Forwarded-for: 10.134.145.2
    14 BackendClose b default
    11 ReqEnd       c 2113606171 1183712178.629895424  
1183712238.683765133 0.000704753 0.000207682 60.053662027
     0 StatAddr       10.134.145.2 0 60 1 1 1 0 0 0 0
    11 StatSess     c 10.134.145.2 49924 60 1 1 1 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712241
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1183712244



I don't know what to make of this... After some time waiting for  
something Varnish processes a POST form the browser but sends a GET  
to the backend? Nothing reaches the browser btw. Strange stuff..  
Maybe someone can shed some light on this...

Thanks for the help!
Andr?


On 2007/07/05, at 12:33, Dag-Erling Sm?rgrav wrote:

> Andr? Cruz <andre.cruz at segula.pt> writes:
>> I'm having problems with POST requests. They reach varnish but  
>> nothing
>> happens and the backend is not contacted.
>>
>> I see this in the log:
>>     11 SessionOpen  c 10.134.145.2 51721
>>
>> ...
>>     11 Debug        c "Pipe Shut read(read)"
>>     11 Debug        c "Pipe Shut write(read)"
>>
>> I found this ticket "http://varnish.projects.linpro.no/ticket/47"
>> which seemed to refer to my problem but even switching to "pipe" does
>> not solve it.
>
> "Pipe Shut" comes from the pipe code, so switching *to* pipe won't  
> make
> any difference because it already *is* pipe.
>
> This is not an error message, BTW, just an indication that a pipe
> session ended.
>
> DES
> -- 
> Dag-Erling Sm?rgrav
> Senior Software Developer
> Linpro AS - www.linpro.no



From andre.cruz at segula.pt  Fri Jul  6 09:37:00 2007
From: andre.cruz at segula.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Fri, 6 Jul 2007 10:37:00 +0100
Subject: Post requests
In-Reply-To: <A3AE30CC-AADB-4C85-B430-8DCED8990E88@segula.pt>
References: <55305F06-58BD-4B01-9C5F-923B70B9895A@segula.pt>
	<87zm2bdr08.fsf@des.linpro.no>
	<A3AE30CC-AADB-4C85-B430-8DCED8990E88@segula.pt>
Message-ID: <463946C2-F656-4918-8FBD-B4442D83AB46@segula.pt>

FYI.

I just tried 1.0.4 and it worked... So it's something new.

Regards,
Andr?

On 2007/07/06, at 10:14, Andr? Cruz wrote:

> I'm still having problems with POST requests. This time I recorded
> more information to try and make sense of this.
>
> Version: 1.1 branch r1656
> cmd line:  ./varnishd -a :8080 -b 127.0.0.1:80 -d -d -n /tmp/pretty
> VCL: the default is used
>
> The browser request as seen through wireshark:
>
> POST /create.bml HTTP/1.1
> Host: andre.localhost:8080
> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:
> 1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
> Accept: text/xml,application/xml,application/xhtml+xml,text/
> html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
> Accept-Language: en-us,en;q=0.5
> Accept-Encoding: gzip,deflate
> Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
> Keep-Alive: 300
> Connection: keep-alive
> Referer: http://andre.localhost:8080/create.bml
> Cookie: ljuniq=btZhgKFp2s9UaAi:1183650844; ljuniq=btZhgKFp2s9UaAi:
> 1183650844
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 164
>
> mode=submit&code=&ssl=&email=&password1=&user=&answer=&captcha_chal=c0 
> %3
> A1183708800%3A3053%3A900%3A7IS6nKjgirIie2gTdXVz%
> 3Ae51274fe8457d7b25ef7c1f6e68fbbae&x=61&y=19
>
>
> Varnish log content:
>
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712178
>      0 WorkThread     0x8679a134 start
>     11 SessionOpen  c 10.134.145.2 49924
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712181
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712184
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712187
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712190
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712193
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712196
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712199
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712202
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712205
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712208
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712211
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712214
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712217
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712220
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712223
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712226
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712229
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712232
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712235
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712238
>     11 SessionClose c pipe
>     11 ReqStart     c 10.134.145.2 49924 2113606171
>     11 RxRequest    c POST
>     11 RxURL        c /create.bml
>     11 RxProtocol   c HTTP/1.1
>     11 RxHeader     c Host: andre.localhost:8080
>     11 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; U; Intel
> Mac OS X; en-US; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
>     11 RxHeader     c Accept: text/xml,application/xml,application/
> xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
>     11 RxHeader     c Accept-Language: en-us,en;q=0.5
>     11 RxHeader     c Accept-Encoding: gzip,deflate
>     11 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>     11 RxHeader     c Keep-Alive: 300
>     11 RxHeader     c Connection: keep-alive
>     11 RxHeader     c Referer: http://andre.localhost:8080/create.bml
>     11 RxHeader     c Cookie: ljuniq=btZhgKFp2s9UaAi:1183650844;
> ljuniq=btZhgKFp2s9UaAi:1183650844
>     11 RxHeader     c Content-Type: application/x-www-form-urlencoded
>     11 RxHeader     c Content-Length: 164
>     11 VCL_call     c recv
>     11 VCL_return   c pipe
>     11 VCL_call     c pipe
>     11 VCL_return   c pipe
>     14 BackendOpen  b default 127.0.0.1 38662 127.0.0.1 80
>     14 BackendXID   b 2113606171
>     11 Backend      c 14 default
>     14 TxRequest    b GET
>     14 TxURL        b /create.bml
>     14 TxProtocol   b HTTP/1.1
>     14 TxHeader     b Host: andre.localhost:8080
>     14 TxHeader     b User-Agent: Mozilla/5.0 (Macintosh; U; Intel
> Mac OS X; en-US; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4
>     14 TxHeader     b Accept: text/xml,application/xml,application/
> xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
>     14 TxHeader     b Accept-Language: en-us,en;q=0.5
>     14 TxHeader     b Accept-Encoding: gzip,deflate
>     14 TxHeader     b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>     14 TxHeader     b Connection: keep-alive
>     14 TxHeader     b Referer: http://andre.localhost:8080/create.bml
>     14 TxHeader     b Cookie: ljuniq=btZhgKFp2s9UaAi:1183650844;
> ljuniq=btZhgKFp2s9UaAi:1183650844
>     14 TxHeader     b Content-Type: application/x-www-form-urlencoded
>     14 TxHeader     b Content-Length: 164
>     14 TxHeader     b X-Varnish: 2113606171
>     14 TxHeader     b X-Forwarded-for: 10.134.145.2
>     14 BackendClose b default
>     11 ReqEnd       c 2113606171 1183712178.629895424
> 1183712238.683765133 0.000704753 0.000207682 60.053662027
>      0 StatAddr       10.134.145.2 0 60 1 1 1 0 0 0 0
>     11 StatSess     c 10.134.145.2 49924 60 1 1 1 0 0 0 0
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712241
>      0 CLI            Rd ping
>      0 CLI            Wr 0 200 PONG 1183712244
>
>
>
> I don't know what to make of this... After some time waiting for
> something Varnish processes a POST form the browser but sends a GET
> to the backend? Nothing reaches the browser btw. Strange stuff..
> Maybe someone can shed some light on this...
>
> Thanks for the help!
> Andr?
>
>
> On 2007/07/05, at 12:33, Dag-Erling Sm?rgrav wrote:
>
>> Andr? Cruz <andre.cruz at segula.pt> writes:
>>> I'm having problems with POST requests. They reach varnish but
>>> nothing
>>> happens and the backend is not contacted.
>>>
>>> I see this in the log:
>>>     11 SessionOpen  c 10.134.145.2 51721
>>>
>>> ...
>>>     11 Debug        c "Pipe Shut read(read)"
>>>     11 Debug        c "Pipe Shut write(read)"
>>>
>>> I found this ticket "http://varnish.projects.linpro.no/ticket/47"
>>> which seemed to refer to my problem but even switching to "pipe"  
>>> does
>>> not solve it.
>>
>> "Pipe Shut" comes from the pipe code, so switching *to* pipe won't
>> make
>> any difference because it already *is* pipe.
>>
>> This is not an error message, BTW, just an indication that a pipe
>> session ended.
>>
>> DES
>> -- 
>> Dag-Erling Sm?rgrav
>> Senior Software Developer
>> Linpro AS - www.linpro.no
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc



From phk at phk.freebsd.dk  Fri Jul  6 10:08:30 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 06 Jul 2007 10:08:30 +0000
Subject: Post requests 
In-Reply-To: Your message of "Fri, 06 Jul 2007 10:14:30 +0100."
	<A3AE30CC-AADB-4C85-B430-8DCED8990E88@segula.pt> 
Message-ID: <9266.1183716510@critter.freebsd.dk>


Sorry for not looking at this earlier.

It was a bug that caused all requests, also piped ones, to be
rewritten to "GET" on the way to the backend.

Fixed in -trunk a minute ago.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Fri Jul  6 10:12:31 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 06 Jul 2007 10:12:31 +0000
Subject: Varnish shutting down under heavy load 
In-Reply-To: Your message of "Fri, 06 Jul 2007 09:48:54 +0530."
	<468DC2AE.3050006@iamcool.net> 
Message-ID: <9323.1183716751@critter.freebsd.dk>


I've seen the automatic restart of the worker process fail here
during my testing, and I suspect you hit the same thing.

I have it on my list of things to investigate RSN.

Do you have any corefiles from varnishd by any chance ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Fri Jul  6 10:15:37 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 06 Jul 2007 10:15:37 +0000
Subject: Q: multiple backends 
In-Reply-To: Your message of "Thu, 05 Jul 2007 20:15:18 +0200."
	<20070705181517.GA5791@falcon> 
Message-ID: <9371.1183716937@critter.freebsd.dk>

In message <20070705181517.GA5791 at falcon>, Christoph writes:
>Hi,
>
>are there any plans to include support for multiple backends in varnish?

You can already use multiple backens in Varnish.

>It would be great to have either fault-tolerance and/or simple
>load-balancing. 

Yes, this is indeed on our plans,  I keep trying to get somebody to
tell me how it should work, but with very little luck :-)

Just saying "fault-tolerance" or "load-balancing" isn't saying much
you see, we need to formulate strategies and algorithms...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Fri Jul  6 10:21:16 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 06 Jul 2007 10:21:16 +0000
Subject: Regarding url purging 
In-Reply-To: Your message of "Thu, 05 Jul 2007 20:13:24 +0200."
	<200707052013.24760.gaute@pht.no> 
Message-ID: <9393.1183717276@critter.freebsd.dk>

In message <200707052013.24760.gaute at pht.no>, Gaute Amundsen writes:
>On Thursday 05 July 2007 19:39, Andr=E9 Cruz wrote:

>> I'm caching pages from different vhosts, say x.com and y.com.
>>
>> If I want to purge all the data from one of the vhosts how do I do
>> it? "url.purge x.com.*" does not work. I'm a bit confused with the
>> syntax since the wiki only talks about paths and not complete URLs.
>
>That's not easily done, as far as I have been able to determine.
>I've submitted a feature request for it..

And I've been thinking about it and may have come up with a solution:

When we hash, we hash on a string that by default contains

	req.http.host "#" req.url "#"

In your case for instance it owuld be:

	x.com#/#

It looks like purges need to (be able) match against this string,
rather than the url alone.

I'm not sure what the exact form of the solution will be, but I
may simply add another CLI command:

	hash.purge $regexp

That matches against the hash string.

The caveat of this is, that if the user customizes the hash string
with vcl_hash(), he has to take this into account with his purge
strings.

But I can see how that could be an advantage also: one could "tag"
object in vcl_hash with a class or token, which can later be used
to purge the tagged objects.

Do we have a ticket for this in trac already ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Fri Jul  6 10:44:03 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 06 Jul 2007 10:44:03 +0000
Subject: Mac and regular expressions 
In-Reply-To: Your message of "Thu, 05 Jul 2007 12:03:38 GMT."
	<loom.20070705T135943-423@post.gmane.org> 
Message-ID: <9514.1183718643@critter.freebsd.dk>

In message <loom.20070705T135943-423 at post.gmane.org>, Martin Aspeli writes:
>Hi,
>
>I think I'm being bitten by a bug related to Mac OS X (10.4) and regular
>expressions.

>$ /usr/local/sbin/varnishd -a localhost:8080  -f /etc/varnish.conf 
>Assert error in mgt_CallCc(), mgt_vcc.c line 214:
>  Condition((dlclose(p)) == 0) not true.
>  errno = 2 (No such file or directory)
>Abort trap

Can you check the man pages for dlclose() on your mac, and see if they
explain what it means when dlclose() fails this way ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From gaute at pht.no  Fri Jul  6 10:59:59 2007
From: gaute at pht.no (Gaute Amundsen)
Date: Fri, 6 Jul 2007 12:59:59 +0200
Subject: Regarding url purging
In-Reply-To: <9393.1183717276@critter.freebsd.dk>
References: <9393.1183717276@critter.freebsd.dk>
Message-ID: <200707061300.00062.gaute@pht.no>

On Friday 06 July 2007 12:21, Poul-Henning Kamp wrote:
> In message <200707052013.24760.gaute at pht.no>, Gaute Amundsen writes:
> >On Thursday 05 July 2007 19:39, Andr=E9 Cruz wrote:
> >> I'm caching pages from different vhosts, say x.com and y.com.
> >>
> >> If I want to purge all the data from one of the vhosts how do I do
> >> it? "url.purge x.com.*" does not work. I'm a bit confused with the
> >> syntax since the wiki only talks about paths and not complete URLs.
> >
> >That's not easily done, as far as I have been able to determine.
> >I've submitted a feature request for it..
>
> And I've been thinking about it and may have come up with a solution:
>
> When we hash, we hash on a string that by default contains
>
> 	req.http.host "#" req.url "#"
>
> In your case for instance it owuld be:
>
> 	x.com#/#
>
> It looks like purges need to (be able) match against this string,
> rather than the url alone.
>
> I'm not sure what the exact form of the solution will be, but I
> may simply add another CLI command:
>
> 	hash.purge $regexp
>
> That matches against the hash string.
>
> The caveat of this is, that if the user customizes the hash string
> with vcl_hash(), he has to take this into account with his purge
> strings.
>
> But I can see how that could be an advantage also: one could "tag"
> object in vcl_hash with a class or token, which can later be used
> to purge the tagged objects.
>
> Do we have a ticket for this in trac already ?

That looks like a perfectly fine way to do it as far as my needs go.
 
Beeing able to saw off the branch youre sitting on, is not an unreasonable 
possibility to expect if you want to play with powertools :)

Anyway I'd expect it would be possible to craft a regexp on the #'s that would 
isolate any additions to the hash from changing the match, if that's what you 
wanted..

Ticket? 
This one I guess: http://varnish.projects.linpro.no/ticket/116

Thanks :)

Gaute

-- 
Programmerer - Pixelhospitalet AS
T?rkoppveien 10, 1570 Dilling
Tlf. 24 12 97 81 - 9074 7344


From fragfutter at gmail.com  Fri Jul  6 11:04:46 2007
From: fragfutter at gmail.com (C. Handel)
Date: Fri, 6 Jul 2007 13:04:46 +0200
Subject: Q: multiple backends
In-Reply-To: <9371.1183716937@critter.freebsd.dk>
References: <20070705181517.GA5791@falcon> <9371.1183716937@critter.freebsd.dk>
Message-ID: <3d62bd5f0707060404v2fdbded5v4891ae36f1524bac@mail.gmail.com>

On 7/6/07, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> In message <20070705181517.GA5791 at falcon>, Christoph writes:
> >Hi,
> >
> >are there any plans to include support for multiple backends in varnish?
>
> You can already use multiple backens in Varnish.
>
> >It would be great to have either fault-tolerance and/or simple
> >load-balancing.
>
> Yes, this is indeed on our plans,  I keep trying to get somebody to
> tell me how it should work, but with very little luck :-)
>
> Just saying "fault-tolerance" or "load-balancing" isn't saying much
> you see, we need to formulate strategies and algorithms...

for fault-tolerance the proposal from anton stonor could be used.
Extend it with the backend-name

># First, we setup decide how to "sniff" that a backend is down
>#
># options_ping: Send a HTTP OPTIONS (Perlbal does that)
># timeout: 	If the backend does not answer within x seconds, it is
>#          	probably down
># icp: 		Abuse the protocol. (Squid + Zope does that)
>backend.default.down_protocol = options_ping | timeout | icp

># What is the timeout limit?
>backend.default.timeout = 30

backend main {
        set backend.host = "IP";
        set backend.port = "80";
}
backend backup {
        set backend.host = "IP";
        set backend.port = "80";
}



then we could use

sub vcl_recv {
   set req.backend = main
   if (backend.main.down) {
      set req.backend = backup
   }
}



for loadbalancing we could either extend the backend definition

backend main {
        set backend.1.host = "IP";
        set backend.1.port = "80";
        set backend.1.weight = 1;
        set backend.2.host = "IP";
        set backend.2.port = "80";
        set backend.2.weight = 1;
}

or introduce a new balancer (i don't know if we could just nest backends)

balancer mainbl {
   set balancer.strategy = "roundrobin";
   # set balancer.strategy = "random";
   backend one {
      set backend.host = IP;
      set backend.port = 80;
      set backend.weight = 1;
   }
   backend two {
      set backend.host = IP;
      set backend.port = 80;
      set backend.weight = 1;
   }
}

and use it

sub vcl_recv {
   set req.backend = mainbl
}



Greetings
   Christoph


From fragfutter at gmail.com  Fri Jul  6 11:25:12 2007
From: fragfutter at gmail.com (C. Handel)
Date: Fri, 6 Jul 2007 13:25:12 +0200
Subject: Cascading Caches, honor Age Header
Message-ID: <3d62bd5f0707060425w7afed601o57ea286ec81b2fb8@mail.gmail.com>

Hi,

When i cascade varnish in front of a squid (didn't test varnish in
front of varnish) it won't honor the Age Header sent from the squid.

If the Squid sends me

Age: 100
Date: Fri, 06 Jul 2007 11:08:20 GMT
Expires: Fri, 06 Jul 2007 11:13:20 GMT

varnish should start with an initial Age of 100. This way i could keep
the expiration of multiple varnish instances in sync.

it might be possible to use the new features mentioned in "New VCL
toys". So after fetching a object from the backend, before storing it
i could

set obj.age = now - obj.date

supposing the time on varnish and backend server is in sync.



Greetings
  Christoph


From ssm at linpro.no  Fri Jul  6 11:41:30 2007
From: ssm at linpro.no (Stig Sandbeck Mathisen)
Date: Fri, 06 Jul 2007 13:41:30 +0200
Subject: Q: multiple backends
In-Reply-To: <9371.1183716937@critter.freebsd.dk> (Poul-Henning Kamp's message
	of "Fri, 06 Jul 2007 10:15:37 +0000")
References: <9371.1183716937@critter.freebsd.dk>
Message-ID: <7xps3569p1.fsf@iostat.e.linpro.no>

"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:

> Yes, this is indeed on our plans, I keep trying to get somebody to
> tell me how it should work, but with very little luck :-)
>
> Just saying "fault-tolerance" or "load-balancing" isn't saying much
> you see, we need to formulate strategies and algorithms...

Common load-balancing strategies are:

* Round robin.  (easy to implement)

* Weighted, i.e. this host can handle twice as much as that host.
  (twice as much what?  Bytes? Requests?)  Some black boxes can do a
  TCP connect or a HTTP GET to the backend to fetch a number
  representing load from the web server, used to weigh backends.

* Weighted based on measured response time from backend.  (Send
  requests to the fastest responding node, one should perhaps base
  this on 2xx responses only)

* Hash on client IP, Host: header, substring of session cookie, url,
  or phase of the moon to select backend.  Some application servers
  generate session cookies where the first n bytes represent the ID of
  a backend server.

What to do with requests that are waiting for a failing backend?

* Return a 5xx message

* Try again on another backend?  (Could have undesirable results, and
  not be possible for all request types.  For instance, it's hard to
  replay a piped request)

How often should one retry a failed backend?  
> Never attribute to malice what can adequately be explained by incompetence.

Any sufficiently advanced incompetence is indistinguishable from malice.

-- 
Stig Sandbeck Mathisen, Linpro


From anup at iamcool.net  Fri Jul  6 12:48:57 2007
From: anup at iamcool.net (Anup Shukla)
Date: Fri, 06 Jul 2007 18:18:57 +0530
Subject: Varnish shutting down under heavy load
In-Reply-To: <9323.1183716751@critter.freebsd.dk>
References: <9323.1183716751@critter.freebsd.dk>
Message-ID: <468E3A39.60408@iamcool.net>

Poul-Henning Kamp wrote:
> I've seen the automatic restart of the worker process fail here
> during my testing, and I suspect you hit the same thing.
>
> I have it on my list of things to investigate RSN.
>
> Do you have any corefiles from varnishd by any chance ?
>
>   

I just set ulimit -c unlimited and started varnish again.

Regards.
A.S


From anup at iamcool.net  Fri Jul  6 13:02:36 2007
From: anup at iamcool.net (Anup Shukla)
Date: Fri, 06 Jul 2007 18:32:36 +0530
Subject: Varnish shutting down under heavy load
In-Reply-To: <468E3A39.60408@iamcool.net>
References: <9323.1183716751@critter.freebsd.dk> <468E3A39.60408@iamcool.net>
Message-ID: <468E3D6C.9080000@iamcool.net>

Anup Shukla wrote:
> Poul-Henning Kamp wrote:
>   
>> I've seen the automatic restart of the worker process fail here
>> during my testing, and I suspect you hit the same thing.
>>
>> I have it on my list of things to investigate RSN.
>>
>> Do you have any corefiles from varnishd by any chance ?
>>
>>   
>>     
>
> I just set ulimit -c unlimited and started varnish again.
>
> Regards.
>   

Did not take it long to go down again.
But i am not able to find any "core" dumps.
Some help as to how to get one will be appreciated.

The logs said

  843 TxHeader     c X-Varnish: 1617761482 1617756991
  843 TxHeader     c Age: 75
  843 TxHeader     c Via: 1.1 varnish
  843 ReqEnd       c 1617761482 1183726324.435635000 
1183726324.435695000 4.934160000 0.000019000 0.000041000
    0 StatAddr       124.64.56.126 0 31 11 15 0 0 1 4243 180969
 1017 SessionOpen  c 58.214.216.252 3364
    0 CLI            Rd vcl.load boot /tmp/vcl.XXnkvV6y
    0 CLI            Wr 0 200 Loaded "/tmp/vcl.XXnkvV6y" as "boot"

    0 Error          CLI read 0 (errno=4)


> A.S
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
>   



From phk at phk.freebsd.dk  Fri Jul  6 13:05:08 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 06 Jul 2007 13:05:08 +0000
Subject: Varnish shutting down under heavy load 
In-Reply-To: Your message of "Fri, 06 Jul 2007 18:32:36 +0530."
	<468E3D6C.9080000@iamcool.net> 
Message-ID: <10028.1183727108@critter.freebsd.dk>

In message <468E3D6C.9080000 at iamcool.net>, Anup Shukla writes:
>Anup Shukla wrote:

>Did not take it long to go down again.
>But i am not able to find any "core" dumps.
>Some help as to how to get one will be appreciated.

Ok, next step:

Run varnishd in the forground by giving it two -d options:

	varnishd -d -d [your other options]



-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From optilude at gmx.net  Fri Jul  6 22:23:32 2007
From: optilude at gmx.net (Martin Aspeli)
Date: Fri, 06 Jul 2007 23:23:32 +0100
Subject: Mac and regular expressions
In-Reply-To: <9514.1183718643@critter.freebsd.dk>
References: <9514.1183718643@critter.freebsd.dk>
Message-ID: <468EC0E4.9070904@gmx.net>

Poul-Henning Kamp wrote:
> In message <loom.20070705T135943-423 at post.gmane.org>, Martin Aspeli writes:
>> Hi,
>>
>> I think I'm being bitten by a bug related to Mac OS X (10.4) and regular
>> expressions.
> 
>> $ /usr/local/sbin/varnishd -a localhost:8080  -f /etc/varnish.conf 
>> Assert error in mgt_CallCc(), mgt_vcc.c line 214:
>>  Condition((dlclose(p)) == 0) not true.
>>  errno = 2 (No such file or directory)
>> Abort trap
> 
> Can you check the man pages for dlclose() on your mac, and see if they
> explain what it means when dlclose() fails this way ?

Yep - it doesn't look too useful, though. I'm guessing the question is, 
"which file is it that it can't find".

$ man dlclose

SYNOPSIS
      #include <dlfcn.h>

      int
      dlclose(void* handle);

DESCRIPTION
      dlclose() releases a reference to the dynamic library or bundle refer-
      enced by handle.  If the reference count drops to 0, the bundle is
      removed from the address space, and handle is rendered invalid.  Just
      before removing a dynamic library or bundle in this way, any 
termination
      routines in it are called.  handle is the value returned by a previous
      call to dlopen.

RETURN VALUES
      If dlclose() is successful, it returns a value of 0.  Otherwise it
      returns -1, and sets an error string that can be retrived with 
dlerror().

$ man dlerror

NAME
      dlerror -- get diagnostic information

SYNOPSIS
      #include <dlfcn.h>

      const char*
      dlerror(void);

DESCRIPTION
      dlerror() returns a null-terminated character string describing 
the last
      error that occurred on this thread during a call to dlopen(), 
dlsym(), or
      dlclose().  If no such error has occurred, dlerror() returns a null
      pointer.  At each call to dlerror(), the error indication is 
reset.  Thus
      in the case of two calls to dlerror(), where the second call 
follows the
      first immediately, the second call will always return a null pointer.

SEE ALSO
      dlopen(3) dlclose(3) dlsym(3) dyld(3)

...

Someone also suggested I invoke cc on /tmp/_.c, which gets:

$ cc /tmp/_.c

/usr/bin/ld: Undefined symbols:
_main
_VRT_GetHdr
_VRT_acl_fini
_VRT_acl_init
_VRT_acl_match
_VRT_alloc_backends
_VRT_count
_VRT_error
_VRT_fini_backend
_VRT_free_backends
_VRT_handling
_VRT_l_backend_host
_VRT_l_backend_port
_VRT_l_obj_ttl
_VRT_r_obj_cacheable
_VRT_r_obj_valid
_VRT_r_req_request
_VRT_set_backend_name

But I suspect that may be because I haven't passed the right parameters 
to cc.

Thanks,
Martin


From anup at iamcool.net  Sat Jul  7 03:57:29 2007
From: anup at iamcool.net (Anup Shukla)
Date: Sat, 07 Jul 2007 09:27:29 +0530
Subject: Varnish shutting down under heavy load
In-Reply-To: <10028.1183727108@critter.freebsd.dk>
References: <10028.1183727108@critter.freebsd.dk>
Message-ID: <468F0F29.2060002@iamcool.net>

Poul-Henning Kamp wrote:
> In message <468E3D6C.9080000 at iamcool.net>, Anup Shukla writes:
>   
>> Anup Shukla wrote:
>>     
>
>   
>> Did not take it long to go down again.
>> But i am not able to find any "core" dumps.
>> Some help as to how to get one will be appreciated.
>>     
>
> Ok, next step:
>
> Run varnishd in the forground by giving it two -d options:
>
> 	varnishd -d -d [your other options]
>   

This is the output from the debug console (a lot of "Child not 
responding" messages scrolled beyond the screen buffer)
=============================
.....
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Cache child died pid=29313 status=0x86
Clean child
Child cleaned
start child pid 2961
Child said (2, 2961): <<Child starts
managed to mmap 8589934592 bytes of 8589934592
Ready
CLI ready
 >>
Child said (2, 2961): <<socktest: linger=0 sndtimeo=0 rcvtimeo=0
 >>
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Cache child died pid=2961 status=0x9
Clean child
Child cleaned
start child pid 5942
Pushing vcls failed:
CLI communication error
Child said (1, 5942): <<Child starts
 >>
unlink /tmp/vcl.XXxtW5RB


varnish.log
=============================
....
  199 TxHeader     c ETag: "56373-7414-43120356ea200"
  199 TxHeader     c Content-Type: application/x-shockwave-flash
  199 TxHeader     c Content-Length: 29716
  199 TxHeader     c X-Varnish: 1888641652
  199 TxHeader     c Age: 0
  199 TxHeader     c Via: 1.1 varnish
  199 ReqEnd       c 1888641652 1183753467.604381000 
1183753469.254338000 0.000404000 0.764325000 0.885632000
    0 StatAddr       218.164.51.156 0 1534 136 274 2 0 188 62079 46278586
    0 CLI            Rd vcl.load boot /tmp/vcl.XXxtW5RB
    0 Error          CLI write failed (errno=32)


The core dump is 11gb in size !!

gdb [varnishd] [corefile]
=============================
Core was generated by `varnishd -d -d -a 192.168.100.109:80 -f 
/etc/varnish/default.vcl -s file,/cache'.
Program terminated with signal 6, Aborted.
Error while mapping shared library sections:
/tmp/vcl.XXxtW5RB: No such file or directory.
Reading symbols from /usr/lib64/libvarnish.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libvarnish.so.0
Reading symbols from /usr/lib64/libvcl.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libvcl.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/tls/librt.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/tls/librt.so.1
Reading symbols from /lib64/tls/libpthread.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/tls/libpthread.so.0
Reading symbols from /lib64/tls/libc.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib64/tls/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Error while reading shared library symbols:
/tmp/vcl.XXxtW5RB: No such file or directory.
#0  0x000000386742e21d in raise () from /lib64/tls/libc.so.6
(gdb) bt
#0  0x000000386742e21d in raise () from /lib64/tls/libc.so.6
#1  0x000000386742fa1e in abort () from /lib64/tls/libc.so.6
#2  0x000000321d502205 in lbv_assert () from /usr/lib64/libvarnish.so.0
#3  0x000000000041b9fc in VSL_MgtInit ()
#4  0x000000000040a611 in Fetch ()
#5  0x00000000004086f3 in CNT_Session ()
#6  0x000000000040ed0b in WRK_Sendfile ()
#7  0x000000000040ef12 in WRK_Sendfile ()
#8  0x0000003868106137 in start_thread () from /lib64/tls/libpthread.so.0
#9  0x00000038674c7113 in clone () from /lib64/tls/libc.so.6

Anything else that i should do?

Regards
A.S



From des at linpro.no  Sat Jul  7 18:50:14 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sat, 07 Jul 2007 20:50:14 +0200
Subject: child process restarts frequently
In-Reply-To: <468DF2D9.3090607@met.no> (=?utf-8?Q?=22H=C3=A5vard_Futs?=
	=?utf-8?Q?=C3=A6ter=22's?= message of "Fri\,
	06 Jul 2007 09\:44\:25 +0200")
References: <468B7A9A.3040706@met.no> <87myybdqj8.fsf@des.linpro.no>
	<468DF2D9.3090607@met.no>
Message-ID: <877ipc9hg9.fsf@des.linpro.no>

H?vard Futs?ter <havard.futseter at met.no> writes:
> Might this perhaps be an indication that Varnish gets into trouble
> when it receives a request for a resource while the cached object for
> this resource is beeing recycled?

Sounds like it.  Poul-Henning, could you take a closer look at the
locking in the expiry code?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ivoras at fer.hr  Fri Jul  6 10:50:32 2007
From: ivoras at fer.hr (Ivan Voras)
Date: Fri, 06 Jul 2007 12:50:32 +0200
Subject: Q: multiple backends
In-Reply-To: <9371.1183716937@critter.freebsd.dk>
References: <9371.1183716937@critter.freebsd.dk>
Message-ID: <468E1E78.3050202@fer.hr>

Poul-Henning Kamp wrote:

> Just saying "fault-tolerance" or "load-balancing" isn't saying much
> you see, we need to formulate strategies and algorithms...

What's wrong with the usual list:

- round-robin
- by load (measure response time from last x requests, send to the faster)
- by content (request .pngs from one server, .jpgs from the other)
- by address (request /dir1/* from one server, /dir2/* from the other)
- weighted (60% from one server, 40% from the other)

and the freaks:
- by hashing the incoming IP (so each IP gets bound to one server, 
useful for cookies)
- by size (learn object sizes, get smaller objects from one server, 
larger from the other)
- by cookie (learn cookies, hash/bin them to servers)

? :)

(this post is a half-joke, I agree there are so many options that it's 
hard to pick the best ones :) )

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070706/69b05934/attachment.bin>

From tiamo at tfr.org  Mon Jul  9 11:34:35 2007
From: tiamo at tfr.org (TiAMO)
Date: Mon, 9 Jul 2007 13:34:35 +0200
Subject: Statistics
Message-ID: <200707091334.35737.tiamo@tfr.org>

Is there anyway to get statistics based on the HTTP/1.1 hosts header, i would 
like to know how many % of the total traffic thats beeing used by each host.

// Fredrik Neij


From phk at phk.freebsd.dk  Mon Jul  9 15:36:02 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 09 Jul 2007 15:36:02 +0000
Subject: Statistics 
In-Reply-To: Your message of "Mon, 09 Jul 2007 13:34:35 +0200."
	<200707091334.35737.tiamo@tfr.org> 
Message-ID: <75918.1183995362@critter.freebsd.dk>

In message <200707091334.35737.tiamo at tfr.org>, TiAMO writes:


>Is there anyway to get statistics based on the HTTP/1.1 hosts header, i would 
>like to know how many % of the total traffic thats beeing used by each host.

We don't have a programme that does this right now, but you can get
some kind of indication for the number of transactions (but not their
size) by using  something like:

	varnishtop -i RxHeader -I Host:

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From denis at zeno.org  Mon Jul  9 20:08:49 2007
From: denis at zeno.org (Denis Ahrens)
Date: Mon, 9 Jul 2007 22:08:49 +0200
Subject: keep-alive and Firefox
Message-ID: <4FC0AFA0-F65F-49B2-BC6B-55B0266E1F64@zeno.org>

Hi

Somewhere around r1640 Firefox started to behave weirdly
while loading pages from varnish.

It seems to have to do with an entry in vcl_recv like:

     if (req.url ~ "random") {
         set req.backend = server2;
         pass;
     }

But Safari and Explorer handle that fine.

I think it has to do with keep-alive.

Denis



From phk at phk.freebsd.dk  Mon Jul  9 20:24:48 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 09 Jul 2007 20:24:48 +0000
Subject: keep-alive and Firefox 
In-Reply-To: Your message of "Mon, 09 Jul 2007 22:08:49 +0200."
	<4FC0AFA0-F65F-49B2-BC6B-55B0266E1F64@zeno.org> 
Message-ID: <96745.1184012688@critter.freebsd.dk>

In message <4FC0AFA0-F65F-49B2-BC6B-55B0266E1F64 at zeno.org>, Denis Ahrens writes
:
>Hi
>
>Somewhere around r1640 Firefox started to behave weirdly
>while loading pages from varnish.
>
>It seems to have to do with an entry in vcl_recv like:
>
>     if (req.url ~ "random") {
>         set req.backend = server2;
>         pass;
>     }
>
>But Safari and Explorer handle that fine.
>
>I think it has to do with keep-alive.

Can you get me at least a varnishlog output ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From denis at zeno.org  Mon Jul  9 21:13:07 2007
From: denis at zeno.org (Denis Ahrens)
Date: Mon, 9 Jul 2007 23:13:07 +0200
Subject: keep-alive and Firefox 
In-Reply-To: <96745.1184012688@critter.freebsd.dk>
References: <96745.1184012688@critter.freebsd.dk>
Message-ID: <8869B04C-3127-4D34-B823-56D46CEDD56E@zeno.org>

On 09.07.2007, at 22:24, Poul-Henning Kamp wrote:

> In message <4FC0AFA0-F65F-49B2-BC6B-55B0266E1F64 at zeno.org>, Denis  
> Ahrens writes
> :
>> Hi
>>
>> Somewhere around r1640 Firefox started to behave weirdly
>> while loading pages from varnish.
>>
>> It seems to have to do with an entry in vcl_recv like:
>>
>>    if (req.url ~ "random") {
>>        set req.backend = server2;
>>        pass;
>>    }
>>
>> But Safari and Explorer handle that fine.
>>
>> I think it has to do with keep-alive.
>
> Can you get me at least a varnishlog output ?

oh, I'm sorry, but while diffing the logoutput
manually I recognized an interesting header:

13 TxHeader     c Age: 3110952656

With r1639 the Age header is only 245s

Maybe this is it.

Denis



From lists at tomster.org  Mon Jul  9 20:28:26 2007
From: lists at tomster.org (Tom Lazar)
Date: Mon, 9 Jul 2007 22:28:26 +0200
Subject: varnish and Nginx
In-Reply-To: <7xtzsm2i15.fsf@iostat.e.linpro.no>
References: <21171599.18191183385150208.JavaMail.root@ms1.startsiden.no>
	<46890BFD.3030003@gmail.com> <7xtzsm2i15.fsf@iostat.e.linpro.no>
Message-ID: <9DE7BA59-0AE2-4AA9-BF01-156FE9DA3CF9@tomster.org>

On Jul 3, 2007, at 7:05 AM, Stig Sandbeck Mathisen wrote:

> Dingo <outbackdingo at gmail.com> writes:
>
>> So its feasible to run Nginx as the server/load balancer and varnish
>> as the front end cache giving potentially a decent high speed/high
>> capacity design.
>
> Yes, those two will complement eachother.

in fact, lovely systems is using that very constellation and  
apparently has had good success with it.

they've discribed their setup at the recent german zope conference.  
perhaps you might find their presentation interesting:

http://www.zope.de/redaktion/dzug/tagung/potsdam-2007/folien/potsdam- 
zope3-performance.pdf/view

hth,

tom

>
> -- 
> Stig Sandbeck Mathisen, Linpro
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>



From lists at tomster.org  Mon Jul  9 22:29:53 2007
From: lists at tomster.org (Tom Lazar)
Date: Tue, 10 Jul 2007 00:29:53 +0200
Subject: varnish and Nginx
In-Reply-To: <9DE7BA59-0AE2-4AA9-BF01-156FE9DA3CF9@tomster.org>
References: <21171599.18191183385150208.JavaMail.root@ms1.startsiden.no>
	<46890BFD.3030003@gmail.com> <7xtzsm2i15.fsf@iostat.e.linpro.no>
	<9DE7BA59-0AE2-4AA9-BF01-156FE9DA3CF9@tomster.org>
Message-ID: <0B872AFF-7AEA-47AB-8B62-6484172D616C@tomster.org>


On Jul 9, 2007, at 10:28 PM, Tom Lazar wrote:

> they've discribed their setup at the recent german zope conference.
> perhaps you might find their presentation interesting:
>
> http://www.zope.de/redaktion/dzug/tagung/potsdam-2007/folien/potsdam-
> zope3-performance.pdf/view

or perhaps not... i just browsed through the pdf and it's not really  
informative without jodok's talk, don't bother with it (i was in the  
room at the time and found the story quite compelling).


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070710/b72800e7/attachment.html>

From phk at phk.freebsd.dk  Tue Jul 10 10:15:07 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 10 Jul 2007 10:15:07 +0000
Subject: Q: multiple backends 
In-Reply-To: Your message of "Fri, 06 Jul 2007 12:50:32 +0200."
	<468E1E78.3050202@fer.hr> 
Message-ID: <3782.1184062507@critter.freebsd.dk>

In message <468E1E78.3050202 at fer.hr>, Ivan Voras writes:

>
>> Just saying "fault-tolerance" or "load-balancing" isn't saying much
>> you see, we need to formulate strategies and algorithms...
>
>What's wrong with the usual list:

>[...]

>(this post is a half-joke, I agree there are so many options that it's=20
>hard to pick the best ones :) )

There is your point right there:  You won't have "all of the above"
any time soon.  What I'm looking for is input in which 20% of the work
gives 80% of the benefit.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From joao.martins-ext_nbs at siemens.com  Tue Jul 10 10:25:17 2007
From: joao.martins-ext_nbs at siemens.com (Martins, Joao (ext))
Date: Tue, 10 Jul 2007 11:25:17 +0100
Subject: Compiling Varnish on Solaris 9 / 10
Message-ID: <EB07A87D94F3EA4DAECC7D9B3BD664BE0118D3D8@PTLISI050MSX.PT001.SIEMENS.NET>

Hi,
 
I would like to try out Varnish to replace a Squid basd solution for reverse proxying. However, our app runs on Solaris, and we're not moving to Linux any time soon. Has anyone here tried to compile Varnish on Solaris? I did a quick configure / make (varnish 1.0.4), and it dies with some missing headers.


Thanks  
--
Jo?o Martins



From joao.martins-ext_nbs at siemens.com  Mon Jul  9 19:25:19 2007
From: joao.martins-ext_nbs at siemens.com (Martins, Joao (ext))
Date: Mon, 9 Jul 2007 20:25:19 +0100
Subject: Compiling varnish on Solaris
Message-ID: <EB07A87D94F3EA4DAECC7D9B3BD664BE0118D1EE@PTLISI050MSX.PT001.SIEMENS.NET>

Hi,
 
I would like to try out Varnish to replace a Squid basd solution for reverse proxying. However, our app runs on Solaris, and we're not moving to Linux any time soon. Has anyone here tryed to compile Varnish on Solaris? I did a quick configure / make (varnsih 1.0.4), and it dies with some missing headers.
 
Thanks
--
Jo?o Martins

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070709/8864837c/attachment.html>

From des at linpro.no  Tue Jul 10 10:29:44 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 10 Jul 2007 12:29:44 +0200
Subject: Compiling varnish on Solaris
In-Reply-To: <EB07A87D94F3EA4DAECC7D9B3BD664BE0118D1EE@PTLISI050MSX.PT001.SIEMENS.NET>
	(Joao Martins's message of "Mon\, 9 Jul 2007 20\:25\:19 +0100")
References: <EB07A87D94F3EA4DAECC7D9B3BD664BE0118D1EE@PTLISI050MSX.PT001.SIEMENS.NET>
Message-ID: <87myy4d013.fsf@des.linpro.no>

"Martins, Joao (ext)" <joao.martins-ext_nbs at siemens.com> writes:
> I would like to try out Varnish to replace a Squid basd solution for
> reverse proxying. However, our app runs on Solaris, and we're not
> moving to Linux any time soon. Has anyone here tryed to compile
> Varnish on Solaris? I did a quick configure / make (varnsih 1.0.4),
> and it dies with some missing headers.

We have patches for Solaris, but lack testing facilities.  I have
received a Solaris 10 media kit from Sun which I will try to install in
VMWare so I can test the patches - hopefully in time for 1.1.  I tried
OpenSolaris Nevada previously, and had no luck running it in VMWare.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From joao.martins-ext_nbs at siemens.com  Tue Jul 10 10:33:19 2007
From: joao.martins-ext_nbs at siemens.com (Martins, Joao (ext))
Date: Tue, 10 Jul 2007 11:33:19 +0100
Subject: Compiling varnish on Solaris
In-Reply-To: <87myy4d013.fsf@des.linpro.no>
Message-ID: <EB07A87D94F3EA4DAECC7D9B3BD664BE0118D3F4@PTLISI050MSX.PT001.SIEMENS.NET>

If you can send me the patches, I can have a go at compiling / running on our machines, and let you know the results.

What compiler is needed? GCC ? SunStudio?


--
./Jo?o

-----Original Message-----
From: Dag-Erling Sm?rgrav [mailto:des at linpro.no] 
Sent: ter?a-feira, 10 de Julho de 2007 11:30
To: Martins, Joao (ext)
Cc: varnish-misc at projects.linpro.no
Subject: Re: Compiling varnish on Solaris

"Martins, Joao (ext)" <joao.martins-ext_nbs at siemens.com> writes:
> I would like to try out Varnish to replace a Squid basd solution for
> reverse proxying. However, our app runs on Solaris, and we're not
> moving to Linux any time soon. Has anyone here tryed to compile
> Varnish on Solaris? I did a quick configure / make (varnsih 1.0.4),
> and it dies with some missing headers.

We have patches for Solaris, but lack testing facilities.  I have
received a Solaris 10 media kit from Sun which I will try to install in
VMWare so I can test the patches - hopefully in time for 1.1.  I tried
OpenSolaris Nevada previously, and had no luck running it in VMWare.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From denis at startsiden.no  Tue Jul 10 11:52:14 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Tue, 10 Jul 2007 13:52:14 +0200 (CEST)
Subject: Q: multiple backends
In-Reply-To: <3782.1184062507@critter.freebsd.dk>
Message-ID: <30304361.52211184068334528.JavaMail.root@ms1.startsiden.no>

----- Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> There is your point right there:  You won't have "all of the above"
> any time soon.  What I'm looking for is input in which 20% of the
> work gives 80% of the benefit.

A huge step would be to have even a basic facility for multiple backends, like round-robin or weighted round-robin (which should theoretically be the easiest models to implement as they require little in the way of monitoring of the backends?). 

Having such an option would enable people to setup simple fault-tolerance and would increase the usability of Varnish a lot. 
More advanced models of dividing the load on the backends are (very) nice to have features imho, the big leap is any functionality at all.

Of course this is all purely my humble opinion.

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From florian.schulze at gmx.net  Tue Jul 10 12:32:03 2007
From: florian.schulze at gmx.net (Florian Schulze)
Date: Tue, 10 Jul 2007 14:32:03 +0200
Subject: Q: multiple backends
References: <3782.1184062507@critter.freebsd.dk>
	<30304361.52211184068334528.JavaMail.root@ms1.startsiden.no>
Message-ID: <op.tu803puxhjnvn3@news.gmane.org>

On Tue, 10 Jul 2007 13:52:14 +0200, Denis Br?khus  
<denis at startsiden.no> wrote:

> ----- Poul-Henning Kamp <phk at phk.freebsd.dk>  
> wrote:
>> There is your point right there:  You won't have "all of the above"
>> any time soon.  What I'm looking for is input in which 20% of the
>> work gives 80% of the benefit.
>
> A huge step would be to have even a basic facility for multiple  
> backends, like round-robin or weighted round-robin (which should  
> theoretically be the easiest models to implement as they require little  
> in the way of monitoring of the backends?).
>
> Having such an option would enable people to setup simple  
> fault-tolerance and would increase the usability of Varnish a lot.
> More advanced models of dividing the load on the backends are (very)  
> nice to have features imho, the big leap is any functionality at all.
>
> Of course this is all purely my humble opinion.
>
> Regards


Hi!

Why don't you use a dedicated load balancer for this? SOmething like  
HAProxy.

Regards,
Florian Schulze



From denis at startsiden.no  Tue Jul 10 12:49:46 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Tue, 10 Jul 2007 14:49:46 +0200 (CEST)
Subject: Q: multiple backends
In-Reply-To: <33239283.52821184071766307.JavaMail.root@ms1.startsiden.no>
Message-ID: <5427057.52851184071786562.JavaMail.root@ms1.startsiden.no>

----- Florian Schulze <florian.schulze at gmx.net> wrote:
> > A huge step would be to have even a basic facility for multiple
> > backends, like round-robin or weighted round-robin (which should
> > theoretically be the easiest models to implement as they require
> > little in the way of monitoring of the backends?).
> > Having such an option would enable people to setup simple
> > fault-tolerance and would increase the usability of Varnish a lot.
> > More advanced models of dividing the load on the backends are (very)
> > nice to have features imho, the big leap is any functionality at
> > all.
> > Of course this is all purely my humble opinion.
> Why don't you use a dedicated load balancer for this? SOmething like
> HAProxy.

Of course there are quite a few alternatives, both software and hardware, open source and proprietary. However Varnish is currently positioned to be placed as near the end user as possible, and having the option of not implementing a separate layer in the design for this kind of functionality would be excellent, both reducing complexity and possible points of failure / bugs.

As previously discussed, other reverse proxies (like nginx and Perlbal) implement such functionality in some form or another (Perlbal being the most advanced of the two it seems). 

I would very much like to have simple loadbalancing built into Varnish instead of having to rely on an additional 3rd party solution in all setups.

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From phk at phk.freebsd.dk  Tue Jul 10 21:35:26 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 10 Jul 2007 21:35:26 +0000
Subject: Varnish can edit headers now, pleast test it
Message-ID: <53412.1184103326@critter.freebsd.dk>


Please comment and test this facility.

Be prepared to increase the "http_workspace" if you do a lot of
string manipulations.  The error handling if you run out of workspace
is not very good, in general VCL will just "not do anything" if it
runs out of space, but not tell you why.

Bug reports, suggestions & ideas are most welcome.

Poul-Henning

------- Forwarded Message

From: phk at projects.linpro.no
Subject: r1667 - in trunk/varnish-cache: bin/varnishd include lib/libvcl
Message-Id: <20070710213047.DC6F71EC464 at projects.linpro.no>
Date: Tue, 10 Jul 2007 23:30:47 +0200 (CEST)

Author: phk
Date: 2007-07-10 23:30:47 +0200 (Tue, 10 Jul 2007)
New Revision: 1667

Log:
Add "regsub" support for string manipulation.

Notice this facility is subject to change!

"regsub" is short for regular expression substitution and it is probably
easiest to explain with some examples:

	sub vcl_recv {
		set req.url = regsub(req.url, "#.*", "");
	}

This will replace the requests URL with the output of the regsub() function

regsub() takes three arguments: the string to be examined, a regular
expression and a replacement string.

In this case, everything after the first '#' is removed (replaced
with nothing).

The replacement string recognizes the following magic sequences:
	&	- insert everything matched by the regexp
	$0	- ditto.
	$1	- replace with the first submatch of the regexp
	$2	- replace with the second submatch of the regexp
	...
	$9	- replace with the ninth submatch of the regexp

(The $0..$9 syntax was chosen over the \0...\9 syntax in order to avoid
a nightmare of escape characters in the VCL source code.  Arguments and
suggestions are welcome).

A more advanced example:

	set bereq.http.ClientIP = regsub(client.ip, "(.*):(.*)", "$2 $1");

The client.ip variable expands to IP:port number, for instance
	127.0.0.1:54662

The regular expression "(.*):(.*)" results in the the following matches:
	& + $0		"127.0.0.1:54662"
	$1		"127.0.0.1"
	$2		"54662"

So the replacement string "$2 $1" results in "54662 127.0.0.1"

And the completed header which is sent to the backend will look like:

	"ClientIP: 54662 127.0.0.1"

An even more advanced example would be:

    set bereq.http.magic = "Client IP = " regsub(client.ip, ":", " port = ");

Where we also exploint the string concatenation ability of the "set" statement.

The result string is built in the request workspace, so you may need
to increase the workspace size if you do a lot of regsub()'s.

Currently there is no decent error handling for running out of workspace.


Modified: trunk/varnish-cache/bin/varnishd/cache_vrt_re.c
===================================================================
- --- trunk/varnish-cache/bin/varnishd/cache_vrt_re.c	2007-07-10 20:43:24 UTC
 (rev 1666)
+++ trunk/varnish-cache/bin/varnishd/cache_vrt_re.c	2007-07-10 21:30:47 UTC
 (rev 1667)
@@ -35,6 +35,7 @@
 
 #include <stdio.h>
 #include <string.h>
+#include <ctype.h>
 #include <stdlib.h>
 #include <regex.h>
 
@@ -100,13 +101,72 @@
 	return (1);
 }
 
- -char *
+const char *
 VRT_regsub(struct sess *sp, const char *str, void *re, const char *sub)
 {
- -	static char foo[4] = "FOO";
- -	(void)sp;
- -	(void)str;
- -	(void)re;
- -	(void)sub;
- -	return (foo);
+	regmatch_t pm[10];
+	regex_t *t;
+	int i, l;
+	char *b, *p, *e;
+	unsigned u, x;
+
+	AN(re);
+	t = re;
+	i = regexec(t, str, 10, pm, 0);
+
+	/* If it didn't match, we can return the original string */
+	if (i == REG_NOMATCH)
+		return(str);
+
+	u = WS_Reserve(sp->http->ws, 0);
+	e = p = b = sp->http->ws->f;
+	e += u;
+
+	/* Copy prefix to match */
+	if (pm[0].rm_so > 0) {
+		if (p + pm[0].rm_so < e)
+			memcpy(p, str, pm[0].rm_so);
+		p += pm[0].rm_so;
+	}
+
+	for ( ; *sub != '\0'; sub++ ) {
+		if (*sub == '&') {
+			l = pm[0].rm_eo - pm[0].rm_so;
+			if (l > 0) {
+				if (p + l < e)
+					memcpy(p, str + pm[0].rm_so, l);
+				p += l;
+			}
+		} else if (*sub == '$' && isdigit(sub[1])) {
+			x = sub[1] - '0';
+			sub++;
+			l = pm[x].rm_eo - pm[x].rm_so;
+			if (l > 0) {
+				if (p + l < e)
+					memcpy(p, str + pm[x].rm_so, l);
+				p += l;
+			}
+		} else {
+			if (p + 1 < e)
+				*p = *sub;
+			p++;
+		}
+	}
+
+	/* Copy suffix to match */
+	l = strlen(str + pm[0].rm_eo);
+	if (l > 0) {
+		if (p + l < e)
+			memcpy(p, str + pm[0].rm_eo, l);
+		p += l;
+	}
+	if (p + 1 < e)
+		*p++ = '\0';
+	xxxassert(p <= e);
+	if (p > e) {
+		WS_Release(sp->http->ws, 0);
+		return (str);
+	} 
+	WS_Release(sp->http->ws, p - b);
+	return (b);
 }

Modified: trunk/varnish-cache/include/vrt.h
===================================================================
- --- trunk/varnish-cache/include/vrt.h	2007-07-10 20:43:24 UTC (rev 1666)
+++ trunk/varnish-cache/include/vrt.h	2007-07-10 21:30:47 UTC (rev 1667)
@@ -68,7 +68,7 @@
 void VRT_re_fini(void *);
 int VRT_re_match(const char *, void *re);
 int VRT_re_test(struct vsb *, const char *, int sub);
- -char *VRT_regsub(struct sess *sp, const char *, void *, const char *);
+const char *VRT_regsub(struct sess *sp, const char *, void *, const char *);
 
 void VRT_count(struct sess *, unsigned);
 int VRT_rewrite(const char *, const char *);

Modified: trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c
===================================================================
- --- trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c	2007-07-10 20:43:24 UTC
 (rev 1666)
+++ trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c	2007-07-10 21:30:47 UTC
 (rev 1667)
@@ -424,7 +424,7 @@
 	vsb_cat(sb, "void VRT_re_fini(void *);\n");
 	vsb_cat(sb, "int VRT_re_match(const char *, void *re);\n");
 	vsb_cat(sb, "int VRT_re_test(struct vsb *, const char *, int sub);\n");
- -	vsb_cat(sb, "char *VRT_regsub(struct sess *sp, const char *, void *, co
nst char *);\n");
+	vsb_cat(sb, "const char *VRT_regsub(struct sess *sp, const char *, void
 *, const char *);\n");
 	vsb_cat(sb, "\n");
 	vsb_cat(sb, "void VRT_count(struct sess *, unsigned);\n");
 	vsb_cat(sb, "int VRT_rewrite(const char *, const char *);\n");

_______________________________________________
varnish-commit mailing list
varnish-commit at projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-commit


------- End of Forwarded Message

============================================================================
From:    Poul-Henning Kamp <phk at critter.freebsd.dk>
Subject: Varnish can edit headers now, pleast test it
To:      varnish-misc at projects.linpro.no
Message-Id: <53367.1184103216 at critter.freebsd.dk>
Date:    Tue, 10 Jul 2007 21:33:36 GMT


Please comment and test this facility.

Be prepared to increase the "http_workspace" if you do a lot of
string manipulations.  The error handling if you run out of workspace
is not very good, in general VCL will just "not do anything" if it
runs out of space, but not tell you why.

Bug reports, suggestions & ideas are most welcome.

Poul-Henning

------- Forwarded Message

From: phk at projects.linpro.no
Subject: r1667 - in trunk/varnish-cache: bin/varnishd include lib/libvcl
Message-Id: <20070710213047.DC6F71EC464 at projects.linpro.no>
Date: Tue, 10 Jul 2007 23:30:47 +0200 (CEST)

Author: phk
Date: 2007-07-10 23:30:47 +0200 (Tue, 10 Jul 2007)
New Revision: 1667

Log:
Add "regsub" support for string manipulation.

Notice this facility is subject to change!

"regsub" is short for regular expression substitution and it is probably
easiest to explain with some examples:

	sub vcl_recv {
		set req.url = regsub(req.url, "#.*", "");
	}

This will replace the requests URL with the output of the regsub() function

regsub() takes three arguments: the string to be examined, a regular
expression and a replacement string.

In this case, everything after the first '#' is removed (replaced
with nothing).

The replacement string recognizes the following magic sequences:
	&	- insert everything matched by the regexp
	$0	- ditto.
	$1	- replace with the first submatch of the regexp
	$2	- replace with the second submatch of the regexp
	...
	$9	- replace with the ninth submatch of the regexp

(The $0..$9 syntax was chosen over the \0...\9 syntax in order to avoid
a nightmare of escape characters in the VCL source code.  Arguments and
suggestions are welcome).

A more advanced example:

	set bereq.http.ClientIP = regsub(client.ip, "(.*):(.*)", "$2 $1");

The client.ip variable expands to IP:port number, for instance
	127.0.0.1:54662

The regular expression "(.*):(.*)" results in the the following matches:
	& + $0		"127.0.0.1:54662"
	$1		"127.0.0.1"
	$2		"54662"

So the replacement string "$2 $1" results in "54662 127.0.0.1"

And the completed header which is sent to the backend will look like:

	"ClientIP: 54662 127.0.0.1"

An even more advanced example would be:

    set bereq.http.magic = "Client IP = " regsub(client.ip, ":", " port = ");

Where we also exploint the string concatenation ability of the "set" statement.

The result string is built in the request workspace, so you may need
to increase the workspace size if you do a lot of regsub()'s.

Currently there is no decent error handling for running out of workspace.

[...]

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


From denis at zeno.org  Wed Jul 11 22:28:30 2007
From: denis at zeno.org (Denis Ahrens)
Date: Thu, 12 Jul 2007 00:28:30 +0200
Subject: logoutput from r1667
Message-ID: <16367464-71EA-4233-9348-241363049866@zeno.org>

Hi

here a logoutput from r1667 with default VCL:

(look at the age header and the http at the end)

    12 SessionOpen  c 66.249.65.83 36289
    12 Debug        c /Kategorien/0/Suche?&o=i#212.202.251.212#
    12 ReqStart     c 66.249.65.83 36289 1350902493
    12 RxRequest    c GET
    12 RxURL        c /Kategorien/0/Suche?&o=i
    12 RxProtocol   c HTTP/1.1
    12 RxHeader     c Host: 212.202.251.212
    12 RxHeader     c Connection: Keep-alive
    12 RxHeader     c User-Agent: Mediapartners-Google/2.1
    12 RxHeader     c Accept-Encoding: gzip
    12 VCL_call     c recv lookup
    12 VCL_call     c hash hash
    12 VCL_call     c miss fetch
    12 Backend      c 13 default
    12 ObjProtocol  c HTTP/1.1
    12 ObjStatus    c 200
    12 ObjResponse  c OK
    12 ObjHeader    c Server: ZenoServer/1.6.10 * DBSERVER/r441 *  
Middleware/277
    12 ObjHeader    c Vary: Accept-Encoding
    12 ObjHeader    c Date: Wed, 11 Jul 2007 22:16:47 GMT
    12 ObjHeader    c Content-Encoding: gzip
    12 ObjHeader    c Content-Type: text/html
    12 ObjHeader    c Cache-Control: s-maxage=300, max-age=120
    12 ObjHeader    c Last-Modified: Tue, 10 Jul 2007 15:02:36 GMT
    12 TTL          c 1350902493 RFC 300 1184192207 1184192207 0 300 0
    12 VCL_call     c fetch insert
    12 Length       c 1749
    12 VCL_call     c deliver deliver
    12 TxProtocol   c HTTP/1.1
    12 TxStatus     c 200
    12 TxResponse   c OK
    12 TxHeader     c Server: ZenoServer/1.6.10 * DBSERVER/r441 *  
Middleware/277
    12 TxHeader     c Vary: Accept-Encoding
    12 TxHeader     c Date: Wed, 11 Jul 2007 22:16:47 GMT
    12 TxHeader     c Content-Encoding: gzip
    12 TxHeader     c Content-Type: text/html
    12 TxHeader     c Cache-Control: s-maxage=300, max-age=120
    12 TxHeader     c Last-Modified: Tue, 10 Jul 2007 15:02:36 GMT
    12 TxHeader     c Content-Length: 1749
    12 TxHeader     c X-Varnish: 1350902493
    12 TxHeader     c Age: 3110775089
    12 TxHeader     c Via: 1.1 varnish
    12 ReqEnd       c 1350902493 1184192207.808972096  
1184192207.820660509 0.001130623 0.011668019 0.000020394

    11 SessionOpen  c 66.249.65.83 36244
    11 HttpError    c Received errno 35
    11 SessionClose c silent
    11 ReqEnd       c 0 1184192211.804295053 1184192211.804295053  
5.047401098 0.000000000 0.000000000

Host is a FreeBSD 6.2p3 system



From phk at phk.freebsd.dk  Wed Jul 11 22:30:45 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 11 Jul 2007 22:30:45 +0000
Subject: logoutput from r1667 
In-Reply-To: Your message of "Thu, 12 Jul 2007 00:28:30 +0200."
	<16367464-71EA-4233-9348-241363049866@zeno.org> 
Message-ID: <60299.1184193045@critter.freebsd.dk>

In message <16367464-71EA-4233-9348-241363049866 at zeno.org>, Denis Ahrens writes
:
>Hi
>
>here a logoutput from r1667 with default VCL:

Can I get you to check your clock relative to the backend ?


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From denis at zeno.org  Wed Jul 11 22:36:45 2007
From: denis at zeno.org (Denis Ahrens)
Date: Thu, 12 Jul 2007 00:36:45 +0200
Subject: logoutput from r1667 
In-Reply-To: <60299.1184193045@critter.freebsd.dk>
References: <60299.1184193045@critter.freebsd.dk>
Message-ID: <CD39EDBC-EB43-4DD3-B938-DC157CE2159A@zeno.org>


On 12.07.2007, at 00:30, Poul-Henning Kamp wrote:

> In message <16367464-71EA-4233-9348-241363049866 at zeno.org>, Denis  
> Ahrens writes
> :
>> Hi
>>
>> here a logoutput from r1667 with default VCL:
>
> Can I get you to check your clock relative to the backend ?

It is in sync with varnish +-1 second

Denis



From denis at zeno.org  Thu Jul 12 00:53:53 2007
From: denis at zeno.org (Denis Ahrens)
Date: Thu, 12 Jul 2007 02:53:53 +0200
Subject: logoutput from r1667 
In-Reply-To: <60299.1184193045@critter.freebsd.dk>
References: <60299.1184193045@critter.freebsd.dk>
Message-ID: <1CD232AD-B501-49DA-8B3A-7ED50E0A5C37@zeno.org>


On 12.07.2007, at 00:30, Poul-Henning Kamp wrote:

> In message <16367464-71EA-4233-9348-241363049866 at zeno.org>, Denis  
> Ahrens writes
> :
>> Hi
>>
>> here a logoutput from r1667 with default VCL:
>
> Can I get you to check your clock relative to the backend ?

Ok, I (ok, actually it was saite) made a patch to show
the actual problem:

before changeset r1639

RES_WriteObj() was responsible for setting t_resp and the output of  
the "Age:" header

after changeset r1639

RES_BuildHTTP() sometimes uses sp->t_resp uninitialized.

Index: bin/varnishd/cache_response.c
===================================================================
--- bin/varnishd/cache_response.c	(revision 1667)
+++ bin/varnishd/cache_response.c	(working copy)
@@ -134,6 +134,11 @@
  	else
  		http_PrintfHeader(sp->wrk, sp->fd, sp->http,
  		    "X-Varnish: %u", sp->xid);
+	if (!sp->t_resp.tv_sec)
+	{
+		http_PrintfHeader(sp->wrk, sp->fd, sp->http, "DEBUG: reorder  
problem occured: sp->t_resp.tv_sec is 0");
+		clock_gettime(CLOCK_REALTIME, &sp->t_resp);
+	}
  	http_PrintfHeader(sp->wrk, sp->fd, sp->http, "Age: %u",
  	    sp->obj->age + sp->t_resp.tv_sec - sp->obj->entered);
  	http_SetHeader(sp->wrk, sp->fd, sp->http, "Via: 1.1 varnish");

Denis



From phk at phk.freebsd.dk  Fri Jul 13 07:58:49 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 13 Jul 2007 07:58:49 +0000
Subject: logoutput from r1667 
In-Reply-To: Your message of "Thu, 12 Jul 2007 02:53:53 +0200."
	<1CD232AD-B501-49DA-8B3A-7ED50E0A5C37@zeno.org> 
Message-ID: <79325.1184313529@critter.freebsd.dk>


This should be fixed now.

Poul-Henning

In message <1CD232AD-B501-49DA-8B3A-7ED50E0A5C37 at zeno.org>, Denis Ahrens writes
:
>
>On 12.07.2007, at 00:30, Poul-Henning Kamp wrote:
>
>> In message <16367464-71EA-4233-9348-241363049866 at zeno.org>, Denis  
>> Ahrens writes
>> :
>>> Hi
>>>
>>> here a logoutput from r1667 with default VCL:
>>
>> Can I get you to check your clock relative to the backend ?
>
>Ok, I (ok, actually it was saite) made a patch to show
>the actual problem:
>
>before changeset r1639
>
>RES_WriteObj() was responsible for setting t_resp and the output of  
>the "Age:" header
>
>after changeset r1639
>
>RES_BuildHTTP() sometimes uses sp->t_resp uninitialized.
>
>Index: bin/varnishd/cache_response.c
>===================================================================
>--- bin/varnishd/cache_response.c	(revision 1667)
>+++ bin/varnishd/cache_response.c	(working copy)
>@@ -134,6 +134,11 @@
>  	else
>  		http_PrintfHeader(sp->wrk, sp->fd, sp->http,
>  		    "X-Varnish: %u", sp->xid);
>+	if (!sp->t_resp.tv_sec)
>+	{
>+		http_PrintfHeader(sp->wrk, sp->fd, sp->http, "DEBUG: reorder  
>problem occured: sp->t_resp.tv_sec is 0");
>+		clock_gettime(CLOCK_REALTIME, &sp->t_resp);
>+	}
>  	http_PrintfHeader(sp->wrk, sp->fd, sp->http, "Age: %u",
>  	    sp->obj->age + sp->t_resp.tv_sec - sp->obj->entered);
>  	http_SetHeader(sp->wrk, sp->fd, sp->http, "Via: 1.1 varnish");
>
>Denis
>
>_______________________________________________
>varnish-misc mailing list
>varnish-misc at projects.linpro.no
>http://projects.linpro.no/mailman/listinfo/varnish-misc
>

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From jean-marc.pouchoulon at ac-montpellier.fr  Fri Jul 13 09:55:59 2007
From: jean-marc.pouchoulon at ac-montpellier.fr (jean-marc pouchoulon)
Date: Fri, 13 Jul 2007 11:55:59 +0200
Subject: varnishnsca segfault
Message-ID: <46974C2F.1080508@ac-montpellier.fr>

Bonjour,

varnishnsca failed with a segfault as I can see with a strace

nanosleep({0, 50000000}, NULL)          = 0
nanosleep({0, 50000000}, NULL)          = 0
nanosleep({0, 50000000}, NULL)          = 0
nanosleep({0, 50000000}, NULL)          = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
nanosleep({0, 50000000}, NULL)          = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
write(1, "DA-EB419BACFB4C}; (R1 1.5); .NET"..., 1024) = 1024
nanosleep({0, 50000000}, NULL)          = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
nanosleep({0, 50000000}, NULL)          = 0
nanosleep({0, 50000000}, NULL)          = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1082, ...}) = 0
nanosleep({0, 50000000}, NULL)          = 0
nanosleep({0, 50000000}, NULL)          = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---

I enabled core dump and I'm waiting another crash

jean-marc





From des at linpro.no  Fri Jul 13 11:38:10 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Fri, 13 Jul 2007 13:38:10 +0200
Subject: varnishnsca segfault
In-Reply-To: <46974C2F.1080508@ac-montpellier.fr> (jean-marc pouchoulon's
	message of "Fri\, 13 Jul 2007 11\:55\:59 +0200")
References: <46974C2F.1080508@ac-montpellier.fr>
Message-ID: <87tzs8a5zx.fsf@des.linpro.no>

jean-marc pouchoulon <jean-marc.pouchoulon at ac-montpellier.fr> writes:
> varnishnsca failed with a segfault as I can see with a strace

This is a known bug in 1.0.4.  Apply r1531 and rebuild.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ric at digitalmarbles.com  Sat Jul 14 10:27:20 2007
From: ric at digitalmarbles.com (Ricardo Newbery)
Date: Sat, 14 Jul 2007 03:27:20 -0700
Subject: Trying to test Vary... but child dies.
Message-ID: <FCDC56A9-D06B-4AAD-8DF9-54F8DE65A0CB@digitalmarbles.com>


I'm trying to test the Vary support.

Tried both the svn trunk and the 1.1 svn branch.  Are these stable  
enough right now to test?

Soon after starting up with...

   varnishd -a x.x.x.x:9079 -b x.x.x.x:9080 -s file,/opt/varnish-1.1/ 
storage/storage.bin -d -d

and visiting one page, then refreshing the page, yields the following  
on the console...

_________________

NB: Limiting size to 2GB on 32 bit architecture to prevent running  
out of
address space.  Specifiy explicit size to override.
file /opt/varnish-1.1/storage/storage.bin size 2147479552 bytes  
(2097148 fs-blocks, 524287 pages)
Using old SHMFILE
rolling(1)...
rolling(2)...
start
CLI <start>
start child pid 22521
200 0

Child said (2, 22521): <<Child starts
managed to mmap 2147479552 bytes of 2147479552
Ready
CLI ready
 >>
Child said (2, 22521): <<socktest: linger=0 sndtimeo=0 rcvtimeo=0
 >>
Child said (2, 22521): <<Assert error in hcl_deref(), hash_classic.c  
line 210:
   Condition(he->hash < hcl_nhash) not true.
   errno = 0 (Success)
 >>
Cache child died pid=22521 status=0x6
Clean child
Child cleaned
______________________

The child dies and never restarts.

Any suggestions?

Ric




From addi at addihetja.com  Mon Jul 16 18:45:10 2007
From: addi at addihetja.com (=?ISO-8859-1?Q?Arn=F3r_Kristj=E1nsson?=)
Date: Mon, 16 Jul 2007 18:45:10 +0000
Subject: Mac and regular expressions
Message-ID: <AF5757F3-AD0D-4C1E-8042-C558605D9113@addihetja.com>


>> and won't unless someone donates a Mac to the core developers?
> I'd rather say "until the core developers get their hands on a Mac",  
> which may happen in a variety of ways. Linpro is an Apple partner,  
> so we may be able to get a good price.

Would remote root access to a Mac OS X Server box help?

A.
>> and won't unless someone donates a Mac to the core developers?
> I'd rather say "until the core developers get their hands on a Mac",  
> which may happen in a variety of ways. Linpro is an Apple partner,  
> so we may be able to get a good price.

Would remote root access to a Mac OS X Server box help?

A.



From des at linpro.no  Mon Jul 16 13:08:40 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 16 Jul 2007 15:08:40 +0200
Subject: Mac and regular expressions
In-Reply-To: <AF5757F3-AD0D-4C1E-8042-C558605D9113@addihetja.com>
	(=?utf-8?Q?=22Arn=C3=B3r=09Kristj=C3=A1nsson=22's?= message of "Mon\, 16
	Jul 2007 18\:45\:10 +0000")
References: <AF5757F3-AD0D-4C1E-8042-C558605D9113@addihetja.com>
Message-ID: <87myxw7axz.fsf@des.linpro.no>

Arn?r Kristj?nsson <addi at addihetja.com> writes:
> Would remote root access to a Mac OS X Server box help?

It would help in the short run; I could at least make sure 1.1 runs on
MacOS X.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From denis at zeno.org  Tue Jul 17 14:18:22 2007
From: denis at zeno.org (Denis Ahrens)
Date: Tue, 17 Jul 2007 16:18:22 +0200
Subject: 304 and header
Message-ID: <4EB9A31B-FF7D-4080-9A31-3403828173CE@zeno.org>

Hi

is it correct that varnish omit some headers when
the answer is a 304?

Denis



From phk at phk.freebsd.dk  Thu Jul 19 07:39:17 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 19 Jul 2007 07:39:17 +0000
Subject: 304 and header 
In-Reply-To: Your message of "Tue, 17 Jul 2007 16:18:22 +0200."
	<4EB9A31B-FF7D-4080-9A31-3403828173CE@zeno.org> 
Message-ID: <1126.1184830757@critter.freebsd.dk>

In message <4EB9A31B-FF7D-4080-9A31-3403828173CE at zeno.org>, Denis Ahrens writes
:
>Hi
>
>is it correct that varnish omit some headers when
>the answer is a 304?

Yes, the 304 is entirely synethetic.

Is that giving you problems ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From denis at zeno.org  Thu Jul 19 15:20:14 2007
From: denis at zeno.org (Denis Ahrens)
Date: Thu, 19 Jul 2007 17:20:14 +0200
Subject: 304 and header 
In-Reply-To: <1126.1184830757@critter.freebsd.dk>
References: <1126.1184830757@critter.freebsd.dk>
Message-ID: <3DC82EB6-CFB9-4FAB-8FA8-84CB5702BF76@zeno.org>


On 19.07.2007, at 09:39, Poul-Henning Kamp wrote:

> In message <4EB9A31B-FF7D-4080-9A31-3403828173CE at zeno.org>, Denis  
> Ahrens writes
> :
>> Hi
>>
>> is it correct that varnish omit some headers when
>> the answer is a 304?
>
> Yes, the 304 is entirely synethetic.
>
> Is that giving you problems ?

We have changing max-age values. but these
do not propagate through when varnish
delivers 304.

Is it possible to fix this myself with VCL?

Denis



From havard.futseter at met.no  Thu Jul 19 15:29:16 2007
From: havard.futseter at met.no (=?UTF-8?B?SMOldmFyZCBGdXRzw6Z0ZXI=?=)
Date: Thu, 19 Jul 2007 17:29:16 +0200
Subject: child process restarts frequently
In-Reply-To: <877ipc9hg9.fsf@des.linpro.no>
References: <468B7A9A.3040706@met.no>
	<87myybdqj8.fsf@des.linpro.no>	<468DF2D9.3090607@met.no>
	<877ipc9hg9.fsf@des.linpro.no>
Message-ID: <469F834C.1080806@met.no>

Dag-Erling Sm?rgrav wrote:
> H?vard Futs?ter <havard.futseter at met.no> writes:
>> Might this perhaps be an indication that Varnish gets into trouble
>> when it receives a request for a resource while the cached object for
>> this resource is beeing recycled?
> 
> Sounds like it.  Poul-Henning, could you take a closer look at the
> locking in the expiry code?
> 
> DES

Hi. I have some more information regarding the above mentioned problem:

We now think the restart is caused by a conflict between the purging 
routine and the timeout routine.

We have the following scenario:

1. url.purge <regexp>.
2. An object that will be affected by the purge times out and is discarded.
3. AFTER ExpPick, but BEFORE ExpKill, there is a request for the same 
object. The object is consequently banned.
4. Immediately after this ban the child process restarts.

Perhaps the expire-system tries to move the object to deathrow twice? 
Speculation only...

For testing purposes we have temporarily removed the url.purge command, 
and have seen no restart after this was done.


-- 
Mvh,
H?vard Futs?ter
Telefon: +47 22 96 32 69
Mobil:   +47 99 69 61 84


From chulmin2 at hotmail.com  Fri Jul 20 07:41:21 2007
From: chulmin2 at hotmail.com (Monty Ree)
Date: Fri, 20 Jul 2007 07:41:21 +0000
Subject: Where is varnish manual or documents?
Message-ID: <BAY101-F139C1392254ECA42FCE54285F40@phx.gbl>

Hello, list.

I'm so sorry that I'm a begineer at varnish.

and I would like to study this varnish for the best performance of reverse 
cache.
As I know, varnish is the best peformance for the reverse proxy, right?

But I can't find any manual or documents about vanish at 
http://varnish.projects.linpro.no/.
Please let me know any documents about varnish.


So thanks for your help..

_________________________________________________________________
????? ??? ?? ????? http://phonebuddy.msn.co.kr/ 



From thomas.westlund at aftenposten.no  Mon Jul 23 12:20:20 2007
From: thomas.westlund at aftenposten.no (Thomas Westlund)
Date: Mon, 23 Jul 2007 14:20:20 +0200
Subject: Problems with varnish 1.1 and obj.status
In-Reply-To: <469F834C.1080806@met.no>
References: <468B7A9A.3040706@met.no> <87myybdqj8.fsf@des.linpro.no>
	<468DF2D9.3090607@met.no> <877ipc9hg9.fsf@des.linpro.no>
	<469F834C.1080806@met.no>
Message-ID: <20070723122020.GA4435@aftenposten.no>

Hi,

I've just uppgraded to varnish 1.1

The upgrade worked just fine and varnish started with my old vcl-file without problems.

However when I added the following to sub vcl_fetch {}

if (obj.status == 404) {
	pass;
} else {
	set obj.ttl = 900s;
insert;
}

Varnish fails at restart with the following response:

Starting varnishd.
Problem loading compiled VCL program:
./bin.U5BscB9E: Undefined symbol "VRT_r_obj_status"

If I remove the "if (obj.status == 404)" statement everything works just fine.

--
Thomas Westlund


From kradziszewski at gmail.com  Mon Jul 23 12:57:42 2007
From: kradziszewski at gmail.com (Kamil Radziszewski)
Date: Mon, 23 Jul 2007 14:57:42 +0200
Subject: Varnish - configuration
Message-ID: <231965b00707230557x3154200t4a63ae07d69f5b7c@mail.gmail.com>

How to configure varnish to cache files from website that is on anoter IP
adress as the varnish ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070723/72a31e42/attachment.html>

From denis at startsiden.no  Mon Jul 23 14:44:41 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Mon, 23 Jul 2007 16:44:41 +0200 (CEST)
Subject: Varnish - configuration
In-Reply-To: <231965b00707230557x3154200t4a63ae07d69f5b7c@mail.gmail.com>
Message-ID: <24671183.91321185201881346.JavaMail.root@ms1.startsiden.no>

From: Kamil Radziszewski <kradziszewski at gmail.com> 
>How to configure varnish to cache files from website that is on anoter IP adress as the varnish ? 

"man vcl" gives you info on the backend directive to use in your vcl config:

backend default {
   set backend.host = "www.example.com";
   set backend.port = "80";
}

You can also use more than one backend, but instead of quoting the entire manpage, it@?s better if you check it yourself really.

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS 
http://www.startsiden.no 



From kradziszewski at gmail.com  Mon Jul 23 17:48:21 2007
From: kradziszewski at gmail.com (Kamil Radziszewski)
Date: Mon, 23 Jul 2007 19:48:21 +0200
Subject: Varnish - configuration
References: 231965b00707230557x3154200t4a63ae07d69f5b7c@mail.gmail.com
Message-ID: <46A4E9E5.4030402@gmail.com>

Allrigth but i don't know how to set the path where the files should be 
cached....i don't know if the default configuration is correct for me ...


From ssm at linpro.no  Mon Jul 23 21:49:03 2007
From: ssm at linpro.no (Stig Sandbeck Mathisen)
Date: Mon, 23 Jul 2007 23:49:03 +0200
Subject: Varnish - configuration
In-Reply-To: <46A4E9E5.4030402@gmail.com> (Kamil Radziszewski's message of
	"Mon, 23 Jul 2007 19:48:21 +0200")
References: <46A4E9E5.4030402@gmail.com>
Message-ID: <7xr6myyenk.fsf@iostat.e.linpro.no>

Kamil Radziszewski <kradziszewski at gmail.com> writes:

> Allrigth but i don't know how to set the path where the files should be 
> cached....i don't know if the default configuration is correct for me ...

That's in the man page of "varnishd".

-- 
Stig Sandbeck Mathisen, Linpro


From chulmin2 at hotmail.com  Tue Jul 24 07:21:56 2007
From: chulmin2 at hotmail.com (Monty Ree)
Date: Tue, 24 Jul 2007 07:21:56 +0000
Subject: What to do best performance at varnish?
Message-ID: <BAY101-F39436FCC8E9A091206113085F00@phx.gbl>

Hello, list.

I would like to set the vernishd with best performance caching.
But I can't find any related documentation,(already saw man page) 
So is there any recommendable config to improve the performance of 
varnishd?
My system is linux(kernel 2.6, centos 4.x)

for example, 
-. which Cache file size is required?
-. Dual CPU and large RAM would be good to performance? 
-. which value is required VARNISH_MAX_THREADS, VARNISH_MIN_THREADS?
-. any sysctl tuning?


Thanks for your help in advance.

_________________________________________________________________
????? ??? ?? ????? http://phonebuddy.msn.co.kr/ 



From phk at phk.freebsd.dk  Tue Jul 24 07:26:27 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 24 Jul 2007 07:26:27 +0000
Subject: What to do best performance at varnish? 
In-Reply-To: Your message of "Tue, 24 Jul 2007 07:21:56 GMT."
	<BAY101-F39436FCC8E9A091206113085F00@phx.gbl> 
Message-ID: <21881.1185261987@critter.freebsd.dk>


We have very little experience with performance tuning yet, as very
few people need to do any, Varnish is very frugal with hardware
resources.

Make sure your cache file is big enough, making it too big won't cost
you anything, so don't worry about it.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Tue Jul 24 07:50:31 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 24 Jul 2007 07:50:31 +0000
Subject: Varnish - configuration 
In-Reply-To: Your message of "Mon, 23 Jul 2007 19:48:21 +0200."
	<46A4E9E5.4030402@gmail.com> 
Message-ID: <21957.1185263431@critter.freebsd.dk>

In message <46A4E9E5.4030402 at gmail.com>, Kamil Radziszewski writes:
>Allrigth but i don't know how to set the path where the files should be 
>cached....i don't know if the default configuration is correct for me ...

By default the cache file is created under /tmp

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Tue Jul 24 08:23:53 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 24 Jul 2007 10:23:53 +0200
Subject: Where is varnish manual or documents?
In-Reply-To: <BAY101-F139C1392254ECA42FCE54285F40@phx.gbl> (Monty Ree's
	message of "Fri\, 20 Jul 2007 07\:41\:21 +0000")
References: <BAY101-F139C1392254ECA42FCE54285F40@phx.gbl>
Message-ID: <87abtm8b1i.fsf@des.linpro.no>

"Monty Ree" <chulmin2 at hotmail.com> writes:
> As I know, varnish is the best peformance for the reverse proxy,
> right?

We think so, yes.

> But I can't find any manual or documents about vanish at
> http://varnish.projects.linpro.no/.  Please let me know any documents
> about varnish.

The software comes with man pages.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jul 24 08:35:35 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 24 Jul 2007 10:35:35 +0200
Subject: What to do best performance at varnish?
In-Reply-To: <BAY101-F39436FCC8E9A091206113085F00@phx.gbl> (Monty Ree's
	message of "Tue\, 24 Jul 2007 07\:21\:56 +0000")
References: <BAY101-F39436FCC8E9A091206113085F00@phx.gbl>
Message-ID: <873aze8ai0.fsf@des.linpro.no>

"Monty Ree" <chulmin2 at hotmail.com> writes:
> - which Cache file size is required?

At least as large as your working set.  Beyond that, you run into the
law of diminishing returns, unless you expect your working set to grow
over time.

> - Dual CPU and large RAM would be good to performance?

Yes.

> - which value is required VARNISH_MAX_THREADS, VARNISH_MIN_THREADS?

Leave them unchanged.  Keep an eye on n_wrk_overflow and n_wrk_drop in
varnishstat; if they increase rapidly under normal working conditions,
increase thread_pool_max.  If n_wrk_overflow increases but n_wrk_drop
doesn't, consider increasing thread_pool_min.  If your traffic comes in
bursts, increasing thread_pool_timeout might help, but in most cases it
won't.

> - any sysctl tuning?

No more than is generally recommended for the amount of memory you have
and the amount of network traffic you expect.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jul 24 08:38:18 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 24 Jul 2007 10:38:18 +0200
Subject: Problems with varnish 1.1 and obj.status
In-Reply-To: <20070723122020.GA4435@aftenposten.no> (Thomas Westlund's message
	of "Mon\, 23 Jul 2007 14\:20\:20 +0200")
References: <468B7A9A.3040706@met.no> <87myybdqj8.fsf@des.linpro.no>
	<468DF2D9.3090607@met.no> <877ipc9hg9.fsf@des.linpro.no>
	<469F834C.1080806@met.no> <20070723122020.GA4435@aftenposten.no>
Message-ID: <87y7h66vt1.fsf@des.linpro.no>

Thomas Westlund <thomas.westlund at aftenposten.no> writes:
> Varnish fails at restart with the following response:
>
> Starting varnishd.
> Problem loading compiled VCL program:
> ./bin.U5BscB9E: Undefined symbol "VRT_r_obj_status"
>
> If I remove the "if (obj.status == 404)" statement everything works just fine.

Looks like an oversight; resp.status is both readable and writable, but
obj.status is write-only.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From chulmin2 at hotmail.com  Tue Jul 24 08:38:11 2007
From: chulmin2 at hotmail.com (Monty Ree)
Date: Tue, 24 Jul 2007 08:38:11 +0000
Subject: curious ab result when -k option included?
Message-ID: <BAY101-F197A610061A0A42826F47F85F00@phx.gbl>

Hello, all.

When I execute ab(apache bentch) to test the varnish performance, I have 
found curious result.
If I set -k option which means Keepalive like below, I can't see the 
result. 

with -k option : Requests per second:    10.96 [#/sec] (mean)
without -k option : Requests per second:    5302 [#/sec] (mean)

ab -k -n 1000 -c 100 http://example.com/images/test.gif

Why this result happens? varnish doesn't support keepalive?


Thanks for your help.

_________________________________________________________________
????? ??? ?? ????? http://phonebuddy.msn.co.kr/ 



From des at linpro.no  Tue Jul 24 08:42:34 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 24 Jul 2007 10:42:34 +0200
Subject: Problems with varnish 1.1 and obj.status
In-Reply-To: <87y7h66vt1.fsf@des.linpro.no> ("Dag-Erling =?utf-8?Q?Sm?=
	=?utf-8?Q?=C3=B8rgrav=22's?= message
	of "Tue\, 24 Jul 2007 10\:38\:18 +0200")
References: <468B7A9A.3040706@met.no> <87myybdqj8.fsf@des.linpro.no>
	<468DF2D9.3090607@met.no> <877ipc9hg9.fsf@des.linpro.no>
	<469F834C.1080806@met.no> <20070723122020.GA4435@aftenposten.no>
	<87y7h66vt1.fsf@des.linpro.no>
Message-ID: <87r6my6vlx.fsf@des.linpro.no>

Dag-Erling Sm?rgrav <des at linpro.no> writes:
> Looks like an oversight; resp.status is both readable and writable, but
> obj.status is write-only.

Try the attached patch.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

-------------- next part --------------
A non-text attachment was scrubbed...
Name: VRT_r_obj_status.diff
Type: text/x-diff
Size: 501 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070724/de077354/attachment.bin>

From des at linpro.no  Tue Jul 24 08:55:14 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 24 Jul 2007 10:55:14 +0200
Subject: curious ab result when -k option included?
In-Reply-To: <BAY101-F197A610061A0A42826F47F85F00@phx.gbl> (Monty Ree's
	message of "Tue\, 24 Jul 2007 08\:38\:11 +0000")
References: <BAY101-F197A610061A0A42826F47F85F00@phx.gbl>
Message-ID: <87myxm6v0t.fsf@des.linpro.no>

"Monty Ree" <chulmin2 at hotmail.com> writes:
> When I execute ab(apache bentch) to test the varnish performance, I
> have found curious result.  If I set -k option which means Keepalive
> like below, I can't see the result.
>
> with -k option : Requests per second:    10.96 [#/sec] (mean)
> without -k option : Requests per second:    5302 [#/sec] (mean)
>
> ab -k -n 1000 -c 100 http://example.com/images/test.gif
>
> Why this result happens? varnish doesn't support keepalive?

I see the same symptom; tcpdump shows that ab does not send a new
request after receiving each answer, but waits for the session to time
out and then reconnects.  I'm not sure if it's a bug in ab or in
Varnish.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jul 24 09:11:31 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 24 Jul 2007 11:11:31 +0200
Subject: curious ab result when -k option included?
In-Reply-To: <87myxm6v0t.fsf@des.linpro.no> ("Dag-Erling =?utf-8?Q?Sm?=
	=?utf-8?Q?=C3=B8rgrav=22's?= message
	of "Tue\, 24 Jul 2007 10\:55\:14 +0200")
References: <BAY101-F197A610061A0A42826F47F85F00@phx.gbl>
	<87myxm6v0t.fsf@des.linpro.no>
Message-ID: <87ir8a6u9o.fsf@des.linpro.no>

Dag-Erling Sm?rgrav <des at linpro.no> writes:
> "Monty Ree" <chulmin2 at hotmail.com> writes:
> > Why this result happens? varnish doesn't support keepalive?
> I see the same symptom; tcpdump shows that ab does not send a new
> request after receiving each answer, but waits for the session to time
> out and then reconnects.  I'm not sure if it's a bug in ab or in
> Varnish.

Varnish always responds with HTTP 1.1 (even though ab sends HTTP 1.0
requests) and does not send "Connection: keep-alive" (since that is the
default in HTTP 1.1).

With the attached patch, I get the following results on my laptop:

Concurrency Level:      200
Time taken for tests:   46.386721 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    100000
Total transferred:      23420240072 bytes
HTML transferred:       23396793740 bytes
Requests per second:    2155.79 [#/sec] (mean)
Time per request:       92.773 [ms] (mean)
Time per request:       0.464 [ms] (mean, across all concurrent requests)
Transfer rate:          493057.66 [Kbytes/sec] received

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

-------------- next part --------------
A non-text attachment was scrubbed...
Name: varnish-keepalive.diff
Type: text/x-diff
Size: 1121 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070724/8f037800/attachment.bin>

From thomas.westlund at aftenposten.no  Tue Jul 24 09:35:39 2007
From: thomas.westlund at aftenposten.no (Thomas Westlund)
Date: Tue, 24 Jul 2007 11:35:39 +0200
Subject: Problems with varnish 1.1 and obj.status
In-Reply-To: <87r6my6vlx.fsf@des.linpro.no>
References: <468B7A9A.3040706@met.no> <87myybdqj8.fsf@des.linpro.no>
	<468DF2D9.3090607@met.no> <877ipc9hg9.fsf@des.linpro.no>
	<469F834C.1080806@met.no> <20070723122020.GA4435@aftenposten.no>
	<87y7h66vt1.fsf@des.linpro.no> <87r6my6vlx.fsf@des.linpro.no>
Message-ID: <20070724093539.GA4580@aftenposten.no>

Hi,


On Tue, Jul 24, 2007 at 10:42:34AM +0200, Dag-Erling Sm?rgrav wrote:
> Dag-Erling Sm?rgrav <des at linpro.no> writes:
> > Looks like an oversight; resp.status is both readable and writable, but
> > obj.status is write-only.
> 
> Try the attached patch.

The patch works, I'm now able to control the caching of 404's.

Thank you very much for the help.


-- 

Thomas Westlund
Aftenposten AS, UNIX/nettverksavd.
Postboks 1, 0051 Oslo
Tlf: +47 98 20 30 33
Fax: +47 22 86 40 74


From kradziszewski at gmail.com  Tue Jul 24 10:45:40 2007
From: kradziszewski at gmail.com (Kamil Radziszewski)
Date: Tue, 24 Jul 2007 12:45:40 +0200
Subject: Squid and Varnish
Message-ID: <231965b00707240345g6a03f83ve333c1fd15f1fa7b@mail.gmail.com>

Can somebody help me with translation squid config file on varnish config ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070724/a35ee834/attachment.html>

From phk at phk.freebsd.dk  Tue Jul 24 11:37:11 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 24 Jul 2007 11:37:11 +0000
Subject: Squid and Varnish 
In-Reply-To: Your message of "Tue, 24 Jul 2007 12:45:40 +0200."
	<231965b00707240345g6a03f83ve333c1fd15f1fa7b@mail.gmail.com> 
Message-ID: <22871.1185277031@critter.freebsd.dk>

In message <231965b00707240345g6a03f83ve333c1fd15f1fa7b at mail.gmail.com>, "Kamil
 Radziszewski" writes:

>Can somebody help me with translation squid config file on varnish config ?

There is no simple way to do that.

You need to know what your squid does, then you need to implement
that in Varnish.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From kradziszewski at gmail.com  Tue Jul 24 21:08:00 2007
From: kradziszewski at gmail.com (Kamil Radziszewski)
Date: Tue, 24 Jul 2007 23:08:00 +0200
Subject: Varnish - memorylimit
Message-ID: <46A66A30.1050104@gmail.com>

How can i set the memory limit for cache ?


From des at linpro.no  Wed Jul 25 07:31:41 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Wed, 25 Jul 2007 09:31:41 +0200
Subject: Varnish - memorylimit
In-Reply-To: <46A66A30.1050104@gmail.com> (Kamil Radziszewski's message of
	"Tue\, 24 Jul 2007 23\:08\:00 +0200")
References: <46A66A30.1050104@gmail.com>
Message-ID: <87y7h55482.fsf@des.linpro.no>

Kamil Radziszewski <kradziszewski at gmail.com> writes:
> How can i set the memory limit for cache ?

There is no memory limit.  The VM system will keep as much of the cache
file in memory as it sees fit.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From kradziszewski at gmail.com  Wed Jul 25 09:22:44 2007
From: kradziszewski at gmail.com (Kamil Radziszewski)
Date: Wed, 25 Jul 2007 11:22:44 +0200
Subject: cache_peer in varnish
Message-ID: <231965b00707250222t2bc8d0a7ub907dbe807f47df5@mail.gmail.com>

How can i replace the squid command "cache_peer" in varnish ??
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070725/3ee4613a/attachment.html>

From rlslog at gmail.com  Wed Jul 25 09:39:32 2007
From: rlslog at gmail.com (Releaselog | RLSLOG.net)
Date: Wed, 25 Jul 2007 11:39:32 +0200
Subject: varnish increased the load
Message-ID: <000301c7ce9f$b4af0780$c600a8c0@r00t>

hey,
I have a dedicated server running Fedora 6 and hosting a Wordpress based 
website with ~80 000 daily unique visitors, already using Wp-Cache plugin 
and eAccelerator. The load was increasing lately a lot because of growing 
number of visitors so I decided to try Varnish as it looks way more powerful 
than Squid and other alternatives. Installation went fine, all is running 
now, but the server load *increased* instead of decreasing. I didn't change 
any configuration, all is set to default (also because of lack of the 
documentation), but as I said, the server load increased at least twice. Do 
you have any idea what could be wrong and what should I check.

thanks

Martin




From perbu at linpro.no  Wed Jul 25 10:04:57 2007
From: perbu at linpro.no (Per Andreas Buer)
Date: Wed, 25 Jul 2007 12:04:57 +0200
Subject: cache_peer in varnish
In-Reply-To: <231965b00707250222t2bc8d0a7ub907dbe807f47df5@mail.gmail.com>
References: <231965b00707250222t2bc8d0a7ub907dbe807f47df5@mail.gmail.com>
Message-ID: <46A72049.9090603@linpro.no>

Kamil Radziszewski wrote:
> How can i replace the squid command "cache_peer" in varnish ??

No. There is no cache peering in Varnish - and most likely there won't
be. In most cases more or less just as fast to fetch the content from
the backend. ICP in Squid is first and foremost designed for use in
forward proxies behind a slow internet connection.


Per.


From des at linpro.no  Wed Jul 25 10:35:30 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Wed, 25 Jul 2007 12:35:30 +0200
Subject: varnish increased the load
In-Reply-To: <000301c7ce9f$b4af0780$c600a8c0@r00t> (rlslog@gmail.com's message
	of "Wed\, 25 Jul 2007 11\:39\:32 +0200")
References: <000301c7ce9f$b4af0780$c600a8c0@r00t>
Message-ID: <87ir886aa5.fsf@des.linpro.no>

"Releaselog | RLSLOG.net" <rlslog at gmail.com> writes:
> I have a dedicated server running Fedora 6 and hosting a Wordpress
> based website with ~80 000 daily unique visitors, already using
> Wp-Cache plugin and eAccelerator.  The load was increasing lately a
> lot because of growing number of visitors so I decided to try Varnish
> as it looks way more powerful than Squid and other alternatives.
> Installation went fine, all is running now, but the server load
> *increased* instead of decreasing. I didn't change any configuration,
> all is set to default (also because of lack of the documentation), but
> as I said, the server load increased at least twice. Do you have any
> idea what could be wrong and what should I check.

The load average is not a reliable indicator of the amount of work your
server is doing.  Depending on OS, pthread implementation and scheduler
strategy, you could have a very high load average on a nearly idle
machine.

Looking at your site, it seems that Varnish is working fine and caching
what it should.  There isn't much more I can say without additional
information (version, command-line parameters, VCL code etc.)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From phk at phk.freebsd.dk  Wed Jul 25 10:42:07 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 25 Jul 2007 10:42:07 +0000
Subject: varnish increased the load 
In-Reply-To: Your message of "Wed, 25 Jul 2007 11:39:32 +0200."
	<000301c7ce9f$b4af0780$c600a8c0@r00t> 
Message-ID: <27278.1185360127@critter.freebsd.dk>

In message <000301c7ce9f$b4af0780$c600a8c0 at r00t>, "Releaselog | RLSLOG.net" wri
tes:


Hi Martin,

You need to tell us what you mean by "load increased", which parameter
are you looking at ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From havard.futseter at met.no  Wed Jul 25 11:44:19 2007
From: havard.futseter at met.no (=?UTF-8?B?SMOldmFyZCBGdXRzw6Z0ZXI=?=)
Date: Wed, 25 Jul 2007 13:44:19 +0200
Subject: child process restarts frequently
In-Reply-To: <469F834C.1080806@met.no>
References: <468B7A9A.3040706@met.no>	<87myybdqj8.fsf@des.linpro.no>	<468DF2D9.3090607@met.no>	<877ipc9hg9.fsf@des.linpro.no>
	<469F834C.1080806@met.no>
Message-ID: <46A73793.2010206@met.no>

Hi. There have not been any replies to my previous mail (seen below), 
and I was wondering if this was because my description of the problem 
was unclear? Or perhaps I did  not supply enough data to find the source 
of the problem?

I would gladly have attached more log-data here, but I am afraid there 
aren't a whole lot more relevant logs available.

Best regards,
H?vard Futs?ter

H?vard Futs?ter wrote:
> Dag-Erling Sm?rgrav wrote:
>> H?vard Futs?ter <havard.futseter at met.no> writes:
>>> Might this perhaps be an indication that Varnish gets into trouble
>>> when it receives a request for a resource while the cached object for
>>> this resource is beeing recycled?
>> Sounds like it.  Poul-Henning, could you take a closer look at the
>> locking in the expiry code?
>>
>> DES
> 
> Hi. I have some more information regarding the above mentioned problem:
> 
> We now think the restart is caused by a conflict between the purging 
> routine and the timeout routine.
> 
> We have the following scenario:
> 
> 1. url.purge <regexp>.
> 2. An object that will be affected by the purge times out and is discarded.
> 3. AFTER ExpPick, but BEFORE ExpKill, there is a request for the same 
> object. The object is consequently banned.
> 4. Immediately after this ban the child process restarts.
> 
> Perhaps the expire-system tries to move the object to deathrow twice? 
> Speculation only...
> 
> For testing purposes we have temporarily removed the url.purge command, 
> and have seen no restart after this was done.
> 
> 


-- 
Mvh,
H?vard Futs?ter
Telefon: +47 22 96 32 69
Mobil:   +47 99 69 61 84


From des at linpro.no  Wed Jul 25 12:57:19 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Wed, 25 Jul 2007 14:57:19 +0200
Subject: child process restarts frequently
In-Reply-To: <46A73793.2010206@met.no> (=?utf-8?Q?=22H=C3=A5vard_Futs?=
	=?utf-8?Q?=C3=A6ter=22's?= message of "Wed\,
	25 Jul 2007 13\:44\:19 +0200")
References: <468B7A9A.3040706@met.no> <87myybdqj8.fsf@des.linpro.no>
	<468DF2D9.3090607@met.no> <877ipc9hg9.fsf@des.linpro.no>
	<469F834C.1080806@met.no> <46A73793.2010206@met.no>
Message-ID: <87ejiw63ps.fsf@des.linpro.no>

H?vard Futs?ter <havard.futseter at met.no> writes:
> Hi. There have not been any replies to my previous mail (seen below),
> and I was wondering if this was because my description of the problem
> was unclear? Or perhaps I did not supply enough data to find the
> source of the problem?

We know what causes it, but it isn't trivial to fix without degrading
performance too much.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From james at nyi.net  Wed Jul 25 18:14:27 2007
From: james at nyi.net (James Quacinella)
Date: Wed, 25 Jul 2007 14:14:27 -0400
Subject: Question about Pass
Message-ID: <46A79303.2090001@nyi.net>

Hello,

 From the man page: "vcl_pass: Called upon entering pass mode. In this 
mode, the request is passed on to the backend, and the backend?s 
response is passed on to the client, but is not entered into the cache. 
Subsequent requests submitted over the same client connection are 
handled normally."

So when running varnishstat, what does 'Cache hits for pass' mean? Sicne 
the above says that pass mode does not enter anything in the cache, but 
it can still use the cache?

thanks for any insight,
james


From andrearo at pvv.ntnu.no  Wed Jul 25 19:11:52 2007
From: andrearo at pvv.ntnu.no (=?ISO-8859-1?Q?Andreas_R=F8sdal?=)
Date: Wed, 25 Jul 2007 21:11:52 +0200 (CEST)
Subject: cache_peer in varnish
In-Reply-To: <46A72049.9090603@linpro.no>
References: <231965b00707250222t2bc8d0a7ub907dbe807f47df5@mail.gmail.com>
	<46A72049.9090603@linpro.no>
Message-ID: <20070725205709.J75759@verden.pvv.ntnu.no>

On Wed, 25 Jul 2007, Per Andreas Buer wrote:
> Kamil Radziszewski wrote:
>> How can i replace the squid command "cache_peer" in varnish ??
>
> No. There is no cache peering in Varnish - and most likely there won't
> be. In most cases more or less just as fast to fetch the content from
> the backend.

Why, do you have any data which indicates that fetching content from the 
backend is as fast as fetching it from a peer? Some backends can take 
several seconds to respond, if they are under heavy load and have to hit 
a database. Fetching data from a cache peer shouldn't take many 
milliseconds. Of course this could vary from system to system...

> ICP in Squid is first and foremost designed for use in
> forward proxies behind a slow internet connection.

What about the hypertext caching protocol, which was designed to replace 
ICP?

  - Andreas


From kradziszewski at gmail.com  Wed Jul 25 19:35:59 2007
From: kradziszewski at gmail.com (Kamil Radziszewski)
Date: Wed, 25 Jul 2007 21:35:59 +0200
Subject: Allways "Not Found"
Message-ID: <46A7A61F.3060103@gmail.com>

I was rnnin my varnish and i think it works but one problem: why i have 
allways
TxStatus     404
TxResponse   Not Fond
??????

That mean the varnish doesn't cache anythink :/


Any ideas ?


From ric at digitalmarbles.com  Wed Jul 25 19:38:49 2007
From: ric at digitalmarbles.com (Ricardo Newbery)
Date: Wed, 25 Jul 2007 12:38:49 -0700
Subject: Conditional GETs
Message-ID: <64E001C5-7D33-42F8-A41E-13EF0EED84F7@digitalmarbles.com>


I assume Varnish supports ETags and conditional GETs "If-Modified- 
Since" (IMS) and "If-None-Match" (INM)?

I'm interested in the heuristic applied.

Are IMS/INM requests relayed to the backend before the content is  
served out of cache?

Are distinct copies of each ETagged page saved in cache or are  
previous versions retained unless purged?  Is it possible to purge a  
single ETagged variant or does the entire set need to be purged?

Do the Cache-Control headers moderating conditional requests ("must- 
revalidate",  "proxy-revalidate", and "no-cache") modify this heuristic?

Ric




From phk at phk.freebsd.dk  Wed Jul 25 19:51:47 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 25 Jul 2007 19:51:47 +0000
Subject: cache_peer in varnish 
In-Reply-To: Your message of "Wed, 25 Jul 2007 21:11:52 +0200."
	<20070725205709.J75759@verden.pvv.ntnu.no> 
Message-ID: <32715.1185393107@critter.freebsd.dk>

In message <20070725205709.J75759 at verden.pvv.ntnu.no>, =?ISO-8859-1?Q?Andreas_R
=F8sdal?= writes:
>On Wed, 25 Jul 2007, Per Andreas Buer wrote:
>> Kamil Radziszewski wrote:
>>> How can i replace the squid command "cache_peer" in varnish ??
>>
>> No. There is no cache peering in Varnish - and most likely there won't
>> be. In most cases more or less just as fast to fetch the content from
>> the backend.
>
>Why, do you have any data which indicates that fetching content from the 
>backend is as fast as fetching it from a peer? Some backends can take 
>several seconds to respond, if they are under heavy load and have to hit 
>a database. Fetching data from a cache peer shouldn't take many 
>milliseconds. Of course this could vary from system to system...

Absent any evidence to the contrary, our current thinking is that
it is a better idea to have a two-layer varnish setup than a peer
to peer protocol.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From chulmin2 at hotmail.com  Thu Jul 26 07:34:31 2007
From: chulmin2 at hotmail.com (Monty Ree)
Date: Thu, 26 Jul 2007 07:34:31 +0000
Subject: how to logout from management interface?
Message-ID: <BAY101-F17A6D8858FCDA0A51110EB85F20@phx.gbl>

Hello, list.

I'm a newbie at varnishd and some questions.

after I entered management interface using telnet,
how to logout from the management interface? 
just ctrl + ] ??

  
Thanks for your time.

_________________________________________________________________
????? ???? ? ?? ???? MSN Hotmail? ?? ???.    
http://www.hotmail.com/ 



From phk at phk.freebsd.dk  Thu Jul 26 07:52:39 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 26 Jul 2007 07:52:39 +0000
Subject: how to logout from management interface? 
In-Reply-To: Your message of "Thu, 26 Jul 2007 07:34:31 GMT."
	<BAY101-F17A6D8858FCDA0A51110EB85F20@phx.gbl> 
Message-ID: <35061.1185436359@critter.freebsd.dk>



If you use telnet:

	CTRL-]
	close


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From rlslog at gmail.com  Wed Jul 25 15:15:46 2007
From: rlslog at gmail.com (Releaselog | RLSLOG.net)
Date: Wed, 25 Jul 2007 17:15:46 +0200
Subject: varnish increased the load 
References: <27278.1185360127@critter.freebsd.dk>
Message-ID: <001601c7cece$ad1c5b10$c600a8c0@r00t>

By load increased, I mean the load which is returned by server after 
"uptime" command. It was previously at values about 1-3 out of the peak and 
now it stays well above 6 most of the time, getting to something as high as 
20 or 40 in the peak time (evenings). Overally, I think it's MySQL database 
which is overloading, but there aren't many things in my mind which could 
help the situation apart from buying next (expensive) server if you say 
Varnish works fine at my box...

Martin

----- Original Message ----- 
From: "Poul-Henning Kamp" <phk at phk.freebsd.dk>
To: "Releaselog | RLSLOG.net" <rlslog at gmail.com>
Cc: <varnish-misc at projects.linpro.no>
Sent: Wednesday, July 25, 2007 12:42 PM
Subject: Re: varnish increased the load


> In message <000301c7ce9f$b4af0780$c600a8c0 at r00t>, "Releaselog | 
> RLSLOG.net" wri
> tes:
>
>
> Hi Martin,
>
> You need to tell us what you mean by "load increased", which parameter
> are you looking at ?
>
> -- 
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by 
> incompetence. 



From des at linpro.no  Thu Jul 26 08:14:27 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 26 Jul 2007 10:14:27 +0200
Subject: Question about Pass
In-Reply-To: <46A79303.2090001@nyi.net> (James Quacinella's message of "Wed\, 
	25 Jul 2007 14\:14\:27 -0400")
References: <46A79303.2090001@nyi.net>
Message-ID: <873azb60po.fsf@des.linpro.no>

James Quacinella <james at nyi.net> writes:
> From the man page: "vcl_pass: Called upon entering pass mode. In this
> mode, the request is passed on to the backend, and the backend's
> response is passed on to the client, but is not entered into the
> cache.  Subsequent requests submitted over the same client connection
> are handled normally."
>
> So when running varnishstat, what does 'Cache hits for pass' mean?
> Sicne the above says that pass mode does not enter anything in the
> cache, but it can still use the cache?

It means we found a record in the cache that indicates that a particular
object should be handled in pass mode.  This can happen if vcl_fetch()
ended with "pass", e.g. if the backend sent a cookie.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Jul 26 08:15:50 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 26 Jul 2007 10:15:50 +0200
Subject: Allways "Not Found"
In-Reply-To: <46A7A61F.3060103@gmail.com> (Kamil Radziszewski's message of
	"Wed\, 25 Jul 2007 21\:35\:59 +0200")
References: <46A7A61F.3060103@gmail.com>
Message-ID: <87y7h34m2x.fsf@des.linpro.no>

Kamil Radziszewski <kradziszewski at gmail.com> writes:
> I was rnnin my varnish and i think it works but one problem: why i have
> allways
> TxStatus     404
> TxResponse   Not Fond
> ??????

Unfortunately, you have not provided any information whatsoever that
could help us debug your problem.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Jul 26 08:17:51 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 26 Jul 2007 10:17:51 +0200
Subject: Conditional GETs
In-Reply-To: <64E001C5-7D33-42F8-A41E-13EF0EED84F7@digitalmarbles.com>
	(Ricardo Newbery's message of "Wed\, 25 Jul 2007 12\:38\:49 -0700")
References: <64E001C5-7D33-42F8-A41E-13EF0EED84F7@digitalmarbles.com>
Message-ID: <87tzrr4lzk.fsf@des.linpro.no>

Ricardo Newbery <ric at digitalmarbles.com> writes:
> I assume Varnish supports ETags and conditional GETs "If-Modified-
> Since" (IMS) and "If-None-Match" (INM)?

Varnish supports IMS but does not know about ETags or INM.

> Do the Cache-Control headers moderating conditional requests ("must-
> revalidate", "proxy-revalidate", and "no-cache") modify this
> heuristic?

In its default configuration, Varnish ignores Cache-Control directives
from clients.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Jul 26 08:23:06 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 26 Jul 2007 10:23:06 +0200
Subject: varnish increased the load
In-Reply-To: <001601c7cece$ad1c5b10$c600a8c0@r00t> (rlslog@gmail.com's message
	of "Wed\, 25 Jul 2007 17\:15\:46 +0200")
References: <27278.1185360127@critter.freebsd.dk>
	<001601c7cece$ad1c5b10$c600a8c0@r00t>
Message-ID: <87ps2f4lqt.fsf@des.linpro.no>

"Releaselog | RLSLOG.net" <rlslog at gmail.com> writes:
> By load increased, I mean the load which is returned by server after
> "uptime" command. It was previously at values about 1-3 out of the
> peak and now it stays well above 6 most of the time, getting to
> something as high as 20 or 40 in the peak time (evenings). Overally, I
> think it's MySQL database which is overloading, but there aren't many
> things in my mind which could help the situation apart from buying
> next (expensive) server if you say Varnish works fine at my box...

You provide very little information, and you don't seem to have made any
attempt to troubleshoot the problem yourself.  All I can say is that
from the looks of the HTTP headers I get from your site, Varnish seems
to cache most of it for 120 seconds.  I have no idea what you used
before Varnish, nor how it was configured, so I can't tell if that's an
improvement.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From Kenneth.Rorvik at hio.no  Thu Jul 26 08:33:41 2007
From: Kenneth.Rorvik at hio.no (=?ISO-8859-1?Q?Kenneth_R=F8rvik?=)
Date: Thu, 26 Jul 2007 10:33:41 +0200
Subject: varnish increased the load
In-Reply-To: <001601c7cece$ad1c5b10$c600a8c0@r00t>
References: <27278.1185360127@critter.freebsd.dk>
	<001601c7cece$ad1c5b10$c600a8c0@r00t>
Message-ID: <46A85C65.6070801@hio.no>

Releaselog | RLSLOG.net wrote:
> By load increased, I mean the load which is returned by server after 
> "uptime" command. It was previously at values about 1-3 out of the peak and 
> now it stays well above 6 most of the time, getting to something as high as 
> 20 or 40 in the peak time (evenings). Overally, I think it's MySQL database 
> which is overloading, but there aren't many things in my mind which could 
> help the situation apart from buying next (expensive) server if you say 
> Varnish works fine at my box...

Keep in mind you are adding processes to the system, which will mostly 
be running - this will probably increase your load. Grossly simplified, 
the load average can be thought of as the number of processes *waiting* 
for execution. Not how much work your CPUs are actually doing.

Important question: Are your "customers" happy? ;)

Other than that, first of all, move the SQL server to a separate box 
with nice I/O and memory. MySQL can really KILL a webserver if it is on 
the same box as apache.

-- 
Kenneth R?rvik, IT HiO
Tlf 22 45 20 83
Kenneth.Rorvik at hio.no


From phk at phk.freebsd.dk  Thu Jul 26 08:32:09 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 26 Jul 2007 08:32:09 +0000
Subject: varnish increased the load 
In-Reply-To: Your message of "Wed, 25 Jul 2007 17:15:46 +0200."
	<001601c7cece$ad1c5b10$c600a8c0@r00t> 
Message-ID: <35292.1185438729@critter.freebsd.dk>

In message <001601c7cece$ad1c5b10$c600a8c0 at r00t>, "Releaselog | RLSLOG.net" wri
tes:

>By load increased, I mean the load which is returned by server after 
>"uptime" command. It was previously at values about 1-3 out of the peak and 
>now it stays well above 6 most of the time, getting to something as high as 
>20 or 40 in the peak time (evenings). Overally, I think it's MySQL database 
>which is overloading, but there aren't many things in my mind which could 
>help the situation apart from buying next (expensive) server if you say 
>Varnish works fine at my box...

It's kind of like a patient saying "Doctor it hurts", doctors really
hate that, at the bare minimum they'd like to know _where_ it hurts
and what you did before it started hurting :-)

Here are some suggestions:

What is your hit-rate in varnish (use varnishstat to see the stats)

Are you happy with that number or should it be higher ?

If it should be higher, you need to use varnishlog to find out why
varnish doesn't cache things (...as long...) as you expect.

A likely culprit is the calculation of cache-TTL, by default Varnish
uses 120 seconds if nothing else can be deduced from the headers
from the backend server.  (The 120 is a parameter you can set
while running).

If your backend doesn't send headers which varnish can use to
calculate the TTL, you can use VCL to change it for specific
content, for instance:

	vcl_fetch {
		if (req.url ~ ".png") {
			obj.ttl = 1h;
		}
	}

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From rad_kam at tlen.pl  Thu Jul 26 08:04:05 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Thu, 26 Jul 2007 10:04:05 +0200
Subject: =?UTF-8?Q?Varnish_don't_want_cache_?=
Message-ID: <3e10e86.3f54be10.46a85575.a52b9@o2.pl>

I don't know why but varnish dont want cache anything , i allways get HttpStatus: recive nothink and RxResponse: not found :/ 

wtf ? :)


From phk at phk.freebsd.dk  Thu Jul 26 08:40:00 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 26 Jul 2007 08:40:00 +0000
Subject: Conditional GETs 
In-Reply-To: Your message of "Wed, 25 Jul 2007 12:38:49 MST."
	<64E001C5-7D33-42F8-A41E-13EF0EED84F7@digitalmarbles.com> 
Message-ID: <35352.1185439200@critter.freebsd.dk>

In message <64E001C5-7D33-42F8-A41E-13EF0EED84F7 at digitalmarbles.com>, Ricardo N
ewbery writes:
>
>I assume Varnish supports ETags and conditional GETs "If-Modified- 
>Since" (IMS) and "If-None-Match" (INM)?

Right now we only support IMS, the ETags are mostly a crutch for
clock-less clients/servers, and varnish isn't relevant in those
cases.

>Are IMS/INM requests relayed to the backend before the content is  
>served out of cache?

No, we check IMS and if it passses, we serve out of cache if cache
is still valid.

>Do the Cache-Control headers moderating conditional requests ("must- 
>revalidate",  "proxy-revalidate", and "no-cache") modify this heuristic?

In general we don't respect cache-control from clients as we are not
a proxy in the RFC2616 sense, but rather part of the server.

Varnish' job is to protect the backend from unnecessary traffic
and since we are in cohort with the backend, we know our content
to be up to date and valid (anything else would be a configuration
error).

If you want to respect cache-control, you can implement it in
vcl_recv()

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Thu Jul 26 08:45:18 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 26 Jul 2007 10:45:18 +0200
Subject: Varnish don't want cache
In-Reply-To: <3e10e86.3f54be10.46a85575.a52b9@o2.pl> (rad kam's message of
	"Thu\, 26 Jul 2007 10\:04\:05 +0200")
References: <3e10e86.3f54be10.46a85575.a52b9@o2.pl>
Message-ID: <87hcnr4kpt.fsf@des.linpro.no>

rad_kam at tlen.pl writes:
> I don't know why but varnish dont want cache anything , i allways get
> HttpStatus: recive nothink and RxResponse: not found :/
>
> wtf ? :)

Yes, "wtf?" is precisely what I always think when I get problem reports
which contain absolutely no useful information.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From phk at phk.freebsd.dk  Thu Jul 26 08:51:28 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 26 Jul 2007 08:51:28 +0000
Subject: Question about Pass 
In-Reply-To: Your message of "Wed, 25 Jul 2007 14:14:27 -0400."
	<46A79303.2090001@nyi.net> 
Message-ID: <35415.1185439888@critter.freebsd.dk>

In message <46A79303.2090001 at nyi.net>, James Quacinella writes:

>So when running varnishstat, what does 'Cache hits for pass' mean? Sicne
>the above says that pass mode does not enter anything in the cache, but
>it can still use the cache?

If you select "pass" in vcl_fetch(), an place-holder object will be
inserted in the cache which is maked "pass this one".

The next request to hit this object, will get a cache-hit, and then
use pass-processing.

The reason for this is that when we have a cache_miss, any
subsequent requests are "stalled" until the backend replies, so
finding out that an object shouldn't be cached all the way down
in vcl_fetch() can potentially be a bottleneck.

Inserting the "pass" object, means that the subsequent requests
of this object don't risk this pile-up.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From ssm at linpro.no  Thu Jul 26 09:06:05 2007
From: ssm at linpro.no (Stig Sandbeck Mathisen)
Date: Thu, 26 Jul 2007 11:06:05 +0200
Subject: Varnish don't want cache
In-Reply-To: <3e10e86.3f54be10.46a85575.a52b9@o2.pl> (rad kam's message of
	"Thu, 26 Jul 2007 10:04:05 +0200")
References: <3e10e86.3f54be10.46a85575.a52b9@o2.pl>
Message-ID: <7x3azbedqa.fsf@iostat.e.linpro.no>

rad_kam at tlen.pl writes:

> I don't know why but varnish dont want cache anything , i allways
> get HttpStatus: recive nothink and RxResponse: not found :/

A bit more information would be really helpful.

> wtf ? :)

vcl!

-- 
Stig Sandbeck Mathisen, Linpro


From chulmin2 at hotmail.com  Thu Jul 26 09:51:59 2007
From: chulmin2 at hotmail.com (Monty Ree)
Date: Thu, 26 Jul 2007 09:51:59 +0000
Subject: additional question about config change?
Message-ID: <BAY101-F3200BC240F7676BE9193B585F20@phx.gbl>

Hello, list.

I have read that config change would be effective like below at management 
interface.
(according to 
http://projects.linpro.no/pipermail/varnish-misc/2007-July/000598.html)

vcl.load newconf /my/new/config
vcl.use newconf

But, if main config file is /etc/default.vcl and varnishd loaded.
After that, if I change /etc/default.vcl(main config file) again, 
above config is valid or not?

If not,what can I do to valid already loaded config file?


Thanks in advance.

_________________________________________________________________
MSN Messenger? ?? ????? ?? ??? ??? ????.   
http://www.msn.co.kr/messenger 



From phk at phk.freebsd.dk  Thu Jul 26 09:54:14 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 26 Jul 2007 09:54:14 +0000
Subject: additional question about config change? 
In-Reply-To: Your message of "Thu, 26 Jul 2007 09:51:59 GMT."
	<BAY101-F3200BC240F7676BE9193B585F20@phx.gbl> 
Message-ID: <35790.1185443654@critter.freebsd.dk>



When you do a vcl.load, the file is read, compiled into a
machine executable format and loaded into varnishd.

Any changes you make to the source file after that will not
affect the compile version which is already loaded.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From ric at digitalmarbles.com  Fri Jul 27 01:19:11 2007
From: ric at digitalmarbles.com (Ricardo Newbery)
Date: Thu, 26 Jul 2007 18:19:11 -0700
Subject: Conditional GETs 
In-Reply-To: <35352.1185439200@critter.freebsd.dk>
References: <35352.1185439200@critter.freebsd.dk>
Message-ID: <E7EB8D13-D7B6-42CD-89F0-524B819B6527@digitalmarbles.com>


On Jul 26, 2007, at 1:40 AM, Poul-Henning Kamp wrote:

> Ricardo Newbery writes:
>>
>> I assume Varnish supports ETags and conditional GETs "If-Modified-
>> Since" (IMS) and "If-None-Match" (INM)?
>
> Right now we only support IMS, the ETags are mostly a crutch for
> clock-less clients/servers, and varnish isn't relevant in those
> cases.



Sorry, what does this mean precisely?  When an ETagged page is served  
through Varnish, does it not cache?  If it does cache, is it possible  
to code the INM test in vcl?

Vary support without ETag concerns me a bit.  See https:// 
bugzilla.mozilla.org/show_bug.cgi?id=269303




>> Are IMS/INM requests relayed to the backend before the content is
>> served out of cache?
>
> No, we check IMS and if it passses, we serve out of cache if cache
> is still valid.


How about an "INM without IMS" request?  Still served out of cache I  
suppose.  Is it possible to modify this behavior via vcl?



>> Do the Cache-Control headers moderating conditional requests ("must-
>> revalidate",  "proxy-revalidate", and "no-cache") modify this  
>> heuristic?
>
> In general we don't respect cache-control from clients as we are not
> a proxy in the RFC2616 sense, but rather part of the server.


Okay.  If I understand correctly, since Varnish doesn't respect cache- 
control from clients, then backend-generated cache-control like "must- 
revalidate",  "proxy-revalidate", and "no-cache" are unnecessary  
since the "max-age" and "s-maxage" response cache-controls are  
sufficient -- at least as far as the Varnish heuristic is concerned.   
Correct?

Does Varnish respect "private" and "no-store" in the response?

How about "public"?

Ric




From phk at phk.freebsd.dk  Fri Jul 27 04:55:58 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 27 Jul 2007 04:55:58 +0000
Subject: Conditional GETs 
In-Reply-To: Your message of "Thu, 26 Jul 2007 18:19:11 MST."
	<E7EB8D13-D7B6-42CD-89F0-524B819B6527@digitalmarbles.com> 
Message-ID: <39352.1185512158@critter.freebsd.dk>

In message <E7EB8D13-D7B6-42CD-89F0-524B819B6527 at digitalmarbles.com>, Ricardo N
ewbery writes:

>Vary support without ETag concerns me a bit.  See https:// 
>bugzilla.mozilla.org/show_bug.cgi?id=269303

Vary will respect any header listed.

>How about an "INM without IMS" request?  Still served out of cache I  
>suppose.  Is it possible to modify this behavior via vcl?

Everything is served out of cache except Cookies and Authentication.

IMS is only special in that it does not return the body unless necessary.

INM is ignored and hence the body is always returned.

>Okay.  If I understand correctly, since Varnish doesn't respect cache- 
>control from clients, then backend-generated cache-control like "must- 
>revalidate",  "proxy-revalidate", and "no-cache" are unnecessary  
>since the "max-age" and "s-maxage" response cache-controls are  
>sufficient -- at least as far as the Varnish heuristic is concerned.   
>Correct?

You may still want to generate the headers for the clients to see.

>Does Varnish respect "private" and "no-store" in the response?

No.  Varnish is considered part of the web-server, and thus these
don't apply to it (unless people implement them in VCL)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From kradziszewski at gmail.com  Fri Jul 27 06:43:29 2007
From: kradziszewski at gmail.com (Kamil Radziszewski)
Date: Fri, 27 Jul 2007 08:43:29 +0200
Subject: Varnish - fetch path
Message-ID: <231965b00707262343s5417ec08yb8141de5514187ef@mail.gmail.com>

i'm wondering where varnish is fetching all files ...in varnishstats i can
see that varnish is fetching, but in the default path (/tmp) i can see just
one file vcl.XXXXXX.....can somebody explaint it?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070727/a2b51048/attachment.html>

From ric at digitalmarbles.com  Fri Jul 27 09:11:17 2007
From: ric at digitalmarbles.com (Ricardo Newbery)
Date: Fri, 27 Jul 2007 02:11:17 -0700
Subject: Conditional GETs 
In-Reply-To: <39352.1185512158@critter.freebsd.dk>
References: <39352.1185512158@critter.freebsd.dk>
Message-ID: <41C0B585-C625-4764-828D-0539817C8F58@digitalmarbles.com>


On Jul 26, 2007, at 9:55 PM, Poul-Henning Kamp wrote:

> Ricardo Newbery writes:
>> How about an "INM without IMS" request?  Still served out of cache I
>> suppose.  Is it possible to modify this behavior via vcl?
>
> Everything is served out of cache except Cookies and Authentication.
>
> IMS is only special in that it does not return the body unless  
> necessary.
>
> INM is ignored and hence the body is always returned.


Yes, but is it possible to get INM behavior via a custom VCL?  In  
other words, Varnish already has a mechanism to serve 304s -- is it  
possible, using VCL, to tie into this mechanism?  And is it possible,  
using VCL, to test for the ETag value in the cached copy?




>> Okay.  If I understand correctly, since Varnish doesn't respect  
>> cache-
>> control from clients, then backend-generated cache-control like  
>> "must-
>> revalidate",  "proxy-revalidate", and "no-cache" are unnecessary
>> since the "max-age" and "s-maxage" response cache-controls are
>> sufficient -- at least as far as the Varnish heuristic is concerned.
>> Correct?
>
> You may still want to generate the headers for the clients to see.


Yes, I understand this.  What I need to do for the clients is a  
separate issue.

I'm a little unsure as I didn't hear an affirmation to my question...  
so does Varnish respect max-age and s-maxage in the response?



>> Does Varnish respect "private" and "no-store" in the response?
>
> No.  Varnish is considered part of the web-server, and thus these
> don't apply to it (unless people implement them in VCL)



Okay...  How should the backend inform Varnish that a certain item  
should not be stored in the cache?

Ric





From eculp at encontacto.net  Fri Jul 27 11:49:02 2007
From: eculp at encontacto.net (eculp at encontacto.net)
Date: Fri, 27 Jul 2007 06:49:02 -0500
Subject: will multiple apache name-based virtual hosts work with varnish.
Message-ID: <20070727064902.e813vmtcg8w80gcc@intranet.encontacto.net>

This is probably a configuration error on my part but in testing  
varnish works as expected except with name-based virtual hosts.

I have apache running on port 8080 and vlc running on port 80 with
varnishd_listen=":80"
varnishd_telnet="localhost:81"

I have no idea what the telnet option is for and the firewall doesn't  
pass 81 maybe that is or will be a problem.

Any suggestions for documentation in addition to the manuals and the  
on site faq will also be appreciated.

Thanks,

ed


From andre.cruz at co.sapo.pt  Fri Jul 27 16:30:13 2007
From: andre.cruz at co.sapo.pt (=?ISO-8859-1?Q?Andr=E9_Cruz?=)
Date: Fri, 27 Jul 2007 17:30:13 +0100
Subject: Regarding changeset 1761
Message-ID: <2A497C19-97A4-46D6-8448-3D27CBFD2F8E@co.sapo.pt>

After reading this changeset:

############
r1761 | cecilihf | 2007-07-25 09:39:10 +0100 (Wed, 25 Jul 2007) | 9  
lines
Changed paths:
    M /trunk/varnish-cache/bin/varnishd/cache_ban.c
    M /trunk/varnish-cache/include/vrt.h
    M /trunk/varnish-cache/lib/libvcl/vcc_action.c
    M /trunk/varnish-cache/lib/libvcl/vcc_fixed_token.c

Implemented http purge with regexp. Example vcl usage:
sub vcl_recv {
         if (req.request == "REPURGE") {
                 purge(req.url);
                 error 404 "Purged";
         }
}
############

I tried the example VCL provided but I get this error:

$ ./varnishd -f ../../etc/blogs.vcl -n /tmp/pretty -d
Expected action, 'if' or '}'
(../../etc/blogs.vcl Line 25 Pos 7)
       purge(req.url);
------#####----------

I'm running r1778. Is this functionality still not implemented?

Thanks,
Andr?

From phk at phk.freebsd.dk  Fri Jul 27 20:15:52 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 27 Jul 2007 20:15:52 +0000
Subject: Varnish - fetch path 
In-Reply-To: Your message of "Fri, 27 Jul 2007 08:43:29 +0200."
	<231965b00707262343s5417ec08yb8141de5514187ef@mail.gmail.com> 
Message-ID: <1615.1185567352@critter.freebsd.dk>

In message <231965b00707262343s5417ec08yb8141de5514187ef at mail.gmail.com>, "Kami
l Radziszewski" writes:

>i'm wondering where varnish is fetching all files ...in varnishstats
>i can see that varnish is fetching, but in the default path (/tmp)
>i can see just one file vcl.XXXXXX.....can somebody explaint it?<br>

Varnish stores all objects in a single file.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From fredrik.neij at teneco.se  Thu Jul 26 20:10:58 2007
From: fredrik.neij at teneco.se (Fredrik Neij)
Date: Thu, 26 Jul 2007 22:10:58 +0200
Subject: Memory leak in varnishncsa ?
Message-ID: <200707262211.00781.fredrik.neij@teneco.se>

root at lb:/tmp# varnishncsa > v.log
Assert error in trimline(), varnishncsa.c line 160:
  Condition(p != NULL) not true.
  errno = 12 (Cannot allocate memory)
Aborted


It just keeps growing until it's out of memory.



From des at linpro.no  Sat Jul 28 08:08:00 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sat, 28 Jul 2007 10:08:00 +0200
Subject: Conditional GETs
In-Reply-To: <41C0B585-C625-4764-828D-0539817C8F58@digitalmarbles.com>
	(Ricardo Newbery's message of "Fri\, 27 Jul 2007 02\:11\:17 -0700")
References: <39352.1185512158@critter.freebsd.dk>
	<41C0B585-C625-4764-828D-0539817C8F58@digitalmarbles.com>
Message-ID: <87tzrpvtlr.fsf@des.linpro.no>

Ricardo Newbery <ric at digitalmarbles.com> writes:
> I'm a little unsure as I didn't hear an affirmation to my question...
> so does Varnish respect max-age and s-maxage in the response?

Yes, see r1386.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Sat Jul 28 08:13:10 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sat, 28 Jul 2007 10:13:10 +0200
Subject: Regarding changeset 1761
In-Reply-To: <2A497C19-97A4-46D6-8448-3D27CBFD2F8E@co.sapo.pt>
	(=?utf-8?Q?=22Andr=C3=A9?= Cruz"'s
	message of "Fri\, 27 Jul 2007 17\:30\:13 +0100")
References: <2A497C19-97A4-46D6-8448-3D27CBFD2F8E@co.sapo.pt>
Message-ID: <87myxhvtd5.fsf@des.linpro.no>

Andr? Cruz <andre.cruz at co.sapo.pt> writes:
> I tried the example VCL provided but I get this error:
>
> $ ./varnishd -f ../../etc/blogs.vcl -n /tmp/pretty -d
> Expected action, 'if' or '}'
> (../../etc/blogs.vcl Line 25 Pos 7)
>        purge(req.url);
> ------#####----------
>
> I'm running r1778. Is this functionality still not implemented?

Check again.  You are almost certainly running the wrong binary.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From kradziszewski at gmail.com  Sat Jul 28 18:24:18 2007
From: kradziszewski at gmail.com (Kamil Radziszewski)
Date: Sat, 28 Jul 2007 20:24:18 +0200
Subject: Strange Error
Message-ID: <46AB89D2.1030607@gmail.com>

Can somebody tell me what does the message mean:

<Error>
 <Code>NoSuchBucket</Code>
  <Message>The specified bucket does not exist</Message>
    <RequestId>EB5F2CE4C66A1060</RequestId>
      <BucketName>my.server.with.varnish</BucketName>
 <HostId>
    NAf7k/r4gv5U/PiVHMjgnJlgPOsgUZvVMBU0N25Fb5R78zAUW3QDzu/oacYaJPK4
 </HostId>
</Error>


??


From phk at phk.freebsd.dk  Sat Jul 28 18:28:23 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 28 Jul 2007 18:28:23 +0000
Subject: Strange Error 
In-Reply-To: Your message of "Sat, 28 Jul 2007 20:24:18 +0200."
	<46AB89D2.1030607@gmail.com> 
Message-ID: <6016.1185647303@critter.freebsd.dk>

In message <46AB89D2.1030607 at gmail.com>, Kamil Radziszewski writes:
>Can somebody tell me what does the message mean:
>
><Error>
> <Code>NoSuchBucket</Code>
>  <Message>The specified bucket does not exist</Message>
>    <RequestId>EB5F2CE4C66A1060</RequestId>
>      <BucketName>my.server.with.varnish</BucketName>
> <HostId>
>    NAf7k/r4gv5U/PiVHMjgnJlgPOsgUZvVMBU0N25Fb5R78zAUW3QDzu/oacYaJPK4
> </HostId>
></Error>

Can you tell me where the message comes from ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Sat Jul 28 19:02:16 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sat, 28 Jul 2007 21:02:16 +0200
Subject: Strange Error
In-Reply-To: <46AB89D2.1030607@gmail.com> (Kamil Radziszewski's message of
	"Sat\, 28 Jul 2007 20\:24\:18 +0200")
References: <46AB89D2.1030607@gmail.com>
Message-ID: <87d4ycwdvr.fsf@des.linpro.no>

Kamil Radziszewski <kradziszewski at gmail.com> writes:
> Can somebody tell me what does the message mean:
>
> <Error>
>  <Code>NoSuchBucket</Code>
>   <Message>The specified bucket does not exist</Message>
>     <RequestId>EB5F2CE4C66A1060</RequestId>
>       <BucketName>my.server.with.varnish</BucketName>
>  <HostId>
>     NAf7k/r4gv5U/PiVHMjgnJlgPOsgUZvVMBU0N25Fb5R78zAUW3QDzu/oacYaJPK4
>  </HostId>
> </Error>

You'll have to ask Amazon, or whoever wrote the software you're running
that uses Amazon S3.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ask at develooper.com  Sat Jul 28 19:01:33 2007
From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Sat, 28 Jul 2007 12:01:33 -0700
Subject: Strange Error
In-Reply-To: <46AB89D2.1030607@gmail.com>
References: <46AB89D2.1030607@gmail.com>
Message-ID: <8382CDC4-01D6-4E76-B0FD-E85EB316C231@develooper.com>


On Jul 28, 2007, at 11:24, Kamil Radziszewski wrote:

> Can somebody tell me what does the message mean:
>
> <Error>
>  <Code>NoSuchBucket</Code>
>   <Message>The specified bucket does not exist</Message>

That looks like an Amazon S3 error.  Are you caching your S3 bucket?


  - ask

-- 
http://develooper.com/ - http://askask.com/




From ric at digitalmarbles.com  Sat Jul 28 22:47:11 2007
From: ric at digitalmarbles.com (Ricardo Newbery)
Date: Sat, 28 Jul 2007 15:47:11 -0700
Subject: Conditional GETs
In-Reply-To: <87tzrpvtlr.fsf@des.linpro.no>
References: <39352.1185512158@critter.freebsd.dk>
	<41C0B585-C625-4764-828D-0539817C8F58@digitalmarbles.com>
	<87tzrpvtlr.fsf@des.linpro.no>
Message-ID: <E003583E-A653-4294-B745-A41CA1FB81ED@digitalmarbles.com>


On Jul 28, 2007, at 1:08 AM, Dag-Erling Sm?rgrav wrote:

> Ricardo Newbery <ric at digitalmarbles.com> writes:
>> I'm a little unsure as I didn't hear an affirmation to my question...
>> so does Varnish respect max-age and s-maxage in the response?
>
> Yes, see r1386.


Ahh... okay.  Thanks.

Now back to the question about INM requests.  After some thought, I  
guess I'm less interested about trying to replicate INM in VCL but am  
still interested in how Varnish refreshes it's own cache.

Since Varnish respects the maxage controls, this implies that when  
the next request comes in after expiry it will generate a conditional  
request (if Last-Modified is set) to refresh it's cache.  Correct?

But no INM to the backend will ever be generated regardless of the  
presence of an ETag.  Correct?

In my own use case (Plone CMS), etags are leveraged as a convenient  
index hash for the server's pagecache in RAM.  But in order to serve  
from the pagecache, we need an INM in the request.  So I'm trying to  
understand the implications of Varnish's ETag behavior on this process.

Ric







From ric at digitalmarbles.com  Sat Jul 28 23:09:41 2007
From: ric at digitalmarbles.com (Ricardo Newbery)
Date: Sat, 28 Jul 2007 16:09:41 -0700
Subject: Conditional GETs
In-Reply-To: <E003583E-A653-4294-B745-A41CA1FB81ED@digitalmarbles.com>
References: <39352.1185512158@critter.freebsd.dk>
	<41C0B585-C625-4764-828D-0539817C8F58@digitalmarbles.com>
	<87tzrpvtlr.fsf@des.linpro.no>
	<E003583E-A653-4294-B745-A41CA1FB81ED@digitalmarbles.com>
Message-ID: <09357B1E-CCE3-4892-8078-73C4ACE2D38F@digitalmarbles.com>


On Jul 28, 2007, at 3:47 PM, Ricardo Newbery wrote:

>
> On Jul 28, 2007, at 1:08 AM, Dag-Erling Sm?rgrav wrote:
>
>> Ricardo Newbery <ric at digitalmarbles.com> writes:
>>> I'm a little unsure as I didn't hear an affirmation to my  
>>> question...
>>> so does Varnish respect max-age and s-maxage in the response?
>>
>> Yes, see r1386.
>
>
> Ahh... okay.  Thanks.
>
> Now back to the question about INM requests.  After some thought, I
> guess I'm less interested about trying to replicate INM in VCL but am
> still interested in how Varnish refreshes it's own cache.
>
> Since Varnish respects the maxage controls, this implies that when
> the next request comes in after expiry it will generate a conditional
> request (if Last-Modified is set) to refresh it's cache.  Correct?
>
> But no INM to the backend will ever be generated regardless of the
> presence of an ETag.  Correct?
>
> In my own use case (Plone CMS), etags are leveraged as a convenient
> index hash for the server's pagecache in RAM.  But in order to serve
> from the pagecache, we need an INM in the request.  So I'm trying to
> understand the implications of Varnish's ETag behavior on this  
> process.
>
> Ric


Correction.  Upon reflection the explanation of the server pagecache  
behavior didn't sound right.  So I checked the code, and I was wrong  
--- the pagecache doesn't need an INM; it regenerates the ETag based  
on the request before querying the pagecache.

Ric





From des at linpro.no  Sun Jul 29 09:10:13 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sun, 29 Jul 2007 11:10:13 +0200
Subject: Conditional GETs
In-Reply-To: <E003583E-A653-4294-B745-A41CA1FB81ED@digitalmarbles.com>
	(Ricardo Newbery's message of "Sat\, 28 Jul 2007 15\:47\:11 -0700")
References: <39352.1185512158@critter.freebsd.dk>
	<41C0B585-C625-4764-828D-0539817C8F58@digitalmarbles.com>
	<87tzrpvtlr.fsf@des.linpro.no>
	<E003583E-A653-4294-B745-A41CA1FB81ED@digitalmarbles.com>
Message-ID: <87wswjvami.fsf@des.linpro.no>

Ricardo Newbery <ric at digitalmarbles.com> writes:
> Since Varnish respects the maxage controls, this implies that when the
> next request comes in after expiry it will generate a conditional
> request (if Last-Modified is set) to refresh it's cache.  Correct?

No, at present it will generate an unconditional request because the
expired version will already have been removed from the cache.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ric at digitalmarbles.com  Sun Jul 29 10:19:39 2007
From: ric at digitalmarbles.com (Ricardo Newbery)
Date: Sun, 29 Jul 2007 03:19:39 -0700
Subject: Conditional GETs
In-Reply-To: <87wswjvami.fsf@des.linpro.no>
References: <39352.1185512158@critter.freebsd.dk>
	<41C0B585-C625-4764-828D-0539817C8F58@digitalmarbles.com>
	<87tzrpvtlr.fsf@des.linpro.no>
	<E003583E-A653-4294-B745-A41CA1FB81ED@digitalmarbles.com>
	<87wswjvami.fsf@des.linpro.no>
Message-ID: <3C8BF0B5-B2BB-4D5D-99C6-F44DDAEFADD1@digitalmarbles.com>


On Jul 29, 2007, at 2:10 AM, Dag-Erling Sm?rgrav wrote:

> Ricardo Newbery <ric at digitalmarbles.com> writes:
>> Since Varnish respects the maxage controls, this implies that when  
>> the
>> next request comes in after expiry it will generate a conditional
>> request (if Last-Modified is set) to refresh it's cache.  Correct?
>
> No, at present it will generate an unconditional request because the
> expired version will already have been removed from the cache.
>
> DES


Okay.  I think I'm getting a better handle on this now.  Thanks for  
your patience.

One more question regarding conditional requests:

What happens when a client sends a conditional request (either INM or  
IMS) and the content is not available in the Varnish cache?  I assume  
that Varnish will just relay the original conditional request through  
to the backend.  But then what happens when the backend responds with  
a 304?  I'm guessing Varnish won't cache a response until a non-304  
is returned.  Correct?

Ric





From des at linpro.no  Sun Jul 29 10:52:05 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sun, 29 Jul 2007 12:52:05 +0200
Subject: Conditional GETs
In-Reply-To: <3C8BF0B5-B2BB-4D5D-99C6-F44DDAEFADD1@digitalmarbles.com>
	(Ricardo Newbery's message of "Sun\, 29 Jul 2007 03\:19\:39 -0700")
References: <39352.1185512158@critter.freebsd.dk>
	<41C0B585-C625-4764-828D-0539817C8F58@digitalmarbles.com>
	<87tzrpvtlr.fsf@des.linpro.no>
	<E003583E-A653-4294-B745-A41CA1FB81ED@digitalmarbles.com>
	<87wswjvami.fsf@des.linpro.no>
	<3C8BF0B5-B2BB-4D5D-99C6-F44DDAEFADD1@digitalmarbles.com>
Message-ID: <87sl77v5wq.fsf@des.linpro.no>

Ricardo Newbery <ric at digitalmarbles.com> writes:
> What happens when a client sends a conditional request (either INM or
> IMS) and the content is not available in the Varnish cache?  I assume
> that Varnish will just relay the original conditional request through
> to the backend.

No, it will strip the conditional header before requesting the page from
the backend.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Sun Jul 29 13:51:21 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sun, 29 Jul 2007 15:51:21 +0200
Subject: Conditional GETs
In-Reply-To: <3C8BF0B5-B2BB-4D5D-99C6-F44DDAEFADD1@digitalmarbles.com>
	(Ricardo Newbery's message of "Sun\, 29 Jul 2007 03\:19\:39 -0700")
References: <39352.1185512158@critter.freebsd.dk>
	<41C0B585-C625-4764-828D-0539817C8F58@digitalmarbles.com>
	<87tzrpvtlr.fsf@des.linpro.no>
	<E003583E-A653-4294-B745-A41CA1FB81ED@digitalmarbles.com>
	<87wswjvami.fsf@des.linpro.no>
	<3C8BF0B5-B2BB-4D5D-99C6-F44DDAEFADD1@digitalmarbles.com>
Message-ID: <87odhvuxly.fsf@des.linpro.no>

Ricardo Newbery <ric at digitalmarbles.com> writes:
> On Jul 29, 2007, at 2:10 AM, Dag-Erling Sm?rgrav wrote:
>> Ricardo Newbery <ric at digitalmarbles.com> writes:
>>> Since Varnish respects the maxage controls, this implies that when
>>> the next request comes in after expiry it will generate a
>>> conditional request (if Last-Modified is set) to refresh it's cache.
>>> Correct?
>> No, at present it will generate an unconditional request because the
>> expired version will already have been removed from the cache.
> What happens when a client sends a conditional request (either INM or
> IMS) and the content is not available in the Varnish cache?  I assume
> that Varnish will just relay the original conditional request through
> to the backend.  But then what happens when the backend responds with
> a 304?  I'm guessing Varnish won't cache a response until a non-304
> is returned.  Correct?

"No, at present it will generate an unconditional request"

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Sun Jul 29 19:15:47 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sun, 29 Jul 2007 21:15:47 +0200
Subject: Conditional GETs
In-Reply-To: <3CBC6A7E-6437-48CF-81A8-B8DDD7E4B877@digitalmarbles.com>
	(Ricardo Newbery's message of "Sun\, 29 Jul 2007 12\:05\:01 -0700")
References: <39352.1185512158@critter.freebsd.dk>
	<41C0B585-C625-4764-828D-0539817C8F58@digitalmarbles.com>
	<87tzrpvtlr.fsf@des.linpro.no>
	<E003583E-A653-4294-B745-A41CA1FB81ED@digitalmarbles.com>
	<87wswjvami.fsf@des.linpro.no>
	<3C8BF0B5-B2BB-4D5D-99C6-F44DDAEFADD1@digitalmarbles.com>
	<87sl77v5wq.fsf@des.linpro.no>
	<3CBC6A7E-6437-48CF-81A8-B8DDD7E4B877@digitalmarbles.com>
Message-ID: <87k5sjuil8.fsf@des.linpro.no>

Ricardo Newbery <ric at digitalmarbles.com> writes:

> On Jul 29, 2007, at 3:52 AM, Dag-Erling Sm?rgrav wrote:
>
>> Ricardo Newbery <ric at digitalmarbles.com> writes:
>>> What happens when a client sends a conditional request (either INM or
>>> IMS) and the content is not available in the Varnish cache?  I assume
>>> that Varnish will just relay the original conditional request through
>>> to the backend.
>>
>> No, it will strip the conditional header before requesting the page
>> from
>> the backend.
>
>
> Okay, I think I understand.  So with Varnish in front, using the
> default VCL, there is no set of circumstances that will ever result
> in a conditional request to the backend.  Correct?

Yes.

> But I can still change this behavior with a custom VCL.  Correct?

Yes, by editing bereq in vcl_miss.

> # perpetually refresh my cache
> vcl_timeout {
>         fetch;
> }

This probably won't, but it's on our list for 2.0.

> # if not available in cache, pass client's IMS to backend
> # and pass response back to client without caching
> vcl_miss {
>         if (req.http.If-Modified-Since) {
>                 pass;
>         }
> }

You probably don't want to do that.  It basically opens up a back door,
allowing anyone to bypass Varnish and hit your backend directly just by
including a bogus IMS in their request.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ric at digitalmarbles.com  Sun Jul 29 19:05:01 2007
From: ric at digitalmarbles.com (Ricardo Newbery)
Date: Sun, 29 Jul 2007 12:05:01 -0700
Subject: Conditional GETs
In-Reply-To: <87sl77v5wq.fsf@des.linpro.no>
References: <39352.1185512158@critter.freebsd.dk>
	<41C0B585-C625-4764-828D-0539817C8F58@digitalmarbles.com>
	<87tzrpvtlr.fsf@des.linpro.no>
	<E003583E-A653-4294-B745-A41CA1FB81ED@digitalmarbles.com>
	<87wswjvami.fsf@des.linpro.no>
	<3C8BF0B5-B2BB-4D5D-99C6-F44DDAEFADD1@digitalmarbles.com>
	<87sl77v5wq.fsf@des.linpro.no>
Message-ID: <3CBC6A7E-6437-48CF-81A8-B8DDD7E4B877@digitalmarbles.com>


On Jul 29, 2007, at 3:52 AM, Dag-Erling Sm?rgrav wrote:

> Ricardo Newbery <ric at digitalmarbles.com> writes:
>> What happens when a client sends a conditional request (either INM or
>> IMS) and the content is not available in the Varnish cache?  I assume
>> that Varnish will just relay the original conditional request through
>> to the backend.
>
> No, it will strip the conditional header before requesting the page  
> from
> the backend.


Okay, I think I understand.  So with Varnish in front, using the  
default VCL, there is no set of circumstances that will ever result  
in a conditional request to the backend.  Correct?

But I can still change this behavior with a custom VCL.  Correct?

Would the following work:


# perpetually refresh my cache
vcl_timeout {
         fetch;
}


# hold on to cache for at least a day
vcl_fetch {
         if (obj.ttl < 86400s) {
                 set obj.ttl = 86400s;
         }


# if not available in cache, pass client's IMS to backend
# and pass response back to client without caching
vcl_miss {
         if (req.http.If-Modified-Since) {
                 pass;
         }
}


If I wanted to selectively apply the first two above, could I just  
leverage a few custom headers like so?


# selectively force a perpetually refreshing cache  (is "resp"  
available here?)
vcl_timeout {
         if (resp.http.X-Cache-Refetch) {
                 fetch;
         }
}


# selectively apply a local ttl without messing up downstream proxies
vcl_fetch {
         if (resp.http.X-Cache-TTL) {
                 set obj.ttl = resp.http.X-Cache-TTL;
         }
}




From phk at phk.freebsd.dk  Sun Jul 29 19:46:03 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 29 Jul 2007 19:46:03 +0000
Subject: Conditional GETs 
In-Reply-To: Your message of "Sun, 29 Jul 2007 12:05:01 MST."
	<3CBC6A7E-6437-48CF-81A8-B8DDD7E4B877@digitalmarbles.com> 
Message-ID: <10838.1185738363@critter.freebsd.dk>

In message <3CBC6A7E-6437-48CF-81A8-B8DDD7E4B877 at digitalmarbles.com>, Ricardo Newbery writes
:
>
>On Jul 29, 2007, at 3:52 AM, Dag-Erling Sm=F8rgrav wrote:
>
>> Ricardo Newbery <ric at digitalmarbles.com> writes:
>>> What happens when a client sends a conditional request (either INM or
>>> IMS) and the content is not available in the Varnish cache?  I assume
>>> that Varnish will just relay the original conditional request through
>>> to the backend.
>>
>> No, it will strip the conditional header before requesting the page  =
>
>> from
>> the backend.
>
>
>Okay, I think I understand.  So with Varnish in front, using the  =
>
>default VCL, there is no set of circumstances that will ever result  =
>
>in a conditional request to the backend.  Correct?

Correct.

>But I can still change this behavior with a custom VCL.  Correct?
>
>Would the following work:
>
>
># perpetually refresh my cache
>vcl_timeout {
>         fetch;
>}

This is not implemented yet, but once we have it...

># if not available in cache, pass client's IMS to backend
># and pass response back to client without caching
>vcl_miss {
>         if (req.http.If-Modified-Since) {
>                 pass;
>         }
>}

This should work.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From rad_kam at tlen.pl  Sun Jul 29 20:16:16 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Sun, 29 Jul 2007 22:16:16 +0200
Subject: =?UTF-8?Q?Pipe_shut_write_and_pipe_shut_read?=
Message-ID: <279f2529.5f73a5f5.46acf590.a0d6c@o2.pl>

what does the pipe shut write and pipe shut write mean ? 


From phk at phk.freebsd.dk  Sun Jul 29 20:37:05 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 29 Jul 2007 20:37:05 +0000
Subject: =?UTF-8?Q?Pipe_shut_write_and_pipe_shut_read?= 
In-Reply-To: Your message of "Sun, 29 Jul 2007 22:16:16 +0200."
	<279f2529.5f73a5f5.46acf590.a0d6c@o2.pl> 
Message-ID: <11088.1185741425@critter.freebsd.dk>

In message <279f2529.5f73a5f5.46acf590.a0d6c at o2.pl>, rad_kam at tlen.pl writes:

>what does the pipe shut write and pipe shut write mean ? 

They're debugging messages of no consequence.  If/When we have no outstanding
pipe issues, we should remove them.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From rad_kam at tlen.pl  Sun Jul 29 20:59:51 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Sun, 29 Jul 2007 22:59:51 +0200
Subject: =?UTF-8?Q?TxHeader_host_and_RxHeader_host?=
Message-ID: <2cb8c7a9.3c7f4d01.46acffc7.319d0@o2.pl>

Both of those should be the same host ? I'm asking because in my varnishlog allways TxHeader and RxHeader host are my server with varnish ... non with backend ..


From des at linpro.no  Sun Jul 29 21:09:45 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sun, 29 Jul 2007 23:09:45 +0200
Subject: TxHeader host and RxHeader host
In-Reply-To: <2cb8c7a9.3c7f4d01.46acffc7.319d0@o2.pl> (rad kam's message of
	"Sun\, 29 Jul 2007 22\:59\:51 +0200")
References: <2cb8c7a9.3c7f4d01.46acffc7.319d0@o2.pl>
Message-ID: <87fy36vrvq.fsf@des.linpro.no>

rad_kam at tlen.pl writes:
> Both of those should be the same host ? I'm asking because in my
> varnishlog allways TxHeader and RxHeader host are my server with
> varnish ... non with backend ..

Rx and Tx mean "receive" and "transmit" respectively.  A request header
will be logged as RxHeader when received from the client, and with
TxHeader when sent to the backend.  Likewise, a response header will be
logged as RxHeader when received from the backend and as TxHeader when
sent to the client.

Varnish uses whatever Host: header it received from the client;
otherwise it wouldn't be able to accelerate servers with multiple
virtual hosts.  If the Host: header you get from the client is not
correct, you can adjust req.http.host it in vcl_recv or bereq.http.host
in vcl_miss, e.g.:

vcl_recv {
    if (req.http.host ~ "^(www\.)?example.com$") {
        req.backend = example_backend;
        req.http.host = "backend.example.com";
    } else {
        error 404 "Unknown virtual host";
    }
}

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rad_kam at tlen.pl  Sun Jul 29 22:30:25 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Mon, 30 Jul 2007 00:30:25 +0200
Subject: =?UTF-8?Q?TxHeader_host_and_RxHeader_host?=
Message-ID: <4e1844df.7d4f18db.46ad1501.cef54@o2.pl>

you wrote:

vcl_recv {
    if (req.http.host ~ "^(www\.)?example.com$") {
        req.backend = example_backend;
        req.http.host = "backend.example.com";
    } else {
        error 404 "Unknown virtual host";
    }
}


i have : 

backend other { 
        set backend.host = "mywebsite";
        set backend.port = "80";
}

if (req.http.host ~ "^my.server.with.varnish")
        {
                set req.backend = other;
        } else {
                error 404 "The host listed in your request is     
                 not recognized";
        }

Is this the same? I'm asking because your configuration won't work ( I have varnish 1.0.2)


From des at linpro.no  Mon Jul 30 05:42:15 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 30 Jul 2007 07:42:15 +0200
Subject: TxHeader host and RxHeader host
In-Reply-To: <4e1844df.7d4f18db.46ad1501.cef54@o2.pl> (rad kam's message of
	"Mon\, 30 Jul 2007 00\:30\:25 +0200")
References: <4e1844df.7d4f18db.46ad1501.cef54@o2.pl>
Message-ID: <87bqduv45k.fsf@des.linpro.no>

rad_kam at tlen.pl writes:
> Is this the same? I'm asking because your configuration won't work ( I
> have varnish 1.0.2)

You could have told us that a little earlier.  Please upgrade to 1.1.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rad_kam at tlen.pl  Mon Jul 30 06:23:29 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Mon, 30 Jul 2007 08:23:29 +0200
Subject: =?UTF-8?Q?Re:_TxHeader_host_and_RxHeader_host?=
Message-ID: <62f7039.79563b7f.46ad83e1.ef59f@o2.pl>

Allready upgraded, but when run varnish get this error: 

Expected action, 'if' or '}'
(/opt/varnish/etc/vcl.conf Line 21 Pos 18)
                 req.backend = apache;
-----------------###########----------



From rad_kam at tlen.pl  Mon Jul 30 06:33:13 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Mon, 30 Jul 2007 08:33:13 +0200
Subject: =?UTF-8?Q?Re:_Re:_TxHeader_host_and_RxHeader_host?=
Message-ID: <314870ed.557ae439.46ad8629.bdd65@o2.pl>

Allright , i forgot write "set" ;)

But have another question: how can I change TxHeader ?


From phk at phk.freebsd.dk  Mon Jul 30 06:57:01 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 30 Jul 2007 06:57:01 +0000
Subject: =?UTF-8?Q?TxHeader_host_and_RxHeader_host?= 
In-Reply-To: Your message of "Sun, 29 Jul 2007 22:59:51 +0200."
	<2cb8c7a9.3c7f4d01.46acffc7.319d0@o2.pl> 
Message-ID: <12856.1185778621@critter.freebsd.dk>

In message <2cb8c7a9.3c7f4d01.46acffc7.319d0 at o2.pl>, rad_kam at tlen.pl writes:

>Both of those should be the same host ? I'm asking because in my
>varnishlog allways TxHeader and RxHeader host are my server with
>varnish ... non with backend ..

Unless you modify them in VCL, they are whatever the client sends.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From steven at x92.net  Mon Jul 30 06:47:44 2007
From: steven at x92.net (Steven Ciaburri)
Date: Sun, 29 Jul 2007 23:47:44 -0700
Subject: Configuration help
Message-ID: <7334df7b0707292347v1745376ao91ae43207b0aed32@mail.gmail.com>

I need a little help building a configuration file that will cache a site
with a simple cookie enabled. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070729/7c2c6053/attachment.html>

From rad_kam at tlen.pl  Mon Jul 30 07:41:35 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Mon, 30 Jul 2007 09:41:35 +0200
Subject: =?UTF-8?Q?Create_worker_thread_failed_12_Cannot_allocate_memory?=
Message-ID: <2a61081.7f4ccae9.46ad962f.40d0d@o2.pl>

This message i receive when my varnish run. I have 6 GB of RAM but when i run varnish 5 GB are free ... the 1 GB is for somethink another ...not for varnish....so what can i do to tell varnish using RAM ??


From des at linpro.no  Mon Jul 30 07:47:48 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 30 Jul 2007 09:47:48 +0200
Subject: Create worker thread failed 12 Cannot allocate memory
In-Reply-To: <2a61081.7f4ccae9.46ad962f.40d0d@o2.pl> (rad kam's message of
	"Mon\, 30 Jul 2007 09\:41\:35 +0200")
References: <2a61081.7f4ccae9.46ad962f.40d0d@o2.pl>
Message-ID: <873az649jv.fsf@des.linpro.no>

rad_kam at tlen.pl writes:
> This message i receive when my varnish run. I have 6 GB of RAM but
> when i run varnish 5 GB are free ... the 1 GB is for somethink another
> ...not for varnish....so what can i do to tell varnish using RAM ??

You are most likely running out of address space.  What platform are you
on?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rad_kam at tlen.pl  Mon Jul 30 08:07:12 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Mon, 30 Jul 2007 10:07:12 +0200
Subject: =?UTF-8?Q?Re:_Create_worker_thread_failed_12_Cannot_allocate_memory?=
Message-ID: <ca6fc2b.251f41c.46ad9c30.3804@o2.pl>

i686 GNU/Linux....i have also set the ulimit value as 500000...and worker proces 2048 but still the same error 


From des at linpro.no  Mon Jul 30 08:16:54 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 30 Jul 2007 10:16:54 +0200
Subject: Create worker thread failed 12 Cannot allocate memory
In-Reply-To: <ca6fc2b.251f41c.46ad9c30.3804@o2.pl> (rad kam's message of "Mon\,
	30 Jul 2007 10\:07\:12 +0200")
References: <ca6fc2b.251f41c.46ad9c30.3804@o2.pl>
Message-ID: <87myxe2tmx.fsf@des.linpro.no>

rad_kam at tlen.pl writes:
> i686 GNU/Linux....

The i386 has a 4 GB address space, 1 GB of which is normally reserved
for the kernel, leaving 3 GB for applications (or even less depending on
the data segment size limit, 'ulimit -d').  A big chunk of that goes to
the program text (including libraries) and bss, and the stacks (one for
each thread).  The cache file can be up to 2 GB (Varnish will refuse to
use more than that on a 32-bit system), and malloc gets whatever is
left, which probably isn't enough if you use a large cache file.

> i have also set the ulimit value as 500000...

*which* ulimit value?

> and worker proces 2048 but still the same error

Increasing the number of worker threads will only make things worse.
You should decrease it to the expected number of concurrent clients.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From admin at adofms.com.au  Mon Jul 30 08:18:08 2007
From: admin at adofms.com.au (ADOFMS Admin, SteveOC)
Date: Mon, 30 Jul 2007 17:48:08 +0930
Subject: Create worker thread failed 12 Cannot allocate memory
In-Reply-To: <ca6fc2b.251f41c.46ad9c30.3804@o2.pl>
References: <ca6fc2b.251f41c.46ad9c30.3804@o2.pl>
Message-ID: <46AD9EC0.2030004@adofms.com.au>

rad_kam at tlen.pl wrote:
> i686 GNU/Linux....i have also set the ulimit value as 500000...and worker proces 2048 but still the same error 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>   
Correct me if Im wrong - but is it possible to address more than 4GB of
RAM with a 32bit linux kernel ?




From des at linpro.no  Mon Jul 30 08:20:35 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 30 Jul 2007 10:20:35 +0200
Subject: Create worker thread failed 12 Cannot allocate memory
In-Reply-To: <46AD9EC0.2030004@adofms.com.au> (ADOFMS Admin's message of "Mon\,
	30 Jul 2007 17\:48\:08 +0930")
References: <ca6fc2b.251f41c.46ad9c30.3804@o2.pl>
	<46AD9EC0.2030004@adofms.com.au>
Message-ID: <87ir822tgs.fsf@des.linpro.no>

"ADOFMS Admin, SteveOC" <admin at adofms.com.au> writes:
> Correct me if Im wrong - but is it possible to address more than 4GB of
> RAM with a 32bit linux kernel ?

Provided your kernel was built with PAE support, yes, but the process
address space is still limited to 3 GB.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rad_kam at tlen.pl  Mon Jul 30 08:26:22 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Mon, 30 Jul 2007 10:26:22 +0200
Subject: =?UTF-8?Q?Re:_Create_worker_thread_failed_12_Cannot_allocate_memory?=
Message-ID: <7ae9ee55.3e29f68f.46ada0ae.8135b@o2.pl>

ulimit -n 50000 :)


From ssm at linpro.no  Mon Jul 30 08:34:52 2007
From: ssm at linpro.no (Stig Sandbeck Mathisen)
Date: Mon, 30 Jul 2007 10:34:52 +0200
Subject: Configuration help
In-Reply-To: <7334df7b0707292347v1745376ao91ae43207b0aed32@mail.gmail.com>
	(Steven Ciaburri's message of "Sun, 29 Jul 2007 23:47:44 -0700")
References: <7334df7b0707292347v1745376ao91ae43207b0aed32@mail.gmail.com>
Message-ID: <7xtzrm70ib.fsf@iostat.e.linpro.no>

"Steven Ciaburri" <steven at x92.net> writes:

> I need a little help building a configuration file that will cache a
> site with a simple cookie enabled.

See the zope-plone.vcl config which is included, and also available at
http://varnish.projects.linpro.no/svn/trunk/varnish-cache/etc/zope-plone.vcl

-- 
Stig Sandbeck Mathisen, Linpro


From des at linpro.no  Mon Jul 30 08:35:53 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 30 Jul 2007 10:35:53 +0200
Subject: Create worker thread failed 12 Cannot allocate memory
In-Reply-To: <7ae9ee55.3e29f68f.46ada0ae.8135b@o2.pl> (rad kam's message of
	"Mon\, 30 Jul 2007 10\:26\:22 +0200")
References: <7ae9ee55.3e29f68f.46ada0ae.8135b@o2.pl>
Message-ID: <87ejiq2sra.fsf@des.linpro.no>

rad_kam at tlen.pl writes:
> ulimit -n 50000 :)

Where is the logic in this?  You run out of address space, and respond
by increasing the number of file descriptors?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rad_kam at tlen.pl  Mon Jul 30 09:21:46 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Mon, 30 Jul 2007 11:21:46 +0200
Subject: =?UTF-8?Q?Re:_Create_worker_thread_failed_12_Cannot_allocate_memory?=
Message-ID: <13a1c6d2.5b8db119.46adadaa.7f646@o2.pl>

i don't know if it's logic...i just try everythink ... is there any possibility to  cache object in RAM ? If not then can you tell waht should i do remove this error ...


From des at linpro.no  Mon Jul 30 09:40:22 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 30 Jul 2007 11:40:22 +0200
Subject: Create worker thread failed 12 Cannot allocate memory
In-Reply-To: <13a1c6d2.5b8db119.46adadaa.7f646@o2.pl> (rad kam's message of
	"Mon\, 30 Jul 2007 11\:21\:46 +0200")
References: <13a1c6d2.5b8db119.46adadaa.7f646@o2.pl>
Message-ID: <87abte2prt.fsf@des.linpro.no>

rad_kam at tlen.pl writes:
> i don't know if it's logic...i just try everythink ... is there any
> possibility to cache object in RAM ?

Varnish caches objects in virtual memory, and leaves it up to the kernel
to decide whether this means RAM or swap.

> If not then can you tell waht should i do remove this error ...

Increase the data size limit
Reduce the number of worker threads
Reduce the size of the cache file
Switch to a 64-bit machine

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rad_kam at tlen.pl  Mon Jul 30 10:04:31 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Mon, 30 Jul 2007 12:04:31 +0200
Subject: =?UTF-8?Q?Re:_Create_worker_thread_failed_12_Cannot_allocate_memory?=
Message-ID: <abcf396.25438c1d.46adb7af.d73c@o2.pl>

maybe can i decrease value of workers and increase value of child process ... is there any chance to do it ?


From des at linpro.no  Mon Jul 30 10:17:48 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 30 Jul 2007 12:17:48 +0200
Subject: Create worker thread failed 12 Cannot allocate memory
In-Reply-To: <abcf396.25438c1d.46adb7af.d73c@o2.pl> (rad kam's message of
	"Mon\, 30 Jul 2007 12\:04\:31 +0200")
References: <abcf396.25438c1d.46adb7af.d73c@o2.pl>
Message-ID: <873az62o1f.fsf@des.linpro.no>

rad_kam at tlen.pl writes:
> maybe can i decrease value of workers and increase value of child
> process...

I don't know what you mean by "increase value of child process".

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rad_kam at tlen.pl  Mon Jul 30 10:25:09 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Mon, 30 Jul 2007 12:25:09 +0200
Subject: =?UTF-8?Q?Fwd:_Re:_Create_worker_thread_failed_12_Cannot_allocate?=
	=?UTF-8?Q?_memory?=
Message-ID: <1587d80e.4f301450.46adbc85.3628f@o2.pl>

Each worker proces has a child process unless i'm wrong ;)


From des at linpro.no  Mon Jul 30 11:23:23 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 30 Jul 2007 13:23:23 +0200
Subject: Fwd: Re: Create worker thread failed 12 Cannot allocate memory
In-Reply-To: <1587d80e.4f301450.46adbc85.3628f@o2.pl> (rad kam's message of
	"Mon\, 30 Jul 2007 12\:25\:09 +0200")
References: <1587d80e.4f301450.46adbc85.3628f@o2.pl>
Message-ID: <87y7gy16fo.fsf@des.linpro.no>

rad_kam at tlen.pl writes:
> Each worker proces has a child process unless i'm wrong ;)

No.  There is one parent process and one child process.  The worker
threads are just that - threads.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From janis.putrams at delfi.lv  Mon Jul 30 11:21:38 2007
From: janis.putrams at delfi.lv (Janis Putrams)
Date: Mon, 30 Jul 2007 14:21:38 +0300
Subject: misc connection closures
Message-ID: <200707301421.38786.janis.putrams@delfi.lv>

Hi!

As this is my first post to this list I would like to thank to all developers of this excellent software.

We put our images behind varnish-1.0.4 serving ~1.5k req/sec and load is staying low which is great.

Though when reaching 2k req/sec we start to encounter connection closures. Connection gets closed immediately after connection is established.

I have raised thread limits but that did not help. I would look into logs if this happened for another application but I am not really sure where to look for varnish.

my configuration:
05:56:16 janis[pts/2]@host ~$ telnet localhost 81
Trying 127.0.0.1.81...
Connected to localhost.
Escape character is '^]'.
param.show
200 709
default_ttl          120 [seconds]
thread_pools         5 [pools]
thread_pool_max      100000 [threads]
thread_pool_min      100 [threads]
thread_pool_timeout  10 [seconds]
overflow_max         100 [%]
http_workspace       8192 [bytes]
sess_timeout         5 [seconds]
pipe_timeout         60 [seconds]
send_timeout         600 [seconds]
auto_restart         on [bool]
fetch_chunksize      128 [kilobytes]
sendfile_threshold   unlimited [bytes]
vcl_trace            off [bool]
listen_address       :80
listen_depth         4096 [connections]
srcaddr_hash         1049 [buckets]
srcaddr_ttl          30 [seconds]
backend_http11       off [bool]
client_http11        off [bool]
ping_interval        3 [seconds]

varnishstat:
00:01:22
Hitrate ratio:       10       19       19
Hitrate avg:     1.0059   1.0096   1.0096

       82740      1016.84      1009.02 Client connections accepted
      182469      2259.17      2225.23 Client requests received
      180928      2256.78      2206.44 Cache hits
           0         0.00         0.00 Cache hits for pass
        1541         2.40        18.79 Cache misses
        1541         2.40        18.79 Backend connections success
           2         0.00         0.02 Backend connections failures
        1507         2.40        18.38 Backend connections reuses
        1511         2.40        18.43 Backend connections recycles
           3         0.00         0.04 Backend connections unused
        1614         6.30        19.68 N struct srcaddr
         311        -3.70         3.79 N active struct srcaddr
         906         0.00        11.05 N struct sess_mem
         875        -1.00        10.67 N struct sess
        1604         2.60        19.56 N struct object
        1604         2.60        19.56 N struct objecthead
        1523         2.40        18.57 N struct smf
           0         0.00         0.00 N small free smf
           2         0.00         0.02 N large free smf
           6         0.00         0.07 N struct vbe_conn
          63         0.20         0.77 N worker threads
          63         0.20         0.77 N worker threads created
           0         0.00         0.00 N worker threads not created
           0         0.00         0.00 N worker threads limited
           0         0.00         0.00 N queued work requests
          63         0.20         0.77 N overflowed work requests
           0         0.00         0.00 N dropped work requests
           0         0.00         0.00 N expired objects
           0         0.00         0.00 N objects on deathrow
           0         0.00         0.00 HTTP header overflows
           0         0.00         0.00 Objects sent with sendfile
       95445      1159.97      1163.96 Objects sent with write
       77391       928.18       943.79 Total Sessions
      182469      2259.37      2225.23 Total Requests
           0         0.00         0.00 Total pipe
           0         0.00         0.00 Total pass
        1541         2.40        18.79 Total fetch
    38575489    476006.89    470432.79 Total header bytes
  7861511491  91999201.08  95872091.35 Total body bytes
       61101       712.18       745.13 Session Closed
         272         5.80         3.32 Session Pipeline
         137         3.00         1.67 Session Read Ahead
      121024      1539.50      1475.90 Session herd
     5833297     71817.11     71137.77 SHM records
      425680      5241.62      5191.22 SHM writes
         128         2.70         1.56 SHM MTX contention      

netstat summary:
      FIN_WAIT1: 93
      TIME_WAIT: 6825
     CLOSE_WAIT: 5
       LAST_ACK: 7
      FIN_WAIT2: 462
    ESTABLISHED: 751
       SYN_RECV: 112
         LISTEN: 7

load:
06:05:55 root[pts/1]@host ~# uptime
 06:05:58 up 52 days, 20:27,  4 users,  load average: 0.55, 0.39, 0.31

process info:
06:19:35 root[pts/1]@host ~/install# ps auxf|egrep [v]arnish
root     25226  0.0  0.0  10020   740 ?        Ss   05:33   0:00 varnishd -f /etc/varnishd/varnishd.conf -T 127.0.0.1:81 -P /var/run/varnishd.pid -s file,/var/spool/varnishd/varnishd_storage.bin,1574176153 -p thread_pools 5 -p listen_depth 4096 -w100,100000,10
root     26973 14.0  0.6 2282680 56608 ?       Sl   06:03   2:13  \_ varnishd -f /etc/varnishd/varnishd.conf -T 127.0.0.1:81 -P /var/run/varnishd.pid -s file,/var/spool/varnishd/varnishd_storage.bin,1574176153 -p thread_pools 5 -p listen_depth 4096 -w100,100000,10

06:19:42 root[pts/1]@host ~/install# uname -a
Linux host 2.6.16.52-1smp #1 SMP Thu May 31 19:32:54 CEST 2007 i686 Intel(R)_Xeon(R)_CPU____________5150__ at _2.66GHz Linux

06:23:04 root[pts/1]@host ~/install# free
             total       used       free     shared    buffers     cached
Mem:          8114        721       7392          0        306        314
-/+ buffers/cache:        101       8013
Swap:         9765          0       9765


connection closure:
06:05:58 root[pts/1]@host ~# telnet host.mydomain.lv 80
Trying 11.11.11.11.80...
Connected to 11.11.11.11.
Escape character is '^]'.
Connection closed by foreign host.

Also in varnishstat Client counters gets reset from time to time. Don't know if it is problem or not.

Thank you in advance.

--
Janis Putrams


From fragfutter at gmail.com  Mon Jul 30 12:58:21 2007
From: fragfutter at gmail.com (C. Handel)
Date: Mon, 30 Jul 2007 14:58:21 +0200
Subject: misc connection closures
In-Reply-To: <200707301421.38786.janis.putrams@delfi.lv>
References: <200707301421.38786.janis.putrams@delfi.lv>
Message-ID: <3d62bd5f0707300558i4fe63de0ldd1886c4147e34ac@mail.gmail.com>

On 7/30/07, Janis Putrams <janis.putrams at delfi.lv> wrote:

> Though when reaching 2k req/sec we start to encounter connection closures.
> Connection gets closed immediately after connection is established.

is there a ulimit from the Operating System? Number of open files/sockets?
(As user running varnish: "ulimit -a")

Greetings
   Christoph


From des at linpro.no  Mon Jul 30 13:42:42 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 30 Jul 2007 15:42:42 +0200
Subject: misc connection closures
In-Reply-To: <200707301421.38786.janis.putrams@delfi.lv> (Janis Putrams's
	message of "Mon\, 30 Jul 2007 14\:21\:38 +0300")
References: <200707301421.38786.janis.putrams@delfi.lv>
Message-ID: <87tzrm0zzh.fsf@des.linpro.no>

Janis Putrams <janis.putrams at delfi.lv> writes:
> We put our images behind varnish-1.0.4 serving ~1.5k req/sec and load
> is staying low which is great.

Varnish 1.0.4 has numerous known bugs and shortcomings.  You should
seriously consider upgrading to 1.1.

> Though when reaching 2k req/sec we start to encounter connection
> closures. Connection gets closed immediately after connection is
> established.

Sounds like you're running out of worker threads.  Varnish will start
bouncing clients off like that when all the worker threads are busy and
there are as many connections waiting in the queue as there are being
processed.

> varnishstat:
> 00:01:22

It's only been running for 82 seconds, so the numbers aren't going to
tell us much.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From janis.putrams at delfi.lv  Mon Jul 30 14:29:41 2007
From: janis.putrams at delfi.lv (Janis Putrams)
Date: Mon, 30 Jul 2007 17:29:41 +0300
Subject: misc connection closures
In-Reply-To: <3d62bd5f0707300558i4fe63de0ldd1886c4147e34ac@mail.gmail.com>
References: <200707301421.38786.janis.putrams@delfi.lv>
	<3d62bd5f0707300558i4fe63de0ldd1886c4147e34ac@mail.gmail.com>
Message-ID: <200707301729.42218.janis.putrams@delfi.lv>

Hi!

You were right.
ulimit -a reported max 1024 open files for the user.
Connections started to be closed when lsof reported that limit. varnish slave 
process also dies when that happens.

setting
/etc/security/limits.conf
* soft nofile 4096
* hard nofile 4096

/etc/pam.d/other
session    required   pam_limits.so

and restarting varnish solved problem.
 
tnx everybody for help. will look into 1.1 as well.

--
Janis

On Monday 30 July 2007 15:58, C. Handel wrote:
> On 7/30/07, Janis Putrams <janis.putrams at delfi.lv> wrote:
> > Though when reaching 2k req/sec we start to encounter connection
> > closures. Connection gets closed immediately after connection is
> > established.
>
> is there a ulimit from the Operating System? Number of open files/sockets?
> (As user running varnish: "ulimit -a")
>
> Greetings
>    Christoph


From james at nyi.net  Mon Jul 30 19:43:17 2007
From: james at nyi.net (James Quacinella)
Date: Mon, 30 Jul 2007 15:43:17 -0400
Subject: Configuration help
In-Reply-To: <7xtzrm70ib.fsf@iostat.e.linpro.no>
References: <7334df7b0707292347v1745376ao91ae43207b0aed32@mail.gmail.com>
	<7xtzrm70ib.fsf@iostat.e.linpro.no>
Message-ID: <46AE3F55.6050102@nyi.net>

Stig Sandbeck Mathisen wrote:
> "Steven Ciaburri" <steven at x92.net> writes:
>   
> See the zope-plone.vcl config which is included, and also available at
> http://varnish.projects.linpro.no/svn/trunk/varnish-cache/etc/zope-plone.vcl

Quick question about that config file:

# We only care about the "__ac.*" cookies, used for authentication
    if (req.http.Cookie && req.http.Cookie ~ 
"__ac(|_(name|password|persistent))=") {
        pass;
    }

Basically, that means if a request comes in with one of those Cookie 
values, its passed to the backend (since if a cookie is given, its 
assumed to be a dynamic page). Would it be possible to do something 
similar but to still cache static images but pass html. I was thinking 
something like req.url ~ "*.html" but not all html pages end with .html. 
Would it be better to use if (req.url ~ "*.(jpg|jpeg|gif)") { lookup } 
or something along those lines.

Anyone every try anything similar?

-- james


From kradziszewski at gmail.com  Mon Jul 30 20:32:57 2007
From: kradziszewski at gmail.com (Kamil Radziszewski)
Date: Mon, 30 Jul 2007 22:32:57 +0200
Subject: TxHeader and Varnish Identify
Message-ID: <46AE4AF9.8030201@gmail.com>

Is there any possibility to hide in header Varnish version ??


From des at linpro.no  Mon Jul 30 21:11:44 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 30 Jul 2007 23:11:44 +0200
Subject: TxHeader and Varnish Identify
In-Reply-To: <46AE4AF9.8030201@gmail.com> (Kamil Radziszewski's message of
	"Mon\, 30 Jul 2007 22\:32\:57 +0200")
References: <46AE4AF9.8030201@gmail.com>
Message-ID: <874pjly4tr.fsf@des.linpro.no>

Kamil Radziszewski <kradziszewski at gmail.com> writes:
> Is there any possibility to hide in header Varnish version ??

Varnish does not emit any headers containing version information.

If you mean the Via: header, the version number it contains is the
protocol version, not the software version.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rad_kam at tlen.pl  Mon Jul 30 21:31:08 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Mon, 30 Jul 2007 23:31:08 +0200
Subject: =?UTF-8?Q?WARNING:_storage_file_size_reduced_to.....?=
Message-ID: <67adc920.7a850742.46ae589c.1b60@o2.pl>

when i run varnish i get this messages: 

WARNING: storage file size reduced to 1073741824 due to system limitations
WARNING: storage file size reduced to 11756049694 (80% of available disk space)
NB: Limiting size to 2GB on 32 bit architecture to prevent running out of
address space.  Specifiy explicit size to override.

What does it mean ?


From eculp at encontacto.net  Mon Jul 30 23:48:33 2007
From: eculp at encontacto.net (eculp at encontacto.net)
Date: Mon, 30 Jul 2007 18:48:33 -0500
Subject: Small varnish 1.1 test with openrealty and joomla.
Message-ID: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>

I have set up varnish for a Joomla/OpenRealty site with a lot of  
dynamic page generation and it seems to be working fine or I have  
convinced myself that it is;)
I'm running the same site on another machine for comparison and  
checking varnishstat, etc.

I only had to change #varnishd_listen=":6081" to :80 in the  
rc.d/varnish of freebsd and add the following to  
etc/varnishd/default.vcl

          sub vcl_recv {
              if (req.request == "GET" && req.http.cookie) {
                  lookup;
              }
          }

          sub vcl_fetch {
              if (obj.http.Set-Cookie) {
                  insert;
              }
          }

to cache the joomla/OpenRealty pages.  It seems to be but I guess I  
need some assurance;)

I have a couple of questions:

   1.  Is the above the best way to cache dynamically generated CMS pages?
   2.  I am running it without vhosts.  I have tried and cannot get vhosts
       to work.  The only thing that I've found that might work would be
       the vcl routines for multiple vhost backends from the vcl man page:

          The following example shows how to support multiple sites
          running on separate backends in the same Varnish instance,
          by selecting backends based on the request URL.

       but that seems to be overkill.  What is the recommended way to
       handle apache virtual hosts?
   3.  I'm also running eaccelerator for php.  Is that a good idea?
   4.  Is there additional documentation on initial set up?  The manuals
       seem to be pretty good, they have gotten me this far.

Thanks for any and all suggestions,

ed



From james at nyi.net  Tue Jul 31 01:58:19 2007
From: james at nyi.net (James)
Date: Mon, 30 Jul 2007 21:58:19 -0400
Subject: Small varnish 1.1 test with openrealty and joomla.
In-Reply-To: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
References: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
Message-ID: <46AE973B.9040407@nyi.net>

eculp at encontacto.net wrote:
> I have a couple of questions:
>
>    1.  Is the above the best way to cache dynamically generated CMS pages?
>   

I'm trying to do something similar and it seems to be the recommended 
way to cache pages even when cookies are present.

>    2.  I am running it without vhosts.  I have tried and cannot get vhosts
>        to work.  The only thing that I've found that might work would be
>        the vcl routines for multiple vhost backends from the vcl man page:
>   

I'll let other people more experienced answer that one.

>    3.  I'm also running eaccelerator for php.  Is that a good idea?
>   

eAccelerator, or any php cache (like xcache or apc) is really a side 
issue from Varnish. I've used all of those before with not much trouble. 
I would recommend its usage.

-- james


From des at linpro.no  Tue Jul 31 06:01:51 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 31 Jul 2007 08:01:51 +0200
Subject: WARNING: storage file size reduced to.....
In-Reply-To: <67adc920.7a850742.46ae589c.1b60@o2.pl> (rad kam's message of
	"Mon\, 30 Jul 2007 23\:31\:08 +0200")
References: <67adc920.7a850742.46ae589c.1b60@o2.pl>
Message-ID: <87tzrlw1ps.fsf@des.linpro.no>

rad_kam at tlen.pl writes:
> WARNING: storage file size reduced to 1073741824 due to system limitations
> WARNING: storage file size reduced to 11756049694 (80% of available disk space)
> NB: Limiting size to 2GB on 32 bit architecture to prevent running out of
> address space.  Specifiy explicit size to override.
>
> What does it mean ?

The maximum cache size on a 32-bit machine is 2 GB (2^31 bytes); in
addition, the size of the cache in bytes has to fit in off_t.  If off_t
is 32 bits, the maximum size is 2^31-1; Varnish will shift the requested
cache size right (i.e. divide it by two) until it fits, so if you
requested 2 GB (2^31, one byte too many) you get 1 GB (2^30).

The second message is the result of a bug which causes Varnish to think
that 1 GB is more than ~14 GB (the amount of free space in the file
system on which you placed the cache file) and set the cache size to 80%
of that, or ~11 GB.

The third message is a consequence of the aforementioned bug; Varnish
notices that ~11 GB is more than 2^31-1 and sets the cache size to
2^31-1 rounded down to the nearest page.

Could you please test the attached patch?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

-------------- next part --------------
A non-text attachment was scrubbed...
Name: fssize.diff
Type: text/x-diff
Size: 1694 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070731/aaf16dda/attachment.bin>

From des at linpro.no  Tue Jul 31 06:09:53 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 31 Jul 2007 08:09:53 +0200
Subject: Small varnish 1.1 test with openrealty and joomla.
In-Reply-To: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
	(eculp@encontacto.net's message of "Mon\, 30 Jul 2007 18\:48\:33 -0500")
References: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
Message-ID: <87ps29w1ce.fsf@des.linpro.no>

eculp at encontacto.net writes:
> sub vcl_fetch {
>     if (obj.http.Set-Cookie) {
>         insert;
>     }
> }

You should strip the Set-Cookie header before inserting the object into
the cache:

sub vcl_fetch {
    if (obj.http.Set-Cookie) {
        remove obj.http.Set-Cookie;
    }
}

(the "insert" is implicit)

> 1.  Is the above the best way to cache dynamically generated CMS
> pages?

It depends on the CMS, really :)

> 2.  I am running it without vhosts.  I have tried and cannot get vhosts
>     to work.  The only thing that I've found that might work would be
>     the vcl routines for multiple vhost backends from the vcl man page:
>
>        The following example shows how to support multiple sites
>        running on separate backends in the same Varnish instance,
>        by selecting backends based on the request URL.
>
>     but that seems to be overkill.  What is the recommended way to
>     handle apache virtual hosts?

The example you mention illustrates how to cache multiple virtual hosts
served by *separate* backends.  If all your virtual hosts are on the
same backend, you shouldn't need to do anything.

> 3.  I'm also running eaccelerator for php.  Is that a good idea?

I'm not familiar with eAccelerator.  If it's a byte-code cache, it will
help speed up cache misses.  If it only caches finished pages, it's
probably redundant but not harmful.

> 4.  Is there additional documentation on initial set up?  The manuals
>     seem to be pretty good, they have gotten me this far.

I keep intending to write more documentation, but like everyone else's
my days only have 24 hours :)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Jul 31 06:12:08 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 31 Jul 2007 08:12:08 +0200
Subject: Memory leak in varnishncsa ?
In-Reply-To: <200707262211.00781.fredrik.neij@teneco.se> (Fredrik Neij's
	message of "Thu\, 26 Jul 2007 22\:10\:58 +0200")
References: <200707262211.00781.fredrik.neij@teneco.se>
Message-ID: <87lkcxw18n.fsf@des.linpro.no>

Fredrik Neij <fredrik.neij at teneco.se> writes:
> root at lb:/tmp# varnishncsa > v.log
> Assert error in trimline(), varnishncsa.c line 160:
>   Condition(p != NULL) not true.
>   errno = 12 (Cannot allocate memory)
> Aborted
>
> It just keeps growing until it's out of memory.

I've looked long and hard at the code, and while I did identify some
memory leaks, they were all in code paths related to the -b option,
which you don't seem to use.

Could you either run varnishncsa under valgrind and send me the results,
or send me the log file so I can try to track down the leak?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From rad_kam at tlen.pl  Tue Jul 31 10:11:14 2007
From: rad_kam at tlen.pl (rad_kam at tlen.pl)
Date: Tue, 31 Jul 2007 12:11:14 +0200
Subject: =?UTF-8?Q?Re:_WARNING:_storage_file_size_reduced_to.....?=
Message-ID: <1ab5fe22.59bd126d.46af0ac2.c6137@o2.pl>

But this issue i get every time even if i set the storage size 5G, 4G 3G even 10G...everytime 

PS: The patch works ;)


From eculp at encontacto.net  Tue Jul 31 10:47:38 2007
From: eculp at encontacto.net (eculp at encontacto.net)
Date: Tue, 31 Jul 2007 05:47:38 -0500
Subject: Small varnish 1.1 test with openrealty and joomla.
In-Reply-To: <87ps29w1ce.fsf@des.linpro.no>
References: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
	<87ps29w1ce.fsf@des.linpro.no>
Message-ID: <20070731054738.a4gp7ri2ow80gcok@intranet.encontacto.net>

Quoting Dag-Erling Sm?rgrav <des at linpro.no>:

> eculp at encontacto.net writes:
>> sub vcl_fetch {
>>     if (obj.http.Set-Cookie) {
>>         insert;
>>     }
>> }
>
> You should strip the Set-Cookie header before inserting the object into
> the cache:
>
> sub vcl_fetch {
>     if (obj.http.Set-Cookie) {
>         remove obj.http.Set-Cookie;
>     }
> }

Substituted the above for the "insert"  Thanks.

> (the "insert" is implicit)
>
>> 1.  Is the above the best way to cache dynamically generated CMS
>> pages?
>
> It depends on the CMS, really :)

That is what I was afraid of. ;)

>> 2.  I am running it without vhosts.  I have tried and cannot get vhosts
>>     to work.  The only thing that I've found that might work would be
>>     the vcl routines for multiple vhost backends from the vcl man page:
>>
>>        The following example shows how to support multiple sites
>>        running on separate backends in the same Varnish instance,
>>        by selecting backends based on the request URL.
>>
>>     but that seems to be overkill.  What is the recommended way to
>>     handle apache virtual hosts?
>
> The example you mention illustrates how to cache multiple virtual hosts
> served by *separate* backends.  If all your virtual hosts are on the
> same backend, you shouldn't need to do anything.

That is what I thought so it must be something in my apache vhost  
configuration.  It really doesn't make sense yet, to me.

At least I now know that it does or should work.  That is the  
important thing for me for now.

>> 3.  I'm also running eaccelerator for php.  Is that a good idea?
>
> I'm not familiar with eAccelerator.  If it's a byte-code cache, it will
> help speed up cache misses.  If it only caches finished pages, it's
> probably redundant but not harmful.
>
>> 4.  Is there additional documentation on initial set up?  The manuals
>>     seem to be pretty good, they have gotten me this far.
>
> I keep intending to write more documentation, but like everyone else's
> my days only have 24 hours :)

If I can get it going, the doc's that exist are pretty good.  I think  
that a solid varnish is more important that the doc's for now.

Thanks for all your help and your work on varnish.

ed

> DES
> --
> Dag-Erling Sm?rgrav
> Senior Software Developer
> Linpro AS - www.linpro.no
>



From eculp at encontacto.net  Tue Jul 31 11:30:44 2007
From: eculp at encontacto.net (eculp at encontacto.net)
Date: Tue, 31 Jul 2007 06:30:44 -0500
Subject: Small varnish 1.1 test with openrealty and joomla.
In-Reply-To: <46AE973B.9040407@nyi.net>
References: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
	<46AE973B.9040407@nyi.net>
Message-ID: <20070731063044.nse9d3phokogkcs4@intranet.encontacto.net>

Quoting James <james at nyi.net>:

> eculp at encontacto.net wrote:
>> I have a couple of questions:
>>
>>   1.  Is the above the best way to cache dynamically generated CMS pages?
>>
>
> I'm trying to do something similar and it seems to be the  
> recommended way to cache pages even when cookies are present.
>
>>   2.  I am running it without vhosts.  I have tried and cannot get vhosts
>>       to work.  The only thing that I've found that might work would be
>>       the vcl routines for multiple vhost backends from the vcl man page:
>>
>
> I'll let other people more experienced answer that one.

If you do happen to get Apache Vhosts running I would certainly  
appreciate your feedback on how you did it.  In my case, the only way  
it seems to work is with the example in the man vcl page:

   The following example shows how to support multiple sites running on sep-
   arate backends in the same Varnish instance, by selecting backends based
   on the request URL.

and then directing them to the vhost url which is actually the same  
backend.  Of course I may somehow be shooting myself in the foot by  
doing this, I'm just begining to test.

The default configuration for varnish has:

# backend default {
#       set backend.host = "127.0.0.1";
#       set backend.port = "8080";
# }

And as far as I can tell kills the the apache vhost directive by  
substituting local host for the vdomain  What I tried was to set the  
default backend to

set backend.host = req.http.host;

but the sintax is incorrect as I expected.  I need to scratch arround  
to figure out a bit about vcl.

Comments?

>
>>   3.  I'm also running eaccelerator for php.  Is that a good idea?
>>
>
> eAccelerator, or any php cache (like xcache or apc) is really a side  
> issue from Varnish. I've used all of those before with not much  
> trouble. I would recommend its usage.

It works, fine, so I see no reason to remove it for now, either.  I  
use apc  on other servers too. I'm trying to decide which I like best.

Thanks again and have a great day,

ed


From des at linpro.no  Tue Jul 31 16:09:35 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 31 Jul 2007 18:09:35 +0200
Subject: Small varnish 1.1 test with openrealty and joomla.
In-Reply-To: <20070731054738.a4gp7ri2ow80gcok@intranet.encontacto.net>
	(eculp@encontacto.net's message of "Tue\, 31 Jul 2007 05\:47\:38 -0500")
References: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
	<87ps29w1ce.fsf@des.linpro.no>
	<20070731054738.a4gp7ri2ow80gcok@intranet.encontacto.net>
Message-ID: <87d4y8wo5c.fsf@des.linpro.no>

eculp at encontacto.net writes:
> Quoting Dag-Erling Sm?rgrav <des at linpro.no>:
> > The example you mention illustrates how to cache multiple virtual hosts
> > served by *separate* backends.  If all your virtual hosts are on the
> > same backend, you shouldn't need to do anything.
> That is what I thought so it must be something in my apache vhost
> configuration.  It really doesn't make sense yet, to me.

Is Varnish passing the correct Host: header to Apache?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From activeco at home.nl  Tue Jul 31 10:56:33 2007
From: activeco at home.nl (s)
Date: Tue, 31 Jul 2007 12:56:33 +0200
Subject: Rewriting HTML?
Message-ID: <1185879394.2303.1.camel@localhost.localdomain>

Does Varnish support rewriting HTML code, especially absolute links in
it?



From des at linpro.no  Tue Jul 31 16:15:09 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 31 Jul 2007 18:15:09 +0200
Subject: Rewriting HTML?
In-Reply-To: <1185879394.2303.1.camel@localhost.localdomain>
	(activeco@home.nl's message of "Tue\, 31 Jul 2007 12\:56\:33 +0200")
References: <1185879394.2303.1.camel@localhost.localdomain>
Message-ID: <87wswgv9bm.fsf@des.linpro.no>

s <activeco at home.nl> writes:
> Does Varnish support rewriting HTML code, especially absolute links in
> it?

No.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From phk at phk.freebsd.dk  Tue Jul 31 16:19:20 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 31 Jul 2007 16:19:20 +0000
Subject: Rewriting HTML? 
In-Reply-To: Your message of "Tue, 31 Jul 2007 12:56:33 +0200."
	<1185879394.2303.1.camel@localhost.localdomain> 
Message-ID: <20965.1185898760@critter.freebsd.dk>

In message <1185879394.2303.1.camel at localhost.localdomain>, s writes:
>Does Varnish support rewriting HTML code, especially absolute links in
>it?

No.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From eculp at encontacto.net  Tue Jul 31 17:16:49 2007
From: eculp at encontacto.net (eculp at encontacto.net)
Date: Tue, 31 Jul 2007 12:16:49 -0500
Subject: Small varnish 1.1 test with openrealty and joomla.
In-Reply-To: <87d4y8wo5c.fsf@des.linpro.no>
References: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
	<87ps29w1ce.fsf@des.linpro.no>
	<20070731054738.a4gp7ri2ow80gcok@intranet.encontacto.net>
	<87d4y8wo5c.fsf@des.linpro.no>
Message-ID: <20070731121649.zjtnms5us0owwgo4@intranet.encontacto.net>

Quoting Dag-Erling Sm?rgrav <des at linpro.no>:

> eculp at encontacto.net writes:
>> Quoting Dag-Erling Sm?rgrav <des at linpro.no>:
>> > The example you mention illustrates how to cache multiple virtual hosts
>> > served by *separate* backends.  If all your virtual hosts are on the
>> > same backend, you shouldn't need to do anything.
>> That is what I thought so it must be something in my apache vhost
>> configuration.  It really doesn't make sense yet, to me.
>
> Is Varnish passing the correct Host: header to Apache?

AFAIK, it is passing what I am/was telling it in default.vcl.  The  
localhost trumps the orginal named base vhost for apache and that is  
what my configuration file was asking for.  So it is correct but not  
what I'm trying to do;)

backend default {
   set backend.host = "127.0.0.1";
   set backend.port = "8080";
}

What I am trying to do is to get it to pass something like:

   set backend.host = req.http.host;

Which, from my limited perspective, would be a solution for this vhost  
issue although there are probably others and hopefully better and/or  
simpler options.

Thanks again, DES, for spending time on this somewhat trivial issue  
that I'm having with my learning curve.

ed

>
> DES
> --
> Dag-Erling Sm?rgrav
> Senior Software Developer
> Linpro AS - www.linpro.no
>



From des at linpro.no  Tue Jul 31 20:12:26 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 31 Jul 2007 22:12:26 +0200
Subject: Small varnish 1.1 test with openrealty and joomla.
In-Reply-To: <20070731121649.zjtnms5us0owwgo4@intranet.encontacto.net>
	(eculp@encontacto.net's message of "Tue\, 31 Jul 2007 12\:16\:49 -0500")
References: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
	<87ps29w1ce.fsf@des.linpro.no>
	<20070731054738.a4gp7ri2ow80gcok@intranet.encontacto.net>
	<87d4y8wo5c.fsf@des.linpro.no>
	<20070731121649.zjtnms5us0owwgo4@intranet.encontacto.net>
Message-ID: <87sl74uyc5.fsf@des.linpro.no>

eculp at encontacto.net writes:
> Quoting Dag-Erling Sm?rgrav <des at linpro.no>:
> > Is Varnish passing the correct Host: header to Apache?
> AFAIK, it is passing what I am/was telling it in default.vcl.

I'm not asking you to guess or speculate; I'm asking you to check your
logs.  Varnish does *not* use backend.host for the Host: header, it uses
whatever it got from the client.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ric at digitalmarbles.com  Tue Jul 31 20:20:58 2007
From: ric at digitalmarbles.com (Ricardo Newbery)
Date: Tue, 31 Jul 2007 13:20:58 -0700
Subject: Small varnish 1.1 test with openrealty and joomla.
In-Reply-To: <20070731121649.zjtnms5us0owwgo4@intranet.encontacto.net>
References: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
	<87ps29w1ce.fsf@des.linpro.no>
	<20070731054738.a4gp7ri2ow80gcok@intranet.encontacto.net>
	<87d4y8wo5c.fsf@des.linpro.no>
	<20070731121649.zjtnms5us0owwgo4@intranet.encontacto.net>
Message-ID: <F6795884-83D5-40A0-8F60-64CB662B5C7E@digitalmarbles.com>


On Jul 31, 2007, at 10:16 AM, eculp at encontacto.net wrote:

> Quoting Dag-Erling Sm?rgrav <des at linpro.no>:
>
>> eculp at encontacto.net writes:
>>> Quoting Dag-Erling Sm?rgrav <des at linpro.no>:
>>>> The example you mention illustrates how to cache multiple  
>>>> virtual hosts
>>>> served by *separate* backends.  If all your virtual hosts are on  
>>>> the
>>>> same backend, you shouldn't need to do anything.
>>> That is what I thought so it must be something in my apache vhost
>>> configuration.  It really doesn't make sense yet, to me.
>>
>> Is Varnish passing the correct Host: header to Apache?
>
> AFAIK, it is passing what I am/was telling it in default.vcl.  The
> localhost trumps the orginal named base vhost for apache and that is
> what my configuration file was asking for.  So it is correct but not
> what I'm trying to do;)
>
> backend default {
>    set backend.host = "127.0.0.1";
>    set backend.port = "8080";
> }
>
> What I am trying to do is to get it to pass something like:
>
>    set backend.host = req.http.host;
>
> Which, from my limited perspective, would be a solution for this vhost
> issue although there are probably others and hopefully better and/or
> simpler options.
>
> Thanks again, DES, for spending time on this somewhat trivial issue
> that I'm having with my learning curve.


The following doesn't work?

vcl_recv {
         set backend.host = req.http.Host;
}

or if it's still missing the Host header, does the following work:

vcl_miss {
         set bereq.http.Host = req.http.Host;
}

and ditto for vcl_pipe and vcl_pass

Ric




From ric at digitalmarbles.com  Tue Jul 31 20:23:55 2007
From: ric at digitalmarbles.com (Ricardo Newbery)
Date: Tue, 31 Jul 2007 13:23:55 -0700
Subject: Small varnish 1.1 test with openrealty and joomla.
In-Reply-To: <F6795884-83D5-40A0-8F60-64CB662B5C7E@digitalmarbles.com>
References: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
	<87ps29w1ce.fsf@des.linpro.no>
	<20070731054738.a4gp7ri2ow80gcok@intranet.encontacto.net>
	<87d4y8wo5c.fsf@des.linpro.no>
	<20070731121649.zjtnms5us0owwgo4@intranet.encontacto.net>
	<F6795884-83D5-40A0-8F60-64CB662B5C7E@digitalmarbles.com>
Message-ID: <F511CF61-A94F-46F2-B9B9-EAEE41A13018@digitalmarbles.com>


On Jul 31, 2007, at 1:20 PM, Ricardo Newbery wrote:

>
> The following doesn't work?
>
> vcl_recv {
>          set backend.host = req.http.Host;
> }
>
> or if it's still missing the Host header, does the following work:
>
> vcl_miss {
>          set bereq.http.Host = req.http.Host;
> }
>
> and ditto for vcl_pipe and vcl_pass
>
> Ric


Scratch that.  DES says that Host already comes from the request.

Ric





From eculp at encontacto.net  Tue Jul 31 22:19:19 2007
From: eculp at encontacto.net (eculp at encontacto.net)
Date: Tue, 31 Jul 2007 17:19:19 -0500
Subject: Small varnish 1.1 test with openrealty and joomla.
In-Reply-To: <87sl74uyc5.fsf@des.linpro.no>
References: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
	<87ps29w1ce.fsf@des.linpro.no>
	<20070731054738.a4gp7ri2ow80gcok@intranet.encontacto.net>
	<87d4y8wo5c.fsf@des.linpro.no>
	<20070731121649.zjtnms5us0owwgo4@intranet.encontacto.net>
	<87sl74uyc5.fsf@des.linpro.no>
Message-ID: <20070731171919.wsgne57as8okgow8@intranet.encontacto.net>

Quoting Dag-Erling Sm?rgrav <des at linpro.no>:

> eculp at encontacto.net writes:
>> Quoting Dag-Erling Sm?rgrav <des at linpro.no>:
>> > Is Varnish passing the correct Host: header to Apache?
>> AFAIK, it is passing what I am/was telling it in default.vcl.
>
> I'm not asking you to guess or speculate; I'm asking you to check your
> logs.  Varnish does *not* use backend.host for the Host: header, it uses
> whatever it got from the client.



This is from the apache log:

127.0.0.1 - - [31/Jul/2007:16:33:22 -0500] "GET /favicon.ico HTTP/1.1"  
404 209 "http://nuevo.ecomania.info/" "Opera/9.22 (X11; Linux i686; U;  
en)"

and displays the index in the root directory under which are all the  
virtual hosts such as nuevo.encomania.info.

The varnishlog -r /var/log/varnish.log is attached (4k) and again my  
interpretation is the same.

If I change the default

backend default {
   set backend.host = "127.0.0.1";
   set backend.port = "8080";
}

to

backend nuevo {
         set backend.host = "nuevo.ecomania.info";
         set backend.port = "8080";
}

with all else the same, it works as expected because  
http://nuevo.ecomania.info is configured as an apache vhost.  It also  
works for this one site if I DO NOT configure virtual hosts.  Maybe  
I'm barking up the wrong tree again.  If so, I apologize.  There is  
still a lot that i don't understand.

Thanks again for your help,

ed
.
>
> DES
> --
> Dag-Erling Sm?rgrav
> Senior Software Developer
> Linpro AS - www.linpro.no
>

-------------- next part --------------
    0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1185919372
    0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1185919375
   15 SessionOpen  c 189.129.12.64 59655
   15 Debug        c "/#nuevo.ecomania.info#"
   15 ReqStart     c 189.129.12.64 59655 1115880254
   15 RxRequest    c GET
   15 RxURL        c /
   15 RxProtocol   c HTTP/1.1
   15 RxHeader     c User-Agent: Opera/9.22 (X11; Linux i686; U; en)
   15 RxHeader     c Host: nuevo.ecomania.info
   15 RxHeader     c Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
   15 RxHeader     c Accept-Language: es,en;q=0.9
   15 RxHeader     c Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1
   15 RxHeader     c Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
   15 RxHeader     c Cookie: __utma=233483905.1461977794.1185903227.1185903227.1185903227.1; __utmc=233483905; __utmz=233483905.1185903227.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); __utma=108653974.1335907909.1185903226.1185903226.1185903226.1; __utmc=108653974; __ut
   15 RxHeader     c Cookie2: $Version=1
   15 RxHeader     c Cache-Control: no-cache
   15 RxHeader     c Connection: Keep-Alive, TE
   15 RxHeader     c TE: deflate, gzip, chunked, identity, trailers
   15 VCL_call     c recv
   15 VCL_return   c lookup
   15 VCL_call     c hash
   15 VCL_return   c hash
   15 Hit          c 1115880252
   15 VCL_call     c hit
   15 VCL_return   c deliver
   15 Length       c 2187
   15 VCL_call     c deliver
   15 VCL_return   c deliver
   15 TxProtocol   c HTTP/1.1
   15 TxStatus     c 200
   15 TxResponse   c OK
   15 TxHeader     c Date: Tue, 31 Jul 2007 22:01:00 GMT
   15 TxHeader     c Server: Apache/2.2.4 (FreeBSD) mod_ssl/2.2.4 OpenSSL/0.9.8e DAV/2 PHP/5.2.3 with Suhosin-Patch
   15 TxHeader     c Content-Type: text/html
   15 TxHeader     c Content-Length: 2187
   15 TxHeader     c X-Varnish: 1115880254 1115880252
   15 TxHeader     c Age: 118
   15 TxHeader     c Via: 1.1 varnish
   15 TxHeader     c Connection: keep-alive
   15 ReqEnd       c 1115880254 1185919378.468985558 1185919378.469070435 0.084838629 0.000037909 0.000046968
    0 StatAddr       189.129.12.64 0 1776 4 7 0 0 4 2021 9375
   15 Debug        c "/favicon.ico#nuevo.ecomania.info#"
   15 ReqStart     c 189.129.12.64 59655 1115880255
   15 RxRequest    c GET
   15 RxURL        c /favicon.ico
   15 RxProtocol   c HTTP/1.1
   15 RxHeader     c User-Agent: Opera/9.22 (X11; Linux i686; U; en)
   15 RxHeader     c Host: nuevo.ecomania.info
   15 RxHeader     c Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
   15 RxHeader     c Accept-Language: es,en;q=0.9
   15 RxHeader     c Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1
   15 RxHeader     c Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
   15 RxHeader     c Referer: http://nuevo.ecomania.info/
   15 RxHeader     c Cache-Control: no-cache
   15 RxHeader     c Connection: Keep-Alive, TE
   15 RxHeader     c TE: deflate, gzip, chunked, identity, trailers
   15 VCL_call     c recv
   15 VCL_return   c lookup
   15 VCL_call     c hash
   15 VCL_return   c hash
   15 Hit          c 1115880253
   15 VCL_call     c hit
   15 VCL_return   c deliver
   15 Length       c 209
   15 VCL_call     c deliver
   15 VCL_return   c deliver
   15 TxProtocol   c HTTP/1.1
   15 TxStatus     c 404
   15 TxResponse   c Not Found
   15 TxHeader     c Date: Tue, 31 Jul 2007 22:01:00 GMT
   15 TxHeader     c Server: Apache/2.2.4 (FreeBSD) mod_ssl/2.2.4 OpenSSL/0.9.8e DAV/2 PHP/5.2.3 with Suhosin-Patch
   15 TxHeader     c Content-Type: text/html; charset=iso-8859-1
   15 TxHeader     c Content-Length: 209
   15 TxHeader     c X-Varnish: 1115880255 1115880253
   15 TxHeader     c Age: 118
   15 TxHeader     c Via: 1.1 varnish
   15 TxHeader     c Connection: keep-alive
   15 ReqEnd       c 1115880255 1185919378.646611929 1185919378.646687508 0.177541494 0.000034094 0.000041485
    0 StatAddr       189.129.12.64 0 1776 4 8 0 0 4 2332 9584
    0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1185919378
    0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1185919381
   15 SessionClose c timeout
   15 StatSess     c 189.129.12.64 59655 nan 1 2 0 0 0 596 2396
    0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1185919384
    0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1185919387
    0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1185919390

From eculp at encontacto.net  Tue Jul 31 22:26:19 2007
From: eculp at encontacto.net (eculp at encontacto.net)
Date: Tue, 31 Jul 2007 17:26:19 -0500
Subject: Small varnish 1.1 test with openrealty and joomla.
	SOLVED
In-Reply-To: <20070731171919.wsgne57as8okgow8@intranet.encontacto.net>
References: <20070730184833.vx2r1t487480o4w4@intranet.encontacto.net>
	<87ps29w1ce.fsf@des.linpro.no>
	<20070731054738.a4gp7ri2ow80gcok@intranet.encontacto.net>
	<87d4y8wo5c.fsf@des.linpro.no>
	<20070731121649.zjtnms5us0owwgo4@intranet.encontacto.net>
	<87sl74uyc5.fsf@des.linpro.no>
	<20070731171919.wsgne57as8okgow8@intranet.encontacto.net>
Message-ID: <20070731172619.mn1oqde7i0wk0k00@intranet.encontacto.net>

Quoting eculp at encontacto.net:

> Quoting Dag-Erling Sm?rgrav <des at linpro.no>:
>
>> eculp at encontacto.net writes:
>>> Quoting Dag-Erling Sm?rgrav <des at linpro.no>:
>>>> Is Varnish passing the correct Host: header to Apache?
>>> AFAIK, it is passing what I am/was telling it in default.vcl.
>>
>> I'm not asking you to guess or speculate; I'm asking you to check your
>> logs.  Varnish does *not* use backend.host for the Host: header, it uses
>> whatever it got from the client.
>
>
>
> This is from the apache log:
>
> 127.0.0.1 - - [31/Jul/2007:16:33:22 -0500] "GET /favicon.ico  
> HTTP/1.1"  404 209 "http://nuevo.ecomania.info/" "Opera/9.22 (X11;  
> Linux i686; U;  en)"

Hi DES,

At the moment that I sent this email I caught the issue. The solution  
is to NOT use 127.0.0.1 but to use the NameVirtualHost 189.129.5.82 in  
the default.vcl.

This works for me:

backend default {
   set backend.host = "189.129.5.82";
   set backend.port = "8080";
}

Thanks again DES for helping me pull my head out of the sand "or where  
ever it was;)

ed
>
> and displays the index in the root directory under which are all the  
>  virtual hosts such as nuevo.encomania.info.
>
> The varnishlog -r /var/log/varnish.log is attached (4k) and again my  
>  interpretation is the same.
>
> If I change the default
>
> backend default {
>   set backend.host = "127.0.0.1";
>   set backend.port = "8080";
> }
>
> to
>
> backend nuevo {
>         set backend.host = "nuevo.ecomania.info";
>         set backend.port = "8080";
> }
>
> with all else the same, it works as expected because   
> http://nuevo.ecomania.info is configured as an apache vhost.  It  
> also  works for this one site if I DO NOT configure virtual hosts.   
> Maybe  I'm barking up the wrong tree again.  If so, I apologize.   
> There is  still a lot that i don't understand.
>
> Thanks again for your help,
>
> ed
> .
>>
>> DES
>> --
>> Dag-Erling Sm?rgrav
>> Senior Software Developer
>> Linpro AS - www.linpro.no
>>
>
>



From activeco at home.nl  Tue Jul 31 22:12:58 2007
From: activeco at home.nl (s)
Date: Wed, 01 Aug 2007 00:12:58 +0200
Subject: Rewriting HTML?
In-Reply-To: <20965.1185898760@critter.freebsd.dk>
References: <20965.1185898760@critter.freebsd.dk>
Message-ID: <1185919979.2303.16.camel@localhost.localdomain>

Ouch.
Pitty, I have to go back to apache regarding it.
Thanks anyway and good luck with the project.

On Tue, 2007-07-31 at 16:19 +0000, Poul-Henning Kamp wrote:
> In message <1185879394.2303.1.camel at localhost.localdomain>, s writes:
> >Does Varnish support rewriting HTML code, especially absolute links in
> >it?
> 
> No.
>