From denis at startsiden.no  Tue May  1 08:36:13 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Tue, 1 May 2007 10:36:13 +0200 (CEST)
Subject: multiple backend + apache bottleneck
In-Reply-To: <4635DDB2.1050400@ac-montpellier.fr>
Message-ID: <16282077.181451178008573692.JavaMail.root@ms1.startsiden.no>

----- jean-marc pouchoulon <jean-marc.pouchoulon at ac-montpellier.fr> wrote:
> Hello all ,
> We are using zope/nuxeo cps  with 8 zeo clients and I test varnish 1.3
> in place of squid 2.6.
> squid uses 8 zeo peers and if I've understood well varnish can  have 
> "only one" host per backend.
> is there is a way to load balance upon multiples hosts with varnish ?

There are multiple ways to achieve this, but nothing is built into varnish (yet atleast). But there are multiple solutions out there you could utilize, either proprietary appliances or software (Alteon/Radware/Cisco/Zeus XTM) or open source solutions. I would guess the latter is most interesting for you, and I have heard good things about perlbal [1]. It is used by a number of high profile, high load sites. 

We are currently running a setup where the pool is divided like this:

           Alteon LB 
           /       \
VarnishServer1     VarnishServer2
       |                  |
Apache2+mod_php1   Apache2+mod_php2

This caters to multiple failure situations; whenever Apache1 goes down Varnish2+Apache2 will serve the load and vice versa. If Varnish1 or Varnish2 goes down, the Alteon will direct traffic straight at the relevant backend (through a backup server config setting), but with a lot less weight than the remaining VarnishServer.

Ideally we would like to build an even more efficient setup, but there are certain limitations of the LB Switch OS we are currently running that prevent this as of now.

The setup I described above is probably not suited to your application though, this was more of an elaboration on one way to setup loadbalancing with Varnish.


[1] http://www.danga.com/perlbal/

-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From des at linpro.no  Tue May  1 13:15:22 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Tue, 01 May 2007 15:15:22 +0200
Subject: Varnishlog -o bug
In-Reply-To: <ujr7is5ie5b.fsf@false.linpro.no> (Dag-Erling
	=?iso-8859-1?Q?Sm=F8rgrav's?= message
	of "Sat, 21 Apr 2007 23:57:52 +0200")
References: <20070420125021.GA75316@fupp.net> <ujr7is5ie5b.fsf@false.linpro.no>
Message-ID: <ujr1wi0wuqd.fsf@false.linpro.no>

des at linpro.no (Dag-Erling Sm?rgrav) writes:
> Anders Nordby <anders at fupp.net> writes:
> > When using varnishlog -o, log entries for cache misses appear on the
> > same line as the preceding log entry:
> It's not just cache misses - it appears to happen whenever two
> VCL_call entries are logged in succession.  I noticed it on friday,
> but haven't had time to look into it yet.

The attached patch should fix it.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: varnishlog.diff
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070501/eca52a47/attachment.ksh>

From perbu at linpro.no  Thu May  3 18:39:38 2007
From: perbu at linpro.no (Per Andreas Buer)
Date: Thu, 03 May 2007 20:39:38 +0200
Subject: multiple backend + apache bottleneck
In-Reply-To: <4635DDB2.1050400@ac-montpellier.fr>
References: <4635DDB2.1050400@ac-montpellier.fr>
Message-ID: <463A2C6A.3040101@linpro.no>

Hi,

jean-marc pouchoulon skrev:
> We are using zope/nuxeo cps  with 8 zeo clients and I test varnish 1.3 
> in place of squid 2.6.
> squid uses 8 zeo peers and if I've understood well varnish can  have 
> "only one" host per backend.
> is there is a way to load balance upon multiples hosts with varnish ?

Do you have a load balancer of choice? You might want to check out pound.


Per.


From jean-marc.pouchoulon at ac-montpellier.fr  Fri May  4 04:11:25 2007
From: jean-marc.pouchoulon at ac-montpellier.fr (jean-marc pouchoulon)
Date: Fri, 04 May 2007 06:11:25 +0200
Subject: multiple backend + apache bottleneck
In-Reply-To: <463A2C6A.3040101@linpro.no>
References: <4635DDB2.1050400@ac-montpellier.fr> <463A2C6A.3040101@linpro.no>
Message-ID: <463AB26D.5050309@ac-montpellier.fr>


>
> Do you have a load balancer of choice? You might want to check out pound.
>
I use now for two days. I will follow this with care:

cisco css (deflate compression ) + 2 varnish 1.0.3  + 2 apache 
(mod_rewrite + mod_proxy_balance) + 8 zeo.

to avoid apache "bottleneck" ( now cisco css and network are the 
"bottlenecks" ... deflate heavily uses css compression card)

I tried to use pound instead of apache balance  but I met some problems 
I didn't understand.

Thanks for all your answers.


jean-marc


"longue vie ? varnish"
>
> Per.




From mramos at co.sapo.pt  Fri May  4 15:14:37 2007
From: mramos at co.sapo.pt (Marco Ramos)
Date: Fri, 04 May 2007 16:14:37 +0100
Subject: multiple backend + apache bottleneck
In-Reply-To: <463A2C6A.3040101@linpro.no>
References: <4635DDB2.1050400@ac-montpellier.fr> <463A2C6A.3040101@linpro.no>
Message-ID: <1178291677.29318.36.camel@supernova>


Hi,

On Thu, 2007-05-03 at 20:39 +0200, Per Andreas Buer wrote:
> Hi,
> 
> jean-marc pouchoulon skrev:
> > We are using zope/nuxeo cps  with 8 zeo clients and I test varnish 1.3 
> > in place of squid 2.6.
> > squid uses 8 zeo peers and if I've understood well varnish can  have 
> > "only one" host per backend.
> > is there is a way to load balance upon multiples hosts with varnish ?
> 
> Do you have a load balancer of choice? You might want to check out pound.

As a load balancer, you can also try perlbal.

Marco Ramos

> 
> 
> Per.
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
> 



From jean-marc.pouchoulon at ac-montpellier.fr  Fri May  4 16:56:14 2007
From: jean-marc.pouchoulon at ac-montpellier.fr (jean-marc pouchoulon)
Date: Fri, 04 May 2007 18:56:14 +0200
Subject: no cache from varnish if max-age=0 + meaning of insert_pass
Message-ID: <463B65AE.5020705@ac-montpellier.fr>

Helo all,

I set max-age=0 and s-max-age=600 in the zope caching policy manager and 
varnish don't cache any content ( except if I force
obj.ttl = 10m in vcl_fetch.)

if max-age=1 varnish is caching the content.

is it the considering behaviour for varnish ? when varnish is supposed 
to cache a content ?

can someone explain the meaning of insert_pass , the difference between 
pass and pipe ?

thanks

jean-marc





From des at linpro.no  Sat May  5 12:49:21 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Sat, 05 May 2007 14:49:21 +0200
Subject: no cache from varnish if max-age=0 + meaning of insert_pass
In-Reply-To: <463B65AE.5020705@ac-montpellier.fr> (jean-marc pouchoulon's
	message of "Fri, 04 May 2007 18:56:14 +0200")
References: <463B65AE.5020705@ac-montpellier.fr>
Message-ID: <ujrvef7juzy.fsf@false.linpro.no>

jean-marc pouchoulon <jean-marc.pouchoulon at ac-montpellier.fr> writes:
> I set max-age=0 and s-max-age=600 in the zope caching policy manager
> and varnish don't cache any content ( except if I force
> obj.ttl = 10m in vcl_fetch.)
>
> if max-age=1 varnish is caching the content.
>
> is it the considering behaviour for varnish ? when varnish is supposed
> to cache a content ?

Varnish currently does not know about s-maxage.  Please try the
attached patch and let me know if it produces the desired result.

> can someone explain the meaning of insert_pass , the difference
> between pass and pipe ?

Check out trunk; insert_pass no longer exists, and pass and pipe are
fairly well documented in the vcl(7) man page.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: s-maxage.diff
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070505/c998683c/attachment.ksh>

From ltning at anduin.net  Mon May  7 16:50:32 2007
From: ltning at anduin.net (=?ISO-8859-1?Q?Eirik_=D8verby?=)
Date: Mon, 7 May 2007 18:50:32 +0200
Subject: Panics/reboots with Varnish
Message-ID: <2669B6F0-E233-41B5-9D16-7EAF6A2E17C0@anduin.net>

Hi all,

I'm running a server (fbsd-amd64, 6.2-STABLE) with a bunch of jails  
(~10). A couple of these are seeing pretty heavy HTTP traffic, so I  
threw in a varnishd in each of the two main offenders, each using the  
httpd in the same jail as back-end. Then I use pf on the host to NAT  
incoming requests to <jail>:http to <jail>:varnishdport .

Now this does indeed lessen the load on the server quite  
dramatically, and also leads to positive reports from users whenever  
this is switched on. I can globally enable/disable this with pfctl -e/ 
pfctl -d, or by modifying+reloading pf config files. In other words -  
when it works, it works great.

Problem: Whenever this is in effect, the box rarely stays up for >48  
hours. Without this in effect, it can stay up for >30 days. I've been  
playing with this since the day varnish 1.0 was released, and from  
what I can tell this is consistent behavior.

I don't have any coredumps - it appears as if the server is just  
booting. I do have a core dump from another situation (where I used a  
mismatched raid controller driver, causing a known panic), so I know  
the whole dump stuff works as it should. I have also swapped memory  
with no change in behavior.

Does anyone have any idea what can be causing this? Which edge cases  
might be touched by varnish that I'm not seeing elsewhere? I can see  
the box is swapping a bit from time to time, but nothing dramatic,  
and that is as it should be, just the kernel doing its job.

Thanks,
/Eirik


From james at nyi.net  Mon May  7 18:16:09 2007
From: james at nyi.net (James Quacinella)
Date: Mon, 07 May 2007 14:16:09 -0400
Subject: VCL and Non-Existent Backend Objects
Message-ID: <463F6CE9.1080206@nyi.net>

Hey,

I was contemplating a setup such as this:

1) Client requests xml file from varnish cache
2) If it exists, fine; else check backend file server running lighttpd
3) If it exists on the backend, again no problem; however, if it doesn't 
exist on the lighttpd backend, I need varnish to check another (dynamic) 
backend, which will dynamically generate it and save it on the lightttpd 
backend for future purposes.

Does anyone have any ideas on how to implement that via VCL? I would 
imagine I would need to be able to see the response headers from the 
original backend, to see if a 404 is generated (like here 
http://varnish.projects.linpro.no/ticket/88).

Thanks!
-- james


From ask at develooper.com  Mon May  7 23:14:09 2007
From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Mon, 7 May 2007 16:14:09 -0700
Subject: don't daemonize option
Message-ID: <D4F0C23D-A50A-49D0-A169-593C53D2FFD3@develooper.com>


I usually run services like varnishd under daemontools/supervise.  I  
realize that varnish already have an option for running a supervisor  
process, but all the same I'd like to have an option to not daemonize  
so I can run varnishd "my way".  :-)


  - ask

-- 
http://develooper.com/ - http://askask.com/




From mramos at co.sapo.pt  Mon May  7 23:29:52 2007
From: mramos at co.sapo.pt (Marco Ramos)
Date: Tue, 8 May 2007 00:29:52 +0100
Subject: don't daemonize option
References: <D4F0C23D-A50A-49D0-A169-593C53D2FFD3@develooper.com>
Message-ID: <000f01c790ff$9d548250$2d86c151@supernova>


Indeed, this would be great :)

----- Original Message ----- 
From: "Ask Bj?rn Hansen" <ask at develooper.com>
To: <varnish-misc at projects.linpro.no>
Sent: Tuesday, May 08, 2007 12:14 AM
Subject: don't daemonize option


>
> I usually run services like varnishd under daemontools/supervise.  I 
> realize that varnish already have an option for running a supervisor 
> process, but all the same I'd like to have an option to not daemonize  so 
> I can run varnishd "my way".  :-)
>
>
>  - ask
>
> -- 
> http://develooper.com/ - http://askask.com/
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
> 



From phk at phk.freebsd.dk  Tue May  8 06:42:04 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 08 May 2007 06:42:04 +0000
Subject: don't daemonize option 
In-Reply-To: Your message of "Mon, 07 May 2007 16:14:09 MST."
	<D4F0C23D-A50A-49D0-A169-593C53D2FFD3@develooper.com> 
Message-ID: <1061.1178606524@critter.freebsd.dk>

In message <D4F0C23D-A50A-49D0-A169-593C53D2FFD3 at develooper.com>, =?ISO-8859-1?
Q?Ask_Bj=F8rn_Hansen?= writes:
>
>I usually run services like varnishd under daemontools/supervise.  I  
>realize that varnish already have an option for running a supervisor  
>process, but all the same I'd like to have an option to not daemonize  
>so I can run varnishd "my way".  :-)

Well, -d (or -d -d) already does that, but we may make a more
dedicated version also, it has been talked about.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Tue May  8 09:26:07 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Tue, 08 May 2007 11:26:07 +0200
Subject: don't daemonize option
In-Reply-To: <D4F0C23D-A50A-49D0-A169-593C53D2FFD3@develooper.com> (Ask
	=?iso-8859-1?Q?Bj=F8rn?=
	Hansen's message of "Mon, 7 May 2007 16:14:09 -0700")
References: <D4F0C23D-A50A-49D0-A169-593C53D2FFD3@develooper.com>
Message-ID: <ujrodkvr7io.fsf@false.linpro.no>

Ask Bj?rn Hansen <ask at develooper.com> writes:
> I usually run services like varnishd under daemontools/supervise.  I
> realize that varnish already have an option for running a supervisor
> process, but all the same I'd like to have an option to not daemonize
> so I can run varnishd "my way".  :-)

The supervisor process is not optional.  It does more than just
supervise, it also handles the management interface, VCL compilation
etc.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue May  8 09:30:14 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Tue, 08 May 2007 11:30:14 +0200
Subject: Panics/reboots with Varnish
In-Reply-To: <2669B6F0-E233-41B5-9D16-7EAF6A2E17C0@anduin.net> (Eirik
	=?iso-8859-1?Q?=D8verby's?=
	message of "Mon, 7 May 2007 18:50:32 +0200")
References: <2669B6F0-E233-41B5-9D16-7EAF6A2E17C0@anduin.net>
Message-ID: <ujrk5vjr7bt.fsf@false.linpro.no>

Eirik ?verby <ltning at anduin.net> writes:
> I don't have any coredumps - it appears as if the server is just
> booting.

Do you have a serial console attached?  Does it print anything when
the server reboots?  If not, can you try again with INVARIANTS enabled
in your kernel?

Have you entirely ruled out a hardware problem?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ask at develooper.com  Tue May  8 11:20:14 2007
From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Tue, 8 May 2007 04:20:14 -0700
Subject: don't daemonize option 
In-Reply-To: <1061.1178606524@critter.freebsd.dk>
References: <1061.1178606524@critter.freebsd.dk>
Message-ID: <4C994308-1A8F-4B8D-926E-F23AAB35D508@develooper.com>


On May 7, 2007, at 11:42 PM, Poul-Henning Kamp wrote:

>> I usually run services like varnishd under daemontools/supervise.  I
>> realize that varnish already have an option for running a supervisor
>> process, but all the same I'd like to have an option to not daemonize
>> so I can run varnishd "my way".  :-)
>
> Well, -d (or -d -d) already does that,

I noticed, but I expected that to bring some debug overhead ...

> but we may make a more dedicated version also, it has been talked  
> about.

Excellent - thanks. :-)


  - ask

-- 
http://develooper.com/ - http://askask.com/




From ryu at hfmus.com  Tue May  8 21:36:56 2007
From: ryu at hfmus.com (Yu, Ryan)
Date: Tue, 8 May 2007 17:36:56 -0400
Subject: Strange Problem with Port 80
Message-ID: <BBE94DB300F1C94EBB1B0DB28E0AA8FCB75E15@NYCVEXCH1.corp.hfmus.com>

I'm having the strangest problem.  I'm using varnish on some test
servers and it works great when I run it on any random port number that
is NOT port 80.

 

When I run it on some random port, it works fast and seems to be doing
what it should be doing.  However, when I have it listen on port 80, for
some reason, it runs like molasses.  I've looked into the possibility of
the network at our datacenter and/or my place of business doing some
sort of bandwidth limiting, but that doesn't appear to be the case.

 

It's noticeably slower when it's run on port 80.  It actually runs
slower than if I didn't run Varnish to cache.

 

I've tried this with the RPM download of 1.0.3, source install of 1.0.3
as well as the latest from svn.

 

Please let me know if further information is required such as
varnishlogs.  I was just hoping to see if anyone else was having this
problem or if there's a simple solution that I'm missing.

 

Btw, I'm running the default install of it without any modifications to
the vcl file.  The only thing that's not default is that I turned off
the management services.

 

--

Cordially,

 

Ryan Yu

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070508/b5fca46d/attachment.html>

From des at linpro.no  Wed May  9 10:03:44 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Wed, 09 May 2007 12:03:44 +0200
Subject: Strange Problem with Port 80
In-Reply-To: <BBE94DB300F1C94EBB1B0DB28E0AA8FCB75E15@NYCVEXCH1.corp.hfmus.com>
	(Ryan Yu's message of "Tue\, 8 May 2007 17\:36\:56 -0400")
References: <BBE94DB300F1C94EBB1B0DB28E0AA8FCB75E15@NYCVEXCH1.corp.hfmus.com>
Message-ID: <873b26pb3z.fsf@des.linpro.no>

"Yu, Ryan" <ryu at hfmus.com> writes:
> I'm having the strangest problem.  I'm using varnish on some test
> servers and it works great when I run it on any random port number that
> is NOT port 80.

My guess is that there is some sort of content-scanning transparent
proxy between you and the server.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ryu at hfmus.com  Wed May  9 12:18:14 2007
From: ryu at hfmus.com (Yu, Ryan)
Date: Wed, 9 May 2007 08:18:14 -0400
Subject: Strange Problem with Port 80
In-Reply-To: <873b26pb3z.fsf@des.linpro.no>
Message-ID: <BBE94DB300F1C94EBB1B0DB28E0AA8FCB75E44@NYCVEXCH1.corp.hfmus.com>

That could be true.  I believe we are actually using Surf Control.

Would it be likely that this proxy would not affect Squid, yet would affect Varnish?

I ask, merely because Squid appears to be working properly.

And I'd much rather use Varnish than Squid at this point and would like to find a solution or at least an explanation so I can do some other testing from a non-proxied location.

--
Cordially,
 
Ryan Yu
-----Original Message-----
From: Dag-Erling Sm?rgrav [mailto:des at linpro.no] 
Sent: Wednesday, May 09, 2007 6:04 AM
To: Yu, Ryan
Cc: varnish-misc at projects.linpro.no
Subject: Re: Strange Problem with Port 80

"Yu, Ryan" <ryu at hfmus.com> writes:
> I'm having the strangest problem.  I'm using varnish on some test
> servers and it works great when I run it on any random port number that
> is NOT port 80.

My guess is that there is some sort of content-scanning transparent
proxy between you and the server.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From perbu at linpro.no  Wed May  9 12:37:02 2007
From: perbu at linpro.no (Per Andreas Buer)
Date: Wed, 09 May 2007 14:37:02 +0200
Subject: Strange Problem with Port 80
In-Reply-To: <BBE94DB300F1C94EBB1B0DB28E0AA8FCB75E44@NYCVEXCH1.corp.hfmus.com>
References: <BBE94DB300F1C94EBB1B0DB28E0AA8FCB75E44@NYCVEXCH1.corp.hfmus.com>
Message-ID: <4641C06E.2060607@linpro.no>

Hi,

You could try and make a tcpdump of a slow session. "tcpdump -n -w
varnish.pcap -e "host $PROXYHOST" might show you what is going on. If
you have no success let me have a look.

Per.

Yu, Ryan wrote:
> That could be true.  I believe we are actually using Surf Control.
>
> Would it be likely that this proxy would not affect Squid, yet would affect Varnish?
>
> I ask, merely because Squid appears to be working properly.
>
> And I'd much rather use Varnish than Squid at this point and would like to find a solution or at least an explanation so I can do some other testing from a non-proxied location.
>
> --
> Cordially,
>  
> Ryan Yu
> -----Original Message-----
> From: Dag-Erling Sm?rgrav [mailto:des at linpro.no] 
> Sent: Wednesday, May 09, 2007 6:04 AM
> To: Yu, Ryan
> Cc: varnish-misc at projects.linpro.no
> Subject: Re: Strange Problem with Port 80
>
> "Yu, Ryan" <ryu at hfmus.com> writes:
>   
>> I'm having the strangest problem.  I'm using varnish on some test
>> servers and it works great when I run it on any random port number that
>> is NOT port 80.
>>     
>
> My guess is that there is some sort of content-scanning transparent
> proxy between you and the server.
>
> DES
>   


-- 
Per Andreas Buer / Linpro AS   t: 21 54 41 21 / m: 958 39 117                   
http://linpro.no/ - Ledende p? Linux og ?pen kildekode.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070509/9e7d003a/attachment.bin>

From denis at zeno.org  Wed May  9 16:00:25 2007
From: denis at zeno.org (Denis Ahrens)
Date: Wed, 9 May 2007 18:00:25 +0200
Subject: Gzip issues with Varnish
Message-ID: <922C786D-54C3-480C-88E2-93DC454B8174@zeno.org>

Hi

The gzip problem can be circumvented with since revision r1398.

Simply add the following to a varnish vcl script:

sub vcl_hash
{
         if (req.http.Accept-Encoding) {
                 set req.hash += req.http.Accept-Encoding;
         }
}

Denis Ahrens



From phk at phk.freebsd.dk  Wed May  9 16:44:55 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 09 May 2007 16:44:55 +0000
Subject: Gzip issues with Varnish 
In-Reply-To: Your message of "Wed, 09 May 2007 18:00:25 +0200."
	<922C786D-54C3-480C-88E2-93DC454B8174@zeno.org> 
Message-ID: <74021.1178729095@critter.freebsd.dk>

In message <922C786D-54C3-480C-88E2-93DC454B8174 at zeno.org>, Denis Ahrens writes
:
>Hi
>
>The gzip problem can be circumvented with since revision r1398.
>
>Simply add the following to a varnish vcl script:
>
>sub vcl_hash
>{
>         if (req.http.Accept-Encoding) {
>                 set req.hash += req.http.Accept-Encoding;
>         }
>}

Well, yes, I guess that does it, but depending on how many
differnet Accept-Encoding headers there are out there, your
hit-date may drop and your cache-size explode

Caveat Emptor

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From jean-marc.pouchoulon at ac-montpellier.fr  Thu May 10 11:11:13 2007
From: jean-marc.pouchoulon at ac-montpellier.fr (jean-marc pouchoulon)
Date: Thu, 10 May 2007 13:11:13 +0200
Subject: vcl question cookie
Message-ID: <4642FDD1.80800@ac-montpellier.fr>

Helo all,

I 'm trying to limit the absence of caching with cookies.


if (req.url ~ "\.pdf$|\.png$|\.gif$|\.jpg$|\.mp3$|\.svf$")  {
      lookup;

     if (req.http.Authenticate || req.http.Cookie ~ "__ac=") {
                        pipe;
     }
}

This is not working , all is cached.
where am I wrong ?

thanks

jean-marc


From des at linpro.no  Thu May 10 11:30:12 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Thu, 10 May 2007 13:30:12 +0200
Subject: vcl question cookie
In-Reply-To: <4642FDD1.80800@ac-montpellier.fr> (jean-marc pouchoulon's
	message of "Thu\, 10 May 2007 13\:11\:13 +0200")
References: <4642FDD1.80800@ac-montpellier.fr>
Message-ID: <876470zzjv.fsf@des.linpro.no>

jean-marc pouchoulon <jean-marc.pouchoulon at ac-montpellier.fr> writes:
> I 'm trying to limit the absence of caching with cookies.
>
> if (req.url ~ "\.pdf$|\.png$|\.gif$|\.jpg$|\.mp3$|\.svf$")  {
>      lookup;
>
>     if (req.http.Authenticate || req.http.Cookie ~ "__ac=") {
>                        pipe;
>     }
> }
>
> This is not working , all is cached.
> where am I wrong ?

"lookup" terminates vcl_recv().  The nested if statement is never
reached.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From jean-marc.pouchoulon at ac-montpellier.fr  Thu May 10 11:45:34 2007
From: jean-marc.pouchoulon at ac-montpellier.fr (jean-marc pouchoulon)
Date: Thu, 10 May 2007 13:45:34 +0200
Subject: vcl question cookie
In-Reply-To: <876470zzjv.fsf@des.linpro.no>
References: <4642FDD1.80800@ac-montpellier.fr> <876470zzjv.fsf@des.linpro.no>
Message-ID: <464305DE.4060107@ac-montpellier.fr>


> "lookup" terminates vcl_recv().  The nested if statement is never
> reached.
>
> DES
>   
in fact I did

if (req.url ~ "\.pdf$|\.png$|\.gif$|\.jpg$|\.mp3$|\.svf$")  {
     

    if (req.http.Authenticate || req.http.Cookie ~ "__ac=") {
                       pipe;
    }
    lookup;

}

I suppose pipe also terminates vcl_recv() 
I tried also
if (req.url ~ "\.pdf$|\.png$|\.gif$|\.jpg$|\.mp3$|\.svf$" && !(req.http.Cookie ~ "__ac=") )  {
         lookup;
     }
it seems slow

Any workaround to accomplish partial cache with cookies ?

thanks again

jean-marc
 



From des at linpro.no  Thu May 10 13:03:31 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Thu, 10 May 2007 15:03:31 +0200
Subject: vcl question cookie
In-Reply-To: <464305DE.4060107@ac-montpellier.fr> (jean-marc pouchoulon's
	message of "Thu\, 10 May 2007 13\:45\:34 +0200")
References: <4642FDD1.80800@ac-montpellier.fr> <876470zzjv.fsf@des.linpro.no>
	<464305DE.4060107@ac-montpellier.fr>
Message-ID: <87wszgygnw.fsf@des.linpro.no>

jean-marc pouchoulon <jean-marc.pouchoulon at ac-montpellier.fr> writes:
> in fact I did
>
> if (req.url ~ "\.pdf$|\.png$|\.gif$|\.jpg$|\.mp3$|\.svf$") {
>        if (req.http.Authenticate || req.http.Cookie ~ "__ac=") {
>                pipe;
>        }
>        lookup;
> }
>
> I suppose pipe also terminates vcl_recv() I tried also
>
> if (req.url ~ "\.pdf$|\.png$|\.gif$|\.jpg$|\.mp3$|\.svf$" &&
>     !(req.http.Cookie ~ "__ac=")) {
>         lookup;
> }
>
> it seems slow

What seems slow?  Does Varnish actually run slowly, or do you just think
the code looks slow?

> Any workaround to accomplish partial cache with cookies ?

Not sure what you're asking for.

If (as in most cases) neither authentication nor cookies actually make
any difference as far as images are concerned, you might as well ignore
them completely:

if (req.url ~ "\.pdf$|\.png$|\.gif$|\.jpg$|\.mp3$|\.svf$") {
       lookup;
}

BTW, your regexp could be more readable:

if (req.url ~ "\.(pdf|png|gif|jpg|mp3|svf)$") {
       lookup;
}

depending on your OS's regexp library, it might also be faster that way
(though you probably won't notice)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From denis at zeno.org  Thu May 10 13:38:25 2007
From: denis at zeno.org (Denis Ahrens)
Date: Thu, 10 May 2007 15:38:25 +0200
Subject: Gzip issues with Varnish 
In-Reply-To: <74021.1178729095@critter.freebsd.dk>
References: <74021.1178729095@critter.freebsd.dk>
Message-ID: <B31688B7-0BFF-49D5-B29F-D58A141B8744@zeno.org>


On 09.05.2007, at 18:44, Poul-Henning Kamp wrote:

> In message <922C786D-54C3-480C-88E2-93DC454B8174 at zeno.org>, Denis  
> Ahrens writes
> :
>> Hi
>>
>> The gzip problem can be circumvented with since revision r1398.
>>
>> Simply add the following to a varnish vcl script:
>>
>> sub vcl_hash
>> {
>>         if (req.http.Accept-Encoding) {
>>                 set req.hash += req.http.Accept-Encoding;
>>         }
>> }
>
> Well, yes, I guess that does it, but depending on how many
> differnet Accept-Encoding headers there are out there, your
> hit-date may drop and your cache-size explode

Ok, I changed it a little to reflect that:

sub vcl_hash
{
         if (req.http.Accept-Encoding ~ "gzip") {
                 set req.hash += "gzip";
         }
         else if (req.http.Accept-Encoding ~ "deflate") {
                 set req.hash += "deflate";
         }
}

We only have gzip support, so this should be ok.

Denis Ahrens



From denis at zeno.org  Thu May 10 13:43:37 2007
From: denis at zeno.org (Denis Ahrens)
Date: Thu, 10 May 2007 15:43:37 +0200
Subject: url.purge and regexp
Message-ID: <2876D9C1-31E0-4232-8F56-6F7CF78F8B60@zeno.org>

Hi

How Iam supposed to purge an URL like this in CLI:

http://www.example.com/page.html?foo=bar&bar=foo

Iam asking because I don't know what will happen with
the "?" in the line. Is it part of the regexp or part
of the url then because escaping with "\" is not
possible (Illegal backslash sequence).

Denis Ahrens



From des at linpro.no  Thu May 10 13:53:14 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Thu, 10 May 2007 15:53:14 +0200
Subject: url.purge and regexp
In-Reply-To: <2876D9C1-31E0-4232-8F56-6F7CF78F8B60@zeno.org> (Denis Ahrens's
	message of "Thu\, 10 May 2007 15\:43\:37 +0200")
References: <2876D9C1-31E0-4232-8F56-6F7CF78F8B60@zeno.org>
Message-ID: <87ps58yed1.fsf@des.linpro.no>

Denis Ahrens <denis at zeno.org> writes:
> How Iam supposed to purge an URL like this in CLI:
>
> http://www.example.com/page.html?foo=bar&bar=foo
>
> Iam asking because I don't know what will happen with
> the "?" in the line. Is it part of the regexp or part
> of the url then because escaping with "\" is not
> possible (Illegal backslash sequence).

You need to escape the backslash:

url.purge http://www.example.com/page.html\\?foo=bar&bar=foo

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Fri May 11 08:55:17 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Fri, 11 May 2007 10:55:17 +0200
Subject: VCL and Non-Existent Backend Objects
In-Reply-To: <463F6CE9.1080206@nyi.net> (James Quacinella's message of "Mon\, 
	07 May 2007 14\:16\:09 -0400")
References: <463F6CE9.1080206@nyi.net>
Message-ID: <877irfvix6.fsf@des.linpro.no>

James Quacinella <james at nyi.net> writes:
> I was contemplating a setup such as this:
>
> 1) Client requests xml file from varnish cache
> 2) If it exists, fine; else check backend file server running lighttpd
> 3) If it exists on the backend, again no problem; however, if it
> doesn't exist on the lighttpd backend, I need varnish to check another
> (dynamic) backend, which will dynamically generate it and save it on
> the lightttpd backend for future purposes.
>
> Does anyone have any ideas on how to implement that via VCL? I would
> imagine I would need to be able to see the response headers from the
> original backend, to see if a 404 is generated (like here
> http://varnish.projects.linpro.no/ticket/88).

I don't think this is supported in the current code base.  Poul-Henning
and I discussed something similar on Monday - the ability to restart a
request, possibly after modifying it a little.

There was also some talk at some point of adding some kind of "policy
check" functionality - the ability to perform a secondary HTTP request
from VCL code, e.g. to check a user's credentials before serving a
cached document which requires authorization, rather than just passing
the entire request on to the backend.

Mind you, these are just ideas - we have no concrete plans to implement
any of this in the near future.  We are currently ramping up work on
2.0, and will most likely be busy with that until fall.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ask at develooper.com  Fri May 11 09:30:35 2007
From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Fri, 11 May 2007 02:30:35 -0700
Subject: VCL and Non-Existent Backend Objects
In-Reply-To: <877irfvix6.fsf@des.linpro.no>
References: <463F6CE9.1080206@nyi.net> <877irfvix6.fsf@des.linpro.no>
Message-ID: <7BD21248-04BC-4609-AE48-48FBFD8009DA@develooper.com>


On May 11, 2007, at 1:55 AM, Dag-Erling Sm?rgrav wrote:

> There was also some talk at some point of adding some kind of "policy
> check" functionality - the ability to perform a secondary HTTP request
> from VCL code, e.g. to check a user's credentials before serving a
> cached document which requires authorization, rather than just passing
> the entire request on to the backend.

Perlbal does this by having the first backend return a special  
"reproxy" header that Perlbal then uses to do the request again to  
another backend.

	http://lists.danga.com/pipermail/perlbal/2007-May/000441.html
	http://code.sixapart.com/svn/perlbal/trunk/doc/reproxying.txt



  - ask

-- 
http://develooper.com/ - http://askask.com/




From phk at phk.freebsd.dk  Fri May 11 15:12:22 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 11 May 2007 15:12:22 +0000
Subject: disable sendfile in Varnish, please
Message-ID: <5362.1178896342@critter.freebsd.dk>


I've nailed three different operating system kernels as having
sendfile(2) issues today, so I would advice all of you to
disable sendfile to avoid the various problems we've seen.

The easiest way is to specify

	-p sendfile_threshold=-1

to varnishd, or by using the CLI:

	param.set sendfile_threshold -1

Only those of you with high paging rates would be in risk of seing
any performance changes as result of this.

While it is quite easy to set up a test environment to look for
sendfile trouble, we cannot do it as part of varnish runtime so I
think we will ship with sendfile_threshold at -1 for the forseeable
future.

Once we start to see kernels where sendfile works correctly in all
instances, we can consider how to seletively enable those.

Alternatively, if nobody sees a performance loss from disabling it
now, we might as well just forget about it in toto.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


From mramos at co.sapo.pt  Mon May 14 10:26:25 2007
From: mramos at co.sapo.pt (Marco Ramos)
Date: Mon, 14 May 2007 11:26:25 +0100
Subject: SNMP support
Message-ID: <1179138385.18576.4.camel@supernova>


Will Varnish have SNMP support in the future? It would be very nice to
have a Varnish MIB to check how things are going.

Marco Ramos



From phk at phk.freebsd.dk  Mon May 14 10:28:26 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 14 May 2007 10:28:26 +0000
Subject: SNMP support 
In-Reply-To: Your message of "Mon, 14 May 2007 11:26:25 +0100."
	<1179138385.18576.4.camel@supernova> 
Message-ID: <65796.1179138506@critter.freebsd.dk>

In message <1179138385.18576.4.camel at supernova>, Marco Ramos writes:
>
>Will Varnish have SNMP support in the future? It would be very nice to
>have a Varnish MIB to check how things are going.

First off, I'm not sure I agree, but I am biased, I've been around
since SNMPv1 :-)

Second, implementing SNMP support in a portable way is unfortunately the
same as implementing all of SNMP, there are no agreed on portable
API's for implementing SNMP. (Another POSIX failure there).

Thirdly, somebody would have to sponsor the work.

So I would say the answer to your question is: "unlikely".

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Mon May 14 10:50:44 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Mon, 14 May 2007 12:50:44 +0200
Subject: SNMP support
In-Reply-To: <65796.1179138506@critter.freebsd.dk> (Poul-Henning Kamp's
	message of "Mon\, 14 May 2007 10\:28\:26 +0000")
References: <65796.1179138506@critter.freebsd.dk>
Message-ID: <87fy5zu1a3.fsf@des.linpro.no>

"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
> Marco Ramos <mramos at co.sapo.pt> writes:
> > Will Varnish have SNMP support in the future? It would be very nice to
> > have a Varnish MIB to check how things are going.
> First off, I'm not sure I agree, but I am biased, I've been around
> since SNMPv1 :-)

I don't think SNMP support has much value in an of itself, but it has
the advantage of being easy to integrate with existing big-name
monitoring tools such as CA Unicenter.

> Second, implementing SNMP support in a portable way is unfortunately the
> same as implementing all of SNMP, there are no agreed on portable
> API's for implementing SNMP. (Another POSIX failure there).
>
> Thirdly, somebody would have to sponsor the work.
>
> So I would say the answer to your question is: "unlikely".

One cheap way of doing it would be to write a bsnmpd plugin that reads
and exports information from the shared memory stats segment.  It would
work out of the box on FreeBSD (provided you enable bsnmpd); on other
systems, you'd have to port bsnmpd first.

Still, it's probably not something we're interested in doing without a
sponsor.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From james at nyi.net  Mon May 14 17:33:20 2007
From: james at nyi.net (James Quacinella)
Date: Mon, 14 May 2007 13:33:20 -0400
Subject: VCL and Non-Existent Backend Objects
In-Reply-To: <7BD21248-04BC-4609-AE48-48FBFD8009DA@develooper.com>
References: <463F6CE9.1080206@nyi.net> <877irfvix6.fsf@des.linpro.no>
	<7BD21248-04BC-4609-AE48-48FBFD8009DA@develooper.com>
Message-ID: <46489D60.1060900@nyi.net>

Ask Bj?rn Hansen wrote:
>
> On May 11, 2007, at 1:55 AM, Dag-Erling Sm?rgrav wrote:
>
>> There was also some talk at some point of adding some kind of "policy
>> check" functionality - the ability to perform a secondary HTTP request
>> from VCL code, e.g. to check a user's credentials before serving a
>> cached document which requires authorization, rather than just passing
>> the entire request on to the backend.
>
> Perlbal does this by having the first backend return a special 
> "reproxy" header that Perlbal then uses to do the request again to 
> another backend.

Thats pretty neat. Too bad the LB software I'm using doesn't really 
support that (at least not now).

-- james



From lists at dirkgomez.de  Mon May 14 22:57:43 2007
From: lists at dirkgomez.de (Dirk Gomez)
Date: Tue, 15 May 2007 00:57:43 +0200
Subject: X-Varnish: what does it mean?
Message-ID: <174F3AA9-0CD5-49AF-A64F-D4926F172F93@dirkgomez.de>

I'm trying to assess whether my varnish setup is doing what I want it  
to do. What is X-Varnish's purpose?

Also am I interpreting the HTTP specs (and the c code) correctly that  
the Age header tells me how long a particular object has been in  
varnish's cache?

-- Dirk


From phk at phk.freebsd.dk  Mon May 14 23:02:12 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 14 May 2007 23:02:12 +0000
Subject: X-Varnish: what does it mean? 
In-Reply-To: Your message of "Tue, 15 May 2007 00:57:43 +0200."
	<174F3AA9-0CD5-49AF-A64F-D4926F172F93@dirkgomez.de> 
Message-ID: <1749.1179183732@critter.freebsd.dk>

In message <174F3AA9-0CD5-49AF-A64F-D4926F172F93 at dirkgomez.de>, Dirk Gomez writ
es:
>I'm trying to assess whether my varnish setup is doing what I want it  
>to do. What is X-Varnish's purpose?

It allows you to find the correct log-entries for the transaction.

All requets in varnish are assigned a XID number, the X-Varnish
tells you what it is, and if a cache-hit was involved, also the
XID of the transaction that put the object in the cache.

>Also am I interpreting the HTTP specs (and the c code) correctly that  
>the Age header tells me how long a particular object has been in  
>varnish's cache?

yes.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From denis at startsiden.no  Wed May 16 06:27:34 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Wed, 16 May 2007 08:27:34 +0200 (CEST)
Subject: "Whitescreen" issue with varnish 1.0.2-2 (debian etch)
In-Reply-To: <14438856.5861179223618395.JavaMail.root@ms1.startsiden.no>
Message-ID: <23453395.11531179296854971.JavaMail.root@ms1.startsiden.no>

Hi all,

I have begun testing varnish on our main server pool, and the initial tests worked smoothly.
However as soon as I put the box under load (gave it a slice of our production traffic) it started "fumbling" requests. I know it's not a very good description, but I have a hard time understanding exactly what goes wrong where.

I did some logging, however due to the fact that the error does not appear until a sufficient amount of traffic hits the system, the logs are quite large. So I really wondered what I should start looking for?  What kind of logmessages?

During my trials I stripped the Varnish config that debian ships to make sure none of those settings were causing trouble. Also I did test running Apache2.2 on it's own on that box, and that works out fine, nothing similar happens. I also disabled sendfile as per PHKs instructions.

I will try to compile a newer build, and do some more testing, but any pointers as to the kind of logentries I should look for would be excellent.

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From Kenneth.Rorvik at hio.no  Wed May 16 06:45:14 2007
From: Kenneth.Rorvik at hio.no (=?UTF-8?B?S2VubmV0aCBSw7hydmlr?=)
Date: Wed, 16 May 2007 08:45:14 +0200
Subject: "Whitescreen" issue with varnish 1.0.2-2 (debian etch)
In-Reply-To: <23453395.11531179296854971.JavaMail.root@ms1.startsiden.no>
References: <23453395.11531179296854971.JavaMail.root@ms1.startsiden.no>
Message-ID: <464AA87A.7050509@hio.no>

I saw the same problems with 1.0.2 on redhat - empty documents returned 
after a while, possibly related to filling up the disk backend file.

1.0.3 fixed it here, I didn't look deeper into it.

-- 
Kenneth R?rvik, IT HiO
Tlf 22 45 20 83
Kenneth.Rorvik at hio.no


From denis at startsiden.no  Wed May 16 08:29:37 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Wed, 16 May 2007 10:29:37 +0200 (CEST)
Subject: "Whitescreen" issue with varnish 1.0.2-2 (debian etch)
In-Reply-To: <464AA87A.7050509@hio.no>
Message-ID: <517789.13131179304177196.JavaMail.root@ms1.startsiden.no>

----- Kenneth R?rvik <Kenneth.Rorvik at hio.no> wrote:
> I saw the same problems with 1.0.2 on redhat - empty documents
> returned 
> after a while, possibly related to filling up the disk backend file.
> 
> 1.0.3 fixed it here, I didn't look deeper into it.

1.0.3 worked here too. I always intended to run 1.0.3, but I wanted to test the etch included package just for the fun of it.
It's a bit scary that the package in etch can give a newcomer to Varnish the impression that it is not really stable/production ready though, I am afraid new users could get a bad start.

I wanted to update the VarnishOnDebian wiki page with a small note on this situation, but it seems I need to be granted edit permissions? 

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From ingvar at linpro.no  Wed May 16 09:15:08 2007
From: ingvar at linpro.no (Ingvar Hagelund)
Date: Wed, 16 May 2007 11:15:08 +0200
Subject: "Whitescreen" issue with varnish 1.0.2-2 (debian etch)
In-Reply-To: <517789.13131179304177196.JavaMail.root@ms1.startsiden.no>
References: <517789.13131179304177196.JavaMail.root@ms1.startsiden.no>
Message-ID: <464ACB9C.4040906@linpro.no>


* Kenneth R?rvik
>> I saw the same problems with 1.0.2 on redhat - empty documents
>> returned after a while, possibly related to filling up the disk backend file.

* Denis Br?khus
>> 1.0.3 fixed it here, I didn't look deeper into it.
> 
> 1.0.3 worked here too. I always intended to run 1.0.3, but I wanted to test the etch included package just for the fun of it.
> It's a bit scary that the package in etch can give a newcomer to Varnish the impression that it is not really stable/production ready though, I am afraid new users could get a bad start.

Rumors says 1.0.4 is on the stairs, banging at the door. It will
probably include an update to the Debian package.

I don't know if changes/updates will trickle into etch over the time.
Stig, Lars?

Ingvar


-- 
N?r alt annet feiler: Symlink


From denis at startsiden.no  Wed May 16 09:08:32 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Wed, 16 May 2007 11:08:32 +0200 (CEST)
Subject: "Whitescreen" issue with varnish 1.0.2-2 (debian etch)
In-Reply-To: <464ACB9C.4040906@linpro.no>
Message-ID: <23073160.14051179306512204.JavaMail.root@ms1.startsiden.no>

----- Ingvar Hagelund <ingvar at linpro.no> wrote:
> Rumors says 1.0.4 is on the stairs, banging at the door. It will
> probably include an update to the Debian package.
> I don't know if changes/updates will trickle into etch over the time.
> Stig, Lars?

For that to happen someone would have to backport all the fixes to 1.0.2, right?
It's probably a more workable alternative to somehow provide an updated debian (etch compatible) package outside the official repository. But this is all speculation on my part of course.

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From des at linpro.no  Wed May 16 13:05:14 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Wed, 16 May 2007 15:05:14 +0200
Subject: "Whitescreen" issue with varnish 1.0.2-2 (debian etch)
In-Reply-To: <517789.13131179304177196.JavaMail.root@ms1.startsiden.no>
	("Denis =?utf-8?Q?Br=C3=A6khus=22's?= message of "Wed\, 16 May 2007
	10\:29\:37 +0200 \(CEST\)")
References: <517789.13131179304177196.JavaMail.root@ms1.startsiden.no>
Message-ID: <87fy5wrkad.fsf@des.linpro.no>

Denis Br?khus <denis at startsiden.no> writes:
> I wanted to update the VarnishOnDebian wiki page with a small note on
> this situation, but it seems I need to be granted edit permissions?

Done.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Wed May 16 13:17:49 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Wed, 16 May 2007 15:17:49 +0200
Subject: Sendfile issues (was: HEADS UP: Varnish on FreeBSD-CURRENT)
In-Reply-To: <ujr1wirma6x.fsf@cat.linpro.no> ("Dag-Erling =?utf-8?Q?Sm?=
	=?utf-8?Q?=C3=B8rgrav=22's?= message
	of "Wed\, 11 Apr 2007 13\:17\:58 +0200")
References: <ujr1wirma6x.fsf@cat.linpro.no>
Message-ID: <874pmcrjpe.fsf@des.linpro.no>

des at linpro.no (Dag-Erling Sm?rgrav) writes:
> Anyone running Varnish on FreeBSD-CURRENT should set the
> sendfile_threshold run-time parameter to -1 (disabling the use of
> sendfile) to work around a bug in the sendfile syscall where the file
> being transferred will be truncated by an amount equivalent to the
> size of the HTTP header.

We have discovered further issues with sendfile on FreeBSD 6 and Linux
2.6, and have decided to disable it by default until we can figure out
whether these are issues in Varnish or in the respective kernels.

The aforementioned bug in FreeBSD 7 still hasn't been fixed, BTW.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ltning at anduin.net  Thu May 17 17:52:14 2007
From: ltning at anduin.net (=?ISO-8859-1?Q?Eirik_=D8verby?=)
Date: Thu, 17 May 2007 19:52:14 +0200
Subject: varnishncsa ignoring SIGHUP?
Message-ID: <3DBB60E5-118A-4C9C-94DA-7A56311A5B4E@anduin.net>

Hi,

I'm suspecting that varnishncsa (from 1.0.3) doesn't actually listen  
to a SIGHUP - I'm unable to make it re-open its logfile.

To work around this I'm currently logging to a FIFO and using another  
tool (flog) to pick up and rotate the logs, but I thought it was  
worth mentioning.

FYI this is on FreeBSD 6.2. Not sure if it has something to do with  
the way I'm starting it or a bug in varnishncsa; I hacked together an  
rc script which, after I discovered this, does what I describe above.  
If anyone has an interest in the script let me know.

/Eirik




From des at linpro.no  Fri May 18 08:14:32 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Fri, 18 May 2007 10:14:32 +0200
Subject: varnishncsa ignoring SIGHUP?
In-Reply-To: <3DBB60E5-118A-4C9C-94DA-7A56311A5B4E@anduin.net> ("Eirik
	=?utf-8?Q?=C3=98verby=22's?= message of "Thu\, 17 May 2007 19\:52\:14
	+0200")
References: <3DBB60E5-118A-4C9C-94DA-7A56311A5B4E@anduin.net>
Message-ID: <87tzuao8ev.fsf@des.linpro.no>

Eirik ?verby <ltning at anduin.net> writes:
> I'm suspecting that varnishncsa (from 1.0.3) doesn't actually listen
> to a SIGHUP - I'm unable to make it re-open its logfile.

Assuming that you didn't do something silly like redirect stdout instead
of using -w, I suspect this may be an effect of aggressive optimization
by the compiler.  Try the following patch:

Index: bin/varnishncsa/varnishncsa.c
===================================================================
--- bin/varnishncsa/varnishncsa.c       (revision 1444)
+++ bin/varnishncsa/varnishncsa.c       (working copy)
@@ -347,7 +347,7 @@
 
 /*--------------------------------------------------------------------*/
 
-static sig_atomic_t reopen;
+static volatile sig_atomic_t reopen;
 
 static void
 sighup(int sig)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From phk at phk.freebsd.dk  Sun May 20 12:21:14 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 20 May 2007 12:21:14 +0000
Subject: disable sendfile in Varnish, please  [UPD]
In-Reply-To: Your message of "Fri, 11 May 2007 15:12:22 GMT."
	<5362.1178896342@critter.freebsd.dk> 
Message-ID: <1368.1179663674@critter.freebsd.dk>


Thanks to kind help from Peter Wemm and John Baldwin from the FreeBSD
project, we now the cause of the sendfile corruption:

Sendfile moves the filebuffers to the socket and as soon as it has
done this, it returns, and the buffers may not have been processed
by TCP yet.

With the recent change to make pass mode use the same codepath as
a normal cached transaction, the filebuffers will get recycled
as soon as sendfile returns, and any other transaction that grabs
that piece of filebuffer may write new data in it, before the
TCP stack is done.

For a normal cached operation, the data in the filebuffer lives on,
until the object expires. and therefore corruption will be a lot
less likely, but not impossible in that case.

I'm investigating the options we have, but for now we can't use
sendfile.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Sun May 20 14:41:32 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sun, 20 May 2007 16:41:32 +0200
Subject: disable sendfile in Varnish, please  [UPD]
In-Reply-To: <1368.1179663674@critter.freebsd.dk> (Poul-Henning Kamp's message
	of "Sun\, 20 May 2007 12\:21\:14 +0000")
References: <1368.1179663674@critter.freebsd.dk>
Message-ID: <87wsz31rs3.fsf@des.linpro.no>

"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
> Sendfile moves the filebuffers to the socket and as soon as it has
> done this, it returns, and the buffers may not have been processed
> by TCP yet.

So we need to be able to block (or at least prevent re-use of the
affected part of the storage file) until sendfile() completes...

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From phk at phk.freebsd.dk  Mon May 21 14:45:49 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 21 May 2007 14:45:49 +0000
Subject: disable sendfile in Varnish, please [UPD] 
In-Reply-To: Your message of "Sun, 20 May 2007 16:41:32 +0200."
	<87wsz31rs3.fsf@des.linpro.no> 
Message-ID: <1916.1179758749@critter.freebsd.dk>

In message <87wsz31rs3.fsf at des.linpro.no>, =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=
 writes:
>"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
>> Sendfile moves the filebuffers to the socket and as soon as it has
>> done this, it returns, and the buffers may not have been processed
>> by TCP yet.
>
>So we need to be able to block (or at least prevent re-use of the
>affected part of the storage file) until sendfile() completes...

No, that gets waaay to complicated.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From ccripy at gmail.com  Tue May 22 00:36:25 2007
From: ccripy at gmail.com (cripy)
Date: Mon, 21 May 2007 20:36:25 -0400
Subject: Too many open files?
Message-ID: <dacbdcd10705211736v27a6cc69w2480c067d99172af@mail.gmail.com>

Hello, varnish suddenly dies after awhile with this message:

>>
Child said (2, 25657): <<Assert error in wrk_thread(), cache_pool.c line
213:
  Condition((pipe(w->pipe)) == 0) not true.
  errno = 24 (Too many open files)
>>


I have set the following in /etc/sysconfig/varnish:
ulimit -n 131072
NFILES=131072
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070521/d0d6f42d/attachment.html>

From des at linpro.no  Tue May 22 07:54:45 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 22 May 2007 09:54:45 +0200
Subject: Too many open files?
In-Reply-To: <dacbdcd10705211736v27a6cc69w2480c067d99172af@mail.gmail.com>
	(ccripy@gmail.com's message of "Mon\, 21 May 2007 20\:36\:25 -0400")
References: <dacbdcd10705211736v27a6cc69w2480c067d99172af@mail.gmail.com>
Message-ID: <87646lpa2i.fsf@des.linpro.no>

cripy <ccripy at gmail.com> writes:
> Hello, varnish suddenly dies after awhile with this message:
>
>>>
> Child said (2, 25657): <<Assert error in wrk_thread(), cache_pool.c line
> 213:
>  Condition((pipe(w->pipe)) == 0) not true.
>  errno = 24 (Too many open files)
>>>
>
>
> I have set the following in /etc/sysconfig/varnish:
> ulimit -n 131072
> NFILES=131072

Can you verify that the limit is actually applied?  e.g. add 'ulimit -a'
to /etc/init.d/varnish, right before where it starts varnishd, run
'/etc/init.d/varnish restart', and check that the file descriptor limit
really is 131072.

Can you also show us the result of typing 'param.show' in the management
interface?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ccripy at gmail.com  Tue May 22 15:44:15 2007
From: ccripy at gmail.com (cripy)
Date: Tue, 22 May 2007 11:44:15 -0400
Subject: Too many open files?
In-Reply-To: <87646lpa2i.fsf@des.linpro.no>
References: <dacbdcd10705211736v27a6cc69w2480c067d99172af@mail.gmail.com>
	<87646lpa2i.fsf@des.linpro.no>
Message-ID: <dacbdcd10705220844i48c7d3bep6c61eac1470e30bf@mail.gmail.com>

Sorry for the list noise.  It was an error in some of my code.  I added some
code in cache_center.c to function cnt_fetch to stop varnish from caching
401 Unauthorized returns.  Anyway, I didn't know varnish's code very well
and wasn't aware that the variable I found that returned me the information
I needed was infact also returning me an fd.  Anyway, for those interested
here is my code that I added with the fix VBE_ClosedFd()
------------
cnt_fetch(struct sess *sp)
{
...
        vc = VBE_GetFd(sp);

        if (http_GetStatus(vc->http) == 401)
                sp->handling = VCL_RET_PASS;
        VBE_ClosedFd(sp->wrk, vc, 0);
...

        if (sp->handling == VCL_RET_ERROR)
                INCOMPL();

-------------------------
On 5/22/07, Dag-Erling Sm?rgrav <des at linpro.no> wrote:
>
> cripy <ccripy at gmail.com> writes:
> > Hello, varnish suddenly dies after awhile with this message:
> >
> >>>
> > Child said (2, 25657): <<Assert error in wrk_thread(), cache_pool.c line
> > 213:
> >  Condition((pipe(w->pipe)) == 0) not true.
> >  errno = 24 (Too many open files)
> >>>
> >
> >
> > I have set the following in /etc/sysconfig/varnish:
> > ulimit -n 131072
> > NFILES=131072
>
> Can you verify that the limit is actually applied?  e.g. add 'ulimit -a'
> to /etc/init.d/varnish, right before where it starts varnishd, run
> '/etc/init.d/varnish restart', and check that the file descriptor limit
> really is 131072.
>
> Can you also show us the result of typing 'param.show' in the management
> interface?
>
> DES
> --
> Dag-Erling Sm?rgrav
> Senior Software Developer
> Linpro AS - www.linpro.no
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070522/0b425f3e/attachment.html>

From des at linpro.no  Thu May 24 10:48:49 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 24 May 2007 12:48:49 +0200
Subject: Trac upgrade
Message-ID: <87sl9mv6ni.fsf@des.linpro.no>

I've upgraded Trac to 0.10.3, and added some plugins which, amongst
other things, include a spam filter and a tool for removing unwanted
ticket updates.  This means I was (finally!) able to remove the ticket
spam we were hit with in early April.

Please let me know if you stumble across something that was broken by
the upgrade.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu May 24 10:48:49 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 24 May 2007 12:48:49 +0200
Subject: Trac upgrade
Message-ID: <87sl9mv6ni.fsf@des.linpro.no>

I've upgraded Trac to 0.10.3, and added some plugins which, amongst
other things, include a spam filter and a tool for removing unwanted
ticket updates.  This means I was (finally!) able to remove the ticket
spam we were hit with in early April.

Please let me know if you stumble across something that was broken by
the upgrade.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From omar.kilani at gmail.com  Tue May 29 05:05:08 2007
From: omar.kilani at gmail.com (Omar Kilani)
Date: Tue, 29 May 2007 15:05:08 +1000
Subject: Cookie inspection
Message-ID: <f0cc3e3e0705282205h3b43065atbc1ee434a6adec94@mail.gmail.com>

Hi there,

I have a web application which supports a bunch of interface/website
languages via Accept-Language headers and/or a user-overridden cookie
value.

I'd like to modify req.hash in vcl_hash to take into account the value
of this particular cookie (similar to what Denis does with
Accept-Encoding here:
http://projects.linpro.no/pipermail/varnish-misc/2007-May/000407.html
-- neat! :)

>From looking through the varnish-cache code, I don't think this type
of operation is supported, and I'm not sure what the best way to
represent this in VCL is.

Including the entire req.http.Cookie in the hash key is suboptimal as
there are a bunch of other cookies set (session cookies, etc)
depending on other variables.

Doing something like req.http.Cookie.LANGUAGE *could* theoretically do
the right thing (and use http_GetHdrField, although some browsers can
send multiple values for the same cookie name... :) and *looks* right,
but obviously has a very different meaning in VCL where it compiles
to:

VRT_GetHdr(sp, 1, "\020Cookie.LANGUAGE:")

I've created a small patch:

http://treehou.se/~omar/cookie-inspection-1.patch

Which implements VRT_GetHdrField and cookie inspection.

This should be a lot more general, but is this something that varnish
could potentially support?

The other thing I'd like to be able to do is have some sort of 'first
match over an array of header values' VCL construct, so you could vary
the hash by whatever languages in Accept-Language your site supports
-- but the cookie thing would be workable too.

Thanks!

Regards,
Omar


From des at linpro.no  Tue May 29 07:53:22 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 29 May 2007 09:53:22 +0200
Subject: Cookie inspection
In-Reply-To: <f0cc3e3e0705282205h3b43065atbc1ee434a6adec94@mail.gmail.com>
	(Omar Kilani's message of "Tue\, 29 May 2007 15\:05\:08 +1000")
References: <f0cc3e3e0705282205h3b43065atbc1ee434a6adec94@mail.gmail.com>
Message-ID: <87hcpwoykt.fsf@des.linpro.no>

"Omar Kilani" <omar.kilani at gmail.com> writes:
> I'd like to modify req.hash in vcl_hash to take into account the value
> of this particular cookie [...]
> From looking through the varnish-cache code, I don't think this type
> of operation is supported, and I'm not sure what the best way to
> represent this in VCL is.

Poul-Henning is working on making multi-value headers accessible as
associative arrays, e.g. "req.http.cache-control[max-age]" for
"Cache-control: max-age".  This implies not only inspection but also
modification of individual fields.  Your patch is a step in the right
direction, though it probably duplicates (or conflicts with) part of
Poul-Henning's work.

In any case, could you please open a ticket with the patch as an
attachment?

> Doing something like req.http.Cookie.LANGUAGE *could* theoretically do
> the right thing (and use http_GetHdrField, although some browsers can
> send multiple values for the same cookie name... :)

AFAIR from RFC2616, if multiple values are provided for the same key,
only the last one applies.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From phk at phk.freebsd.dk  Tue May 29 08:16:59 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 29 May 2007 08:16:59 +0000
Subject: Cookie inspection 
In-Reply-To: Your message of "Tue, 29 May 2007 15:05:08 +1000."
	<f0cc3e3e0705282205h3b43065atbc1ee434a6adec94@mail.gmail.com> 
Message-ID: <16610.1180426619@critter.freebsd.dk>

In message <f0cc3e3e0705282205h3b43065atbc1ee434a6adec94 at mail.gmail.com>, "Omar
 Kilani" writes:

>Including the entire req.http.Cookie in the hash key is suboptimal as
>there are a bunch of other cookies set (session cookies, etc)
>depending on other variables.
>
>Doing something like req.http.Cookie.LANGUAGE *could* theoretically do
>the right thing (and use http_GetHdrField, although some browsers can
>send multiple values for the same cookie name... :) and *looks* right,
>but obviously has a very different meaning in VCL where it compiles
>to:

The Planned syntax is:
	req.http.Cookie[language]

and just as for http headers, the headers will be read sequentially
and the last found value is used.

>http://treehou.se/~omar/cookie-inspection-1.patch

Not bad :-)

>The other thing I'd like to be able to do is have some sort of 'first
>match over an array of header values' VCL construct, so you could vary
>the hash by whatever languages in Accept-Language your site supports

Does your backend include a proper Vary: header when it selects on
language ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Tue May 29 09:18:00 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 29 May 2007 11:18:00 +0200
Subject: Cookie inspection
In-Reply-To: <16610.1180426619@critter.freebsd.dk> (Poul-Henning Kamp's
	message of "Tue\, 29 May 2007 08\:16\:59 +0000")
References: <16610.1180426619@critter.freebsd.dk>
Message-ID: <87lkf8ng3b.fsf@des.linpro.no>

"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
> Does your backend include a proper Vary: header when it selects on
> language ?

Applications which use cookies should emit "Vary: Cookie".

The HTTP protocol provides a mechanism for selecting the language of a
page (client sends Accept-Language, server sends Content-Language), so
there should be no reason to use cookies for that.

The part of the patch that makes ';' equivalent to ',' is problematic,
as ';' is normally used to separate subfields within individual header
values (e.g. "Content-Type: text/html;charset=utf-8").  Whoever wrote
RFC 2109 (which uses ';' instead of ',' to separate multiple values) is
an idiot...

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From admin at adofms.com.au  Tue May 29 09:20:54 2007
From: admin at adofms.com.au (admin)
Date: Tue, 29 May 2007 18:50:54 +0930
Subject: My Varnish project
Message-ID: <465BF076.9030804@adofms.com.au>

Hello,

Thank you very much for an excellent product.  I wish to use Varnish
cache as a front end to my system, but I have some problems in using it,
and the way forward probably involves hacking the code. I will run this
past you all first to get your opinion.



Background:

Our site (http://www.adofms.com.au) is a traditional LAMP application,
using PHP / Apache / MySQL.  Users login to the application, and upon
login, a session is created in the database, and a cookie is set for the
user.

The ADOFMS system contains a large number of Units - Fuel Card -
Vehicles for the users to maintain. When a user logs in and gets a valid
session, they can only see the vehicles and cards associated with their
unit. So far so good.

Problem that I have is that the session identification is done entirely
by the cookie - I do not repeat the session ID in the URL.

So for example -

User number 1 visits :
http://www.adofms.com.au/vehicles.php   - and gets a list of vehicles
that they own.

User number 2 visits
http://www.adofms.com.au/vehicles.php - and they get a different list of
vehicles to what user 1 sees.

So, of course, with Varnish, the vehicles.php output from user 1 is
cached, and presented to user 2. This is very quick and efficient, but
not what we want. I need things to be cached on a session by session basis.


I believe I have 3 options here :

1) Re-write the whole PHP application to repeat the session ID as part
of every URL in the system. That is do-able, but boring.

2) Cook up some VCL code to cache pages on a per-session basis, by
appending the req.http.Cookie value to the URL before it is stored in
the cache, and then doing the same thing when looking up the cache. VCL
does not easily allow this to happen though ???? Correct me if I am wrong.

3) Hack the source code of Varnish to use the session ID (from the
cookie) to segregate cached results by session.


I am going to have a go at method 3) anyway, but would like your opinion
before starting out on this adventure  :)   I would think its not too
hard, since I know exactly what I am trying to achieve, which is often
half the battle when coding. I have some similar enhancements that would
suit my application well. I will explain them here:

Example for extra enchancement :

I have a URL such as

http://www.adofms.com.au/vehicles.php?op=view&id=12345
  - Displays the full details of vehicle ID 12345, which can generate a
lot of SQL calls to create. I would love to cache the results of this on
a per user basis.

If the user updates the vehicle - the naming conventions in this
application guarantee that the backend receives a HTTP POST request with
a URL of :
http://www.adofms.com.au/vehicles.php?op=update .. followed by a call to
redisplay the record, which in this case is
http://www.adofms.com.au/vehicles.php?op=view&id=12345&refresh=1

Now - If I am going to hack the Varnish code anyway, I can get around
this by intercepting POST requests that have an op=update GET variable
set, and an id=<some value> POST variable set .. and if so, remove the
existing vehicles.php?op=view&id=<some value>   entry from the cache.

Again, running this idea past you for feedback. I understand that such a
change would be very specific to our application and the way it works,
but thats OK with me.

Another alternative would be to make VCL a little more powerful - a
couple of simple ways of changing the key value for the "obj" variable
would go a long way I think .. and the ability to grab GET and POST
variables would be very handy too.


Would appreciate your thoughts on these issues.

Thank you
Steve OConnor
ADOFMS Chief Developer



From phk at phk.freebsd.dk  Tue May 29 09:55:15 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 29 May 2007 09:55:15 +0000
Subject: My Varnish project 
In-Reply-To: Your message of "Tue, 29 May 2007 18:50:54 +0930."
	<465BF076.9030804@adofms.com.au> 
Message-ID: <46139.1180432515@critter.freebsd.dk>

In message <465BF076.9030804 at adofms.com.au>, admin writes:

>2) Cook up some VCL code to cache pages on a per-session basis, by
>appending the req.http.Cookie value to the URL before it is stored in
>the cache, and then doing the same thing when looking up the cache. VCL
>does not easily allow this to happen though ???? Correct me if I am wrong.

Actually, we just added a facility to do this:

	vcl_hash {
		req.hash += req.http.cookie;
	}

It will (not yet) do the right thing if there are multiple cookie
header lines, but that is in the works.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From admin at adofms.com.au  Tue May 29 10:18:58 2007
From: admin at adofms.com.au (admin)
Date: Tue, 29 May 2007 19:48:58 +0930
Subject: My Varnish project
In-Reply-To: <46139.1180432515@critter.freebsd.dk>
References: <46139.1180432515@critter.freebsd.dk>
Message-ID: <465BFE12.5070001@adofms.com.au>

Poul-Henning Kamp wrote:
> In message <465BF076.9030804 at adofms.com.au>, admin writes:
>
>   
>> 2) Cook up some VCL code to cache pages on a per-session basis, by
>> appending the req.http.Cookie value to the URL before it is stored in
>> the cache, and then doing the same thing when looking up the cache. VCL
>> does not easily allow this to happen though ???? Correct me if I am wrong.
>>     
>
> Actually, we just added a facility to do this:
>
> 	vcl_hash {
> 		req.hash += req.http.cookie;
> 	}
>
> It will (not yet) do the right thing if there are multiple cookie
> header lines, but that is in the works.
>
>   
Excellent - I am on gentoo, using varnish-1.0.4

The man pages for VCL do mention a vcl_hash interface, but it says it's
not implemented yet. Wasnt sure what it was, but I assume that is called
whenever the hash value for the key of the object is calculated ? 

That would be before every insert command, and before every lookup
command ?  In that case, its exactly what I need. I notice that req.hash
is not mentioned in the man pages either.

Will make sure that I have the latest version and have a play again. 
Actually - it wont take a minute, and its all quiet here now (8pm in
Australia), so Ill try it on the production servers now .. :)

Steve


From phk at phk.freebsd.dk  Tue May 29 10:20:27 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 29 May 2007 10:20:27 +0000
Subject: My Varnish project 
In-Reply-To: Your message of "Tue, 29 May 2007 19:48:58 +0930."
	<465BFE12.5070001@adofms.com.au> 
Message-ID: <72869.1180434027@critter.freebsd.dk>

In message <465BFE12.5070001 at adofms.com.au>, admin writes:
>Poul-Henning Kamp wrote:

>The man pages for VCL do mention a vcl_hash interface, but it says it's
>not implemented yet. Wasnt sure what it was, but I assume that is called
>whenever the hash value for the key of the object is calculated ? 

Yes.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Tue May 29 10:28:29 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 29 May 2007 12:28:29 +0200
Subject: My Varnish project
In-Reply-To: <465BFE12.5070001@adofms.com.au> (admin@adofms.com.au's message
	of "Tue\, 29 May 2007 19\:48\:58 +0930")
References: <46139.1180432515@critter.freebsd.dk>
	<465BFE12.5070001@adofms.com.au>
Message-ID: <874plvorea.fsf@des.linpro.no>

admin <admin at adofms.com.au> writes:
> Poul-Henning Kamp <phk at phk.freebsd.dk> writes:
> > Actually, we just added a facility to do this:
> >
> > 	vcl_hash {
> > 		req.hash += req.http.cookie;
> > 	}
> >
> > It will (not yet) do the right thing if there are multiple cookie
> > header lines, but that is in the works.
> Excellent - I am on gentoo, using varnish-1.0.4

1.0.4 does not have this functionality, you'll have to check out the
latest code from Subversion and build / install manually.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From admin at adofms.com.au  Tue May 29 10:29:20 2007
From: admin at adofms.com.au (ADOFMS Admin, SteveOC)
Date: Tue, 29 May 2007 19:59:20 +0930
Subject: My Varnish project
In-Reply-To: <72869.1180434027@critter.freebsd.dk>
References: <72869.1180434027@critter.freebsd.dk>
Message-ID: <465C0080.4050606@adofms.com.au>

Poul-Henning Kamp wrote:
> In message <465BFE12.5070001 at adofms.com.au>, admin writes:
>   
>> Poul-Henning Kamp wrote:
>>     
>
>   
>> The man pages for VCL do mention a vcl_hash interface, but it says it's
>> not implemented yet. Wasnt sure what it was, but I assume that is called
>> whenever the hash value for the key of the object is calculated ? 
>>     
>
> Yes.
>
>   
Thanks

I have this now :
         sub vcl_hash {
             req.hash += req.http.cookie;
         }

and get this when I run varnishd :

(/etc/varnish/adofms.vcl Line 22 Pos 14)
             req.hash += req.http.cookie;
-------------########--------------------

I think I need a more up to date source file ?

When I emerge varnish , it fetches and builds :
mirror://sourceforge/varnish/varnish-1.0.4.tar.gz

Is there a newer version in SVN or something ?




From admin at adofms.com.au  Tue May 29 10:33:23 2007
From: admin at adofms.com.au (ADOFMS Admin, SteveOC)
Date: Tue, 29 May 2007 20:03:23 +0930
Subject: My Varnish project
In-Reply-To: <874plvorea.fsf@des.linpro.no>
References: <46139.1180432515@critter.freebsd.dk>	<465BFE12.5070001@adofms.com.au>
	<874plvorea.fsf@des.linpro.no>
Message-ID: <465C0173.4050704@adofms.com.au>

Dag-Erling Sm?rgrav wrote:
> admin <admin at adofms.com.au> writes:
>   
>> Poul-Henning Kamp <phk at phk.freebsd.dk> writes:
>>     
>>> Actually, we just added a facility to do this:
>>>
>>> 	vcl_hash {
>>> 		req.hash += req.http.cookie;
>>> 	}
>>>
>>> It will (not yet) do the right thing if there are multiple cookie
>>> header lines, but that is in the works.
>>>       
>> Excellent - I am on gentoo, using varnish-1.0.4
>>     
>
> 1.0.4 does not have this functionality, you'll have to check out the
> latest code from Subversion and build / install manually.
>
> DES
>   
You guys are great :)

Doing this now ... shouldnt be long

SteveOC


From des at linpro.no  Tue May 29 10:51:45 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 29 May 2007 12:51:45 +0200
Subject: My Varnish project
In-Reply-To: <465C0080.4050606@adofms.com.au> (ADOFMS Admin's message of "Tue\,
	29 May 2007 19\:59\:20 +0930")
References: <72869.1180434027@critter.freebsd.dk>
	<465C0080.4050606@adofms.com.au>
Message-ID: <87wsyrnbr2.fsf@des.linpro.no>

"ADOFMS Admin, SteveOC" <admin at adofms.com.au> writes:
> I have this now :
>          sub vcl_hash {
>              req.hash += req.http.cookie;
>          }
>
> and get this when I run varnishd :
>
> (/etc/varnish/adofms.vcl Line 22 Pos 14)
>              req.hash += req.http.cookie;
> -------------########--------------------
>
> I think I need a more up to date source file ?

You need the latest sources from Subversion, and you left out the "set"
keyword.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From omar.kilani at gmail.com  Tue May 29 11:01:12 2007
From: omar.kilani at gmail.com (Omar Kilani)
Date: Tue, 29 May 2007 21:01:12 +1000
Subject: Cookie inspection
In-Reply-To: <16610.1180426619@critter.freebsd.dk>
References: <f0cc3e3e0705282205h3b43065atbc1ee434a6adec94@mail.gmail.com>
	<16610.1180426619@critter.freebsd.dk>
Message-ID: <f0cc3e3e0705290401j555afab0qd3affb5c328a0b3d@mail.gmail.com>

On 5/29/07, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> In message <f0cc3e3e0705282205h3b43065atbc1ee434a6adec94 at mail.gmail.com>, "Omar
>  Kilani" writes:
>
> >Doing something like req.http.Cookie.LANGUAGE *could* theoretically do
> >the right thing (and use http_GetHdrField, although some browsers can
> >send multiple values for the same cookie name... :) and *looks* right,
> >but obviously has a very different meaning in VCL where it compiles
> >to:
>
> The Planned syntax is:
>         req.http.Cookie[language]
>
> and just as for http headers, the headers will be read sequentially
> and the last found value is used.

That makes a lot more sense. :)

Out of curiosity, are things like this documented on Trac or somewhere else?

I probably missed it searching the tickets - has this work been started?

> >http://treehou.se/~omar/cookie-inspection-1.patch
>
> Not bad :-)

Hehe - I really just meant to illustrate what I was getting at with
the patch. ;)

It took longer to write the email than to code up the patch, which I
think says a lot about the hackability of the code and how easy it is
to understand.

> >The other thing I'd like to be able to do is have some sort of 'first
> >match over an array of header values' VCL construct, so you could vary
> >the hash by whatever languages in Accept-Language your site supports
>
> Does your backend include a proper Vary: header when it selects on
> language ?

Yes, but I'm not sure how Vary on Accept-Language is supposed to work
- does it just add the entire value to the hash key or?

Wouldn't the hit rate be low on the cached content if this was the case?

For example, I'd serve en-US to all these browsers (whose language
settings I haven't changed):

Opera (which sends the following Accept-Language value):

en,ja;q=0.9,fr;q=0.8,de;q=0.7,es;q=0.6,it;q=0.5,nl;q=0.4,sv;q=0.3,nb;q=0.2,da;q=0.1,fi;q=0.1,pt;q=0.1,lv;q=0.1,zh-CN;q=0.1,zh-TW;q=0.1,ko;q=0.1,en;q=0.1

Safari:

en

Camino:

en,ja;q=0.9,fr;q=0.9,de;q=0.8,es;q=0.8,it;q=0.7,nl;q=0.6,sv;q=0.6,nb;q=0.5,da;q=0.4,fi;q=0.4,pt;q=0.3,lv;q=0.3,zh-Hans;q=0.2,zh-Hant;q=0.1,ko;q=0.1

But would they all be cached independently and served different
versions of the same document?

Thanks.

Regards,
Omar


From admin at adofms.com.au  Tue May 29 11:46:17 2007
From: admin at adofms.com.au (ADOFMS Admin, SteveOC)
Date: Tue, 29 May 2007 21:16:17 +0930
Subject: My Varnish project
In-Reply-To: <87wsyrnbr2.fsf@des.linpro.no>
References: <72869.1180434027@critter.freebsd.dk>	<465C0080.4050606@adofms.com.au>
	<87wsyrnbr2.fsf@des.linpro.no>
Message-ID: <465C1289.1040105@adofms.com.au>

Just to finish this thread then :

I am currently running the latest SVN release of Varnish on our
production server, and implemented the hash upgrade using just VCL.

My VCL setup has :

         sub vcl_hash {
             set req.hash += req.url;
             set req.hash += req.http.host;
             set req.hash += req.http.cookie;
             hash;
         }

Very easy, but it also needs to insert into the cache on vcl_fetch when
resp.http.Set-Cookie is true.  (ie - DONT pass when set-cookie is set)

Its now working fine, and passed a few obvious test cases.

I will run this on our production machine for the next 24 hours. We will
have about 400 login sessions tomorrow with live users each doing an
hours worth of work, so it will be good to keep an eye on things during
this period.

Also - I am building this one with x86_64, and everything looks fine to
me. Its worth flagging it in Gentoo with at least an ~amd64 keyword.

The codebase looks very clean - I will spend some time experimenting
soon, making up some new VCL extensions to solve my other problems and
tailor Varnish to exactly fit my application. My aim here is to make
sure that all the hackery can be done using simple VCL variables, so
that others can get Varnish fitting in with other weird setups too.

I really like this project, and its good break from PHP coding. I miss
my C compiler !.

Thanks again guys.
SteveOC


From thomas.westlund at aftenposten.no  Wed May 30 07:03:45 2007
From: thomas.westlund at aftenposten.no (Thomas Westlund)
Date: Wed, 30 May 2007 09:03:45 +0200
Subject: My Varnish project
In-Reply-To: <465C1289.1040105@adofms.com.au>
References: <72869.1180434027@critter.freebsd.dk>
	<465C0080.4050606@adofms.com.au> <87wsyrnbr2.fsf@des.linpro.no>
	<465C1289.1040105@adofms.com.au>
Message-ID: <20070530070345.GB3993@aftenposten.no>

Hi,

If I would like to do a check based on the response code from the backend how can I do this in VCL

I would like to be able to not cache 404's or to set the timeout to a very low number for 404's.

I have checked the vcl man page, but I'm unable to find which statement to use for the if clause.

-- 

Thomas Westlund
Aftenposten AS, UNIX/nettverksavd.
Postboks 1, 0051 Oslo
Tlf: +47 98 20 30 33
Fax: +47 22 86 40 74


From thomas.westlund at aftenposten.no  Wed May 30 07:04:46 2007
From: thomas.westlund at aftenposten.no (Thomas Westlund)
Date: Wed, 30 May 2007 09:04:46 +0200
Subject: Checking http response codes.
In-Reply-To: <20070530070345.GB3993@aftenposten.no>
References: <72869.1180434027@critter.freebsd.dk>
	<465C0080.4050606@adofms.com.au> <87wsyrnbr2.fsf@des.linpro.no>
	<465C1289.1040105@adofms.com.au>
	<20070530070345.GB3993@aftenposten.no>
Message-ID: <20070530070446.GC3993@aftenposten.no>

Hi,

Sorry this message got sent whith the wrong subject :/

On Wed, May 30, 2007 at 09:03:45AM +0200, Thomas Westlund wrote:
> Hi,
> 
> If I would like to do a check based on the response code from the backend how can I do this in VCL
> 
> I would like to be able to not cache 404's or to set the timeout to a very low number for 404's.
> 
> I have checked the vcl man page, but I'm unable to find which statement to use for the if clause.
> 
> -- 
> 
> Thomas Westlund
> Aftenposten AS, UNIX/nettverksavd.
> Postboks 1, 0051 Oslo
> Tlf: +47 98 20 30 33
> Fax: +47 22 86 40 74

-- 

Thomas Westlund
Aftenposten AS, UNIX/nettverksavd.
Postboks 1, 0051 Oslo
Tlf: +47 98 20 30 33
Fax: +47 22 86 40 74


From thomas.westlund at aftenposten.no  Wed May 30 07:26:27 2007
From: thomas.westlund at aftenposten.no (Thomas Westlund)
Date: Wed, 30 May 2007 09:26:27 +0200
Subject: Checking http response codes
Message-ID: <20070530072627.GA30518@aftenposten.no>

Hi,

I'm having av bit of keyboard trouble this morning, so forgive me if i post this question more than once, (too early still perhaps ;)

If I would like to do a check based on the response code from the backend how can I do this in VCL

I would like to be able to not cache 404's or to set the timeout to a very low number for 404's.

I have checked the vcl man page, but I'm unable to find which statement to use for the if clause

-- 

Thomas Westlund
Aftenposten AS, UNIX/nettverksavd.
Postboks 1, 0051 Oslo
Tlf: +47 98 20 30 33
Fax: +47 22 86 40 74


From ccripy at gmail.com  Wed May 30 15:55:53 2007
From: ccripy at gmail.com (cripy)
Date: Wed, 30 May 2007 11:55:53 -0400
Subject: Checking http response codes
In-Reply-To: <20070530072627.GA30518@aftenposten.no>
References: <20070530072627.GA30518@aftenposten.no>
Message-ID: <dacbdcd10705300855m67e654dcs45b5b42c4e5c86bc@mail.gmail.com>

It's currently not possible to do this in VCL that I know of...I have come
up with a solution for a similar issue I had with varnish cache'ing 401
Unauthorized responses, thus denying legit authenticated users the proper
content because they are being serviced the 401.  What I did is jumped into
the function cnt_fetch in bin/varnishd/cache_center.c here's a snippet...

--- /tmp/clean/varnish-1.0.4/bin/varnishd/cache_center.c        2007-05-20
10:38:16.000000000 -0700
+++ cache_center.c      2007-05-23 14:55:43.000000000 -0700
@@ -286,7 +286,8 @@
 cnt_fetch(struct sess *sp)
 {

-
+       struct vbe_conn *vc;
+
        if (Fetch(sp)) {
                sp->obj->cacheable = 0;
                HSH_Unbusy(sp->obj);
@@ -300,6 +301,11 @@
        RFC2616_cache_policy(sp, &sp->obj->http);       /* XXX -> VCL */

        VCL_fetch_method(sp);
+       vc = VBE_GetFd(sp);
+
+       if (http_GetStatus(vc->http) == 401)
+               sp->handling = VCL_RET_PASS;
+        VBE_ClosedFd(sp->wrk, vc, 0);

        if (sp->handling == VCL_RET_ERROR)
                INCOMPL();
@@ -353,6 +359,64 @@
        sp->step = STP_DONE;
        return (0);
 }



On 5/30/07, Thomas Westlund <thomas.westlund at aftenposten.no> wrote:
>
> Hi,
>
> I'm having av bit of keyboard trouble this morning, so forgive me if i
> post this question more than once, (too early still perhaps ;)
>
> If I would like to do a check based on the response code from the backend
> how can I do this in VCL
>
> I would like to be able to not cache 404's or to set the timeout to a very
> low number for 404's.
>
> I have checked the vcl man page, but I'm unable to find which statement to
> use for the if clause
>
> --
>
> Thomas Westlund
> Aftenposten AS, UNIX/nettverksavd.
> Postboks 1, 0051 Oslo
> Tlf: +47 98 20 30 33
> Fax: +47 22 86 40 74
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070530/7eaa44ca/attachment.html>

From nwmcsween at gmail.com  Wed May 30 22:32:22 2007
From: nwmcsween at gmail.com (Nathan McSween)
Date: Wed, 30 May 2007 15:32:22 -0700
Subject: Varnish and ultramokey L7 switching?
Message-ID: <000001c7a30a$65790960$c91fa105@octane>

I highly doubt this is possible but is there any way I could implement layer
7 load balancing while using varnish and without multiple machines?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20070530/ee423d8f/attachment.html>