From des at linpro.no  Tue May  1 18:50:24 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Tue, 01 May 2007 20:50:24 +0200
Subject: Child crashes when the backend is down
Message-ID: <ujrejm0v0nj.fsf@false.linpro.no>

This is probably what's killing my customer up north:

> % varnishd -a localhost:8080 -b localhost:80 -sfile,/tmp,4m -d -d

[note that there is no server at localhost:80]

> file /tmp/varnish.TIGksM (unlinked) size 4194304 bytes (1024 fs-blocks, 1024 pages)
> Using old SHMFILE
> rolling(1)...
> rolling(2)...
> start
> CLI <start>
> start child pid 7023
> 200 0
>
> Child said (2, 7023): <<Child starts
> managed to mmap 4194304 bytes of 4194304
> Ready
> CLI ready
> >>

[now point firefox at http://localhost:8080/]

> Child said (2, 7023): <<socktest: linger=0 sndtimeo=0 rcvtimeo=0
> >>
> Child not responding to ping

[because it is busy dumping core]

> Cache child died pid=7023 status=0x8b

[0x8b = SIGSEGV]

> Clean child
> Child cleaned
> start child pid 7039
> Child said (2, 7039): <<Child starts
> managed to mmap 4194304 bytes of 4194304
> Ready
> CLI ready
> >>
> Manager got SIGINT
> Clean child
> Child stopping
> unlink /tmp/vcl.XX9MHO5n

[let's look at that core file]

> % gdb =varnishd ~/core
> GNU gdb 6.6-debian
> Copyright (C) 2006 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu"...
> Using host libthread_db library "/lib/libthread_db.so.1".
> Error while mapping shared library sections:
> /tmp/vcl.XX9MHO5n: No such file or directory.
> Reading symbols from /opt/varnish/lib/libvarnish.so.0...done.
> Loaded symbols for /opt/varnish/lib/libvarnish.so.0
> Reading symbols from /opt/varnish/lib/libvcl.so.0...done.
> Loaded symbols for /opt/varnish/lib/libvcl.so.0
> Reading symbols from /lib/libdl.so.2...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /lib/librt.so.1...done.
> Loaded symbols for /lib/librt.so.1
> Reading symbols from /lib/libpthread.so.0...done.
> Loaded symbols for /lib/libpthread.so.0
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /lib/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib/libnss_files.so.2...done.
> Loaded symbols for /lib/libnss_files.so.2
> Symbol file not found for /tmp/vcl.XX9MHO5n
> Core was generated by `varnishd -a localhost:8080 -b localhost:80 -sfile,/tmp,4m -d -d'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x00002afef513d3d9 in pthread_mutex_lock () from /lib/libpthread.so.0
> (gdb) where
> #0  0x00002afef513d3d9 in pthread_mutex_lock () from /lib/libpthread.so.0
> #1  0x000000000040cfa3 in HSH_Unbusy (o=0x64e840) at cache_hash.c:195
> #2  0x00000000004095fc in cnt_fetch (sp=0x64c308) at cache_center.c:292
> #3  0x000000000040a428 in CNT_Session (sp=0x64c308) at steps.h:40
> #4  0x0000000000411c2b in wrk_do_one (w=0x43002c50) at cache_pool.c:191
> #5  0x0000000000411fe7 in wrk_thread (priv=0x632340) at cache_pool.c:231
> #6  0x00002afef513b2a5 in start_thread () from /lib/libpthread.so.0
> #7  0x00002afef541f61d in clone () from /lib/libc.so.6
> #8  0x0000000000000000 in ?? ()
> (gdb) up
> #1  0x000000000040cfa3 in HSH_Unbusy (o=0x64e840) at cache_hash.c:195
> 195             LOCK(&o->objhead->mtx);
> (gdb) l
> 190             CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
> 191             assert(o->busy);
> 192             assert(o->refcnt > 0);
> 193             if (o->cacheable)
> 194                     EXP_Insert(o);
> 195             LOCK(&o->objhead->mtx);
> 196             o->busy = 0;
> 197             UNLOCK(&o->objhead->mtx);
> 198             while (1) {
> 199                     sp = TAILQ_FIRST(&o->waitinglist);
> (gdb) p *o
> $1 = {magic = 847584578, refcnt = 1, xid = 780407261, objhead = 0x0,
>   heap_idx = 0, ban_seq = 0, pass = 0, response = 0, valid = 0, cacheable = 0,
>   busy = 1, len = 0, age = 0, entered = 0, ttl = 0, last_modified = 0, http = {
>     magic = 1680389577, s = 0x0, t = 0x0, v = 0x0, f = 0x0, e = 0x0,
>     conds = 0 '\0', logtag = HTTP_Rx, hd = {{b = 0x0,
>         e = 0x0} <repeats 32 times>}, hdf = '\0' <repeats 31 times>, nhd = 0},
>   list = {tqe_next = 0x0, tqe_prev = 0x0}, deathrow = {tqe_next = 0x0,
>     tqe_prev = 0x0}, store = {tqh_first = 0x0, tqh_last = 0x64eb18},
>   waitinglist = {tqh_first = 0x0, tqh_last = 0x64eb28}}

There is no o->objhead because no object was ever retrieved and
Fetch() returned early.

Why is o->busy protected by o->objhead->mtx?  Why not have a mutex
directly in struct object instead?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue May  1 19:21:59 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Tue, 01 May 2007 21:21:59 +0200
Subject: vcl and cache-control header
In-Reply-To: <ujr4pmxyigf.fsf@false.linpro.no> (Dag-Erling
	=?iso-8859-1?Q?Sm=F8rgrav's?= message
	of "Mon, 30 Apr 2007 17:45:20 +0200")
References: <3947.1177442385@critter.freebsd.dk>
	<2FE660AF-943E-4D8E-8041-8E34EE354779@zeno.org>
	<ujr4pmxyigf.fsf@false.linpro.no>
Message-ID: <ujrabwouz6w.fsf@false.linpro.no>

des at linpro.no (Dag-Erling Sm?rgrav) writes:
> OK, I'm not sure we've ever tested going from hit to pass, although
> it's meant to work.  Could you please open a ticket about this?  Call
> it "returning pass from vcl_hit crashes the child" or something.

Looking at the code, I can definitely say that it doesn't work,
although it was clearly intended (and promptly forgotten) at some
point.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From phk at phk.freebsd.dk  Wed May  2 07:34:16 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 02 May 2007 07:34:16 +0000
Subject: Child crashes when the backend is down 
In-Reply-To: Your message of "Tue, 01 May 2007 20:50:24 +0200."
	<ujrejm0v0nj.fsf@false.linpro.no> 
Message-ID: <45172.1178091256@critter.freebsd.dk>

In message <ujrejm0v0nj.fsf at false.linpro.no>, Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= writes:

>This is probably what's killing my customer up north:

>There is no o->objhead because no object was ever retrieved and
>Fetch() returned early.
>
>Why is o->busy protected by o->objhead->mtx?  Why not have a mutex
>directly in struct object instead?

o->busy is protected by the objhead mutex because otherwise we would have
to grab all the objects mutexes in the hash lookup.

The busy flag is very seldomly manipulated, and this is much more efficient.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Wed May  2 08:43:06 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Wed, 02 May 2007 10:43:06 +0200
Subject: Child crashes when the backend is down
In-Reply-To: <45172.1178091256@critter.freebsd.dk> (Poul-Henning Kamp's
	message of "Wed, 02 May 2007 07:34:16 +0000")
References: <45172.1178091256@critter.freebsd.dk>
Message-ID: <ujr1whzvco5.fsf@false.linpro.no>

"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
> o->busy is protected by the objhead mutex because otherwise we would have
> to grab all the objects mutexes in the hash lookup.
>
> The busy flag is very seldomly manipulated, and this is much more efficient.

Fine, but what is the locking protocol when o has no objhead?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From phk at phk.freebsd.dk  Wed May  2 08:46:14 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 02 May 2007 08:46:14 +0000
Subject: Child crashes when the backend is down 
In-Reply-To: Your message of "Wed, 02 May 2007 10:43:06 +0200."
	<ujr1whzvco5.fsf@false.linpro.no> 
Message-ID: <45542.1178095574@critter.freebsd.dk>

In message <ujr1whzvco5.fsf at false.linpro.no>, Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= writes:
>"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
>> o->busy is protected by the objhead mutex because otherwise we would have
>> to grab all the objects mutexes in the hash lookup.
>>
>> The busy flag is very seldomly manipulated, and this is much more efficie=
>nt.
>
>Fine, but what is the locking protocol when o has no objhead?

Obviously nobody else will ever check the busy flag in that case,
so the locking isn't necessary.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Wed May  2 13:59:08 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Wed, 02 May 2007 15:59:08 +0200
Subject: Child crashes when the backend is down
In-Reply-To: <45542.1178095574@critter.freebsd.dk> (Poul-Henning Kamp's
	message of "Wed, 02 May 2007 08:46:14 +0000")
References: <45542.1178095574@critter.freebsd.dk>
Message-ID: <ujrabwnz5qr.fsf@false.linpro.no>

"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
> Obviously nobody else will ever check the busy flag in that case,
> so the locking isn't necessary.

HSH_Ref() and HSH_Deref() already check that o->objhead != NULL before
trying to lock the mutex, so I applied the same logic to HSH_Unbusy().
Is this a suitable fix?

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: varnish-unbusy.diff
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20070502/fd801d36/attachment.ksh>

From phk at phk.freebsd.dk  Wed May  2 13:59:47 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 02 May 2007 13:59:47 +0000
Subject: Child crashes when the backend is down 
In-Reply-To: Your message of "Wed, 02 May 2007 15:59:08 +0200."
	<ujrabwnz5qr.fsf@false.linpro.no> 
Message-ID: <47251.1178114387@critter.freebsd.dk>

In message <ujrabwnz5qr.fsf at false.linpro.no>, Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= writes:
>--=-=-=
>Content-Type: text/plain; charset=iso-8859-1
>Content-Transfer-Encoding: quoted-printable
>
>"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
>> Obviously nobody else will ever check the busy flag in that case,
>> so the locking isn't necessary.
>
>HSH_Ref() and HSH_Deref() already check that o->objhead !=3D NULL before
>trying to lock the mutex, so I applied the same logic to HSH_Unbusy().
>Is this a suitable fix?

Yes, I think so.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From barry at automattic.com  Fri May 11 02:31:51 2007
From: barry at automattic.com (Barry Abrahamson)
Date: Thu, 10 May 2007 19:31:51 -0700
Subject: Varnish crashing in production
Message-ID: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>

Hi,

We are currently testing Varnish as a reverse proxy to serve images.   
We are running into problems where the varnishd child (and sometimes  
parent), will die unexpectedly.  When the child process dies, the  
parent restarts it and when the parent process dies, it is restarted  
by our monitoring systems.  Sometimes this happens every 5 minutes,  
sometimes it can stay running up to 2 hours.  I don't see any pattern  
as to when it is failing.  In our development environment, with no  
"real" traffic, but the same configuration, we are unable to  
replicate the crash.  The vcl file we are using is basically the  
default one, except in the fetch subroutine, the we only insert  
objects if they have a certain header and pass everything else.  The  
core dumps created as a result of the crash all show this:

Program terminated with signal 6, Aborted.
Cannot access memory at address 0x2a9566be08

But I am not really sure where to go from here to track down the  
cause of this issue.

I have tried using many different cache sizes -- currently it is set  
to 1GB which is less than the amount of RAM in the machine.  The  
restarts occur way before the cache is full.  Any help would be much  
appreciated.

Thanks,

Barry



From ask at develooper.com  Fri May 11 05:26:50 2007
From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Thu, 10 May 2007 22:26:50 -0700
Subject: Varnish crashing in production
In-Reply-To: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
Message-ID: <B358E220-602B-42C4-860A-45D917D58F77@develooper.com>


On May 10, 2007, at 7:31 PM, Barry Abrahamson wrote:

> We are currently testing Varnish as a reverse proxy to serve  
> images.  We are running into problems where the varnishd child (and  
> sometimes parent), will die unexpectedly.

FWIW I am seeing something similar.  I haven't seen anything in the  
syslog or tried running varnishlog for long enough to catch it  
happening (which I why I didn't post).  In this installation (on  
RHEL4) increasing the cache size did seem to help on the frequency.

It never happens when it's idle-ish (<100-150 rps) but in the busy  
hours (2-500rps) it happens frequently.   In the cache hit ratio (I  
love munin!) you can see that pretty clearly - stable or slowly  
increasing average ratio during the off-peak hours - when it's busy  
it drops every time there's a crash...

	http://tmp.askask.com/2007/05/varnish_cachehitratio-day.png


  - ask

-- 
http://develooper.com/ - http://askask.com/




From des at linpro.no  Fri May 11 06:57:06 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Fri, 11 May 2007 08:57:06 +0200
Subject: Varnish crashing in production
In-Reply-To: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com> (Barry
	Abrahamson's message of "Thu\, 10 May 2007 19\:31\:51 -0700")
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
Message-ID: <87d517x2yl.fsf@des.linpro.no>

Barry Abrahamson <barry at automattic.com> writes:
> We are currently testing Varnish as a reverse proxy to serve images.
> We are running into problems where the varnishd child (and sometimes
> parent), will die unexpectedly.  When the child process dies, the
> parent restarts it and when the parent process dies, it is restarted
> by our monitoring systems.  Sometimes this happens every 5 minutes,
> sometimes it can stay running up to 2 hours.  I don't see any pattern
> as to when it is failing.  In our development environment, with no
> "real" traffic, but the same configuration, we are unable to
> replicate the crash.  The vcl file we are using is basically the
> default one, except in the fetch subroutine, the we only insert
> objects if they have a certain header and pass everything else.  The
> core dumps created as a result of the crash all show this:

1.0.3 has a known bug which will cause the child process to crash if
something unexpected happens while retrieving an object from the
backend.  Please try again with the tip of the 1.0 branch.

> Program terminated with signal 6, Aborted.

That's an assertion failure.  If you run the management process in the
foreground (start varnishd manually with -d -d, then type "start" to
kick off the child process) you should see an error message when the
child dies.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From perbu at linpro.no  Fri May 11 07:59:17 2007
From: perbu at linpro.no (Per Andreas Buer)
Date: Fri, 11 May 2007 09:59:17 +0200
Subject: Varnish crashing in production
In-Reply-To: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
Message-ID: <46442255.7050305@linpro.no>

Barry Abrahamson wrote:
> Hi,
>
> We are currently testing Varnish as a reverse proxy to serve images. 
> We are running into problems where the varnishd child (and sometimes
> parent), will die unexpectedly.

If you could supply
 * varnish version
 * OS version
 * how you invoke varnish (command line)
 * architecture
people might help you better.

>   ..
> I have tried using many different cache sizes -- currently it is set
> to 1GB which is less than the amount of RAM in the machine.  The
> restarts occur way before the cache is full.  Any help would be much
> appreciated.

Use should use a much bigger cache. The cache should never fill up - so
use 36-150GB is you can.


-- 
Per Andreas Buer / Linpro AS   t: 21 54 41 21 / m: 958 39 117                   
http://linpro.no/ - Ledende p? Linux og ?pen kildekode.



From andrearo at pvv.ntnu.no  Fri May 11 08:05:09 2007
From: andrearo at pvv.ntnu.no (=?ISO-8859-1?Q?Andreas_R=F8sdal?=)
Date: Fri, 11 May 2007 10:05:09 +0200 (CEST)
Subject: Varnish crashing in production
In-Reply-To: <B358E220-602B-42C4-860A-45D917D58F77@develooper.com>
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
	<B358E220-602B-42C4-860A-45D917D58F77@develooper.com>
Message-ID: <20070511100242.M33832@verden.pvv.ntnu.no>

On Thu, 10 May 2007, Ask Bj?rn Hansen wrote:
> On May 10, 2007, at 7:31 PM, Barry Abrahamson wrote:
>
>> We are currently testing Varnish as a reverse proxy to serve images.  We 
>> are running into problems where the varnishd child (and sometimes parent), 
>> will die unexpectedly.
>
> FWIW I am seeing something similar.  I haven't seen anything in the syslog or 
> tried running varnishlog for long enough to catch it happening (which I why I 
> didn't post).  In this installation (on RHEL4) increasing the cache size did 
> seem to help on the frequency.
>
> It never happens when it's idle-ish (<100-150 rps) but in the busy hours 
> (2-500rps) it happens frequently.   In the cache hit ratio (I love munin!) 
> you can see that pretty clearly - stable or slowly increasing average ratio 
> during the off-peak hours - when it's busy it drops every time there's a 
> crash...
>
> 	http://tmp.askask.com/2007/05/varnish_cachehitratio-day.png

Interesting..  How did you setup rrdtool to graph the hit ratio?


  - Andreas R.

From barry at automattic.com  Fri May 11 08:18:22 2007
From: barry at automattic.com (Barry Abrahamson)
Date: Fri, 11 May 2007 01:18:22 -0700
Subject: Varnish crashing in production
In-Reply-To: <87d517x2yl.fsf@des.linpro.no>
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
	<87d517x2yl.fsf@des.linpro.no>
Message-ID: <3967FAB3-FBBC-41CB-A735-DB77735E00FD@automattic.com>


On May 10, 2007, at 11:57 PM, Dag-Erling Sm?rgrav wrote:

> 1.0.3 has a known bug which will cause the child process to crash if
> something unexpected happens while retrieving an object from the
> backend.  Please try again with the tip of the 1.0 branch.

Thanks for the info.  I will try the upgrade and let you know how it  
goes.

>
>> Program terminated with signal 6, Aborted.
>
> That's an assertion failure.  If you run the management process in the
> foreground (start varnishd manually with -d -d, then type "start" to
> kick off the child process) you should see an error message when the
> child dies.

I will try this as well.  Thanks again.

Barry



From barry at automattic.com  Fri May 11 08:21:21 2007
From: barry at automattic.com (Barry Abrahamson)
Date: Fri, 11 May 2007 01:21:21 -0700
Subject: Varnish crashing in production
In-Reply-To: <46442255.7050305@linpro.no>
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
	<46442255.7050305@linpro.no>
Message-ID: <5D9792FC-0409-433F-809B-631436111C95@automattic.com>

On May 11, 2007, at 12:59 AM, Per Andreas Buer wrote:


>  * varnish version

1.0.3

>  * OS version

Debian Sarge AMD64

>  * how you invoke varnish (command line)

/usr/local/sbin/varnishd -a[IP address] -f/etc/varnish/varnish.vcl - 
sfile,/var/lib/varnish/cache,8G -t300s -T[IP address]:6969

>  * architecture

x86_64

> Use should use a much bigger cache. The cache should never fill up  
> - so
> use 36-150GB is you can.

Ok, I will make the cache bigger.


Thanks,

Barry




From ask at develooper.com  Fri May 11 08:39:39 2007
From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Fri, 11 May 2007 01:39:39 -0700
Subject: Varnish crashing in production
In-Reply-To: <20070511100242.M33832@verden.pvv.ntnu.no>
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
	<B358E220-602B-42C4-860A-45D917D58F77@develooper.com>
	<20070511100242.M33832@verden.pvv.ntnu.no>
Message-ID: <19522D99-D243-40BD-AB15-B855E04AC2BC@develooper.com>


On May 11, 2007, at 1:05 AM, Andreas R?sdal wrote:

>> 	http://tmp.askask.com/2007/05/varnish_cachehitratio-day.png
>
> Interesting..  How did you setup rrdtool to graph the hit ratio?

Munin - http://munin.projects.linpro.no/ - and this plugin: http:// 
xrl.us/wbmz (Link to muninexchange.projects.linpro.no)



   - ask

-- 
http://develooper.com/ - http://askask.com/




From ask at develooper.com  Fri May 11 08:41:12 2007
From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Fri, 11 May 2007 01:41:12 -0700
Subject: Varnish crashing in production
In-Reply-To: <5D9792FC-0409-433F-809B-631436111C95@automattic.com>
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
	<46442255.7050305@linpro.no>
	<5D9792FC-0409-433F-809B-631436111C95@automattic.com>
Message-ID: <D87B0EBE-D260-4124-BB09-0A17760DAFFB@develooper.com>


> Use should use a much bigger cache. The cache should never fill up  
> - so
> use 36-150GB is you can.

Ah - that is probably my problem.  It is on $client server and for  
some reason they are using a 32-bit installation so I am limited to  
2GB (which for how this particular data is used is fine).

I am using trunk at r1394 - so it shouldn't be the 1.0.3 issue.


   - ask

-- 
http://develooper.com/ - http://askask.com/




From des at linpro.no  Fri May 11 08:41:55 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Fri, 11 May 2007 10:41:55 +0200
Subject: Varnish crashing in production
In-Reply-To: <5D9792FC-0409-433F-809B-631436111C95@automattic.com> (Barry
	Abrahamson's message of "Fri\, 11 May 2007 01\:21\:21 -0700")
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
	<46442255.7050305@linpro.no>
	<5D9792FC-0409-433F-809B-631436111C95@automattic.com>
Message-ID: <87k5vfvjjg.fsf@des.linpro.no>

Barry Abrahamson <barry at automattic.com> writes:
> On May 11, 2007, at 12:59 AM, Per Andreas Buer wrote:
> > Use should use a much bigger cache. The cache should never fill up -
> > so use 36-150GB is you can.
> Ok, I will make the cache bigger.

Within reason - there is no point in using a cache file that is many
times the size of your data set.  I'm not sure how the allocator works
in detail, but somewhere between 1x and 2x the size of your data set
should be sufficient.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From perbu at linpro.no  Fri May 11 08:46:22 2007
From: perbu at linpro.no (Per Andreas Buer)
Date: Fri, 11 May 2007 10:46:22 +0200
Subject: Varnish crashing in production
In-Reply-To: <87k5vfvjjg.fsf@des.linpro.no>
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>	<46442255.7050305@linpro.no>	<5D9792FC-0409-433F-809B-631436111C95@automattic.com>
	<87k5vfvjjg.fsf@des.linpro.no>
Message-ID: <46442D5E.4020505@linpro.no>

Dag-Erling Sm?rgrav wrote:
> Within reason - there is no point in using a cache file that is many
> times the size of your data set.  I'm not sure how the allocator works
> in detail, but somewhere between 1x and 2x the size of your data set
> should be sufficient.
>   

Is an "out of memory" situation handled properly now?


-- 
Per Andreas Buer / Linpro AS   t: 21 54 41 21 / m: 958 39 117                   
http://linpro.no/ - Ledende p? Linux og ?pen kildekode.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20070511/14b38d20/attachment.bin>

From barry at automattic.com  Fri May 11 08:47:20 2007
From: barry at automattic.com (Barry Abrahamson)
Date: Fri, 11 May 2007 01:47:20 -0700
Subject: Varnish crashing in production
In-Reply-To: <87k5vfvjjg.fsf@des.linpro.no>
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
	<46442255.7050305@linpro.no>
	<5D9792FC-0409-433F-809B-631436111C95@automattic.com>
	<87k5vfvjjg.fsf@des.linpro.no>
Message-ID: <041B2C7C-55C7-4535-B806-93A643D415D4@automattic.com>

> Within reason - there is no point in using a cache file that is many
> times the size of your data set.  I'm not sure how the allocator works
> in detail, but somewhere between 1x and 2x the size of your data set
> should be sufficient.

Varnish is sitting in front of about 600GB of data currently.  We add  
a header to files that were requested more than X times in Y seconds  
and tell varnish to cache anything with that header and don't cache  
rest.  When we were trying to cache everything we were running into  
serious IO issues (not to mention that we would run out of space  
eventually).   I am not sure how big that dataset is going to get.   
The Expires headers on the content is 1 week.  Once the crashing  
issue is resolved and varnish stays running for a while I will be  
able to get a better idea.

Thanks again,

Barry


From des at linpro.no  Fri May 11 08:48:15 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Fri, 11 May 2007 10:48:15 +0200
Subject: Varnish crashing in production
In-Reply-To: <46442D5E.4020505@linpro.no> (Per Andreas Buer's message of "Fri\,
	11 May 2007 10\:46\:22 +0200")
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
	<46442255.7050305@linpro.no>
	<5D9792FC-0409-433F-809B-631436111C95@automattic.com>
	<87k5vfvjjg.fsf@des.linpro.no> <46442D5E.4020505@linpro.no>
Message-ID: <87bqgrvj8w.fsf@des.linpro.no>

Per Andreas Buer <perbu at linpro.no> writes:
> Dag-Erling Sm?rgrav wrote:
> > Within reason - there is no point in using a cache file that is many
> > times the size of your data set.  I'm not sure how the allocator works
> > in detail, but somewhere between 1x and 2x the size of your data set
> > should be sufficient.
> Is an "out of memory" situation handled properly now?

No, but it won't occur if your cache is larger than your data set (and I
mean "data set", not "working set" - thanks to the likes of Google, most
of your data set will be in your working set at some point or other)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From barry at automattic.com  Fri May 11 08:51:56 2007
From: barry at automattic.com (Barry Abrahamson)
Date: Fri, 11 May 2007 01:51:56 -0700
Subject: Varnish crashing in production
In-Reply-To: <87bqgrvj8w.fsf@des.linpro.no>
References: <5E1695DB-A29D-49D5-B0C1-9AC293F7F74D@automattic.com>
	<46442255.7050305@linpro.no>
	<5D9792FC-0409-433F-809B-631436111C95@automattic.com>
	<87k5vfvjjg.fsf@des.linpro.no> <46442D5E.4020505@linpro.no>
	<87bqgrvj8w.fsf@des.linpro.no>
Message-ID: <DB54B261-9D62-4909-ABC9-447D7A2FE1E5@automattic.com>


On May 11, 2007, at 1:48 AM, Dag-Erling Sm?rgrav wrote:

> Per Andreas Buer <perbu at linpro.no> writes:
>> Dag-Erling Sm?rgrav wrote:
>>> Within reason - there is no point in using a cache file that is many
>>> times the size of your data set.  I'm not sure how the allocator  
>>> works
>>> in detail, but somewhere between 1x and 2x the size of your data set
>>> should be sufficient.
>> Is an "out of memory" situation handled properly now?
>
> No, but it won't occur if your cache is larger than your data set  
> (and I
> mean "data set", not "working set" - thanks to the likes of Google,  
> most
> of your data set will be in your working set at some point or other)

Yeah, it doesn't really apply to images (which is all we are caching  
right now), but once we start looking at caching actual html pages,  
robots will kill the cache (going on the assumption we don't have  
space to cache everything).  I think by keeping track of the number  
of hits within a given time period and only caching "hot" content, it  
should resolve this issue.

Barry





From andrearo at pvv.ntnu.no  Fri May 11 14:53:53 2007
From: andrearo at pvv.ntnu.no (=?ISO-8859-1?Q?Andreas_R=F8sdal?=)
Date: Fri, 11 May 2007 16:53:53 +0200 (CEST)
Subject: Varnish ICP?
Message-ID: <20070511165155.D37842@verden.pvv.ntnu.no>

Hello!  Is ICPv2 included in the plans for varnish 2.0?

  - Andreas R.


From phk at phk.freebsd.dk  Fri May 11 14:58:55 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 11 May 2007 14:58:55 +0000
Subject: Varnish ICP? 
In-Reply-To: Your message of "Fri, 11 May 2007 16:53:53 +0200."
	<20070511165155.D37842@verden.pvv.ntnu.no> 
Message-ID: <5150.1178895535@critter.freebsd.dk>

In message <20070511165155.D37842 at verden.pvv.ntnu.no>, =?ISO-8859-1?Q?Andreas_R
=F8sdal?= writes:

>Hello!  Is ICPv2 included in the plans for varnish 2.0?

No.

Our thinking on ICP in general is that a two-layer setup is a better
use of HW-resources.

ICP was for when several client side caches are on the same LAN but have
slow lines out into the world.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From andrearo at pvv.ntnu.no  Sun May 13 12:52:37 2007
From: andrearo at pvv.ntnu.no (=?ISO-8859-1?Q?Andreas_R=F8sdal?=)
Date: Sun, 13 May 2007 14:52:37 +0200 (CEST)
Subject: sendfile changes
Message-ID: <20070513144559.D67788@verden.pvv.ntnu.no>

Hello,
I have a question about a recent change done in SVN trunk:
http://varnish.projects.linpro.no/changeset/1417

"Make the sendfile threshold inifinity for now, we have evidence of
sendfile not doing it's job in a number of operating system (-versions ?)"

What does "not doing it's job" mean here? Sending corrupt or incomplete 
data?

   - Andreas R.


From phk at phk.freebsd.dk  Sun May 13 12:53:32 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 13 May 2007 12:53:32 +0000
Subject: sendfile changes 
In-Reply-To: Your message of "Sun, 13 May 2007 14:52:37 +0200."
	<20070513144559.D67788@verden.pvv.ntnu.no> 
Message-ID: <59179.1179060812@critter.freebsd.dk>

In message <20070513144559.D67788 at verden.pvv.ntnu.no>, =?ISO-8859-1?Q?Andreas_R
=F8sdal?= writes:
>Hello,
>I have a question about a recent change done in SVN trunk:
>http://varnish.projects.linpro.no/changeset/1417
>
>"Make the sendfile threshold inifinity for now, we have evidence of
>sendfile not doing it's job in a number of operating system (-versions ?)"
>
>What does "not doing it's job" mean here? Sending corrupt or incomplete 
>data?

One, the other or even Both, depending on your kernel.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From andrearo at pvv.ntnu.no  Sun May 13 13:55:29 2007
From: andrearo at pvv.ntnu.no (=?ISO-8859-1?Q?Andreas_R=F8sdal?=)
Date: Sun, 13 May 2007 15:55:29 +0200 (CEST)
Subject: sendfile changes 
In-Reply-To: <59179.1179060812@critter.freebsd.dk>
References: <59179.1179060812@critter.freebsd.dk>
Message-ID: <20070513155325.O68470@verden.pvv.ntnu.no>

On Sun, 13 May 2007, Poul-Henning Kamp wrote:
> In message <20070513144559.D67788 at verden.pvv.ntnu.no>, =?ISO-8859-1?Q?Andreas_R
> =F8sdal?= writes:
>> Hello,
>> I have a question about a recent change done in SVN trunk:
>> http://varnish.projects.linpro.no/changeset/1417
>>
>> "Make the sendfile threshold inifinity for now, we have evidence of
>> sendfile not doing it's job in a number of operating system (-versions ?)"
>>
>> What does "not doing it's job" mean here? Sending corrupt or incomplete
>> data?
>
> One, the other or even Both, depending on your kernel.

Yes, I've seen both. What is the best workaround for this problem?
Should I install the varnish from SVN trunk?

  - Andreas R.


From phk at phk.freebsd.dk  Sun May 13 15:47:30 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 13 May 2007 15:47:30 +0000
Subject: sendfile changes 
In-Reply-To: Your message of "Sun, 13 May 2007 15:55:29 +0200."
	<20070513155325.O68470@verden.pvv.ntnu.no> 
Message-ID: <59757.1179071250@critter.freebsd.dk>

In message <20070513155325.O68470 at verden.pvv.ntnu.no>, =?ISO-8859-1?Q?Andreas_R
=F8sdal?= writes:
>On Sun, 13 May 2007, Poul-Henning Kamp wrote:
>> In message <20070513144559.D67788 at verden.pvv.ntnu.no>, =?ISO-8859-1?Q?Andreas_R
>> =F8sdal?= writes:
>>> Hello,
>>> I have a question about a recent change done in SVN trunk:
>>> http://varnish.projects.linpro.no/changeset/1417
>>>
>>> "Make the sendfile threshold inifinity for now, we have evidence of
>>> sendfile not doing it's job in a number of operating system (-versions ?)"
>>>
>>> What does "not doing it's job" mean here? Sending corrupt or incomplete
>>> data?
>>
>> One, the other or even Both, depending on your kernel.
>
>Yes, I've seen both. What is the best workaround for this problem?
>Should I install the varnish from SVN trunk?

Just set the parameter "sendfile_threshold" to -1



-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Mon May 14 08:53:06 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Mon, 14 May 2007 10:53:06 +0200
Subject: sendfile changes
In-Reply-To: <20070513155325.O68470@verden.pvv.ntnu.no> ("Andreas
	=?utf-8?Q?R=C3=B8sdal=22's?=
	message of "Sun\, 13 May 2007 15\:55\:29 +0200 \(CEST\)")
References: <59179.1179060812@critter.freebsd.dk>
	<20070513155325.O68470@verden.pvv.ntnu.no>
Message-ID: <87ps53u6q5.fsf@des.linpro.no>

Andreas R?sdal <andrearo at pvv.ntnu.no> writes:
> Poul-Henning Kamp <phk at phk.freebsd.dk> writes:
> > Andreas R?sdal <andrearo at pvv.ntnu.no> writes:
> > > What does "not doing it's job" mean here? Sending corrupt or
> > > incomplete data?
> > One, the other or even Both, depending on your kernel.
> Yes, I've seen both. What is the best workaround for this problem?
> Should I install the varnish from SVN trunk?

telnet to the management port and type

  param.set sendfile_threshold -1

and add "-p sendfile_threshold=-1" to your startup flags
(VARNISHD_PARAMS in /etc/sysconfig/varnish)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Mon May 14 14:50:58 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Mon, 14 May 2007 16:50:58 +0200
Subject: Binding to "any"
Message-ID: <871whjtq5p.fsf@des.linpro.no>

Currently, there is a known issue with varnishd on FreeBSD, where if you
run it with "-a :80" it will only bind to IN6ADDR_ANY instead of both
INADDR_ANY and IN6ADDR_ANY.

This is not easy to solve.  The obvious fix, which is to bind to *all*
addresses returned by getaddrinfo() instead of just the first one, will
work on FreeBSD, but not on Linux, where you can't bind to the same port
on both INADDR_ANY and IN6ADDR_ANY.

On Linux, the current behaviour is good enough - binding to either
INADDR_ANY or IN6ADDR_ANY will give you both.

On FreeBSD, if we want both *and* the kernel has IPv6 support compiled
in, we can bind to IN6ADDR_ANY and clear the IPV6_V6ONLY socket option
(it defaults to on).  If the kernel does *not* have IPv6 support
compiled in, getaddrinfo() will still return both INADDR_ANY and
IN6ADDR_ANY, but we binding to IN6ADDR_ANY will fail.

I don't see any good way out of this mess - we're pretty much forced to
special-case the "any" case.

I've looked at other network servers, such as Apache and OpenSSH, and
they pretty much universally solve (or rather avoid) this issue by
explicitly specifying either 0.0.0.0 (INADDR_ANY) or :: (IN6ADDR_ANY) as
the default listening address.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From darix at web.de  Mon May 14 15:50:27 2007
From: darix at web.de (Marcus Rueckert)
Date: Mon, 14 May 2007 17:50:27 +0200
Subject: Binding to "any"
In-Reply-To: <871whjtq5p.fsf@des.linpro.no>
References: <871whjtq5p.fsf@des.linpro.no>
Message-ID: <20070514155027.GT3490@pixel.global-banlist.de>

On 2007-05-14 16:50:58 +0200, Dag-Erling Sm?rgrav wrote:
> Currently, there is a known issue with varnishd on FreeBSD, where if you
> run it with "-a :80" it will only bind to IN6ADDR_ANY instead of both
> INADDR_ANY and IN6ADDR_ANY.
> 
> This is not easy to solve.  The obvious fix, which is to bind to *all*
> addresses returned by getaddrinfo() instead of just the first one, will
> work on FreeBSD, but not on Linux, where you can't bind to the same port
> on both INADDR_ANY and IN6ADDR_ANY.
> 
> On Linux, the current behaviour is good enough - binding to either
> INADDR_ANY or IN6ADDR_ANY will give you both.

bind the ipv6 binds with IPV6_ONLY. than you get the same behavior as
on *BSD. postfix does that e.g. you just need to detect the socket type.

> On FreeBSD, if we want both *and* the kernel has IPv6 support compiled
> in, we can bind to IN6ADDR_ANY and clear the IPV6_V6ONLY socket option
> (it defaults to on).  If the kernel does *not* have IPv6 support
> compiled in, getaddrinfo() will still return both INADDR_ANY and
> IN6ADDR_ANY, but we binding to IN6ADDR_ANY will fail.
> 
> I don't see any good way out of this mess - we're pretty much forced to
> special-case the "any" case.
> 
> I've looked at other network servers, such as Apache and OpenSSH, and
> they pretty much universally solve (or rather avoid) this issue by
> explicitly specifying either 0.0.0.0 (INADDR_ANY) or :: (IN6ADDR_ANY) as
> the default listening address.

think so too.

    darix

-- 
           openSUSE - SUSE Linux is my linux
               openSUSE is good for you
                   www.opensuse.org


From phk at phk.freebsd.dk  Mon May 14 23:27:48 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 14 May 2007 23:27:48 +0000
Subject: Binding to "any" 
In-Reply-To: Your message of "Mon, 14 May 2007 16:50:58 +0200."
	<871whjtq5p.fsf@des.linpro.no> 
Message-ID: <2134.1179185268@critter.freebsd.dk>

In message <871whjtq5p.fsf at des.linpro.no>, Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=
 writes:

>Currently, there is a known issue with varnishd on FreeBSD, where if you
>run it with "-a :80" it will only bind to IN6ADDR_ANY instead of both
>INADDR_ANY and IN6ADDR_ANY.

I think the failure condition in the wildcard case should be "couldn't
bind to any address", wouldn't that solve it ?

It could be generalized to "for each -a argument, fail only if no
binding was achieved at all".

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Tue May 15 08:50:02 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Tue, 15 May 2007 10:50:02 +0200
Subject: Binding to "any"
In-Reply-To: <20070514155027.GT3490@pixel.global-banlist.de> (Marcus
	Rueckert's message of "Mon\, 14 May 2007 17\:50\:27 +0200")
References: <871whjtq5p.fsf@des.linpro.no>
	<20070514155027.GT3490@pixel.global-banlist.de>
Message-ID: <87wszasc79.fsf@des.linpro.no>

Marcus Rueckert <darix at web.de> writes:
> On 2007-05-14 16:50:58 +0200, Dag-Erling Sm?rgrav wrote:
> > On Linux, the current behaviour is good enough - binding to either
> > INADDR_ANY or IN6ADDR_ANY will give you both.
> bind the ipv6 binds with IPV6_ONLY. than you get the same behavior as
> on *BSD. postfix does that e.g. you just need to detect the socket type.

OK, so the only practical difference is that IPV6_V6ONLY is the default
on Linux?  Don't know why I never thought of that.

This still means we need to modify the listening socket setup code to
handle multiple sockets resulting from a single specification, for
instance ":80" -> { 0.0.0.0:80, [::]:80 }.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From darix at web.de  Tue May 15 09:33:45 2007
From: darix at web.de (Marcus Rueckert)
Date: Tue, 15 May 2007 11:33:45 +0200
Subject: Binding to "any"
In-Reply-To: <87wszasc79.fsf@des.linpro.no>
References: <871whjtq5p.fsf@des.linpro.no>
	<20070514155027.GT3490@pixel.global-banlist.de>
	<87wszasc79.fsf@des.linpro.no>
Message-ID: <20070515093345.GV3490@pixel.global-banlist.de>

On 2007-05-15 10:50:02 +0200, Dag-Erling Sm?rgrav wrote:
> Date: Tue, 15 May 2007 10:50:02 +0200
> From: Dag-Erling Sm?rgrav <des at linpro.no>
> Subject: Re: Binding to "any"
> To: Marcus Rueckert <darix at web.de>
> Cc: varnish-dev at projects.linpro.no
> X-DSPAM-Result: Innocent
> X-DSPAM-Confidence: 0.9933
> X-DSPAM-Probability: 0.0000
> 
> Marcus Rueckert <darix at web.de> writes:
> > On 2007-05-14 16:50:58 +0200, Dag-Erling Sm?rgrav wrote:
> > > On Linux, the current behaviour is good enough - binding to either
> > > INADDR_ANY or IN6ADDR_ANY will give you both.
> > bind the ipv6 binds with IPV6_ONLY. than you get the same behavior as
> > on *BSD. postfix does that e.g. you just need to detect the socket type.
> 
> OK, so the only practical difference is that IPV6_V6ONLY is the default
> on Linux?  Don't know why I never thought of that.

no. ipv6_only is default on *bsd. linux defaults to ipv6_only off.

> This still means we need to modify the listening socket setup code to
> handle multiple sockets resulting from a single specification, for
> instance ":80" -> { 0.0.0.0:80, [::]:80 }.

yes.

    darix

-- 
           openSUSE - SUSE Linux is my linux
               openSUSE is good for you
                   www.opensuse.org


From des at linpro.no  Tue May 15 09:35:51 2007
From: des at linpro.no (Dag-Erling =?utf-8?Q?Sm=C3=B8rgrav?=)
Date: Tue, 15 May 2007 11:35:51 +0200
Subject: Binding to "any"
In-Reply-To: <20070515093345.GV3490@pixel.global-banlist.de> (Marcus
	Rueckert's message of "Tue\, 15 May 2007 11\:33\:45 +0200")
References: <871whjtq5p.fsf@des.linpro.no>
	<20070514155027.GT3490@pixel.global-banlist.de>
	<87wszasc79.fsf@des.linpro.no>
	<20070515093345.GV3490@pixel.global-banlist.de>
Message-ID: <87lkfqsa2w.fsf@des.linpro.no>

Marcus Rueckert <darix at web.de> writes:
> On 2007-05-15 10:50:02 +0200, Dag-Erling Sm?rgrav wrote:
> > OK, so the only practical difference is that IPV6_V6ONLY is the default
> > on Linux?  Don't know why I never thought of that.
> no. ipv6_only is default on *bsd. linux defaults to ipv6_only off.

That's what I meant, even if it isn't what I wrote :)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From andrearo at pvv.ntnu.no  Fri May 18 08:46:50 2007
From: andrearo at pvv.ntnu.no (=?ISO-8859-1?Q?Andreas_R=F8sdal?=)
Date: Fri, 18 May 2007 10:46:50 +0200 (CEST)
Subject: Varnish ICP? 
In-Reply-To: <5150.1178895535@critter.freebsd.dk>
References: <5150.1178895535@critter.freebsd.dk>
Message-ID: <20070518101851.F47501@verden.pvv.ntnu.no>

On Fri, 11 May 2007, Poul-Henning Kamp wrote:
> In message <20070511165155.D37842 at verden.pvv.ntnu.no>, =?ISO-8859-1?Q?Andreas_R
> =F8sdal?= writes:
>
>> Hello!  Is ICPv2 included in the plans for varnish 2.0?
>
> No.
>
> Our thinking on ICP in general is that a two-layer setup is a better
> use of HW-resources.

It would be interesting to know a little more about how to configure 
Varnish in a two-layer setup. I guess this would be of interest to others 
deploying varnish as well.

Would such a two-layer setup be comparable to a squid cache hierarchy in 
surrogate mode?

For instance, if you had four servers with equal hardware resources to 
use as Varnish caches, how should these be optimally configured?

- A two-layer setup would mean a hierarchy with two siblings in 
two layers.
- An one layer setup would mean four siblings in one layer.

If I understand this correctly, the two-layer setup would have a lower 
mean time before failure, but a higher cache hit ratio, than the one layer 
setup.



  - Andreas



From phk at phk.freebsd.dk  Wed May 23 14:52:15 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 23 May 2007 14:52:15 +0000
Subject: Varnish ICP? 
In-Reply-To: Your message of "Fri, 18 May 2007 10:46:50 +0200."
	<20070518101851.F47501@verden.pvv.ntnu.no> 
Message-ID: <5475.1179931935@critter.freebsd.dk>

In message <20070518101851.F47501 at verden.pvv.ntnu.no>, =?ISO-8859-1?Q?Andreas_R
=F8sdal?= writes:

>> Our thinking on ICP in general is that a two-layer setup is a better
>> use of HW-resources.
>
>It would be interesting to know a little more about how to configure 
>Varnish in a two-layer setup. I guess this would be of interest to others 
>deploying varnish as well.

Sorry about the delay, I've been travelling for most of the last week.

Right now, I don't know of anybody who has created a two-layer
Varnish setup, so most of what we have on the table is conjecture.

The overall argument against a peer protocol is that peer servers
are unlikely to have to document we are looking for, simply because
this machine doesn't have it.  We see the same probabilities for
the documents on all servers.

The reason why this argument comes out the other way for squid, is
that squid's peer facility were built for the case where the real
object were behind a slow and expensive WAN line, whereas asking
the peer is just a high-speed trip across the LAN.  In the time it
took to establish a TCP connection across the WAN we could ask all
our peers and even if the probability of a hit was very low, a hit
saved us precious WAN time.

In our case the only benefit is a minute reduction in backend load
and a longer delay for all transactions that eventually goes to the
backend after the peer check.

Hold this up against the implementation of a pretty nasty protocol
with a lot of state and a geometric increase in activity on all
involved Varnish machines, and the benefit is hard to spot.

Compare this to setting up a two-level Varnish:

	frontends --- concentrator --- backend

If any of the frontends have document X, it is also on the
concentrator, so instead of asking N frontends to find it,
we are guaranteed to get it by asking the concentrator.

And, more importantly, if the concentrator does not have it,
it will pick it up from the backend for us.

In other words, we need no extra code, there are no "blind alleys"
for the request and overall performance will better.

Obviously, this all pressumes that the concentrator has the hardware
it needs, but if you are in the bandwidth class where you have this
problem, a 64GB RAM machine is probably not going to shock you.

But as I said, this is all conjecture and handwaving, but I'm pretty
sure it holds up.

Obviously, if your dataset is enormous, things may not look
this way and other ways to skin this rhubarb may be necessary.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Thu May 24 15:14:38 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 24 May 2007 17:14:38 +0200
Subject: cnt_deliver()
Message-ID: <87r6p6tfs1.fsf@des.linpro.no>

from cnt_deliver():

	if (sp->obj->objhead != NULL && sp->obj->pass) {
		/* we will no longer need the storage */
		HSH_Freestore(sp->obj);
	}
	HSH_Deref(sp->obj);
	sp->obj = NULL;

isn't this redundant?  In the PASS case, there are no other references
to the object, and HSH_Deref() will call HSH_Freestore().

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu May 24 15:27:45 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 24 May 2007 17:27:45 +0200
Subject: Request for review: synthetic objects
Message-ID: <87ps4qtf66.fsf@des.linpro.no>

The attached patch adds the beginning of an API for synthetic objects.

Brief overview:

 - moved http_msg array from cache_response.c to cache_http.c,
   introduced http_StatusMessage() lookup function

 - introduced http_Put{Protocol,Status,Response} to complement
   http_PrintfHeader().

 - introduced SYN_ErrorPage() in a new file, cache_synthetic.c.
   SYN_ErrorPage() populates the session's current object with the
   specified error code and a corresponding HTML error page; it is the
   caller's responsibility to ensure that the session has a suitable
   object (i.e. one that doesn't already have headers or a body)

 - rewrote RES_Error() to simply call SYN_ErrorPage() (with ttl = 0) and
   RES_WriteObj().

 - rewrote cnt_fetch() to use SYN_ErrorPage() to create a 503 page with
   a TTL of 30 seconds when Fetch() fails.

 - removed the call to RES_Error() in cache_backend.c; the error
   trickles back up to cnt_fetch() anyway.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

-------------- next part --------------
A non-text attachment was scrubbed...
Name: varnish-synthetic.diff
Type: text/x-diff
Size: 16320 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20070524/4fa90eaa/attachment.bin>

From des at linpro.no  Fri May 25 10:03:54 2007
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Fri, 25 May 2007 12:03:54 +0200
Subject: Request for review: synthetic objects
In-Reply-To: <87ps4qtf66.fsf@des.linpro.no> ("Dag-Erling =?utf-8?Q?Sm?=
	=?utf-8?Q?=C3=B8rgrav=22's?= message
	of "Thu\, 24 May 2007 17\:27\:45 +0200")
References: <87ps4qtf66.fsf@des.linpro.no>
Message-ID: <87ps4pqkxh.fsf@des.linpro.no>

Dag-Erling Sm?rgrav <des at linpro.no> writes:
> The attached patch adds the beginning of an API for synthetic objects.

Poul-Henning replied off-list, the patch has been committed.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no