From marco.molinari at gmail.com  Mon Feb  4 19:05:48 2008
From: marco.molinari at gmail.com (Marco Molinari)
Date: Mon, 4 Feb 2008 20:05:48 +0100
Subject: A few users see a blank page
Message-ID: <e950eb1e0802041105r235a012am800b97525c6d9255@mail.gmail.com>

Hi,
I recently deployed Varnish and its performance is simply incredible.
I'm having a problem, though, which made me jump back to Squid. In the
week I used Varnish I had some users mailing me that they couldn't see
the site anymore, they just saw a blank page.
It seems this affects a small percentage of the users, as I got just
about 10 emails in a week and my users are in the 10000s, and in fact
I didn't notice any problem in the stats; so I setup two servers, one
with Squid and the other with Varnish, and the affected users see the
site only through Squid.

I'm trying to understand what's happening and with the help of some
users here's what I discovered:
- every user who has problems connects from an office (completely
different places, of course)
- at least one user connects via a proxy (she has a header that says
"Via: 1.1 SRVFW1")
- Apache serves the page, which is logged with the correct size
- they can see very small pages (a test I did just printed "Thanks " +
the timestamp)

This happens both with 1.1.2 and trunk.

On standard log files I can't find anything, I couldn't even know it
was happening if those users didn't email me. I'd like to have more
data before submitting a bug report; do you have any suggestion on how
and what I could collect?

Thanks
Marco Molinari


From ak at openplans.org  Wed Feb  6 01:57:27 2008
From: ak at openplans.org (Arne Kepp)
Date: Tue, 05 Feb 2008 20:57:27 -0500
Subject: A few users see a blank page
In-Reply-To: <e950eb1e0802041105r235a012am800b97525c6d9255@mail.gmail.com>
References: <e950eb1e0802041105r235a012am800b97525c6d9255@mail.gmail.com>
Message-ID: <47A91407.5000703@openplans.org>

(Maybe this should be on -misc ?)

I've seen one odd (to me) problem like this that I was able to reproduce 
consistently, thanks to the help of a user. If you are serving PHP, and 
PHP is gzip'ing the output then clients behind a Squid proxy (on their 
end) only get blank pages.

Squid records something like "TCP_MISS/200 416 GET <script name>? - 
DIRECT/<ip address> text/html" in the logs, but doesn't appear to get 
the page.

Moving gzip compression from PHP to Apache solved this particular 
problem, but I still have users reporting blank pages every now and 
then. I've been accused of blocking all sorts of clients for this reason.

My VCL file is simple, it's the standard logic + lookup for image and 
css extensions and pass for everything else (also tried pipe).

-Arne

Marco Molinari wrote:
> Hi,
> I recently deployed Varnish and its performance is simply incredible.
> I'm having a problem, though, which made me jump back to Squid. In the
> week I used Varnish I had some users mailing me that they couldn't see
> the site anymore, they just saw a blank page.
> It seems this affects a small percentage of the users, as I got just
> about 10 emails in a week and my users are in the 10000s, and in fact
> I didn't notice any problem in the stats; so I setup two servers, one
> with Squid and the other with Varnish, and the affected users see the
> site only through Squid.
>
> I'm trying to understand what's happening and with the help of some
> users here's what I discovered:
> - every user who has problems connects from an office (completely
> different places, of course)
> - at least one user connects via a proxy (she has a header that says
> "Via: 1.1 SRVFW1")
> - Apache serves the page, which is logged with the correct size
> - they can see very small pages (a test I did just printed "Thanks " +
> the timestamp)
>
> This happens both with 1.1.2 and trunk.
>
> On standard log files I can't find anything, I couldn't even know it
> was happening if those users didn't email me. I'd like to have more
> data before submitting a bug report; do you have any suggestion on how
> and what I could collect?
>
> Thanks
> Marco Molinari
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-dev
>
> !DSPAM:4038,47a7621398828992556831!
>
>   



From des at linpro.no  Thu Feb 14 10:00:31 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 14 Feb 2008 11:00:31 +0100
Subject: how to build the nagios-plugin?
In-Reply-To: <BB714A64-E194-46D0-A3EF-179F303BBB58@lovelysystems.com> (Jodok
	Batlogg's message of "Sat, 26 Jan 2008 23:39:04 +0100")
References: <BB714A64-E194-46D0-A3EF-179F303BBB58@lovelysystems.com>
Message-ID: <ujr4pcb6eo0.fsf@login.linpro.no>

Jodok Batlogg <jodok at lovelysystems.com> writes:
> i'm trying to build the nagios-plugin for varnish but i fail already
> at the ./configure step...
>
> checking for VARNISHAPI... configure: error: Package requirements
> (varnishapi) were not met:
>
> No package 'varnishapi' found

Sorry for the late response.

If you install a sufficiantly new version of Varnish, it will install
the necessary pkg-config file.

If you installed Varnish from .rpm or .deb, you need to install the
devel package as well (IIRC, varnish-libs-devel for .rpm and
libvarnish0-dev for .deb)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Feb 14 10:01:51 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 14 Feb 2008 11:01:51 +0100
Subject: race condition in ticket 144 - 1.1 backport fix? [patch]
In-Reply-To: <4785839E.1020503@papasian.org> (Daniel Papasian's message of
	"Wed, 09 Jan 2008 21:31:58 -0500")
References: <4785839E.1020503@papasian.org>
Message-ID: <ujrzlu3501c.fsf@login.linpro.no>

Daniel Papasian <daniel at papasian.org> writes:
> I see that in 1.2 the backend code is significantly different and more
> flexible, and I suspect the bug does not exist there.  Are there plans
> to backport this backend code to 1.1?

Sorry, no...

> If not, I have a patch that I believe fixes the issue on the 1.1
> branch by acquiring a lock before the addr structure is used.

I suspect that would severely impact performance, but I'll have a look
at it.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Feb 14 10:04:59 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 14 Feb 2008 11:04:59 +0100
Subject: development efforts on the Solaris side.
In-Reply-To: <79838813-4375-4B63-ACB7-3538293BE8E6@omniti.com> (Theo
	Schlossnagle's message of "Wed, 9 Jan 2008 11:42:36 -0500")
References: <82666.1199871612@critter.freebsd.dk>
	<79838813-4375-4B63-ACB7-3538293BE8E6@omniti.com>
Message-ID: <ujrve4r4zw4.fsf@login.linpro.no>

Theo Schlossnagle <theo at omniti.com> writes:
> Solaris guarantees that the address ranges (and file data)
> references as the source for data in calls to both sendfile() and
> sendfilev() will not be touched after control returns to the caller.

Thanks, that means we can enable sendfile() on Solaris.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Thu Feb 14 10:48:29 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 14 Feb 2008 11:48:29 +0100
Subject: development efforts on the Solaris side.
In-Reply-To: <03B017F1-86E3-45AD-9387-10CD3B80AC32@omniti.com> (Theo
	Schlossnagle's message of "Wed, 9 Jan 2008 09:27:13 -0500")
References: <82666.1199871612@critter.freebsd.dk>
	<03B017F1-86E3-45AD-9387-10CD3B80AC32@omniti.com>
Message-ID: <ujrr6ff4xvm.fsf@login.linpro.no>

Theo Schlossnagle <theo at omniti.com> writes:
> [re: stv = stevedores = stevedores->next;]
>
> [...]  Was worried that the assumption that both assignments were
> atomic.  One is, then the next is.  Perhaps that doesn't matter.  It
> does seem like a hard to trigger race condition to me because while
> the assignment is atomic, the access to stevedores->next is not
> guaranteed to be view consistent in that assignment:
>
> T1: get stevedores->next in R(T1,a)
> T2: get stevedores->next in R(T2,a)
> T1: set stevedores to R(T1,a)
> T2: set stevedores to T(T2,a)
>
> Now T1 and T2 both "advanced the pointer" but they did the same work
> and the stv they have is the same.  Is that not a problem?  Perhaps
> I don't understand the impact of that scenario correctly.

There is a race, but it's irrelevant.  The point of this code is to
spread the load between the stevedores.  It doesn't matter if two
simultaneours allocations go to the same stevedore as long as *all*
allocations don't go to the same stevedore.  It will balance out in
the long run.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From theo at omniti.com  Thu Feb 14 12:32:19 2008
From: theo at omniti.com (Theo Schlossnagle)
Date: Thu, 14 Feb 2008 07:32:19 -0500
Subject: development efforts on the Solaris side.
In-Reply-To: <ujrr6ff4xvm.fsf@login.linpro.no>
References: <82666.1199871612@critter.freebsd.dk>
	<03B017F1-86E3-45AD-9387-10CD3B80AC32@omniti.com>
	<ujrr6ff4xvm.fsf@login.linpro.no>
Message-ID: <9AD339B4-8F8A-4263-9810-369BB108F6B5@omniti.com>


On Feb 14, 2008, at 5:48 AM, Dag-Erling Sm?rgrav wrote:

> Theo Schlossnagle <theo at omniti.com> writes:
>> [re: stv = stevedores = stevedores->next;]
>>
>> [...]  Was worried that the assumption that both assignments were
>> atomic.  One is, then the next is.  Perhaps that doesn't matter.  It
>> does seem like a hard to trigger race condition to me because while
>> the assignment is atomic, the access to stevedores->next is not
>> guaranteed to be view consistent in that assignment:
>>
>> T1: get stevedores->next in R(T1,a)
>> T2: get stevedores->next in R(T2,a)
>> T1: set stevedores to R(T1,a)
>> T2: set stevedores to T(T2,a)
>>
>> Now T1 and T2 both "advanced the pointer" but they did the same work
>> and the stv they have is the same.  Is that not a problem?  Perhaps
>> I don't understand the impact of that scenario correctly.
>
> There is a race, but it's irrelevant.  The point of this code is to
> spread the load between the stevedores.  It doesn't matter if two
> simultaneours allocations go to the same stevedore as long as *all*
> allocations don't go to the same stevedore.  It will balance out in
> the long run.

The comment made me think that the code was relying on different  
threads getting different stevedores for safety reasons -- not  
efficiency.  Thanks for clearing that up.

--
Theo Schlossnagle
Esoteric Curio -- http://lethargy.org/
OmniTI Computer Consulting, Inc. -- http://omniti.com/



From dave at cheney.net  Mon Feb 18 02:34:47 2008
From: dave at cheney.net (Dave Cheney)
Date: Mon, 18 Feb 2008 13:34:47 +1100
Subject: Backend syntax changed
Message-ID: <5A36E0C0-F398-4D00-9026-12E4D8AC31EC@cheney.net>

Hi,

I compiled r2505 from svn but the backend syntax has changed since  
this page

http://varnish.projects.linpro.no/wiki/Backends

was written. Can anyone please supply with the new round_robbin syntax  
for the following old style config

backend_round_robin default {
	set backend.set = {
		{ "172.16.0.72", "http" }
		{ "172.16.0.73", "http" }
	};
}

Many thanks

Dave Cheney



From phk at phk.freebsd.dk  Mon Feb 18 09:04:11 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 18 Feb 2008 09:04:11 +0000
Subject: Backend syntax changed 
In-Reply-To: Your message of "Mon, 18 Feb 2008 13:34:47 +1100."
	<5A36E0C0-F398-4D00-9026-12E4D8AC31EC@cheney.net> 
Message-ID: <27102.1203325451@critter.freebsd.dk>

In message <5A36E0C0-F398-4D00-9026-12E4D8AC31EC at cheney.net>, Dave Cheney write
s:
>Hi,
>
>I compiled r2505 from svn but the backend syntax has changed since  
>this page
>
>http://varnish.projects.linpro.no/wiki/Backends
>
>was written. Can anyone please supply with the new round_robbin syntax  
>for the following old style config
>
>backend_round_robin default {
>	set backend.set = {
>		{ "172.16.0.72", "http" }
>		{ "172.16.0.73", "http" }
>	};
>}

Hi Dave,

I have not converted the round_robin code yet, but you should be get
roughly the same effect with a 2 part random:

	director default random {
		{
			.backend = { .host = "172.16.0.72"; }
			.weight = 1;
		}
		{
			.backend = { .host = "172.16.0.73"; }
			.weight = 1;
		}
	}


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From dave at cheney.net  Mon Feb 18 12:09:30 2008
From: dave at cheney.net (Dave Cheney)
Date: Mon, 18 Feb 2008 23:09:30 +1100
Subject: Backend syntax changed 
In-Reply-To: <27102.1203325451@critter.freebsd.dk>
References: <27102.1203325451@critter.freebsd.dk>
Message-ID: <63AF1321-FF05-41A5-8CCC-B8DAAD70DC0C@cheney.net>

Thank you for your kind assistance Poul-Henning.

Cheers

Dave

On 18/02/2008, at 8:04 PM, Poul-Henning Kamp wrote:

> In message <5A36E0C0-F398-4D00-9026-12E4D8AC31EC at cheney.net>, Dave  
> Cheney write
> s:
>> Hi,
>>
>> I compiled r2505 from svn but the backend syntax has changed since
>> this page
>>
>> http://varnish.projects.linpro.no/wiki/Backends
>>
>> was written. Can anyone please supply with the new round_robbin  
>> syntax
>> for the following old style config
>>
>> backend_round_robin default {
>> 	set backend.set = {
>> 		{ "172.16.0.72", "http" }
>> 		{ "172.16.0.73", "http" }
>> 	};
>> }
>
> Hi Dave,
>
> I have not converted the round_robin code yet, but you should be get
> roughly the same effect with a 2 part random:
>
> 	director default random {
> 		{
> 			.backend = { .host = "172.16.0.72"; }
> 			.weight = 1;
> 		}
> 		{
> 			.backend = { .host = "172.16.0.73"; }
> 			.weight = 1;
> 		}
> 	}
>
>
> -- 
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by  
> incompetence.



From ay at vg.no  Tue Feb 19 13:59:52 2008
From: ay at vg.no (Audun Ytterdal)
Date: Tue, 19 Feb 2008 14:59:52 +0100
Subject: Varnish only using some of reserved ram, IO-trashing
Message-ID: <47BAE0D8.8050504@vg.no>

I've got 3 varnish servers with 32GB of RAM

Two of them run CentOS4.5 and one runs CentOS5

The problem is with the CentOS5 box

Varnish sees all of the 32 GB of ram, but only uses 3-4GB of it, while 
trashing the server with high IO-wait and load.

Varnish running as follows
/usr/sbin/varnishd -a :80 -f /etc/varnish/xxxx.vcl -T 127.0.0.1:82 -t 
120 -w 2,1000,30 -u varnish -g varnish -p thread_pools 1 -p 
client_http11 on -p rush_exponent 10 -s 
file,/var/varnish/varnish_storage.bin,30G -P /var/run/varnish.pid

Anyone care to take a look?

Working server (CentOS4.5)
==========================
cpu(s):  3.0% us,  1.0% sy,  0.0% ni, 95.3% id,  0.0% wa,  0.1% hi, 0.7%si
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19139 varnish   16   0 32.1g  19g  18g S   18 61.5   1333:30 varnishd

uname -r; ulimit -a
2.6.9-55.0.2.ELsmp
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 268287
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

/usr/sbin/varnishd -a :80 -f /etc/varnish/nettby.vcl -T 127.0.0.1:82 -t 
120 -w 2,1000,30 -u varnish -g varnish -p thread_pools 1 -p 
client_http11 on -p rush_exponent 10 -s 
file,/var/varnish/varnish_storage.bin,30G -P /var/run/varnish.pid

# varnishstat -1
client_conn         162710315       332.27 Client connections accepted
client_req          415626452       848.75 Client requests received
cache_hit           410058375       837.38 Cache hits
cache_hitpass         3187366         6.51 Cache hits for pass
cache_miss            2377826         4.86 Cache misses
backend_conn          5565194        11.36 Backend connections success
backend_fail              558         0.00 Backend connections failures
backend_reuse               0         0.00 Backend connections reuses
backend_recycle             0         0.00 Backend connections recycles
backend_unused            991         0.00 Backend connections unused
n_srcaddr                5558          .   N struct srcaddr
n_srcaddr_act            1260          .   N active struct srcaddr
n_sess_mem               8097          .   N struct sess_mem
n_sess                   2968          .   N struct sess
n_object              1249250          .   N struct object
n_objecthead          1249804          .   N struct objecthead
n_smf                 2389608          .   N struct smf
n_smf_frag                 16          .   N small free smf
n_smf_large                 1          .   N large free smf
n_vbe_conn                  0          .   N struct vbe_conn
n_wrk                      65          .   N worker threads
n_wrk_create            57012         0.12 N worker threads created
n_wrk_failed                0         0.00 N worker threads not created
n_wrk_max               39549         0.08 N worker threads limited
n_wrk_queue                 0         0.00 N queued work requests
n_wrk_overflow          96561         0.20 N overflowed work requests
n_wrk_drop                  0         0.00 N dropped work requests
n_expired             1183055          .   N expired objects
n_lru_nuked                 0          .   N LRU nuked objects
n_lru_saved                 0          .   N LRU saved objects
n_deathrow                  0          .   N objects on deathrow
losthdr                   171         0.00 HTTP header overflows
n_objsendfile               0         0.00 Objects sent with sendfile
n_objwrite          280398080       572.60 Objects sent with write
s_sess              162710284       332.27 Total Sessions
s_req               415627912       848.75 Total Requests
s_pipe                      0         0.00 Total pipe
s_pass                      0         0.00 Total pass
s_fetch               5565187        11.36 Total fetch
s_hdrbytes       122863296770    250899.13 Total header bytes
s_bodybytes      578238928802   1180821.68 Total body bytes
sess_closed           3290353         6.72 Session Closed
sess_pipeline               0         0.00 Session Pipeline
sess_readahead        1571272         3.21 Session Read Ahead
sess_herd           413387877       844.18 Session herd
shm_records       17029260317     34775.45 SHM records
shm_writes          908226596      1854.69 SHM writes
shm_cont               504250         1.03 SHM MTX contention
sm_nreq               7997521        16.33 allocator requests
sm_nobj               2389591          .   outstanding allocations
sm_balloc         21814378496          .   bytes allocated
sm_bfree          10397876224          .   bytes free
backend_req           5565190        11.36 Backend requests made


Problem Server (CentOS5)
========================
Cpu(s):  2.3%us,  0.9%sy,  0.0%ni, 59.3%id, 36.6%wa,  0.3%hi,  0.7%si, 
0.0%st
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9353 varnish   15   0 33.0g 3.0g 2.8g S   15  9.6  16:47.07 varnishd

Notice 36% IO-wait. It's using the file instead of mapping memory

# uname -r; ulimit -a
2.6.18-53.1.13.el5
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 268287
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 327680
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 268287
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

# varnishstat -1
client_conn           4474131       327.99 Client connections accepted
client_req            9736045       713.73 Client requests received
cache_hit             9439400       691.99 Cache hits
cache_hitpass               0         0.00 Cache hits for pass
cache_miss             295479        21.66 Cache misses
backend_conn           295485        21.66 Backend connections success
backend_fail                0         0.00 Backend connections failures
backend_reuse               0         0.00 Backend connections reuses
backend_recycle             0         0.00 Backend connections recycles
backend_unused              0         0.00 Backend connections unused
n_srcaddr                5361          .   N struct srcaddr
n_srcaddr_act            1297          .   N active struct srcaddr
n_sess_mem               2909          .   N struct sess_mem
n_sess                   2356          .   N struct sess
n_object               292447          .   N struct object
n_objecthead           292486          .   N struct objecthead
n_smf                  579235          .   N struct smf
n_smf_frag                  1          .   N small free smf
n_smf_large                 1          .   N large free smf
n_vbe_conn                 21          .   N struct vbe_conn
n_bereq                   126          .   N struct bereq
n_wrk                     225          .   N worker threads
n_wrk_create             3298         0.24 N worker threads created
n_wrk_failed                0         0.00 N worker threads not created
n_wrk_max                   0         0.00 N worker threads limited
n_wrk_queue                 0         0.00 N queued work requests
n_wrk_overflow           3298         0.24 N overflowed work requests
n_wrk_drop                  0         0.00 N dropped work requests
n_backend                   1          .   N backends
n_expired                5976          .   N expired objects
n_lru_nuked                 0          .   N LRU nuked objects
n_lru_saved                 0          .   N LRU saved objects
n_lru_moved           7282571          .   N LRU moved objects
n_deathrow                  0          .   N objects on deathrow
losthdr                     0         0.00 HTTP header overflows
n_objsendfile               0         0.00 Objects sent with sendfile
n_objwrite            7526263       551.74 Objects sent with write
s_sess                4474103       327.99 Total Sessions
s_req                 9736204       713.75 Total Requests
s_pipe                      0         0.00 Total pipe
s_pass                      0         0.00 Total pass
s_fetch                295483        21.66 Total fetch
s_hdrbytes         3017925993    221239.35 Total header bytes
s_bodybytes       15705176320   1151321.48 Total body bytes
sess_closed            125264         9.18 Session Closed
sess_pipeline               0         0.00 Session Pipeline
sess_readahead          42404         3.11 Session Read Ahead
sess_herd             9622386       705.40 Session herd
shm_records         403217255     29559.22 SHM records
shm_writes           23235793      1703.38 SHM writes
shm_cont               114977         8.43 SHM MTX contention
sm_nreq                593953        43.54 allocator requests
sm_nobj                579233          .   outstanding allocations
sm_balloc          4619284480          .   bytes allocated
sm_bfree          27592970240          .   bytes free
backend_req            295485        21.66 Backend requests made


*****************************************************************
Denne fotnoten bekrefter at denne e-postmeldingen ble
skannet av MailSweeper og funnet fri for virus.
*****************************************************************
This footnote confirms that this email message has been 
swept by MailSweeper for the presence of computer viruses.
*****************************************************************



From dave at cheney.net  Sat Feb 23 05:01:31 2008
From: dave at cheney.net (Dave Cheney)
Date: Sat, 23 Feb 2008 16:01:31 +1100
Subject: Adjusting the parents watchdog timer
Message-ID: <4EBBA239-675D-4D99-BF3B-16A439953AF8@cheney.net>

Hi,

	I am experiencing child restarts every 8 to 24 hours due to the child  
failing to respond to the parents ping. Is it possible to extend the  
ping timeout from the parent to try to avoid this behavior? The cache  
in question sits behind a load balancer so if the child is not  
restarted the load balancer will shift to the other host.

Cheers

Dave


From anders at fupp.net  Mon Feb 25 10:46:30 2008
From: anders at fupp.net (Anders Nordby)
Date: Mon, 25 Feb 2008 11:46:30 +0100
Subject: Adjusting the parents watchdog timer
In-Reply-To: <4EBBA239-675D-4D99-BF3B-16A439953AF8@cheney.net>
References: <4EBBA239-675D-4D99-BF3B-16A439953AF8@cheney.net>
Message-ID: <20080225104629.GA31964@fupp.net>

Hi,

Yes. Just set ping_interval to a higher number than 3 (default number of
seconds), or to 0 if you want to disable it completely. If you start
Varnish with the -T option, you can telnet to the management interface
and see which variables you can poke.

-A.

On Sat, Feb 23, 2008 at 04:01:31PM +1100, Dave Cheney wrote:
> 	I am experiencing child restarts every 8 to 24 hours due to the child  
> failing to respond to the parents ping. Is it possible to extend the  
> ping timeout from the parent to try to avoid this behavior? The cache  
> in question sits behind a load balancer so if the child is not  
> restarted the load balancer will shift to the other host.
> 
> Cheers
> 
> Dave
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-dev


-- 
Anders.


From dave at cheney.net  Mon Feb 25 14:00:23 2008
From: dave at cheney.net (Dave Cheney)
Date: Tue, 26 Feb 2008 01:00:23 +1100
Subject: Adjusting the parents watchdog timer
In-Reply-To: <20080225104629.GA31964@fupp.net>
References: <4EBBA239-675D-4D99-BF3B-16A439953AF8@cheney.net>
	<20080225104629.GA31964@fupp.net>
Message-ID: <E0E00746-C323-43F0-B7A9-572CD95800F7@cheney.net>

Thanks for the tip Andres. I've pushed the ping timeout up to 15  
seconds to see what that will turn up (looking for evidence of the  
machines swapping like crazy or the loadavg shooting through the roof).

Cheers

Dave

On 25/02/2008, at 9:46 PM, Anders Nordby wrote:

> Hi,
>
> Yes. Just set ping_interval to a higher number than 3 (default  
> number of
> seconds), or to 0 if you want to disable it completely. If you start
> Varnish with the -T option, you can telnet to the management interface
> and see which variables you can poke.
>
> -A.
>
> On Sat, Feb 23, 2008 at 04:01:31PM +1100, Dave Cheney wrote:
>> 	I am experiencing child restarts every 8 to 24 hours due to the  
>> child
>> failing to respond to the parents ping. Is it possible to extend the
>> ping timeout from the parent to try to avoid this behavior? The cache
>> in question sits behind a load balancer so if the child is not
>> restarted the load balancer will shift to the other host.
>>
>> Cheers
>>
>> Dave
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at projects.linpro.no
>> http://projects.linpro.no/mailman/listinfo/varnish-dev
>
>
> -- 
> Anders.



From dave at cheney.net  Tue Feb 26 22:19:19 2008
From: dave at cheney.net (Dave Cheney)
Date: Wed, 27 Feb 2008 09:19:19 +1100
Subject: Adjusting the parents watchdog timer
In-Reply-To: <E0E00746-C323-43F0-B7A9-572CD95800F7@cheney.net>
References: <4EBBA239-675D-4D99-BF3B-16A439953AF8@cheney.net>
	<20080225104629.GA31964@fupp.net>
	<E0E00746-C323-43F0-B7A9-572CD95800F7@cheney.net>
Message-ID: <5F33B9BE-E875-41AE-940E-5CEFB0391C8B@cheney.net>

Adjusting the timeout to 15 seconds has keep a machine which was  
previously restarting the child up to 4 times a day stable for over 24  
hours.

Clearly the watchdog timeout exists for a reason and adjusting it this  
high is only a placebo. I'll try reducing the timeout slowly  to  
figure out exactly what is causing the child to timeout.

Cheers

Dave

On 26/02/2008, at 1:00 AM, Dave Cheney wrote:

> Thanks for the tip Andres. I've pushed the ping timeout up to 15
> seconds to see what that will turn up (looking for evidence of the
> machines swapping like crazy or the loadavg shooting through the  
> roof).
>
> Cheers
>
> Dave
>
> On 25/02/2008, at 9:46 PM, Anders Nordby wrote:
>
>> Hi,
>>
>> Yes. Just set ping_interval to a higher number than 3 (default
>> number of
>> seconds), or to 0 if you want to disable it completely. If you start
>> Varnish with the -T option, you can telnet to the management  
>> interface
>> and see which variables you can poke.
>>
>> -A.
>>
>> On Sat, Feb 23, 2008 at 04:01:31PM +1100, Dave Cheney wrote:
>>> 	I am experiencing child restarts every 8 to 24 hours due to the
>>> child
>>> failing to respond to the parents ping. Is it possible to extend the
>>> ping timeout from the parent to try to avoid this behavior? The  
>>> cache
>>> in question sits behind a load balancer so if the child is not
>>> restarted the load balancer will shift to the other host.
>>>
>>> Cheers
>>>
>>> Dave
>>> _______________________________________________
>>> varnish-dev mailing list
>>> varnish-dev at projects.linpro.no
>>> http://projects.linpro.no/mailman/listinfo/varnish-dev
>>
>>
>> -- 
>> Anders.
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-dev



From phk at phk.freebsd.dk  Tue Feb 26 22:33:32 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 26 Feb 2008 22:33:32 +0000
Subject: Adjusting the parents watchdog timer 
In-Reply-To: Your message of "Wed, 27 Feb 2008 09:19:19 +1100."
	<5F33B9BE-E875-41AE-940E-5CEFB0391C8B@cheney.net> 
Message-ID: <14061.1204065212@critter.freebsd.dk>

In message <5F33B9BE-E875-41AE-940E-5CEFB0391C8B at cheney.net>, Dave Cheney write
s:

>Adjusting the timeout to 15 seconds has keep a machine which was  
>previously restarting the child up to 4 times a day stable for over 24  
>hours.
>
>Clearly the watchdog timeout exists for a reason and adjusting it this  
>high is only a placebo. I'll try reducing the timeout slowly  to  
>figure out exactly what is causing the child to timeout.

The default value is very much a number pulled out of thin air, and
it is certainly not inconceiveable that it needs tweaking.

In general I would expect 5 seconds to be enough, and I would like to
find out why they are not.

It may help if you plot (using munin ?) the activity on the machine
and see if you can correlate the hickups with activitity.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From dave at cheney.net  Tue Feb 26 22:42:41 2008
From: dave at cheney.net (Dave Cheney)
Date: Wed, 27 Feb 2008 09:42:41 +1100
Subject: Adjusting the parents watchdog timer 
In-Reply-To: <14061.1204065212@critter.freebsd.dk>
References: <14061.1204065212@critter.freebsd.dk>
Message-ID: <C4552EF2-19E7-44E0-A458-7A6D39549C2B@cheney.net>

[root at rado ~]# uname -a
Linux rado.redbubble.com 2.6.23.14-64.fc7 #1 SMP Sun Jan 20 22:20:19  
EST 2008 x86_64 x86_64 x86_64 GNU/Linux

I'm using cacti to graph the usual system stats with a 5 minute  
resolution. There is some indication of a loadavg jump in concert with  
each child restart which may indicate thread pileup.

I haven't looked at the PING code but i'm guessing its a connect, a  
dummy request and a close, so thread starvation or mutex contention  
could cause a blockage.

I'll reduce the timeout to 5 seconds and see what happens.

Cheers

Dave

On 27/02/2008, at 9:33 AM, Poul-Henning Kamp wrote:

> The default value is very much a number pulled out of thin air, and
> it is certainly not inconceiveable that it needs tweaking.
>
> In general I would expect 5 seconds to be enough, and I would like to
> find out why they are not.
>
> It may help if you plot (using munin ?) the activity on the machine
> and see if you can correlate the hickups with activitity.



From phk at phk.freebsd.dk  Tue Feb 26 22:47:10 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 26 Feb 2008 22:47:10 +0000
Subject: Adjusting the parents watchdog timer 
In-Reply-To: Your message of "Wed, 27 Feb 2008 09:42:41 +1100."
	<C4552EF2-19E7-44E0-A458-7A6D39549C2B@cheney.net> 
Message-ID: <14152.1204066030@critter.freebsd.dk>

In message <C4552EF2-19E7-44E0-A458-7A6D39549C2B at cheney.net>, Dave Cheney write
s:

>I haven't looked at the PING code but i'm guessing its a connect, a  
>dummy request and a close, so thread starvation or mutex contention  
>could cause a blockage.

Not quite.

The master process has a pipe to the client for issuing commands
(purge, load vcl etc) and every $ping_interval it sends a "PING"
message that simply should elicit a PONG reply to show that the
child is responding.

You can also do this at any other varnish CLI:

	ping
	200 15      
	PONG 1204065967

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From aotto at mosso.com  Tue Feb 26 22:58:39 2008
From: aotto at mosso.com (Adrian Otto)
Date: Tue, 26 Feb 2008 14:58:39 -0800
Subject: Adjusting the parents watchdog timer 
In-Reply-To: <C4552EF2-19E7-44E0-A458-7A6D39549C2B@cheney.net>
References: <14061.1204065212@critter.freebsd.dk>
	<C4552EF2-19E7-44E0-A458-7A6D39549C2B@cheney.net>
Message-ID: <D68EB175-B1F3-4358-8B32-635277EAED9C@mosso.com>

Dave,

For what it's worth, I've also seen the condition you described. My  
solution was to simply disable the watchdog timer. I did not have the  
energy to debug it, because it only seemed to happen when I was  
pushing rather ridiculously high test loads. As an experiment, it  
might be interesting to just dedicate one additional thread to  
responding to something just like PING/PONG in addition to the PING/ 
PONG already there, and use a separate client to generate checks. For  
sake of discussion, call this PING2. You'll also want to adjust PING/ 
PONG so that the parent logs a failure instead of actually restarting  
the child. If the PING/PONG fails, but the PING2 continues to work,  
then it points to an internal issue within varnishd. However, if they  
both quit working, that would indicate external an condition as the  
more likely cause.

One way to catch it is to change the parent PING code to start gdb  
and attach it to your varnishd process when it fails to respond,  
rather than killing it. That way you can stop and look at exactly  
what's happening inside varnishd at the time of the condition. I'd  
first look at things like the thread count, and get a backtrace of  
all the threads to see if something is obviously stuck somewhere. If  
you don't see it right away, consider just running gcore inside gdb  
to generate a core file, and perhaps one of us could take a look.

Regards,

Adrian

On Feb 26, 2008, at 2:42 PM, Dave Cheney wrote:

> [root at rado ~]# uname -a
> Linux rado.redbubble.com 2.6.23.14-64.fc7 #1 SMP Sun Jan 20 22:20:19
> EST 2008 x86_64 x86_64 x86_64 GNU/Linux
>
> I'm using cacti to graph the usual system stats with a 5 minute
> resolution. There is some indication of a loadavg jump in concert with
> each child restart which may indicate thread pileup.
>
> I haven't looked at the PING code but i'm guessing its a connect, a
> dummy request and a close, so thread starvation or mutex contention
> could cause a blockage.
>
> I'll reduce the timeout to 5 seconds and see what happens.
>
> Cheers
>
> Dave
>
> On 27/02/2008, at 9:33 AM, Poul-Henning Kamp wrote:
>
>> The default value is very much a number pulled out of thin air, and
>> it is certainly not inconceiveable that it needs tweaking.
>>
>> In general I would expect 5 seconds to be enough, and I would like to
>> find out why they are not.
>>
>> It may help if you plot (using munin ?) the activity on the machine
>> and see if you can correlate the hickups with activitity.
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-dev



From phk at phk.freebsd.dk  Tue Feb 26 23:44:44 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 26 Feb 2008 23:44:44 +0000
Subject: Adjusting the parents watchdog timer 
In-Reply-To: Your message of "Tue, 26 Feb 2008 14:58:39 PST."
	<D68EB175-B1F3-4358-8B32-635277EAED9C@mosso.com> 
Message-ID: <14526.1204069484@critter.freebsd.dk>

In message <D68EB175-B1F3-4358-8B32-635277EAED9C at mosso.com>, Adrian Otto writes
:

>As an experiment, it  
>might be interesting to just dedicate one additional thread to  
>responding to something just like PING/PONG in addition to the PING/ 
>PONG already there, and use a separate client to generate checks.

The cache-process already dedicates a thread to handling the CLI
pipe, so that wouldn't prove anything.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From dave at cheney.net  Wed Feb 27 03:53:26 2008
From: dave at cheney.net (Dave Cheney)
Date: Wed, 27 Feb 2008 14:53:26 +1100
Subject: Adjusting the parents watchdog timer 
In-Reply-To: <D68EB175-B1F3-4358-8B32-635277EAED9C@mosso.com>
References: <14061.1204065212@critter.freebsd.dk>
	<C4552EF2-19E7-44E0-A458-7A6D39549C2B@cheney.net>
	<D68EB175-B1F3-4358-8B32-635277EAED9C@mosso.com>
Message-ID: <5C21D2B2-735C-4328-9F62-C6CAD699C151@cheney.net>

The embarrassing thing is these machines are doing 40req/sec not the  
5000 that others reportedly handle. I've tried to strip out all the  
non essential services running on these machines, leaving nothing  
running by varnish, but there may be some periodic IO sapping cron job  
that I have missed and is causing >>that<< much of a periodic load  
that the watchdog timer fires.

I've reduced the watchdog to 5 seconds now, i'll see how that goes and  
if it can give any hints.

Cheers

Dave


On 27/02/2008, at 9:58 AM, Adrian Otto wrote:

> For what it's worth, I've also seen the condition you described. My
> solution was to simply disable the watchdog timer. I did not have the
> energy to debug it, because it only seemed to happen when I was
> pushing rather ridiculously high test loads.



From skanga at gmail.com  Tue Feb 19 22:23:52 2008
From: skanga at gmail.com (Shiraz Kanga)
Date: Tue, 19 Feb 2008 14:23:52 -0800
Subject: Load Balancing Algorithms in VCL
Message-ID: <47BB56F8.4080902@gmail.com>


Will it be possible to implement custom Load Balancing algorithms in 
VCL? There are numerous algorithms that can be used to distribute load like
Round Robin, Weighted Round Robin, Least Connections, Fastest Response, 
Most Bandwidth, etc.

Each of these algorithms can be made sticky i.e. only balance NEW 
requests and send existing ones to the same backend as they were 
previously sent to (to reuse caches, etc).

In order to support these VCL will need access to various statistics like
* Number of active requests per server
* Number of requests per second per server
* Avg response time per server
* Bandwidth per server

Are these stats available in the VCL environment?

Thanks

shiraz
skanga at gmail.com



From maxime at carpediem.fr  Fri Feb 22 21:55:25 2008
From: maxime at carpediem.fr (Maxime Cd)
Date: Fri, 22 Feb 2008 22:55:25 +0100
Subject: Varnish only using some of reserved ram, IO-trashing
In-Reply-To: <47BAE0D8.8050504@vg.no>
References: <47BAE0D8.8050504@vg.no>
Message-ID: <47BF44CD.6060903@carpediem.fr>

Audun Ytterdal wrote:
> I've got 3 varnish servers with 32GB of RAM
>
> Two of them run CentOS4.5 and one runs CentOS5
>
> The problem is with the CentOS5 box
>
> Varnish sees all of the 32 GB of ram, but only uses 3-4GB of it, while 
> trashing the server with high IO-wait and load.

I'm not a kernel expert, but it sounds like a 32 bits memory address 
limit. Your CentOS4.5 are very likely running in 64 bits mode, while the 
CentOS 5.0 is in 32 bit mode.



Maxime.



From chris at nmedia.net  Mon Feb 25 15:26:00 2008
From: chris at nmedia.net (Chris Cappuccio)
Date: Mon, 25 Feb 2008 07:26:00 -0800
Subject: varnish on OpenBSD
Message-ID: <20080225152600.GB3426@ref.nmedia.net>

On OpenBSD, the OS headers include sys/ucred.h through sys/mount.h.

For this to work, conftest needs to include sys/param.h before sys/mount.h
to get the definition for NGROUPS.  I'm not sure if NetBSD also requires this,
or not.  It seems like an odd dependency.

configure: WARNING: sys/mount.h: present but cannot be compiled
configure: WARNING: sys/mount.h:     check for missing prerequisite headers?
configure: WARNING: sys/mount.h: see the Autoconf documentation
configure: WARNING: sys/mount.h:     section "Present But Cannot Be Compiled"
configure: WARNING: sys/mount.h: proceeding with the preprocessor's result
configure: WARNING: sys/mount.h: in the future, the compiler will take precedence
configure: WARNING:     ## --------------------------------------------- ##
configure: WARNING:     ## Report this to varnish-dev at projects.linpro.no ##
configure: WARNING:     ## --------------------------------------------- ##

---

configure:20770: checking sys/mount.h usability
configure:20787: gcc -c -g -O2  conftest.c >&5
In file included from /usr/include/sys/mount.h:40,
                 from conftest.c:59:
/usr/include/sys/ucred.h:46: error: `NGROUPS' undeclared here (not in a function)
configure:20793: $? = 1
configure: failed program was:
| /* confdefs.h.  */
| #define PACKAGE_NAME "Varnish"
| #define PACKAGE_TARNAME "varnish"
| #define PACKAGE_VERSION "1.1.2"
| #define _GNU_SOURCE 1
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_DLFCN_H 1
| #define STDC_HEADERS 1
| #define HAVE_SYS_WAIT_H 1
| #define TIME_WITH_SYS_TIME 1
| #define HAVE_SYS_SOCKET_H 1
| /* end confdefs.h.  */
| #include <stdio.h>
| #ifdef HAVE_SYS_TYPES_H
| # include <sys/types.h>
| #endif
| #ifdef HAVE_SYS_STAT_H
| # include <sys/stat.h>
| #endif
| #ifdef STDC_HEADERS
| # include <stdlib.h>
| # include <stddef.h>
| #else
| # ifdef HAVE_STDLIB_H
| #  include <stdlib.h>
| # endif
| #endif
| #ifdef HAVE_STRING_H
| # if !defined STDC_HEADERS && defined HAVE_MEMORY_H
| #  include <memory.h>
| # endif
| # include <string.h>
| #endif
| #ifdef HAVE_STRINGS_H
| # include <strings.h>
| #endif
| #ifdef HAVE_INTTYPES_H
| # include <inttypes.h>
| #endif
| #ifdef HAVE_STDINT_H
| # include <stdint.h>
| #endif
| #ifdef HAVE_UNISTD_H
| # include <unistd.h>
| #endif
| #include <sys/mount.h>
configure:20807: result: no
configure:20811: checking sys/mount.h presence
configure:20826: gcc -E  conftest.c
configure:20832: $? = 0
configure:20846: result: yes
configure:20859: WARNING: sys/mount.h: present but cannot be compiled
configure:20861: WARNING: sys/mount.h:     check for missing prerequisite headers?
configure:20863: WARNING: sys/mount.h: see the Autoconf documentation
configure:20865: WARNING: sys/mount.h:     section "Present But Cannot Be Compiled"
configure:20867: WARNING: sys/mount.h: proceeding with the preprocessor's resultconfigure:20869: WARNING: sys/mount.h: in the future, the compiler will take precedence
configure:20879: checking for sys/mount.h
configure:20887: result: yes