From barry at automattic.com  Sun Jun  1 15:58:54 2008
From: barry at automattic.com (Barry Abrahamson)
Date: Sun, 1 Jun 2008 11:58:54 -0400
Subject: Multiple varnish instances per server?
Message-ID: <58FB0490-B012-4FEA-A885-C98A0D25D0D9@automattic.com>

Hi,

Is anyone running multiple varnish instances per server (one per disk  
or similar?)

We are currently running a single varnish instance per server using  
the file backend.  Machines are Dual Opteron 2218, 4GB RAM, and 2  
250GB SATA drives.  We have the cache file on a software RAID 0  
array.  Our cache size is set to 300GB, but once we get to 100GB or  
so, IO starts to get very spiky, causing loads to spike into the 100  
range.  Our expires are rather long (1-2 weeks).  My initial thoughts  
were that this was caused by cache file fragmentation, but we are  
seeing similar issues when using the malloc backend.  We were thinking  
that running 2 instances per server with smaller cache files (one per  
physical disk), may improve our IO problems.  Is there any performance  
benefit/detriment to running multiple varnish instances per server?   
Is there a performance hit for having a large cache?

Request rates aren't that high (50-150/sec), but the cached files are  
all images, some of which can be rather big (3MB).

Also, is anyone else seeing similar issues under similar workloads?
--
Barry Abrahamson | Systems Wrangler | Automattic
Blog: http://barry.wordpress.com








From phk at phk.freebsd.dk  Sun Jun  1 16:31:07 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 01 Jun 2008 16:31:07 +0000
Subject: Multiple varnish instances per server? 
In-Reply-To: Your message of "Sun, 01 Jun 2008 11:58:54 -0400."
	<58FB0490-B012-4FEA-A885-C98A0D25D0D9@automattic.com> 
Message-ID: <1286.1212337867@critter.freebsd.dk>

In message <58FB0490-B012-4FEA-A885-C98A0D25D0D9 at automattic.com>, Barry Abrahams
on writes:
>Hi,
>
>Is anyone running multiple varnish instances per server (one per disk  
>or similar?)

Just remember to set the -n argument to something different.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From michael at dynamine.net  Sun Jun  1 17:38:24 2008
From: michael at dynamine.net (Michael S. Fischer)
Date: Sun, 1 Jun 2008 10:38:24 -0700
Subject: Multiple varnish instances per server?
In-Reply-To: <58FB0490-B012-4FEA-A885-C98A0D25D0D9@automattic.com>
References: <58FB0490-B012-4FEA-A885-C98A0D25D0D9@automattic.com>
Message-ID: <86db848d0806011038r49776b7awefec1a381684dbf9@mail.gmail.com>

Why are you using Varnish to serve primarily images?  Modern webservers
serve static files very efficiently off the filesystem.
Best regards,

--Michael

On Sun, Jun 1, 2008 at 8:58 AM, Barry Abrahamson <barry at automattic.com>
wrote:

> Hi,
>
> Is anyone running multiple varnish instances per server (one per disk
> or similar?)
>
> We are currently running a single varnish instance per server using
> the file backend.  Machines are Dual Opteron 2218, 4GB RAM, and 2
> 250GB SATA drives.  We have the cache file on a software RAID 0
> array.  Our cache size is set to 300GB, but once we get to 100GB or
> so, IO starts to get very spiky, causing loads to spike into the 100
> range.  Our expires are rather long (1-2 weeks).  My initial thoughts
> were that this was caused by cache file fragmentation, but we are
> seeing similar issues when using the malloc backend.  We were thinking
> that running 2 instances per server with smaller cache files (one per
> physical disk), may improve our IO problems.  Is there any performance
> benefit/detriment to running multiple varnish instances per server?
> Is there a performance hit for having a large cache?
>
> Request rates aren't that high (50-150/sec), but the cached files are
> all images, some of which can be rather big (3MB).
>
> Also, is anyone else seeing similar issues under similar workloads?
> --
> Barry Abrahamson | Systems Wrangler | Automattic
> Blog: http://barry.wordpress.com
>
>
>
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080601/d5257854/attachment.html>

From barry at automattic.com  Sun Jun  1 17:47:53 2008
From: barry at automattic.com (Barry Abrahamson)
Date: Sun, 1 Jun 2008 13:47:53 -0400
Subject: Multiple varnish instances per server?
In-Reply-To: <86db848d0806011038r49776b7awefec1a381684dbf9@mail.gmail.com>
References: <58FB0490-B012-4FEA-A885-C98A0D25D0D9@automattic.com>
	<86db848d0806011038r49776b7awefec1a381684dbf9@mail.gmail.com>
Message-ID: <791351F3-C3A4-4DDF-9032-C34327D4D2C2@automattic.com>

\On Jun 1, 2008, at 1:38 PM, Michael S. Fischer wrote:

> Why are you using Varnish to serve primarily images?  Modern  
> webservers serve static files very efficiently off the filesystem.

Because we have about 6TB of content and are using Varnish as the  
"hot" cache and S3 as the "cold" store.

--
Barry Abrahamson | Systems Wrangler | Automattic
Blog: http://barry.wordpress.com








From phk at phk.freebsd.dk  Mon Jun  2 12:41:10 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 02 Jun 2008 12:41:10 +0000
Subject: Conditional caching question 
In-Reply-To: Your message of "Sat, 31 May 2008 12:43:58 -0300."
	<4841723E.2020109@eastlink.ca> 
Message-ID: <5225.1212410470@critter.freebsd.dk>

In message <4841723E.2020109 at eastlink.ca>, David Pratt writes:

>Hi. In most cases, I want a request to be passed to a backend where it 
>will be handled by server. If frequency is high, however; I want to add 
>the object to varnish cache and have varnish handle it. I am not worried 
>about a mechanism to keeping track of frequency of requests. Question is 
>what is available to me to add an object/path to the varnish cache if it 
>was originally passed? 

That's the problem, we don't have any place to hold statistics for
objects that get "pass" treatment, so we would have no way of knowing
that a particular object was suitable for caching.

You could write a program that extracts that information from the
varnishlog on the fly (sort of like varnishtop), but then we need
a mechanism to get from there to tell varnishd that this or that
object should be cached from now on.

I wouldn't say that your way of using varnish is backwards relative
to the design objectives, but you do come close, since we assumed
caching by default, and pass as exception, rather than the other
way around.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From chris.shenton at nasa.gov  Mon Jun  2 14:57:17 2008
From: chris.shenton at nasa.gov (Chris Shenton)
Date: Mon, 2 Jun 2008 10:57:17 -0400
Subject: Blew away .../var/varnish/HOSTNAME/ -- broke varnishstat,
	how to recover?
Message-ID: <209C76B7-DE29-4033-AB6C-E687B6B97644@nasa.gov>

We're using varnish for a public site, works beautifully and handled  
getting slashdotted gracefully -- many thanks!

We built it with "zc.buildout" which creates .../parts/varnish/install/ 
var/varnish/FQDN/... and others.  In a recent update, I accidentally  
blew away the .../var/varnish/FQDN directory which contained a few  
files that seem necessary for "varnishstat" and perhaps others to work.

  bin.XXkP5ZYv
  _.c
  _.vsl

$ varnishstat
Cannot open /usr/local/mastersite_buildout/parts/varnish/install/var/ 
varnish/FQDN/_.vsl: No such file or directory

Varnish continues to serve well, so I haven't restarted.

What's the recommended way to re-create this directory?  I'm assuming  
restarting Varnish will re-create it but want to be sure.

We have to fill out pounds of paperwork in order to take any outage on  
a public server, no matter how small.  Is there a way to restart  
Varnish without any downtime -- to continue accepting but holding  
connections until restarted, rather like Apache's "apachectl graceful"  
does?  Other ideas?

Thanks.



From phk at phk.freebsd.dk  Mon Jun  2 15:19:06 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 02 Jun 2008 15:19:06 +0000
Subject: Blew away .../var/varnish/HOSTNAME/ -- broke varnishstat,
	how to recover? 
In-Reply-To: Your message of "Mon, 02 Jun 2008 10:57:17 -0400."
	<209C76B7-DE29-4033-AB6C-E687B6B97644@nasa.gov> 
Message-ID: <5797.1212419946@critter.freebsd.dk>

In message <209C76B7-DE29-4033-AB6C-E687B6B97644 at nasa.gov>, Chris Shenton write
s:

>What's the recommended way to re-create this directory?  I'm assuming  
>restarting Varnish will re-create it but want to be sure.

Yes, restarting varnish (completely: ie both manager and child process)
should recreate it with no side effects.

>We have to fill out pounds of paperwork in order to take any outage on  
>a public server, no matter how small.  Is there a way to restart  
>Varnish without any downtime -- to continue accepting but holding  
>connections until restarted, rather like Apache's "apachectl graceful"  
>does?  Other ideas?

Varnish doesn't spend long time on startup, so I would just get it
done and over with.

You are in no rush however, so you can schedule it for whenever
you want.

(In the meantime, if you have enabled the telnet option, you can still
see the stats through the CLI interface.)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From chris.shenton at nasa.gov  Mon Jun  2 15:30:30 2008
From: chris.shenton at nasa.gov (Chris Shenton)
Date: Mon, 2 Jun 2008 11:30:30 -0400
Subject: Blew away .../var/varnish/HOSTNAME/ -- broke varnishstat,
	how to recover? 
In-Reply-To: <5797.1212419946@critter.freebsd.dk>
References: <5797.1212419946@critter.freebsd.dk>
Message-ID: <6E857806-920D-4E04-A1F6-9ED6E441C713@nasa.gov>


On Jun 2, 2008, at 11:19 AM, Poul-Henning Kamp wrote:
>
> Yes, restarting varnish (completely: ie both manager and child  
> process)
> should recreate it with no side effects.
>
> Varnish doesn't spend long time on startup, so I would just get it
> done and over with.

Just tried it on a QA server and it restarts in about 0.25 seconds,  
and re-creates the directory with the _.vsl file.  Have to see if my  
overlords are OK with that.  Thanks.
>
>
> (In the meantime, if you have enabled the telnet option, you can still
> see the stats through the CLI interface.)

I do have that and can ask it for stats. Thanks for the reminder. 


From erik at okcupid.com  Mon Jun  2 15:54:46 2008
From: erik at okcupid.com (Erik Steigler)
Date: Mon, 02 Jun 2008 11:54:46 -0400
Subject: Varnish/Swap usage and network timeouts
Message-ID: <484417C6.3020705@okcupid.com>

Hello,
I have 2 machines running varnish which handles around 1000 requests per 
second each and sometimes the machines will just stop responding to any 
network communication. A coworker used wireshark to check the connection 
and saw a whole lot of tcp retransmissions. Normally I would suspect 
some sort of network buffer issue but there is nothing in the logs at 
all. This is on FreeBSD  6.3 with a default 64 bit kernel with bge 
network cards. They are both running varnish-trunk revision 2635. 
Attached is the VCL file that is currently running. Varnish is started 
with the following options:
file,/vol/data1/varnish.cache,90% -t 259200 -h classic,500009 -p 
lru_interval=3600

The only suspicious things I've found are: Varnish says it is using 372G 
of virtual memory when I told it to use a 58gb file and there is only 
2gb of physical memory in the machine and there is pretty constant 
swapping going on, mostly under 500K but it seems to be pretty constant 
no matter the number of connections and such. The amount of swap space 
in the machine is 4gb and things are using 1.1gb.

Threads don't seem to be a problem, as the highest I've seen it is at 
500 and that was after an extended period of time around 250 or so. So I 
don't think I'm running into thread issues

So I guess I'm asking if anyone has any other suggestions other then 
reinstalling to FreeBSD 7.0 which seems to be the best solution I can 
see. If that is the best solution, what is the optimal FreeBSD 
configuration for Varnish? Lots of swap space? No swap space? Currently 
the mmap-ed file is on an SSD so one alternative I thought of was to 
make the entire SSD part of swap and use the malloc option and get rid 
of any regular disk basked swap.

Thanks for any help,
Erik


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: default.vcl
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080602/18744435/attachment.ksh>

From michael at dynamine.net  Mon Jun  2 16:14:49 2008
From: michael at dynamine.net (Michael S. Fischer)
Date: Mon, 2 Jun 2008 09:14:49 -0700
Subject: Blew away .../var/varnish/HOSTNAME/ -- broke varnishstat,
	how to recover?
In-Reply-To: <209C76B7-DE29-4033-AB6C-E687B6B97644@nasa.gov>
References: <209C76B7-DE29-4033-AB6C-E687B6B97644@nasa.gov>
Message-ID: <86db848d0806020914q38c2d44bpc55b1b1a06fa31f1@mail.gmail.com>

On Mon, Jun 2, 2008 at 7:57 AM, Chris Shenton <chris.shenton at nasa.gov>
wrote:

> We have to fill out pounds of paperwork in order to take any outage on
> a public server, no matter how small.  Is there a way to restart
> Varnish without any downtime -- to continue accepting but holding
> connections until restarted, rather like Apache's "apachectl graceful"
> does?  Other ideas?


Can you avoid the problem by putting your Varnish servers behind a load
balancer?  That way, you can preemptively disable the server from taking
traffic on the LB side prior to restarting Varnish, thereby eliminating any
perceivable customer effect.

Also, be careful about using "apachectl graceful" (especially combined with
log rotation), as connections that are currently idle but which may never
receive traffic again will not be terminated.  I consider it too leaky to
use.

Best regards,

--Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080602/8551d5c2/attachment.html>

From anders at fupp.net  Wed Jun  4 08:20:54 2008
From: anders at fupp.net (Anders Nordby)
Date: Wed, 4 Jun 2008 10:20:54 +0200
Subject: Varnish/Swap usage and network timeouts
In-Reply-To: <484417C6.3020705@okcupid.com>
References: <484417C6.3020705@okcupid.com>
Message-ID: <20080604082054.GA87105@fupp.net>

Hi,

Please try:

- upgrade to FreeBSD 7.0, and use SCHED_ULE instead of SCHED_4BSD.

- use -s malloc,<n>G instead of -s file. Set up large swap areas
according to your needs.

- if you get "swap zone exhausted, increase kern.maxswzone" error
messages, try increasing kern.maxswzone. Default is 32 MB, try 256 MB or
something like that: kern.maxswzone="268435456"

- checking http://varnish.projects.linpro.no/wiki/Performance. :-)

Cheers,
Anders.

On Mon, Jun 02, 2008 at 11:54:46AM -0400, Erik Steigler wrote:
> Hello,
> I have 2 machines running varnish which handles around 1000 requests per 
> second each and sometimes the machines will just stop responding to any 
> network communication. A coworker used wireshark to check the connection 
> and saw a whole lot of tcp retransmissions. Normally I would suspect 
> some sort of network buffer issue but there is nothing in the logs at 
> all. This is on FreeBSD  6.3 with a default 64 bit kernel with bge 
> network cards. They are both running varnish-trunk revision 2635. 
> Attached is the VCL file that is currently running. Varnish is started 
> with the following options:
> file,/vol/data1/varnish.cache,90% -t 259200 -h classic,500009 -p 
> lru_interval=3600
> 
> The only suspicious things I've found are: Varnish says it is using 372G 
> of virtual memory when I told it to use a 58gb file and there is only 
> 2gb of physical memory in the machine and there is pretty constant 
> swapping going on, mostly under 500K but it seems to be pretty constant 
> no matter the number of connections and such. The amount of swap space 
> in the machine is 4gb and things are using 1.1gb.
> 
> Threads don't seem to be a problem, as the highest I've seen it is at 
> 500 and that was after an extended period of time around 250 or so. So I 
> don't think I'm running into thread issues
> 
> So I guess I'm asking if anyone has any other suggestions other then 
> reinstalling to FreeBSD 7.0 which seems to be the best solution I can 
> see. If that is the best solution, what is the optimal FreeBSD 
> configuration for Varnish? Lots of swap space? No swap space? Currently 
> the mmap-ed file is on an SSD so one alternative I thought of was to 
> make the entire SSD part of swap and use the malloc option and get rid 
> of any regular disk basked swap.
> 
> Thanks for any help,
> Erik
> 
> 

> backend default {
> 	.host = "xzy.com";
> 	.port = "80";
> }
> 
> sub vcl_recv {
>   if (req.url ~ "\.js|css$") {
> 	if (req.http.Accept-Encoding ~ "gzip") {
> 		set req.http.Accept-Encoding = "gzip";
> 	} elsif (req.http.Accept-Encoding ~ "deflate") {
> 		set req.http.Accept-Encoding = "deflate";
> 	} else {
> 		unset req.http.accept-encoding;
> 	}
>   } else {
> 	unset req.http.accept-encoding;
>   }
> 
> 
>   unset		req.http.Cookie;
>   set		req.grace = 10m;
> }
> 
> sub vcl_fetch {
>   set		obj.http.P3P = "CP='NOI CURa ADMa DEVa TAIa OUR BUS IND UNI COM NAV INT', policyref='http://www.okcupid.com/w3c/p3p.xml'";
>   set		obj.grace = 10m;
>   if (obj.ttl < 259200s) {
> 	set	obj.ttl = 259200s;
>   }
> }
> 

> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc



-- 
Anders.


From wichert at wiggy.net  Wed Jun  4 07:54:31 2008
From: wichert at wiggy.net (Wichert Akkerman)
Date: Wed, 04 Jun 2008 09:54:31 +0200
Subject: Aiming for Varnish 2.0
In-Reply-To: <12466.1212270499@critter.freebsd.dk>
References: <12466.1212270499@critter.freebsd.dk>
Message-ID: <48464A37.3000202@wiggy.net>

Poul-Henning Kamp wrote:
> We are putting Varnish 2.0 right in the cross-hairs now, and that means
> that I'll be trying to close as many outstanding issues and tickets as
> possible.
>    
Thanks, I'll play with it this week.

I haven't seen a list of changes for 2.0, but one thing I was wondering 
is if it adds request processing time statistics. One of the most useful 
statistics we got from Squid was request miss services times, hit 
service times and overal service times. I have not seen anything 
similar  in varnishstat output or the munin plugin.

Wichert.

-- 
Wichert Akkerman<wichert at wiggy.net>    It is simple to make things.
http://www.wiggy.net/                  It is hard to make things simple.



From alex.taggart at gmail.com  Wed Jun  4 23:38:46 2008
From: alex.taggart at gmail.com (Alexander)
Date: Wed, 4 Jun 2008 23:38:46 +0000 (UTC)
Subject: Accessing req and resp header values
Message-ID: <loom.20080604T232548-357@post.gmane.org>

I am trying to implement a bit of logic to change the Content-Type of the
response, but part of the logic requires knowing the User-agent, which is in the
request.

Example:

sub vcl_deliver {
    if (req.http.User-Agent ~ "MSIE 6" 
        && resp.http.Content-Type ~ "application/xhtml+xml") {
        set resp.http.Content-Type = "text/html";
    }
    deliver;
}

Result:
Variable 'req.http.User-Agent' not accessible in method 'vcl_deliver'

I haven't found any documentation stating what variables are available in which
functions, but so far I haven't been able to figure out any way of doing this
short of messing with the hash and having unnecessary duplicates in the cache.

Any suggestions?



From barry at automattic.com  Thu Jun  5 06:03:15 2008
From: barry at automattic.com (Barry Abrahamson)
Date: Thu, 5 Jun 2008 02:03:15 -0400
Subject: Conditional caching question 
In-Reply-To: <5225.1212410470@critter.freebsd.dk>
References: <5225.1212410470@critter.freebsd.dk>
Message-ID: <FF7F8DE2-3EB5-408D-A918-90222591F9E1@automattic.com>

On Jun 2, 2008, at 8:41 AM, Poul-Henning Kamp wrote:

> In message <4841723E.2020109 at eastlink.ca>, David Pratt writes:
>
>> Hi. In most cases, I want a request to be passed to a backend where  
>> it
>> will be handled by server. If frequency is high, however; I want to  
>> add
>> the object to varnish cache and have varnish handle it. I am not  
>> worried
>> about a mechanism to keeping track of frequency of requests.  
>> Question is
>> what is available to me to add an object/path to the varnish cache  
>> if it
>> was originally passed?
>
> I wouldn't say that your way of using varnish is backwards relative
> to the design objectives, but you do come close, since we assumed
> caching by default, and pass as exception, rather than the other
> way around.

We do this on WordPress.com to avoid filling our caches with  
infrequently requested data.  The way we handle it is when an object  
reaches a certain req/sec threshold, we send a header from the backend  
and then have varnish configured to only insert objects into the cache  
which contain this custom header.  Based on phk's reply, I guess we  
are using varnish in a somewhat backwards manner as well, since we  
assume pass as the detault, insert as the exception.

This used to work in 1.0.3.  I have started to look into upgrading to  
trunk, and it doesn't seem to work so well anymore.  It looks like the  
first time the URL is requested, if it is passed because it hasn't  
reached that threshold and the header hasn't been set, all subsequent  
requests are automatically "pass" ed.  These show up as "Cache hits  
for pass" in varnishstat.  Any way around this?


--
Barry Abrahamson | Systems Wrangler | Automattic
Blog: http://barry.wordpress.com








From phk at phk.freebsd.dk  Thu Jun  5 07:54:32 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 05 Jun 2008 07:54:32 +0000
Subject: Conditional caching question 
In-Reply-To: Your message of "Thu, 05 Jun 2008 02:03:15 -0400."
	<FF7F8DE2-3EB5-408D-A918-90222591F9E1@automattic.com> 
Message-ID: <3603.1212652472@critter.freebsd.dk>

In message <FF7F8DE2-3EB5-408D-A918-90222591F9E1 at automattic.com>, Barry Abraham
son writes:
>On Jun 2, 2008, at 8:41 AM, Poul-Henning Kamp wrote:

>We do this on WordPress.com to avoid filling our caches with  
>infrequently requested data.  The way we handle it is when an object  
>reaches a certain req/sec threshold, we send a header from the backend  
>and then have varnish configured to only insert objects into the cache  
>which contain this custom header.  Based on phk's reply, I guess we  
>are using varnish in a somewhat backwards manner as well, since we  
>assume pass as the detault, insert as the exception.

Not at all, the "backwards" aspect was if Varnish was supposed to
be able to figure out when to start caching an object.

Having the backend tell Varnish what to cache and how long is
exactly how Varnish was designed to work.

>This used to work in 1.0.3.  I have started to look into upgrading to  
>trunk, and it doesn't seem to work so well anymore.  It looks like the  
>first time the URL is requested, if it is passed because it hasn't  
>reached that threshold and the header hasn't been set, all subsequent  
>requests are automatically "pass" ed.  These show up as "Cache hits  
>for pass" in varnishstat.  Any way around this?

When you do _not_ detect the magic header, set the TTL lower, that
controls how long time the "this should be passed" cache object
lives.

If that doesn't work, it's a bug.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From riham.aldakkak at espace.com.eg  Thu Jun  5 09:25:58 2008
From: riham.aldakkak at espace.com.eg (Riham Aldakkak)
Date: Thu, 5 Jun 2008 12:25:58 +0300
Subject: varnishtop ranking
Message-ID: <d2b26bf0806050225g5f3fd06dhe7546457dffcc593@mail.gmail.com>

The varnishtop displays a list of most frequent logs .... It displays 3
columns: 1 representing the frequency , the Header & the value ...
What does this frequenct represent ? The number of repetition of this log in
the shared memory , for example , or what exactly ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080605/d293fcb9/attachment.html>

From phk at phk.freebsd.dk  Thu Jun  5 09:29:32 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 05 Jun 2008 09:29:32 +0000
Subject: Accessing req and resp header values 
In-Reply-To: Your message of "Wed, 04 Jun 2008 23:38:46 GMT."
	<loom.20080604T232548-357@post.gmane.org> 
Message-ID: <6451.1212658172@critter.freebsd.dk>

In message <loom.20080604T232548-357 at post.gmane.org>, Alexander writes:

>Result:
>Variable 'req.http.User-Agent' not accessible in method 'vcl_deliver'
>
>I haven't found any documentation stating what variables are available in which
>functions, but so far I haven't been able to figure out any way of doing this
>short of messing with the hash and having unnecessary duplicates in the cache.

It's in the secret supplement to the master code book :-)

I can't remember why you do not have access to the request in vcl_deliver{}
there was a good reason at the time, but there may be better reasons to
fix that reason.

Please open a ticket for this, and we'll look at it.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Thu Jun  5 09:37:27 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 05 Jun 2008 09:37:27 +0000
Subject: varnishtop ranking 
In-Reply-To: Your message of "Thu, 05 Jun 2008 12:25:58 +0300."
	<d2b26bf0806050225g5f3fd06dhe7546457dffcc593@mail.gmail.com> 
Message-ID: <6499.1212658647@critter.freebsd.dk>

In message <d2b26bf0806050225g5f3fd06dhe7546457dffcc593 at mail.gmail.com>, "Riham
 Aldakkak" writes:

>The varnishtop displays a list of most frequent logs .... It displays 3
>columns: 1 representing the frequency , the Header & the value ...
>What does this frequenct represent ? The number of repetition of this log in
>the shared memory , for example , or what exactly ?

It's an exponential decay of the frequency of arrival if I remember
right, think of it as a "bogo-frequency".

If anybody can come up with a better method/metric, I'm all ears.

Varnishtop was just something I threw together in 15 minutes it never
got the love and attention it really deserves.

And it would make great starting point for hacking varnish code.

(he said knowingly, nude, nudge, wink, wink, ya'know what I mean ?) :-)

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From anders at fupp.net  Thu Jun  5 16:43:33 2008
From: anders at fupp.net (Anders Nordby)
Date: Thu, 5 Jun 2008 18:43:33 +0200
Subject: Aiming for Varnish 2.0
In-Reply-To: <48464A37.3000202@wiggy.net>
References: <12466.1212270499@critter.freebsd.dk> <48464A37.3000202@wiggy.net>
Message-ID: <20080605164333.GA46178@fupp.net>

Hi,

On Wed, Jun 04, 2008 at 09:54:31AM +0200, Wichert Akkerman wrote:
>> We are putting Varnish 2.0 right in the cross-hairs now, and that means
>> that I'll be trying to close as many outstanding issues and tickets as
>> possible.
>>    
> Thanks, I'll play with it this week.
> I haven't seen a list of changes for 2.0, but one thing I was wondering 
> is if it adds request processing time statistics. One of the most useful 
> statistics we got from Squid was request miss services times, hit 
> service times and overal service times. I have not seen anything 
> similar  in varnishstat output or the munin plugin.

I agree that this is very useful.

You can also just monitor HTTP response time. Example:
http://muninexchange.projects.linpro.no/?search=&cid=10&os%5B4%5D=on&os%5B7%5D=on&os%5B3%5D=on&os%5B2%5D=on&os%5B5%5D=on&os%5B8%5D=on&os%5B1%5D=on&os%5B6%5D=on&pid=158

Bye,

-- 
Anders.


From rbp at occam.com.br  Thu Jun  5 22:32:18 2008
From: rbp at occam.com.br (Rodrigo Pimentel)
Date: Thu, 5 Jun 2008 19:32:18 -0300
Subject: Varnish + ESI in production
Message-ID: <4706efb00806051532j1d3d82fercaf9fefd9ac686c4@mail.gmail.com>

Hi,

I was wondering if there's any report of varnish + esi being used in
production, particularly in heavy traffic sites (but reports otherwise
would be nice too). I'm considering it for a project and have already
gathered some information, but I'd love to hear more about your
experiences.

Thanks!

Cheers,


    rbp
-- 
Rodrigo Bernardo Pimentel <rbp at occam.com.br>
http://occam.com.br
+55 11 3628-9723
+55 11 9213-3252


From wichert at wiggy.net  Fri Jun  6 11:20:11 2008
From: wichert at wiggy.net (Wichert Akkerman)
Date: Fri, 06 Jun 2008 13:20:11 +0200
Subject: Aiming for Varnish 2.0
In-Reply-To: <12466.1212270499@critter.freebsd.dk>
References: <12466.1212270499@critter.freebsd.dk>
Message-ID: <48491D6B.4080005@wiggy.net>

Poul-Henning Kamp wrote:
> We are putting Varnish 2.0 right in the cross-hairs now, and that means
> that I'll be trying to close as many outstanding issues and tickets as
> possible.
>
> I will strongly urge you all to run -trunk and help flush out the
> remaining bugs.
>    

We had a varnish trunk instance stop working today. All backend servers 
were running correctly but requests going through varnish hung. The 
varnishlog output is below. After restarting the varnish daemon 
everything worked normally again.

    54 SessionOpen  c 10.121.10.125 54873
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750905.378312826 1212750905.378312826 
0.000177622 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83586 82633 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54873 0 1 0 0 0 0 0 0
    54 SessionOpen  c 10.121.10.125 54875
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750906.390101671 1212750906.390101671 
0.000184059 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83587 82634 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54875 0 1 0 0 0 0 0 0
    54 SessionOpen  c 10.121.10.125 54877
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750907.398921728 1212750907.398921728 
0.000149965 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83588 82635 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54877 0 1 0 0 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1212750907
    54 SessionOpen  c 10.121.10.125 54879
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750908.408715248 1212750908.408715248 
0.000169516 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83589 82636 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54879 0 1 0 0 0 0 0 0
    54 SessionOpen  c 10.121.10.125 54881
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750909.418306112 1212750909.418306112 
0.000190735 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83591 82637 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54881 0 1 0 0 0 0 0 0
    54 SessionOpen  c 10.121.10.125 54883
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750910.429055929 1212750910.429055929 
0.000201941 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83592 82638 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54883 0 1 0 0 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1212750910
    54 SessionOpen  c 10.121.10.125 54885
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750911.438663244 1212750911.438663244 
0.000112534 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83593 82639 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54885 0 1 0 0 0 0 0 0
    54 SessionOpen  c 10.121.10.125 54887
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750912.448537588 1212750912.448537588 
0.000156164 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83594 82640 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54887 0 1 0 0 0 0 0 0
     0 ExpKill        456634866 0
    54 SessionOpen  c 10.121.10.125 54889
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750913.458305836 1212750913.458305836 
0.000211954 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83595 82641 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54889 0 1 0 0 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1212750913
    54 SessionOpen  c 10.121.10.125 54891
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750914.468340158 1212750914.468340158 
0.000185251 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83596 82642 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54891 0 1 0 0 0 0 0 0
    54 SessionOpen  c 10.121.10.125 54893
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750915.479013205 1212750915.479013205 
0.000136852 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83597 82643 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54893 0 1 0 0 0 0 0 0
    54 SessionOpen  c 10.121.10.125 54895
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750916.488635540 1212750916.488635540 
0.000167847 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83598 82644 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54895 0 1 0 0 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1212750916
    54 SessionOpen  c 10.121.10.125 54897
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750917.498711586 1212750917.498711586 
0.000220776 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83599 82645 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54897 0 1 0 0 0 0 0 0
    54 SessionOpen  c 10.121.10.125 54899
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750918.508799791 1212750918.508799791 
0.000160933 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83600 82646 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54899 0 1 0 0 0 0 0 0
    54 SessionOpen  c 10.121.10.125 54901
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750919.518652439 1212750919.518652439 
0.000189543 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83601 82647 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54901 0 1 0 0 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1212750919
    54 SessionOpen  c 10.121.10.125 54903
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750920.528769732 1212750920.528769732 
0.000162125 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83602 82648 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54903 0 1 0 0 0 0 0 0
    54 SessionOpen  c 10.121.10.125 54905
    54 HttpError    c Received nothing
    54 SessionClose c silent
    54 ReqEnd       c 0 1212750921.538681984 1212750921.538681984 
0.000258446 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83603 82649 0 0 0 0 0 0
    54 StatSess     c 10.121.10.125 54905 0 1 0 0 0 0 0 0
    54 SessionOpen  c 10.121.10.84 41839
    54 ReqStart     c 10.121.10.84 41839 456634871
    54 RxRequest    c GET
    54 RxURL        c /
    54 RxProtocol   c HTTP/1.0
    54 RxHeader     c User-Agent: w3m/0.5.1+cvs-1.946
    54 RxHeader     c Accept: text/*, image/*, audio/*, application/*
    54 RxHeader     c Accept-Encoding: gzip, compress, bzip, bzip2, deflate
    54 RxHeader     c Accept-Language: en;q=1.0
    54 RxHeader     c Host: plone.customer.int
    54 VCL_call     c recv
    54 VCL_return   c lookup
    54 VCL_call     c hash
    54 VCL_return   c hash
    54 Debug        c "on waiting list on obj 456633699"
     0 StatAddr       10.121.10.84 0 nan 22 20 0 0 2 2397 16350
    55 SessionOpen  c 10.121.10.125 54907
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750922.548995256 1212750922.548995256 
0.000127077 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83604 82650 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54907 0 1 0 0 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1212750922
    55 SessionOpen  c 10.121.10.125 54909
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750923.558841467 1212750923.558841467 
0.000233412 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83605 82651 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54909 0 1 0 0 0 0 0 0
    55 SessionOpen  c 10.121.10.125 54911
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750924.568663836 1212750924.568663836 
0.000154972 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83606 82652 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54911 0 1 0 0 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1212750925
    55 SessionOpen  c 10.121.10.125 54913
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750925.578656435 1212750925.578656435 
0.000115395 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83607 82653 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54913 0 1 0 0 0 0 0 0
    55 SessionOpen  c 10.121.10.125 54915
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750926.588635683 1212750926.588635683 
0.000172138 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83608 82654 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54915 0 1 0 0 0 0 0 0
    55 SessionOpen  c 10.121.10.125 54917
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750927.599018574 1212750927.599018574 
0.000161171 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83609 82655 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54917 0 1 0 0 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1212750928
    55 SessionOpen  c 10.121.10.125 54919
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750928.609158754 1212750928.609158754 
0.000053167 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83610 82656 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54919 0 1 0 0 0 0 0 0
    55 SessionOpen  c 10.121.10.125 54921
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750929.620168686 1212750929.620168686 
0.000217199 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83611 82657 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54921 0 1 0 0 0 0 0 0
    55 SessionOpen  c 10.121.10.125 54923
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750930.629410744 1212750930.629410744 
0.000182867 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83612 82658 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54923 0 1 0 0 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1212750931
    55 SessionOpen  c 10.121.10.125 54925
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750931.639029741 1212750931.639029741 
0.000146389 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83613 82659 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54925 0 1 0 0 0 0 0 0
    55 SessionOpen  c 10.121.10.125 54927
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750932.649048805 1212750932.649048805 
0.000166893 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83614 82660 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54927 0 1 0 0 0 0 0 0
    55 SessionOpen  c 10.121.10.125 54929
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750933.659260511 1212750933.659260511 
0.000162840 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83615 82661 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54929 0 1 0 0 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1212750934
    55 SessionOpen  c 10.121.10.125 54931
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750934.671282768 1212750934.671282768 
0.000159502 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83616 82662 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54931 0 1 0 0 0 0 0 0
    55 SessionOpen  c 10.121.10.125 54933
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750935.678945065 1212750935.678945065 
0.000104427 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83617 82663 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54933 0 1 0 0 0 0 0 0
    55 SessionOpen  c 10.121.10.125 54935
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750936.689173222 1212750936.689173222 
0.000151396 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83618 82664 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54935 0 1 0 0 0 0 0 0
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1212750937
    55 SessionOpen  c 10.121.10.125 54937
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750937.699398041 1212750937.699398041 
0.000175238 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83619 82665 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54937 0 1 0 0 0 0 0 0
    55 SessionOpen  c 10.121.10.125 54939
    55 HttpError    c Received nothing
    55 SessionClose c silent
    55 ReqEnd       c 0 1212750938.709848642 1212750938.709848642 
0.000188351 0.000000000 0.000000000
     0 StatAddr       10.121.10.125 0 83620 82666 0 0 0 0 0 0
    55 StatSess     c 10.121.10.125 54939 0 1 0 0 0 0 0 0


-- 
Wichert Akkerman<wichert at wiggy.net>    It is simple to make things.
http://www.wiggy.net/                  It is hard to make things simple.



From phk at phk.freebsd.dk  Fri Jun  6 11:28:50 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 06 Jun 2008 11:28:50 +0000
Subject: Aiming for Varnish 2.0 
In-Reply-To: Your message of "Fri, 06 Jun 2008 13:20:11 +0200."
	<48491D6B.4080005@wiggy.net> 
Message-ID: <14267.1212751730@critter.freebsd.dk>

In message <48491D6B.4080005 at wiggy.net>, Wichert Akkerman writes:
>Poul-Henning Kamp wrote:
>> We are putting Varnish 2.0 right in the cross-hairs now, and that means
>> that I'll be trying to close as many outstanding issues and tickets as
>> possible.
>>
>> I will strongly urge you all to run -trunk and help flush out the
>> remaining bugs.
>>    
>
>We had a varnish trunk instance stop working today.

Make sure you have commit #2651.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From chris.shenton at nasa.gov  Fri Jun  6 14:18:47 2008
From: chris.shenton at nasa.gov (Chris Shenton)
Date: Fri, 6 Jun 2008 10:18:47 -0400
Subject: Varnish + ESI in production
In-Reply-To: <4706efb00806051532j1d3d82fercaf9fefd9ac686c4@mail.gmail.com>
References: <4706efb00806051532j1d3d82fercaf9fefd9ac686c4@mail.gmail.com>
Message-ID: <3E98FA8F-D591-446A-A415-C9AC738A1B92@nasa.gov>


On Jun 5, 2008, at 6:32 PM, Rodrigo Pimentel wrote:

> Hi,
>
> I was wondering if there's any report of varnish + esi being used in
> production, particularly in heavy traffic sites (but reports otherwise
> would be nice too). I'm considering it for a project and have already
> gathered some information, but I'd love to hear more about your
> experiences.

We're using it for http://nasascience.nasa.gov/ and it allowed us to  
withstand Reddit, Digg, Slashdot when we went live.   Great stuff.



From tech at safira.com  Fri Jun  6 17:06:50 2008
From: tech at safira.com (Adriano Nagel)
Date: Fri, 6 Jun 2008 14:06:50 -0300
Subject: Varnish + ESI in production
In-Reply-To: <80a65d4e0806060957k5ed5b77ds44732b1103eb68c8@mail.gmail.com>
References: <4706efb00806051532j1d3d82fercaf9fefd9ac686c4@mail.gmail.com>
	<3E98FA8F-D591-446A-A415-C9AC738A1B92@nasa.gov>
	<80a65d4e0806060957k5ed5b77ds44732b1103eb68c8@mail.gmail.com>
Message-ID: <80a65d4e0806061006h5270e538u56a84d204f60dc18@mail.gmail.com>

> On Fri, Jun 6, 2008 at 11:18 AM, Chris Shenton <chris.shenton at nasa.gov> wrote:
>>
>> We're using it for http://nasascience.nasa.gov/ and it allowed us to
>> withstand Reddit, Digg, Slashdot when we went live.   Great stuff.

[repost from my subscriber's address]

Hey, very good news indeed. Should go straight to a testimonials page
in the project site.

So, are you using trunk? [edit: I mean, what revision?]

Question for the developers: regarding the roadmap, is there any ETA
for a stable Varnish release that implements ESI? Will there be a 1.2
release, or will you go straight to 2.0?

Thanks!

--
Adriano


From chris.shenton at nasa.gov  Fri Jun  6 18:47:02 2008
From: chris.shenton at nasa.gov (Chris Shenton)
Date: Fri, 6 Jun 2008 14:47:02 -0400
Subject: Varnish + ESI in production
In-Reply-To: <48494EDC.5030308@endpoint.com>
References: <4706efb00806051532j1d3d82fercaf9fefd9ac686c4@mail.gmail.com>
	<3E98FA8F-D591-446A-A415-C9AC738A1B92@nasa.gov>
	<48494EDC.5030308@endpoint.com>
Message-ID: <52902CA0-8693-4F5D-9671-645773178587@nasa.gov>


On Jun 6, 2008, at 10:51 AM, JT Justman wrote:

> Chris Shenton wrote:
>> We're using it for http://nasascience.nasa.gov/ and it allowed us  
>> to  withstand Reddit, Digg, Slashdot when we went live.   Great  
>> stuff.
>
> Cool! How many includes do you have? What are your TTLs like? And  
> what trunk are you running?
>

CC'ing the list as a few other folks asked similar questions privately.

Er, sorry, I think I read your message too quickly and lead to  
confusion. We're not (yet) using ESIs, pretty much just the stock  
settings.  We've got it fronting Apache which multiplexes to 4 backend  
Zope instances running Plone.

We're running varnish-1.1.2. I don't think I'd be comfortable living  
on the bleeding edge for a public site, not unless I learned a LOT  
more about varnish anyway :-)




From anr at safira.com  Sun Jun  8 21:05:47 2008
From: anr at safira.com (Adriano Nagel)
Date: Sun, 8 Jun 2008 18:05:47 -0300
Subject: ESI is generating extraneous output for some clients
Message-ID: <80a65d4e0806081405v8433ebdibae59f1029b98e2b@mail.gmail.com>

Hi,

When using ESI (trunk r2652), varnish is adding extraneous characters
to the output sent to the client, depending on the client being used.

For example, for this html file:

<html>
<body>
<h1>Teste ESI 8</h1>
<b><esi:include src="/cgi-bin/date" /></b>
</body>
</html>

and this vcl snippet:

    } elseif (req.url ~ "^/t(este)?.html$") {
        remove obj.http.Last-Modified;
        esi;  /* Do ESI processing */
        set obj.ttl = 1m;

"wget http://localhost/teste.html" generates this output:

26
<html>
<body>
<h1>Teste ESI 8</h1>
<b>
19
Sun Jun  8 17:37:43 2008

15
</b>
</body>
</html>

0

telnet generates the same output.

On the other hand, curl & safari get the expected output, eg

$ curl http://localhost/teste.html
<html>
<body>
<h1>Teste ESI 8</h1>
<b>Sun Jun  8 17:51:49 2008
</b>
</body>
</html>

Can anyone duplicate this problem?

Thanks,

--
Adriano


From phk at phk.freebsd.dk  Sun Jun  8 21:09:19 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 08 Jun 2008 21:09:19 +0000
Subject: ESI is generating extraneous output for some clients 
In-Reply-To: Your message of "Sun, 08 Jun 2008 18:05:47 -0300."
	<80a65d4e0806081405v8433ebdibae59f1029b98e2b@mail.gmail.com> 
Message-ID: <4171.1212959359@critter.freebsd.dk>

In message <80a65d4e0806081405v8433ebdibae59f1029b98e2b at mail.gmail.com>, "Adria
no Nagel" writes:
>Hi,
>
>When using ESI (trunk r2652), varnish is adding extraneous characters
>to the output sent to the client, depending on the client being used.
>
>For example, for this html file:
>
><html>
><body>
><h1>Teste ESI 8</h1>
><b><esi:include src="/cgi-bin/date" /></b>
></body>
></html>
>
>and this vcl snippet:
>
>    } elseif (req.url ~ "^/t(este)?.html$") {
>        remove obj.http.Last-Modified;
>        esi;  /* Do ESI processing */
>        set obj.ttl = 1m;
>
>"wget http://localhost/teste.html" generates this output:
>
>26
><html>
><body>
><h1>Teste ESI 8</h1>
><b>
>19
>Sun Jun  8 17:37:43 2008
>
>15
></b>
></body>
></html>
>
>0

When processing ESI documents, Varnish will send "chunked" encoding
and the number lines you see ("26", "19", "15" and "0") are the
chunked headers, they say (in hex) how many characters follow.

So for telnet, the above is absolutely correct, but wget shouldn't
show these lines.

>On the other hand, curl & safari get the expected output, eg

As they obviously handle the chunked encoding correctly.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From anr at safira.com  Sun Jun  8 21:23:43 2008
From: anr at safira.com (Adriano Nagel)
Date: Sun, 8 Jun 2008 18:23:43 -0300
Subject: ESI is generating extraneous output for some clients
In-Reply-To: <4171.1212959359@critter.freebsd.dk>
References: <80a65d4e0806081405v8433ebdibae59f1029b98e2b@mail.gmail.com>
	<4171.1212959359@critter.freebsd.dk>
Message-ID: <80a65d4e0806081423t81b6a07i46db007b0aa8656f@mail.gmail.com>

On Sun, Jun 8, 2008 at 6:09 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> When processing ESI documents, Varnish will send "chunked" encoding
> and the number lines you see ("26", "19", "15" and "0") are the
> chunked headers, they say (in hex) how many characters follow.

Aha, I didn't know about chunked encoding, and missed the
"Transfer-Encoding" header hint.

Thanks,

--
Adriano


From fairwinds at eastlink.ca  Mon Jun  9 03:32:51 2008
From: fairwinds at eastlink.ca (David Pratt)
Date: Mon, 09 Jun 2008 00:32:51 -0300
Subject: Conditional caching question
In-Reply-To: <FF7F8DE2-3EB5-408D-A918-90222591F9E1@automattic.com>
References: <5225.1212410470@critter.freebsd.dk>
	<FF7F8DE2-3EB5-408D-A918-90222591F9E1@automattic.com>
Message-ID: <484CA463.6090502@eastlink.ca>

Hi. Sorry for not getting back sooner. The use case I have is have 
backend determine when frequency reaches a threshold and set ttl 
dynamically based on the rate. My goal is to tune cache using rates and 
ttl values to ensure only most requested makes it into the cache as 
opposed to adding 'bulk'. I am simply experimenting at this stage but 
believe it can result in a sort of 'smart caching' to maximize the 
benefit of the cache for content in demand while minimizing the overall 
cache resource to deliver the resources.

I am sure that somewhere here is an algorithm for that can provide some 
decent cache regulation for utilization and size. Its all a bit of a 
balancing act with the responsible for the backend as well I realize. 
Many thanks.

Regards
David


Barry Abrahamson wrote:
> On Jun 2, 2008, at 8:41 AM, Poul-Henning Kamp wrote:
> 
>> In message <4841723E.2020109 at eastlink.ca>, David Pratt writes:
>>
>>> Hi. In most cases, I want a request to be passed to a backend where it
>>> will be handled by server. If frequency is high, however; I want to add
>>> the object to varnish cache and have varnish handle it. I am not worried
>>> about a mechanism to keeping track of frequency of requests. Question is
>>> what is available to me to add an object/path to the varnish cache if it
>>> was originally passed?
>>
>> I wouldn't say that your way of using varnish is backwards relative
>> to the design objectives, but you do come close, since we assumed
>> caching by default, and pass as exception, rather than the other
>> way around.
> 
> We do this on WordPress.com to avoid filling our caches with 
> infrequently requested data.  The way we handle it is when an object 
> reaches a certain req/sec threshold, we send a header from the backend 
> and then have varnish configured to only insert objects into the cache 
> which contain this custom header.  Based on phk's reply, I guess we are 
> using varnish in a somewhat backwards manner as well, since we assume 
> pass as the detault, insert as the exception.
> 
> This used to work in 1.0.3.  I have started to look into upgrading to 
> trunk, and it doesn't seem to work so well anymore.  It looks like the 
> first time the URL is requested, if it is passed because it hasn't 
> reached that threshold and the header hasn't been set, all subsequent 
> requests are automatically "pass" ed.  These show up as "Cache hits for 
> pass" in varnishstat.  Any way around this?
> 
> 
> -- 
> Barry Abrahamson | Systems Wrangler | Automattic
> Blog: http://barry.wordpress.com
> 
> 
> 
> 
> 
> 
> 



From phk at phk.freebsd.dk  Mon Jun  9 07:05:42 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 09 Jun 2008 07:05:42 +0000
Subject: Conditional caching question 
In-Reply-To: Your message of "Mon, 09 Jun 2008 00:32:51 -0300."
	<484CA463.6090502@eastlink.ca> 
Message-ID: <6132.1212995142@critter.freebsd.dk>

In message <484CA463.6090502 at eastlink.ca>, David Pratt writes:

>Hi. Sorry for not getting back sooner. The use case I have is have 
>backend determine when frequency reaches a threshold and set ttl 
>dynamically based on the rate. [...]

And Varnish is a perfect frontend for that.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From wichert at wiggy.net  Mon Jun  9 07:13:26 2008
From: wichert at wiggy.net (Wichert Akkerman)
Date: Mon, 09 Jun 2008 09:13:26 +0200
Subject: guru meditation errors on timeouts?
Message-ID: <484CD816.5040404@wiggy.net>

We've upgraded to current varnish trunk but are seeing a fair number of guru 
meditation errors appear. My hunch is that those appear when the backend 
server takes too long to respond. Is there a way to verify that and to 
change the used timeout?

Wichert.


-- 
Wichert Akkerman <wichert at wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.



From phk at phk.freebsd.dk  Mon Jun  9 07:16:07 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 09 Jun 2008 07:16:07 +0000
Subject: guru meditation errors on timeouts? 
In-Reply-To: Your message of "Mon, 09 Jun 2008 09:13:26 +0200."
	<484CD816.5040404@wiggy.net> 
Message-ID: <6209.1212995767@critter.freebsd.dk>

In message <484CD816.5040404 at wiggy.net>, Wichert Akkerman writes:

>We've upgraded to current varnish trunk but are seeing a fair number of guru 
>meditation errors appear. My hunch is that those appear when the backend 
>server takes too long to respond. Is there a way to verify that and to 
>change the used timeout?

Do you have a varnishlog where this happens ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From wichert at wiggy.net  Mon Jun  9 14:32:00 2008
From: wichert at wiggy.net (Wichert Akkerman)
Date: Mon, 09 Jun 2008 16:32:00 +0200
Subject: guru meditation errors on timeouts?
In-Reply-To: <6209.1212995767@critter.freebsd.dk>
References: <6209.1212995767@critter.freebsd.dk>
Message-ID: <484D3EE0.9010105@wiggy.net>

Poul-Henning Kamp wrote:
> In message <484CD816.5040404 at wiggy.net>, Wichert Akkerman writes:
> 
>> We've upgraded to current varnish trunk but are seeing a fair number of guru 
>> meditation errors appear. My hunch is that those appear when the backend 
>> server takes too long to respond. Is there a way to verify that and to 
>> change the used timeout?
> 
> Do you have a varnishlog where this happens ?

I do now:

    12 SessionOpen  c 10.121.10.84 53022
    12 ReqStart     c 10.121.10.84 53022 1318563041
    12 RxRequest    c GET
    12 RxURL        c /nordic/elkjop-datter/sectors/ce/newsletter
    12 RxProtocol   c HTTP/1.1
    12 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 
10_5_3; nb-no) AppleWebKit/525.18 (KHTML, like Gecko) Version/
3.1.1 Safari/525.20
    12 RxHeader     c Referer: 
http://plone.elkjop.int/nordic/elkjop-datter/sectors/ce
    12 RxHeader     c Accept: 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    12 RxHeader     c Accept-Language: nb-no
    12 RxHeader     c Accept-Encoding: gzip, deflate
    12 RxHeader     c Cookie: 
tree-s="eJzTyCkw5NLIKTDiClZ3hALXLEdbda4CY65EoIQJSNYUSdYjPBIka8aVCAR6AAoUED0"; 
mainchain="943ed3c25e6d44497de
b3fe274f98a96"; _ZopeId="94470895A3ZdBnkIwXc"; 
__ac="#######################################="
    12 RxHeader     c Connection: keep-alive
    12 RxHeader     c Host: plone.elkjop.int
    12 VCL_call     c recv
    12 VCL_return   c lookup
    12 VCL_call     c hash
    12 VCL_return   c hash
    12 VCL_call     c miss
    12 VCL_return   c fetch
    13 TxRequest    b GET
    13 TxURL        b 
/VirtualHostBase/http/plone.elkjop.int:80/eli/VirtualHostRoot/nordic/elkjop-datter/sectors/ce/newsletter
    13 TxProtocol   b HTTP/1.1
    13 TxHeader     b User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 
10_5_3; nb-no) AppleWebKit/525.18 (KHTML, like Gecko) Version/
3.1.1 Safari/525.20
    13 TxHeader     b Referer: 
http://plone.elkjop.int/nordic/elkjop-datter/sectors/ce
    13 TxHeader     b Accept: 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    13 TxHeader     b Accept-Language: nb-no
    13 TxHeader     b Accept-Encoding: gzip, deflate
    13 TxHeader     b Cookie: 
tree-s="eJzTyCkw5NLIKTDiClZ3hALXLEdbda4CY65EoIQJSNYUSdYjPBIka8aVCAR6AAoUED0"; 
mainchain="943ed3c25e6d44497de
b3fe274f98a96"; _ZopeId="94470895A3ZdBnkIwXc"; 
__ac="#######################################="
    13 TxHeader     b Host: plone.elkjop.int
    13 TxHeader     b X-Varnish: 1318563041
    13 TxHeader     b X-Forwarded-For: 10.121.10.84
    13 BackendClose b lb01
    12 VCL_call     c fetch
     0 Debug        - "VCL_error(0, (null))"
    12 VCL_return   c error
    12 SessionClose c error returned
    12 TxProtocol   c HTTP/1.0
    12 TxStatus     c 503
    12 TxResponse   c Service Unavailable
    12 TxHeader     c Date: Mon, 09 Jun 2008 13:56:24 GMT
    12 TxHeader     c Server: Varnish
    12 TxHeader     c Retry-After: 0
    12 TxHeader     c Content-Type: text/html; charset=utf-8
    12 TxHeader     c X-Varnish: 1318563041
    12 TxHeader     c Connection: close
    12 ReqEnd       c 1318563041 1213019784.562031031 1213019784.562954903 
0.032984257 nan nan




-- 
Wichert Akkerman <wichert at wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.



From duja at torlen.net  Mon Jun  9 19:02:02 2008
From: duja at torlen.net (Erik Torlen)
Date: Mon, 09 Jun 2008 21:02:02 +0200
Subject: Strange Opera error in POST
In-Reply-To: <3ncizfw40ga7izx.230520081634@torlen.net>
References: <3ncizfw40ga7izx.230520081634@torlen.net>
Message-ID: <484D7E2A.3060804@torlen.net>

Ok, I have gathered some more info about the problem.

What I have is a form login that has 3 input fields, username, password 
and security code (from captcha).

When opera is making the POST it receives a 200 OK and NOT a 302 Moved 
Temp. as expected.

This is the request that Opera is making:

   13 RxRequest    c POST
   13 RxURL        c /takelogin.php
   13 RxProtocol   c HTTP/1.1
   13 RxHeader     c User-Agent: Opera/9.27 (Windows NT 5.1; U; sv)
   13 RxHeader     c Host: mydomain.com:8080
   13 RxHeader     c Accept: text/html, application/xml;q=0.9, 
application/xhtml+xml, image/png, image/jpeg, image/gif, 
image/x-xbitmap, */*;q=0.1
   13 RxHeader     c Accept-Language: sv-SE,sv;q=0.9,en;q=0.8
   13 RxHeader     c Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1
   13 RxHeader     c Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
   13 RxHeader     c Referer: http://mydomain.com:8080/login.php
   13 RxHeader     c Cookie: PHPSESSID=f3a60f178adede632c761dca745054cf
   13 RxHeader     c Cookie2: $Version=1
   13 RxHeader     c Connection: Keep-Alive, TE
   13 RxHeader     c TE: deflate, gzip, chunked, identity, trailers
   13 RxHeader     c Content-Length: 63
   13 RxHeader     c Content-Type: application/x-www-form-urlencoded

My vcl has this code when POST are received:

        if(req.request != "GET" && req.request != "HEAD") {
                set req.http.Connection = "close";
                pass;
        }

This code usually works with FF and IE but NOT with Opera.

If I remove "set req.http.Connection = "close";"

the login process works with no problem.

I have had problems with POSTs before, thats why I've been using 
Connection = "close" on POSTs.

Here is the whole error log:
    0 CLI          - Rd ping
    0 CLI          - Wr 0 200 PONG 1213029670
   13 SessionClose - timeout
   13 StatSess     - 82.182.xx.xx 1306 0 1 2 0 0 2 744 6750
    0 CLI          - Rd ping
    0 CLI          - Wr 0 200 PONG 1213029673
   13 SessionOpen  c 82.182.xx.xx 1307 0.0.0.0:8080
   13 ReqStart     c 82.182.xx.xx 1307 803913450
   13 RxRequest    c POST
   13 RxURL        c /takelogin.php
   13 RxProtocol   c HTTP/1.1
   13 RxHeader     c User-Agent: Opera/9.27 (Windows NT 5.1; U; sv)
   13 RxHeader     c Host: mydomain.com:8080
   13 RxHeader     c Accept: text/html, application/xml;q=0.9, 
application/xhtml+xml, image/png, image/jpeg, image/gif, 
image/x-xbitmap, */*;q=0.1
   13 RxHeader     c Accept-Language: sv-SE,sv;q=0.9,en;q=0.8
   13 RxHeader     c Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1
   13 RxHeader     c Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
   13 RxHeader     c Referer: http://mydomain.com:8080/login.php
   13 RxHeader     c Cookie: PHPSESSID=f3a60f178adede632c761dca745054cf
   13 RxHeader     c Cookie2: $Version=1
   13 RxHeader     c Connection: Keep-Alive, TE
   13 RxHeader     c TE: deflate, gzip, chunked, identity, trailers
   13 RxHeader     c Content-Length: 63
   13 RxHeader     c Content-Type: application/x-www-form-urlencoded
   13 VCL_call     c recv
   13 VCL_return   c pass
   13 VCL_call     c pass
   13 VCL_return   c pass
   14 TxRequest    - POST
   14 TxURL        - /takelogin.php
   14 TxProtocol   - HTTP/1.1
   14 TxHeader     - User-Agent: Opera/9.27 (Windows NT 5.1; U; sv)
   14 TxHeader     - Host: mydomain.com:8080
   14 TxHeader     - Accept: text/html, application/xml;q=0.9, 
application/xhtml+xml, image/png, image/jpeg, image/gif, 
image/x-xbitmap, */*;q=0.1
   14 TxHeader     - Accept-Language: sv-SE,sv;q=0.9,en;q=0.8
   14 TxHeader     - Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1
   14 TxHeader     - Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
   14 TxHeader     - Referer: http://mydomain.com:8080/login.php
   14 TxHeader     - Cookie: PHPSESSID=f3a60f178adede632c761dca745054cf
   14 TxHeader     - Cookie2: $Version=1
   14 TxHeader     - Content-Type: application/x-www-form-urlencoded
   14 TxHeader     - X-Varnish: 803913450
   14 TxHeader     - X-Forwarded-For: 82.182.xx.xx
   14 RxProtocol   - HTTP/1.1
   14 RxStatus     - 200
   14 RxResponse   - OK
   14 RxHeader     - Date: Mon, 09 Jun 2008 16:41:14 GMT
   14 RxHeader     - Server: Apache
   14 RxHeader     - X-Powered-By: PHP/5.2.0-8+etch10
   14 RxHeader     - Content-Length: 23
   14 RxHeader     - Content-Type: text/html; charset=UTF-8
   13 ObjProtocol  c HTTP/1.1
   13 ObjStatus    c 200
   13 ObjResponse  c OK
   13 ObjHeader    c Date: Mon, 09 Jun 2008 16:41:14 GMT
   13 ObjHeader    c Server: Apache
   13 ObjHeader    c X-Powered-By: PHP/5.2.0-8+etch10
   13 ObjHeader    c Content-Type: text/html; charset=UTF-8
   14 BackendReuse - mydomain
   13 TTL          c 803913450 RFC 120 1213029674 1213029674 0 0 0
   13 VCL_call     c fetch
   13 VCL_return   c pass
   13 Length       c 23
   13 VCL_call     c deliver
   13 VCL_return   c deliver
   13 TxProtocol   c HTTP/1.1
   13 TxStatus     c 200
   13 TxResponse   c OK
   13 TxHeader     c Server: Apache
   13 TxHeader     c X-Powered-By: PHP/5.2.0-8+etch10
   13 TxHeader     c Content-Type: text/html; charset=UTF-8
   13 TxHeader     c Content-Length: 23
   13 TxHeader     c Date: Mon, 09 Jun 2008 16:41:14 GMT
   13 TxHeader     c X-Varnish: 803913450
   13 TxHeader     c Age: 0
   13 TxHeader     c Via: 1.1 varnish
   13 TxHeader     c Connection: keep-alive
   13 ReqEnd       c 803913450 1213029674.708000898 1213029674.780754805 
0.003840923 0.072657347 0.000096560
    0 StatAddr     - 82.182.xx.xx 0 76 14 47 0 0 30 12274 157175


# varnishd -V
varnishd (varnish-trunk)
Copyright (c) 2006-2008 Linpro AS / Verdens Gang AS

trunk updated 080609 20:00

/ Duja


duja at torlen.net skrev:
> Hi,
>
> Im wondering if anyone have seen some strange behaviour from Opera (9.27) when doing a POST through varnish?
>
> >From my debugging:
> Opera doesn't send any POST data when doing it through varnish . 
> Opera sends a Cookie2 header - Cookie2: $version=1
>
> In this moment I dont have so much info from the debugging. I will reply as soon as I have some more to share.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
>   



From phk at phk.freebsd.dk  Mon Jun  9 19:07:07 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 09 Jun 2008 19:07:07 +0000
Subject: Strange Opera error in POST 
In-Reply-To: Your message of "Mon, 09 Jun 2008 21:02:02 +0200."
	<484D7E2A.3060804@torlen.net> 
Message-ID: <21808.1213038427@critter.freebsd.dk>

In message <484D7E2A.3060804 at torlen.net>, Erik Torlen writes:

>My vcl has this code when POST are received:
>
>        if(req.request != "GET" && req.request != "HEAD") {
>                set req.http.Connection = "close";
>                pass;
>        }
>
>This code usually works with FF and IE but NOT with Opera.
>
>If I remove "set req.http.Connection = "close";"
>
>the login process works with no problem.
>
>I have had problems with POSTs before, thats why I've been using 
>Connection = "close" on POSTs.

I'm not sure I can say much here...

The "close" trick is mostly, if not only, relevant for "pipe" mode,
where it prevents more than the first request from the client from
being piped.  It doesn't really do anything positive for "pass".

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From duja at torlen.net  Mon Jun  9 19:15:18 2008
From: duja at torlen.net (Erik Torlen)
Date: Mon, 09 Jun 2008 21:15:18 +0200
Subject: Strange Opera error in POST
In-Reply-To: <21808.1213038427@critter.freebsd.dk>
References: <21808.1213038427@critter.freebsd.dk>
Message-ID: <484D8146.6000202@torlen.net>

Yes, Im aware of that. But I did have some problem in an earlier version 
of trunk (and especially 1.1.2), but with the latest it seems better.

I have seen alot of updates on trunk the latest days, what's the status 
of it, is it stable for production use?

/ Duja

Poul-Henning Kamp skrev:
> In message <484D7E2A.3060804 at torlen.net>, Erik Torlen writes:
>
>   
>> My vcl has this code when POST are received:
>>
>>        if(req.request != "GET" && req.request != "HEAD") {
>>                set req.http.Connection = "close";
>>                pass;
>>        }
>>
>> This code usually works with FF and IE but NOT with Opera.
>>
>> If I remove "set req.http.Connection = "close";"
>>
>> the login process works with no problem.
>>
>> I have had problems with POSTs before, thats why I've been using 
>> Connection = "close" on POSTs.
>>     
>
> I'm not sure I can say much here...
>
> The "close" trick is mostly, if not only, relevant for "pipe" mode,
> where it prevents more than the first request from the client from
> being piped.  It doesn't really do anything positive for "pass".
>
>   



From duja at torlen.net  Mon Jun  9 19:25:58 2008
From: duja at torlen.net (Erik Torlen)
Date: Mon, 09 Jun 2008 21:25:58 +0200
Subject: Strange Opera error in POST
In-Reply-To: <21808.1213038427@critter.freebsd.dk>
References: <21808.1213038427@critter.freebsd.dk>
Message-ID: <484D83C6.8030301@torlen.net>

What I should mention is that the problem can be solved by another method.

If I remove the Cookie2 AND TE header it all works out fine:

       remove req.http.Cookie2;
       remove req.http.TE;

        if(req.request != "GET" && req.request != "HEAD") {
                set req.http.Connection = "close";
                pass;
        }

This works!

But I must remove BOTH Cookie2 and TE header, else it wont work.

/ Duja

Poul-Henning Kamp skrev:
> In message <484D7E2A.3060804 at torlen.net>, Erik Torlen writes:
>
>   
>> My vcl has this code when POST are received:
>>
>>        if(req.request != "GET" && req.request != "HEAD") {
>>                set req.http.Connection = "close";
>>                pass;
>>        }
>>
>> This code usually works with FF and IE but NOT with Opera.
>>
>> If I remove "set req.http.Connection = "close";"
>>
>> the login process works with no problem.
>>
>> I have had problems with POSTs before, thats why I've been using 
>> Connection = "close" on POSTs.
>>     
>
> I'm not sure I can say much here...
>
> The "close" trick is mostly, if not only, relevant for "pipe" mode,
> where it prevents more than the first request from the client from
> being piped.  It doesn't really do anything positive for "pass".
>
>   



From phk at phk.freebsd.dk  Mon Jun  9 20:14:07 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 09 Jun 2008 20:14:07 +0000
Subject: Strange Opera error in POST 
In-Reply-To: Your message of "Mon, 09 Jun 2008 21:25:58 +0200."
	<484D83C6.8030301@torlen.net> 
Message-ID: <22479.1213042447@critter.freebsd.dk>


Can I get you to record a varnish log where you do that and one 
where you don't, and then open a ticket with this issue ?

Poul-Henning

In message <484D83C6.8030301 at torlen.net>, Erik Torlen writes:
>What I should mention is that the problem can be solved by another method.
>
>If I remove the Cookie2 AND TE header it all works out fine:
>
>       remove req.http.Cookie2;
>       remove req.http.TE;
>
>        if(req.request != "GET" && req.request != "HEAD") {
>                set req.http.Connection = "close";
>                pass;
>        }
>
>This works!
>
>But I must remove BOTH Cookie2 and TE header, else it wont work.
>
>/ Duja
>
>Poul-Henning Kamp skrev:
>> In message <484D7E2A.3060804 at torlen.net>, Erik Torlen writes:
>>
>>   
>>> My vcl has this code when POST are received:
>>>
>>>        if(req.request != "GET" && req.request != "HEAD") {
>>>                set req.http.Connection = "close";
>>>                pass;
>>>        }
>>>
>>> This code usually works with FF and IE but NOT with Opera.
>>>
>>> If I remove "set req.http.Connection = "close";"
>>>
>>> the login process works with no problem.
>>>
>>> I have had problems with POSTs before, thats why I've been using 
>>> Connection = "close" on POSTs.
>>>     
>>
>> I'm not sure I can say much here...
>>
>> The "close" trick is mostly, if not only, relevant for "pipe" mode,
>> where it prevents more than the first request from the client from
>> being piped.  It doesn't really do anything positive for "pass".
>>
>>   
>

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From duja at torlen.net  Tue Jun 10 07:01:50 2008
From: duja at torlen.net (duja at torlen.net)
Date: Tue, 10 Jun 2008 09:01:50 +0200
Subject: Strange Opera error in POST
Message-ID: <uo3imyy4x4s62ja.100620080901@torlen.net>

Sure, will do that as soon as possible.

Duja

Original Message -----------------------

Can I get you to record a varnish log where you do that and one 
where you don't, and then open a ticket with this issue ?

Poul-Henning



From oliver.oli+0815 at gmail.com  Thu Jun 12 09:49:43 2008
From: oliver.oli+0815 at gmail.com (Oliver Oli)
Date: Thu, 12 Jun 2008 11:49:43 +0200
Subject: Flash Range header workaround?
Message-ID: <66f535320806120249w41a9097v816710e17ceb572a@mail.gmail.com>

Hello,

I haven't found any user manual on the Varnish website... it seems to
me that varnish is quite flexible, but i don't know where to start.
Any pointer to some documentation / tutorial is welcome.

I have a simple use case:
I would like to request a partial document from a Flash movie [1].
Unfortunatly the "Range" header is blocked by flash. I wonder if I can
use Varnish for a work-around. like rewriting

http://varnish.localhost/mysong.mp3?range=10000-30000

to

http://backend.localhost/mysong.mp3
and a "Range: bytes=10000-30000" header

or rewriting some custom "X-Oli-Range: " header to "Range: "


Is Varnish able to cache partial documents?

- Oli



[1] http://developer.amazonwebservices.com/connect/thread.jspa?threadID=21377&tstart=0


From oliver.oli+0815 at gmail.com  Sun Jun 15 20:28:37 2008
From: oliver.oli+0815 at gmail.com (Oliver Oli)
Date: Sun, 15 Jun 2008 22:28:37 +0200
Subject: Using Range requests with cached files
Message-ID: <66f535320806151328i14d1413eye00099634552cbdb@mail.gmail.com>

Hello,

how does varnish handle Range request? I'm getting varying results.
Sometimes it seems that a Range request is delivered from the cache,
sometimes it seems that varnish is receiving the whole file from the
backend first and sometimes it serves the whole file completely
ignoring the Range header. I cannot see any pattern in varnish's
behaviour.

Does vernish support the Range header at all?

I'm using the default.vcl from the debian package (built from the 1.2
branch) with "set obj.ttl=30d" in sub vcl_fetch.


From phk at phk.freebsd.dk  Sun Jun 15 21:14:16 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sun, 15 Jun 2008 21:14:16 +0000
Subject: Using Range requests with cached files 
In-Reply-To: Your message of "Sun, 15 Jun 2008 22:28:37 +0200."
	<66f535320806151328i14d1413eye00099634552cbdb@mail.gmail.com> 
Message-ID: <11875.1213564456@critter.freebsd.dk>

In message <66f535320806151328i14d1413eye00099634552cbdb at mail.gmail.com>, "Oliv
er Oli" writes:
>Hello,
>
>how does varnish handle Range request?

"not at all"

I have put it on the "things to do after 2.0" wiki page.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From oliver.oli+0815 at gmail.com  Sun Jun 15 22:12:51 2008
From: oliver.oli+0815 at gmail.com (Oliver Oli)
Date: Mon, 16 Jun 2008 00:12:51 +0200
Subject: Using Range requests with cached files
In-Reply-To: <11875.1213564456@critter.freebsd.dk>
References: <66f535320806151328i14d1413eye00099634552cbdb@mail.gmail.com>
	<11875.1213564456@critter.freebsd.dk>
Message-ID: <66f535320806151512o5f4e9532q19539c2f610f6719@mail.gmail.com>

Hello Poul-Henning,

good to know. Is there a way to cache partial responses anyway
(ignoring the fact that many requests for differing ranges would blow
up the cache)? I tried to add req.http.Range to req.hash, but I'm
oviously missing something, because it's not cached.

sub vcl_hash {
        set req.hash += req.url;
        set req.hash += req.http.Range;
        set req.hash += req.http.host;
        hash;
}




On Sun, Jun 15, 2008 at 11:14 PM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> In message <66f535320806151328i14d1413eye00099634552cbdb at mail.gmail.com>, "Oliv
> er Oli" writes:
>>Hello,
>>
>>how does varnish handle Range request?
>
> "not at all"
>
> I have put it on the "things to do after 2.0" wiki page.


From perbu at linpro.no  Mon Jun 16 12:02:32 2008
From: perbu at linpro.no (Per Buer)
Date: Mon, 16 Jun 2008 14:02:32 +0200
Subject: Using Range requests with cached files
In-Reply-To: <11875.1213564456@critter.freebsd.dk>
References: <11875.1213564456@critter.freebsd.dk>
Message-ID: <48565658.6040500@linpro.no>

Poul-Henning Kamp skrev:
> In message <66f535320806151328i14d1413eye00099634552cbdb at mail.gmail.com>, "Oliv
> er Oli" writes:
>> Hello,
>>
>> how does varnish handle Range request?
> 
> "not at all"

Wouldn't range requests "just work" with a proper implementation of Vary:?



Per.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080616/dd12360e/attachment.bin>

From phk at phk.freebsd.dk  Mon Jun 16 12:06:16 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 16 Jun 2008 12:06:16 +0000
Subject: Using Range requests with cached files 
In-Reply-To: Your message of "Mon, 16 Jun 2008 14:02:32 +0200."
	<48565658.6040500@linpro.no> 
Message-ID: <18868.1213617976@critter.freebsd.dk>

In message <48565658.6040500 at linpro.no>, Per Buer writes:

>>> Hello,
>>>
>>> how does varnish handle Range request?
>> 
>> "not at all"
>
>Wouldn't range requests "just work" with a proper implementation of Vary: ?

No, Range is orthogonal to Vary (which we fully support for quite some
time already).

It's not a big deal to support Range, it just hasn't made the 2.0 list.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From karl at vomba.com  Mon Jun 16 11:40:59 2008
From: karl at vomba.com (Karl Bernard)
Date: Mon, 16 Jun 2008 07:40:59 -0400
Subject: Varnishstat in trunk
Message-ID: <4856514B.8020308@vomba.com>


I wanted to test the trunk version, but I am unable to compile 
varnishstat and varnishtop.

I have no compilation error, they are simply not there after my make and 
make install.

I also seem unable to get varnishadm working:

[root at cdn-web08 varnish-cache]# varnishadm -T 6082 help
connect(): Invalid argument
An error occured in receiving status.

I must be doing something wrong, any body can make suggetsion?

Karl


From tfheen at linpro.no  Mon Jun 16 13:34:07 2008
From: tfheen at linpro.no (Tollef Fog Heen)
Date: Mon, 16 Jun 2008 15:34:07 +0200
Subject: Varnishstat in trunk
In-Reply-To: <4856514B.8020308@vomba.com> (Karl Bernard's message of "Mon\,
	16 Jun 2008 07\:40\:59 -0400")
References: <4856514B.8020308@vomba.com>
Message-ID: <87lk15wlkg.fsf@luxevop.linpro.no>

* Karl Bernard 

| I must be doing something wrong, any body can make suggetsion?

Do you have the ncurses header files installed?  libncurses5-dev is
the package name on Ubuntu and Debian.

-- 
Tollef Fog Heen / Linpro AS    t: 21 54 41 73

UNIX is user friendly, it's just picky about who its friends are


From andrearo at pvv.ntnu.no  Mon Jun 16 18:00:07 2008
From: andrearo at pvv.ntnu.no (Andreas R.)
Date: Mon, 16 Jun 2008 20:00:07 +0200 (CEST)
Subject: Pushing vcls failed: No VCL_conf symbol
Message-ID: <Pine.BSF.4.60.0806161956040.30853@bacchus.pvv.ntnu.no>

Hello!

Im trying to run varnish 1.1 installed as rpm for OpenSuSE.
However, varnish never starts, and I get the following
output when running varnish from the command line with debugging:



$ /usr/sbin/varnishd -d -f /etc/varnish/vcl.conf -T127.0.0.1:6082 -s 
file,/var/cache/varnish,524288

file /var/cache/varnish/varnish.3znroi (unlinked) size 524288 bytes (128 
fs-blocks, 128 pages)
Using old SHMFILE
New Pid 11616

rolling(1)...
rolling(2)...

CLI <>
200 0

start
CLI <start>
start child pid 11617
Pushing vcls failed:
No VCL_conf symbol

Child said (1, 11617): <<Child starts
managed to mmap 524288 bytes of 524288
Ready
CLI ready
>>
unlink ./bin.XXv3b25u
k 1 rev 16



From phk at phk.freebsd.dk  Mon Jun 16 22:30:42 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 16 Jun 2008 22:30:42 +0000
Subject: "varnishtest" program
Message-ID: <37440.1213655442@critter.freebsd.dk>


As those of you who follow the commit list will have seen, I have
spent a couple of days rewriting my internal work-tool and committed
it under the name "varnishtest".

Varnishtest is a script driven program which can set up a carefully
choreographed dance between some threads acting as backends, some
threads acting as clients and a varnish process.

varnishtest is not complete yet, I need to implement the ability
to flip VCL programs and examine statistics, but already now you
can do quite advanced testing with just the basic features which
are there.

If anybody have a sadistic streak in them, I'll be very happy to
receive scripts that show how varnish is broken :-)

Attached below a quick tour of a varnishtest run.

Poul-Henning


Here is the first script that managed to get a request through a
varnish instance, with comments:

	# Start a varnish instance "v1"
	varnish v1 -arg "-b localhost:9080" -start

	# Create a server thread
	server s1 {
		# Receive a request
		rxreq 
		# Send a standard response
		txresp -hdr "Connection: close" -body "012345\n"
	}

	# Start the server thread
	server s1 -start 

	# Create a client thread
	client c1 {
		# Send a request
		txreq -url "/"
		# Wait for a response
		rxresp
		# Insist that it be a success
		expect resp.status == 200
	}

	# Run the client
	client c1 -run

	# Wait for the server to die
	server s1 -wait

	# (Forcefully) Stop the varnish instance.
	varnish v1 -stop

The output, running this script looks as follows.

The "bargraph" at the beginning of the line is an indication of the
level of detail in the line.

The second field where the message comes from

The rest of the line is anyones guess :-)

#    TEST tests/b00000.vtc starting
###  v1   CMD: cd ../varnishd && ./varnishd -d -d -n v1 -a :9081 -T :9001 -b localhost:9080
###  v1   opening CLI connection
#### v1   debug| NB: Storage size limited to 2GB on 32 bit architecture,\n
#### v1   debug| NB: otherwise we could run out of address space.\n
#### v1   debug| storage_file: filename: ./varnish.Shkoq5 (unlinked) size 2047 MB.\n
###  v1   CLI connection fd = 3
#### v1   CLI TX| start
#### v1   debug| Using old SHMFILE\n
#### v1   debug| Notice: locking SHMFILE in core failed: Operation not permitted\n
#### v1   debug| bind(): Address already in use\n
#### v1   debug| rolling(1)...
#### v1   debug| \n
#### v1   debug| rolling(2)...\n
#### v1   debug| Debugging mode, enter "start" to start child\n
###  v1   CLI 200 <start>
##   s1   Starting server
###  s1   listen on :9080 (fd 6)
##   c1   Starting client
##   c1   Waiting for client
##   s1   started on :9080
##   c1   started
###  c1   connect to :9081
###  c1   connected to :9081 fd is 8
#### c1   | GET / HTTP/1.1\r\n
#### c1   | \r\n
###  c1   rxresp
#### s1   Accepted socket 7
###  s1   rxreq
#### s1   | GET / HTTP/1.1\r\n
#### s1   | X-Varnish: 422080121\r\n
#### s1   | X-Forwarded-For: 127.0.0.1\r\n
#### s1   | Host: localhost\r\n
#### s1   | \r\n
#### s1   http[ 0] | GET
#### s1   http[ 1] | /
#### s1   http[ 2] | HTTP/1.1
#### s1   http[ 3] | X-Varnish: 422080121
#### s1   http[ 4] | X-Forwarded-For: 127.0.0.1
#### s1   http[ 5] | Host: localhost
#### s1   | HTTP/1.1 200 Ok\r\n
#### s1   | Connection: close\r\n
#### s1   | \r\n
#### s1   | 012345\n
#### s1   | \r\n
##   s1   ending
#### c1   | HTTP/1.1 200 Ok\r\n
#### c1   | Content-Length: 9\r\n
#### c1   | Date: Mon, 16 Jun 2008 22:16:55 GMT\r\n
#### c1   | X-Varnish: 422080121\r\n
#### c1   | Age: 0\r\n
#### c1   | Via: 1.1 varnish\r\n
#### c1   | Connection: keep-alive\r\n
#### c1   | \r\n
#### c1   http[ 0] | HTTP/1.1
#### c1   http[ 1] | 200
#### c1   http[ 2] | Ok
#### c1   http[ 3] | Content-Length: 9
#### c1   http[ 4] | Date: Mon, 16 Jun 2008 22:16:55 GMT
#### c1   http[ 5] | X-Varnish: 422080121
#### c1   http[ 6] | Age: 0
#### c1   http[ 7] | Via: 1.1 varnish
#### c1   http[ 8] | Connection: keep-alive
#### c1   EXPECT resp.status (200) == 200 (200) match
##   c1   ending
##   s1   Waiting for server
#### v1   CLI TX| stop
###  v1   CLI 200 <stop>
#    TEST tests/b00000.vtc completed

If instead of 200 we had expected 201 with the line:
	expect resp.status == 201

The output would have ended with:

#### c1   http[ 0] | HTTP/1.1
#### c1   http[ 1] | 200
#### c1   http[ 2] | Ok
#### c1   http[ 3] | Content-Length: 9
#### c1   http[ 4] | Date: Mon, 16 Jun 2008 22:26:35 GMT
#### c1   http[ 5] | X-Varnish: 648043653 648043652
#### c1   http[ 6] | Age: 6
#### c1   http[ 7] | Via: 1.1 varnish
#### c1   http[ 8] | Connection: keep-alive
---- c1   EXPECT resp.status (200) == 201 (201) failed


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Mon Jun 16 22:33:57 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 16 Jun 2008 22:33:57 +0000
Subject: "varnishtest" program 
In-Reply-To: Your message of "Mon, 16 Jun 2008 22:30:42 GMT."
	<37440.1213655442@critter.freebsd.dk> 
Message-ID: <37490.1213655637@critter.freebsd.dk>

In message <37440.1213655442 at critter.freebsd.dk>, Poul-Henning Kamp writes:

>Varnishtest is a script driven program which can set up a carefully
>choreographed dance between some threads acting as backends, some
>threads acting as clients and a varnish process.
>
>If anybody have a sadistic streak in them, I'll be very happy to
>receive scripts that show how varnish is broken :-)

And of course: tickets with varnishtest scripts in the to reproduce
the problem earn brownie points :-)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From duja at torlen.net  Tue Jun 17 10:37:08 2008
From: duja at torlen.net (duja at torlen.net)
Date: Tue, 17 Jun 2008 12:37:08 +0200
Subject: Question about threads
Message-ID: <nag4kt1wjkwwpwd.170620081237@torlen.net>

I recently made a loadtest against through varnish. 

First I received a very high response time and found out that varnish was maxing the maximum nr of threads. 

I updated thread_min = 5 and thread_max = 300 and recevied much better resp. times.

Then I increased the nr of concurrent users and made another loadtest. The strange thing here was that I received high resp. times but the threads stopped at 238.

The "N worker threads not created" increased rapidly.

I increased the threads again and changed listen_depth to 2048.

Here is all the numbers:
         238         0.00         0.22 N worker threads created
        1318         4.98         1.21 N worker threads not created

    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"

default_ttl                120 [seconds]
thread_pools               2 [pools] <<<<
thread_pool_max            400 [threads] <<<<
thread_pool_min            10 [threads] <<<<
thread_pool_timeout        120 [seconds]
thread_pool_purge_delay    1000 [milliseconds]
thread_pool_add_threshold  2 [requests]
thread_pool_add_delay      10 [milliseconds]
thread_pool_fail_delay     200 [milliseconds]
overflow_max               100 [%]
rush_exponent              3 [requests per request]
sess_workspace             8192 [bytes]
obj_workspace              8192 [bytes]
sess_timeout               5 [seconds]
pipe_timeout               60 [seconds]
send_timeout               600 [seconds]
auto_restart               on [bool]
fetch_chunksize            128 [kilobytes]
vcl_trace                  off [bool]
listen_address             0.0.0.0:80
listen_depth               2048 [connections] <<<<
srcaddr_hash               1049 [buckets]
srcaddr_ttl                30 [seconds]
backend_http11             off [bool]
client_http11              off [bool]
cli_timeout                5 [seconds]
ping_interval              3 [seconds]
lru_interval               2 [seconds]
cc_command                 exec cc -fpic -shared -Wl,-x -o %o %s
max_restarts               4 [restarts]
max_esi_includes           5 [restarts]
cache_vbe_conns            off [bool]
connect_timeout            400 [ms]
cli_buffer                 8192 [bytes]
diag_bitmap                0x0 [bitmap]

Why do I get "Create worker thread failed 12 Cannot allocate memory" when I had 1900MB free RAM and 65GB free Disk on the server? Any ideas?

If "N worker threads not created" is increasing, is that a bad sign?

Thanks
Duja



From lapo at lapo.it  Tue Jun 17 12:11:56 2008
From: lapo at lapo.it (Lapo Luchini)
Date: Tue, 17 Jun 2008 14:11:56 +0200
Subject: honoring browser reload request
Message-ID: <g389mc$6rc$1@ger.gmane.org>

Using latest varnish from FreeBSD ports (version 1.1.2), is the 
following VCL the "correct" solution to support browser shift-reloads to 
get fresh content and actually update the cache?

sub vcl_hit {
     if (req.http.Cache-Control ~ "no-cache") {
         set obj.ttl = 0s;
         pass;
     }
}

As far as I understood vcl(7), this means:
1. fetch the object from cache (implicit in the fact that we're inside 
vcl_hit)
2. set ttl to 0s, expiring it (so next requests will fetch new content)
3. pass to backend (so *this* request see new content)

Does this do what I expect it to do?
Is there a way to avoid hitting the backend twice?
Is there a way to do that in vcl_recv?
(I guess not, since there's no "obj" object available yet, there)
I guess I could use purge_url(...), except I have virtual hosts and it 
wants a regexp and req.url could contain some special char (could it?).

-- 
Lapo Luchini - http://lapo.it/



From michael at dynamine.net  Tue Jun 17 20:17:52 2008
From: michael at dynamine.net (Michael S. Fischer)
Date: Tue, 17 Jun 2008 13:17:52 -0700
Subject: Question about threads
In-Reply-To: <nag4kt1wjkwwpwd.170620081237@torlen.net>
References: <nag4kt1wjkwwpwd.170620081237@torlen.net>
Message-ID: <86db848d0806171317p679a2eb7v6541b773a0fdb8b5@mail.gmail.com>

Raising the number of threads will not significantly improve Varnish
concurrency in most cases.  I did a test a few months ago using 4 CPUs on
RHEL 4.6 with very high request concurrency and a very low
request-per-connection ratio (i.e., 1:1, no keepalives) and found that the
magic number is about 16 threads/CPU.  You should raise overflow_max to a
very high value (10000% worked just fine for us).

Under optimal operating conditions you should not see the "threads not
created" value increasing like this.

Best regards,

--Michael

On Tue, Jun 17, 2008 at 3:37 AM, <duja at torlen.net> wrote:

> I recently made a loadtest against through varnish.
>
> First I received a very high response time and found out that varnish was
> maxing the maximum nr of threads.
>
> I updated thread_min = 5 and thread_max = 300 and recevied much better
> resp. times.
>
> Then I increased the nr of concurrent users and made another loadtest. The
> strange thing here was that I received high resp. times but the threads
> stopped at 238.
>
> The "N worker threads not created" increased rapidly.
>
> I increased the threads again and changed listen_depth to 2048.
>
> Here is all the numbers:
>         238         0.00         0.22 N worker threads created
>        1318         4.98         1.21 N worker threads not created
>
>    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
>    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
>    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
>    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
>    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
>    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
>    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
>    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
>
> default_ttl                120 [seconds]
> thread_pools               2 [pools] <<<<
> thread_pool_max            400 [threads] <<<<
> thread_pool_min            10 [threads] <<<<
> thread_pool_timeout        120 [seconds]
> thread_pool_purge_delay    1000 [milliseconds]
> thread_pool_add_threshold  2 [requests]
> thread_pool_add_delay      10 [milliseconds]
> thread_pool_fail_delay     200 [milliseconds]
> overflow_max               100 [%]
> rush_exponent              3 [requests per request]
> sess_workspace             8192 [bytes]
> obj_workspace              8192 [bytes]
> sess_timeout               5 [seconds]
> pipe_timeout               60 [seconds]
> send_timeout               600 [seconds]
> auto_restart               on [bool]
> fetch_chunksize            128 [kilobytes]
> vcl_trace                  off [bool]
> listen_address             0.0.0.0:80
> listen_depth               2048 [connections] <<<<
> srcaddr_hash               1049 [buckets]
> srcaddr_ttl                30 [seconds]
> backend_http11             off [bool]
> client_http11              off [bool]
> cli_timeout                5 [seconds]
> ping_interval              3 [seconds]
> lru_interval               2 [seconds]
> cc_command                 exec cc -fpic -shared -Wl,-x -o %o %s
> max_restarts               4 [restarts]
> max_esi_includes           5 [restarts]
> cache_vbe_conns            off [bool]
> connect_timeout            400 [ms]
> cli_buffer                 8192 [bytes]
> diag_bitmap                0x0 [bitmap]
>
> Why do I get "Create worker thread failed 12 Cannot allocate memory" when I
> had 1900MB free RAM and 65GB free Disk on the server? Any ideas?
>
> If "N worker threads not created" is increasing, is that a bad sign?
>
> Thanks
> Duja
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080617/1e8e96c5/attachment.html>

From morfoh at opensde.org  Wed Jun 18 00:48:10 2008
From: morfoh at opensde.org (Christian Wiese)
Date: Wed, 18 Jun 2008 02:48:10 +0200
Subject: Strange browser hickups with varnish
Message-ID: <20080618024810.3d58b767@opensde.org>

Hi Folks,

we're currently evaluating varnish using it to cache all sorts of static
content. It mostly works with minimal configuration, but we are
experiencing really strange browser hangs.

First, the site loads without problem. Then, after clicking a bit
around, the browser (tested: firefox 1.5, 2.0, IE6) hangs while loading
a small JavaScript file. This happens regardless if the script comes
from the cache or not. Once the browser has been into this state, you
can't get it out of it. No reload, no shift-reload, nothing.

However, restarting it works. For a while...

The funny thing is that this seems to be a client thing. When I have a
browser hanging, I can fetch the same file with curl or wget just fine.

Any idea what yould cause that?

Thanks in advance for any suggestion.

Cheers,
Chris


From jt at endpoint.com  Wed Jun 18 04:28:50 2008
From: jt at endpoint.com (JT Justman)
Date: Tue, 17 Jun 2008 21:28:50 -0700
Subject: Strange browser hickups with varnish
In-Reply-To: <20080618024810.3d58b767@opensde.org>
References: <20080618024810.3d58b767@opensde.org>
Message-ID: <48588F02.10706@endpoint.com>

Christian Wiese wrote:
> The funny thing is that this seems to be a client thing. When I have a
> browser hanging, I can fetch the same file with curl or wget just fine.
>

This could be due to Varnish having different versions in different 
encodings (gzip/deflate/etc). Take a look at the encoding and see if 
this is the cause of differences between browsers.

As to the larger issue, I don't have much idea, but I think you should 
post more information, at least your varnishlog output.

JT

-- 
End Point Corporation
http://www.endpoint.com


From kradziszewski at gmail.com  Wed Jun 18 06:30:15 2008
From: kradziszewski at gmail.com (Kamil Radziszewski)
Date: Wed, 18 Jun 2008 08:30:15 +0200
Subject: Rewrite 
Message-ID: <4858AB77.2000100@gmail.com>

Hi, how can i rewrite url in varnish like this:


http://mydomain.com/([0-9]+)/([0-9]+)/([0-9]+)/([0-9]+)\.[0-9]+\.jpg  => 
http://backend/$1/$2/$3/$4.jpg

??




From rafael.umann at terra.com.br  Wed Jun 18 11:51:00 2008
From: rafael.umann at terra.com.br (Rafael Umann)
Date: Wed, 18 Jun 2008 08:51:00 -0300
Subject: Question about threads
In-Reply-To: <nag4kt1wjkwwpwd.170620081237@torlen.net>
References: <nag4kt1wjkwwpwd.170620081237@torlen.net>
Message-ID: <20080618085100.0473890b@sup-dig.wrk.terra.com.br>


If it is a 32bits system, probably the problem is that your stack size
is 10Mb. So 238 * 10mb = ~2gb

I decreased my stack size to 512Kb. Using 1gb storage files i can now
open almost 1900 threads using all the 2gb that 32bits can alloc. So, my
Varnish makes 20000 hits/second serving clients!

##### linux

# cat /etc/security/limits.conf 
* soft nofile 131072
* hard nofile 131072
* soft stack 512
* hard stack 512


# cat /etc/sysctl.conf

# Tuning webservers kernel >= 2.6.22 - Rafael Umann 20080502
net.core.somaxconn=8192
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_fin_timeout=3
net.ipv4.tcp_keepalive_time=30
net.ipv4.tcp_max_orphans=262144
net.ipv4.tcp_max_syn_backlog=262144

net.ipv4.tcp_max_tw_buckets=1440000
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1
net.ipv4.ip_local_port_range=2048 61000
net.core.rmem_max=33554432
net.core.wmem_max=33554432

net.ipv4.tcp_syncookies=1
net.ipv4.tcp_abort_on_overflow=1

net.core.rmem_default=65536
net.core.wmem_default=65536
net.core.netdev_max_backlog=262144

net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_mem=16777216 16777216 16777216

sunrpc.tcp_slot_table_entries=128
sunrpc.udp_slot_table_entries=128

fs.file-max=5049800

vm.min_free_kbytes=204800
vm.lowmem_reserve_ratio=256 256
vm.page-cluster=15
vm.swappiness=100
vm.overcommit_memory=1

# DSR - DO NOT USE IT IF YOU DONT USE DIRECT SERVER RESPONSE!!!!!
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

# Tests with new parameters - Rafael Umann 20080613
net.ipv4.tcp_no_metrics_save=1

net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2


Try it out and let us now!

[]s,


On Tue, 17 Jun 2008 12:37:08 +0200
<duja at torlen.net> wrote:

> I recently made a loadtest against through varnish. 
> 
> First I received a very high response time and found out that varnish
> was maxing the maximum nr of threads. 
> 
> I updated thread_min = 5 and thread_max = 300 and recevied much
> better resp. times.
> 
> Then I increased the nr of concurrent users and made another
> loadtest. The strange thing here was that I received high resp. times
> but the threads stopped at 238.
> 
> The "N worker threads not created" increased rapidly.
> 
> I increased the threads again and changed listen_depth to 2048.
> 
> Here is all the numbers:
>          238         0.00         0.22 N worker threads created
>         1318         4.98         1.21 N worker threads not created
> 
>     0 Debug        - "Create worker thread failed 12 Cannot allocate
> memory" 0 Debug        - "Create worker thread failed 12 Cannot
> allocate memory" 0 Debug        - "Create worker thread failed 12
> Cannot allocate memory" 0 Debug        - "Create worker thread failed
> 12 Cannot allocate memory" 0 Debug        - "Create worker thread
> failed 12 Cannot allocate memory" 0 Debug        - "Create worker
> thread failed 12 Cannot allocate memory" 0 Debug        - "Create
> worker thread failed 12 Cannot allocate memory" 0 Debug        -
> "Create worker thread failed 12 Cannot allocate memory"
> 
> default_ttl                120 [seconds]
> thread_pools               2 [pools] <<<<
> thread_pool_max            400 [threads] <<<<
> thread_pool_min            10 [threads] <<<<
> thread_pool_timeout        120 [seconds]
> thread_pool_purge_delay    1000 [milliseconds]
> thread_pool_add_threshold  2 [requests]
> thread_pool_add_delay      10 [milliseconds]
> thread_pool_fail_delay     200 [milliseconds]
> overflow_max               100 [%]
> rush_exponent              3 [requests per request]
> sess_workspace             8192 [bytes]
> obj_workspace              8192 [bytes]
> sess_timeout               5 [seconds]
> pipe_timeout               60 [seconds]
> send_timeout               600 [seconds]
> auto_restart               on [bool]
> fetch_chunksize            128 [kilobytes]
> vcl_trace                  off [bool]
> listen_address             0.0.0.0:80
> listen_depth               2048 [connections] <<<<
> srcaddr_hash               1049 [buckets]
> srcaddr_ttl                30 [seconds]
> backend_http11             off [bool]
> client_http11              off [bool]
> cli_timeout                5 [seconds]
> ping_interval              3 [seconds]
> lru_interval               2 [seconds]
> cc_command                 exec cc -fpic -shared -Wl,-x -o %o %s
> max_restarts               4 [restarts]
> max_esi_includes           5 [restarts]
> cache_vbe_conns            off [bool]
> connect_timeout            400 [ms]
> cli_buffer                 8192 [bytes]
> diag_bitmap                0x0 [bitmap]
> 
> Why do I get "Create worker thread failed 12 Cannot allocate memory"
> when I had 1900MB free RAM and 65GB free Disk on the server? Any
> ideas?
> 
> If "N worker threads not created" is increasing, is that a bad sign?
> 
> Thanks
> Duja
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
> 
> E-mail verificado pelo Terra Anti-Spam.
> Para classificar como spam ou n?o spam, visite
> http://mail.terra.com.br/cgi-bin/reportspam.cgi?+_d=SCY1ODAyNDQ3I3Blcm0hdGVycmEmMSwxMjEzNjk5MDI5LjExNzYyNy4yOTM5OS50cmlidW5lLnRlcnJhLmNvbSw1OTIz
> Verifique periodicamente a pasta Spam para garantir que apenas
> mensagens indesejadas sejam classificadas como Spam.
> 
> Esta mensagem foi verificada pelo E-mail Protegido Terra.
> Atualizado em 17/06/2008
> 


-- 
Rafael Umann <rafael.umann at corp.terra.com.br>
Suporte Engenharia 1
Terra Networks Brasil S/A 
Tel: 55 (51) 3284-4344


From phk at phk.freebsd.dk  Wed Jun 18 13:21:20 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 18 Jun 2008 13:21:20 +0000
Subject: Rewrite 
In-Reply-To: Your message of "Wed, 18 Jun 2008 08:30:15 +0200."
	<4858AB77.2000100@gmail.com> 
Message-ID: <83817.1213795280@critter.freebsd.dk>

In message <4858AB77.2000100 at gmail.com>, Kamil Radziszewski writes:
>Hi, how can i rewrite url in varnish like this:
>
>
>http://mydomain.com/([0-9]+)/([0-9]+)/([0-9]+)/([0-9]+)\.[0-9]+\.jpg  => 
>http://backend/$1/$2/$3/$4.jpg


	set req.url = regsub(req.url, $regexp, $replacement);

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From morfoh at opensde.org  Wed Jun 18 17:56:57 2008
From: morfoh at opensde.org (Christian Wiese)
Date: Wed, 18 Jun 2008 19:56:57 +0200
Subject: Strange browser hickups with varnish
In-Reply-To: <48588F02.10706@endpoint.com>
References: <20080618024810.3d58b767@opensde.org> <48588F02.10706@endpoint.com>
Message-ID: <20080618195657.3614e1cd@opensde.org>

Hi JT,

thanks a lot for your answer and sorry for that late reply!
We found the problem in the vcl:

------------------------------------snip----------------------------------
 sub vcl_recv {
-	if (req.http.Cookie) {
+	if (req.request == "GET" &&
+		req.http.Cookie &&
+		(req.url ~ "/(images|res|inc)/" ||
+		req.url ~ "/slice/" ||
+		req.url ~ "/info/(resolveplaylist|cmenu)" ||
+		req.url ~ "/info/player/player.js" ||
+		req.url ~ "/info/streamchannel/getmetadata")) {
 		lookup;
 	}
------------------------------------snip----------------------------------

We currently test our application heavily, but everything looks fine so
far!

Thanks a lot to all people that made varnish happen :)

Cheers,
Chris

Am Tue, 17 Jun 2008 21:28:50 -0700
schrieb JT Justman <jt at endpoint.com>:

> Christian Wiese wrote:
> > The funny thing is that this seems to be a client thing. When I
> > have a browser hanging, I can fetch the same file with curl or wget
> > just fine.
> >
> 
> This could be due to Varnish having different versions in different 
> encodings (gzip/deflate/etc). Take a look at the encoding and see if 
> this is the cause of differences between browsers.
> 
> As to the larger issue, I don't have much idea, but I think you
> should post more information, at least your varnishlog output.
> 
> JT
> 


From michael at dynamine.net  Wed Jun 18 20:24:52 2008
From: michael at dynamine.net (Michael S. Fischer)
Date: Wed, 18 Jun 2008 13:24:52 -0700
Subject: Question about threads
In-Reply-To: <20080618085100.0473890b@sup-dig.wrk.terra.com.br>
References: <nag4kt1wjkwwpwd.170620081237@torlen.net>
	<20080618085100.0473890b@sup-dig.wrk.terra.com.br>
Message-ID: <86db848d0806181324w731c9d91x58d39846b46196ad@mail.gmail.com>

On Wed, Jun 18, 2008 at 4:51 AM, Rafael Umann <rafael.umann at terra.com.br>
wrote:

>
> If it is a 32bits system, probably the problem is that your stack size
> is 10Mb. So 238 * 10mb = ~2gb
>
> I decreased my stack size to 512Kb. Using 1gb storage files i can now
> open almost 1900 threads using all the 2gb that 32bits can alloc. So, my
> Varnish makes 20000 hits/second serving clients!


What is your request:connection ratio?

Best regards,

--Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080618/f2fd8510/attachment.html>

From armdan20 at gmail.com  Thu Jun 19 08:41:09 2008
From: armdan20 at gmail.com (andan andan)
Date: Thu, 19 Jun 2008 10:41:09 +0200
Subject: CLI sessions and syslog
Message-ID: <c1dd32e40806190141v2ab8884du59cf753a134f84df@mail.gmail.com>

Hello.

It is possible to disable sysloging for the CLI sessions, or at least,
change the priority or facility. ?

We are using Varnish (trunk) with LVS,  pinging varnish to test if the
instance is alive, logging these sessions every 5 seconds is very
annoying.

Thanks in advance.
BR.


From rafael.umann at terra.com.br  Thu Jun 19 12:37:22 2008
From: rafael.umann at terra.com.br (Rafael Umann)
Date: Thu, 19 Jun 2008 09:37:22 -0300
Subject: Question about threads
In-Reply-To: <86db848d0806181324w731c9d91x58d39846b46196ad@mail.gmail.com>
References: <nag4kt1wjkwwpwd.170620081237@torlen.net>
	<20080618085100.0473890b@sup-dig.wrk.terra.com.br>
	<86db848d0806181324w731c9d91x58d39846b46196ad@mail.gmail.com>
Message-ID: <20080619093722.0d290417@sup-dig.wrk.terra.com.br>

On Wed, 18 Jun 2008 13:24:52 -0700
"Michael S. Fischer" <michael at dynamine.net> wrote:

> On Wed, Jun 18, 2008 at 4:51 AM, Rafael Umann
> <rafael.umann at terra.com.br> wrote:
> 
> >
> > If it is a 32bits system, probably the problem is that your stack
> > size is 10Mb. So 238 * 10mb = ~2gb
> >
> > I decreased my stack size to 512Kb. Using 1gb storage files i can
> > now open almost 1900 threads using all the 2gb that 32bits can
> > alloc. So, my Varnish makes 20000 hits/second serving clients!
> 
> 
> What is your request:connection ratio?
> 
> Best regards,
> 
> --Michael
> 

Unfortunately now i dont have servers doing 20000 hits/second, and
thats why i dont have stats for you. I have 6 servers runing this
service now, each one doing 5500 hits/sec with 10% CPU usage, and this
infrastructure scales to 20000 hits/sec each server. For test purpose
only i let just 2 servers with all the traffic, so i saw this limit i
told you.

In this 20000 hits/sec im seeing two bottle necks:

1. The 32bits arch (cant open threads and the storage file is too
small), so im moving into 64bits.
2. The cpu usage of the "listener process" with 20000 hits/sec is
getting to 100% in one CPU (we also made a patch to improve CPU usage,
and it is avaliable in this last trunk version.. without this patch
varnish was doing 8000 hits/sec). But anyway, i spoke to Poul, and he
told me that the "listener process" will be changed soon (i really hope
so, because i would love to use more than 30% of my cpu!).

Anyway, varnish is doing a great job for me!!!

Some stats taken now with 5500 hits/sec:

# netstat -nap|grep :80 |grep SYN_REC |wc -l
166

# netstat -nap|grep :80 |grep ESTAB |wc -l
17139

# netstat -nap|grep :80 |grep FIN |wc -l
1454

# netstat -nap|grep :80 |grep TIME_W |wc -l
7766


# varnishstat

Hitrate ratio:       10       13       13
Hitrate avg:     0.9990   0.9990   0.9990

    36189916       476.66       250.71 Client connections accepted
   404804204      5494.13      2804.30 Client requests received
   391586079      5335.24      2712.74 Cache hits
    10975643       157.89        76.03 Cache hits for pass
     1125709         1.00         7.80 Cache misses
    12101360       158.89        83.83 Backend connections success
           0         0.00         0.00 Backend connections failures
           0         0.00         0.00 Backend connections reuses
    11139673       141.90        77.17 Backend connections recycles
         350        -1.00         0.00 Backend connections unused
       13447          .            .   N struct srcaddr
       11353          .            .   N active struct srcaddr
       30351          .            .   N struct sess_mem
       17391          .            .   N struct sess
       47069          .            .   N struct object
       47080          .            .   N struct objecthead
       97813          .            .   N struct smf
        4185          .            .   N small free smf
           5          .            .   N large free smf
           0          .            .   N struct vbe_conn
         556          .            .   N worker threads
         556         0.00         0.00 N worker threads created
           0         0.00         0.00 N worker threads not created
           0         0.00         0.00 N worker threads limited
           0         0.00         0.00 N queued work requests
         556         0.00         0.00 N overflowed work requests
           0         0.00         0.00 N dropped work requests
           0          .            .   N expired objects
     1079159          .            .   N LRU nuked objects
           0          .            .   N LRU saved objects
           0          .            .   N objects on deathrow
          48         0.00         0.00 HTTP header overflows
           0         0.00         0.00 Objects sent with sendfile
   135051222      1812.72       935.58 Objects sent with write
    36189893       494.65       250.71 Total Sessions
   404810596      5491.13      2804.35 Total Requests
           0         0.00         0.00 Total pipe
           0         0.00         0.00 Total pass
    12101371       158.89        83.83 Total fetch
 93389657948   1269372.98    646962.32 Total header bytes
418107385660   5962840.69   2896463.38 Total body bytes
     5412491        89.94        37.50 Session Closed
           0         0.00         0.00 Session Pipeline
     1623343         0.00        11.25 Session Read Ahead
   398293975      5409.19      2759.20 Session herd
 15780035398    212996.82    109317.12 SHM records
   516154306      6307.55      3575.69 SHM writes
     2661815        33.98        18.44 SHM MTX contention
    23279696       242.83       161.27 allocator requests
       93623          .            .   outstanding allocations
   910299136          .            .   bytes allocated


[]s,
-- 
Rafael Umann <rafael.umann at corp.terra.com.br>
Suporte Engenharia 1
Terra Networks Brasil S/A 
Tel: +55 (51) 3284-4344


From max.clark at gmail.com  Thu Jun 19 14:13:35 2008
From: max.clark at gmail.com (Max Clark)
Date: Thu, 19 Jun 2008 07:13:35 -0700
Subject: Core Dump
Message-ID: <2fa1e1780806190713y1f5ab85lde938eaf914e9c64@mail.gmail.com>

I am running #2651 trunk. Can someone on the list send me a link to
debugging instructions so I can post here / on trac?

Thanks,
Max

Jun 19 11:16:48 can05 varnishd: Child (781) said <<Assert error in
STV_alloc(), stevedore.c line 71:>>
Jun 19 11:16:48 can05 kernel: pid 781 (varnishd), uid 101: exited on signal 6
Jun 19 11:16:48 can05 varnishd: Child (781) said <<  Condition((st) !=
NULL) not true.>>
Jun 19 11:16:48 can05 varnishd: Child (2934) said <<Child starts>>
Jun 19 11:16:48 can05 varnishd: Child (2934) said <<managed to mmap
1073741824 bytes of 1073741824>>
Jun 19 11:16:48 can05 varnishd: Child (2934) said <<Ready>>
Jun 19 11:16:49 can05 kernel: pid 780 (varnishd), uid 0: exited on
signal 6 (core dumped)


From michael at dynamine.net  Thu Jun 19 15:05:21 2008
From: michael at dynamine.net (Michael S. Fischer)
Date: Thu, 19 Jun 2008 08:05:21 -0700
Subject: Question about threads
In-Reply-To: <20080619093722.0d290417@sup-dig.wrk.terra.com.br>
References: <nag4kt1wjkwwpwd.170620081237@torlen.net>
	<20080618085100.0473890b@sup-dig.wrk.terra.com.br>
	<86db848d0806181324w731c9d91x58d39846b46196ad@mail.gmail.com>
	<20080619093722.0d290417@sup-dig.wrk.terra.com.br>
Message-ID: <86db848d0806190805r55ca9bwe1e22f9918bc628a@mail.gmail.com>

On Thu, Jun 19, 2008 at 5:37 AM, Rafael Umann <rafael.umann at terra.com.br>
wrote:

> > What is your request:connection ratio?
>
> Unfortunately now i dont have servers doing 20000 hits/second, and
> thats why i dont have stats for you.


Actually, it's right there in your varnishstat output:

   36189916       476.66       250.71 Client connections accepted
  404804204      5494.13      2804.30 Client requests received

Your request:connection ratio is > 10:1!  This is a very good situation to
be in.  varnish doesn't have to spawn nearly as many threads as it would if
the ratio were lower, as is common at many other sites.


> I have 6 servers runing this
> service now, each one doing 5500 hits/sec with 10% CPU usage, and this
> infrastructure scales to 20000 hits/sec each server.


It's probably inaccurate to assume that things will scale linearly :)

Best regards,

--Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080619/d96b3831/attachment.html>

From rafael.umann at terra.com.br  Thu Jun 19 15:32:32 2008
From: rafael.umann at terra.com.br (Rafael Umann)
Date: Thu, 19 Jun 2008 12:32:32 -0300
Subject: Question about threads
In-Reply-To: <86db848d0806190805r55ca9bwe1e22f9918bc628a@mail.gmail.com>
References: <nag4kt1wjkwwpwd.170620081237@torlen.net>
	<20080618085100.0473890b@sup-dig.wrk.terra.com.br>
	<86db848d0806181324w731c9d91x58d39846b46196ad@mail.gmail.com>
	<20080619093722.0d290417@sup-dig.wrk.terra.com.br>
	<86db848d0806190805r55ca9bwe1e22f9918bc628a@mail.gmail.com>
Message-ID: <20080619123232.3e0af9bb@sup-dig.wrk.terra.com.br>

On Thu, 19 Jun 2008 08:05:21 -0700
"Michael S. Fischer" <michael at dynamine.net> wrote:

> On Thu, Jun 19, 2008 at 5:37 AM, Rafael Umann
> <rafael.umann at terra.com.br> wrote:
> 
> > > What is your request:connection ratio?
> >
> > Unfortunately now i dont have servers doing 20000 hits/second, and
> > thats why i dont have stats for you.
> 
> 
> Actually, it's right there in your varnishstat output:
> 
>    36189916       476.66       250.71 Client connections accepted
>   404804204      5494.13      2804.30 Client requests received
> 
> Your request:connection ratio is > 10:1!  This is a very good
> situation to be in.  varnish doesn't have to spawn nearly as many
> threads as it would if the ratio were lower, as is common at many
> other sites.

OK, what i meant is that i dont have the stats for the amount of
hits/second that i told u. Anyway, with this amount of traffic, with
this timeouts that i set, it will probablly be the same.

> 
> > I have 6 servers runing this
> > service now, each one doing 5500 hits/sec with 10% CPU usage, and
> > this infrastructure scales to 20000 hits/sec each server.
> 
> 
> It's probably inaccurate to assume that things will scale linearly :)

Its not with Varnish :) actually i was surprised with this, but it
scales almost linearly to the 20000 hits/sec (ok, ok, with a litle bit
of response time decrease), but it dont go any further.

> 
> Best regards,
> 
> --Michael
> 
> E-mail verificado pelo Terra Anti-Spam.
> Para classificar como spam ou n??o spam, visite
> http://mail.terra.com.br/cgi-bin/reportspam.cgi?+_d=SCY1ODAyNDQ3I3Blcm0hdGVycmEmMSwxMjEzODg3OTIzLjk1ODU0MC4xNjgyMC5nYW5hbm9xdWUudGVycmEuY29tLDU0Mzc=
> Verifique periodicamente a pasta Spam para garantir que apenas
> mensagens indesejadas sejam classificadas como Spam.
> 
> Esta mensagem foi verificada pelo E-mail Protegido Terra.
> Atualizado em 19/06/2008


-- 
Rafael Umann <rafael.umann at corp.terra.com.br>
Suporte Engenharia 1
Terra Networks Brasil S/A 
Tel: 55 (51) 3284-4344


From max.clark at gmail.com  Thu Jun 19 22:48:47 2008
From: max.clark at gmail.com (Max Clark)
Date: Thu, 19 Jun 2008 15:48:47 -0700
Subject: Hostname in error pages
Message-ID: <2fa1e1780806191548g1360d4b8v1cb518eb300fe963@mail.gmail.com>

Hello,

With multiple servers behind a load balancer, how do I add the
hostname to the error page (i.e. 503 service unavailable) so we can
identify the system giving the error?

Thanks,
Max


From phk at phk.freebsd.dk  Fri Jun 20 07:12:12 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 20 Jun 2008 07:12:12 +0000
Subject: CLI sessions and syslog 
In-Reply-To: Your message of "Thu, 19 Jun 2008 10:41:09 +0200."
	<c1dd32e40806190141v2ab8884du59cf753a134f84df@mail.gmail.com> 
Message-ID: <10360.1213945932@critter.freebsd.dk>

In message <c1dd32e40806190141v2ab8884du59cf753a134f84df at mail.gmail.com>, "anda
n andan" writes:
>Hello.
>
>It is possible to disable sysloging for the CLI sessions, or at least,
>change the priority or facility. ?

Not without hacking the code, search for "syslog" in mgt_cli.c.

>We are using Varnish (trunk) with LVS,  pinging varnish to test if the
>instance is alive, logging these sessions every 5 seconds is very
>annoying.

They are sent on LOG_INFO which is just one level above LOG_DEBUG, so
they shouldn't be spit on the console or anything ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From likuku.public at gmail.com  Fri Jun 20 09:42:01 2008
From: likuku.public at gmail.com (kuku li)
Date: Fri, 20 Jun 2008 17:42:01 +0800
Subject: varnish will just restart itself as the virtual memory goes to 3G
Message-ID: <32e012170806200242m6f5a7cb0ma90434e6eb4224e6@mail.gmail.com>

Hello,

we have been running varnish for a while but noticed that varnish will just
restart itself as the virtual memory goes to 3G(from the linux top command)
and the cache hit rate consequently drop to almost 0%.  Is it a known bug or
we just missed something important?

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080620/21ef904f/attachment.html>

From phk at phk.freebsd.dk  Fri Jun 20 10:25:07 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 20 Jun 2008 10:25:07 +0000
Subject: varnish will just restart itself as the virtual memory goes to 3G
In-Reply-To: Your message of "Fri, 20 Jun 2008 17:42:01 +0800."
	<32e012170806200242m6f5a7cb0ma90434e6eb4224e6@mail.gmail.com> 
Message-ID: <16310.1213957507@critter.freebsd.dk>

In message <32e012170806200242m6f5a7cb0ma90434e6eb4224e6 at mail.gmail.com>, "kuku
 li" writes:

>we have been running varnish for a while but noticed that varnish will just
>restart itself as the virtual memory goes to 3G(from the linux top command)
>and the cache hit rate consequently drop to almost 0%.  Is it a known bug or
>we just missed something important?

Varnish will use any amount of virtual memory you give it, make sure
it has enough :-)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Fri Jun 20 12:30:49 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 20 Jun 2008 12:30:49 +0000
Subject: HEADSUP: regsub() changes in -trunk
Message-ID: <18920.1213965049@critter.freebsd.dk>


Those of you running -trunk should take note of this commit if
you use regsub() in your VCL.

I apologize for this kind of disruptive change, but the replacement
syntax was badly chosen and I needed to fix it before 2.0

Poul-Henning


----------------------------------------------------------------------------
Author: phk
Date: 2008-06-20 13:58:25 +0200 (Fri, 20 Jun 2008)
New Revision: 2741

Modified:
   trunk/varnish-cache/bin/varnishd/cache_vrt_re.c
   trunk/varnish-cache/bin/varnishtest/tests/c00001.vtc
Log:

NB: FLAGDAY!

Make an executive decision, and change the regsub() replacement specifiers to
get something which is consistent and nontroubling for URL and query strings.

Since $ and & both are heavily used in query strings, we (DES & I)
have chosen to use \0 ... \9 for replacement indicators, with \0
being the "all matched text" replacement and \1...\9 replacing
with tne N'th paranthesized subexpressions.

   regsub("_barf_", "(b)(a)(r)(f)", "\0\4\3\2\\p") -> "_barffra\p_"

----------------------------------------------------------------------------

--
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


From wichert at wiggy.net  Fri Jun 20 14:29:11 2008
From: wichert at wiggy.net (Wichert Akkerman)
Date: Fri, 20 Jun 2008 16:29:11 +0200
Subject: HEADSUP: regsub() changes in -trunk
In-Reply-To: <18920.1213965049@critter.freebsd.dk>
References: <18920.1213965049@critter.freebsd.dk>
Message-ID: <20080620142911.GA25912@wiggy.net>

Previously Poul-Henning Kamp wrote:
> Those of you running -trunk should take note of this commit if
> you use regsub() in your VCL.
> 
> I apologize for this kind of disruptive change, but the replacement
> syntax was badly chosen and I needed to fix it before 2.0

This and the changes from
http://varnish.projects.linpro.no/changeset/2739 are not reflected in
vcl(7) yet.

Do you expect the VCL to stablize with the 2.0 release? It is a bit
painful that you need a different VCL file for 1.0, 1.2, 2.0-tp1 and
2.0-after-tp1. There are tools (such as plone.recipe.varnish) that try
to gnerate a VCL and that is becoming quite painful by now.

Wichert.

-- 
Wichert Akkerman <wichert at wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.


From phk at phk.freebsd.dk  Fri Jun 20 15:29:20 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 20 Jun 2008 15:29:20 +0000
Subject: HEADSUP: regsub() changes in -trunk 
In-Reply-To: Your message of "Fri, 20 Jun 2008 16:29:11 +0200."
	<20080620142911.GA25912@wiggy.net> 
Message-ID: <29756.1213975760@critter.freebsd.dk>

In message <20080620142911.GA25912 at wiggy.net>, Wichert Akkerman writes:
>Previously Poul-Henning Kamp wrote:
>> Those of you running -trunk should take note of this commit if
>> you use regsub() in your VCL.
>> 
>> I apologize for this kind of disruptive change, but the replacement
>> syntax was badly chosen and I needed to fix it before 2.0
>
>This and the changes from
>http://varnish.projects.linpro.no/changeset/2739 are not reflected in
>vcl(7) yet.
>
>Do you expect the VCL to stablize with the 2.0 release? It is a bit
>painful that you need a different VCL file for 1.0, 1.2, 2.0-tp1 and
>2.0-after-tp1. There are tools (such as plone.recipe.varnish) that try
>to gnerate a VCL and that is becoming quite painful by now.

My intent is that VCL generally should be stable inside major branches
so the VCL in 2.0 should work on all 2.x versions.

Right now -trunk is edging really close to what 2.0 should be, and
that means a number of minor and major tweaks, such as these happen
now.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From damien.sarazin at smile.fr  Fri Jun 20 16:40:50 2008
From: damien.sarazin at smile.fr (Damien Sarazin)
Date: Fri, 20 Jun 2008 18:40:50 +0200
Subject: Little question about the releases of Varnish
Message-ID: <485BDD92.10708@smile.fr>

Hello,

i am new to Varnish and i have some questions about the differents 
versions. First i installed the 1.1.2 release with the .deb packages but 
i browsed the mail list, and i saw many answers where people where were 
advising to install a newer version from the svn.

In fact i would like to use Varnish as a reverse proxy on a backend with 
several apache virtual hosts. I also need to be able to retrieve the 
logs for each virtual host on the backend so that i can process them 
with awstats.

My question is : should i continue to work with the 1.1.2 release or is 
the 1.2 version stable enough so i can use this last one ?

Sincerely yours,

-- 
Damien S.




From michael at dynamine.net  Sat Jun 21 00:51:57 2008
From: michael at dynamine.net (Michael S. Fischer)
Date: Fri, 20 Jun 2008 17:51:57 -0700
Subject: varnish will just restart itself as the virtual memory goes to 3G
In-Reply-To: <32e012170806200242m6f5a7cb0ma90434e6eb4224e6@mail.gmail.com>
References: <32e012170806200242m6f5a7cb0ma90434e6eb4224e6@mail.gmail.com>
Message-ID: <86db848d0806201751h4ab23155r361d45764adcdaa3@mail.gmail.com>

This sounds an awful lot like "no PAE kernel" -- i.e., 32 bits and a really
old OS.

--Michael

On Fri, Jun 20, 2008 at 2:42 AM, kuku li <likuku.public at gmail.com> wrote:

> Hello,
>
> we have been running varnish for a while but noticed that varnish will just
> restart itself as the virtual memory goes to 3G(from the linux top command)
> and the cache hit rate consequently drop to almost 0%.  Is it a known bug or
> we just missed something important?
>
> Thanks.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080620/fe361d01/attachment.html>

From phk at phk.freebsd.dk  Sat Jun 21 09:59:50 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 21 Jun 2008 09:59:50 +0000
Subject: Little question about the releases of Varnish 
In-Reply-To: Your message of "Fri, 20 Jun 2008 18:40:50 +0200."
	<485BDD92.10708@smile.fr> 
Message-ID: <3360.1214042390@critter.freebsd.dk>

In message <485BDD92.10708 at smile.fr>, Damien Sarazin writes:

>My question is : should i continue to work with the 1.1.2 release or is 
>the 1.2 version stable enough so i can use this last one ?

Actually, we are heading into the 2.0 release cycle, so you should
grab a -trunk copy from SVN and look at that.

The first 2.0 release preview will be rolled real soon now.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Sat Jun 21 10:11:50 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 21 Jun 2008 10:11:50 +0000
Subject: pre-2.0 Trac Ticket Triage
Message-ID: <3462.1214043110@critter.freebsd.dk>


As some of you will have discovered, I am busy going through the
tickets in Trac in preparation for 2.0 and have closed all those
tickets that I deemed to be of no help to the project any longer.

Since the 2.0 release will likely increase interest in Varnish
considerably, I want to avoid having 1.x related bugs cluttering
the database, when the default answer to any problem with 1.x
pretty soon will be: Update to 2.x.

Quite a number of tickets I have closed were "heisenbugs" at the
time of filing, and the majority of them relates to the two major
race conditions we had: kqueue/sess and backend/addr.  Obviously
I have no iron-clad guarantee that there is not another bug
hiding there as well, but I trust my users to open new tickets
in that case.

Other more planning-oriented tickets have been moved to the wiki.

If you feel any of your tickets were closed in error, please don't
hessitate to say so.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


From oliver.oli+0815 at gmail.com  Sun Jun 22 11:17:28 2008
From: oliver.oli+0815 at gmail.com (Oliver Oli)
Date: Sun, 22 Jun 2008 13:17:28 +0200
Subject: Little question about the releases of Varnish
In-Reply-To: <3360.1214042390@critter.freebsd.dk>
References: <485BDD92.10708@smile.fr> <3360.1214042390@critter.freebsd.dk>
Message-ID: <66f535320806220417q3a9a8b8ay9f74a43b14edf2e7@mail.gmail.com>

are there any debian buildpackage scripts available?

On Sat, Jun 21, 2008 at 11:59 AM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> In message <485BDD92.10708 at smile.fr>, Damien Sarazin writes:
>
>>My question is : should i continue to work with the 1.1.2 release or is
>>the 1.2 version stable enough so i can use this last one ?
>
> Actually, we are heading into the 2.0 release cycle, so you should
> grab a -trunk copy from SVN and look at that.
>
> The first 2.0 release preview will be rolled real soon now.
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>


From anders at fupp.net  Sun Jun 22 08:30:46 2008
From: anders at fupp.net (Anders Nordby)
Date: Sun, 22 Jun 2008 10:30:46 +0200
Subject: Varnishstat in trunk
In-Reply-To: <4856514B.8020308@vomba.com>
References: <4856514B.8020308@vomba.com>
Message-ID: <20080622083046.GA61470@fupp.net>

Hi,

On Mon, Jun 16, 2008 at 07:40:59AM -0400, Karl Bernard wrote:
> I wanted to test the trunk version, but I am unable to compile 
> varnishstat and varnishtop.
> 
> I have no compilation error, they are simply not there after my make and 
> make install.

Check your configure/build logs.

> I also seem unable to get varnishadm working:
> 
> [root at cdn-web08 varnish-cache]# varnishadm -T 6082 help
> connect(): Invalid argument
> An error occured in receiving status.
> 
> I must be doing something wrong, any body can make suggetsion?

You have an IPv6 localhost address in /etc/hosts, but aren't using IPv6?
Please comment it out if you have it.

Bye,

-- 
Anders.


From duja at torlen.net  Mon Jun 23 08:07:59 2008
From: duja at torlen.net (duja at torlen.net)
Date: Mon, 23 Jun 2008 10:07:59 +0200
Subject: Question about threads
Message-ID: <1fgi7aymrnf7fpf.230620081007@torlen.net>

Thanks for the tips, I will test this and come back with the result.

/ E

Original Message -----------------------
On Wed, 18 Jun 2008 13:24:52 -0700
"Michael S. Fischer" <michael at dynamine.net> wrote:

> On Wed, Jun 18, 2008 at 4:51 AM, Rafael Umann
> <rafael.umann at terra.com.br> wrote:
> 
> >
> > If it is a 32bits system, probably the problem is that your stack
> > size is 10Mb. So 238 * 10mb = ~2gb
> >
> > I decreased my stack size to 512Kb. Using 1gb storage files i can
> > now open almost 1900 threads using all the 2gb that 32bits can
> > alloc. So, my Varnish makes 20000 hits/second serving clients!
> 
> 
> What is your request:connection ratio?
> 
> Best regards,
> 
> --Michael
> 

Unfortunately now i dont have servers doing 20000 hits/second, and
thats why i dont have stats for you. I have 6 servers runing this
service now, each one doing 5500 hits/sec with 10% CPU usage, and this
infrastructure scales to 20000 hits/sec each server. For test purpose
only i let just 2 servers with all the traffic, so i saw this limit i
told you.

In this 20000 hits/sec im seeing two bottle necks:

1. The 32bits arch (cant open threads and the storage file is too
small), so im moving into 64bits.
2. The cpu usage of the "listener process" with 20000 hits/sec is
getting to 100% in one CPU (we also made a patch to improve CPU usage,
and it is avaliable in this last trunk version.. without this patch
varnish was doing 8000 hits/sec). But anyway, i spoke to Poul, and he
told me that the "listener process" will be changed soon (i really hope
so, because i would love to use more than 30% of my cpu!).

Anyway, varnish is doing a great job for me!!!

Some stats taken now with 5500 hits/sec:

# netstat -nap|grep :80 |grep SYN_REC |wc -l
166

# netstat -nap|grep :80 |grep ESTAB |wc -l
17139

# netstat -nap|grep :80 |grep FIN |wc -l
1454

# netstat -nap|grep :80 |grep TIME_W |wc -l
7766


# varnishstat

Hitrate ratio:       10       13       13
Hitrate avg:     0.9990   0.9990   0.9990

    36189916       476.66       250.71 Client connections accepted
   404804204      5494.13      2804.30 Client requests received
   391586079      5335.24      2712.74 Cache hits
    10975643       157.89        76.03 Cache hits for pass
     1125709         1.00         7.80 Cache misses
    12101360       158.89        83.83 Backend connections success
           0         0.00         0.00 Backend connections failures
           0         0.00         0.00 Backend connections reuses
    11139673       141.90        77.17 Backend connections recycles
         350        -1.00         0.00 Backend connections unused
       13447          .            .   N struct srcaddr
       11353          .            .   N active struct srcaddr
       30351          .            .   N struct sess_mem
       17391          .            .   N struct sess
       47069          .            .   N struct object
       47080          .            .   N struct objecthead
       97813          .            .   N struct smf
        4185          .            .   N small free smf
           5          .            .   N large free smf
           0          .            .   N struct vbe_conn
         556          .            .   N worker threads
         556         0.00         0.00 N worker threads created
           0         0.00         0.00 N worker threads not created
           0         0.00         0.00 N worker threads limited
           0         0.00         0.00 N queued work requests
         556         0.00         0.00 N overflowed work requests
           0         0.00         0.00 N dropped work requests
           0          .            .   N expired objects
     1079159          .            .   N LRU nuked objects
           0          .            .   N LRU saved objects
           0          .            .   N objects on deathrow
          48         0.00         0.00 HTTP header overflows
           0         0.00         0.00 Objects sent with sendfile
   135051222      1812.72       935.58 Objects sent with write
    36189893       494.65       250.71 Total Sessions
   404810596      5491.13      2804.35 Total Requests
           0         0.00         0.00 Total pipe
           0         0.00         0.00 Total pass
    12101371       158.89        83.83 Total fetch
 93389657948   1269372.98    646962.32 Total header bytes
418107385660   5962840.69   2896463.38 Total body bytes
     5412491        89.94        37.50 Session Closed
           0         0.00         0.00 Session Pipeline
     1623343         0.00        11.25 Session Read Ahead
   398293975      5409.19      2759.20 Session herd
 15780035398    212996.82    109317.12 SHM records
   516154306      6307.55      3575.69 SHM writes
     2661815        33.98        18.44 SHM MTX contention
    23279696       242.83       161.27 allocator requests
       93623          .            .   outstanding allocations
   910299136          .            .   bytes allocated


[]s,
-- 
Rafael Umann <rafael.umann at corp.terra.com.br>
Suporte Engenharia 1
Terra Networks Brasil S/A 
Tel: +55 (51) 3284-4344



From phk at phk.freebsd.dk  Mon Jun 23 08:14:55 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 23 Jun 2008 08:14:55 +0000
Subject: Question about threads 
In-Reply-To: Your message of "Mon, 23 Jun 2008 10:07:59 +0200."
	<1fgi7aymrnf7fpf.230620081007@torlen.net> 
Message-ID: <12818.1214208895@critter.freebsd.dk>

In message <1fgi7aymrnf7fpf.230620081007 at torlen.net>, duja at torlen.net writes:

>1. The 32bits arch (cant open threads and the storage file is too
>small), so im moving into 64bits.

Yes, 32bit is generally not big enough to Varnish for non-trivial
workloads.

>2. The cpu usage of the "listener process" with 20000 hits/sec is
>getting to 100% in one CPU.

You may want to try to update to a -trunk version and play a bit
with the session_linger parameter, every time you get a tick in
the "Session Linger" counter, that's one request that took a
shortcut.

I have an idea for a more comprehensive shortcut for new sessions
but that will have to wait until after 2.0.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From tfheen at linpro.no  Mon Jun 23 13:27:10 2008
From: tfheen at linpro.no (Tollef Fog Heen)
Date: Mon, 23 Jun 2008 15:27:10 +0200
Subject: Little question about the releases of Varnish
In-Reply-To: <66f535320806220417q3a9a8b8ay9f74a43b14edf2e7@mail.gmail.com>
	(Oliver Oli's message of "Sun, 22 Jun 2008 13:17:28 +0200")
References: <485BDD92.10708@smile.fr> <3360.1214042390@critter.freebsd.dk>
	<66f535320806220417q3a9a8b8ay9f74a43b14edf2e7@mail.gmail.com>
Message-ID: <87mylc1du9.fsf@linpro.no>

* "Oliver Oli" 

| are there any debian buildpackage scripts available?

Yes, you get them in the debian/ directory if you do a subversion
checkout.

-- 
Tollef Fog Heen / Linpro AS    t: 21 54 41 73

UNIX is user friendly, it's just picky about who its friends are


From duja at torlen.net  Mon Jun 23 19:20:59 2008
From: duja at torlen.net (Erik Torlen)
Date: Mon, 23 Jun 2008 21:20:59 +0200
Subject: Question about threads
In-Reply-To: <1fgi7aymrnf7fpf.230620081007@torlen.net>
References: <1fgi7aymrnf7fpf.230620081007@torlen.net>
Message-ID: <485FF79B.4050808@torlen.net>

I still have the same problem :(
The threads are created up to 238 where they are stopped, even if I set 
threads_max = 1000 and threads_pools = 2 (or 3).

I also tested the tips and decreased the stack sixe to 512 and increased 
overflow_max to 10000% .

Any idea what could be wrong?

/ E

duja at torlen.net skrev:
> Thanks for the tips, I will test this and come back with the result.
>
> / E
>
>
>   



From rafael.umann at terra.com.br  Mon Jun 23 20:46:19 2008
From: rafael.umann at terra.com.br (Rafael Umann)
Date: Mon, 23 Jun 2008 17:46:19 -0300
Subject: Question about threads
In-Reply-To: <485FF79B.4050808@torlen.net>
References: <1fgi7aymrnf7fpf.230620081007@torlen.net>
	<485FF79B.4050808@torlen.net>
Message-ID: <20080623174619.203e2ae4@sup-dig.wrk.terra.com.br>


how about your storage file size?

[]s,

On Mon, 23 Jun 2008 21:20:59 +0200
Erik Torlen <duja at torlen.net> wrote:

> I still have the same problem :(
> The threads are created up to 238 where they are stopped, even if I
> set threads_max = 1000 and threads_pools = 2 (or 3).
> 
> I also tested the tips and decreased the stack sixe to 512 and
> increased overflow_max to 10000% .
> 
> Any idea what could be wrong?
> 
> / E
> 
> duja at torlen.net skrev:
> > Thanks for the tips, I will test this and come back with the result.
> >
> > / E
> >
> >
> >   
> 
> 
> E-mail verificado pelo Terra Anti-Spam.
> Para classificar como spam ou n??o spam, visite
> http://mail.terra.com.br/cgi-bin/reportspam.cgi?+_d=SCY1ODAyNDQ3I3Blcm0hdGVycmEmMSwxMjE0MjQ4ODYwLjkzNTUyMC4yNTIxOS5nYW5hbm9xdWUudGVycmEuY29tLDE3MTA=
> Verifique periodicamente a pasta Spam para garantir que apenas
> mensagens indesejadas sejam classificadas como Spam.
> 
> Esta mensagem foi verificada pelo E-mail Protegido Terra.
> Atualizado em 23/06/2008
> 


-- 
Rafael Umann <rafael.umann at corp.terra.com.br>
Suporte Engenharia 1
Terra Networks Brasil S/A 
Tel: 55 (51) 3284-4344


From phk at phk.freebsd.dk  Mon Jun 23 20:50:51 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 23 Jun 2008 20:50:51 +0000
Subject: Question about threads 
In-Reply-To: Your message of "Mon, 23 Jun 2008 21:20:59 +0200."
	<485FF79B.4050808@torlen.net> 
Message-ID: <17105.1214254251@critter.freebsd.dk>

In message <485FF79B.4050808 at torlen.net>, Erik Torlen writes:
>I still have the same problem :(
>The threads are created up to 238 where they are stopped, even if I set 
>threads_max = 1000 and threads_pools = 2 (or 3).
>
>I also tested the tips and decreased the stack sixe to 512 and increased 
>overflow_max to 10000% .
>
>Any idea what could be wrong?

Are you running on a 32bit or 64bit system ?

On a 32bit system you may simply be running out of address-space...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From jianing at gmail.com  Mon Jun 23 23:47:33 2008
From: jianing at gmail.com (Jianing Hu)
Date: Mon, 23 Jun 2008 16:47:33 -0700
Subject: varnish performance problem
Message-ID: <c7e725270806231647q5567963te9bfaee54571c83c@mail.gmail.com>

We are testing varnish as a potential replacement for our legacy squid
setup on a dual core 8GB RAM server. We've noticed that varnish
performance would suddenly deteriorate after running for a while. The
server load is at around 0.2 before all of a sudden it climbs to over
20 and keeps rising. Top shows varnish is using more than 90% of
memory when this happens.

We know that there's much large varnish instances running out there,
so it must be something with our setup. Does anyone know where we
should be looking at? I've attached the command we use to run varnish
and the vcl file below. I can provide more information if that helps.

varnishd -a localhost:80 -T localhost:6082 -f default.vcl -u varnish
-g varnish -s file,/mnt/varnish/varnish_storage.bin,20G

backend default {
    set backend.host = "x.x.x.x";
    set backend.port = "80";
}
sub vcl_recv {
    if (req.request != "GET" && req.request != "HEAD") {
        pipe;
    }
    if (req.http.Expect) {
        pipe;
    }
    if (req.http.Authenticate) {
        pass;
    }
    lookup;
}
sub vcl_pipe {
    pipe;
}
sub vcl_pass {
    pass;
}
sub vcl_hash {
    set req.hash += req.url;
    set req.hash += req.http.host;
    hash;
}
sub vcl_hit {
    if (!obj.cacheable) {
        pass;
    }
    deliver;
}
sub vcl_miss {
    fetch;
}
sub vcl_fetch {
    if (!obj.valid) {
        error;
    }
    if (!obj.cacheable) {
        pass;
    }
    if (obj.http.Set-Cookie) {
        pass;
    }
    insert;
}
sub vcl_deliver {
    deliver;
}
sub vcl_timeout {
    discard;
}
sub vcl_discard {
    discard;
}


From armdan20 at gmail.com  Tue Jun 24 07:18:01 2008
From: armdan20 at gmail.com (andan andan)
Date: Tue, 24 Jun 2008 09:18:01 +0200
Subject: Varnish trunk rev 2780 refuses to start
Message-ID: <c1dd32e40806240018y11027ab2u8c8ac4d4513fdbe2@mail.gmail.com>

Hi all.

We have upgraded to the latest trunk version (2780), but Varnish
refuses to start property, the syslog says:

Jun 24 07:09:06 vvarnish varnishd[2280]: child (2281) Started
Jun 24 07:09:06 vvarnish varnishd[2280]: Pushing vcls failed: CLI
communication error
Jun 24 07:09:06 vvarnish varnishd[2280]: Child (2281) said Closed fds:
3 4 5 6 10 11 13 14
Jun 24 07:09:06 vvarnish varnishd[2280]: Child (2281) said Child starts
Jun 24 07:09:06 vvarnish varnishd[2280]: Child (2281) said managed to
mmap 0 bytes of 734003200

Our configuration is:

DAEMON_OPTS="-a :80 \
             -T 127.0.0.1:6082 \
             -f /etc/varnish/conf.vcl \
             -u varnish -g varnish \
             -s file,/var/lib/varnish/varnish_storage.bin,700MB \
             -t 240 \
             -p thread_pools=1 \
             -p thread_pool_max=5000"

CentOS 5.1, rpm built from trunk, revision 2780.

Previously, we were using revision 2735, this works fine.

Thanks.
BR.


From phk at phk.freebsd.dk  Tue Jun 24 10:15:51 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 24 Jun 2008 10:15:51 +0000
Subject: Varnish trunk rev 2780 refuses to start 
In-Reply-To: Your message of "Tue, 24 Jun 2008 09:18:01 +0200."
	<c1dd32e40806240018y11027ab2u8c8ac4d4513fdbe2@mail.gmail.com> 
Message-ID: <57013.1214302551@critter.freebsd.dk>

In message <c1dd32e40806240018y11027ab2u8c8ac4d4513fdbe2 at mail.gmail.com>, "anda
n andan" writes:


Can you try #2786 pleae, I think I have fixed it now.

Poul-Henning

>Hi all.
>
>We have upgraded to the latest trunk version (2780), but Varnish
>refuses to start property, the syslog says:
>
>Jun 24 07:09:06 vvarnish varnishd[2280]: child (2281) Started
>Jun 24 07:09:06 vvarnish varnishd[2280]: Pushing vcls failed: CLI
>communication error
>Jun 24 07:09:06 vvarnish varnishd[2280]: Child (2281) said Closed fds:
>3 4 5 6 10 11 13 14
>Jun 24 07:09:06 vvarnish varnishd[2280]: Child (2281) said Child starts
>Jun 24 07:09:06 vvarnish varnishd[2280]: Child (2281) said managed to
>mmap 0 bytes of 734003200
>
>Our configuration is:
>
>DAEMON_OPTS="-a :80 \
>             -T 127.0.0.1:6082 \
>             -f /etc/varnish/conf.vcl \
>             -u varnish -g varnish \
>             -s file,/var/lib/varnish/varnish_storage.bin,700MB \
>             -t 240 \
>             -p thread_pools=1 \
>             -p thread_pool_max=5000"
>
>CentOS 5.1, rpm built from trunk, revision 2780.
>
>Previously, we were using revision 2735, this works fine.
>
>Thanks.
>BR.
>_______________________________________________
>varnish-misc mailing list
>varnish-misc at projects.linpro.no
>http://projects.linpro.no/mailman/listinfo/varnish-misc
>

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From armdan20 at gmail.com  Tue Jun 24 10:29:49 2008
From: armdan20 at gmail.com (andan andan)
Date: Tue, 24 Jun 2008 12:29:49 +0200
Subject: Varnish trunk rev 2780 refuses to start
In-Reply-To: <57013.1214302551@critter.freebsd.dk>
References: <c1dd32e40806240018y11027ab2u8c8ac4d4513fdbe2@mail.gmail.com>
	<57013.1214302551@critter.freebsd.dk>
Message-ID: <c1dd32e40806240329w1fd6bb8ekc6e0743aaae9ccde@mail.gmail.com>

2008/6/24 Poul-Henning Kamp <phk at phk.freebsd.dk>:
> In message <c1dd32e40806240018y11027ab2u8c8ac4d4513fdbe2 at mail.gmail.com>, "anda
> n andan" writes:
>
>
> Can you try #2786 pleae, I think I have fixed it now.

Working again !

Thank you very much.
BR.


From jianing at gmail.com  Tue Jun 24 18:10:54 2008
From: jianing at gmail.com (Jianing Hu)
Date: Tue, 24 Jun 2008 11:10:54 -0700
Subject: varnish performance problem
Message-ID: <c7e725270806241110g22c7f69fk744d121981910a5c@mail.gmail.com>

Sorry if this is a duplicate post. I sent the following yesterday but
didn't see it appearing in the mailing list.

We are testing varnish as a potential replacement for our legacy squid
setup on a dual core 8GB RAM server. We've noticed that varnish
performance would suddenly deteriorate after running for a while. The
server load is at around 0.2 before all of a sudden it climbs to over
20 and keeps rising. Top shows varnish is using more than 90% of
memory when this happens.

We know that there's much large varnish instances running out there,
so it must be something with our setup. Does anyone know where we
should be looking at? I've attached the command we use to run varnish
and the vcl file below. I can provide more information if that helps.

varnishd -a localhost:80 -T localhost:6082 -f default.vcl -u varnish
-g varnish -s file,/mnt/varnish/varnish_storage.bin,20G

backend default {
   set backend.host = "x.x.x.x";
   set backend.port = "80";
}
sub vcl_recv {
   if (req.request != "GET" && req.request != "HEAD") {
       pipe;
   }
   if (req.http.Expect) {
       pipe;
   }
   if (req.http.Authenticate) {
       pass;
   }
   lookup;
}
sub vcl_pipe {
   pipe;
}
sub vcl_pass {
   pass;
}
sub vcl_hash {
   set req.hash += req.url;
   set req.hash += req.http.host;
   hash;
}
sub vcl_hit {
   if (!obj.cacheable) {
       pass;
   }
   deliver;
}
sub vcl_miss {
   fetch;
}
sub vcl_fetch {
   if (!obj.valid) {
       error;
   }
   if (!obj.cacheable) {
       pass;
   }
   if (obj.http.Set-Cookie) {
       pass;
   }
   insert;
}
sub vcl_deliver {
   deliver;
}
sub vcl_timeout {
   discard;
}
sub vcl_discard {
   discard;
}


From phk at phk.freebsd.dk  Tue Jun 24 18:15:33 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 24 Jun 2008 18:15:33 +0000
Subject: varnish performance problem 
In-Reply-To: Your message of "Tue, 24 Jun 2008 11:10:54 MST."
	<c7e725270806241110g22c7f69fk744d121981910a5c@mail.gmail.com> 
Message-ID: <64928.1214331333@critter.freebsd.dk>

In message <c7e725270806241110g22c7f69fk744d121981910a5c at mail.gmail.com>, "Jian
ing Hu" writes:
>Sorry if this is a duplicate post. I sent the following yesterday but
>didn't see it appearing in the mailing list.
>
>We are testing varnish as a potential replacement for our legacy squid
>setup on a dual core 8GB RAM server. We've noticed that varnish
>performance would suddenly deteriorate after running for a while. The
>server load is at around 0.2 before all of a sudden it climbs to over
>20 and keeps rising. Top shows varnish is using more than 90% of
>memory when this happens.

Are you running on a 32bit or 64 bit machine ?

What does varnishstat say about how much you have loaded in the
cache when performance deteriorates ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From duja at torlen.net  Tue Jun 24 19:04:46 2008
From: duja at torlen.net (Erik Torlen)
Date: Tue, 24 Jun 2008 21:04:46 +0200
Subject: Question about threads
In-Reply-To: <17105.1214254251@critter.freebsd.dk>
References: <17105.1214254251@critter.freebsd.dk>
Message-ID: <4861454E.1030909@torlen.net>

Im running 32bit. But I think that I have succeded creating more then 
238 threads before on another system with the same setup.

Anyway, 64bit might be the thing to have...

If I want to have Debian, is it AMD64 version that I should go for? (OT)

/ E

Poul-Henning Kamp skrev:
> In message <485FF79B.4050808 at torlen.net>, Erik Torlen writes:
>   
>> I still have the same problem :(
>> The threads are created up to 238 where they are stopped, even if I set 
>> threads_max = 1000 and threads_pools = 2 (or 3).
>>
>> I also tested the tips and decreased the stack sixe to 512 and increased 
>> overflow_max to 10000% .
>>
>> Any idea what could be wrong?
>>     
>
> Are you running on a 32bit or 64bit system ?
>
> On a 32bit system you may simply be running out of address-space...
>
>   



From phk at phk.freebsd.dk  Tue Jun 24 19:15:03 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 24 Jun 2008 19:15:03 +0000
Subject: Question about threads 
In-Reply-To: Your message of "Tue, 24 Jun 2008 21:04:46 +0200."
	<4861454E.1030909@torlen.net> 
Message-ID: <65509.1214334903@critter.freebsd.dk>

In message <4861454E.1030909 at torlen.net>, Erik Torlen writes:
>Im running 32bit. But I think that I have succeded creating more then 
>238 threads before on another system with the same setup.
>
>Anyway, 64bit might be the thing to have...
>
>If I want to have Debian, is it AMD64 version that I should go for? (OT)

Yes.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From perbu at linpro.no  Tue Jun 24 19:24:55 2008
From: perbu at linpro.no (Per Buer)
Date: Tue, 24 Jun 2008 21:24:55 +0200
Subject: Question about threads
In-Reply-To: <4861454E.1030909@torlen.net>
References: <17105.1214254251@critter.freebsd.dk> <4861454E.1030909@torlen.net>
Message-ID: <48614A07.5060000@linpro.no>

Erik Torlen skrev:
> Im running 32bit. But I think that I have succeded creating more then 
> 238 threads before on another system with the same setup.
> 
> Anyway, 64bit might be the thing to have...

Maybe the init script should issue a warning that a 32bit arch is only
usable as a test enviroment.


Per.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080624/0c2602fa/attachment.bin>

From phk at phk.freebsd.dk  Tue Jun 24 19:30:53 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 24 Jun 2008 19:30:53 +0000
Subject: Question about threads 
In-Reply-To: Your message of "Tue, 24 Jun 2008 21:24:55 +0200."
	<48614A07.5060000@linpro.no> 
Message-ID: <65595.1214335853@critter.freebsd.dk>

In message <48614A07.5060000 at linpro.no>, Per Buer writes:

>Maybe the init script should issue a warning that a 32bit arch is only
>usable as a test enviroment.

Well, Varnish is generally usable in 32bit, provided you have
very small content, so I'm hessitant to rule it entirely out,
but yes, we clearly need to push the 64bit angle.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From darryl.dixon at winterhouseconsulting.com  Tue Jun 24 20:39:09 2008
From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting)
Date: Wed, 25 Jun 2008 08:39:09 +1200 (NZST)
Subject: honoring browser reload request
Message-ID: <52544.203.97.62.217.1214339949.squirrel@services.directender.co.nz>

> is the
>following VCL the "correct" solution to support browser shift-reloads to
>get fresh content and actually update the cache?
>
>sub vcl_hit {
>     if (req.http.Cache-Control ~ "no-cache") {
>         set obj.ttl = 0s;
>         pass;
>     }
>}
>

Only kind-of; IE and Mozilla send different things with a SHIFT (or CTRL)
+ Reload, we do virtually what you have outlined above, but with a slight
addition:

in vcl_recv, we shortcut and send such requests straight to lookup:

    /* Honour Cache-Control: and Pragma: ... */
    if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~
".*no-cache.*") {
        lookup;
    }

...and then in vcl_recv we do similar to you:
    /* Honour Cache-Control: and Pragma: ... */
    if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~
".*no-cache.*") {
        set obj.ttl = 0s;
        pass;
    }


>As far as I understood vcl(7), this means:
>1. fetch the object from cache (implicit in the fact that we're inside
>vcl_hit)
>2. set ttl to 0s, expiring it (so next requests will fetch new content)
>3. pass to backend (so *this* request see new content)
>Does this do what I expect it to do?

Yes, this configuration does indeed do what you expect (purge the item, go
get a fresh one from the backend)

>Is there a way to avoid hitting the backend twice?

You're only hitting the backend once if you're running the code from vcl_hit

>Is there a way to do that in vcl_recv?

Not really, but see our shortcut above

>(I guess not, since there's no "obj" object available yet, there)
>I guess I could use purge_url(...), except I have virtual hosts and it
>wants a regexp and req.url could contain some special char (could it?).

We tried with purge_url but it was just too hard and didn't want to do
what we expected. We've had good results with our configuration above.

Hope this helps,

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com



From darryl.dixon at winterhouseconsulting.com  Tue Jun 24 21:12:33 2008
From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting)
Date: Wed, 25 Jun 2008 09:12:33 +1200 (NZST)
Subject: honoring browser reload request
In-Reply-To: <52544.203.97.62.217.1214339949.squirrel@services.directender.co.nz>
References: <52544.203.97.62.217.1214339949.squirrel@services.directender.co.nz>
Message-ID: <44608.203.97.62.217.1214341953.squirrel@services.directender.co.nz>

> in vcl_recv, we shortcut and send such requests straight to lookup:
>
>     /* Honour Cache-Control: and Pragma: ... */
>     if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~
> ".*no-cache.*") {
>         lookup;
>     }
>
> ...and then in vcl_recv we do similar to you:

Bother, of course I meant: ..and then in *vcl_hit* we do similar to you... :)

>     /* Honour Cache-Control: and Pragma: ... */
>     if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~
> ".*no-cache.*") {
>         set obj.ttl = 0s;
>         pass;
>     }
>

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com


From rafael.umann at terra.com.br  Tue Jun 24 22:14:35 2008
From: rafael.umann at terra.com.br (Rafael Umann)
Date: Tue, 24 Jun 2008 19:14:35 -0300
Subject: Question about threads
In-Reply-To: <48614A07.5060000@linpro.no>
References: <17105.1214254251@critter.freebsd.dk> <4861454E.1030909@torlen.net>
	<48614A07.5060000@linpro.no>
Message-ID: <20080624191435.3fbd08cd@sup-dig.wrk.terra.com.br>

On Tue, 24 Jun 2008 21:24:55 +0200
Per Buer <perbu at linpro.no> wrote:

> Erik Torlen skrev:
> > Im running 32bit. But I think that I have succeded creating more
> > then 238 threads before on another system with the same setup.
> > 
> > Anyway, 64bit might be the thing to have...
> 
> Maybe the init script should issue a warning that a 32bit arch is only
> usable as a test enviroment.

Well, i disagree with that. Im runing 6 servers with 50000 hits/second
in a 32 bit system and this servers can go to 115000 hits/second. This
is a LOT of traffic to consider as test :)

> 
> 
> Per.
> 


-- 
Rafael Umann <rafael.umann at corp.terra.com.br>
Suporte Engenharia 1
Terra Networks Brasil S/A 
Tel: 55 (51) 3284-4344


From drupalnut at gmail.com  Wed Jun 25 04:51:56 2008
From: drupalnut at gmail.com (Jack Tuhman)
Date: Wed, 25 Jun 2008 00:51:56 -0400
Subject: Varnish Questions
Message-ID: <f401c82a0806242151o5e81f156xf25f499abadf52af@mail.gmail.com>

Hello list,

I was about to setup another squid install for a drupal site that is going
to be getting some high traffic and I found varnish.  I think that product
is really great but I am having some issues.


First hardware.
I have varnish and the web server running on the same quad core dual xeon
box 8gb of ram on debian.  I have used the debian package to install varnish
and it installed  version1.0.2-2

There is a database for the site running on another box.

Here are my 2 questions, I hope someone out there can point me in the right
direction.

1. connection reset issues
I ran
 siege -c 250 -r 1000 -i -d 1 http://<myip>

This is a sample of the output I get
HTTP/1.1 200   0.41 secs:    5548 bytes ==> /
HTTP/1.1 200   0.41 secs:    5548 bytes ==> /
HTTP/1.1 200   0.42 secs:    5548 bytes ==> /
HTTP/1.1 200   0.41 secs:    5548 bytes ==> /
Error: socket: read error Connection reset by peer sock.c:455: Connection
reset by peer
Error: socket: read error Connection reset by peer sock.c:455: Connection
reset by peer
Error: socket: read error Connection reset by peer sock.c:455: Connection
reset by peer
Error: socket: read error Connection reset by peer sock.c:455: Connection
reset by peer
Error: socket: read error Connection reset by peer sock.c:455: Connection
reset by peer
HTTP/1.1 200   1.17 secs:    5548 bytes ==> /
HTTP/1.1 200   1.94 secs:    5548 bytes ==> /
HTTP/1.1 200   0.63 secs:    5548 bytes ==> /
HTTP/1.1 200   0.63 secs:    5548 bytes ==> /
HTTP/1.1 200   0.62 secs:    5548 bytes ==> /

At the end of the run I get Availability of around 45%  If I run  siege
directly to the webserver, I get an Availability of 98%. (it also takes
longer)

2. caching pages
I have updated content on my main site, and would like to clear the varnish
cache,  I telnet into the management interface and run purge * (I also tried
purge *. , purge .* , purge *.*) I then clear my browser cache, and refresh
the page, I get the old page.  If I visit the server directly, I get the new
page?


Below is my vcl.conf file I have comment out the cookie part because I would
like it to pass those requests to the webserver.  (I guess it could cache
the front page)


backend default {
        set backend.host = "10.10.0.10";
        set backend.port = "81";
}

sub vcl_recv {
        if (req.request == "POST") {
                pipe;
        }

        # force lookup even when cookies are present
#        if (req.request == "GET" && req.http.cookie) {
#                lookup;
#        }
 if (req.request == "GET" && req.url ~ "\.(gif|jpg|swf|css|js)$") {
    lookup;
 }
}

sub vcl_fetch {
        # force minimum ttl of 180 seconds
        if (obj.ttl < 180s) {
                set obj.ttl = 180s;
        }
}


Thanks for any help someone can give.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080625/927dd21b/attachment.html>

From darryl.dixon at winterhouseconsulting.com  Wed Jun 25 05:17:12 2008
From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting)
Date: Wed, 25 Jun 2008 17:17:12 +1200 (NZST)
Subject: Varnish Questions
Message-ID: <3392.203.97.62.217.1214371032.squirrel@services.directender.co.nz>

> Hello list,
>

Hi Jack,

>
> 1. connection reset issues

Can't comment specifically on these, but your varnish version is very old.
1.1.2 is good and stable for us, and -trunk is apparently pretty stable,
too...
>
> 2. caching pages
> I have updated content on my main site, and would like to clear the
> varnish
> cache,  I telnet into the management interface and run purge * (I also
> tried
> purge *. , purge .* , purge *.*) I then clear my browser cache, and
> refresh
> the page, I get the old page.  If I visit the server directly, I get the
> new
> page?
>

it's 'url.purge' in 1.1.2, not sure about your version.

>
> Below is my vcl.conf file I have comment out the cookie part because I
[...snip...]

It's important to know/remember that the vcl you write does not *replace*
the default vcl, it just runs first, unless you specifically return a
different action (eg, lookup, pass, etc). The cookie stuff you hashed out
will still run in the default vcl afterward with your current setup. See:
http://varnish.projects.linpro.no/wiki/VCLExampleDefault
...for more information on this (be aware the default presented there is
probably not correct for your current version).

> Thanks for any help someone can give.

You're welcome ;)

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com



From drupalnut at gmail.com  Fri Jun 27 01:00:46 2008
From: drupalnut at gmail.com (Jack Tuhman)
Date: Thu, 26 Jun 2008 21:00:46 -0400
Subject: cache files pass php
Message-ID: <f401c82a0806261800h634d16b2lf86692705f28e5ab@mail.gmail.com>

Is there an easy vcl trick to get varnish to always pass php requests to the
webserver and to cache images, css javascript, etc?

I would like

apache->varnish->client

If the client requests a document, the php is run on the webserver, varnish
takes it and downloads it to the client so that the apache thread can be
used for other things. Varnish can cache the images and text files as they
don't change.

Thoughts?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20080626/b81b045c/attachment.html>

From darryl.dixon at winterhouseconsulting.com  Fri Jun 27 01:34:24 2008
From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting)
Date: Fri, 27 Jun 2008 13:34:24 +1200 (NZST)
Subject: cache files pass php
In-Reply-To: <f401c82a0806261800h634d16b2lf86692705f28e5ab@mail.gmail.com>
References: <f401c82a0806261800h634d16b2lf86692705f28e5ab@mail.gmail.com>
Message-ID: <20032.203.97.62.217.1214530464.squirrel@services.directender.co.nz>

> Is there an easy vcl trick to get varnish to always pass php requests to
> the
> webserver and to cache images, css javascript, etc?
>
> I would like
>
> apache->varnish->client
>
> If the client requests a document, the php is run on the webserver,
> varnish
> takes it and downloads it to the client so that the apache thread can be
> used for other things. Varnish can cache the images and text files as they
> don't change.
>
> Thoughts?

Hi Jack,

This seems straightforward, in vcl_recv you would need something like:
sub vcl_recv {
if (req.url ~
"\.(ico|gif|jpeg|jpg|png|swf|css|htc|js|bz2|gz|zip|xls|ppt|pdf|doc|rar)")
{
    lookup;
}
pass;
}

The idea is that items that you 'know' are cacheable get looked up in the
cache and fetched from the backend if necessary, and that everything else
otherwise gets passed straight to the backend for servicing.

(note that this is an extremely abbreviated example that probably omits
several important considerations like cookies, etc. Including, depending
on what headers, etc the backend emits, you may want some more code in
vcl_fetch to decide whether something is insert-ed into the cache or
pass-ed onwards)

Hope this helps,

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com




From oliver.oli+0815 at gmail.com  Fri Jun 27 13:29:23 2008
From: oliver.oli+0815 at gmail.com (Oliver Oli)
Date: Fri, 27 Jun 2008 15:29:23 +0200
Subject: Varnish 2.0 Debian buildpackage errors
Message-ID: <66f535320806270629y3d10c8c8td5c5ce0a7565e3d0@mail.gmail.com>

Any idea why it doesn't build?

~/svn/varnish-2.0/varnish-cache$ dpkg-buildpackage
dpkg-buildpackage: set CFLAGS to default value: -g -O2
dpkg-buildpackage: set CPPFLAGS to default value:
dpkg-buildpackage: set LDFLAGS to default value:
dpkg-buildpackage: set FFLAGS to default value: -g -O2
dpkg-buildpackage: set CXXFLAGS to default value: -g -O2
dpkg-buildpackage: source package varnish
dpkg-buildpackage: source version 2.0~tp2-0
dpkg-buildpackage: source changed by Stig Sandbeck Mathisen <ssm at linpro.no>
dpkg-buildpackage: host architecture i386
 fakeroot debian/rules clean
dh_testdir
dh_testroot
rm -f build-stamp
[ -f config.sub.orig ] && mv config.sub.orig config.sub || true
[ -f config.guess.orig ] && mv config.sub.orig config.guess || true
[ -f Makefile ] && make distclean
make: *** [clean] Error 1
dpkg-buildpackage: failure: fakeroot debian/rules clean gave error exit status 2


From skye at f4.ca  Fri Jun 27 18:16:49 2008
From: skye at f4.ca (Skye Poier Nott)
Date: Fri, 27 Jun 2008 11:16:49 -0700
Subject: A couple of simple questions...
Message-ID: <255AD360-3015-4E47-9861-1781763B5CBE@f4.ca>

Hello... a few beginner questions:

1. When you restart Varnish the cache is always purged?

2. If I have a section like this in vcl_fetch, is the default_ttl  
parameter basically ignored? (assuming default_ttl is less than the  
value below).  So in other words, this VCL code enforces a minimum TTL?

	if (obj.ttl < 36h) {
		set obj.ttl = 36h;
	}

3. When I want to use Varnish in a WAN transparent accelerator mode,  
if I don't specify any "backend default" is the default to fetch the  
document from the hostname in the URL?

Thanks,
Skye



From tfheen at linpro.no  Mon Jun 30 12:17:04 2008
From: tfheen at linpro.no (Tollef Fog Heen)
Date: Mon, 30 Jun 2008 14:17:04 +0200
Subject: A couple of simple questions...
In-Reply-To: <255AD360-3015-4E47-9861-1781763B5CBE@f4.ca> (Skye Poier Nott's
	message of "Fri\, 27 Jun 2008 11\:16\:49 -0700")
References: <255AD360-3015-4E47-9861-1781763B5CBE@f4.ca>
Message-ID: <87abh3azi7.fsf@luxevop.linpro.no>

* Skye Poier Nott 

| 1. When you restart Varnish the cache is always purged?

Correct, we don't have persistent caches yet.

| 2. If I have a section like this in vcl_fetch, is the default_ttl  
| parameter basically ignored? (assuming default_ttl is less than the  
| value below).  So in other words, this VCL code enforces a minimum TTL?
| 
| 	if (obj.ttl < 36h) {
| 		set obj.ttl = 36h;
| 	}

At first glance, this looks fine tome.

| 3. When I want to use Varnish in a WAN transparent accelerator mode,  
| if I don't specify any "backend default" is the default to fetch the  
| document from the hostname in the URL?

You have to define a default backend.

-- 
Tollef Fog Heen / Linpro AS    t: 21 54 41 73

UNIX is user friendly, it's just picky about who its friends are


From wichert at wiggy.net  Mon Jun 30 13:18:05 2008
From: wichert at wiggy.net (Wichert Akkerman)
Date: Mon, 30 Jun 2008 15:18:05 +0200
Subject: grace handling
Message-ID: <4868DD0D.5070108@wiggy.net>

Is there any documentation on how req.grace and obj.grace are used? I 
could not find any mention of grace handling in the manpages or the wiki.

Wichert.

-- 
Wichert Akkerman <wichert at wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.




From skye at F4.ca  Mon Jun 30 18:36:01 2008
From: skye at F4.ca (Skye Poier Nott)
Date: Mon, 30 Jun 2008 11:36:01 -0700
Subject: A couple of simple questions...
In-Reply-To: <35A17A9B-42B0-41F0-9E19-520159F05D37@F4.ca>
References: <255AD360-3015-4E47-9861-1781763B5CBE@f4.ca>
	<87abh3azi7.fsf@luxevop.linpro.no>
	<35A17A9B-42B0-41F0-9E19-520159F05D37@F4.ca>
Message-ID: <782A146C-D220-4077-BEAF-9D8BDB3EC240@F4.ca>

(Whoops, meant to send this to the list)

>
> | 3. When I want to use Varnish in a WAN transparent accelerator mode,
> | if I don't specify any "backend default" is the default to fetch the
> | document from the hostname in the URL?
>
> You have to define a default backend.

Hmm, OK.  Is the scenario I describe above even possible with  
Varnish?  Can you build a dynamic backend at vcl_recv() time out of  
req.http.host or something like that?

Thanks,
Skye




From fragfutter at gmail.com  Mon Jun 30 18:40:53 2008
From: fragfutter at gmail.com (C. Handel)
Date: Mon, 30 Jun 2008 20:40:53 +0200
Subject: A couple of simple questions...
In-Reply-To: <782A146C-D220-4077-BEAF-9D8BDB3EC240@F4.ca>
References: <255AD360-3015-4E47-9861-1781763B5CBE@f4.ca>
	<87abh3azi7.fsf@luxevop.linpro.no>
	<35A17A9B-42B0-41F0-9E19-520159F05D37@F4.ca>
	<782A146C-D220-4077-BEAF-9D8BDB3EC240@F4.ca>
Message-ID: <3d62bd5f0806301140q5c02027erd930be1d0829b175@mail.gmail.com>

On Mon, Jun 30, 2008 at 8:36 PM, Skye Poier Nott <skye at f4.ca> wrote:
> (Whoops, meant to send this to the list)
>
>>
>> | 3. When I want to use Varnish in a WAN transparent accelerator mode,
>> | if I don't specify any "backend default" is the default to fetch the
>> | document from the hostname in the URL?
>>
>> You have to define a default backend.
>
> Hmm, OK.  Is the scenario I describe above even possible with
> Varnish?  Can you build a dynamic backend at vcl_recv() time out of
> req.http.host or something like that?

What you are describing is a classic Proxy (->Squid).
Varnish is used as accelerator for a given Plattform.

Greetings
   Christoph


From skye at F4.ca  Mon Jun 30 20:50:38 2008
From: skye at F4.ca (Skye Poier Nott)
Date: Mon, 30 Jun 2008 13:50:38 -0700
Subject: Reloading default.vcl
Message-ID: <A5F6E7FC-A98A-43C6-B7A6-97CDD7555720@F4.ca>

Is it possible to reload /usr/local/etc/varnish/default.vcl without  
restarting?

 From the docs it looks like one way to do this would be through the  
telnet interface:

vcl.load some-unique-name /usr/local/etc/varnish/default.vcl
vcl.use some-unique-name

each time the default.vcl is changed, unless there's reload option I  
haven't found.

Thanks,
Skye



From phk at phk.freebsd.dk  Mon Jun 30 21:54:01 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 30 Jun 2008 21:54:01 +0000
Subject: Reloading default.vcl 
In-Reply-To: Your message of "Mon, 30 Jun 2008 13:50:38 MST."
	<A5F6E7FC-A98A-43C6-B7A6-97CDD7555720@F4.ca> 
Message-ID: <3083.1214862841@critter.freebsd.dk>

In message <A5F6E7FC-A98A-43C6-B7A6-97CDD7555720 at F4.ca>, Skye Poier Nott writes
:
>Is it possible to reload /usr/local/etc/varnish/default.vcl without  
>restarting?
>
> From the docs it looks like one way to do this would be through the  
>telnet interface:
>
>vcl.load some-unique-name /usr/local/etc/varnish/default.vcl
>vcl.use some-unique-name
>
>each time the default.vcl is changed, unless there's reload option I  
>haven't found.

Correct.

Afterwards you can
	vcl.discard old-unique-name

if you want to get rid of it.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From darryl.dixon at winterhouseconsulting.com  Mon Jun 30 22:02:52 2008
From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting)
Date: Tue, 1 Jul 2008 10:02:52 +1200 (NZST)
Subject: Reloading default.vcl
In-Reply-To: <3083.1214862841@critter.freebsd.dk>
References: <3083.1214862841@critter.freebsd.dk>
Message-ID: <64320.203.97.62.217.1214863372.squirrel@services.directender.co.nz>

> In message <A5F6E7FC-A98A-43C6-B7A6-97CDD7555720 at F4.ca>, Skye Poier Nott
> writes
> :
>>Is it possible to reload /usr/local/etc/varnish/default.vcl without
>>restarting?
>>
>> From the docs it looks like one way to do this would be through the
>>telnet interface:
>>
>>vcl.load some-unique-name /usr/local/etc/varnish/default.vcl
>>vcl.use some-unique-name
>>
>>each time the default.vcl is changed, unless there's reload option I
>>haven't found.
>
> Correct.
>
> Afterwards you can
> 	vcl.discard old-unique-name
>
> if you want to get rid of it.
>

It is pretty easy to automate this all with expect(1) as well, if an
'automated' reload solution is required :)

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com