From jwood at guba.com  Fri Mar  2 01:18:59 2007
From: jwood at guba.com (Joel Wood)
Date: Thu, 1 Mar 2007 17:18:59 -0800
Subject: varnish problems on Debian 3.1, kernel  2.6.19.2
Message-ID: <FA11B6A9-3E7F-4B63-A055-4A7DEC5744B4@guba.com>

Hello All,

I'm trying to test out varnish 1.0.3 on Debian 3.1 with a 2.6.19.2  
kernel, and I seem to be having some difficulties.

I run varnish with the following options:

sbin/varnishd -T 127.0.0.1:8082 -s file,/home/guba/varnish/var/  -a  
0.0.0.0:9081 -b timothy:8080

varnish starts fine, and runs.  When I request a file with wget there  
is a pause of several seconds between the request and the delivery of  
the file.  As I get the file, I see:

    12 SessionClose c timeout

in the output from varnishlog.

If I use curl, I get the file immediately.

Any help would be greatly appreciated.

-Joel 
   


From des at linpro.no  Fri Mar  2 11:06:27 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Fri, 02 Mar 2007 12:06:27 +0100
Subject: varnish problems on Debian 3.1, kernel  2.6.19.2
In-Reply-To: <FA11B6A9-3E7F-4B63-A055-4A7DEC5744B4@guba.com> (Joel Wood's
	message of "Thu, 1 Mar 2007 17:18:59 -0800")
References: <FA11B6A9-3E7F-4B63-A055-4A7DEC5744B4@guba.com>
Message-ID: <ujrps7rykek.fsf@cat.linpro.no>

Joel Wood <jwood at guba.com> writes:
> varnish starts fine, and runs.  When I request a file with wget there
> is a pause of several seconds between the request and the delivery of
> the file.  As I get the file, I see:
>
>    12 SessionClose c timeout
>
> in the output from varnishlog.
>
> If I use curl, I get the file immediately.

Try turning off keepalive on the backend.

If that doesn't help, post a full log of the transaction with both
wget and curl (remember to restart varnishd, or at least clear the
cache, before each attempt)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From lists at dirkgomez.de  Wed Mar 14 05:40:06 2007
From: lists at dirkgomez.de (Dirk Gomez)
Date: Wed, 14 Mar 2007 06:40:06 +0100
Subject: Varnish as an HTTP accelator for multiple Zope instances on
	multiple machines 
Message-ID: <C2604441-3817-4975-817A-70AC5DB1C2B4@dirkgomez.de>

Hi list,

I'm trying to use varnish as an HTTP accelerator for a Zope virtual  
hosting shop. HTTP requests are to be dispatched based on the  
hostname to different Zope backends which reside on different machines.

Varnish runs on Debian testing and my setup doesn't work. The  
vcl.conf is simple and looks correct. (The conciseness of vcl.conf  
was the prime reason to ditch squid btw) This is vcl.conf:

backend default {
         set backend.host = "127.0.0.1";
         set backend.port = "7080";
}


backend thishost {
         set backend.host = "127.0.0.1";
         set backend.port = "7080";
}

backend thathost {
         set backend.host = "192.168.100.199";
         set backend.port = "9080";
}

sub vcl_recv {
         if (req.request == "POST") {
                 pipe;
         }

         # force lookup even when cookies are present
         if (req.request == "GET" && req.http.cookie) {
                 lookup;
         }

              if (req.http.host ~ "^hop.metatest.de") {
                  set req.backend = thishost;
              } elsif (req.http.host ~ "^zop.metatest.de") {
                  set req.backend = thathost;
              } else {
                  error 404 "Unknown virtual host";
              }


}

sub vcl_fetch {
         # force minimum ttl of 180 seconds
         if (obj.ttl < 180s) {
                 set obj.ttl = 180s;
         }
}

/etc/default/varnish hasn't been changed.

varnishlog gives me this output:

    13 RxHeader     c Connection: keep-alive
    13 RxHeader     c Host: zop.metatest.de
    13 VCL_call     c recv
    13 VCL_return   c lookup
    13 VCL_call     c miss
    13 VCL_return   c fetch
    17 BackendClose   default
    17 BackendOpen  b default 127.0.0.1 46677 127.0.0.1 7080

So varnish goes to 127.0.0.1 instead of 192.168.100.199.

Any pointers what is wrong?

Thanks a lot!

-- Dirk




From des at linpro.no  Wed Mar 14 09:15:46 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Wed, 14 Mar 2007 10:15:46 +0100
Subject: Varnish as an HTTP accelator for multiple Zope instances on
	multiple machines
In-Reply-To: <C2604441-3817-4975-817A-70AC5DB1C2B4@dirkgomez.de> (Dirk Gomez's
	message of "Wed, 14 Mar 2007 06:40:06 +0100")
References: <C2604441-3817-4975-817A-70AC5DB1C2B4@dirkgomez.de>
Message-ID: <ujrk5xk424d.fsf@cat.linpro.no>

Dirk Gomez <lists at dirkgomez.de> writes:
> sub vcl_recv {
>         if (req.request == "POST") {
>                 pipe;
>         }
>
>         # force lookup even when cookies are present
>         if (req.request == "GET" && req.http.cookie) {
>                 lookup;
>         }
>
>              if (req.http.host ~ "^hop.metatest.de") {
>                  set req.backend = thishost;
>              } elsif (req.http.host ~ "^zop.metatest.de") {
>                  set req.backend = thathost;
>              } else {
>                  error 404 "Unknown virtual host";
>              }
>
>
> }

You should select the backend *before* checking for POST or cookies.

> varnishlog gives me this output:
>
>    13 RxHeader     c Connection: keep-alive
>    13 RxHeader     c Host: zop.metatest.de
>    13 VCL_call     c recv
>    13 VCL_return   c lookup
>    13 VCL_call     c miss
>    13 VCL_return   c fetch
>    17 BackendClose   default
>    17 BackendOpen  b default 127.0.0.1 46677 127.0.0.1 7080
>
> So varnish goes to 127.0.0.1 instead of 192.168.100.199.

Most likely, the client sent a cookie.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From phk at phk.freebsd.dk  Wed Mar 14 09:59:29 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 14 Mar 2007 09:59:29 +0000
Subject: Varnish as an HTTP accelator for multiple Zope instances on
	multiple machines 
In-Reply-To: Your message of "Wed, 14 Mar 2007 06:40:06 +0100."
	<C2604441-3817-4975-817A-70AC5DB1C2B4@dirkgomez.de> 
Message-ID: <45859.1173866369@critter.freebsd.dk>

In message <C2604441-3817-4975-817A-70AC5DB1C2B4 at dirkgomez.de>, Dirk Gomez writ
es:

>varnishlog gives me this output:
>
>    13 RxHeader     c Connection: keep-alive
>    13 RxHeader     c Host: zop.metatest.de
>    13 VCL_call     c recv
>    13 VCL_return   c lookup
>    13 VCL_call     c miss
>    13 VCL_return   c fetch
>    17 BackendClose   default
>    17 BackendOpen  b default 127.0.0.1 46677 127.0.0.1 7080
>
>So varnish goes to 127.0.0.1 instead of 192.168.100.199.
>
>Any pointers what is wrong?

In cases like this, set the parameter vcl_trace to true
and varnish will log where it goes in the VCL code by
line and character:

   11 VCL_call     c recv
   11 VCL_trace    c 1 1.22
                     n l.c

n = serial number of token (ignore this)
l = line number
c = character in line

   11 VCL_trace    c 2 2.9
   11 VCL_trace    c 5 5.5
   11 VCL_trace    c 6 5.9
   11 VCL_trace    c 8 8.5
   11 VCL_trace    c 9 8.9
   11 VCL_trace    c 10 8.34
   11 VCL_trace    c 12 11.5
   11 VCL_return   c lookup

In this case it was the default VCL:

  1	sub default_vcl_recv {
	....5...10...15...20.^ = 1.22
  2	    if (req.request != \"GET\" && req.request != \"HEAD\") {
	....5...^ = 2.9
  3		pipe;
  4	    }
  5	    if (req.http.Expect) {
	....^ = 5.5
	....5...^ = 5.9
  6		pipe;
  7	    }
  8	    if (req.http.Authenticate || req.http.Cookie) {
	....^ = 8.5
	....5...^ = 8.9
	....5...10...15...20...25...30...^ = 8.34
  9		pass;
 10	    }
 11	    lookup;
	....^ = 11.5
	}


Remember to count TAB characters as one char only.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From lists at dirkgomez.de  Wed Mar 14 15:41:38 2007
From: lists at dirkgomez.de (Dirk Gomez)
Date: Wed, 14 Mar 2007 16:41:38 +0100
Subject: Varnish as an HTTP accelator for multiple Zope instances on
	multiple machines
In-Reply-To: <ujrk5xk424d.fsf@cat.linpro.no>
References: <C2604441-3817-4975-817A-70AC5DB1C2B4@dirkgomez.de>
	<ujrk5xk424d.fsf@cat.linpro.no>
Message-ID: <17C5189E-4499-4900-B91B-60BE41336F52@dirkgomez.de>

Dag, thanks a lot. That exactly solved my problem!

-- Dirk


From stonorn at giraffen.dk  Mon Mar 19 20:54:43 2007
From: stonorn at giraffen.dk (Anton Stonor)
Date: Mon, 19 Mar 2007 21:54:43 +0100
Subject: Checking for purge prevents caching
Message-ID: <45FEF893.6020102@giraffen.dk>

I don't know if this is a weird bug or my lack of vcl knowledge, so here 
we go:

I want this behavior: When somebody hits <ctrl>-F5 in a browser, varnish 
will purge the current page from the cache.

Firefox puts this in its request header when you do that:

Cache-Control: no-cache

So I inserted this piece of vcl:

--
sub vcl_hit {
   if(req.http.Cache-Control == "no-cache") {
     set obj.ttl = 0s;
     error 200 "Hey, we got a purge";
   }
}
--

At first glance that kind of works. When somebody hits <ctrl>-F5 the 
page gets purged and I get my custom error message.

However suddenly the page is not served from the cache _at all_. That is 
unless I just hit F5, _then_ I get a cached page. Hitting F5 sends this 
request header in Firefox:

Cache-Control: max-age=0

I don't send any caching headers on the backend if that matters, just 
relying on the default Varnish fallback to 120 sec.

Any ideas?


/Anton


From phk at phk.freebsd.dk  Sat Mar 24 22:29:30 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 24 Mar 2007 22:29:30 +0000
Subject: VCL compiler changes
Message-ID: <76939.1174775370@critter.freebsd.dk>


Those of you subscribed to the subversion commitlogs will have noticed
that I am working on the VCL compiler right now.

The two most recent changes is that multiple definitions of the
vcl methods are concatenated so that

	sub vcl_pipe {
		foo;
	}
	sub vcl_pipe {
		bar;
	}

is the same as

	sub vcl_pipe {
		foo;
		bar;
	}

Together with the other change, that adds

	include "somefile" ;

to the syntax, this should allow us to implement a sort of VCL
library which can easily be shared.

For instance, a vcl file implementing the squid purge functionality
could look like:

	sub vcl_recv {
		if (req.request == "PURGE") {
			lookup;
		}
	}

	sub vcl_hit {
		if (req.request == "PURGE") {
			set obj.ttl = 0s;
			error 200 "Purged.";
		}
	}

	sub vcl_miss {
		if (req.request == "PURGE") {
			error 404 "Not in cache.";
		}
	}

Wanting to apply an acl to the PURGE functionality, it could be
called from a VCL program as such:


	sub vcl_recv {
		if (req.request == "PURGE") {
			if (!client.ip ~ acl_purge) {
				error 405 "Not allowed.";
			}
		}
	}

	include "squid_purge.vcl";


If we come up with a set of such generally useful VCL snippets,
we could decide to add the "library" to the binary so tha tone
could just write:

	include "@squid_purge"

and have The Right Thing Happen.

As always: feedback and input most welcome!

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Wed Mar 28 07:45:33 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Wed, 28 Mar 2007 09:45:33 +0200
Subject: Checking for purge prevents caching
In-Reply-To: <45FEF893.6020102@giraffen.dk> (Anton Stonor's message of "Mon,
	19 Mar 2007 21:54:43 +0100")
References: <45FEF893.6020102@giraffen.dk>
Message-ID: <ujrr6r9reua.fsf@cat.linpro.no>

Anton Stonor <stonorn at giraffen.dk> writes:
> At first glance that kind of works. When somebody hits <ctrl>-F5 the
> page gets purged and I get my custom error message.
>
> However suddenly the page is not served from the cache _at all_. That
> is unless I just hit F5, _then_ I get a cached page. Hitting F5 sends
> this request header in Firefox:
>
> Cache-Control: max-age=0
>
> I don't send any caching headers on the backend if that matters, just
> relying on the default Varnish fallback to 120 sec.
>
> Any ideas?

Hard to tell without logs...

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From stonorn at giraffen.dk  Wed Mar 28 19:38:24 2007
From: stonorn at giraffen.dk (Anton Stonor)
Date: Wed, 28 Mar 2007 21:38:24 +0200
Subject: Checking for purge prevents caching
In-Reply-To: <ujrr6r9reua.fsf@cat.linpro.no>
References: <45FEF893.6020102@giraffen.dk> <ujrr6r9reua.fsf@cat.linpro.no>
Message-ID: <460AC430.6050309@giraffen.dk>

Dag-Erling Sm?rgrav wrote:

> Anton Stonor <stonorn at giraffen.dk> writes:
>> At first glance that kind of works. When somebody hits <ctrl>-F5 the
>> page gets purged and I get my custom error message.
>>
>> However suddenly the page is not served from the cache _at all_. That
>> is unless I just hit F5, _then_ I get a cached page. Hitting F5 sends
>> this request header in Firefox:
>>
>> Cache-Control: max-age=0
>>
>> I don't send any caching headers on the backend if that matters, just
>> relying on the default Varnish fallback to 120 sec.
>>
>> Any ideas?
> 
> Hard to tell without logs...

OK, here it is:


FIRST "NORMAL" REQUEST - it should be served from the cache
-----------------------------------------------------------

   0 StatAddr       127.0.0.1 0 264 2 6 0 0 2 1672 145
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1175105738
     0 CLI            Rd vcl.load boot /tmp/vcl.XXu1YKyr
     0 CLI            Wr 0 200 Loaded "/tmp/vcl.XXu1YKyr" as "boot"

     0 CLI            Rd vcl.use boot
     0 CLI            Wr 0 200
     0 CLI            Rd start
     0 CLI            Wr 0 200
     0 WorkThread     0xae04f1b4 start
    13 SessionOpen  c 127.0.0.1 42443
    13 ReqStart     c 127.0.0.1 42443 1440795409
    13 RxRequest    c GET
    13 RxURL        c /headerstuff
    13 RxProtocol   c HTTP/1.1
    13 RxHeader     c Host: localhost:9008
    13 RxHeader     c User-Agent: Mozilla/5.0 (X11; U; Linux i686; 
da-DK; rv:1.8.0.5) Gecko/20060731 Ubuntu/dapper-security Firefox/1.5.0.5
    13 RxHeader     c Accept: 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    13 RxHeader     c Accept-Language: da,en-us;q=0.7,en;q=0.3
    13 RxHeader     c Accept-Encoding: gzip,deflate
    13 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    13 RxHeader     c Keep-Alive: 300
    13 RxHeader     c Connection: keep-alive
    13 VCL_call     c recv
    13 VCL_return   c lookup
    13 VCL_call     c miss
    13 VCL_return   c fetch
    16 BackendOpen  b default 127.0.0.1 54436 127.0.0.1 8080
    16 BackendXID   b 1440795409
    13 Backend      c 16 default
    16 TxRequest    b GET
    16 TxURL        b /headerstuff
    16 TxProtocol   b HTTP/1.1
    16 TxHeader     b Host: localhost:9008
    16 TxHeader     b User-Agent: Mozilla/5.0 (X11; U; Linux i686; 
da-DK; rv:1.8.0.5) Gecko/20060731 Ubuntu/dapper-security Firefox/1.5.0.5
    16 TxHeader     b Accept: 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    16 TxHeader     b Accept-Language: da,en-us;q=0.7,en;q=0.3
    16 TxHeader     b Accept-Encoding: gzip,deflate
    16 TxHeader     b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    16 TxHeader     b X-Varnish: 1440795409
    16 TxHeader     b X-Forwarded-for: 127.0.0.1
    16 RxProtocol   b HTTP/1.1
    16 RxStatus     b 200
    16 RxResponse   b OK
    16 RxHeader     b Server: Zope/(Zope 2.10.2-final, python 2.4.3, 
linux2) ZServer/1.1
    16 RxHeader     b Date: Wed, 28 Mar 2007 18:15:40 GMT
    16 RxHeader     b Content-Length: 29
    16 RxHeader     b Content-Type: text/plain; charset=iso-8859-15
    13 TTL          c 1440795409 RFC 120 1175105740 1175105740 0 0 0
    13 VCL_call     c fetch
    13 VCL_return   c insert
    13 ObjProtocol  c HTTP/1.1
    13 ObjStatus    c 200
    13 ObjResponse  c OK
    13 ObjHeader    c Server: Zope/(Zope 2.10.2-final, python 2.4.3, 
linux2) ZServer/1.1
    13 ObjHeader    c Date: Wed, 28 Mar 2007 18:15:40 GMT
    13 ObjHeader    c Content-Type: text/plain; charset=iso-8859-15
    13 ObjHeader    c Content-Length: 29
    16 BackendReuse b default
    13 Length       c 29
    13 TxProtocol   c HTTP/1.1
    13 TxStatus     c 200
    13 TxResponse   c OK
    13 TxHeader     c Server: Zope/(Zope 2.10.2-final, python 2.4.3, 
linux2) ZServer/1.1
    13 TxHeader     c Date: Wed, 28 Mar 2007 18:15:40 GMT
    13 TxHeader     c Content-Type: text/plain; charset=iso-8859-15
    13 TxHeader     c Content-Length: 29
    13 TxHeader     c X-Varnish: 1440795409
    13 TxHeader     c Age: 0
    13 TxHeader     c Via: 1.1 varnish
    13 ReqEnd       c 1440795409 1175105740.472784000 
1175105740.642282000 0.000369000 0.009144000 0.160354000


SECOND "NORMAL" REQUEST - now, it should really be in the cache
---------------------------------------------------------------

     0 StatAddr       127.0.0.1 0 0 1 1 0 0 1 240 29
     0 CLI            Rd vcl.load boot /tmp/vcl.XXu1YKyr
     0 CLI            Wr 0 200 Loaded "/tmp/vcl.XXu1YKyr" as "boot"

     0 CLI            Rd vcl.use boot
     0 CLI            Wr 0 200
     0 CLI            Rd start
     0 CLI            Wr 0 200
     0 WorkThread     0xae04f1b4 start
    13 SessionOpen  c 127.0.0.1 42445
    13 ReqStart     c 127.0.0.1 42445 555485910
    13 RxRequest    c GET
    13 RxURL        c /headerstuff
    13 RxProtocol   c HTTP/1.1
    13 RxHeader     c Host: localhost:9008
    13 RxHeader     c User-Agent: Mozilla/5.0 (X11; U; Linux i686; 
da-DK; rv:1.8.0.5) Gecko/20060731 Ubuntu/dapper-security Firefox/1.5.0.5
    13 RxHeader     c Accept: 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    13 RxHeader     c Accept-Language: da,en-us;q=0.7,en;q=0.3
    13 RxHeader     c Accept-Encoding: gzip,deflate
    13 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    13 RxHeader     c Keep-Alive: 300
    13 RxHeader     c Connection: keep-alive
    13 VCL_call     c recv
    13 VCL_return   c lookup
    13 VCL_call     c miss
    13 VCL_return   c fetch
    16 BackendOpen  b default 127.0.0.1 54438 127.0.0.1 8080
    16 BackendXID   b 555485910
    13 Backend      c 16 default
    16 TxRequest    b GET
    16 TxURL        b /headerstuff
    16 TxProtocol   b HTTP/1.1
    16 TxHeader     b Host: localhost:9008
    16 TxHeader     b User-Agent: Mozilla/5.0 (X11; U; Linux i686; 
da-DK; rv:1.8.0.5) Gecko/20060731 Ubuntu/dapper-security Firefox/1.5.0.5
    16 TxHeader     b Accept: 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    16 TxHeader     b Accept-Language: da,en-us;q=0.7,en;q=0.3
    16 TxHeader     b Accept-Encoding: gzip,deflate
    16 TxHeader     b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    16 TxHeader     b X-Varnish: 555485910
    16 TxHeader     b X-Forwarded-for: 127.0.0.1
    16 RxProtocol   b HTTP/1.1
    16 RxStatus     b 200
    16 RxResponse   b OK
    16 RxHeader     b Server: Zope/(Zope 2.10.2-final, python 2.4.3, 
linux2) ZServer/1.1
    16 RxHeader     b Date: Wed, 28 Mar 2007 18:15:41 GMT
    16 RxHeader     b Content-Length: 29
    16 RxHeader     b Content-Type: text/plain; charset=iso-8859-15
    13 TTL          c 555485910 RFC 120 1175105741 1175105741 0 0 0
    13 VCL_call     c fetch
    13 VCL_return   c insert
    13 ObjProtocol  c HTTP/1.1
    13 ObjStatus    c 200
    13 ObjResponse  c OK
    13 ObjHeader    c Server: Zope/(Zope 2.10.2-final, python 2.4.3, 
linux2) ZServer/1.1
    13 ObjHeader    c Date: Wed, 28 Mar 2007 18:15:41 GMT
    13 ObjHeader    c Content-Type: text/plain; charset=iso-8859-15
    13 ObjHeader    c Content-Length: 29
    16 BackendReuse b default
    13 Length       c 29
    13 TxProtocol   c HTTP/1.1
    13 TxStatus     c 200
    13 TxResponse   c OK
    13 TxHeader     c Server: Zope/(Zope 2.10.2-final, python 2.4.3, 
linux2) ZServer/1.1
    13 TxHeader     c Date: Wed, 28 Mar 2007 18:15:41 GMT
    13 TxHeader     c Content-Type: text/plain; charset=iso-8859-15
    13 TxHeader     c Content-Length: 29
    13 TxHeader     c X-Varnish: 555485910
    13 TxHeader     c Age: 0
    13 TxHeader     c Via: 1.1 varnish
    13 ReqEnd       c 555485910 1175105741.342312000 
1175105741.518589000 0.000377000 0.007488000 0.168789000


JUST PRESS F5 - we get it from the cache
-----------------------------------------

     0 StatAddr       127.0.0.1 0 0 1 1 0 0 1 240 29
    13 ReqStart     c 127.0.0.1 42447 1020536477
    13 RxRequest    c GET
    13 RxURL        c /headerstuff
    13 RxProtocol   c HTTP/1.1
    13 RxHeader     c Host: localhost:9008
    13 RxHeader     c User-Agent: Mozilla/5.0 (X11; U; Linux i686; 
da-DK; rv:1.8.0.5) Gecko/20060731 Ubuntu/dapper-security Firefox/1.5.0.5
    13 RxHeader     c Accept: 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
    13 RxHeader     c Accept-Language: da,en-us;q=0.7,en;q=0.3
    13 RxHeader     c Accept-Encoding: gzip,deflate
    13 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    13 RxHeader     c Keep-Alive: 300
    13 RxHeader     c Connection: keep-alive
    13 RxHeader     c Cache-Control: max-age=0
    13 VCL_call     c recv
    13 VCL_return   c lookup
    13 Hit          c 1020536476
    13 VCL_call     c hit
    13 VCL_return   c deliver
    13 Length       c 29
    13 TxProtocol   c HTTP/1.1
    13 TxStatus     c 200
    13 TxResponse   c OK
    13 TxHeader     c Server: Zope/(Zope 2.10.2-final, python 2.4.3, 
linux2) ZServer/1.1
    13 TxHeader     c Date: Wed, 28 Mar 2007 18:15:43 GMT
    13 TxHeader     c Content-Type: text/plain; charset=iso-8859-15
    13 TxHeader     c Content-Length: 29
    13 TxHeader     c X-Varnish: 1020536477 1020536476
    13 TxHeader     c Age: 3
    13 TxHeader     c Via: 1.1 varnish
    13 ReqEnd       c 1020536477 1175105746.436478000 
1175105746.561595000 2.482832000 0.000047000 0.125070000



From cnkang at gmail.com  Thu Mar 29 05:06:41 2007
From: cnkang at gmail.com (Kang Liu)
Date: Thu, 29 Mar 2007 13:06:41 +0800
Subject: vcl file for caching flash video?
Message-ID: <8e98d9eb0703282206i7b567cfenf6d9427e69fa9301@mail.gmail.com>

Hi,
I'm testing varnish for caching flash video content for a youtube
style site. I wrote a very simple vcl file for the site.
the backend is lighttpd, contents in the backend are all flash videos
(.flv files).
Is there any problem in my vcl file?

backend default {
    set backend.host = "IP address";
    set backend.port = "http";
}

sub vcl_recv {
    if (req.request != "GET" && req.request != "HEAD") {
        pipe;
    }
    if (req.http.Expect) {
        pipe;
    }
    lookup;
}//no cookie, no auth needed; just look up the cache...

sub vcl_hit {
    deliver;
}//deliver each hit from cache

sub vcl_miss {
    fetch;
}

sub vcl_fetch {
    if (!obj.valid) {
        error;
    }
    set obj.ttl = 31536000s;
    insert;
}//flash video never change, so cache everything for 1 year

sub vcl_timeout {
    discard;
}


From phk at phk.freebsd.dk  Thu Mar 29 05:39:42 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 29 Mar 2007 05:39:42 +0000
Subject: vcl file for caching flash video? 
In-Reply-To: Your message of "Thu, 29 Mar 2007 13:06:41 +0800."
	<8e98d9eb0703282206i7b567cfenf6d9427e69fa9301@mail.gmail.com> 
Message-ID: <12166.1175146782@critter.freebsd.dk>

In message <8e98d9eb0703282206i7b567cfenf6d9427e69fa9301 at mail.gmail.com>, "Kang
 Liu" writes:
>Hi,
>I'm testing varnish for caching flash video content for a youtube
>style site. I wrote a very simple vcl file for the site.
>the backend is lighttpd, contents in the backend are all flash videos
>(.flv files).
>Is there any problem in my vcl file?

None that I see.

You don't need to copy the default VCL, it's always used if you
have not specified anything, so you should not need the vcl_hit, 
vcl_miss and vcl_timeout functions.

VCL understands more time units than seconds, so you could have
written:
	set obj.ttl = 365d;

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From phk at phk.freebsd.dk  Thu Mar 29 07:18:18 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 29 Mar 2007 07:18:18 +0000
Subject: Checking for purge prevents caching 
In-Reply-To: Your message of "Wed, 28 Mar 2007 21:38:24 +0200."
	<460AC430.6050309@giraffen.dk> 
Message-ID: <12620.1175152698@critter.freebsd.dk>

In message <460AC430.6050309 at giraffen.dk>, Anton Stonor writes:
>Dag-Erling Sm?rgrav wrote:
>
>OK, here it is:

I don't see anything wrong in this logfile ?

>SECOND "NORMAL" REQUEST - now, it should really be in the cache
>---------------------------------------------------------------
>
>     0 StatAddr       127.0.0.1 0 0 1 1 0 0 1 240 29
>     0 CLI            Rd vcl.load boot /tmp/vcl.XXu1YKyr
>     0 CLI            Wr 0 200 Loaded "/tmp/vcl.XXu1YKyr" as "boot"

You restarted varnish here, didn't you ?


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From stonorn at giraffen.dk  Thu Mar 29 13:21:36 2007
From: stonorn at giraffen.dk (Anton Stonor)
Date: Thu, 29 Mar 2007 15:21:36 +0200
Subject: Checking for purge prevents caching
In-Reply-To: <12620.1175152698@critter.freebsd.dk>
References: <12620.1175152698@critter.freebsd.dk>
Message-ID: <460BBD60.5030601@giraffen.dk>

Poul-Henning Kamp wrote:

>> SECOND "NORMAL" REQUEST - now, it should really be in the cache
>> ---------------------------------------------------------------
>>
>>     0 StatAddr       127.0.0.1 0 0 1 1 0 0 1 240 29
>>     0 CLI            Rd vcl.load boot /tmp/vcl.XXu1YKyr
>>     0 CLI            Wr 0 200 Loaded "/tmp/vcl.XXu1YKyr" as "boot"
> 
> You restarted varnish here, didn't you ?

Ah that is a restart? Nope, I didn't restart anything.

These lines appear on every "normal" request except for f5 and ctrl-f5. 
When I remove the VCL, Varnish does not restart, but are running smooth.

What is the best way to dig deeper into why the restarts happen under 
this condition?

/Anton






From dwetzel at nerim.net  Thu Mar 29 16:20:48 2007
From: dwetzel at nerim.net (Damien Wetzel)
Date: Thu, 29 Mar 2007 18:20:48 +0200
Subject: log files
Message-ID: <17931.59232.890894.380967@dwetzel@nerim.net>

Hello,
could someone tell me if the shmlogs are saved in a textfile and
where is it ?
Thanks,
-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Damien WETZEL                                   ("`-/")_.-'"``-._
ATANAR TECHNOLOGIES                              . . `; -._    )-;-,_`)
                                               (v_,)'  _  )`-.\  ``-'
Phone:+33 1 45 43 02 90                        _.- _..-_/ / ((.'
- So much to do, so little time -            ((,.-'   ((,/            
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



From des at linpro.no  Thu Mar 29 16:56:37 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Thu, 29 Mar 2007 18:56:37 +0200
Subject: log files
In-Reply-To: <17931.59232.890894.380967@dwetzel@nerim.net> (Damien Wetzel's
	message of "Thu, 29 Mar 2007 18:20:48 +0200")
References: <17931.59232.890894.380967@dwetzel@nerim.net>
Message-ID: <ujrvegkng3e.fsf@cat.linpro.no>

Damien Wetzel <dwetzel at nerim.net> writes:
> could someone tell me if the shmlogs are saved in a textfile and
> where is it ?

No, you have to run varnishlog and / or varnishncsa yourself to
capture the logs to a file.  For the next release, I will work with
the packagers to make sure binary packages ship with an init script
which starts varnishlog.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From dwetzel at nerim.net  Thu Mar 29 17:49:19 2007
From: dwetzel at nerim.net (Damien Wetzel)
Date: Thu, 29 Mar 2007 19:49:19 +0200
Subject: log files
In-Reply-To: <ujrvegkng3e.fsf@cat.linpro.no>
References: <17931.59232.890894.380967@dwetzel@nerim.net>
	<ujrvegkng3e.fsf@cat.linpro.no>
Message-ID: <17931.64543.652972.864603@dwetzel@nerim.net>

thanks,
i think it could be interresting to add a field in the ncsa logs telling
if the object was delivered from the cache or had to be fetched.
Is there a way to get w3C log ?
and last is there a ay to remove ping pong lines from the logs ?
Damien,
Dag-Erling Sm?rgrav writes:
 > Damien Wetzel <dwetzel at nerim.net> writes:
 > > could someone tell me if the shmlogs are saved in a textfile and
 > > where is it ?
 > 
 > No, you have to run varnishlog and / or varnishncsa yourself to
 > capture the logs to a file.  For the next release, I will work with
 > the packagers to make sure binary packages ship with an init script
 > which starts varnishlog.
 > 
 > DES
 > -- 
 > Dag-Erling Sm?rgrav
 > Senior Software Developer
 > Linpro AS - www.linpro.no



From phk at phk.freebsd.dk  Thu Mar 29 21:50:50 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 29 Mar 2007 21:50:50 +0000
Subject: Checking for purge prevents caching 
In-Reply-To: Your message of "Thu, 29 Mar 2007 15:21:36 +0200."
	<460BBD60.5030601@giraffen.dk> 
Message-ID: <15503.1175205050@critter.freebsd.dk>

In message <460BBD60.5030601 at giraffen.dk>, Anton Stonor writes:

>What is the best way to dig deeper into why the restarts happen under 
>this condition?

Run varnishd with "-d -d" options and watch what it says

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Fri Mar 30 07:00:18 2007
From: des at linpro.no (Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?=)
Date: Fri, 30 Mar 2007 09:00:18 +0200
Subject: log files
In-Reply-To: <17931.64543.652972.864603@dwetzel@nerim.net> (Damien Wetzel's
	message of "Thu, 29 Mar 2007 19:49:19 +0200")
References: <17931.59232.890894.380967@dwetzel@nerim.net>
	<ujrvegkng3e.fsf@cat.linpro.no>
	<17931.64543.652972.864603@dwetzel@nerim.net>
Message-ID: <ujrhcs3nrlp.fsf@cat.linpro.no>

Damien Wetzel <dwetzel at nerim.net> writes:
> i think it could be interresting to add a field in the ncsa logs telling
> if the object was delivered from the cache or had to be fetched.
> Is there a way to get w3C log ?

That's what varnishncsa does.

> and last is there a ay to remove ping pong lines from the logs ?

All the information you need is in the man page.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From dwetzel at nerim.net  Fri Mar 30 13:45:37 2007
From: dwetzel at nerim.net (Damien Wetzel)
Date: Fri, 30 Mar 2007 15:45:37 +0200
Subject: log files
In-Reply-To: <ujrhcs3nrlp.fsf@cat.linpro.no>
References: <17931.59232.890894.380967@dwetzel@nerim.net>
	<ujrvegkng3e.fsf@cat.linpro.no>
	<17931.64543.652972.864603@dwetzel@nerim.net>
	<ujrhcs3nrlp.fsf@cat.linpro.no>
Message-ID: <17933.5249.247592.184781@dwetzel@nerim.net>

Dag-Erling Sm?rgrav writes:
 > Damien Wetzel <dwetzel at nerim.net> writes:
 > > i think it could be interresting to add a field in the ncsa logs telling
 > > if the object was delivered from the cache or had to be fetched.
 > > Is there a way to get w3C log ?
 > 
 > That's what varnishncsa does.
i haven't seen any fields using varnishncsa which refers to cache hits or miss
the man talk about the possibility to get the logs in combined format but
i don't how to do it

 > 
 > > and last is there a ay to remove ping pong lines from the logs ?
 > 
 > All the information you need is in the man page.
it's my bad the man pages weren't installed after the make install command.

PS: I don't think ESI for a v2 are a good addon.
ESI is mainly an akamai tool to catch customers but is not of real use.
the standard hasn't changed since 2002 and doesn't seem to be widely used.

 > 
 > DES
 > -- 
 > Dag-Erling Sm?rgrav
 > Senior Software Developer
 > Linpro AS - www.linpro.no



From olivier.dobberkau at dkd.de  Fri Mar 30 19:40:49 2007
From: olivier.dobberkau at dkd.de (Olivier Dobberkau)
Date: Fri, 30 Mar 2007 21:40:49 +0200
Subject: Questions about Setup
Message-ID: <C2333461.35210%olivier.dobberkau@dkd.de>

Hello!

I just started to look into varnish. We want to use it with TYPO3 as varnish
seems to be quite powerfull.

I have some question about the initial setup and hope to can shed some light
on me.

As far as i understood varnish can bei used in different setups:

A) On a single Server. Listening on Port 80 forwarding to localhost on a
special port.

B) On a server forwarding to different backends.

My Question, as I did not really understand the vcl conf stuff, can someone
post his vcl conf with some more explanations for one of the mentioned
szenarios.

Thanks a lot in advance.

Olivier

PS: Does anyone a working binary for macos x.. But that is different story
for now.
-- 

Olivier Dobberkau
. . . . . . . . . . . . . .
Je TYPO3, desto d.k.d

d.k.d Internet Service GmbH
Kaiserstr. 79
60329 Frankfurt/Main

Registergericht: Amtsgericht Frankfurt am Main
Registernummer: HRB 45590
Gesch?ftsf?hrer: Olivier Dobberkau, G?tz Wegenast.

fon:  069 - 43 05 61-70
fax:  069 - 43 05 61-90
mail: olivier.dobberkau at dkd.de
home: http://www.dkd.de


aktuelle TYPO3-Projekte:
www.dosb.de - Relaunch
www.gesundheit.de - Relaunch
www.wwf.de - barrierefrei, Relaunch


From denis at startsiden.no  Sat Mar 31 09:48:32 2007
From: denis at startsiden.no (=?utf-8?Q?Denis_Br=C3=A6khus?=)
Date: Sat, 31 Mar 2007 11:48:32 +0200 (CEST)
Subject: Questions about Setup
In-Reply-To: <C2333461.35210%olivier.dobberkau@dkd.de>
Message-ID: <19969305.21711175334512466.JavaMail.root@ms1.startsiden.no>

----- Olivier Dobberkau <olivier.dobberkau at dkd.de> wrote:
> As far as i understood varnish can bei used in different setups:
> 
> A) On a single Server. Listening on Port 80 forwarding to localhost on
> a special port.
> 
> B) On a server forwarding to different backends.

You can make varnish listen on any port you want really, but port 80 will obviously be the right one for most cases.
Varnish can forward to any number of backends, be it localhost or any other server (doesn't matter too much for Varnish).

In my case we have two varnish front end servers that each hand off requests to one of two backend servers. (The reason for having two varnish servers is simply fail-over considerations, from my observations one of them could easily handle all of the load by itself.)


> My Question, as I did not really understand the vcl conf stuff, can
> someone post his vcl conf with some more explanations for one of the
> mentioned szenarios.

Maybe it would be better if you asked questions about the things you do not understand? Our config is mainly the normal config, add the backend block, and some specific exceptions for site urls we are sure we do not want cached in any way.

Regards
-- 
Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS
http://www.startsiden.no



From asjo at koldfront.dk  Sat Mar 31 15:20:14 2007
From: asjo at koldfront.dk (Adam =?iso-8859-1?Q?Sj=F8gren?=)
Date: Sat, 31 Mar 2007 17:20:14 +0200
Subject: Virtual hosts and logfiles
Message-ID: <87mz1tfnip.fsf@topper.koldfront.dk>

  Hi.


For fun I'm experimenting with running Varnish in front of an Apache
that runs some virtual hosts; it looks like it is working very well so
far.

Having Varnish in front means that Apache logs the local IP/hostname
for all requests, which makes the logs somewhat less useful than
before.

I wonder if anyone has solved how to make varnishncsa only log
requests to a specific virtualhost? (Then I would just run a
varnishncsa for each virtualhost; problem solved).

Looking at the man-page, I'm guessing that the '-i' flag perhaps could
be used, but I haven't been able to dig up what "tag" means in this
context. Maybe I'm way off.

Any pointers where to look?


  Best regards,

     Adam

-- 
 "Q: Who tells you what to do?                                Adam Sj?gren
  A: The Bell Telephone Company. The telephone rings     asjo at koldfront.dk
     and I do what I'm told to do."



From phk at phk.freebsd.dk  Sat Mar 31 16:19:35 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 31 Mar 2007 16:19:35 +0000
Subject: Virtual hosts and logfiles 
In-Reply-To: Your message of "Sat, 31 Mar 2007 17:20:14 +0200."
	<87mz1tfnip.fsf@topper.koldfront.dk> 
Message-ID: <81146.1175357975@critter.freebsd.dk>

In message <87mz1tfnip.fsf at topper.koldfront.dk>, Adam =?iso-8859-1?Q?Sj=F8gren?= writes:

>I wonder if anyone has solved how to make varnishncsa only log
>requests to a specific virtualhost? (Then I would just run a
>varnishncsa for each virtualhost; problem solved).

I am not sure you can do that with varnishncsa at present, but it
should probably grow support for that.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From asjo at koldfront.dk  Sat Mar 31 16:35:06 2007
From: asjo at koldfront.dk (Adam =?iso-8859-1?Q?Sj=F8gren?=)
Date: Sat, 31 Mar 2007 18:35:06 +0200
Subject: Delay when fetching a page without Content-Length
Message-ID: <87abxtfk1x.fsf@topper.koldfront.dk>

In my experiments, I've run into a long(ish) delay when I ask for a
url behind http auth:

 $ time wget -S http://www.koldfront.local/stats/
 [...?]
 real    0m16.799s
 [...]

Going directly to Apache:

 $ time wget -S http://www.koldfront.local:8080/stats/
 [...?]
 real    0m0.013s
 [...]

I have previously observed such delays on pages where I didn't return
a Content-Length header (or, embarrassingly, did it wrong) - is this
related, or should it just work when Apache sends 'Transfer-Encoding:
chunked'?


(I have configured Varnish to 'pass' if there is an WWW-Authenticate
or Authorization-header? - I hope that is correct).


  Best regards,

    Adam


?  --18:27:24--  http://www.koldfront.local/stats/
            => `index.html'
 Resolving www.koldfront.local... 192.168.1.111
 Connecting to www.koldfront.local|192.168.1.111|:80... connected.
 HTTP request sent, awaiting response... 
   HTTP/1.1 401 Authorization Required
   Date: Sat, 31 Mar 2007 16:27:26 GMT
   Server: Apache/1.3.33 (Debian GNU/Linux) mod_fastcgi/2.4.2 mod_perl/1.29
   WWW-Authenticate: Basic realm="Statistics"
   Content-Type: text/html; charset=iso-8859-1
   X-Varnish: 1997136530
   X-Forwarded-for: 192.168.1.160
   Transfer-Encoding: chunked
 Authorization failed.

?  --18:28:14--  http://www.koldfront.local:8080/stats/
            => `index.html'
 Resolving www.koldfront.local... 192.168.1.111
 Connecting to www.koldfront.local|192.168.1.111|:8080... connected.
 HTTP request sent, awaiting response... 
   HTTP/1.1 401 Authorization Required
   Date: Sat, 31 Mar 2007 16:28:17 GMT
   Server: Apache/1.3.33 (Debian GNU/Linux) mod_fastcgi/2.4.2 mod_perl/1.29
   WWW-Authenticate: Basic realm="Statistics"
   Connection: close
   Content-Type: text/html; charset=iso-8859-1
   X-Pad: avoid browser bug
 Authorization failed.

? <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=416940>

-- 
 "Subdued flamboyance"                                        Adam Sj?gren
                                                         asjo at koldfront.dk



From phk at phk.freebsd.dk  Sat Mar 31 17:17:10 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 31 Mar 2007 17:17:10 +0000
Subject: Delay when fetching a page without Content-Length 
In-Reply-To: Your message of "Sat, 31 Mar 2007 18:35:06 +0200."
	<87abxtfk1x.fsf@topper.koldfront.dk> 
Message-ID: <81364.1175361430@critter.freebsd.dk>


Hi Adam,

Which version of Varnish are you running ?

Can you send me the output from varnishlog for the transaction ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From asjo at koldfront.dk  Sat Mar 31 18:11:45 2007
From: asjo at koldfront.dk (Adam =?iso-8859-1?Q?Sj=F8gren?=)
Date: Sat, 31 Mar 2007 20:11:45 +0200
Subject: Delay when fetching a page without Content-Length
References: <87abxtfk1x.fsf@topper.koldfront.dk>
	<81364.1175361430@critter.freebsd.dk>
Message-ID: <87648hffku.fsf@topper.koldfront.dk>

On Sat, 31 Mar 2007 17:17:10 +0000, Poul-Henning wrote:

> Which version of Varnish are you running ?

The one from Debian unstable backported to stable, i.e. 1.0.3:

 $ /usr/sbin/varnishd -V
 varnishd (varnish-1.0.3)
 Copyright (c) 2006 Linpro AS / Verdens Gang AS
 $ 

> Can you send me the output from varnishlog for the transaction ?

Sure:

 $ varnishlog
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1175364221
    13 SessionOpen  c 192.168.1.160 52579
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1175364224
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1175364227
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1175364230
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1175364233
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1175364236
    13 ReqStart     c 192.168.1.160 52579 1997137143
    13 RxRequest    c GET
    13 RxURL        c /stats/
    13 RxProtocol   c HTTP/1.0
    13 RxHeader     c User-Agent: Wget/1.10.2
    13 RxHeader     c Accept: */*
    13 RxHeader     c Host: www.koldfront.local
    13 RxHeader     c Connection: Keep-Alive
    13 VCL_call     c recv
    13 VCL_return   c lookup
    13 VCL_call     c miss
    13 VCL_return   c fetch
    17 BackendOpen  b default 127.0.0.1 1027 127.0.0.1 8080
    17 BackendXID   b 1997137143
    13 Backend      c 17 default
    17 TxRequest    b GET
    17 TxURL        b /stats/
    17 TxProtocol   b HTTP/1.1
    17 TxHeader     b User-Agent: Wget/1.10.2
    17 TxHeader     b Accept: */*
    17 TxHeader     b Host: www.koldfront.local
    17 TxHeader     b X-Varnish: 1997137143
    17 TxHeader     b X-Forwarded-for: 192.168.1.160
    17 RxProtocol   b HTTP/1.1
    17 RxStatus     b 401
    17 RxResponse   b Authorization Required
    17 RxHeader     b Date: Sat, 31 Mar 2007 18:03:42 GMT
    17 RxHeader     b Server: Apache/1.3.33 (Debian GNU/Linux) mod_fastcgi/2.4.2 mod_perl/1.29
    17 RxHeader     b WWW-Authenticate: Basic realm="Statistics"
    17 RxHeader     b Transfer-Encoding: chunked
    17 RxHeader     b Content-Type: text/html; charset=iso-8859-1
    13 TTL          c 1997137143 RFC 120 1175364222 1175364222 0 0 0
    13 VCL_call     c fetch
    13 VCL_return   c pass
    13 RxProtocol   c HTTP/1.1
    13 RxStatus     c 401
    13 RxResponse   c Authorization Required
    13 RxHeader     c Date: Sat, 31 Mar 2007 18:03:42 GMT
    13 RxHeader     c Server: Apache/1.3.33 (Debian GNU/Linux) mod_fastcgi/2.4.2 mod_perl/1.29
    13 RxHeader     c WWW-Authenticate: Basic realm="Statistics"
    13 RxHeader     c Content-Type: text/html; charset=iso-8859-1
    13 RxHeader     c X-Varnish: 1997137143
    13 RxHeader     c X-Forwarded-for: 192.168.1.160
    13 RxHeader     c Transfer-Encoding: chunked
    17 BackendReuse b default
    13 ReqEnd       c 1997137143 1175364222.294967044 1175364238.516689601 0.000751112 0.001383653 16.220338904
     0 StatAddr       192.168.1.160 0 8161 36 37 0 20 11 11486 92507
    13 HttpError    c Received errno 104
    13 SessionClose c no request
    13 StatSess     c 192.168.1.160 52579 16 1 1 0 1 0 322 529
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1175364239
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1175364242
 ^C
 $ 

(If I run ngrep I see the request being sent to Apache and the reply
back immediately, and then the pause).


Let me know if I should supply more information.


  Best regards,

    Adam

-- 
 "we push onward. to you, it is 2005, to us, it is            Adam Sj?gren
  2011. we are always far ahead."                        asjo at koldfront.dk



From phk at phk.freebsd.dk  Sat Mar 31 18:18:36 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 31 Mar 2007 18:18:36 +0000
Subject: Delay when fetching a page without Content-Length 
In-Reply-To: Your message of "Sat, 31 Mar 2007 20:11:45 +0200."
	<87648hffku.fsf@topper.koldfront.dk> 
Message-ID: <86144.1175365116@critter.freebsd.dk>

In message <87648hffku.fsf at topper.koldfront.dk>, Adam =?iso-8859-1?Q?Sj=F8gren?= writes:
>On Sat, 31 Mar 2007 17:17:10 +0000, Poul-Henning wrote:
>
>> Which version of Varnish are you running ?
>
>The one from Debian unstable backported to stable, i.e. 1.0.3:
>
>    13 VCL_call     c fetch
>    13 VCL_return   c pass

Hmm, you're running into a bug in the pass implementation which
have since been fixed.

Try using pipe mode for authenticated requests instead, by putting
this in your vcl_recv function:

	if (req.http.Authenticate || req.http.Cookie) {
		pipe;
	}
 


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From asjo at koldfront.dk  Sat Mar 31 18:35:59 2007
From: asjo at koldfront.dk (Adam =?iso-8859-1?Q?Sj=F8gren?=)
Date: Sat, 31 Mar 2007 20:35:59 +0200
Subject: Delay when fetching a page without Content-Length
References: <87648hffku.fsf@topper.koldfront.dk>
	<86144.1175365116@critter.freebsd.dk>
Message-ID: <87veghdzw0.fsf@topper.koldfront.dk>

On Sat, 31 Mar 2007 18:18:36 +0000, Poul-Henning wrote:

> Hmm, you're running into a bug in the pass implementation which
> have since been fixed.

Ah, great.

(Where can I read about meaning of these? I have scanned through
vcl(7), varnishd(1) and the FAQ).

> Try using pipe mode for authenticated requests instead, by putting
> this in your vcl_recv function:

> 	if (req.http.Authenticate || req.http.Cookie) {
> 		pipe;
> 	}

Hm. I think I have done something wrong, because that made it worse:

 $ time wget -S http://www.koldfront.dk/stats/
 --20:28:26--  http://www.koldfront.dk/stats/
            => `index.html'
 Resolving www.koldfront.dk... 192.168.1.111
 Connecting to www.koldfront.dk|192.168.1.111|:80... connected.
 HTTP request sent, awaiting response... No data received.
 Retrying.

 --20:28:27--  http://www.koldfront.dk/stats/
   (try: 2) => `index.html'
 Connecting to www.koldfront.dk|192.168.1.111|:80... connected.
 HTTP request sent, awaiting response... No data received.
 Retrying.
 [...]

 $ varnishlog
     0 CLI            Rd ping
     0 CLI            Wr 0 200 PONG 1175365706
     0 WorkThread     0xb4391830 start
    13 SessionOpen  c 192.168.1.160 40834
     0 Debug          "INCOMPLETE AT: cnt_fetch(332)"
     0 CLI            Rd vcl.load boot /tmp/vcl.XXZGJC2j
     0 CLI            Wr 0 200 Loaded "/tmp/vcl.XXZGJC2j" as "boot"

     0 CLI            Rd vcl.use boot
     0 CLI            Wr 0 200 
     0 CLI            Rd start
     0 CLI            Wr 0 200 
     0 WorkThread     0xb4391830 start
    13 SessionOpen  c 192.168.1.160 40835
     0 Debug          "INCOMPLETE AT: cnt_fetch(332)"
     0 CLI            Rd vcl.load boot /tmp/vcl.XXZGJC2j
     0 CLI            Wr 0 200 Loaded "/tmp/vcl.XXZGJC2j" as "boot"

The modified vcl.conf I was using looks like this:

 # This is a basic vcl.conf file for varnish.
 # Modifying this file should be where you store your modifications to
 # varnish. Settnigs here will override defaults.

 backend default {
         set backend.host = "127.0.0.1";
         set backend.port = "8080";       
 }

 sub vcl_recv {
         if (req.http.authorization) {
 #               pass;
                 pipe;
         }

         if (req.request == "POST") {
                 pipe;
         }

         # force lookup even when cookies are present
         #if (req.request == "GET" && req.http.cookie) {
         #        lookup;
         #}
 }

 sub vcl_fetch {
         if (resp.http.www-authenticate) {
 #               pass;
                 pipe;
         }

         # force minimum ttl of 180 seconds
         if (obj.ttl < 180s) {
                 set obj.ttl = 180s;
         }
 }


I will check out from svn and give that a spin...


  Best regards,

    Adam

-- 
 "Fr?n och med nu s? ?r 's? snart som m?jligt' 53             Adam Sj?gren
  timmar!"                                               asjo at koldfront.dk



From phk at phk.freebsd.dk  Sat Mar 31 19:22:37 2007
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 31 Mar 2007 19:22:37 +0000
Subject: Delay when fetching a page without Content-Length 
In-Reply-To: Your message of "Sat, 31 Mar 2007 20:35:59 +0200."
	<87veghdzw0.fsf@topper.koldfront.dk> 
Message-ID: <86505.1175368957@critter.freebsd.dk>

In message <87veghdzw0.fsf at topper.koldfront.dk>, Adam =?iso-8859-1?Q?Sj=F8gren?= writes:
>On Sat, 31 Mar 2007 18:18:36 +0000, Poul-Henning wrote:
>
>> Hmm, you're running into a bug in the pass implementation which
>> have since been fixed.
>
>Ah, great.
>
>(Where can I read about meaning of these? I have scanned through
>vcl(7), varnishd(1) and the FAQ).
>
>> Try using pipe mode for authenticated requests instead, by putting
>> this in your vcl_recv function:
>
>> 	if (req.http.Authenticate || req.http.Cookie) {
>> 		pipe;
>> 	}
>
>Hm. I think I have done something wrong, because that made it worse:
>
> sub vcl_recv {
>         if (req.http.authorization) {

Should have been:  req.http.authenticate

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From asjo at koldfront.dk  Sat Mar 31 19:44:14 2007
From: asjo at koldfront.dk (Adam =?iso-8859-1?Q?Sj=F8gren?=)
Date: Sat, 31 Mar 2007 21:44:14 +0200
Subject: Delay when fetching a page without Content-Length
References: <87veghdzw0.fsf@topper.koldfront.dk>
	<86505.1175368957@critter.freebsd.dk>
Message-ID: <87ejn5dwq9.fsf@topper.koldfront.dk>

On Sat, 31 Mar 2007 19:22:37 +0000, Poul-Henning wrote:

>> if (req.http.authorization) {

> Should have been:  req.http.authenticate

Well, no such header is ever sent, from what I can see, but I'll give
it a go.

No difference - still ~16s delay.


First the browser asks for the page and the server answers with
WWW-Authenticate: Basic realm="...". The browser then asks me for
username and password, and goes on to retry the request, this time
with an Authorization: Basic ... header added.

The delay occurs on the first request.


  Best regards,

     Adam

-- 
 "Yakka foob mog. Grug pubbawup zink wattoom gazork.          Adam Sj?gren
  Chumble spuzz."                                        asjo at koldfront.dk