From lluo_2wire at yahoo.com  Sun Mar  2 08:29:59 2008
From: lluo_2wire at yahoo.com (Louis Luo)
Date: Sun, 2 Mar 2008 00:29:59 -0800 (PST)
Subject: SSL support
Message-ID: <284832.44453.qm@web63907.mail.re1.yahoo.com>

Hi, I am a new user of Varnish. I read the FAQ section about HTTPS. Does it mean Varnish doesn't support HTTPS requests? If so, do you have plan to support it in the future? Thanks.

L.




      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ 



From perbu at linpro.no  Sun Mar  2 16:11:43 2008
From: perbu at linpro.no (Per Andreas Buer)
Date: Sun, 02 Mar 2008 17:11:43 +0100
Subject: SSL support
In-Reply-To: <284832.44453.qm@web63907.mail.re1.yahoo.com>
References: <284832.44453.qm@web63907.mail.re1.yahoo.com>
Message-ID: <47CAD1BF.40504@linpro.no>

Hi,

Louis Luo skrev:
> Hi, I am a new user of Varnish. I read the FAQ section about HTTPS. Does it mean Varnish doesn't support HTTPS requests?

Varnish does not support https.

> If so, do you have plan to support it in the future? Thanks.

As far as I know there are no concrete plans to include HTTPS support. I 
guess it would be a rather simple task to include HTTPS support so it 
might be done on some point.

Someone just needs to want it enough to pay for it.


Per.


From jim at bonetruck.org  Sat Mar  1 20:47:33 2008
From: jim at bonetruck.org (Jim Razmus)
Date: Sat, 1 Mar 2008 15:47:33 -0500
Subject: Configure Error with 1.1.2 on OpenBSD 4.3-beta sparc64
Message-ID: <20080301204733.GA16206@mail.bonetruck.org>

As per the instructions, here ya go:

configure: WARNING: sys/mount.h: present but cannot be compiled
configure: WARNING: sys/mount.h:     check for missing prerequisite headers?
configure: WARNING: sys/mount.h: see the Autoconf documentation
configure: WARNING: sys/mount.h:     section "Present But Cannot Be Compiled"
configure: WARNING: sys/mount.h: proceeding with the preprocessor's result
configure: WARNING: sys/mount.h: in the future, the compiler will take precedence
configure: WARNING:     ## --------------------------------------------- ##
configure: WARNING:     ## Report this to varnish-dev at projects.linpro.no ##
configure: WARNING:     ## --------------------------------------------- ##

HTH,
Jim


From des at linpro.no  Mon Mar  3 13:29:28 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 03 Mar 2008 14:29:28 +0100
Subject: SSL support
In-Reply-To: <47CAD1BF.40504@linpro.no> (Per Andreas Buer's message of "Sun,
	02 Mar 2008 17:11:43 +0100")
References: <284832.44453.qm@web63907.mail.re1.yahoo.com>
	<47CAD1BF.40504@linpro.no>
Message-ID: <ujrd4qcx7ev.fsf@login.linpro.no>

Per Andreas Buer <perbu at linpro.no> writes:
> As far as I know there are no concrete plans to include HTTPS
> support. I guess it would be a rather simple task to include HTTPS
> support so it might be done on some point.

Actually, it would be very complicated...

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Mon Mar  3 14:58:59 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 03 Mar 2008 15:58:59 +0100
Subject: Load Balancing Algorithms in VCL
In-Reply-To: <47BB56F8.4080902@gmail.com> (Shiraz Kanga's message of "Tue, 19
	Feb 2008 14:23:52 -0800")
References: <47BB56F8.4080902@gmail.com>
Message-ID: <ujrve43x39o.fsf@login.linpro.no>

Shiraz Kanga <skanga at gmail.com> writes:
> In order to support these VCL will need access to various statistics like
> * Number of active requests per server
> * Number of requests per second per server
> * Avg response time per server
> * Bandwidth per server
>
> Are these stats available in the VCL environment?

Not presently, and our current take on backend load balancing is that
it is best done in C, probably with pluggable algorithms (or directors
as they are called in the code).

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From perbu at linpro.no  Mon Mar  3 19:14:29 2008
From: perbu at linpro.no (Per Andreas Buer)
Date: Mon, 03 Mar 2008 20:14:29 +0100
Subject: SSL support
In-Reply-To: <ujrd4qcx7ev.fsf@login.linpro.no>
References: <284832.44453.qm@web63907.mail.re1.yahoo.com>	<47CAD1BF.40504@linpro.no>
	<ujrd4qcx7ev.fsf@login.linpro.no>
Message-ID: <47CC4E15.1070906@linpro.no>

Dag-Erling Sm?rgrav skrev:
> 
> Actually, it would be very complicated...

Oh. I guess I'm wrong, then. I though it was the simple matter of 
linking in some SSL library and let it do its magic. Whats the complication?


Per.


From caleb.anthony at gmail.com  Mon Mar  3 22:52:14 2008
From: caleb.anthony at gmail.com (Caleb Anthony)
Date: Mon, 3 Mar 2008 15:52:14 -0700
Subject: SSL support
In-Reply-To: <47CC4E15.1070906@linpro.no>
References: <284832.44453.qm@web63907.mail.re1.yahoo.com>
	<47CAD1BF.40504@linpro.no> <ujrd4qcx7ev.fsf@login.linpro.no>
	<47CC4E15.1070906@linpro.no>
Message-ID: <33afbbe70803031452j2c0d0837t8ea4bcff2249a17e@mail.gmail.com>

You could always setup an SSL frontend that will then communicate with
Varnish on the backend.

Check out Pound:

http://www.apsis.ch/pound/

On 3/3/08, Per Andreas Buer <perbu at linpro.no> wrote:
> Dag-Erling Sm?rgrav skrev:
> >
> > Actually, it would be very complicated...
>
> Oh. I guess I'm wrong, then. I though it was the simple matter of
> linking in some SSL library and let it do its magic. Whats the complication?
>
>
> Per.
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-dev
>


From lluo_2wire at yahoo.com  Mon Mar  3 23:38:07 2008
From: lluo_2wire at yahoo.com (Louis Luo)
Date: Mon, 3 Mar 2008 15:38:07 -0800 (PST)
Subject: SSL support
Message-ID: <647725.78771.qm@web63907.mail.re1.yahoo.com>

I am trying to use Nginx's SSL feature to offload SSL connections. But wouldn't this impose extra overhead?BTW, some ticket mentioned that compression would be added to 2.0. How is it now? I need this feature badly. Do you guys have some early work that I can try?Thanks,L.> You could always setup an SSL frontend that will then communicate with> Varnish on the backend.>> Check out Pound:>> http://www.apsis.ch/pound/>>On 3/3/08, Per Andreas Buer <perbu at linpro.no> wrote:> > Dag-Erling Sm?rgrav skrev:
> > >
> > > Actually, it would be very complicated...
> >
> > Oh. I guess I'm wrong, then. I though it was the simple matter of
> > linking in some SSL library and let it do its magic. Whats the complication?
> >
> >
> > Per.
> > _______________________________________________
> > varnish-dev mailing list
> > varnish-dev at projects.linpro.no> > http://projects.linpro.no/mailman/listinfo/varnish-dev> >







      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs


From des at linpro.no  Tue Mar  4 09:47:40 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 04 Mar 2008 10:47:40 +0100
Subject: SSL support
In-Reply-To: <47CC4E15.1070906@linpro.no> (Per Andreas Buer's message of "Mon, 
	03 Mar 2008 20:14:29 +0100")
References: <284832.44453.qm@web63907.mail.re1.yahoo.com>
	<47CAD1BF.40504@linpro.no> <ujrd4qcx7ev.fsf@login.linpro.no>
	<47CC4E15.1070906@linpro.no>
Message-ID: <ujr7igix1kz.fsf@login.linpro.no>

Per Andreas Buer <perbu at linpro.no> writes:
> Dag-Erling Sm?rgrav <des at linpro.no> writes:
> > Actually, it would be very complicated...
> Oh. I guess I'm wrong, then. I though it was the simple matter of
> linking in some SSL library and let it do its magic. Whats the
> complication?

We would need to add a translation layer between the code that talks
to the client and the code that handles requests.  The former is not
entirely centralized, so some cleanup would be required.  Performance
would suffer, possibly even in the non-SSL case.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Mar  4 09:54:24 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 04 Mar 2008 10:54:24 +0100
Subject: Load Balancing Algorithms in VCL
In-Reply-To: <47CC5A63.7090403@gmail.com> (Shiraz Kanga's message of "Mon, 03
	Mar 2008 12:06:59 -0800")
References: <47BB56F8.4080902@gmail.com> <ujrve43x39o.fsf@login.linpro.no>
	<47CC5A63.7090403@gmail.com>
Message-ID: <ujrpruavmpb.fsf@login.linpro.no>

Shiraz Kanga <skanga at gmail.com> writes:
> Thanks for your reply. Would you mind explaining why load balancing is
> "best done in C" especially since VCL should have the same performance
> as C as it goes through the C compiler.

I don't see why I have to explain our decision.  This is the way
things are.  If it doesn't work for you, tell us what you're trying to
do (not how you're trying to do it) and we'll think about it.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From phk at phk.freebsd.dk  Tue Mar  4 11:34:47 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 04 Mar 2008 11:34:47 +0000
Subject: Load Balancing Algorithms in VCL 
In-Reply-To: Your message of "Tue, 19 Feb 2008 14:23:52 PST."
	<47BB56F8.4080902@gmail.com> 
Message-ID: <15817.1204630487@critter.freebsd.dk>

In message <47BB56F8.4080902 at gmail.com>, Shiraz Kanga writes:
>
>Will it be possible to implement custom Load Balancing algorithms in 
>VCL? There are numerous algorithms that can be used to distribute load like
>Round Robin, Weighted Round Robin, Least Connections, Fastest Response, 
>Most Bandwidth, etc.

The plan is to implement such "directors" in C (we already have "random")
and I'm not convinced that doing it in VCL makes much sense, but can
be persuaded otherwise by good arguments.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From jesus at omniti.com  Tue Mar  4 13:34:42 2008
From: jesus at omniti.com (Theo Schlossnagle)
Date: Tue, 4 Mar 2008 08:34:42 -0500
Subject: Load Balancing Algorithms in VCL 
In-Reply-To: <15817.1204630487@critter.freebsd.dk>
References: <15817.1204630487@critter.freebsd.dk>
Message-ID: <0C6FC463-FE47-4E26-8DFB-1182B8B0307C@omniti.com>


On Mar 4, 2008, at 6:34 AM, Poul-Henning Kamp wrote:

> In message <47BB56F8.4080902 at gmail.com>, Shiraz Kanga writes:
>>
>> Will it be possible to implement custom Load Balancing algorithms in
>> VCL? There are numerous algorithms that can be used to distribute  
>> load like
>> Round Robin, Weighted Round Robin, Least Connections, Fastest  
>> Response,
>> Most Bandwidth, etc.
>
> The plan is to implement such "directors" in C (we already have  
> "random")
> and I'm not convinced that doing it in VCL makes much sense, but can
> be persuaded otherwise by good arguments.


Load balancing algorithms can be complicated over larger sets of  
backends (if that is the desired use case).  mod_backhand was written  
specifically as a platform to test different algorithms in the lab.   
What I learned from that is that if you have a good (and very simple)  
API to implement new algorithms, then it is easy enough to test.  For  
example, we found interesting and successful approaches using cost- 
benefit-based algorithms and randomized-log2-window algorithms (much  
better than simple random).  As a note, simple random performs really  
well in the lab (better than least-connections and fastest-response)  
and in practice, while it doesn't do as well, it still does better  
than most alternatives -- so good first algorithm selection.

On the other hand, VCL compiles to C.  It seems you have an excellent  
platform to expose that algorithm composition up the stack and make it  
even more accessible without compromising efficiency.  At the end of  
the day, it would only be a convenience thing.  If not in VCL, I hope  
that it could be at least dlopened, so algorithms can be tested post- 
install.

Best regards,

Theo

--
Theo Schlossnagle
Esoteric Curio -- http://lethargy.org/
OmniTI Computer Consulting, Inc. -- http://omniti.com/



From des at linpro.no  Wed Mar  5 10:48:47 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Wed, 05 Mar 2008 11:48:47 +0100
Subject: SSL support
In-Reply-To: <647725.78771.qm@web63907.mail.re1.yahoo.com> (Louis Luo's
	message of "Mon\, 3 Mar 2008 15\:38\:07 -0800 \(PST\)")
References: <647725.78771.qm@web63907.mail.re1.yahoo.com>
Message-ID: <87tzjlzbsg.fsf@des.linpro.no>

Louis Luo <lluo_2wire at yahoo.com> writes:
> BTW, some ticket mentioned that compression would be added to 2.0.
> How is it now?  I need this feature badly.  Do you guys have some
> early work that I can try?

That was a wishlist item, not a promised feature.  However, Varnish 2.0
supports Vary, so if can enable compression on your backend, it will
"just work" - but DO NOT UNDER ANY CIRCUMSTANCES use the sample
configuration at the top of Apache's mod_deflate documentation, as it
will effectively make every compressable document uncacheable.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From skanga at gmail.com  Mon Mar  3 20:06:59 2008
From: skanga at gmail.com (Shiraz Kanga)
Date: Mon, 03 Mar 2008 12:06:59 -0800
Subject: Load Balancing Algorithms in VCL
In-Reply-To: <ujrve43x39o.fsf@login.linpro.no>
References: <47BB56F8.4080902@gmail.com> <ujrve43x39o.fsf@login.linpro.no>
Message-ID: <47CC5A63.7090403@gmail.com>

Dag-Erling Sm?rgrav wrote:
> Shiraz Kanga <skanga at gmail.com> writes:
>   
>> In order to support these VCL will need access to various statistics like
>> * Number of active requests per server
>> * Number of requests per second per server
>> * Avg response time per server
>> * Bandwidth per server
>>
>> Are these stats available in the VCL environment?
>>     
>
> Not presently, and our current take on backend load balancing is that
> it is best done in C, probably with pluggable algorithms (or directors
> as they are called in the code).
>
> DES
>   
Thanks for your reply. Would you mind explaining why load balancing is 
"best done in C" especially since VCL should have the same performance 
as C as it goes through the C compiler.

shiraz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20080303/30dec5df/attachment.html>

From lluo_2wire at yahoo.com  Wed Mar  5 19:02:01 2008
From: lluo_2wire at yahoo.com (Louis Luo)
Date: Wed, 5 Mar 2008 11:02:01 -0800 (PST)
Subject: SSL support
Message-ID: <690776.86841.qm@web63910.mail.re1.yahoo.com>

Thanks for your info. But do you mean Varnish 2.0 won't support compression by itself, but rely on the backend to compress HTML pages? If so, then can it cache compressed pages? 


I need to offload the compression work from backend to the proxy, so probably cannot take advantage of this Vary feature. I was wondering how much work there would be to add this compression feature. If this is not that challenging, our team might do some work to port lighty's mod_deflate to Varnish, if that won't conflict with your plan.


Thanks.


L.
> That was a wishlist item, not a promised feature.  However, Varnish 2.0 supports Vary, so if can enable compression on your backend, it will "just work" - but DO NOT UNDER ANY CIRCUMSTANCES use the sample configuration at the top of Apache's mod_deflate documentation, as it will effectively make every compressable document uncacheable. DES




      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ 



From des at linpro.no  Thu Mar  6 11:06:05 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Thu, 06 Mar 2008 12:06:05 +0100
Subject: SSL support
In-Reply-To: <690776.86841.qm@web63910.mail.re1.yahoo.com> (Louis Luo's
	message of "Wed\, 5 Mar 2008 11\:02\:01 -0800 \(PST\)")
References: <690776.86841.qm@web63910.mail.re1.yahoo.com>
Message-ID: <87mypcw1r6.fsf@des.linpro.no>

Louis Luo <lluo_2wire at yahoo.com> writes:
> Thanks for your info. But do you mean Varnish 2.0 won't support
> compression by itself, but rely on the backend to compress HTML pages?
> If so, then can it cache compressed pages?

Yes, and yes.

> I need to offload the compression work from backend to the proxy

Why?  A properly configured cache should reduce the backend load to a
point where you can afford compression.

> so probably cannot take advantage of this Vary feature. I was
> wondering how much work there would be to add this compression
> feature. If this is not that challenging, our team might do some work
> to port lighty's mod_deflate to Varnish, if that won't conflict with
> your plan.

Compression itself is easy.  The hard part is figuring out when and how
to do it; ideally, we should compress a document only if and only if we
get a request from a client that supports compression, and when we do,
we should cache it with the same ttl as the original object.  I imagine
that some of the mechanisms that were introduced to support Vary can
also be used to support native compression.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From smiley at npr.org  Fri Mar  7 21:10:18 2008
From: smiley at npr.org (Shain Miley)
Date: Fri, 07 Mar 2008 16:10:18 -0500
Subject: Caching issue
Message-ID: <47D1AF3A.50409@npr.org>

Hello all,
First let me say thanks to everyone for taking time to work on this 
project...I am still in the testing phase, however if all goes well 
Varnish is going to come in quite handy.

So here is my problem:

I would like to use varnish to cache urls for say a period of 30 
minutes.  The caching seems fine as long as I use the same browser to 
request the url.  But I am looking for a stop gap solution at this time 
so in the event that the web server becomes unavailable....varnish can 
serve the docs for a little while, so we can get things back in order.  
Anyway..as it stands right now..I can request a url from wget or GET or 
firefox... and turn off the webserver and I can refresh the page and all 
is well.

What I cannot do...is request a url using GET on one machine..then stop 
the webserver and request that same url from firefox (different machine) 
and get a valid page...I get a 503 error...Here is the vcl.conf file 
(which I took most of from the web) I am using:

Any help would be great!

Thanks,

Shain

sub vcl_recv {
        if (req.request != "GET" && req.request != "HEAD") {
                # PURGE request if zope asks nicely
                if (req.request == "PURGE") {
                        if (!client.ip ~ purge) {
                              error 405 "Not allowed.";
                }
                lookup;
                }
                pipe;
        }
        if (req.http.Expect) {
                pipe;
        }

        if (req.http.Authenticate || req.http.Authorization) {
                pass;
        }

        # We only care about the "__ac.*" cookies, used for authentication
        if (req.http.Cookie && req.http.Cookie ~ 
"__ac(|_(name|password|persistent))=") {
                pass;
        }

        # File type that we will always cache
        if (req.request == "GET" && req.url ~ 
"\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|js|vsd|doc|ppt|pps|xls|pdf|mp3|mp4|m4a|ogg|mov|avi|wmv|sxw|zip|gz|bz2|tgz|tar)$") 
{
            lookup;
        }

        if (req.request == "POST") {
                pipe;
        }

        # force lookup even when cookies are present
        if (req.request == "GET" && req.http.cookie) {
                lookup;
        }
        lookup;
}

sub vcl_fetch {

if (obj.ttl < 1800) {
                set obj.ttl = 1800s;
        }
}

# Do the PURGE thing
sub vcl_hit {
        if (req.request == "PURGE") {
                set obj.ttl = 0s;
                error 200 "Purged";
        }
}

sub vcl_miss {
        if (req.request == "PURGE") {
                error 404 "Not in cache";
        }
}





From aotto at mosso.com  Fri Mar  7 21:54:58 2008
From: aotto at mosso.com (Adrian Otto)
Date: Fri, 7 Mar 2008 13:54:58 -0800
Subject: Caching issue
In-Reply-To: <47D1AF3A.50409@npr.org>
References: <47D1AF3A.50409@npr.org>
Message-ID: <BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>

Shain,

I can't say for certain without looking at it myself, but I wonder if  
when you request the URL from the second machine using firefox you  
are actually doing a "refresh" that's setting a Cache-Control header  
that's forcing varnish to consult the origin server. I suggest  
connecting to your varnish with "varnishlog" and watching the headers  
that each client sends varnish. If you see headers like this:

    13 RxHeader     c Pragma: no-cache
    13 RxHeader     c Cache-Control: no-cache

Then you know that's what's happening. In general, it sounds like you  
are looking for an "off-line" mode where the cache server simply  
serves stale content if an origin server can not be reached. If that  
feature is present in varnish, I'm not aware of it.

Adrian


On Mar 7, 2008, at 1:10 PM, Shain Miley wrote:

> Hello all,
> First let me say thanks to everyone for taking time to work on this
> project...I am still in the testing phase, however if all goes well
> Varnish is going to come in quite handy.
>
> So here is my problem:
>
> I would like to use varnish to cache urls for say a period of 30
> minutes.  The caching seems fine as long as I use the same browser to
> request the url.  But I am looking for a stop gap solution at this  
> time
> so in the event that the web server becomes unavailable....varnish can
> serve the docs for a little while, so we can get things back in order.
> Anyway..as it stands right now..I can request a url from wget or  
> GET or
> firefox... and turn off the webserver and I can refresh the page  
> and all
> is well.
>
> What I cannot do...is request a url using GET on one machine..then  
> stop
> the webserver and request that same url from firefox (different  
> machine)
> and get a valid page...I get a 503 error...Here is the vcl.conf file
> (which I took most of from the web) I am using:
>
> Any help would be great!
>
> Thanks,
>
> Shain
>
> sub vcl_recv {
>         if (req.request != "GET" && req.request != "HEAD") {
>                 # PURGE request if zope asks nicely
>                 if (req.request == "PURGE") {
>                         if (!client.ip ~ purge) {
>                               error 405 "Not allowed.";
>                 }
>                 lookup;
>                 }
>                 pipe;
>         }
>         if (req.http.Expect) {
>                 pipe;
>         }
>
>         if (req.http.Authenticate || req.http.Authorization) {
>                 pass;
>         }
>
>         # We only care about the "__ac.*" cookies, used for  
> authentication
>         if (req.http.Cookie && req.http.Cookie ~
> "__ac(|_(name|password|persistent))=") {
>                 pass;
>         }
>
>         # File type that we will always cache
>         if (req.request == "GET" && req.url ~
> "\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg| 
> swf|ico|css|js|vsd|doc|ppt|pps|xls|pdf|mp3|mp4|m4a|ogg|mov|avi|wmv| 
> sxw|zip|gz|bz2|tgz|tar)$")
> {
>             lookup;
>         }
>
>         if (req.request == "POST") {
>                 pipe;
>         }
>
>         # force lookup even when cookies are present
>         if (req.request == "GET" && req.http.cookie) {
>                 lookup;
>         }
>         lookup;
> }
>
> sub vcl_fetch {
>
> if (obj.ttl < 1800) {
>                 set obj.ttl = 1800s;
>         }
> }
>
> # Do the PURGE thing
> sub vcl_hit {
>         if (req.request == "PURGE") {
>                 set obj.ttl = 0s;
>                 error 200 "Purged";
>         }
> }
>
> sub vcl_miss {
>         if (req.request == "PURGE") {
>                 error 404 "Not in cache";
>         }
> }
>
>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-dev



From smiley at npr.org  Fri Mar  7 22:24:48 2008
From: smiley at npr.org (Shain Miley)
Date: Fri, 07 Mar 2008 17:24:48 -0500
Subject: Caching issue
In-Reply-To: <BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>
References: <47D1AF3A.50409@npr.org>
	<BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>
Message-ID: <47D1C0B0.4030007@npr.org>

Adrian,
Thanks for the info...here is the output from my request (ip's 
changed)..in case this can shed some light on things:

   0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1204928472
    0 WorkThread     0x66b1d1b0 start
   12 SessionOpen  c 172.1.1.1 3690
    0 WorkThread     0x6531d1b0 end
   12 ReqStart     c 172.1.1.1 3690 84896454
   12 RxRequest    c GET
   12 RxURL        c /index.html
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Accept: image/gif, image/x-xbitmap, image/jpeg, 
image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint, 
application/msword, application/x-shockwave-flash, */*
   12 RxHeader     c Accept-Language: en-us
   12 RxHeader     c UA-CPU: x86
   12 RxHeader     c Accept-Encoding: gzip, deflate
   12 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; 
Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
   12 RxHeader     c Host: host.xxx.com:8080
   12 RxHeader     c Connection: Keep-Alive
   12 RxHeader     c Cookie: v1st=9771DC737ED120FD; 
GUID=000CD0009ECB070B15D1037761626364
   12 VCL_call     c recv
   12 VCL_return   c lookup
   12 VCL_call     c hash
   12 VCL_return   c hash
   12 VCL_call     c miss
   12 VCL_return   c fetch
   12 Length       c 455
   12 VCL_call     c deliver
   12 VCL_return   c deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 503
   12 TxResponse   c Service Unavailable
   12 TxHeader     c Server: Varnish
   12 TxHeader     c Retry-After: 30
   12 TxHeader     c Content-Type: text/html; charset=utf-8
   12 TxHeader     c Content-Length: 455
   12 TxHeader     c Date: Fri, 07 Mar 2008 22:21:14 GMT
   12 TxHeader     c X-Varnish: 84896454
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 ReqEnd       c 84896454 1204928473.923037052 1204928474.937938929 
0.004235029 1.014863968 0.000037909
    0 StatAddr       172.1.1.1 0 5172 23 249 0 0 11 63867 204568
    0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1204928475



Shain

Adrian Otto wrote:
> Shain,
>
> I can't say for certain without looking at it myself, but I wonder if 
> when you request the URL from the second machine using firefox you are 
> actually doing a "refresh" that's setting a Cache-Control header 
> that's forcing varnish to consult the origin server. I suggest 
> connecting to your varnish with "varnishlog" and watching the headers 
> that each client sends varnish. If you see headers like this:
>
>    13 RxHeader     c Pragma: no-cache
>    13 RxHeader     c Cache-Control: no-cache
>
> Then you know that's what's happening. In general, it sounds like you 
> are looking for an "off-line" mode where the cache server simply 
> serves stale content if an origin server can not be reached. If that 
> feature is present in varnish, I'm not aware of it.
>
> Adrian
>
>
> On Mar 7, 2008, at 1:10 PM, Shain Miley wrote:
>
>> Hello all,
>> First let me say thanks to everyone for taking time to work on this
>> project...I am still in the testing phase, however if all goes well
>> Varnish is going to come in quite handy.
>>
>> So here is my problem:
>>
>> I would like to use varnish to cache urls for say a period of 30
>> minutes.  The caching seems fine as long as I use the same browser to
>> request the url.  But I am looking for a stop gap solution at this time
>> so in the event that the web server becomes unavailable....varnish can
>> serve the docs for a little while, so we can get things back in order.
>> Anyway..as it stands right now..I can request a url from wget or GET or
>> firefox... and turn off the webserver and I can refresh the page and all
>> is well.
>>
>> What I cannot do...is request a url using GET on one machine..then stop
>> the webserver and request that same url from firefox (different machine)
>> and get a valid page...I get a 503 error...Here is the vcl.conf file
>> (which I took most of from the web) I am using:
>>
>> Any help would be great!
>>
>> Thanks,
>>
>> Shain
>>
>> sub vcl_recv {
>>         if (req.request != "GET" && req.request != "HEAD") {
>>                 # PURGE request if zope asks nicely
>>                 if (req.request == "PURGE") {
>>                         if (!client.ip ~ purge) {
>>                               error 405 "Not allowed.";
>>                 }
>>                 lookup;
>>                 }
>>                 pipe;
>>         }
>>         if (req.http.Expect) {
>>                 pipe;
>>         }
>>
>>         if (req.http.Authenticate || req.http.Authorization) {
>>                 pass;
>>         }
>>
>>         # We only care about the "__ac.*" cookies, used for 
>> authentication
>>         if (req.http.Cookie && req.http.Cookie ~
>> "__ac(|_(name|password|persistent))=") {
>>                 pass;
>>         }
>>
>>         # File type that we will always cache
>>         if (req.request == "GET" && req.url ~
>> "\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|js|vsd|doc|ppt|pps|xls|pdf|mp3|mp4|m4a|ogg|mov|avi|wmv|sxw|zip|gz|bz2|tgz|tar)$") 
>>
>> {
>>             lookup;
>>         }
>>
>>         if (req.request == "POST") {
>>                 pipe;
>>         }
>>
>>         # force lookup even when cookies are present
>>         if (req.request == "GET" && req.http.cookie) {
>>                 lookup;
>>         }
>>         lookup;
>> }
>>
>> sub vcl_fetch {
>>
>> if (obj.ttl < 1800) {
>>                 set obj.ttl = 1800s;
>>         }
>> }
>>
>> # Do the PURGE thing
>> sub vcl_hit {
>>         if (req.request == "PURGE") {
>>                 set obj.ttl = 0s;
>>                 error 200 "Purged";
>>         }
>> }
>>
>> sub vcl_miss {
>>         if (req.request == "PURGE") {
>>                 error 404 "Not in cache";
>>         }
>> }
>>
>>
>>
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at projects.linpro.no
>> http://projects.linpro.no/mailman/listinfo/varnish-dev
>
>



From aotto at mosso.com  Fri Mar  7 22:42:14 2008
From: aotto at mosso.com (Adrian Otto)
Date: Fri, 7 Mar 2008 14:42:14 -0800
Subject: Caching issue
In-Reply-To: <47D1C0B0.4030007@npr.org>
References: <47D1AF3A.50409@npr.org>
	<BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>
	<47D1C0B0.4030007@npr.org>
Message-ID: <E4E6557C-58B8-40BC-9A57-FC69C7E5DA2C@mosso.com>

Shain,

Based on your trace, that browser is getting a "miss" and then can't  
fetch the document from the origin server. I did notice that there is  
a cookie set, which might be what's causing the miss. Now, you could  
get around that with the right VCL file configuration. I'm not  
certain if there is a VCL configuration that would cause varnish to  
simply disregard the Pragma and Cache-Control headers from the  
client, because anyone doing a shift+reload while your origin server  
is down is going to get sent back to the origin server as well.

Adrian


From phk at phk.freebsd.dk  Fri Mar  7 22:56:05 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 07 Mar 2008 22:56:05 +0000
Subject: Caching issue 
In-Reply-To: Your message of "Fri, 07 Mar 2008 17:24:48 EST."
	<47D1C0B0.4030007@npr.org> 
Message-ID: <24918.1204930565@critter.freebsd.dk>

In message <47D1C0B0.4030007 at npr.org>, Shain Miley writes:

>   12 VCL_call     c miss
>   12 VCL_return   c fetch

Varnish didn't find a cache-hit and I guess you didn't log the
backend transaction here, so I don't know what the backend told us.

>   12 Length       c 455
>   12 VCL_call     c deliver
>   12 VCL_return   c deliver
>   12 TxProtocol   c HTTP/1.1
>   12 TxStatus     c 503
>   12 TxResponse   c Service Unavailable

It looks a lot like it didn't even get hold of the backend...

Possibly because it couldn't resolve the name of it or because
it didn't have a route to the backend.

It looks like you ran varnislog with a -c argument, try leaving
that out so that the backend transaction also gets logged.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From smiley at npr.org  Fri Mar  7 23:06:02 2008
From: smiley at npr.org (Shain Miley)
Date: Fri, 07 Mar 2008 18:06:02 -0500
Subject: Caching issue
In-Reply-To: <24918.1204930565@critter.freebsd.dk>
References: <24918.1204930565@critter.freebsd.dk>
Message-ID: <47D1CA5A.5050902@npr.org>

I did run varnishlog from the command line without any arguments...it 
could not reach the webserver because I turned it off...I was hoping to 
setup varnish in such a way that if the webserver fails...varnish will 
keep serving pages.

For example: if user A requests index.html and varnish caches it...then 
5 minutes later user B requests index.html (and varnish still has it in 
the cache) then it should just serve up the file..without ever going to 
check the backend.  I know that it might not be the number one intended 
use for varnish...but that is what I am trying to do...any suggestions?

Thanks in advance,
Shain

Poul-Henning Kamp wrote:
> In message <47D1C0B0.4030007 at npr.org>, Shain Miley writes:
>
>   
>>   12 VCL_call     c miss
>>   12 VCL_return   c fetch
>>     
>
> Varnish didn't find a cache-hit and I guess you didn't log the
> backend transaction here, so I don't know what the backend told us.
>
>   
>>   12 Length       c 455
>>   12 VCL_call     c deliver
>>   12 VCL_return   c deliver
>>   12 TxProtocol   c HTTP/1.1
>>   12 TxStatus     c 503
>>   12 TxResponse   c Service Unavailable
>>     
>
> It looks a lot like it didn't even get hold of the backend...
>
> Possibly because it couldn't resolve the name of it or because
> it didn't have a route to the backend.
>
> It looks like you ran varnislog with a -c argument, try leaving
> that out so that the backend transaction also gets logged.
>
>
>   



From daniel at papasian.org  Fri Mar  7 22:53:36 2008
From: daniel at papasian.org (Daniel Papasian)
Date: Fri, 07 Mar 2008 17:53:36 -0500
Subject: Caching issue
In-Reply-To: <E4E6557C-58B8-40BC-9A57-FC69C7E5DA2C@mosso.com>
References: <47D1AF3A.50409@npr.org>	<BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>	<47D1C0B0.4030007@npr.org>
	<E4E6557C-58B8-40BC-9A57-FC69C7E5DA2C@mosso.com>
Message-ID: <47D1C770.2080608@papasian.org>

Adrian Otto wrote:
> I'm not certain if there is a VCL configuration that would cause varnish to  
> simply disregard the Pragma and Cache-Control headers from the  
> client, because anyone doing a shift+reload while your origin server  
> is down is going to get sent back to the origin server as well.

No, I believe it's a simple matter to simply disregard the Pragma and 
Cache-Control headers on incoming requests.  Most

Most simply, if you wanted to have every request do a lookup from the cache:

sub vcl_recv {
   lookup;
}

If the default.vcl I have lying around is still current, then the 
default vcl_recv doesn't appear to be looking at Cache-Control or 
Pragma, either.

But obviously you probably want to do some checks against 
req.http.Cache-Control, req.http.Pragma, req.http.cookie, and perhaps 
any other request header, before just blindly doing a lookup in the 
cache.  There are cases, however, where blindly doing a lookup could 
make sense - such as when you are in full control of not only all of the 
backends, but also all of the clients (e.g. you are putting varnish in 
front of a web application that is used by another one of your systems)

Daniel


From SMiley at npr.org  Sat Mar  8 02:17:35 2008
From: SMiley at npr.org (Shain Miley)
Date: Fri, 7 Mar 2008 21:17:35 -0500
Subject: Caching issue
References: <47D1AF3A.50409@npr.org>	<BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>	<47D1C0B0.4030007@npr.org>
	<E4E6557C-58B8-40BC-9A57-FC69C7E5DA2C@mosso.com>
	<47D1C770.2080608@papasian.org>
Message-ID: <3EE3DE9B48263E4AB242CACE3874560507FBBF@hq-exch01.npr_usa.ad.npr.org>

Thanks alot for the help...I have been trying some of these ideas with little success...even the simple example of 'lookup everything in vcl_recv' did not work...seems to me that should check the cache for everything it receives..which it might have been doing....but it still showed a cache miss..and showed me the 503 error...does anyone know how the matching works...does it only take a hash of the url..or does it use more then that...

Shain


-----Original Message-----
From: varnish-dev-bounces at projects.linpro.no on behalf of Daniel Papasian
Sent: Fri 3/7/2008 5:53 PM
To: Adrian Otto
Cc: varnish-dev at projects.linpro.no
Subject: Re: Caching issue
 
Adrian Otto wrote:
> I'm not certain if there is a VCL configuration that would cause varnish to  
> simply disregard the Pragma and Cache-Control headers from the  
> client, because anyone doing a shift+reload while your origin server  
> is down is going to get sent back to the origin server as well.

No, I believe it's a simple matter to simply disregard the Pragma and 
Cache-Control headers on incoming requests.  Most

Most simply, if you wanted to have every request do a lookup from the cache:

sub vcl_recv {
   lookup;
}

If the default.vcl I have lying around is still current, then the 
default vcl_recv doesn't appear to be looking at Cache-Control or 
Pragma, either.

But obviously you probably want to do some checks against 
req.http.Cache-Control, req.http.Pragma, req.http.cookie, and perhaps 
any other request header, before just blindly doing a lookup in the 
cache.  There are cases, however, where blindly doing a lookup could 
make sense - such as when you are in full control of not only all of the 
backends, but also all of the clients (e.g. you are putting varnish in 
front of a web application that is used by another one of your systems)

Daniel
_______________________________________________
varnish-dev mailing list
varnish-dev at projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20080307/b2adfdf3/attachment.html>

From phk at phk.freebsd.dk  Sat Mar  8 08:41:28 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Sat, 08 Mar 2008 08:41:28 +0000
Subject: Caching issue 
In-Reply-To: Your message of "Fri, 07 Mar 2008 18:06:02 EST."
	<47D1CA5A.5050902@npr.org> 
Message-ID: <26710.1204965688@critter.freebsd.dk>

In message <47D1CA5A.5050902 at npr.org>, Shain Miley writes:
>I did run varnishlog from the command line without any arguments...it 
>could not reach the webserver because I turned it off...I was hoping to 
>setup varnish in such a way that if the webserver fails...varnish will 
>keep serving pages.
>
>For example: if user A requests index.html and varnish caches it...then 
>5 minutes later user B requests index.html (and varnish still has it in 
>the cache) then it should just serve up the file..without ever going to 
>check the backend.  I know that it might not be the number one intended 
>use for varnish...but that is what I am trying to do...any suggestions?

That's certainly possible, but you need to get it set up right.

By default Varnish will respect whatever the backend says about
caching, and in this case you want to override that.

I don't know if your backend said "do not cache this" or "cache only
15 seconds", but whatever it is, you need to override it in VCL.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Sat Mar  8 15:56:14 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sat, 08 Mar 2008 16:56:14 +0100
Subject: Caching issue
In-Reply-To: <47D1AF3A.50409@npr.org> (Shain Miley's message of "Fri\, 07 Mar
	2008 16\:10\:18 -0500")
References: <47D1AF3A.50409@npr.org>
Message-ID: <878x0tfbvl.fsf@des.linpro.no>

Shain Miley <smiley at npr.org> writes:
> What I cannot do...is request a url using GET on one machine..then stop 
> the webserver and request that same url from firefox (different machine) 
> and get a valid page...I get a 503 error...

Most likely because Firefox has a cookie that triggers the following
code:

>         # We only care about the "__ac.*" cookies, used for authentication
>         if (req.http.Cookie && req.http.Cookie ~ 
> "__ac(|_(name|password|persistent))=") {
>                 pass;
>         }

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Sat Mar  8 15:56:51 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sat, 08 Mar 2008 16:56:51 +0100
Subject: Caching issue
In-Reply-To: <BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com> (Adrian Otto's
	message of "Fri\, 7 Mar 2008 13\:54\:58 -0800")
References: <47D1AF3A.50409@npr.org>
	<BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>
Message-ID: <874pbhfbuk.fsf@des.linpro.no>

Adrian Otto <aotto at mosso.com> writes:
> I can't say for certain without looking at it myself, but I wonder if  
> when you request the URL from the second machine using firefox you  
> are actually doing a "refresh" that's setting a Cache-Control header  
> that's forcing varnish to consult the origin server.

For the millionth time, Varnish *intentionally* *does* *not* *obey*
these headers.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Sat Mar  8 15:58:57 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sat, 08 Mar 2008 16:58:57 +0100
Subject: Caching issue
In-Reply-To: <47D1C0B0.4030007@npr.org> (Shain Miley's message of "Fri\, 07 Mar
	2008 17\:24\:48 -0500")
References: <47D1AF3A.50409@npr.org>
	<BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>
	<47D1C0B0.4030007@npr.org>
Message-ID: <87zlt9dx6m.fsf@des.linpro.no>

Shain Miley <smiley at npr.org> writes:
> Thanks for the info...here is the output from my request (ip's
> changed)..in case this can shed some light on things:

This is useless if you don't also show us a request that worked.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From SMiley at npr.org  Sat Mar  8 23:02:41 2008
From: SMiley at npr.org (Shain Miley)
Date: Sat, 8 Mar 2008 18:02:41 -0500
Subject: Caching issue
References: <47D1AF3A.50409@npr.org><BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com><47D1C0B0.4030007@npr.org>
	<87zlt9dx6m.fsf@des.linpro.no>
Message-ID: <3EE3DE9B48263E4AB242CACE3874560507FBC3@hq-exch01.npr_usa.ad.npr.org>

Fair enough..here you go:

I used Firefox and IE from the same machine:

First the cache hit:

   12 SessionOpen  c 172.1.1.1 3203
   12 ReqStart     c 172.1.1.1 3203 42332617
   12 RxRequest    c GET
   12 RxURL        c /templates/topics/topic.php?topicId=1006
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Host: server11.npr.org:8080
   12 RxHeader     c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12
   12 RxHeader     c Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=$
   12 RxHeader     c Accept-Language: en-us,en;q=0.5
   12 RxHeader     c Accept-Encoding: gzip,deflate
   12 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
   12 RxHeader     c Keep-Alive: 300
   12 RxHeader     c Connection: keep-alive
   12 RxHeader     c Cookie: v1st=7ED84E9096FDA5B; GUID=0003D141E48D07D140F1C40561626364
   12 RxHeader     c If-None-Match: "jpd--899294354.37421"
   12 VCL_call     c recv
   12 VCL_return   c lookup
   12 VCL_call     c hash
   12 VCL_return   c hash
   12 Hit          c 42332555
   12 VCL_call     c hit
   12 VCL_return   c deliver
   12 Length       c 9137
   12 VCL_call     c deliver
   12 VCL_return   c deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache
   12 TxHeader     c X-Powered-By: PHP/5.2.3
   12 TxHeader     c X-Cache: jpcache vv2 - npr-burn
   12 TxHeader     c ETag: "jpd--899294354.37421"
   12 TxHeader     c Cache-Control: max-age=0
   12 TxHeader     c Expires: Sat, 08 Mar 2008 22:41:22 GMT
   12 TxHeader     c Vary: Accept-Encoding
   12 TxHeader     c Content-Encoding: gzip
   12 TxHeader     c Content-Encoding: gzip
   12 TxHeader     c Content-Type: text/html
   12 TxHeader     c Content-Length: 9137
   12 TxHeader     c Date: Sat, 08 Mar 2008 22:43:05 GMT
   12 TxHeader     c X-Varnish: 42332617 42332555
   12 TxHeader     c Age: 102
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 ReqEnd       c 42332617 1205016185.289316893 1205016185.289417028 0.006116867 0.000038147 0.000061989


Now the cache miss:

12 SessionOpen  c 172.1.1.1 3211
   12 ReqStart     c 172.1.1.1 3211 42332632
   12 RxRequest    c GET
   12 RxURL        c /templates/topics/topic.php?topicId=1006
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Accept: */*
   12 RxHeader     c Accept-Language: en-us
   12 RxHeader     c Accept-Encoding: gzip, deflate
   12 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
   12 RxHeader     c Host: server11.npr.org:8080
   12 RxHeader     c Connection: Keep-Alive
   12 RxHeader     c Cookie: v1st=2E524136463554A; GUID=000748BD59C807AC295102B561626364; LE4=+5SoM6V421+314+4
   12 VCL_call     c recv
   12 VCL_return   c lookup
   12 VCL_call     c hash
   12 VCL_return   c hash
   12 VCL_call     c miss
   12 VCL_return   c fetch
   12 Length       c 455
   12 VCL_call     c deliver
   12 VCL_return   c deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 503
   12 TxResponse   c Service Unavailable
   12 TxHeader     c Server: Varnish
   12 TxHeader     c Retry-After: 30
   12 TxHeader     c Content-Type: text/html; charset=utf-8
   12 TxHeader     c Content-Length: 455
   12 TxHeader     c Date: Sat, 08 Mar 2008 22:43:15 GMT
   12 TxHeader     c X-Varnish: 42332632
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 ReqEnd       c 42332632 1205016194.493165016 1205016195.508080959 0.005739927 1.014875889 0.000040054


And here is the current vcl.conf:

#
# This is a basic VCL configuration file for varnish.  See the vcl(7)
# man page for details on VCL syntax and semantics.
#
# $Id: default.vcl 1929 2007-08-29 15:37:59Z des $
#

# Default backend definition.  Set this to point to your content
# server.

backend default {
set backend.host = "172.31.2.61"; // use your own backend ip address
set backend.port = "80"; // use your own backend port
}

sub vcl_recv {

  # Remove the "Cookie:" header from the request.
  remove req.http.Set-Cookie;
  remove req.http.Cache-Control;


if (req.request == "GET" && req.url ~ "\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|js|vsd|doc)$") {
            lookup;
        }

if (req.http.Cache-Control ~ "no-cache") {
       lookup;
}
if (req.http.Cache-Control == "max-age=0") {
       lookup;
}
  # Do a "lookup" in the cache.  This goes to "vcl_hit", or to
  # "vcl_miss" and then "vcl_fetch"
  lookup;
}

# Do the PURGE thing
sub vcl_hit {
}
sub vcl_miss {
}

sub vcl_fetch {
if (obj.ttl < 7200s) {
set obj.ttl = 7200s;
        }
insert;

 if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" || obj.http.Cache-Control ~ "max-age=0") {
        insert;
        }

  if (obj.ttl < 3600s) {
    set obj.ttl = 3600s;
  }
insert;
}

sub vcl_hash {

set req.hash += req.url;
hash;
}


Thanks,

Shain

-----Original Message-----
From: Dag-Erling Sm?rgrav [mailto:des at linpro.no]
Sent: Sat 3/8/2008 10:58 AM
To: Shain Miley
Cc: Adrian Otto; varnish-dev at projects.linpro.no
Subject: Re: Caching issue
 
Shain Miley <smiley at npr.org> writes:
> Thanks for the info...here is the output from my request (ip's
> changed)..in case this can shed some light on things:

This is useless if you don't also show us a request that worked.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20080308/32b20cfc/attachment.html>

From des at linpro.no  Sun Mar  9 09:49:52 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sun, 09 Mar 2008 10:49:52 +0100
Subject: Caching issue
In-Reply-To: <3EE3DE9B48263E4AB242CACE3874560507FBC3@hq-exch01.npr_usa.ad.npr.org>
	(Shain Miley's message of "Sat\, 8 Mar 2008 18\:02\:41 -0500")
References: <47D1AF3A.50409@npr.org>
	<BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>
	<47D1C0B0.4030007@npr.org> <87zlt9dx6m.fsf@des.linpro.no>
	<3EE3DE9B48263E4AB242CACE3874560507FBC3@hq-exch01.npr_usa.ad.npr.org>
Message-ID: <87ve3wdy67.fsf@des.linpro.no>

"Shain Miley" <SMiley at npr.org> writes:
> First the cache hit:
>
>    12 SessionOpen  c 172.1.1.1 3203
>    12 ReqStart     c 172.1.1.1 3203 42332617
>    12 RxRequest    c GET
>    12 RxURL        c /templates/topics/topic.php?topicId=1006
>    12 RxProtocol   c HTTP/1.1
>    12 RxHeader     c Host: server11.npr.org:8080
>    12 RxHeader     c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12
>    12 RxHeader     c Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=$
>    12 RxHeader     c Accept-Language: en-us,en;q=0.5
>    12 RxHeader     c Accept-Encoding: gzip,deflate
>    12 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>    12 RxHeader     c Keep-Alive: 300
>    12 RxHeader     c Connection: keep-alive
>    12 RxHeader     c Cookie: v1st=7ED84E9096FDA5B; GUID=0003D141E48D07D140F1C40561626364
>    12 RxHeader     c If-None-Match: "jpd--899294354.37421"
>    12 VCL_call     c recv
>    12 VCL_return   c lookup
>    12 VCL_call     c hash
>    12 VCL_return   c hash
>    12 Hit          c 42332555
>    12 VCL_call     c hit
>    12 VCL_return   c deliver
>    12 Length       c 9137
>    12 VCL_call     c deliver
>    12 VCL_return   c deliver
>    12 TxProtocol   c HTTP/1.1
>    12 TxStatus     c 200
>    12 TxResponse   c OK
>    12 TxHeader     c Server: Apache
>    12 TxHeader     c X-Powered-By: PHP/5.2.3
>    12 TxHeader     c X-Cache: jpcache vv2 - npr-burn
>    12 TxHeader     c ETag: "jpd--899294354.37421"
>    12 TxHeader     c Cache-Control: max-age=0
>    12 TxHeader     c Expires: Sat, 08 Mar 2008 22:41:22 GMT
>    12 TxHeader     c Vary: Accept-Encoding
>    12 TxHeader     c Content-Encoding: gzip
>    12 TxHeader     c Content-Encoding: gzip
>    12 TxHeader     c Content-Type: text/html
>    12 TxHeader     c Content-Length: 9137
>    12 TxHeader     c Date: Sat, 08 Mar 2008 22:43:05 GMT
>    12 TxHeader     c X-Varnish: 42332617 42332555
>    12 TxHeader     c Age: 102
>    12 TxHeader     c Via: 1.1 varnish
>    12 TxHeader     c Connection: keep-alive
>    12 ReqEnd       c 42332617 1205016185.289316893 1205016185.289417028 0.006116867 0.000038147 0.000061989

Where is the backend traffic?  Please use both -b and -c.

> Now the cache miss:
>
>    12 SessionOpen  c 172.1.1.1 3211
>    12 ReqStart     c 172.1.1.1 3211 42332632
>    12 RxRequest    c GET
>    12 RxURL        c /templates/topics/topic.php?topicId=1006
>    12 RxProtocol   c HTTP/1.1
>    12 RxHeader     c Accept: */*
>    12 RxHeader     c Accept-Language: en-us
>    12 RxHeader     c Accept-Encoding: gzip, deflate
>    12 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
>    12 RxHeader     c Host: server11.npr.org:8080
>    12 RxHeader     c Connection: Keep-Alive
>    12 RxHeader     c Cookie: v1st=2E524136463554A; GUID=000748BD59C807AC295102B561626364; LE4=+5SoM6V421+314+4
>    12 VCL_call     c recv
>    12 VCL_return   c lookup
>    12 VCL_call     c hash
>    12 VCL_return   c hash
>    12 VCL_call     c miss
>    12 VCL_return   c fetch
>    12 Length       c 455
>    12 VCL_call     c deliver
>    12 VCL_return   c deliver
>    12 TxProtocol   c HTTP/1.1
>    12 TxStatus     c 503
>    12 TxResponse   c Service Unavailable
>    12 TxHeader     c Server: Varnish
>    12 TxHeader     c Retry-After: 30
>    12 TxHeader     c Content-Type: text/html; charset=utf-8
>    12 TxHeader     c Content-Length: 455
>    12 TxHeader     c Date: Sat, 08 Mar 2008 22:43:15 GMT
>    12 TxHeader     c X-Varnish: 42332632
>    12 TxHeader     c Age: 0
>    12 TxHeader     c Via: 1.1 varnish
>    12 TxHeader     c Connection: keep-alive
>    12 ReqEnd       c 42332632 1205016194.493165016 1205016195.508080959 0.005739927 1.014875889 0.000040054

This client sent a different Accept-Encoding header than the first one,
so from Varnish's perspective, these two requests are actually for two
different objects.

> And here is the current vcl.conf:

Is it the same vcl.conf that was in effect when the above requests were
logged?

> #
> # This is a basic VCL configuration file for varnish.  See the vcl(7)
> # man page for details on VCL syntax and semantics.
> #
> # $Id: default.vcl 1929 2007-08-29 15:37:59Z des $
> #
>
> # Default backend definition.  Set this to point to your content
> # server.
>
> backend default {
> set backend.host = "172.31.2.61"; // use your own backend ip address
> set backend.port = "80"; // use your own backend port
> }
>
> sub vcl_recv {
>
>   # Remove the "Cookie:" header from the request.
>   remove req.http.Set-Cookie;
>   remove req.http.Cache-Control;
>
>
> if (req.request == "GET" && req.url ~ "\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|js|vsd|doc)$") {
>             lookup;
>         }
>
> if (req.http.Cache-Control ~ "no-cache") {
>        lookup;
> }
> if (req.http.Cache-Control == "max-age=0") {
>        lookup;
> }

This is useless, because a) you've already removed that header and b)
Varnish doesn't obey it anyway.

>   # Do a "lookup" in the cache.  This goes to "vcl_hit", or to
>   # "vcl_miss" and then "vcl_fetch"
>   lookup;

If you're going to do a lookup anyway, why bother with all of the above?

This will break badly with POST, btw.

> }
>
> # Do the PURGE thing
> sub vcl_hit {
> }
> sub vcl_miss {
> }
>
> sub vcl_fetch {
> if (obj.ttl < 7200s) {
> set obj.ttl = 7200s;
>         }
> insert;
>
>  if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" || obj.http.Cache-Control ~ "max-age=0") {
>         insert;
>         }

This is useless because a) the "insert" above terminates vcl_fetch and
b) Varnish does not obey Pragma or Cache-Control anyway.

>
>   if (obj.ttl < 3600s) {
>     set obj.ttl = 3600s;
>   }
> insert;

Redundant.

> }
>
> sub vcl_hash {
>
> set req.hash += req.url;
> hash;
> }

Why not host?  Why define vcl_hash at all?  The default should suffice.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From SMiley at npr.org  Sun Mar  9 16:57:23 2008
From: SMiley at npr.org (Shain Miley)
Date: Sun, 9 Mar 2008 12:57:23 -0400
Subject: Caching issue
References: <47D1AF3A.50409@npr.org><BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com><47D1C0B0.4030007@npr.org>
	<87zlt9dx6m.fsf@des.linpro.no><3EE3DE9B48263E4AB242CACE3874560507FBC3@hq-exch01.npr_usa.ad.npr.org>
	<87ve3wdy67.fsf@des.linpro.no>
Message-ID: <3EE3DE9B48263E4AB242CACE3874560507FBC4@hq-exch01.npr_usa.ad.npr.org>

The output from below is a result of 'varnishlog > log.file'.  According to the man page if I don't use '-b' or '-c' both are assumed..so I don't know why any of the logging would be missing? Am I missing somthing?
 
In terms of the 'different Accept-Encoding header ' being set by the other browser..do you mean the 'Vary' header?  The reason I ask is because both set Accept-Encoding to 'gzip:deflate' so they are the same there.  If it is the result of 'Vary' can I simply remove that header and achive what I am looking for?
 
I would also like to know if anyone knows how I can better debug the hash that is being created...ie...I would like you know exaclty what information is being used in it's creation, so I can get to a plcae really where only the url is being used as the hash...that way it will match on any machine, from any browser, as long as the url has been cached.
 
I know that the vcl.conf file that I provided has a lot of redundant info in it...but I was trying to make sure it does a lookup and insert no matter what and I wanted to be sure that no case would be missed...cause it is still not working for me.
 
Thanks again for taking time to help me,

Shain
 
 

________________________________

From: Dag-Erling Sm?rgrav [mailto:des at linpro.no]
Sent: Sun 3/9/2008 5:49 AM
To: Shain Miley
Cc: Adrian Otto; varnish-dev at projects.linpro.no
Subject: Re: Caching issue



"Shain Miley" <SMiley at npr.org> writes:
> First the cache hit:
>
>    12 SessionOpen  c 172.1.1.1 3203
>    12 ReqStart     c 172.1.1.1 3203 42332617
>    12 RxRequest    c GET
>    12 RxURL        c /templates/topics/topic.php?topicId=1006
>    12 RxProtocol   c HTTP/1.1
>    12 RxHeader     c Host: server11.npr.org:8080
>    12 RxHeader     c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12
>    12 RxHeader     c Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=$
>    12 RxHeader     c Accept-Language: en-us,en;q=0.5
>    12 RxHeader     c Accept-Encoding: gzip,deflate
>    12 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>    12 RxHeader     c Keep-Alive: 300
>    12 RxHeader     c Connection: keep-alive
>    12 RxHeader     c Cookie: v1st=7ED84E9096FDA5B; GUID=0003D141E48D07D140F1C40561626364
>    12 RxHeader     c If-None-Match: "jpd--899294354.37421"
>    12 VCL_call     c recv
>    12 VCL_return   c lookup
>    12 VCL_call     c hash
>    12 VCL_return   c hash
>    12 Hit          c 42332555
>    12 VCL_call     c hit
>    12 VCL_return   c deliver
>    12 Length       c 9137
>    12 VCL_call     c deliver
>    12 VCL_return   c deliver
>    12 TxProtocol   c HTTP/1.1
>    12 TxStatus     c 200
>    12 TxResponse   c OK
>    12 TxHeader     c Server: Apache
>    12 TxHeader     c X-Powered-By: PHP/5.2.3
>    12 TxHeader     c X-Cache: jpcache vv2 - npr-burn
>    12 TxHeader     c ETag: "jpd--899294354.37421"
>    12 TxHeader     c Cache-Control: max-age=0
>    12 TxHeader     c Expires: Sat, 08 Mar 2008 22:41:22 GMT
>    12 TxHeader     c Vary: Accept-Encoding
>    12 TxHeader     c Content-Encoding: gzip
>    12 TxHeader     c Content-Encoding: gzip
>    12 TxHeader     c Content-Type: text/html
>    12 TxHeader     c Content-Length: 9137
>    12 TxHeader     c Date: Sat, 08 Mar 2008 22:43:05 GMT
>    12 TxHeader     c X-Varnish: 42332617 42332555
>    12 TxHeader     c Age: 102
>    12 TxHeader     c Via: 1.1 varnish
>    12 TxHeader     c Connection: keep-alive
>    12 ReqEnd       c 42332617 1205016185.289316893 1205016185.289417028 0.006116867 0.000038147 0.000061989

Where is the backend traffic?  Please use both -b and -c.

> Now the cache miss:
>
>    12 SessionOpen  c 172.1.1.1 3211
>    12 ReqStart     c 172.1.1.1 3211 42332632
>    12 RxRequest    c GET
>    12 RxURL        c /templates/topics/topic.php?topicId=1006
>    12 RxProtocol   c HTTP/1.1
>    12 RxHeader     c Accept: */*
>    12 RxHeader     c Accept-Language: en-us
>    12 RxHeader     c Accept-Encoding: gzip, deflate
>    12 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
>    12 RxHeader     c Host: server11.npr.org:8080
>    12 RxHeader     c Connection: Keep-Alive
>    12 RxHeader     c Cookie: v1st=2E524136463554A; GUID=000748BD59C807AC295102B561626364; LE4=+5SoM6V421+314+4
>    12 VCL_call     c recv
>    12 VCL_return   c lookup
>    12 VCL_call     c hash
>    12 VCL_return   c hash
>    12 VCL_call     c miss
>    12 VCL_return   c fetch
>    12 Length       c 455
>    12 VCL_call     c deliver
>    12 VCL_return   c deliver
>    12 TxProtocol   c HTTP/1.1
>    12 TxStatus     c 503
>    12 TxResponse   c Service Unavailable
>    12 TxHeader     c Server: Varnish
>    12 TxHeader     c Retry-After: 30
>    12 TxHeader     c Content-Type: text/html; charset=utf-8
>    12 TxHeader     c Content-Length: 455
>    12 TxHeader     c Date: Sat, 08 Mar 2008 22:43:15 GMT
>    12 TxHeader     c X-Varnish: 42332632
>    12 TxHeader     c Age: 0
>    12 TxHeader     c Via: 1.1 varnish
>    12 TxHeader     c Connection: keep-alive
>    12 ReqEnd       c 42332632 1205016194.493165016 1205016195.508080959 0.005739927 1.014875889 0.000040054

This client sent a different Accept-Encoding header than the first one,
so from Varnish's perspective, these two requests are actually for two
different objects.

> And here is the current vcl.conf:

Is it the same vcl.conf that was in effect when the above requests were
logged?

> #
> # This is a basic VCL configuration file for varnish.  See the vcl(7)
> # man page for details on VCL syntax and semantics.
> #
> # $Id: default.vcl 1929 2007-08-29 15:37:59Z des $
> #
>
> # Default backend definition.  Set this to point to your content
> # server.
>
> backend default {
> set backend.host = "172.31.2.61"; // use your own backend ip address
> set backend.port = "80"; // use your own backend port
> }
>
> sub vcl_recv {
>
>   # Remove the "Cookie:" header from the request.
>   remove req.http.Set-Cookie;
>   remove req.http.Cache-Control;
>
>
> if (req.request == "GET" && req.url ~ "\.(html|php|gif|jpg|swf|css|js|png|jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|css|js|vsd|doc)$") {
>             lookup;
>         }
>
> if (req.http.Cache-Control ~ "no-cache") {
>        lookup;
> }
> if (req.http.Cache-Control == "max-age=0") {
>        lookup;
> }

This is useless, because a) you've already removed that header and b)
Varnish doesn't obey it anyway.

>   # Do a "lookup" in the cache.  This goes to "vcl_hit", or to
>   # "vcl_miss" and then "vcl_fetch"
>   lookup;

If you're going to do a lookup anyway, why bother with all of the above?

This will break badly with POST, btw.

> }
>
> # Do the PURGE thing
> sub vcl_hit {
> }
> sub vcl_miss {
> }
>
> sub vcl_fetch {
> if (obj.ttl < 7200s) {
> set obj.ttl = 7200s;
>         }
> insert;
>
>  if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" || obj.http.Cache-Control ~ "max-age=0") {
>         insert;
>         }

This is useless because a) the "insert" above terminates vcl_fetch and
b) Varnish does not obey Pragma or Cache-Control anyway.

>
>   if (obj.ttl < 3600s) {
>     set obj.ttl = 3600s;
>   }
> insert;

Redundant.

> }
>
> sub vcl_hash {
>
> set req.hash += req.url;
> hash;
> }

Why not host?  Why define vcl_hash at all?  The default should suffice.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20080309/20370a6b/attachment.html>

From des at linpro.no  Mon Mar 10 08:51:45 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Mon, 10 Mar 2008 09:51:45 +0100
Subject: Caching issue
In-Reply-To: <3EE3DE9B48263E4AB242CACE3874560507FBC4@hq-exch01.npr_usa.ad.npr.org>
	(Shain Miley's message of "Sun\, 9 Mar 2008 12\:57\:23 -0400")
References: <47D1AF3A.50409@npr.org>
	<BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>
	<47D1C0B0.4030007@npr.org> <87zlt9dx6m.fsf@des.linpro.no>
	<3EE3DE9B48263E4AB242CACE3874560507FBC3@hq-exch01.npr_usa.ad.npr.org>
	<87ve3wdy67.fsf@des.linpro.no>
	<3EE3DE9B48263E4AB242CACE3874560507FBC4@hq-exch01.npr_usa.ad.npr.org>
Message-ID: <87pru3dkri.fsf@des.linpro.no>

"Shain Miley" <SMiley at npr.org> writes:
> The output from below is a result of 'varnishlog > log.file'.
> According to the man page if I don't use '-b' or '-c' both are
> assumed.

Best practice for Varnish logging is to store the raw log data in a file
which can later be read back by varnishlog and varnishncsa.

> I don't know why any of the logging would be missing? Am I missing
> somthing?

The backend traffic is missing; I can't tell why since I don't know
precisely what you did.

> In terms of the 'different Accept-Encoding header ' being set by the
> other browser..do you mean the 'Vary' header?

No.  The client sends Accept-*, the server sends Vary.  The Vary header
tells Varnish which of the Accept-* headers are relevant.

> The reason I ask is because both set Accept-Encoding to 'gzip:deflate'
> so they are the same there.

No, they aren't.  Look closer.

> If it is the result of 'Vary' can I simply remove that header and
> achive what I am looking for?

Not unless you also strip any incoming Accept* headers.

> I would also like to know if anyone knows how I can better debug the
> hash that is being created...ie...I would like you know exaclty what
> information is being used in it's creation, so I can get to a plcae
> really where only the url is being used as the hash...

The way your vcl_hash is set up only the URL is being used in the hash.
Vary support cannot be implemented correctly by just including it in the
hash string, so it isn't.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From smiley at npr.org  Mon Mar 10 22:41:11 2008
From: smiley at npr.org (Shain Miley)
Date: Mon, 10 Mar 2008 18:41:11 -0400
Subject: Caching issue
In-Reply-To: <87pru3dkri.fsf@des.linpro.no>
References: <47D1AF3A.50409@npr.org>	<BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>	<47D1C0B0.4030007@npr.org>
	<87zlt9dx6m.fsf@des.linpro.no>	<3EE3DE9B48263E4AB242CACE3874560507FBC3@hq-exch01.npr_usa.ad.npr.org>	<87ve3wdy67.fsf@des.linpro.no>	<3EE3DE9B48263E4AB242CACE3874560507FBC4@hq-exch01.npr_usa.ad.npr.org>
	<87pru3dkri.fsf@des.linpro.no>
Message-ID: <47D5B907.6000308@npr.org>

Here is the output from an original request...nothing is cached at this 
point..I hope it has the backend info you need.  I started 
varnish...requested index.html from firefox, stopped varnish to clear 
it's cache, then requested index.html from IE...here are the results 
(varnishlog > file_name).

PS...thank you to everyone who is giving up some of their time to give 
me hand...


FIREFOX:

    0 CLI            Rd ping
    0 CLI            Wr 0 200 PONG 1205188277
   12 SessionOpen  c 172.30.4.23 50780
   12 ReqStart     c 172.30.4.23 50780 1501045688
   12 RxRequest    c GET
   12 RxURL        c /index.html
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Host: server1.npr.org:8080
   12 RxHeader     c User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; 
rv:1.8.1.12) Gecko/20080207 Ubuntu/7.10 (gutsy) Firefox/2.0.0.12
   12 RxHeader     c Accept: 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
   12 RxHeader     c Accept-Language: en-us,en;q=0.5
   12 RxHeader     c Accept-Encoding: gzip,deflate
   12 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
   12 RxHeader     c Keep-Alive: 300
   12 RxHeader     c Connection: keep-alive
   12 RxHeader     c
   12 VCL_call     c recv
   12 VCL_return   c lookup
   12 VCL_call     c hash
   12 VCL_return   c hash
   12 VCL_call     c miss
   12 VCL_return   c fetch
   15 BackendClose   default
   15 BackendOpen  b default 172.1.1.2 36260 172.1.1.2 80
   15 BackendXID   b 1501045688
   12 Backend      c 15 default
   15 TxRequest    b GET
   15 TxURL        b /index.html
   15 TxProtocol   b HTTP/1.1
   15 TxHeader     b Host: server1.npr.org:8080
   15 TxHeader     b User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; 
rv:1.8.1.12) Gecko/20080207 Ubuntu/7.10 (gutsy) Firefox/2.0.0.12
   15 TxHeader     b Accept: 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
   15 TxHeader     b Accept-Language: en-us,en;q=0.5
   15 TxHeader     b Accept-Encoding: gzip,deflate
   15 TxHeader     b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
   15 TxHeader     b
   15 TxHeader     b X-Varnish: 1501045688
   15 TxHeader     b X-Forwarded-for: 172.30.4.23
   15 RxProtocol   b HTTP/1.1
   15 RxStatus     b 200
   15 RxResponse   b OK
   15 RxHeader     b Date: Mon, 10 Mar 2008 22:31:17 GMT
   15 RxHeader     b Server: Apache
   15 RxHeader     b Last-Modified: Mon, 10 Mar 2008 22:14:00 GMT
   15 RxHeader     b ETag: "11cbc-8553fa00"
   15 RxHeader     b Accept-Ranges: bytes
   15 RxHeader     b Cache-Control: max-age=0
   15 RxHeader     b Expires: Mon, 10 Mar 2008 22:31:17 GMT
   15 RxHeader     b Vary: Accept-Encoding
   15 RxHeader     b Content-Encoding: gzip
   15 RxHeader     b Content-Length: 17243
   15 RxHeader     b Content-Type: text/html
   12 ObjProtocol  c HTTP/1.1
   12 ObjStatus    c 200
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Mon, 10 Mar 2008 22:31:17 GMT
   12 ObjHeader    c Server: Apache
   12 ObjHeader    c Last-Modified: Mon, 10 Mar 2008 22:14:00 GMT
   12 ObjHeader    c ETag: "11cbc-8553fa00"
   12 ObjHeader    c Cache-Control: max-age=0
   12 ObjHeader    c Expires: Mon, 10 Mar 2008 22:31:17 GMT
   12 ObjHeader    c Vary: Accept-Encoding
   12 ObjHeader    c Content-Encoding: gzip
   12 ObjHeader    c Content-Type: text/html
   15 BackendReuse b default
   12 TTL          c 1501045688 RFC 0 1205188277 1205188277 1205188277 0 0
   12 VCL_call     c fetch
   12 TTL          c 1501045688 VCL 7200 1205188278
   12 VCL_return   c insert
   12 Length       c 17243
   12 VCL_call     c deliver
   12 VCL_return   c deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache
   12 TxHeader     c Last-Modified: Mon, 10 Mar 2008 22:14:00 GMT
   12 TxHeader     c ETag: "11cbc-8553fa00"
   12 TxHeader     c Cache-Control: max-age=0
   12 TxHeader     c Expires: Mon, 10 Mar 2008 22:31:17 GMT
   12 TxHeader     c Vary: Accept-Encoding
   12 TxHeader     c Content-Encoding: gzip
   12 TxHeader     c Content-Type: text/html
   12 TxHeader     c Content-Length: 17243
   12 TxHeader     c Date: Mon, 10 Mar 2008 22:31:17 GMT
   12 TxHeader     c X-Varnish: 1501045688
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 ReqEnd       c 1501045688 1205188277.778860092 
1205188277.822711945 0.000495195 0.008096933 0.035754919



IE:

 0 WorkThread     0x69a4d1b0 start
   12 SessionOpen  c 172.30.4.50 4319
   12 ReqStart     c 172.30.4.50 4319 1501045670
   12 RxRequest    c GET
   12 RxURL        c /index.html
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Accept: image/gif, image/x-xbitmap, image/jpeg, 
image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint, 
application/msword, application/x-shockwave-flash, */*
   12 RxHeader     c Accept-Language: en-us
   12 RxHeader     c UA-CPU: x86
   12 RxHeader     c Accept-Encoding: gzip, deflate
   12 RxHeader     c If-Modified-Since: Mon, 10 Mar 2008 21:33:26 GMT
   12 RxHeader     c If-None-Match: "11bd4-f4401580"
   12 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; 
Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
   12 RxHeader     c Host: server1.npr.org:8080
   12 RxHeader     c Connection: Keep-Alive
   12 RxHeader     c Cookie: v1st=FB94D3E930840FF1; 
GUID=000CD0009ECB070B15D1037761626364
   12 VCL_call     c recv
   12 VCL_return   c lookup
   12 VCL_call     c hash
   12 VCL_return   c hash
   12 VCL_call     c miss
   12 VCL_return   c fetch
   15 BackendOpen  b default 172.1.1.2 35345 172.1.1.2 80
   15 BackendXID   b 1501045670
   12 Backend      c 15 default
   15 TxRequest    b GET
   15 TxURL        b /index.html
   15 TxProtocol   b HTTP/1.1
   15 TxHeader     b Accept: image/gif, image/x-xbitmap, image/jpeg, 
image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint, 
application/msword, application/x-shockwave-flash, */*
   15 TxHeader     b Accept-Language: en-us
   15 TxHeader     b UA-CPU: x86
   15 TxHeader     b Accept-Encoding: gzip, deflate
   15 TxHeader     b User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; 
Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
   15 TxHeader     b Host: server1.npr.org:8080
   15 TxHeader     b Cookie: v1st=FB94D3E930840FF1; 
GUID=000CD0009ECB070B15D1037761626364
   15 TxHeader     b X-Varnish: 1501045670
   15 TxHeader     b X-Forwarded-for: 172.30.4.50
   15 RxProtocol   b HTTP/1.1
   15 RxStatus     b 200
   15 RxResponse   b OK
   15 RxHeader     b Date: Mon, 10 Mar 2008 22:29:20 GMT
   15 RxHeader     b Server: Apache
   15 RxHeader     b Last-Modified: Mon, 10 Mar 2008 22:14:00 GMT
   15 RxHeader     b ETag: "11cbc-8553fa00"
   15 RxHeader     b Accept-Ranges: bytes
   15 RxHeader     b Cache-Control: max-age=0
   15 RxHeader     b Expires: Mon, 10 Mar 2008 22:29:20 GMT
   15 RxHeader     b Vary: Accept-Encoding
   15 RxHeader     b Content-Encoding: gzip
   15 RxHeader     b Content-Length: 17243
   15 RxHeader     b Content-Type: text/html
   12 ObjProtocol  c HTTP/1.1
   12 ObjStatus    c 200
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Mon, 10 Mar 2008 22:29:20 GMT
   12 ObjHeader    c Server: Apache
   12 ObjHeader    c Last-Modified: Mon, 10 Mar 2008 22:14:00 GMT
   12 ObjHeader    c ETag: "11cbc-8553fa00"
   12 ObjHeader    c Cache-Control: max-age=0
   12 ObjHeader    c Expires: Mon, 10 Mar 2008 22:29:20 GMT   12 
VCL_call     c fetch
   12 TTL          c 1501045670 VCL 7200 1205188160
   12 VCL_return   c insert
   12 Length       c 17243
   12 VCL_call     c deliver
   12 VCL_return   c deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache
   12 TxHeader     c Last-Modified: Mon, 10 Mar 2008 22:14:00 GMT
   12 TxHeader     c ETag: "11cbc-8553fa00"
   12 TxHeader     c Cache-Control: max-age=0
   12 TxHeader     c Expires: Mon, 10 Mar 2008 22:29:20 GMT
   12 TxHeader     c Vary: Accept-Encoding
   12 TxHeader     c Content-Encoding: gzip
   12 TxHeader     c Content-Type: text/html
   12 TxHeader     c Content-Length: 17243
   12 TxHeader     c Date: Mon, 10 Mar 2008 22:29:20 GMT
   12 TxHeader     c X-Varnish: 1501045670
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 ReqEnd       c 1501045670 1205188160.207285881 
1205188160.217762947 0.000722885 0.010413170 0.000063896
   12 ObjHeader    c Vary: Accept-Encoding
   12 ObjHeader    c Content-Encoding: gzip
   12 ObjHeader    c Content-Type: text/html
   15 BackendReuse b default
   12 TTL          c 1501045670 RFC 0 1205188160 1205188160 1205188160 0 0





Dag-Erling Sm?rgrav wrote:
> "Shain Miley" <SMiley at .org> writes:
>   
>> The output from below is a result of 'varnishlog > log.file'.
>> According to the man page if I don't use '-b' or '-c' both are
>> assumed.
>>     
>
> Best practice for Varnish logging is to store the raw log data in a file
> which can later be read back by varnishlog and varnishncsa.
>
>   
>> I don't know why any of the logging would be missing? Am I missing
>> somthing?
>>     
>
> The backend traffic is missing; I can't tell why since I don't know
> precisely what you did.
>
>   
>> In terms of the 'different Accept-Encoding header ' being set by the
>> other browser..do you mean the 'Vary' header?
>>     
>
> No.  The client sends Accept-*, the server sends Vary.  The Vary header
> tells Varnish which of the Accept-* headers are relevant.
>
>   
>> The reason I ask is because both set Accept-Encoding to 'gzip:deflate'
>> so they are the same there.
>>     
>
> No, they aren't.  Look closer.
>
>   
>> If it is the result of 'Vary' can I simply remove that header and
>> achive what I am looking for?
>>     
>
> Not unless you also strip any incoming Accept* headers.
>
>   
>> I would also like to know if anyone knows how I can better debug the
>> hash that is being created...ie...I would like you know exaclty what
>> information is being used in it's creation, so I can get to a plcae
>> really where only the url is being used as the hash...
>>     
>
> The way your vcl_hash is set up only the URL is being used in the hash.
> Vary support cannot be implemented correctly by just including it in the
> hash string, so it isn't.
>
> DES
>   



From phk at phk.freebsd.dk  Mon Mar 10 22:54:19 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 10 Mar 2008 22:54:19 +0000
Subject: Caching issue 
In-Reply-To: Your message of "Mon, 10 Mar 2008 18:41:11 -0400."
	<47D5B907.6000308@npr.org> 
Message-ID: <3038.1205189659@critter.freebsd.dk>


My guess is  that Vary processing is getting you:


Firefox sends:
	   12 RxHeader     c Accept-Encoding: gzip,deflate
which goes to the backend:
	   15 TxHeader     b Accept-Encoding: gzip,deflate
which the backend uses:
	   15 RxHeader     b Vary: Accept-Encoding
	   15 RxHeader     b Content-Encoding: gzip

Then IE sends:
	   12 RxHeader     c Accept-Encoding: gzip, deflate

Which you will notice, has a space between "gzip," and "deflate"
thus not matching the Vary string already on record in the
cached object.

Varnish requires exact matching, the logic to find out if they
match semantically is not trivial to implement.

You could try the following in vcl_recv:

	if (req.http.accept-encoding) {
		set req.http.accept-encoding = regsub(
		    req.http.accept-encoding,
		    "gzip, *", "gzip,");
	}

to get rid of the space(s), but it is not guaranteed to get all cases.

Alternatively, the more brutal:

	if (req.http.accept-encoding ~ "gzip") {
		set req.http.accept-encoding = "gzip";
	} else {
		unset req.http.accept-encoding;
	}

Will get the desired effect in all cases, provided your backend does
not attempt deflate as fallback.

And finally, giving up on gzip'ing in toto, which is probably a bad choice:

	unset req.http.accept-encoding;

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From jyri at virkki.com  Mon Mar 10 23:35:42 2008
From: jyri at virkki.com (Jyri J. Virkki)
Date: Mon, 10 Mar 2008 16:35:42 -0700
Subject: cache_acceptor_poll.c oddities (Solaris)
Message-ID: <20080310233542.GA28527@localhost.localdomain>


Hello,

Starting varnish (current code from svn) on Solaris, without any
requests I see it sits there with one thread using up all of 1 cpu,
stuck in a poll loop (per truss).

I see on Solaris it ends up using cache_acceptor_poll.c by default and
the problem is that vca_main() calls poll with a pollfd mostly full of
bad data.

Happens because vca_poll() is called earlier once with one fd (fd=7
here) and so it initializes pollfd[7], but all of pollfd[1..6] remain
uninitialized. I see code in vca_pollspace() that presumably attempted to
initialize the space but doesn't really (the loop only sets "p->fd = -1"
with a constant p so only the 0th entry is initialized).  It seems as if
the code was really wanting to do something like shown in diff below?

Presumably this must be working correctly on other platforms so before
filing a bug I thought I'd ask if I'm missing something obvious.
I'll try it on my Linux box later to see what's different. (This diff
isn't enough to make it work at all on Solaris, there are other
problems I see, but wanted to clarify this bit first).

Thanks...


Index: cache_acceptor_poll.c
===================================================================
--- cache_acceptor_poll.c       (revision 2590)
+++ cache_acceptor_poll.c       (working copy)
@@ -58,6 +58,7 @@
 {
        struct pollfd *p;
        unsigned u, v;
+       unsigned oldnpoll = npoll;
 
        if (fd < npoll)
                return;
@@ -69,26 +70,40 @@
        p = realloc(pollfd, u * sizeof *p);
        XXXAN(p);       /* close offending fd */
        memset(p + npoll, 0, (u - npoll) * sizeof *p);
-       for (v = npoll ; v <= u; v++)
-               p->fd = -1;
+       for (v = oldnpoll ; v < u; v++)
+               (p+v)->fd = -1;
        pollfd = p;
        npoll = u;
 }


-- 
Jyri J. Virkki - Santa Cruz, CA






-- 


From des at linpro.no  Tue Mar 11 09:15:00 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 11 Mar 2008 10:15:00 +0100
Subject: Caching issue
In-Reply-To: <47D5B907.6000308@npr.org> (Shain Miley's message of "Mon\, 10 Mar
	2008 18\:41\:11 -0400")
References: <47D1AF3A.50409@npr.org>
	<BC5548FB-6291-4E7B-A0B0-DC2F498759AB@mosso.com>
	<47D1C0B0.4030007@npr.org> <87zlt9dx6m.fsf@des.linpro.no>
	<3EE3DE9B48263E4AB242CACE3874560507FBC3@hq-exch01.npr_usa.ad.npr.org>
	<87ve3wdy67.fsf@des.linpro.no>
	<3EE3DE9B48263E4AB242CACE3874560507FBC4@hq-exch01.npr_usa.ad.npr.org>
	<87pru3dkri.fsf@des.linpro.no> <47D5B907.6000308@npr.org>
Message-ID: <87r6ehd3l7.fsf@des.linpro.no>

Shain Miley <smiley at npr.org> writes:
> Here is the output from an original request...nothing is cached at
> this point..I hope it has the backend info you need.

Well, I don't need anything since I've already found the problem :)
But yes, the backend information is there (lines that have 'b' instead
of 'c' in the third column)

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Mar 11 09:22:58 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 11 Mar 2008 10:22:58 +0100
Subject: Caching issue
In-Reply-To: <3038.1205189659@critter.freebsd.dk> (Poul-Henning Kamp's message
	of "Mon\, 10 Mar 2008 22\:54\:19 +0000")
References: <3038.1205189659@critter.freebsd.dk>
Message-ID: <87k5k9d37x.fsf@des.linpro.no>

"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
> You could try the following in vcl_recv:
>
> 	if (req.http.accept-encoding) {
> 		set req.http.accept-encoding = regsub(
> 		    req.http.accept-encoding,
> 		    "gzip, *", "gzip,");
> 	}
>
> to get rid of the space(s), but it is not guaranteed to get all cases.
>
> Alternatively, the more brutal:
>
> 	if (req.http.accept-encoding ~ "gzip") {
> 		set req.http.accept-encoding = "gzip";
> 	} else {
> 		unset req.http.accept-encoding;
> 	}
>
> Will get the desired effect in all cases, provided your backend does
> not attempt deflate as fallback.

A slightly more complex solution, to cover all bases without losing
functionality:

set req.http.accept-encoding = regsub(req.http.accept-encoding,
    "^ *gzip, *deflate *$", "gzip,deflate");
set req.http.accept-encoding = regsub(req.http.accept-encoding,
    "^ *deflate, *gzip *$", "gzip,deflate");

This should take care of >99% of cases; I don't know of any browser that
supports only one of the two, or supports anything *but* those two.

However, Apache only supports gzip, so Poul-Henning's solution is
sufficient in this case.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From phk at phk.freebsd.dk  Tue Mar 11 10:04:03 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 11 Mar 2008 10:04:03 +0000
Subject: Caching issue 
In-Reply-To: Your message of "Tue, 11 Mar 2008 10:22:58 +0100."
	<87k5k9d37x.fsf@des.linpro.no> 
Message-ID: <5567.1205229843@critter.freebsd.dk>


Is this FAQ fodder ?

In message <87k5k9d37x.fsf at des.linpro.no>, =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=
 writes:
>"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
>> You could try the following in vcl_recv:
>>
>> 	if (req.http.accept-encoding) {
>> 		set req.http.accept-encoding =3D regsub(
>> 		    req.http.accept-encoding,
>> 		    "gzip, *", "gzip,");
>> 	}
>>
>> to get rid of the space(s), but it is not guaranteed to get all cases.
>>
>> Alternatively, the more brutal:
>>
>> 	if (req.http.accept-encoding ~ "gzip") {
>> 		set req.http.accept-encoding =3D "gzip";
>> 	} else {
>> 		unset req.http.accept-encoding;
>> 	}
>>
>> Will get the desired effect in all cases, provided your backend does
>> not attempt deflate as fallback.
>
>A slightly more complex solution, to cover all bases without losing
>functionality:
>
>set req.http.accept-encoding =3D regsub(req.http.accept-encoding,
>    "^ *gzip, *deflate *$", "gzip,deflate");
>set req.http.accept-encoding =3D regsub(req.http.accept-encoding,
>    "^ *deflate, *gzip *$", "gzip,deflate");
>
>This should take care of >99% of cases; I don't know of any browser that
>supports only one of the two, or supports anything *but* those two.
>
>However, Apache only supports gzip, so Poul-Henning's solution is
>sufficient in this case.
>
>DES
>--=20
>Dag-Erling Sm=C3=B8rgrav
>Senior Software Developer
>Linpro AS - www.linpro.no
>

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From des at linpro.no  Tue Mar 11 11:01:11 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 11 Mar 2008 12:01:11 +0100
Subject: Caching issue
In-Reply-To: <5567.1205229843@critter.freebsd.dk> (Poul-Henning Kamp's message
	of "Tue\, 11 Mar 2008 10\:04\:03 +0000")
References: <5567.1205229843@critter.freebsd.dk>
Message-ID: <877ig9cyo8.fsf@des.linpro.no>

"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
> Is this FAQ fodder ?

Yes, I'll stick it in the wiki.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Tue Mar 11 11:04:25 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Tue, 11 Mar 2008 12:04:25 +0100
Subject: cache_acceptor_poll.c oddities (Solaris)
In-Reply-To: <20080310233542.GA28527@localhost.localdomain> (Jyri J. Virkki's
	message of "Mon\, 10 Mar 2008 16\:35\:42 -0700")
References: <20080310233542.GA28527@localhost.localdomain>
Message-ID: <873aqxcyiu.fsf@des.linpro.no>

"Jyri J. Virkki" <jyri at virkki.com> writes:
> Starting varnish (current code from svn) on Solaris, without any
> requests I see it sits there with one thread using up all of 1 cpu,
> stuck in a poll loop (per truss).

The poll code has seen very little maintenance or testing lately, simply
because there are better alternatives on all the platforms we officially
support.  Theo Schlossnagle has an acceptor implementation that uses
Solaris event ports; you may have more luck with that than with poll.

In any case, thank you for the patch; I will commit it as soon as
possible.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From ay at vg.no  Tue Mar 11 15:31:49 2008
From: ay at vg.no (Audun Ytterdal)
Date: Tue, 11 Mar 2008 16:31:49 +0100
Subject: SSL support
In-Reply-To: <87tzjlzbsg.fsf@des.linpro.no>
References: <647725.78771.qm@web63907.mail.re1.yahoo.com> 
	<87tzjlzbsg.fsf@des.linpro.no>
Message-ID: <47D6A5E5.9090604@vg.no>

Dag-Erling Sm?rgrav wrote:
> Louis Luo <lluo_2wire at yahoo.com> writes:
>> BTW, some ticket mentioned that compression would be added to 2.0.
>> How is it now?  I need this feature badly.  Do you guys have some
>> early work that I can try?
> 
> That was a wishlist item, not a promised feature.  However, Varnish 2.0
> supports Vary, so if can enable compression on your backend, it will
> "just work" - but DO NOT UNDER ANY CIRCUMSTANCES use the sample
> configuration at the top of Apache's mod_deflate documentation, as it
> will effectively make every compressable document uncacheable.

You mean

"AddOutputFilterByType DEFLATE text/html text/plain text/xml"

Why will that make documents uncacheable?

Would be sad for non-gzipable browser, but that would be fixed by

"Header append Vary User-Agent"

Or?


--
Audun


*****************************************************************
Denne fotnoten bekrefter at denne e-postmeldingen ble
skannet av MailSweeper og funnet fri for virus.
*****************************************************************
This footnote confirms that this email message has been 
swept by MailSweeper for the presence of computer viruses.
*****************************************************************



From des at linpro.no  Wed Mar 12 16:54:33 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Wed, 12 Mar 2008 17:54:33 +0100
Subject: SSL support
In-Reply-To: <47D6A5E5.9090604@vg.no> (Audun Ytterdal's message of "Tue\, 11
	Mar 2008 16\:31\:49 +0100")
References: <647725.78771.qm@web63907.mail.re1.yahoo.com>
	<87tzjlzbsg.fsf@des.linpro.no> <47D6A5E5.9090604@vg.no>
Message-ID: <87zlt37uie.fsf@des.linpro.no>

Audun Ytterdal <ay at vg.no> writes:
> Dag-Erling Sm?rgrav wrote:
> > That was a wishlist item, not a promised feature.  However, Varnish 2.0
> > supports Vary, so if can enable compression on your backend, it will
> > "just work" - but DO NOT UNDER ANY CIRCUMSTANCES use the sample
> > configuration at the top of Apache's mod_deflate documentation, as it
> > will effectively make every compressable document uncacheable.
>
> You mean
>
> "AddOutputFilterByType DEFLATE text/html text/plain text/xml"
>
> Why will that make documents uncacheable?

No, that part is fine.  I was referring to the "Compress everything
except images" example right below it.

> Would be sad for non-gzipable browser, but that would be fixed by
>
> "Header append Vary User-Agent"

Absolutely not.  This will force Varnish to cache a separate copy of the
document for every single User-Agent string it encounters.

Browsers that do not support gzip or deflate do not need special
treatment: they do not send Accept-Encoding: and will therefore get an
uncompressed version of the page.

The only browsers that advertise gzip / deflate support but do not
actually support it (and therefore need the BrowserMatch rules) are
early versions of Netscape 4.  According to thecounter.com's December
2007 statistics, Netscape 4's market share is 0.00067 - less than one in
one thousand.

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From des at linpro.no  Wed Mar 12 17:50:16 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Wed, 12 Mar 2008 18:50:16 +0100
Subject: Caching issue
In-Reply-To: <877ig9cyo8.fsf@des.linpro.no> ("Dag-Erling =?utf-8?Q?Sm?=
	=?utf-8?Q?=C3=B8rgrav=22's?= message
	of "Tue\, 11 Mar 2008 12\:01\:11 +0100")
References: <5567.1205229843@critter.freebsd.dk> <877ig9cyo8.fsf@des.linpro.no>
Message-ID: <87ve3r7rxj.fsf@des.linpro.no>

Dag-Erling Sm?rgrav <des at linpro.no> writes:
> "Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
> > Is this FAQ fodder ?
> Yes, I'll stick it in the wiki.

I added a FAQ entry which links to a separate page on the subject:

http://varnish.projects.linpro.no/wiki/FAQ/Compression

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no


From jodok at lovelysystems.com  Wed Mar 12 18:26:03 2008
From: jodok at lovelysystems.com (Jodok Batlogg)
Date: Wed, 12 Mar 2008 19:26:03 +0100
Subject: varnish consuming too much memory?
Message-ID: <ABDACB78-4994-43B5-A775-986EA2198A9A@lovelysystems.com>

hi,

i'm wondering how varnish determines how much memory it should  
consume...
we're running multiple varnishes on one box and now and then one  
instance starts to use more and more memory.
right now it's using 65% out of 16GB ram, which is pretty much :)

varnishlog shows that pretty much traffic is going on :)
varnishhist shows that the requests are beeing served really fast
varnishtop shows: the following:



list length  
569 
                                                                                                       hu

   4537.38 VCL_retur deliver
   2655.35 TxProtoco HTTP/1.1
   2397.58 RxProtoco HTTP/1.0
   2397.58 RxHeader  Host: www.zoomer.de
   2397.58 RxHeader  Connection: close
   2397.58 VCL_call  recv
   2397.58 VCL_retur lookup
   2397.58 VCL_call  hash
   2397.58 VCL_retur hash
   2397.58 VCL_call  deliver
   2397.58 TxHeader  Via: 1.1 varnish
   2397.58 TxHeader  Connection: close
   2397.58 SessionCl Connection: close
   2378.81 RxRequest GET
   2261.07 TxHeader  Server: nginx/0.5.35
   2261.07 TxHeader  Keep-Alive: timeout=20
   2252.21 TxStatus  200
   2245.30 TxHeader  X-Powered-By: Zope (www.zope.org), Python (www.python.org 
)
   2236.44 TxRespons Ok
   2139.80 VCL_call  hit
   1450.22 RxHeader  Accept: */*
   1028.78 TxHeader  Content-Type: text/html;charset=utf-8
    826.47 RxHeader  Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    820.48 RxHeader  Accept-Encoding: gzip,deflate
    796.95 RxHeader  User-Agent: Wget/1.10.2
    796.95 RxHeader  Authorization: Basic aHVtYm9sZHQ6QWxleDA3
    737.62 RxHeader  Accept-Encoding: gzip, deflate
    720.28 RxHeader  Accept-Language: de-de,de;q=0.8,en- 
us;q=0.5,en;q=0.3
    685.15 RxHeader  Cookie: uid=5BbkCkfYHlkLGChJBTgCAg==
    614.14 RxHeader  User-Agent: Mozilla/5.0 (Windows; U; Windows NT  
5.1; de; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.1
    585.94 RxHeader  Accept: text/xml,application/xml,application/xhtml 
+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/
    484.43 TxHeader  Content-Type: text/xml;charset=utf-8
    480.47 RxURL     /xml/generic/humboldt/top1.xml
    480.47 Hit       350226425
    480.47 Length    1555
    480.47 TxHeader  Last-Modified: Wed, 12 Mar 2008 18:16:06 GMT


any idea?

jodok

--
"Beautiful is better than ugly."
   -- The Zen of Python, by Tim Peters

Jodok Batlogg, Lovely Systems GmbH
Schmelzh?tterstra?e 26a, 6850 Dornbirn, Austria
mobile: +43 676 5683591, phone: +43 5572 908060

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2454 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20080312/d1822930/attachment.bin>

From phk at phk.freebsd.dk  Wed Mar 12 18:59:49 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 12 Mar 2008 18:59:49 +0000
Subject: varnish consuming too much memory? 
In-Reply-To: Your message of "Wed, 12 Mar 2008 19:26:03 +0100."
	<ABDACB78-4994-43B5-A775-986EA2198A9A@lovelysystems.com> 
Message-ID: <88506.1205348389@critter.freebsd.dk>


>i'm wondering how varnish determines how much memory it should  
>consume...

It uses all memory it can for caching.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From jodok at lovelysystems.com  Wed Mar 12 22:03:59 2008
From: jodok at lovelysystems.com (Jodok Batlogg)
Date: Wed, 12 Mar 2008 23:03:59 +0100
Subject: varnish consuming too much memory? 
In-Reply-To: <88506.1205348389@critter.freebsd.dk>
References: <88506.1205348389@critter.freebsd.dk>
Message-ID: <33CD53F7-74C0-447C-8E0C-8F40365B7D7D@lovelysystems.com>


On 12.03.2008, at 19:59, Poul-Henning Kamp wrote:

>
>> i'm wondering how varnish determines how much memory it should
>> consume...
>
> It uses all memory it can for caching.

- is there a way to limit it?
i'd like to assign the 4 varnishes running on this box 3 GB each and  
leave 4GB for linux disk/nfs cache

- what effect has the size of the VARNISH_STORAGE_SIZE?
is this cache persistent? when are objects moved from memory to disk?

thanks

jodok

>
>
> -- 
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by  
> incompetence.

--
"Beautiful is better than ugly."
   -- The Zen of Python, by Tim Peters

Jodok Batlogg, Lovely Systems GmbH
Schmelzh?tterstra?e 26a, 6850 Dornbirn, Austria
mobile: +43 676 5683591, phone: +43 5572 908060

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2454 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20080312/2339e655/attachment.bin>

From phk at phk.freebsd.dk  Thu Mar 13 09:56:52 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 13 Mar 2008 09:56:52 +0000
Subject: varnish consuming too much memory? 
In-Reply-To: Your message of "Wed, 12 Mar 2008 23:03:59 +0100."
	<33CD53F7-74C0-447C-8E0C-8F40365B7D7D@lovelysystems.com> 
Message-ID: <30140.1205402212@critter.freebsd.dk>

In message <33CD53F7-74C0-447C-8E0C-8F40365B7D7D at lovelysystems.com>, Jodok Batl
ogg writes:

>>> i'm wondering how varnish determines how much memory it should
>>> consume...
>>
>> It uses all memory it can for caching.
>
>- is there a way to limit it?

no.

>i'd like to assign the 4 varnishes running on this box 3 GB each and  
>leave 4GB for linux disk/nfs cache

Why are you running multiple varnishes on the same box ?

>- what effect has the size of the VARNISH_STORAGE_SIZE?

That sets the size of the file varnish maps for storage, and in a sort
of way limits the amount of memory varnish can use, but also severely
limits your hit-rate when it runs full.

>is this cache persistent? when are objects moved from memory to disk?

No.

Whenever the kernel VM subsystem decide it should happen.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From jodok at lovelysystems.com  Thu Mar 13 10:27:32 2008
From: jodok at lovelysystems.com (Jodok Batlogg)
Date: Thu, 13 Mar 2008 11:27:32 +0100
Subject: varnish consuming too much memory? 
In-Reply-To: <30140.1205402212@critter.freebsd.dk>
References: <30140.1205402212@critter.freebsd.dk>
Message-ID: <B1F957D0-3A98-441A-B8D6-F940EBDA5E91@lovelysystems.com>

On 13.03.2008, at 10:56, Poul-Henning Kamp wrote:

> In message <33CD53F7-74C0-447C-8E0C-8F40365B7D7D at lovelysystems.com>,  
> Jodok Batl
> ogg writes:
>
>>>> i'm wondering how varnish determines how much memory it should
>>>> consume...
>>>
>>> It uses all memory it can for caching.
>>
>> - is there a way to limit it?
>
> no.
>
>> i'd like to assign the 4 varnishes running on this box 3 GB each and
>> leave 4GB for linux disk/nfs cache
>
> Why are you running multiple varnishes on the same box ?

we're hosting multiple sites on our cluster. the cluster has two load  
balanced reverse proxies that handle delivery of static content  
(nginx) and caching (varnish).
the different sites are running under different users. i'd like to  
share the resources "fair" between them. the sites have different  
rules for content expiry, allowed purge hosts,...

some of the static files are beeing delivered from NFS, with netfs  
we're using linux cache memory to cache the files as well.
right now (one) varnish eats up all memory, leaves no memory for  
caching and the system starts swapping.


>> - what effect has the size of the VARNISH_STORAGE_SIZE?
>
> That sets the size of the file varnish maps for storage, and in a sort
> of way limits the amount of memory varnish can use, but also severely
> limits your hit-rate when it runs full.

can you elaborate or point me to some documentation?

>> is this cache persistent? when are objects moved from memory to disk?
>
> No.
>
> Whenever the kernel VM subsystem decide it should happen.

so it is a kind of home-made swap memory for varnish?

thanks

jodok

>
>
> -- 
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by  
> incompetence.

--
"Beautiful is better than ugly."
   -- The Zen of Python, by Tim Peters

Jodok Batlogg, Lovely Systems GmbH
Schmelzh?tterstra?e 26a, 6850 Dornbirn, Austria
mobile: +43 676 5683591, phone: +43 5572 908060

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2454 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20080313/f1cfc89c/attachment.bin>

From dave at cheney.net  Thu Mar 13 10:47:52 2008
From: dave at cheney.net (Dave Cheney)
Date: Thu, 13 Mar 2008 21:47:52 +1100
Subject: varnish consuming too much memory? 
In-Reply-To: <B1F957D0-3A98-441A-B8D6-F940EBDA5E91@lovelysystems.com>
References: <30140.1205402212@critter.freebsd.dk>
	<B1F957D0-3A98-441A-B8D6-F940EBDA5E91@lovelysystems.com>
Message-ID: <BFFA7925-122A-45CA-ABB5-1DEC0B2C5E47@cheney.net>

Hi Jodok

	Please take a look at

	http://varnish.projects.linpro.no/wiki/ArchitectNotes

	It should give you a pretty clear idea about the design choices that  
the varnish team have made.

Cheers

Dave

On 13/03/2008, at 9:27 PM, Jodok Batlogg wrote:

> can you elaborate or point me to some documentation?
>
>>> is this cache persistent? when are objects moved from memory to  
>>> disk?
>>
>> No.
>>
>> Whenever the kernel VM subsystem decide it should happen.
>
> so it is a kind of home-made swap memory for varnish?
>
> thanks
>
> jodok

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20080313/b3571f8b/attachment.html>

From ask at develooper.com  Thu Mar 13 14:43:11 2008
From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Thu, 13 Mar 2008 07:43:11 -0700
Subject: varnish consuming too much memory? 
In-Reply-To: <B1F957D0-3A98-441A-B8D6-F940EBDA5E91@lovelysystems.com>
References: <30140.1205402212@critter.freebsd.dk>
	<B1F957D0-3A98-441A-B8D6-F940EBDA5E91@lovelysystems.com>
Message-ID: <5230871D-B903-463F-905D-6E8DCFF032C5@develooper.com>


On Mar 13, 2008, at 3:27, Jodok Batlogg wrote:

> the different sites are running under different users. i'd like to  
> share the resources "fair" between them.

If your platform supports virtualization then you can split up your  
system into N virtual boxes to limit how much real memory varnish uses  
for each...



  - ask

-- 
http://develooper.com/ - http://askask.com/




From jodok at lovelysystems.com  Thu Mar 13 21:35:31 2008
From: jodok at lovelysystems.com (Jodok Batlogg)
Date: Thu, 13 Mar 2008 22:35:31 +0100
Subject: varnish consuming too much memory? 
In-Reply-To: <BFFA7925-122A-45CA-ABB5-1DEC0B2C5E47@cheney.net>
References: <30140.1205402212@critter.freebsd.dk>
	<B1F957D0-3A98-441A-B8D6-F940EBDA5E91@lovelysystems.com>
	<BFFA7925-122A-45CA-ABB5-1DEC0B2C5E47@cheney.net>
Message-ID: <861F88BE-E0A9-46C2-A835-AAEF5446C4B7@lovelysystems.com>


On 13.03.2008, at 11:47, Dave Cheney wrote:

> Hi Jodok
>
> 	Please take a look at
>
> 	http://varnish.projects.linpro.no/wiki/ArchitectNotes
>
> 	It should give you a pretty clear idea about the design choices  
> that the varnish team have made.

i've read that document beforehand - but i don't get the point why it  
should be impossible to limit system resources?

jodok

>
> Cheers
>
> Dave
>
> On 13/03/2008, at 9:27 PM, Jodok Batlogg wrote:
>
>> can you elaborate or point me to some documentation?
>>
>>>> is this cache persistent? when are objects moved from memory to  
>>>> disk?
>>>
>>> No.
>>>
>>> Whenever the kernel VM subsystem decide it should happen.
>>
>> so it is a kind of home-made swap memory for varnish?
>>
>> thanks
>>
>> jodok
>

--
"Beautiful is better than ugly."
   -- The Zen of Python, by Tim Peters

Jodok Batlogg, Lovely Systems GmbH
Schmelzh?tterstra?e 26a, 6850 Dornbirn, Austria
mobile: +43 676 5683591, phone: +43 5572 908060



From jodok at lovelysystems.com  Thu Mar 13 21:38:13 2008
From: jodok at lovelysystems.com (Jodok Batlogg)
Date: Thu, 13 Mar 2008 22:38:13 +0100
Subject: varnish consuming too much memory? 
In-Reply-To: <5230871D-B903-463F-905D-6E8DCFF032C5@develooper.com>
References: <30140.1205402212@critter.freebsd.dk>
	<B1F957D0-3A98-441A-B8D6-F940EBDA5E91@lovelysystems.com>
	<5230871D-B903-463F-905D-6E8DCFF032C5@develooper.com>
Message-ID: <7FC400FB-7BB0-4979-81DE-7EE538AED27A@lovelysystems.com>

On 13.03.2008, at 15:43, Ask Bj?rn Hansen wrote:

>
> On Mar 13, 2008, at 3:27, Jodok Batlogg wrote:
>
>> the different sites are running under different users. i'd like to  
>> share the resources "fair" between them.
>
> If your platform supports virtualization then you can split up your  
> system into N virtual boxes to limit how much real memory varnish  
> uses for each...

well - if you're serving 500mbit/s you don't want a virtualized  
system...
probably i really need to create a more complicated vcl that allows to  
run all applications within the same cache space.

but still it's open how it is possible to reserve some GB of RAM for  
cache memory...

or is it really true that i need dedicated boxes for caching only?

jodok

>
>
>
>
> - ask
>
> -- 
> http://develooper.com/ - http://askask.com/
>
>

--
"Beautiful is better than ugly."
   -- The Zen of Python, by Tim Peters

Jodok Batlogg, Lovely Systems GmbH
Schmelzh?tterstra?e 26a, 6850 Dornbirn, Austria
mobile: +43 676 5683591, phone: +43 5572 908060



From phk at phk.freebsd.dk  Thu Mar 13 21:45:09 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 13 Mar 2008 21:45:09 +0000
Subject: varnish consuming too much memory? 
In-Reply-To: Your message of "Thu, 13 Mar 2008 22:38:13 +0100."
	<7FC400FB-7BB0-4979-81DE-7EE538AED27A@lovelysystems.com> 
Message-ID: <39650.1205444709@critter.freebsd.dk>

In message <7FC400FB-7BB0-4979-81DE-7EE538AED27A at lovelysystems.com>, Jodok Batl
ogg writes:

>or is it really true that i need dedicated boxes for caching only?

If your operating system has a VM system worth its salt, no.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From jodok at lovelysystems.com  Thu Mar 13 21:50:46 2008
From: jodok at lovelysystems.com (Jodok Batlogg)
Date: Thu, 13 Mar 2008 22:50:46 +0100
Subject: varnish consuming too much memory? 
In-Reply-To: <39650.1205444709@critter.freebsd.dk>
References: <39650.1205444709@critter.freebsd.dk>
Message-ID: <438E9577-CF3C-4328-B108-402128196ED6@lovelysystems.com>


On 13.03.2008, at 22:45, Poul-Henning Kamp wrote:

> In message <7FC400FB-7BB0-4979-81DE-7EE538AED27A at lovelysystems.com>,  
> Jodok Batl
> ogg writes:
>
>> or is it really true that i need dedicated boxes for caching only?
>
> If your operating system has a VM system worth its salt, no.

so you mean - in case the box is busy accessing objects from cache  
memory and busy accessing some items from varnish cache and another  
application is busy and accesses some other objects from memory  
frequently my 2.6 linux kernel is intelligent enough to decide what to  
keep in memory and what to put to swap.
that would be great :)

what is varnish's disk file exactly for? is varnish using "real"  
memory only and "swapping" to its own file?

thanks

jodok

>
>
>
> -- 
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by  
> incompetence.

--
"Beautiful is better than ugly."
   -- The Zen of Python, by Tim Peters

Jodok Batlogg, Lovely Systems GmbH
Schmelzh?tterstra?e 26a, 6850 Dornbirn, Austria
mobile: +43 676 5683591, phone: +43 5572 908060



From phk at phk.freebsd.dk  Thu Mar 13 22:00:45 2008
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 13 Mar 2008 22:00:45 +0000
Subject: varnish consuming too much memory? 
In-Reply-To: Your message of "Thu, 13 Mar 2008 22:50:46 +0100."
	<438E9577-CF3C-4328-B108-402128196ED6@lovelysystems.com> 
Message-ID: <39754.1205445645@critter.freebsd.dk>

In message <438E9577-CF3C-4328-B108-402128196ED6 at lovelysystems.com>, Jodok Batl
ogg writes:

>what is varnish's disk file exactly for? is varnish using "real" =20
>memory only and "swapping" to its own file?

it is the backingstore to which the kernel can page the virtual memory 
which Varnish uses.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From jyri at virkki.com  Sat Mar 15 00:49:16 2008
From: jyri at virkki.com (Jyri J. Virkki)
Date: Fri, 14 Mar 2008 17:49:16 -0700
Subject: cache_acceptor_poll.c oddities (Solaris)
In-Reply-To: <873aqxcyiu.fsf@des.linpro.no>
References: <20080310233542.GA28527@localhost.localdomain>
	<873aqxcyiu.fsf@des.linpro.no>
Message-ID: <20080315004916.GA5904@localhost.localdomain>

Once upon a time Dag-Erling Sm?rgrav wrote:
>
> The poll code has seen very little maintenance or testing lately, simply
> because there are better alternatives on all the platforms we officially
> support.  Theo Schlossnagle has an acceptor implementation that uses
> Solaris event ports; you may have more luck with that than with poll.
> 
> In any case, thank you for the patch; I will commit it as soon as
> possible.

While I'd meant it as more of an illustrative than final patch, I see
you massaged it during checkin already, so thanks!

The other immediate problem I saw I've filed as ticket 222[1] and
included the patch there (related to IOV_MAX limit on Solaris, which I
see from the archives was discussed earlier and the define was changed,
but didn't quite work).

I realize the poll implementation isn't that interesting but I figured
I'd start with it as an initial experiment and add an event port
alternative later, but nice to hear it's been done.

Theo, are you planning on contributing it to the main trunk to make it
easier to access and keep in sync?



[1]http://varnish.projects.linpro.no/ticket/222
-- 
Jyri J. Virkki - Santa Cruz, CA






-- 


From jesus at omniti.com  Sat Mar 15 16:25:36 2008
From: jesus at omniti.com (Theo Schlossnagle)
Date: Sat, 15 Mar 2008 12:25:36 -0400
Subject: cache_acceptor_poll.c oddities (Solaris)
In-Reply-To: <20080315004916.GA5904@localhost.localdomain>
References: <20080310233542.GA28527@localhost.localdomain>
	<873aqxcyiu.fsf@des.linpro.no>
	<20080315004916.GA5904@localhost.localdomain>
Message-ID: <F6235C04-84BE-4268-B8C8-C3C4D8FE1C0F@omniti.com>


On Mar 14, 2008, at 8:49 PM, Jyri J. Virkki wrote:

> Once upon a time Dag-Erling Sm?rgrav wrote:
>>
>> The poll code has seen very little maintenance or testing lately,  
>> simply
>> because there are better alternatives on all the platforms we  
>> officially
>> support.  Theo Schlossnagle has an acceptor implementation that uses
>> Solaris event ports; you may have more luck with that than with poll.
>>
>> In any case, thank you for the patch; I will commit it as soon as
>> possible.
>
> While I'd meant it as more of an illustrative than final patch, I see
> you massaged it during checkin already, so thanks!
>
> The other immediate problem I saw I've filed as ticket 222[1] and
> included the patch there (related to IOV_MAX limit on Solaris, which I
> see from the archives was discussed earlier and the define was  
> changed,
> but didn't quite work).
>
> I realize the poll implementation isn't that interesting but I figured
> I'd start with it as an initial experiment and add an event port
> alternative later, but nice to hear it's been done.
>
> Theo, are you planning on contributing it to the main trunk to make it
> easier to access and keep in sync?

Here's a more recent patch:

http://lethargy.org/~jesus/misc/varnish-cache-2543-sol10-2.diff

I'd love to have the changes put back.  There are a few outstanding  
issues:

1) the ping/pong stuff (cross notify) is done to leverage the  
efficiency of event ports, but it is a bad hack from the integration  
perspective.  I can fix this when I get some time -- don't have enough  
of that at the moment.
2) The socket timeouts aren't supported which is bad and nontrivial to  
fix.  However, adding a very thin read(v)/write(v)/sendfile(v)  
abstraction layer would make this very easy and make the SSL  
featureset trivial to add as well.  I could add this in if the  
developers are interested in it?
3) The VCL compiler stuff is changes to support a short-coming of the  
Sun Studio toolchain.  And while the change should work everywhere,  
phk expressed some dislike for it.  Apparently, it's a system("")  
call, so I could do some trickery by chaining commands to work around  
the issue.  I think my work-around is more to-the-point.  I doubt that  
will be taken back, which is fine -- it's subjective argument over  
which approach is better.
4) The IOV_MAX stuff is still an issue as the Sun Studio C  
preprocessor doesn't work with
#if (IOV_MAX < (HTTP_HDR_MAX * 2))

Aside from those issues.  The patch is working well for us in  
production.  We've hit no issues with (2) at this point which is the  
only part that is bonafide technical problem.

For 2) above, the approach I would take is to change the session fd to  
be an:

typedef struct varnish_io_opset {
   int (*read)(void *op_ctx, void *buf, size_t nbyte);
   int (*write)(void *op_ctx, const void *buf, size_t nbyte);
   ... readv ...
   ... writev ...
   ... sendfile ...

} varnish_io_opset_t;
typedef struct varnish_socket {
   varnish_io_opset_t *ops;
   void *op_ctx;
} varnish_socket_t;

This would also obviate the IOV_MAX trickery as Solaris would get its  
own I/O opset.

Best regards,

Theo

--
Theo Schlossnagle
Esoteric Curio -- http://lethargy.org/
OmniTI Computer Consulting, Inc. -- http://omniti.com/



From des at linpro.no  Sun Mar 16 14:06:23 2008
From: des at linpro.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=)
Date: Sun, 16 Mar 2008 15:06:23 +0100
Subject: cache_acceptor_poll.c oddities (Solaris)
In-Reply-To: <20080315004916.GA5904@localhost.localdomain> (Jyri J. Virkki's
	message of "Fri\, 14 Mar 2008 17\:49\:16 -0700")
References: <20080310233542.GA28527@localhost.localdomain>
	<873aqxcyiu.fsf@des.linpro.no>
	<20080315004916.GA5904@localhost.localdomain>
Message-ID: <87skyqix0g.fsf@des.linpro.no>

"Jyri J. Virkki" <jyri at virkki.com> writes:
> The other immediate problem I saw I've filed as ticket 222[1] and
> included the patch there (related to IOV_MAX limit on Solaris, which I
> see from the archives was discussed earlier and the define was changed,
> but didn't quite work).

Thanks

DES
-- 
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no