From ntwrkd at gmail.com  Tue Jun  1 03:29:12 2010
From: ntwrkd at gmail.com (ntwrkd)
Date: Mon, 31 May 2010 20:29:12 -0700
Subject: memory usage tuning
Message-ID: <AANLkTikG31hi8Dh_M52_eZ8jf8EAVwU_ho2lL5VXtsBN@mail.gmail.com>

Is there a way to tune the memory footprint of varnish cache.
it was discussed somewhat in this thread: http://varnish-cache.org/ticket/546
but it would be nice to see formal guidelines for how one might reduce
the memory footprint of their varnish cache (if it doesn't exist
already)

thanks in advance.



From jdzstz at gmail.com  Tue Jun  1 18:09:38 2010
From: jdzstz at gmail.com (=?ISO-8859-1?Q?Jorge_D=EDaz?=)
Date: Tue, 1 Jun 2010 20:09:38 +0200
Subject: Problems with varnishncsa and duplicated ReqEnd (tickets #633, 
	#685 and #709) - Solaris 10
In-Reply-To: <AANLkTileEWRZKnKQOR_wyCRtO44GPoPzM6NSZKFiqh35@mail.gmail.com>
References: <AANLkTileEWRZKnKQOR_wyCRtO44GPoPzM6NSZKFiqh35@mail.gmail.com>
Message-ID: <AANLkTikIsI8dZSVaRewFzZJmO1dUAbnfmMkW-pKoUTnv@mail.gmail.com>

Hello,

About bugs of varnishncsa in tickets
#633<http://varnish-cache.org/ticket/633>and
#685 <http://varnish-cache.org/ticket/685>  I have modified varnishncsa.c,

It ignores duplicated ReqEnd and dump bogus in stderr ?may be changed to
system error log???

   1. Get XID from ReqEnd
   2. If XID == 0 then ignore ReqEnd
   3. If XID != = but bogus == 1 then print bogus request in standard error.


It works me ok in Solaris 10, and avoid coredumps.


Thank you



2010/5/31 Jorge D?az <jdzstz at gmail.com>

> I am testing Varnish (r4576 <http://varnish-cache.org/changeset/4576>) in
> Solaris 10 5.10 Generic_120011-14 sun4v sparc SUNW,Sun-Fire-T2000.
> We are using now Apache with mod_cache and we are planning to switch to
> Varnish, I have followed the instructions in
> http://letsgetdugg.com/2009/12/04/varnish-on-solaris/<http://letsgetdugg.com/2009/12/04/varnish-on-solaris/>
>
> After fixing the LINGER problem (ticket #649<http://varnish-cache.org/ticket/649>), varnish seems to work ok, we have also make some little tests in
> production enviroment, switching to Varnish during 3 hours in one of our
> eight Apache Servers.
>
> The only problem that is stopping us to making the change are some strange
> behaviour with varnishncsa.
> Original varnishncsa.c version generate coredumps and fixed file with #685<http://varnish-cache.org/ticket/685>workaround generates a lot of bogus requests.
>
> After some investigations I have found that the root of the problems is a
> duplicated ReqEnd record when there is a SessionClose EOF, I have filled the
> bug in new ticket #709 <http://varnish-cache.org/ticket/709>  The
> duplicated ReqEnd has no XID
>
> Tickets #633 <http://varnish-cache.org/ticket/633> and #685<http://varnish-cache.org/ticket/685>
> they are almost the same. I think it would be nice to control the bogus
> requests:
>
>    1. Get XID from ReqEnd
>    2. If XID == 0 then ignore ReqEnd
>    3. If XID != = but bogus == 1 then print bogus request in standard
>    error.
>
> I attach to the mail two varnishncsa.c fixes:
>
>    1. using workaround of #685  and adding extra logging
>    2. retrieving xid and testing "if (!lp->bogus && lp->xid > 0)" and
>    adding extra logging
>
> I have to change the extra logging to stderr.
>
>
> *I need help with #709 <http://varnish-cache.org/ticket/709> , somebody
> has any clue about SessionClose EOF problem?? This issue is reported also in
> varnish-misc list:
>
> http://lists.varnish-cache.org/pipermail/varnish-misc/2009-December/003395.html
> *
> I attach my preproduction logs (with a BOGUS error and a duplicated ReqEnd)
> and my modified varnishncsa:
>
>     7 TxHeader     c X-backend: prepro
>     7 ReqEnd       c 1937859944 1275326684.354095459 1275326684.376112938
> 0.000319004 0.021736622 0.000280857
>     7 SessionClose c EOF
>     7 ReqEnd       c 0 1275326684.386904716 1275326684.386904716
> 0.010791779 0.000000000 0.000000000
>     7 StatSess     c 212.170.156.253 31325 0 1 1 0 0 1 288 0
>
> Thank you
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100601/2d027711/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: varnishncsa.c
Type: application/octet-stream
Size: 14603 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100601/2d027711/attachment.obj>

From perbu at varnish-software.com  Wed Jun  2 18:54:26 2010
From: perbu at varnish-software.com (Per Buer)
Date: Wed, 2 Jun 2010 20:54:26 +0200
Subject: removing the old man pages
Message-ID: <AANLkTilY8rFB7xfpwihqM58y1LLQatr5S8dc5dpzGz9l@mail.gmail.com>

Hi.

I guess we can remove the old man pages from trunk now. I'm not all
done with new pages yet but I guess I will be within a day or two. I
don't have enough make-foo to do this myself.

-- 
Per Buer,  Varnish Software
Phone: +47 21 54 41 21 / Mobile: +47 958 39 117 / skype: per.buer



From varnish-dev at projects.linpro.no  Fri Jun  4 11:33:17 2010
From: varnish-dev at projects.linpro.no (PurchaseViagra Online)
Date: Fri, 4 Jun 2010 07:33:17 -0400
Subject: Check our site, Mr. varnish-dev! You can save 80%. on
Message-ID: <20100604113329.E8A7B1380CB@projects.linpro.no>

An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100604/3ce9a3ab/attachment.html>

From lee.doolan at volcanomail.com  Sun Jun  6 06:54:22 2010
From: lee.doolan at volcanomail.com (lee doolan)
Date: Sat, 5 Jun 2010 23:54:22 -0700
Subject: [PATCH] Concatenate Set-Cookie headers before a call to vcl_fetch()
Message-ID: <20100605235422.4E0F3B7@resin14.mta.everyone.net>


I have modified a 2.1.2el5 distribution slightly.  

All of the set-cookie headers from a backend response are
concatenated, with separators between them, and then the resulting
string is written into a header named x-set-cookies.  This is done
just before a call to vcl_fetch().  After the call to vcl_fetch(), the
x-set-cookies header is removed.  The code is here, in github:

     http://github.com/leed25d/ljVarnishPatch

The code may be slightly rough around the edges as I have not done any
C coding for around 10 years, but the modification works well enough
to make our system admins pretty happy.

--lee



From sky at crucially.net  Sun Jun  6 20:48:34 2010
From: sky at crucially.net (sky at crucially.net)
Date: Sun, 6 Jun 2010 20:48:34 +0000
Subject: [PATCH] Concatenate Set-Cookie headers before a call to
	vcl_fetch()
In-Reply-To: <20100605235422.4E0F3B7@resin14.mta.everyone.net>
References: <20100605235422.4E0F3B7@resin14.mta.everyone.net>
Message-ID: <1056011767-1275857320-cardhu_decombobulator_blackberry.rim.net-958953443-@bda578.bisx.prod.on.blackberry>

Just curious, what is the use case?

Thanks
Artur
Sent via BlackBerry by AT&T

-----Original Message-----
From: "lee doolan" <lee.doolan at volcanomail.com>
Date: Sat, 5 Jun 2010 23:54:22 
To: <varnish-dev at varnish-cache.org>
Subject: [PATCH] Concatenate Set-Cookie headers before a call to vcl_fetch()


I have modified a 2.1.2el5 distribution slightly.  

All of the set-cookie headers from a backend response are
concatenated, with separators between them, and then the resulting
string is written into a header named x-set-cookies.  This is done
just before a call to vcl_fetch().  After the call to vcl_fetch(), the
x-set-cookies header is removed.  The code is here, in github:

     http://github.com/leed25d/ljVarnishPatch

The code may be slightly rough around the edges as I have not done any
C coding for around 10 years, but the modification works well enough
to make our system admins pretty happy.

--lee

_______________________________________________
varnish-dev mailing list
varnish-dev at varnish-cache.org
http://lists.varnish-cache.org/mailman/listinfo/varnish-dev

From lee.doolan at volcanomail.com  Sun Jun  6 21:33:04 2010
From: lee.doolan at volcanomail.com (lee doolan)
Date: Sun, 6 Jun 2010 14:33:04 -0700
Subject: [PATCH] Concatenate Set-Cookie headers before a call to
	vcl_fetch()
Message-ID: <20100606143304.4E01F88@resin14.mta.everyone.net>


Well, the 'Set-Cookie' header is unique amongst the headers of an HTTP
response in the respect that it can occur multiple times.  VCL has no
facility for looping over these headers and deciding which ones to
remove and which ones to keep.  With this patch, you have the option
of
    First, remove all the Set-Cookie headers like this:
        remove beresp.http.Set-Cookie;

    then, replace the ones that you want to keep by extracting them
    from the X-Set-Cookies header with regsuball(). ie:
        set beresp.http.X-Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "keep1 regex"); 
        set beresp.http.X-Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "keep2 regex"); 
        set beresp.http.X-Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "and so forth"); 

        



--- sky at crucially.net wrote:

From: sky at crucially.net
To: lee.doolan at volcanomail.com,varnish-dev at varnish-cache.org
Subject: Re: [PATCH] Concatenate Set-Cookie headers before a call to vcl_fetch()
Date: Sun, 6 Jun 2010 20:48:34 +0000

Just curious, what is the use case?

Thanks
Artur
Sent via BlackBerry by AT&T

-----Original Message-----
From: "lee doolan" <lee.doolan at volcanomail.com>
Date: Sat, 5 Jun 2010 23:54:22 
To: <varnish-dev at varnish-cache.org>
Subject: [PATCH] Concatenate Set-Cookie headers before a call to vcl_fetch()


I have modified a 2.1.2el5 distribution slightly.  

All of the set-cookie headers from a backend response are
concatenated, with separators between them, and then the resulting
string is written into a header named x-set-cookies.  This is done
just before a call to vcl_fetch().  After the call to vcl_fetch(), the
x-set-cookies header is removed.  The code is here, in github:

     http://github.com/leed25d/ljVarnishPatch

The code may be slightly rough around the edges as I have not done any
C coding for around 10 years, but the modification works well enough
to make our system admins pretty happy.

--lee

_______________________________________________
varnish-dev mailing list
varnish-dev at varnish-cache.org
http://lists.varnish-cache.org/mailman/listinfo/varnish-dev





From lee.doolan at volcanomail.com  Mon Jun  7 05:18:48 2010
From: lee.doolan at volcanomail.com (lee doolan)
Date: Sun, 6 Jun 2010 22:18:48 -0700
Subject: [PATCH] Concatenate Set-Cookie headers before a call
	to	vcl_fetch()
Message-ID: <20100606221848.4E03FF0@resin14.mta.everyone.net>

I need to correct a nasty error.  These lines
        set beresp.http.X-Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "keep1 regex"); 
        set beresp.http.X-Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "keep2 regex"); 
        set beresp.http.X-Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "and so forth"); 

should read like this:
        set beresp.http.Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "keep1 regex"); 
        set beresp.http.Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "keep2 regex"); 
        set beresp.http.Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "and so forth"); 

--- lee.doolan at volcanomail.com wrote:

From: "lee doolan" <lee.doolan at volcanomail.com>
To: <sky at crucially.net>
Cc: varnish-dev at varnish-cache.org
Subject: Re: [PATCH] Concatenate Set-Cookie headers before a call to	vcl_fetch()
Date: Sun, 6 Jun 2010 14:33:04 -0700


Well, the 'Set-Cookie' header is unique amongst the headers of an HTTP
response in the respect that it can occur multiple times.  VCL has no
facility for looping over these headers and deciding which ones to
remove and which ones to keep.  With this patch, you have the option
of
    First, remove all the Set-Cookie headers like this:
        remove beresp.http.Set-Cookie;

    then, replace the ones that you want to keep by extracting them
    from the X-Set-Cookies header with regsuball(). ie:
        set beresp.http.X-Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "keep1 regex"); 
        set beresp.http.X-Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "keep2 regex"); 
        set beresp.http.X-Set-Cookie = regsuball(beresp.http.X-Set-Cookies, "and so forth"); 

        



--- sky at crucially.net wrote:

From: sky at crucially.net
To: lee.doolan at volcanomail.com,varnish-dev at varnish-cache.org
Subject: Re: [PATCH] Concatenate Set-Cookie headers before a call to vcl_fetch()
Date: Sun, 6 Jun 2010 20:48:34 +0000

Just curious, what is the use case?

Thanks
Artur
Sent via BlackBerry by AT&T

-----Original Message-----
From: "lee doolan" <lee.doolan at volcanomail.com>
Date: Sat, 5 Jun 2010 23:54:22 
To: <varnish-dev at varnish-cache.org>
Subject: [PATCH] Concatenate Set-Cookie headers before a call to vcl_fetch()


I have modified a 2.1.2el5 distribution slightly.  

All of the set-cookie headers from a backend response are
concatenated, with separators between them, and then the resulting
string is written into a header named x-set-cookies.  This is done
just before a call to vcl_fetch().  After the call to vcl_fetch(), the
x-set-cookies header is removed.  The code is here, in github:

     http://github.com/leed25d/ljVarnishPatch

The code may be slightly rough around the edges as I have not done any
C coding for around 10 years, but the modification works well enough
to make our system admins pretty happy.

--lee

_______________________________________________
varnish-dev mailing list
varnish-dev at varnish-cache.org
http://lists.varnish-cache.org/mailman/listinfo/varnish-dev



_______________________________________________
varnish-dev mailing list
varnish-dev at varnish-cache.org
http://lists.varnish-cache.org/mailman/listinfo/varnish-dev





From l at lrowe.co.uk  Mon Jun  7 17:47:32 2010
From: l at lrowe.co.uk (Laurence Rowe)
Date: Mon, 7 Jun 2010 18:47:32 +0100
Subject: [PATCH] Concatenate Set-Cookie headers before a call to 
	vcl_fetch()
In-Reply-To: <20100606143304.4E01F88@resin14.mta.everyone.net>
References: <20100606143304.4E01F88@resin14.mta.everyone.net>
Message-ID: <AANLkTimgLK0AxEvMFwsXdnu-tb9F6ejshTpWKu9GRsu_@mail.gmail.com>

On 6 June 2010 22:33, lee doolan <lee.doolan at volcanomail.com> wrote:
>
> Well, the 'Set-Cookie' header is unique amongst the headers of an HTTP
> response in the respect that it can occur multiple times. ?VCL has no
> facility for looping over these headers and deciding which ones to
> remove and which ones to keep.

This is true for all headers. See:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

"""
Multiple message-header fields with the same field-name MAY be present
in a message if and only if the entire field-value for that header
field is defined as a comma-separated list [i.e., #(values)]. It MUST
be possible to combine the multiple header fields into one
"field-name: field-value" pair, without changing the semantics of the
message, by appending each subsequent field-value to the first, each
separated by a comma. The order in which header fields with the same
field-name are received is therefore significant to the interpretation
of the combined field value, and thus a proxy MUST NOT change the
order of these field values when a message is forwarded.
"""

So it should be fine for a proxy like Varnish to combine the multiple
headers into a single header, no need for any special X-Set-Cookies.

Laurence



From sky at crucially.net  Mon Jun  7 17:49:16 2010
From: sky at crucially.net (Artur Bergman)
Date: Mon, 7 Jun 2010 10:49:16 -0700
Subject: [PATCH] Concatenate Set-Cookie headers before a call to
	vcl_fetch()
In-Reply-To: <AANLkTimgLK0AxEvMFwsXdnu-tb9F6ejshTpWKu9GRsu_@mail.gmail.com>
References: <20100606143304.4E01F88@resin14.mta.everyone.net>
	<AANLkTimgLK0AxEvMFwsXdnu-tb9F6ejshTpWKu9GRsu_@mail.gmail.com>
Message-ID: <394F9D22-7E3F-4D5E-88CB-A818CEAE58AA@crucially.net>

No, browser break if you do that.

(Nginx source bug reports about this)

Artur

On Jun 7, 2010, at 10:47 AM, Laurence Rowe wrote:

> On 6 June 2010 22:33, lee doolan <lee.doolan at volcanomail.com> wrote:
>>
>> Well, the 'Set-Cookie' header is unique amongst the headers of an  
>> HTTP
>> response in the respect that it can occur multiple times.  VCL has no
>> facility for looping over these headers and deciding which ones to
>> remove and which ones to keep.
>
> This is true for all headers. See:
> http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2
>
> """
> Multiple message-header fields with the same field-name MAY be present
> in a message if and only if the entire field-value for that header
> field is defined as a comma-separated list [i.e., #(values)]. It MUST
> be possible to combine the multiple header fields into one
> "field-name: field-value" pair, without changing the semantics of the
> message, by appending each subsequent field-value to the first, each
> separated by a comma. The order in which header fields with the same
> field-name are received is therefore significant to the interpretation
> of the combined field value, and thus a proxy MUST NOT change the
> order of these field values when a message is forwarded.
> """
>
> So it should be fine for a proxy like Varnish to combine the multiple
> headers into a single header, no need for any special X-Set-Cookies.
>
> Laurence




From varnish at octo.it  Tue Jun  8 20:38:08 2010
From: varnish at octo.it (Florian Forster)
Date: Tue, 8 Jun 2010 22:38:08 +0200
Subject: [PATCH] Link "libvarnishapi" against "libvarnish".
Message-ID: <20100608203807.GF26178@verplant.org>

Hi,

I noticed that "libvarnishapi" uses some symbols found in
"libvarnish"?[0] but doesn't link against that library. If a program
simply links with "-lvarnishapi" [1], those symbols can't be found and
the link will fail.

The source of the problem is probably that "libvarnishapi" uses some
source files from the "libvarnish" directory.

The attached patch solves the problem by linking "libvarnishapi" with
"libvarnish", so a dynamic linker can resolve this dependency
automatically. You might want to add "libvarnish" and "pcre" to the
"Libs.private" field in the .pc file in order to fix static linking.

Best regards,
?octo

[0] At least "VRE_compile" and "VRE_exec".
[1] As propagated by the "varnishapi.pc" pkg-config file.
-- 
Florian octo Forster
Hacker in training
GnuPG: 0x91523C3D
http://verplant.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: link-libvarnishapi.patch
Type: text/x-diff
Size: 406 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100608/f0e12923/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100608/f0e12923/attachment-0001.bin>

From phk at phk.freebsd.dk  Tue Jun  8 20:56:37 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 08 Jun 2010 20:56:37 +0000
Subject: [PATCH] Link "libvarnishapi" against "libvarnish". 
In-Reply-To: Your message of "Tue, 08 Jun 2010 22:38:08 +0200."
	<20100608203807.GF26178@verplant.org> 
Message-ID: <58048.1276030597@critter.freebsd.dk>

In message <20100608203807.GF26178 at verplant.org>, Florian Forster writes:

>I noticed that "libvarnishapi" uses some symbols found in
>"libvarnish"=C2=A0[0] but doesn't link against that library. If a program
>simply links with "-lvarnishapi" [1], those symbols can't be found and
>the link will fail.

Actually I'm trying to uravel that issue right now in -trunk
libvarnishapi shall not link against libvarnish, the entire point
of having a separate lib for the exposed API would be lost.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From varnish at octo.it  Thu Jun 10 10:17:37 2010
From: varnish at octo.it (Florian Forster)
Date: Thu, 10 Jun 2010 12:17:37 +0200
Subject: [PATCH] Link "libvarnishapi" against "libvarnish".
In-Reply-To: <58048.1276030597@critter.freebsd.dk>
References: <20100608203807.GF26178@verplant.org>
	<58048.1276030597@critter.freebsd.dk>
Message-ID: <20100610101737.GL26178@verplant.org>

Hi,

On Tue, Jun 08, 2010 at 08:56:37PM +0000, Poul-Henning Kamp wrote:
> Actually I'm trying to uravel that issue right now in -trunk
> libvarnishapi shall not link against libvarnish, the entire point of
> having a separate lib for the exposed API would be lost.

after your changes the library has the missing "VRE_compile" and
"VRE_exec" symbols and is also correctly linked with libpcre. Thanks for
fixing this :)

However, I now run into the problem that "pthread_once" cannot be
resolved. Oddly enough, "pthread_mutex_*" are resolved correctly. I'm
assuming some GNU libc oddity here.

The attached patch fixed the problem for me by adding "@PTHREAD_LIBS@"
to the pkg-config file for libvarnishapi.

Regards,
?octo
-- 
Florian octo Forster
Hacker in training
GnuPG: 0x91523C3D
http://verplant.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100610/4544bc7a/attachment.bin>

From varnish at octo.it  Thu Jun 10 11:10:56 2010
From: varnish at octo.it (Florian Forster)
Date: Thu, 10 Jun 2010 13:10:56 +0200
Subject: [PATCH] Link "libvarnishapi" against "libvarnish".
In-Reply-To: <20100610101737.GL26178@verplant.org>
References: <20100608203807.GF26178@verplant.org>
	<58048.1276030597@critter.freebsd.dk>
	<20100610101737.GL26178@verplant.org>
Message-ID: <20100610111056.GN26178@verplant.org>

On Thu, Jun 10, 2010 at 12:17:37PM +0200, Florian Forster wrote:
> The attached patch ?

Argh, I'll forget my own head next.. Patch is attached.

?octo
-- 
Florian octo Forster
Hacker in training
GnuPG: 0x91523C3D
http://verplant.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: link-libvarnishapi.patch
Type: text/x-diff
Size: 347 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100610/c5097bfe/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100610/c5097bfe/attachment-0001.bin>

From joec1502 at gmail.com  Fri Jun 11 18:57:28 2010
From: joec1502 at gmail.com (Joe Chen)
Date: Fri, 11 Jun 2010 12:57:28 -0600
Subject: Preloading Cached Contents into Varnish?
Message-ID: <AANLkTik8_FQQq3NXLay9T14E4Yv0CVppyAhGYoq_pDKi@mail.gmail.com>

Would someone kindly tell me how and where I should start in adding a
cache-preloading functionality?

I like Varnish's performance in caching and try to add a capability of
preloading cached contents into Varnish server.

My understanding is that I need to find a right spot for inserting hash
index, and then right spot for inserting cached contents into cache storage.

I also intend to use memory as only cache storage, not hard disk.

Any help would be appreciated very much,

--Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100611/33263b9e/attachment.html>

From aotto at mosso.com  Fri Jun 11 19:23:07 2010
From: aotto at mosso.com (Adrian Otto)
Date: Fri, 11 Jun 2010 12:23:07 -0700
Subject: Preloading Cached Contents into Varnish?
In-Reply-To: <AANLkTik8_FQQq3NXLay9T14E4Yv0CVppyAhGYoq_pDKi@mail.gmail.com>
References: <AANLkTik8_FQQq3NXLay9T14E4Yv0CVppyAhGYoq_pDKi@mail.gmail.com>
Message-ID: <0B16693A-B4A6-456C-A131-CF16A44F20D7@mosso.com>

Joe,

On Jun 11, 2010, at 11:57 AM, Joe Chen wrote:

> Would someone kindly tell me how and where I should start in adding  
> a cache-preloading functionality?
>
> I like Varnish's performance in caching and try to add a capability  
> of preloading cached contents into Varnish server.
>
> My understanding is that I need to find a right spot for inserting  
> hash index, and then right spot for inserting cached contents into  
> cache storage.

Why not just implement a simple HTTP client script to walk the list of  
URL's you want to load on some suitable interval? This way you don't  
need any code modifications.

> I also intend to use memory as only cache storage, not hard disk.

Simply define the storage file/volume to be smaller than your main  
memory minus some capacity for the server to run in. In my use case on  
a server with 32 GB RAM, I use a 30GB storage volume, and I see  
practically no disk access because the cached pages always fit in  
memory.

Regards,

Adrian



From joec1502 at gmail.com  Fri Jun 11 20:40:55 2010
From: joec1502 at gmail.com (Joe Chen)
Date: Fri, 11 Jun 2010 14:40:55 -0600
Subject: Preloading Cached Contents into Varnish?
In-Reply-To: <0B16693A-B4A6-456C-A131-CF16A44F20D7@mosso.com>
References: <AANLkTik8_FQQq3NXLay9T14E4Yv0CVppyAhGYoq_pDKi@mail.gmail.com>
	<0B16693A-B4A6-456C-A131-CF16A44F20D7@mosso.com>
Message-ID: <AANLkTiltTowYx8hksJ1pFuwrALvOkZu0GJAzCASI9smF@mail.gmail.com>

On Fri, Jun 11, 2010 at 1:23 PM, Adrian Otto <aotto at mosso.com> wrote:
>
> I like Varnish's performance in caching and try to add a capability of
>> preloading cached contents into Varnish server.
>>
>> My understanding is that I need to find a right spot for inserting hash
>> index, and then right spot for inserting cached contents into cache storage.
>>
>
> Why not just implement a simple HTTP client script to walk the list of
> URL's you want to load on some suitable interval? This way you don't need
> any code modifications.
>
> This would work if I could pull the contents.  Unfortunately, my cached
contents can not be pulled in, can only be pushed in by an outside  server.
So my own program will receive the pushed-in contents in terms of http
header and body, and then inserted them into the Varnish cache storage.

>
>  I also intend to use memory as only cache storage, not hard disk.
>>
>
> Simply define the storage file/volume to be smaller than your main memory
> minus some capacity for the server to run in. In my use case on a server
> with 32 GB RAM, I use a 30GB storage volume, and I see practically no disk
> access because the cached pages always fit in memory.
>

Didn't known that, I'll definitely try it.  But my running system will be
disk-less, so not sure if I have to change the code or not.

Thanks,

--Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100611/9b93f4ff/attachment.html>

From joec1502 at gmail.com  Mon Jun 14 19:23:12 2010
From: joec1502 at gmail.com (Joe Chen)
Date: Mon, 14 Jun 2010 13:23:12 -0600
Subject: Preloading Cached Contents into Varnish?
In-Reply-To: <AANLkTiltTowYx8hksJ1pFuwrALvOkZu0GJAzCASI9smF@mail.gmail.com>
References: <AANLkTik8_FQQq3NXLay9T14E4Yv0CVppyAhGYoq_pDKi@mail.gmail.com>
	<0B16693A-B4A6-456C-A131-CF16A44F20D7@mosso.com>
	<AANLkTiltTowYx8hksJ1pFuwrALvOkZu0GJAzCASI9smF@mail.gmail.com>
Message-ID: <AANLkTinjTfuJuiXl0rDZcQiKDcUCj74w6YMTMvxx0KF8@mail.gmail.com>

I haven't received any reply besides Adrian's one. Is this emailing list
active?  I still need someone to point me to a good direction of adding
preloaded cache in the storage.

--Joe


On Fri, Jun 11, 2010 at 2:40 PM, Joe Chen <joec1502 at gmail.com> wrote:

>
>
> On Fri, Jun 11, 2010 at 1:23 PM, Adrian Otto <aotto at mosso.com> wrote:
>>
>>  I like Varnish's performance in caching and try to add a capability of
>>> preloading cached contents into Varnish server.
>>>
>>> My understanding is that I need to find a right spot for inserting hash
>>> index, and then right spot for inserting cached contents into cache storage.
>>>
>>
>> Why not just implement a simple HTTP client script to walk the list of
>> URL's you want to load on some suitable interval? This way you don't need
>> any code modifications.
>>
>> This would work if I could pull the contents.  Unfortunately, my cached
> contents can not be pulled in, can only be pushed in by an outside  server.
> So my own program will receive the pushed-in contents in terms of http
> header and body, and then inserted them into the Varnish cache storage.
>
>>
>>  I also intend to use memory as only cache storage, not hard disk.
>>>
>>
>> Simply define the storage file/volume to be smaller than your main memory
>> minus some capacity for the server to run in. In my use case on a server
>> with 32 GB RAM, I use a 30GB storage volume, and I see practically no disk
>> access because the cached pages always fit in memory.
>>
>
> Didn't known that, I'll definitely try it.  But my running system will be
> disk-less, so not sure if I have to change the code or not.
>
> Thanks,
>
> --Joe
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100614/48f02aee/attachment.html>

From phk at phk.freebsd.dk  Mon Jun 14 19:35:48 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 14 Jun 2010 19:35:48 +0000
Subject: Preloading Cached Contents into Varnish? 
In-Reply-To: Your message of "Mon, 14 Jun 2010 13:23:12 CST."
	<AANLkTinjTfuJuiXl0rDZcQiKDcUCj74w6YMTMvxx0KF8@mail.gmail.com> 
Message-ID: <17325.1276544148@critter.freebsd.dk>

In message <AANLkTinjTfuJuiXl0rDZcQiKDcUCj74w6YMTMvxx0KF8 at mail.gmail.com>, Joe 
Chen writes:

>I haven't received any reply besides Adrian's one. Is this emailing list
>active?  I still need someone to point me to a good direction of adding
>preloaded cache in the storage.

Sorry for the delay.

>>> Why not just implement a simple HTTP client script to walk the list of
>>> URL's you want to load on some suitable interval? This way you don't need
>>> any code modifications.

This actually is the canonical answer: adding a lot of code to
Varnish to do what a few lines of script can do, would not make much
sense.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From joec1502 at gmail.com  Mon Jun 14 20:41:23 2010
From: joec1502 at gmail.com (Joe Chen)
Date: Mon, 14 Jun 2010 14:41:23 -0600
Subject: Preloading Cached Contents into Varnish?
In-Reply-To: <17325.1276544148@critter.freebsd.dk>
References: <AANLkTinjTfuJuiXl0rDZcQiKDcUCj74w6YMTMvxx0KF8@mail.gmail.com>
	<17325.1276544148@critter.freebsd.dk>
Message-ID: <AANLkTin88QPuNhY1MPbHVmMBFeJV4u-cq8hQdeNvCTIQ@mail.gmail.com>

On Mon, Jun 14, 2010 at 1:35 PM, Poul-Henning Kamp <phk at phk.freebsd.dk>wrote:

> Chen writes:
> >>> Why not just implement a simple HTTP client script to walk the list of
> >>> URL's you want to load on some suitable interval? This way you don't
> need
> >>> any code modifications.
>
> This actually is the canonical answer: adding a lot of code to
> Varnish to do what a few lines of script can do, would not make much
> sense.
>
> Again, It would have worked for me if I could decide what sources are to
pull by a script.  The fact is that I cannot initiate a pull, all has to be
pushed in by multicast to reduce the traffic loads (there will be many many
platforms running Varnish to receive the same pushed in cached contents).
 So I cannot pull them by a script.

Thanks for your answering me,

--Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100614/0aad3ace/attachment.html>

From sky at crucially.net  Mon Jun 14 20:42:32 2010
From: sky at crucially.net (Artur Bergman)
Date: Mon, 14 Jun 2010 13:42:32 -0700
Subject: Preloading Cached Contents into Varnish?
In-Reply-To: <AANLkTin88QPuNhY1MPbHVmMBFeJV4u-cq8hQdeNvCTIQ@mail.gmail.com>
References: <AANLkTinjTfuJuiXl0rDZcQiKDcUCj74w6YMTMvxx0KF8@mail.gmail.com>
	<17325.1276544148@critter.freebsd.dk>
	<AANLkTin88QPuNhY1MPbHVmMBFeJV4u-cq8hQdeNvCTIQ@mail.gmail.com>
Message-ID: <357BEE7D-5183-455E-97BE-A182A2E35D3C@crucially.net>


On Jun 14, 2010, at 1:41 PM, Joe Chen wrote:

>
>
> On Mon, Jun 14, 2010 at 1:35 PM, Poul-Henning Kamp  
> <phk at phk.freebsd.dk> wrote:
> Chen writes:
> >>> Why not just implement a simple HTTP client script to walk the  
> list of
> >>> URL's you want to load on some suitable interval? This way you  
> don't need
> >>> any code modifications.
>
> This actually is the canonical answer: adding a lot of code to
> Varnish to do what a few lines of script can do, would not make much
> sense.
>
> Again, It would have worked for me if I could decide what sources  
> are to pull by a script.  The fact is that I cannot initiate a pull,  
> all has to be pushed in by multicast to reduce the traffic loads  
> (there will be many many platforms running Varnish to receive the  
> same pushed in cached contents).  So I cannot pull them by a script.
>
> Thanks for your answering me,
>
> --Joe


Yes you can, you can write a 10 line script that listens to the  
multicast and issues the pull.

Artur

If you'd like to unsubscribe and stop receiving these emails click here: http://u10956.sendgrid.info/wf/unsubscribe?rp=wrU9d1VlqIiuL6N0zVMze4Ep%2FR7u99vtLlE%2BlH3ENQCZNSISh1iSpv072pg1%2B%2Bb9VrC6nKrdFGGFRqYJ2yJmFA%3D%3D.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100614/17c9b36f/attachment.html>

From aotto at mosso.com  Mon Jun 14 22:34:00 2010
From: aotto at mosso.com (Adrian Otto)
Date: Mon, 14 Jun 2010 15:34:00 -0700
Subject: Anyone interested in memcached protocol?
In-Reply-To: <AANLkTin88QPuNhY1MPbHVmMBFeJV4u-cq8hQdeNvCTIQ@mail.gmail.com>
References: <AANLkTinjTfuJuiXl0rDZcQiKDcUCj74w6YMTMvxx0KF8@mail.gmail.com>
	<17325.1276544148@critter.freebsd.dk>
	<AANLkTin88QPuNhY1MPbHVmMBFeJV4u-cq8hQdeNvCTIQ@mail.gmail.com>
Message-ID: <BA387579-0AA7-4AC7-9E80-DD136D800A19@mosso.com>

Varnish Devs,

In considering the recent question from Joe Chen (Re: Preloading  
Cached Contents into Varnish?) it renewed an interest of mine to put a  
memcached client API interface on Varnish. There is code in the  
libmemcahced project that should streamline the implementation from  
the protocol perspective. This way any memcached client could operate  
on the contents of the Varnish cache, which could be used as a push  
update mechanism. With SASL auth implemented, this could be done  
securely. I recognize that there are ways to accomplish this with  
memcached using VM and simply allowing it to swap, or using the  
storage engine interface with a persistence engine on the back. I'd  
like to have the flexibility to use a varnish cluster as both a web  
cache and an object cache. This has large potential benefits from an  
operational perspective, where we'd like to keep the number of servers  
in a given configuration to a minimum. It also opens up new  
possibilities where a memcached client can act upon the contents of  
the cache in cases where web clients are the target consumers of the  
content.

Imagine this example of a database driven web site accelerated with  
Varnish:

Web Client -> Varnish -> Apache LAMP APP -> MySQL.

Basic Setup

1.1) Page is accessed by web client, and HTML generated by the app as  
the result of a MySQL query.
1.2) Cached content used until TTL reached.
1.3) Clients accessing expired pages repeat the procedure at step 1.1.

Basic Setup + Auto-Purge

2.1) Page is accessed by web client, and HTML generated by the app as  
the result of a MySQL query.
2.2) Clients use cached content until TTL reached, or until content is  
purged.
2.3) Row changes in MySQL database, triggering blackhole storage  
engine on a replica database, replication log acted upon by watcher  
daemon.
2.4) Watcher daemon expires the cached page asynchronously, and re- 
requests it to re-cache it. If not already cached, does not re-load it.

The advantage here is faster convergence of the cached content and the  
database contents. This setup has a problem because there is a  
possible race in 2.4 between expiring the content, and re-caching it.  
It's possible for the client to get forwarded all the way to the back- 
end and wait for the application to regenerate the content, rather  
than always getting a cached version. This is especially true if  
generating the content takes a long time, and/or if the object is  
constantly being accessed.

New Setup with memcached Protocol Integration

3.1) Page is accessed by web client, and HTML generated by the app as  
the result of a MySQL query.
3.2) Clients use cached content until TTL expires.
3.3) Row changes in MySQL database, triggering blackhole engine on  
replica database, replication log acted upon by watcher daemon.
3.4) Watcher daemon performs a CAS operation to swap in a fresh  
version of the cached content.

The advantage is that the race in 2.4 is eliminated in 3.4 because the  
replacement was both asynchronous and atomic. Clients always get a  
cached result for content that has not expired or been evicted from  
the cache.

You might argue that having a CAS purge feature in varnish might be  
equally helpful for this use case. I'm simply trying to illustrate one  
possible integration where the additional protocol availability would  
be useful. Actually using varnish directly as an object cache is  
another use case that seems equally compelling.

My question is if anyone else has a use case for this capability, and  
would you have an interest in having this added as a new feature? If  
so, I may be willing to contribute.

Thanks,

Adrian Otto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100614/33e214ba/attachment.html>

From sky at crucially.net  Mon Jun 14 22:37:04 2010
From: sky at crucially.net (Artur Bergman)
Date: Mon, 14 Jun 2010 15:37:04 -0700
Subject: Anyone interested in memcached protocol?
In-Reply-To: <BA387579-0AA7-4AC7-9E80-DD136D800A19@mosso.com>
References: <AANLkTinjTfuJuiXl0rDZcQiKDcUCj74w6YMTMvxx0KF8@mail.gmail.com>
	<17325.1276544148@critter.freebsd.dk>
	<AANLkTin88QPuNhY1MPbHVmMBFeJV4u-cq8hQdeNvCTIQ@mail.gmail.com>
	<BA387579-0AA7-4AC7-9E80-DD136D800A19@mosso.com>
Message-ID: <A4EE19F0-7A47-48FC-9AD7-408A3211AE60@crucially.net>


On Jun 14, 2010, at 3:34 PM, Adrian Otto wrote:

>
> The advantage is that the race in 2.4 is eliminated in 3.4 because  
> the replacement was both asynchronous and atomic. Clients always get  
> a cached result for content that has not expired or been evicted  
> from the cache.
>
>


The words atomic and asynchronous and cache and invalidation do not  
belong together.

Artur



From magnus at hagander.net  Tue Jun 15 07:39:30 2010
From: magnus at hagander.net (Magnus Hagander)
Date: Tue, 15 Jun 2010 09:39:30 +0200
Subject: Anyone interested in memcached protocol?
In-Reply-To: <BA387579-0AA7-4AC7-9E80-DD136D800A19@mosso.com>
References: <AANLkTinjTfuJuiXl0rDZcQiKDcUCj74w6YMTMvxx0KF8@mail.gmail.com>
	<17325.1276544148@critter.freebsd.dk>
	<AANLkTin88QPuNhY1MPbHVmMBFeJV4u-cq8hQdeNvCTIQ@mail.gmail.com>
	<BA387579-0AA7-4AC7-9E80-DD136D800A19@mosso.com>
Message-ID: <AANLkTim305c5HrHgSKTFQeMWtcRGIv8d5d79WilQ1654@mail.gmail.com>

On Tue, Jun 15, 2010 at 00:34, Adrian Otto <aotto at mosso.com> wrote:
> Varnish Devs,
> In considering the recent question from Joe Chen (Re: Preloading Cached
> Contents into Varnish?) it renewed an interest of mine to put a memcached
> client API interface on Varnish. There is code in the libmemcahced project
> that should streamline the implementation from the protocol perspective.
> This way any memcached client could operate on the contents of the Varnish
> cache, which could be used as a push update mechanism. With SASL auth
> implemented, this could be done securely. I recognize that there are ways to
> accomplish this with memcached using VM and simply allowing it to swap, or
> using the storage engine interface with a persistence engine on the back.
> I'd like to have?the flexibility to use a varnish cluster as both a web
> cache and an object cache. This has large potential benefits from an
> operational perspective, where we'd like to keep the number of servers in a
> given configuration to a minimum. It also opens up new possibilities where a
> memcached client can act upon the contents of the cache in cases where web
> clients are the target consumers of the content.
> Imagine this example of a database driven web site accelerated with Varnish:
> Web Client -> Varnish -> Apache LAMP APP -> MySQL.
> Basic Setup
> 1.1) Page is accessed by web client, and HTML generated by the app as the
> result of a MySQL query.
> 1.2) Cached content used until TTL reached.
> 1.3) Clients accessing expired pages repeat the procedure at step 1.1.
> Basic Setup + Auto-Purge
> 2.1) Page is accessed by web client, and HTML generated by the app as the
> result of a MySQL query.
> 2.2) Clients use cached content until TTL reached, or until content is
> purged.
> 2.3) Row changes in MySQL database, triggering blackhole storage engine on a
> replica database, replication log acted upon by watcher daemon.
> 2.4) Watcher daemon expires the cached page asynchronously, and re-requests
> it to re-cache it. If not already cached, does not re-load it.
> The advantage here is faster convergence of the cached content and the

Just as a point, isn't this a lot simpler to do with just a trigger on
the table firing off a regular purge request to varnish? No need for
replication or an extra watcher daemon or anything like that. I guess
it could be a problem if varnish is slow to respond, but really,
varnish has never been slow to respond in any of the cases I've done
it :-) And even with a deamon, using a trigger ought to be much
easier, no?

(Granted, I've not done this on MySQL specificall, only PostgreSQL,
but since it has triggers by now, I don't see why it wouldn't work)


> database contents. This setup has a problem because there is a possible race
> in 2.4 between expiring the content, and re-caching it. It's possible for
> the client to get forwarded all the way to the back-end and wait for the
> application to regenerate the content, rather than always getting a cached
> version. This is especially true if generating the content takes a long
> time, and/or if the object is constantly being accessed.

Shouldn't grace-mode be able to handle this? Granted it will get the
old content until the refresh has finished, but other than that? (And
we're talking asynchronous changes anyway, so that shouldn't make a
huge difference)


> New Setup with memcached Protocol Integration
> 3.1) Page is accessed by web client, and HTML generated by the app as the
> result of a MySQL query.
> 3.2) Clients use cached content until TTL expires.
> 3.3) Row changes in MySQL database, triggering blackhole engine on replica
> database, replication log acted upon by watcher daemon.
> 3.4) Watcher daemon performs a CAS operation to swap in a fresh version of
> the cached content.
> The advantage is that the race in 2.4 is eliminated in 3.4 because the

If it's ok to serve stale content while it's reloaded, grace should do
it, I think, no need for the custom stuff.

If it's not ok, then I think you need to do a distributed transaction
with the database, which is way beyond anything varnish can (or
should!) do.

Or am I missing something? :-)


> Actually using varnish directly as an object cache is another use case that
> seems equally compelling.

But we already have memcached for that, no? Why do we need another
product to do it?

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/



From paul.mansfield at taptu.com  Tue Jun 15 10:15:28 2010
From: paul.mansfield at taptu.com (Paul Mansfield)
Date: Tue, 15 Jun 2010 11:15:28 +0100
Subject: Preloading Cached Contents into Varnish?
In-Reply-To: <17325.1276544148@critter.freebsd.dk>
References: <17325.1276544148@critter.freebsd.dk>
Message-ID: <4C1752C0.60907@taptu.com>

On 14/06/10 20:35, Poul-Henning Kamp wrote:
> In message <AANLkTinjTfuJuiXl0rDZcQiKDcUCj74w6YMTMvxx0KF8 at mail.gmail.com>, Joe 
> Chen writes:
> 
>> I haven't received any reply besides Adrian's one. Is this emailing list
>> active?  I still need someone to point me to a good direction of adding
>> preloaded cache in the storage.
> 
> Sorry for the delay.
> 
>>>> Why not just implement a simple HTTP client script to walk the list of
>>>> URL's you want to load on some suitable interval? This way you don't need
>>>> any code modifications.
> 
> This actually is the canonical answer: adding a lot of code to
> Varnish to do what a few lines of script can do, would not make much
> sense.

I have followed this thread with some interest because I read it as a
request to do what Akamai do, namely that when web client fetches the
HTML the cache parses the HTML and fetches the page assets so that when
the client fetches them they are already loaded into the cache; this is
ideal for anyone running a very large web site with lots of dynamic
pages and a huge library of static content which can't all be loaded
into varnish, and where, say, 10% of the items are likely to be very
popular but you don't know which.

However, I see that the OP seems to want simply to pre-seed the cache
with popular content which is unlikely to change.




From phk at phk.freebsd.dk  Mon Jun 21 10:29:53 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 21 Jun 2010 10:29:53 +0000
Subject: 2.1.3 candidate patch available
Message-ID: <40969.1277116193@critter.freebsd.dk>


	http://phk.freebsd.dk/misc/_2.1.3.patch

This is revision 0 of my candidate patch for the 2.1.3 release.

Please test this if you can, and report if any of the bugs mentioned
below are not fixed:

Contents:
        Add #include vmb.h / vmb.c (..., r4977)

        Critbit.c:
                use malloc for y-structs
                To reduce objhdr/obj delta and reduce CPU-wastage.

        #700 (r4829):
                Allow TAB in HTTP reason field.

        #702 (r4865):
                Off by one in Range length calculation

        #704 (r4866):
                Range "from-end" miscalculation

        #722:
                Fix issue with director cleanup on vcl.discard

        B-Heap
                Slight performance improvement

        Expect{Err} bug in vcc_action.c
                Panic instead of error message

        TCP_Check() of getsockname in VRT_r_server_ip() &c.
                Solaris bug-apeacement

        #719 (r4973, r4974, r4975)
                ESI panic when element spans malloc segments

        Remove link from error in default.vcl (r4867)


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.



From phk at phk.freebsd.dk  Mon Jun 21 11:16:27 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 21 Jun 2010 11:16:27 +0000
Subject: 2.1.3 candidate patch available 
In-Reply-To: Your message of "Mon, 21 Jun 2010 10:29:53 GMT."
	<40969.1277116193@critter.freebsd.dk> 
Message-ID: <55165.1277118987@critter.freebsd.dk>

In message <40969.1277116193 at critter.freebsd.dk>, Poul-Henning Kamp writes:
>
>	http://phk.freebsd.dk/misc/_2.1.3.patch

Sorry, here is the correct URL:

	http://phk.freebsd.dk/misc/_.2.1.3.patch

>
>This is revision 0 of my candidate patch for the 2.1.3 release.
>
>Please test this if you can, and report if any of the bugs mentioned
>below are not fixed:
>
>Contents:
>        Add #include vmb.h / vmb.c (..., r4977)
>
>        Critbit.c:
>                use malloc for y-structs
>                To reduce objhdr/obj delta and reduce CPU-wastage.
>
>        #700 (r4829):
>                Allow TAB in HTTP reason field.
>
>        #702 (r4865):
>                Off by one in Range length calculation
>
>        #704 (r4866):
>                Range "from-end" miscalculation
>
>        #722:
>                Fix issue with director cleanup on vcl.discard
>
>        B-Heap
>                Slight performance improvement
>
>        Expect{Err} bug in vcc_action.c
>                Panic instead of error message
>
>        TCP_Check() of getsockname in VRT_r_server_ip() &c.
>                Solaris bug-apeacement
>
>        #719 (r4973, r4974, r4975)
>                ESI panic when element spans malloc segments
>
>        Remove link from error in default.vcl (r4867)
>
>
>-- 
>Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
>phk at FreeBSD.ORG         | TCP/IP since RFC 956
>FreeBSD committer       | BSD since 4.3-tahoe
>Never attribute to malice what can adequately be explained by incompetence.
>
>_______________________________________________
>varnish-dev mailing list
>varnish-dev at varnish-cache.org
>http://lists.varnish-cache.org/mailman/listinfo/varnish-dev
>

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From andersonbrown8 at gmail.com  Thu Jun 24 11:12:10 2010
From: andersonbrown8 at gmail.com (Anderson Brown)
Date: Thu, 24 Jun 2010 07:12:10 -0400
Subject: Maximum Object Size to be Cached
Message-ID: <AANLkTinINTh5jpfGOdQQcn_gJeTAn0RbUCXRWUZT-B-O@mail.gmail.com>

Hi Folks -

I had been doing some testing, and I have file sizes ranging from
1-8k, 9-24k, 25-50k.  I noticed everything gets cached upto 769-1024k.
 But 1025k onwards doesn't get cached.  Is this a configuration
setting I can increase?

Thanks.



From perbu at varnish-software.com  Thu Jun 24 15:37:54 2010
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 24 Jun 2010 17:37:54 +0200
Subject: Maximum Object Size to be Cached
In-Reply-To: <AANLkTinINTh5jpfGOdQQcn_gJeTAn0RbUCXRWUZT-B-O@mail.gmail.com>
References: <AANLkTinINTh5jpfGOdQQcn_gJeTAn0RbUCXRWUZT-B-O@mail.gmail.com>
Message-ID: <AANLkTinPawkWHAR4XQTvBbNn2NGiXWeI0jW3gTNUP-yI@mail.gmail.com>

Hi,

On Thu, Jun 24, 2010 at 1:12 PM, Anderson Brown
<andersonbrown8 at gmail.com> wrote:
> Hi Folks -
>
> I had been doing some testing, and I have file sizes ranging from
> 1-8k, 9-24k, 25-50k. ?I noticed everything gets cached upto 769-1024k.
> ?But 1025k onwards doesn't get cached. ?Is this a configuration
> setting I can increase?

AFAIK Varnish doesn't care about the object size. Could you show us
what varnishlog says about how the big objects are handled?


-- 
Per Buer,  Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer



From phk at phk.freebsd.dk  Mon Jun 28 16:55:18 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 28 Jun 2010 16:55:18 +0000
Subject: 2.1.3 (rev2) candidate patch available 
Message-ID: <83017.1277744118@critter.freebsd.dk>


I have updated the 2.1.3 candidate patch at

	http://phk.freebsd.dk/misc/_.2.1.3.patch

Please help test it.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From captkiddo at gmail.com  Mon Jun 28 17:47:35 2010
From: captkiddo at gmail.com (David Brown)
Date: Mon, 28 Jun 2010 13:47:35 -0400
Subject: Varnish showing hex characters from Jakarta-Tomcat connector protocol
Message-ID: <AANLkTinQeX14NVc6TkE1tUNOCabdPcSYEtq160Iqi6gq@mail.gmail.com>

We are seeing an unusual behavior attempting to use Varnish to front our web
systems. Currently we use Squid in reverse proxy config for our site.


Our current setup is Tomcat 5.5 -> JK connector 1.2.30 -> SunOne 6.1sp12 ->
Squid 2.7


When we put Varnish 2.1.2 in place of Squid we see hex characters showing up
in content from Tomcat, typically "1ff8".


I tracked down the characters to the AJP/1.3 protocol. According to the
protocol it is the size of the following body part (bytes count in hex).


Various browsers, curl, wget and squid behave correctly and do not display
these characters. Varnish seems to insist on including it in the body of the
content. This only happens when Varnish stores the object in cache (lookup),
when it is set to pass or pipe it does not show up. (we really want to cache
some of this content though)


Here's a bit of a tcpdump to show what I am getting. Line 0150 shows the
1ff8 that is getting included. The content starts right after that at
@CHARSET.


It seems to me Varnish is starting to read the body copy too early and is
including data from the AJP protocol stack into the body.


Is there a way to fix this?




0000   00 1b 63 b1 22 e4 00 12 f2 1d 63 00 08 00 45 00  ..c.".....c...E.

0010   05 8c be fd 40 00 3e 06 6f e4 c0 a8 b4 f7 c0 a8  .... at .>.o.......

0020   d2 41 00 50 d3 54 5c 16 8d 9f 6a 4f 79 7f 80 10  .A.P.T\...jOy...

0030   c0 60 d4 4c 00 00 01 01 08 0a 4e 5a 92 87 1e 8d  .`.L......NZ....

0040   92 de 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f  ..HTTP/1.1 200 O

0050   4b 0d 0a 53 65 72 76 65 72 3a 20 53 75 6e 2d 4f  K..Server: Sun-O

0060   4e 45 2d 57 65 62 2d 53 65 72 76 65 72 2f 36 2e  NE-Web-Server/6.

0070   31 0d 0a 44 61 74 65 3a 20 4d 6f 6e 2c 20 32 38  1..Date: Mon, 28

0080   20 4a 75 6e 20 32 30 31 30 20 31 35 3a 33 37 3a   Jun 2010 15:37:

0090   32 30 20 47 4d 54 0d 0a 43 61 63 68 65 2d 63 6f  20 GMT..Cache-co

00a0   6e 74 72 6f 6c 3a 20 70 75 62 6c 69 63 2c 70 72  ntrol: public,pr

00b0   69 76 61 74 65 0d 0a 45 54 61 67 3a 20 57 2f 22  ivate..ETag: W/"

00c0   32 32 30 39 38 2d 31 32 37 37 33 30 31 39 38 34  22098-1277301984

00d0   30 30 30 22 0d 0a 4c 61 73 74 2d 4d 6f 64 69 66  000"..Last-Modif

00e0   69 65 64 3a 20 57 65 64 2c 20 32 33 20 4a 75 6e  ied: Wed, 23 Jun

00f0   20 32 30 31 30 20 31 34 3a 30 36 3a 32 34 20 47   2010 14:06:24 G

0100   4d 54 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65  MT..Content-Type

0110   3a 20 74 65 78 74 2f 63 73 73 0d 0a 43 6f 6e 74  : text/css..Cont

0120   65 6e 74 2d 4c 65 6e 67 74 68 3a 20 32 32 30 39  ent-Length: 2209

0130   38 0d 0a 54 72 61 6e 73 66 65 72 2d 65 6e 63 6f  8..Transfer-enco

0140   64 69 6e 67 3a 20 63 68 75 6e 6b 65 64 0d 0a 0d  ding: chunked...

0150   0a 31 66 66 38 0d 0a 40 43 48 41 52 53 45 54 20  .1ff8.. at CHARSET

0160   22 55 54 46 2d 38 22 3b 23 74 70 63 72 7b 62 61  "UTF-8";#tpcr{ba

0170   63 6b 67 72 6f 75 6e 64 2d 63 6f 6c 6f 72 3a 57  ckground-color:W

0180   68 69 74 65 3b 6d 61 72 67 69 6e 3a 31 30 70 78  hite;margin:10px

0190   20 30 20 32 30 70 78 20 30 3b 7d 0a 23 74 70 63   0 20px 0;}.#tpc

01a0   72 20 68 33 7b 66 6f 6e 74 2d 73 69 7a 65 3a 32  r h3{font-size:2

01b0   36 70 78 3b 6d 61 72 67 69 6e 2d 62 6f 74 74 6f  6px;margin-botto

01c0   6d 3a 32 30 70 78 3b 66 6f 6e 74 2d 77 65 69 67  m:20px;font-weig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100628/c4c2cdf6/attachment.html>

From phk at phk.freebsd.dk  Mon Jun 28 19:14:26 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 28 Jun 2010 19:14:26 +0000
Subject: Varnish showing hex characters from Jakarta-Tomcat connector
	protocol 
In-Reply-To: Your message of "Mon, 28 Jun 2010 13:47:35 -0400."
	<AANLkTinQeX14NVc6TkE1tUNOCabdPcSYEtq160Iqi6gq@mail.gmail.com> 
Message-ID: <83447.1277752466@critter.freebsd.dk>

In message <AANLkTinQeX14NVc6TkE1tUNOCabdPcSYEtq160Iqi6gq at mail.gmail.com>, Davi
d Brown writes:

>When we put Varnish 2.1.2 in place of Squid we see hex characters showing up
>in content from Tomcat, typically "1ff8".

This is what is called "HTTP/1.1 chunked encoding" headers.

Somehow one end does not realize that chunked encoding is being used.

Check Transfer-Encoding headers and HTTP version fields are not
disturbed.



-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From captkiddo at gmail.com  Tue Jun 29 16:31:53 2010
From: captkiddo at gmail.com (David Brown)
Date: Tue, 29 Jun 2010 12:31:53 -0400
Subject: Varnish showing hex characters from Jakarta-Tomcat connector 
	protocol
In-Reply-To: <83447.1277752466@critter.freebsd.dk>
References: <AANLkTinQeX14NVc6TkE1tUNOCabdPcSYEtq160Iqi6gq@mail.gmail.com>
	<83447.1277752466@critter.freebsd.dk>
Message-ID: <AANLkTikRDbvceKr14-PKP8RuIC9Djyab0kLiehPG2Ea4@mail.gmail.com>

I checked and it seems the Transfer-Encoding header is being stripped by
Varnish. I can see it when I go directly to the web server but not when I
view the file through Varnish.


I worked around the problem by adding this to my VCL


In sub vcl_fetch


 # Is Transfer-Encoding in the response, if so store it.

if (beresp.http.Transfer-Encoding ~ "chunked") {

set beresp.http.X-Encoding = beresp.http.Transfer-Encoding;

}




In sub vcl_deliver


 # Putting back the Transfer-Encoding header

if (resp.http.X-Encoding == "chunked") {

set resp.http.Transfer-Encoding = "chunked";

}



It is odd that Varnish only seems to strip the header when the content come
from Tomcat through the JK connector. The same files delivered from the same
web server, but as flat files, don't have the same thing happen.


Perhaps Tomcat or the JK connector is doing some odd formating.



Thanks for your help.




On Mon, Jun 28, 2010 at 3:14 PM, Poul-Henning Kamp <phk at phk.freebsd.dk>wrote:

> In message <AANLkTinQeX14NVc6TkE1tUNOCabdPcSYEtq160Iqi6gq at mail.gmail.com>,
> Davi
> d Brown writes:
>
> >When we put Varnish 2.1.2 in place of Squid we see hex characters showing
> up
> >in content from Tomcat, typically "1ff8".
>
> This is what is called "HTTP/1.1 chunked encoding" headers.
>
> Somehow one end does not realize that chunked encoding is being used.
>
> Check Transfer-Encoding headers and HTTP version fields are not
> disturbed.
>
>
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100629/7701665d/attachment.html>

From phk at phk.freebsd.dk  Tue Jun 29 21:22:44 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 29 Jun 2010 21:22:44 +0000
Subject: Varnish showing hex characters from Jakarta-Tomcat connector
	protocol 
In-Reply-To: Your message of "Tue, 29 Jun 2010 12:31:53 -0400."
	<AANLkTikRDbvceKr14-PKP8RuIC9Djyab0kLiehPG2Ea4@mail.gmail.com> 
Message-ID: <73267.1277846564@critter.freebsd.dk>

In message <AANLkTikRDbvceKr14-PKP8RuIC9Djyab0kLiehPG2Ea4 at mail.gmail.com>, Davi
d Brown writes:

>I checked and it seems the Transfer-Encoding header is being stripped by
>Varnish. I can see it when I go directly to the web server but not when I
>view the file through Varnish.
>
>I worked around the problem by adding this to my VCL

That sounds positively wrong.

Is it possible that you can capture a varnishlog of one of these
transaction and send to me ?


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From dave at cheney.net  Tue Jun 29 22:47:34 2010
From: dave at cheney.net (Dave Cheney)
Date: Wed, 30 Jun 2010 08:47:34 +1000
Subject: Varnish showing hex characters from Jakarta-Tomcat connector 
	protocol
In-Reply-To: <83447.1277752466@critter.freebsd.dk>
References: <AANLkTinQeX14NVc6TkE1tUNOCabdPcSYEtq160Iqi6gq@mail.gmail.com>
	<83447.1277752466@critter.freebsd.dk>
Message-ID: <AANLkTilOnsyCkwt8pgMhcBimRpPyGqksqOJZ0TKGGvC-@mail.gmail.com>

The response headers has both a content length and a transferencoding
set. The spec says that the transfer encoding takes precident over
content length, but I would view the presence of both as a bug. I
would start looking at the code path from sun one server to
squid/varnish as tomcat talking ajp does not introduce either header
by default.

Cheers

Dace

On Tuesday, June 29, 2010, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> In message <AANLkTinQeX14NVc6TkE1tUNOCabdPcSYEtq160Iqi6gq at mail.gmail.com>, Davi
> d Brown writes:
>
>>When we put Varnish 2.1.2 in place of Squid we see hex characters showing up
>>in content from Tomcat, typically "1ff8".
>
> This is what is called "HTTP/1.1 chunked encoding" headers.
>
> Somehow one end does not realize that chunked encoding is being used.
>
> Check Transfer-Encoding headers and HTTP version fields are not
> disturbed.
>
>
>
> --
> Poul-Henning Kamp ? ? ? | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG ? ? ? ? | TCP/IP since RFC 956
> FreeBSD committer ? ? ? | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-dev
>



From captkiddo at gmail.com  Wed Jun 30 13:08:16 2010
From: captkiddo at gmail.com (David Brown)
Date: Wed, 30 Jun 2010 09:08:16 -0400
Subject: Varnish showing hex characters from Jakarta-Tomcat connector 
	protocol
In-Reply-To: <73267.1277846564@critter.freebsd.dk>
References: <AANLkTikRDbvceKr14-PKP8RuIC9Djyab0kLiehPG2Ea4@mail.gmail.com>
	<73267.1277846564@critter.freebsd.dk>
Message-ID: <AANLkTil5v0EXmsBo7yvODMDfr-S37xhcgw3aKRIZL9_I@mail.gmail.com>

You were right, it was wrong. The change I made to the VCL fixed the one
problem but screwed up the rest of the site so I had to back it out.

Attached is a varnish log of the activity. I have some tcpdumps at various
points too if you need them.

I still think it's connected to tomcat and specifically the JK connector. It
seems the JK connector has an 8K limit (AJP 1.3) and breaks up large files
in the transmission between tomcat and the webserver. The characters may be
(?) part of JK's breakup of the file. My reasoning is because I can take the
same file and place it directly on the web server and no problem, only when
it comes from tomcat/jk do we have these issues. ( I can't convince the
programmers to take static content out of their java apps and put it on the
server )



On Tue, Jun 29, 2010 at 5:22 PM, Poul-Henning Kamp <phk at phk.freebsd.dk>wrote:

> In message <AANLkTikRDbvceKr14-PKP8RuIC9Djyab0kLiehPG2Ea4 at mail.gmail.com>,
> Davi
> d Brown writes:
>
> >I checked and it seems the Transfer-Encoding header is being stripped by
> >Varnish. I can see it when I go directly to the web server but not when I
> >view the file through Varnish.
> >
> >I worked around the problem by adding this to my VCL
>
> That sounds positively wrong.
>
> Is it possible that you can capture a varnishlog of one of these
> transaction and send to me ?
>
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100630/8f6ee35a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: varnish.log
Type: application/octet-stream
Size: 90564 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100630/8f6ee35a/attachment.obj>

From mail at viliar.net.ru  Tue Jun 15 14:08:40 2010
From: mail at viliar.net.ru (mail at viliar.net.ru)
Date: Tue, 15 Jun 2010 14:08:40 -0000
Subject: trunk: fail to compile.
Message-ID: <fbf4bef38edfb911ca94689bc8d6a217@mxt.alx.in>

Source from trunk (r4956). Fail on tests.

==============================================
1 of 174 tests failed
Please report to varnish-dev at varnish-cache.org
==============================================

Assert error in cmd_http_send(), vtc_http.c line 727:
  Condition(i == strlen(av[1])) not true.
  errno = 104 (Connection reset by peer)
/bin/sh: line 4:  3695 ????????? ???????         ./varnishtest ${dir}$tst
FAIL: ./tests/r00387.vtc
#    top  TEST ././tests/r00400.vtc passed (0.492s)
PASS: ./tests/r00400.vtc




sysinfo.

# cat /etc/redhat-release 
CentOS release 5.5 (Final)

# uname -a
Linux gaianewi386 2.6.18-194.3.1.el5.028stab069.6xen #1 SMP Wed May 26
18:35:38 MSD 2010 i686 i686 i386 GNU/Linux

# gcc -v
Using built-in specs.
Target: i386-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-libgcj-multifile
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada
--enable-java-awt=gtk --disable-dssi --enable-plugin
--with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
--host=i386-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)

Don't think it depends on gcc version, but I will try with gcc44 (tech
preview in CentOS5).



From mail at viliar.net.ru  Tue Jun 15 14:10:20 2010
From: mail at viliar.net.ru (mail at viliar.net.ru)
Date: Tue, 15 Jun 2010 14:10:20 -0000
Subject: sorry, wrong subject. fail to pass tests.
Message-ID: <0283896509e272bc62fc1337f5e293ee@mxt.alx.in>

Source from trunk (r4956). Fail on tests.

==============================================
1 of 174 tests failed
Please report to varnish-dev at varnish-cache.org
==============================================

Assert error in cmd_http_send(), vtc_http.c line 727:
  Condition(i == strlen(av[1])) not true.
  errno = 104 (Connection reset by peer)
/bin/sh: line 4:  3695 ????????? ???????         ./varnishtest ${dir}$tst
FAIL: ./tests/r00387.vtc
#    top  TEST ././tests/r00400.vtc passed (0.492s)
PASS: ./tests/r00400.vtc




sysinfo.

# cat /etc/redhat-release 
CentOS release 5.5 (Final)

# uname -a
Linux gaianewi386 2.6.18-194.3.1.el5.028stab069.6xen #1 SMP Wed May 26
18:35:38 MSD 2010 i686 i686 i386 GNU/Linux

# gcc -v
Using built-in specs.
Target: i386-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-libgcj-multifile
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada
--enable-java-awt=gtk --disable-dssi --enable-plugin
--with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre
--with-cpu=generic
--host=i386-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)

Don't think it depends on gcc version, but I will try with gcc44 (tech
preview in CentOS5).




From frankamp at gmail.com  Thu Jun 17 22:22:52 2010
From: frankamp at gmail.com (Joshua Frankamp)
Date: Thu, 17 Jun 2010 22:22:52 -0000
Subject: grace for all
Message-ID: <AANLkTilqC31PTTNbpK5jAPnaoL0AA3vTY7i95DnPSpYv@mail.gmail.com>

Hello,

I would like to propose a?objective to your roadmap. Both of these
build on the idea of grace.

1. Grace for All w/ Backend Fill. Three requests hit varnish
simultaneously. The leader gets punished, as grace only applies to the
latter two. They are the lucky ones, but the first must wait. I would
propose giving all three grace, _and_ when the item has been
retrieved, it can go to /dev/null after populating the cache through
vlc_fetch. It stands to reason that if the graced content is good
enough for #2 and #3, it must be also valid for #1. There are
considerations here, and it complicates because you have two code
paths emerging from one, essentially spawning another to do the
backend. I would imagine this as another grace option off by default.

2. vlc_recv should be able to call miss in addition to pass and
lookup. This will allow a use to force a cached item to refresh under
certain circumstances. For example if I have a special process hit
varnish and provide req.http.X-Cache-Builder I could choose to jump
from vlc_recv to miss, thereby forcing a backend lookup, that would
replace cache without _any_ real user getting the misfortune of
missing cache. This would NOT require async processing or thread
splitting as option 1, but would allow those of us who want to keep
cache fresh for our users to build it. In our case, we already have a
system that can crawl our own site with special headers, and it can
afford to go slow.

Either of these extend grace to all in a manner of speaking, no
request deserves to be punished. :-)

Note: I have searched the archive before posting, and I saw several
references to people misunderstanding grace or wanting prefetch to be
implemented and hoping for features like these. The developer
responses were mostly "it doesn't work like that" which I understand.
I am suggesting the idea is still important, but perhaps there are
other ways to accomplish something. #2 seems straitforward and
relatively side effect free.

Thanks,
- Joshua Frankamp