From andersonbrown8 at gmail.com  Wed Sep  1 02:03:40 2010
From: andersonbrown8 at gmail.com (Anderson Brown)
Date: Tue, 31 Aug 2010 22:03:40 -0400
Subject: HTTP 400 returned for a HTTP request over 28 headers?
In-Reply-To: <AANLkTinOq=0vxHr2jsdrv360UxMbOe1H8XYRVC_RJZPF@mail.gmail.com>
References: <AANLkTi=heOR3ro4n7jXxn3yUwmWFSfF4aS6fRHmMnSQe@mail.gmail.com>
	<AANLkTimFgXapb3-mOLUv1V_Y0NSfM8gCfqr1QhQc4nDU@mail.gmail.com>
	<AANLkTi=7g4CrW0yta5oMN4A1rT5Axqc6JA0TU6FsM=cP@mail.gmail.com>
	<AANLkTimcnj1ZLNcktYr3b6xagrhL0dLRP4+qVTZENgow@mail.gmail.com>
	<AANLkTinOq=0vxHr2jsdrv360UxMbOe1H8XYRVC_RJZPF@mail.gmail.com>
Message-ID: <AANLkTinshDadrAXxeuGwOaR+aWZxp4j1ObyR7if9mOpa@mail.gmail.com>

Thanks All.  I had to:

+ Uninstall the 2.0.6 varnish installed via yum.
+ Download the 2.1.x rpm from sourceforge and install that.
+ -p http_headers=128 worked like a charm.

Kristian - yes, I didn't have those NOT WORK on there; just added it on the
email.

Thank you for the quick turnaround.  Appreciate it.
Anderson.


On Tue, Aug 31, 2010 at 2:56 PM, T. Pascal <t_pascal at zennet.com> wrote:

> On Tue, Aug 31, 2010 at 7:10 AM, Per Buer <perbu at varnish-software.com>
> wrote:
> > Hi,
> >
> > On Tue, Aug 31, 2010 at 3:52 PM, Anderson Brown <
> andersonbrown8 at gmail.com>
> > wrote:
> >> However, when I try to restart:
> >>
> >> root at machine>/etc/init.d/varnish restart
> >> Stopping varnish HTTP accelerator:                         [  OK  ]
> >> Starting varnish HTTP accelerator:                         [FAILED]
> >>
> >> I see nothing in /var/log/messages nor in varnishlog.
> >
> > It should be logged somewhere, if not - that would be a bug. I'm not sure
> > where it would be appropriate to log such things on RHEL.
> > wrt the parameter - I _think_ this was made a run-time option in 2.1. It
> was
> > a compile time option in 2.0.
> >
> In my centos servers with the sourceforge RPMs, it is logged in
> /var/log/messages.
>
> >> root at machine>varnishd -V
> >> varnishd (varnish-2.0.6)
> >> Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS
> >>
> >> Am I using an older version?  Just did the varnish install via yum-EPEL.
> >
> > As 2.1 is not 100% compatible with 2.0 EPEL for EL 5.0 is stuck on 2.0.
> > However, there are RPM's available.
> >
> The RPMs are available on sourceforge (a link exists from the
> downloads section on the main varnish-cache.org home page); I have
> used even 2.13 without problems.
>
> I am not sure if there will be compatibility issues if you want to
> jump back to epel for Varnish at some point (when 2.1 is available
> there).
>
> There is also a bug with the DAEMON options startup script from these
> RPMs.  You will want to change the single quotes in
> /etc/init.d/varnish with double quotes for the DAEMON_OPT line (as
> epel does).
>
> -T
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100831/8244cf9d/attachment.html>

From andersonbrown8 at gmail.com  Wed Sep  1 02:03:40 2010
From: andersonbrown8 at gmail.com (Anderson Brown)
Date: Tue, 31 Aug 2010 22:03:40 -0400
Subject: HTTP 400 returned for a HTTP request over 28 headers?
In-Reply-To: <AANLkTinOq=0vxHr2jsdrv360UxMbOe1H8XYRVC_RJZPF@mail.gmail.com>
References: <AANLkTi=heOR3ro4n7jXxn3yUwmWFSfF4aS6fRHmMnSQe@mail.gmail.com>
	<AANLkTimFgXapb3-mOLUv1V_Y0NSfM8gCfqr1QhQc4nDU@mail.gmail.com>
	<AANLkTi=7g4CrW0yta5oMN4A1rT5Axqc6JA0TU6FsM=cP@mail.gmail.com>
	<AANLkTimcnj1ZLNcktYr3b6xagrhL0dLRP4+qVTZENgow@mail.gmail.com>
	<AANLkTinOq=0vxHr2jsdrv360UxMbOe1H8XYRVC_RJZPF@mail.gmail.com>
Message-ID: <AANLkTinshDadrAXxeuGwOaR+aWZxp4j1ObyR7if9mOpa@mail.gmail.com>

Thanks All.  I had to:

+ Uninstall the 2.0.6 varnish installed via yum.
+ Download the 2.1.x rpm from sourceforge and install that.
+ -p http_headers=128 worked like a charm.

Kristian - yes, I didn't have those NOT WORK on there; just added it on the
email.

Thank you for the quick turnaround.  Appreciate it.
Anderson.


On Tue, Aug 31, 2010 at 2:56 PM, T. Pascal <t_pascal at zennet.com> wrote:

> On Tue, Aug 31, 2010 at 7:10 AM, Per Buer <perbu at varnish-software.com>
> wrote:
> > Hi,
> >
> > On Tue, Aug 31, 2010 at 3:52 PM, Anderson Brown <
> andersonbrown8 at gmail.com>
> > wrote:
> >> However, when I try to restart:
> >>
> >> root at machine>/etc/init.d/varnish restart
> >> Stopping varnish HTTP accelerator:                         [  OK  ]
> >> Starting varnish HTTP accelerator:                         [FAILED]
> >>
> >> I see nothing in /var/log/messages nor in varnishlog.
> >
> > It should be logged somewhere, if not - that would be a bug. I'm not sure
> > where it would be appropriate to log such things on RHEL.
> > wrt the parameter - I _think_ this was made a run-time option in 2.1. It
> was
> > a compile time option in 2.0.
> >
> In my centos servers with the sourceforge RPMs, it is logged in
> /var/log/messages.
>
> >> root at machine>varnishd -V
> >> varnishd (varnish-2.0.6)
> >> Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS
> >>
> >> Am I using an older version?  Just did the varnish install via yum-EPEL.
> >
> > As 2.1 is not 100% compatible with 2.0 EPEL for EL 5.0 is stuck on 2.0.
> > However, there are RPM's available.
> >
> The RPMs are available on sourceforge (a link exists from the
> downloads section on the main varnish-cache.org home page); I have
> used even 2.13 without problems.
>
> I am not sure if there will be compatibility issues if you want to
> jump back to epel for Varnish at some point (when 2.1 is available
> there).
>
> There is also a bug with the DAEMON options startup script from these
> RPMs.  You will want to change the single quotes in
> /etc/init.d/varnish with double quotes for the DAEMON_OPT line (as
> epel does).
>
> -T
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100831/8244cf9d/attachment-0001.html>

From andersonbrown8 at gmail.com  Wed Sep  1 02:03:40 2010
From: andersonbrown8 at gmail.com (Anderson Brown)
Date: Tue, 31 Aug 2010 22:03:40 -0400
Subject: HTTP 400 returned for a HTTP request over 28 headers?
In-Reply-To: <AANLkTinOq=0vxHr2jsdrv360UxMbOe1H8XYRVC_RJZPF@mail.gmail.com>
References: <AANLkTi=heOR3ro4n7jXxn3yUwmWFSfF4aS6fRHmMnSQe@mail.gmail.com>
	<AANLkTimFgXapb3-mOLUv1V_Y0NSfM8gCfqr1QhQc4nDU@mail.gmail.com>
	<AANLkTi=7g4CrW0yta5oMN4A1rT5Axqc6JA0TU6FsM=cP@mail.gmail.com>
	<AANLkTimcnj1ZLNcktYr3b6xagrhL0dLRP4+qVTZENgow@mail.gmail.com>
	<AANLkTinOq=0vxHr2jsdrv360UxMbOe1H8XYRVC_RJZPF@mail.gmail.com>
Message-ID: <AANLkTinshDadrAXxeuGwOaR+aWZxp4j1ObyR7if9mOpa@mail.gmail.com>

Thanks All.  I had to:

+ Uninstall the 2.0.6 varnish installed via yum.
+ Download the 2.1.x rpm from sourceforge and install that.
+ -p http_headers=128 worked like a charm.

Kristian - yes, I didn't have those NOT WORK on there; just added it on the
email.

Thank you for the quick turnaround.  Appreciate it.
Anderson.


On Tue, Aug 31, 2010 at 2:56 PM, T. Pascal <t_pascal at zennet.com> wrote:

> On Tue, Aug 31, 2010 at 7:10 AM, Per Buer <perbu at varnish-software.com>
> wrote:
> > Hi,
> >
> > On Tue, Aug 31, 2010 at 3:52 PM, Anderson Brown <
> andersonbrown8 at gmail.com>
> > wrote:
> >> However, when I try to restart:
> >>
> >> root at machine>/etc/init.d/varnish restart
> >> Stopping varnish HTTP accelerator:                         [  OK  ]
> >> Starting varnish HTTP accelerator:                         [FAILED]
> >>
> >> I see nothing in /var/log/messages nor in varnishlog.
> >
> > It should be logged somewhere, if not - that would be a bug. I'm not sure
> > where it would be appropriate to log such things on RHEL.
> > wrt the parameter - I _think_ this was made a run-time option in 2.1. It
> was
> > a compile time option in 2.0.
> >
> In my centos servers with the sourceforge RPMs, it is logged in
> /var/log/messages.
>
> >> root at machine>varnishd -V
> >> varnishd (varnish-2.0.6)
> >> Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS
> >>
> >> Am I using an older version?  Just did the varnish install via yum-EPEL.
> >
> > As 2.1 is not 100% compatible with 2.0 EPEL for EL 5.0 is stuck on 2.0.
> > However, there are RPM's available.
> >
> The RPMs are available on sourceforge (a link exists from the
> downloads section on the main varnish-cache.org home page); I have
> used even 2.13 without problems.
>
> I am not sure if there will be compatibility issues if you want to
> jump back to epel for Varnish at some point (when 2.1 is available
> there).
>
> There is also a bug with the DAEMON options startup script from these
> RPMs.  You will want to change the single quotes in
> /etc/init.d/varnish with double quotes for the DAEMON_OPT line (as
> epel does).
>
> -T
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100831/8244cf9d/attachment-0002.html>

From andersonbrown8 at gmail.com  Wed Sep  1 02:03:40 2010
From: andersonbrown8 at gmail.com (Anderson Brown)
Date: Tue, 31 Aug 2010 22:03:40 -0400
Subject: HTTP 400 returned for a HTTP request over 28 headers?
In-Reply-To: <AANLkTinOq=0vxHr2jsdrv360UxMbOe1H8XYRVC_RJZPF@mail.gmail.com>
References: <AANLkTi=heOR3ro4n7jXxn3yUwmWFSfF4aS6fRHmMnSQe@mail.gmail.com>
	<AANLkTimFgXapb3-mOLUv1V_Y0NSfM8gCfqr1QhQc4nDU@mail.gmail.com>
	<AANLkTi=7g4CrW0yta5oMN4A1rT5Axqc6JA0TU6FsM=cP@mail.gmail.com>
	<AANLkTimcnj1ZLNcktYr3b6xagrhL0dLRP4+qVTZENgow@mail.gmail.com>
	<AANLkTinOq=0vxHr2jsdrv360UxMbOe1H8XYRVC_RJZPF@mail.gmail.com>
Message-ID: <AANLkTinshDadrAXxeuGwOaR+aWZxp4j1ObyR7if9mOpa@mail.gmail.com>

Thanks All.  I had to:

+ Uninstall the 2.0.6 varnish installed via yum.
+ Download the 2.1.x rpm from sourceforge and install that.
+ -p http_headers=128 worked like a charm.

Kristian - yes, I didn't have those NOT WORK on there; just added it on the
email.

Thank you for the quick turnaround.  Appreciate it.
Anderson.


On Tue, Aug 31, 2010 at 2:56 PM, T. Pascal <t_pascal at zennet.com> wrote:

> On Tue, Aug 31, 2010 at 7:10 AM, Per Buer <perbu at varnish-software.com>
> wrote:
> > Hi,
> >
> > On Tue, Aug 31, 2010 at 3:52 PM, Anderson Brown <
> andersonbrown8 at gmail.com>
> > wrote:
> >> However, when I try to restart:
> >>
> >> root at machine>/etc/init.d/varnish restart
> >> Stopping varnish HTTP accelerator:                         [  OK  ]
> >> Starting varnish HTTP accelerator:                         [FAILED]
> >>
> >> I see nothing in /var/log/messages nor in varnishlog.
> >
> > It should be logged somewhere, if not - that would be a bug. I'm not sure
> > where it would be appropriate to log such things on RHEL.
> > wrt the parameter - I _think_ this was made a run-time option in 2.1. It
> was
> > a compile time option in 2.0.
> >
> In my centos servers with the sourceforge RPMs, it is logged in
> /var/log/messages.
>
> >> root at machine>varnishd -V
> >> varnishd (varnish-2.0.6)
> >> Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS
> >>
> >> Am I using an older version?  Just did the varnish install via yum-EPEL.
> >
> > As 2.1 is not 100% compatible with 2.0 EPEL for EL 5.0 is stuck on 2.0.
> > However, there are RPM's available.
> >
> The RPMs are available on sourceforge (a link exists from the
> downloads section on the main varnish-cache.org home page); I have
> used even 2.13 without problems.
>
> I am not sure if there will be compatibility issues if you want to
> jump back to epel for Varnish at some point (when 2.1 is available
> there).
>
> There is also a bug with the DAEMON options startup script from these
> RPMs.  You will want to change the single quotes in
> /etc/init.d/varnish with double quotes for the DAEMON_OPT line (as
> epel does).
>
> -T
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100831/8244cf9d/attachment-0003.html>

From andersonbrown8 at gmail.com  Wed Sep  1 02:03:40 2010
From: andersonbrown8 at gmail.com (Anderson Brown)
Date: Tue, 31 Aug 2010 22:03:40 -0400
Subject: HTTP 400 returned for a HTTP request over 28 headers?
In-Reply-To: <AANLkTinOq=0vxHr2jsdrv360UxMbOe1H8XYRVC_RJZPF@mail.gmail.com>
References: <AANLkTi=heOR3ro4n7jXxn3yUwmWFSfF4aS6fRHmMnSQe@mail.gmail.com>
	<AANLkTimFgXapb3-mOLUv1V_Y0NSfM8gCfqr1QhQc4nDU@mail.gmail.com>
	<AANLkTi=7g4CrW0yta5oMN4A1rT5Axqc6JA0TU6FsM=cP@mail.gmail.com>
	<AANLkTimcnj1ZLNcktYr3b6xagrhL0dLRP4+qVTZENgow@mail.gmail.com>
	<AANLkTinOq=0vxHr2jsdrv360UxMbOe1H8XYRVC_RJZPF@mail.gmail.com>
Message-ID: <AANLkTinshDadrAXxeuGwOaR+aWZxp4j1ObyR7if9mOpa@mail.gmail.com>

Thanks All.  I had to:

+ Uninstall the 2.0.6 varnish installed via yum.
+ Download the 2.1.x rpm from sourceforge and install that.
+ -p http_headers=128 worked like a charm.

Kristian - yes, I didn't have those NOT WORK on there; just added it on the
email.

Thank you for the quick turnaround.  Appreciate it.
Anderson.


On Tue, Aug 31, 2010 at 2:56 PM, T. Pascal <t_pascal at zennet.com> wrote:

> On Tue, Aug 31, 2010 at 7:10 AM, Per Buer <perbu at varnish-software.com>
> wrote:
> > Hi,
> >
> > On Tue, Aug 31, 2010 at 3:52 PM, Anderson Brown <
> andersonbrown8 at gmail.com>
> > wrote:
> >> However, when I try to restart:
> >>
> >> root at machine>/etc/init.d/varnish restart
> >> Stopping varnish HTTP accelerator:                         [  OK  ]
> >> Starting varnish HTTP accelerator:                         [FAILED]
> >>
> >> I see nothing in /var/log/messages nor in varnishlog.
> >
> > It should be logged somewhere, if not - that would be a bug. I'm not sure
> > where it would be appropriate to log such things on RHEL.
> > wrt the parameter - I _think_ this was made a run-time option in 2.1. It
> was
> > a compile time option in 2.0.
> >
> In my centos servers with the sourceforge RPMs, it is logged in
> /var/log/messages.
>
> >> root at machine>varnishd -V
> >> varnishd (varnish-2.0.6)
> >> Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS
> >>
> >> Am I using an older version?  Just did the varnish install via yum-EPEL.
> >
> > As 2.1 is not 100% compatible with 2.0 EPEL for EL 5.0 is stuck on 2.0.
> > However, there are RPM's available.
> >
> The RPMs are available on sourceforge (a link exists from the
> downloads section on the main varnish-cache.org home page); I have
> used even 2.13 without problems.
>
> I am not sure if there will be compatibility issues if you want to
> jump back to epel for Varnish at some point (when 2.1 is available
> there).
>
> There is also a bug with the DAEMON options startup script from these
> RPMs.  You will want to change the single quotes in
> /etc/init.d/varnish with double quotes for the DAEMON_OPT line (as
> epel does).
>
> -T
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100831/8244cf9d/attachment-0004.html>

From slink at schokola.de  Thu Sep  2 14:57:30 2010
From: slink at schokola.de (Nils Goroll)
Date: Thu, 02 Sep 2010 16:57:30 +0200
Subject: Backend get-if-modified-since
Message-ID: <4C7FBB5A.4050605@schokola.de>

Hi,

I need a new feature to always send backend get-if-modified-since requests if
the respective object already exists in the cache and update TTL, Expires etc.

Has anyone already looked into this?

If not, I will implement this feature.

Nils



From kristian at varnish-software.com  Mon Sep  6 14:45:01 2010
From: kristian at varnish-software.com (Kristian Lyngstol)
Date: Mon, 6 Sep 2010 16:45:01 +0200
Subject: DNS-dir tests and .ident
Message-ID: <20100906144458.GC3475@sunrider>

Hi,

I added v00029.vtc which does a rudimentary test of the DNS director. It
tests VCL and that it works for localhost and not for non-localhost. I'll
add some -badvcl tests after I'm finished with this mail.

In the process I noticed that .ident was removed from the backend. Is it
really as simple as just removing it from the backends I generate? IE: not
call vcc_EmitBeIdent(). It seems to pass the tests, but it seemed a bit too
easy.

- Kristian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: Digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100906/c6c0619b/attachment.bin>

From phk at phk.freebsd.dk  Mon Sep  6 15:37:54 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 06 Sep 2010 15:37:54 +0000
Subject: DNS-dir tests and .ident 
In-Reply-To: Your message of "Mon, 06 Sep 2010 16:45:01 +0200."
	<20100906144458.GC3475@sunrider> 
Message-ID: <1897.1283787474@critter.freebsd.dk>

In message <20100906144458.GC3475 at sunrider>, Kristian Lyngstol writes:

>In the process I noticed that .ident was removed from the backend. Is it
>really as simple as just removing it from the backends I generate? IE: not
>call vcc_EmitBeIdent(). It seems to pass the tests, but it seemed a bit too
>easy.

Yes, the ident is no longer used to identify backends, we go by
{name,ipv4,ipv6} triplet now.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From slink at schokola.de  Tue Sep  7 11:23:21 2010
From: slink at schokola.de (Nils Goroll)
Date: Tue, 07 Sep 2010 13:23:21 +0200
Subject: Backend get-if-modified-since
In-Reply-To: <4C7FBB5A.4050605@schokola.de>
References: <4C7FBB5A.4050605@schokola.de>
Message-ID: <4C8620A9.1000308@schokola.de>

Hi,

> If not, I will implement this feature.

OK, because there were no answers, we're going to implement this feature. As
with almost all features/fixes I've implemented, patches will be submitted to trac.

I'll check first if this can be done in VCL, but I remember to have thought
about this already and last time I was of the opinion that the feature needs to
get implemented in C.

Thanks, Nils



From slink at schokola.de  Tue Sep  7 12:38:09 2010
From: slink at schokola.de (Nils Goroll)
Date: Tue, 07 Sep 2010 14:38:09 +0200
Subject: Backend get-if-modified-since
In-Reply-To: <4C8620A9.1000308@schokola.de>
References: <4C7FBB5A.4050605@schokola.de> <4C8620A9.1000308@schokola.de>
Message-ID: <4C863231.6070604@schokola.de>

sorry, I've overlooked a reply in private. Someone has already started work on
this. We'll collaborate if possible.



From slink at schokola.de  Thu Sep  9 12:19:10 2010
From: slink at schokola.de (Nils Goroll)
Date: Thu, 09 Sep 2010 14:19:10 +0200
Subject: svn access changed?
Message-ID: <4C88D0BE.1090802@schokola.de>

Hi,

I am getting a "path not found" error trying to access the SVN repo. Have I
missed a change?

Thanks, Nils

haggis:~/Devel/varnish-svn/trunk$ svn info| head -3
Path: .
URL: http://www.varnish-cache.org/svn/trunk
Repository Root: http://www.varnish-cache.org/svn

haggis:~/Devel/varnish-svn/trunk$ svn up
svn: 'http://www.varnish-cache.org/svn/trunk' path not found




From perbu at varnish-software.com  Thu Sep  9 15:34:36 2010
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 9 Sep 2010 17:34:36 +0200
Subject: svn access changed?
In-Reply-To: <4C88D0BE.1090802@schokola.de>
References: <4C88D0BE.1090802@schokola.de>
Message-ID: <AANLkTi=81egkLwPsNMbrjBjK_kWtJVC4-W95VX5w9Me7@mail.gmail.com>

On Thu, Sep 9, 2010 at 2:19 PM, Nils Goroll <slink at schokola.de> wrote:

> Hi,
>
> I am getting a "path not found" error trying to access the SVN repo. Have I
> missed a change?
>

Sorry, our bad. We're pointing www.varnish-cache.org at a new server as of
today. I added a rule to proxy /svn back to the old one. Should work now.

-- 
Per Buer,  Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100909/fee7cb4a/attachment.html>

From bisho at tuenti.com  Thu Sep  9 18:34:24 2010
From: bisho at tuenti.com (=?UTF-8?Q?Guillermo_P=C3=A9rez?=)
Date: Thu, 9 Sep 2010 20:34:24 +0200
Subject: It's possible to have dinamic backends?
Message-ID: <AANLkTimDcS=EcVxJBQv8bBBjRQHKgGT2wBjFtf1i++8_@mail.gmail.com>

We want to deploy several Varnish as a cache & routing layer to a lot
of backends with different contents. We have developed a C module for
the rewriting and calculation of which backend host to use, but I have
not found a way to dynamically configure which server to use as
backend.

The use case is the following: We have images assigned to "storage
ids". Each storage id is replicated in at least two image backend
servers. We can move the storage id from one backend server to other
to rebalance, when one backend fails, etc. Our module will calculate
the actual mapping of storage id -> backend server to query.

With a short list of backends images1, 2, 3 its easy and manageable to
configure them in the backends section, and then do set req.backend =
images1. But when your number of backends is several dozens, it's a
pain to manage.

sub vcl_recv {
	C{
		if (get_server_for_url) {
			VRT_SetHdr(sp, HDR_REQ, "\010X-Panel:",
					get_server_for_url(VRT_r_req_url(sp))
				);
		}
	}C

	if (req.http.X-Panel = "1") {
		set req.backend = server1;
	} else if (req.http.X-Panel = "2") {
		set req.backend = server2;
....

So, going back to the point, there is any way of configuring the host
to use for a request, without it being defined in the backends
section? I can generate the IP if needed.

Thanks a lot.

-- 
Guille -?????- <bisho at tuenti.com>
:wq



From ssm at debian.org  Fri Sep 10 07:53:44 2010
From: ssm at debian.org (Stig Sandbeck Mathisen)
Date: Fri, 10 Sep 2010 09:53:44 +0200
Subject: Two patches for varnish
Message-ID: <7xy6baxcdz.fsf@fsck.linpro.no>


I've received one patch from Faidon Liambotis for a race condition in
"make test" causing it to fail at times on the autobuilder network.

This commit is available at
http://git.debian.org/?p=pkg-varnish/pkg-varnish.git;a=commit;h=e16e97ae29b393e1f739696578904ddab84bcf04

Also, there's a typo in the changelog entry for 2.1.1, a commit that
fixes this is available at
http://git.debian.org/?p=pkg-varnish/pkg-varnish.git;a=commit;h=e8a6889b22005db40cb4d57d631baed386638640

-- 
Stig Sandbeck Mathisen



From slink at schokola.de  Fri Sep 10 11:41:22 2010
From: slink at schokola.de (Nils Goroll)
Date: Fri, 10 Sep 2010 13:41:22 +0200
Subject: svn access changed?
In-Reply-To: <AANLkTi=81egkLwPsNMbrjBjK_kWtJVC4-W95VX5w9Me7@mail.gmail.com>
References: <4C88D0BE.1090802@schokola.de>
	<AANLkTi=81egkLwPsNMbrjBjK_kWtJVC4-W95VX5w9Me7@mail.gmail.com>
Message-ID: <4C8A1962.7010800@schokola.de>

> Should work now.

It does. Thank you, Per.

Nils



From slink at schokola.de  Fri Sep 10 11:53:57 2010
From: slink at schokola.de (Nils Goroll)
Date: Fri, 10 Sep 2010 13:53:57 +0200
Subject: It's possible to have dinamic backends?
In-Reply-To: <AANLkTimDcS=EcVxJBQv8bBBjRQHKgGT2wBjFtf1i++8_@mail.gmail.com>
References: <AANLkTimDcS=EcVxJBQv8bBBjRQHKgGT2wBjFtf1i++8_@mail.gmail.com>
Message-ID: <4C8A1C55.5010805@schokola.de>

Hi Guillermo,

> So, going back to the point, there is any way of configuring the host
> to use for a request, without it being defined in the backends
> section? I can generate the IP if needed.

I usually use perl code to generate the VCL, that way can just generate the
backend selector

	if (req.http.X-Panel = "1") {
		set req.backend = server1;
	} else if (req.http.X-Panel = "2") {
		set req.backend = server2;
....

An alternative is the VSLP director I have written. I haven't published it yet,
as I have quite a lot of open tickets with patches I proposed, and I don't see
much value in preparing patches if the core developers don't have enough
resources to review them. The director is ready, I am using it in production on
a high traffic site.

In
http://lists.varnish-cache.org/pipermail/varnish-dev/2010-March/002420.html
I have described the basic design.

IIUC, the VSLP director would do what you need. VCL inline code would look
something like

	C{
		if (get_server_for_url) {
			/* pseudo-VCL: set dir.key = get_server_for_url(req.url) */
			VRT_l_dir_key_int(sp, get_server_for_url(VRT_r_req_url(sp)));
		}
	}C

Server selection is then as

	server = key % nservers

nserver being the number of servers, plus additional logic when the server is
down etc.


Nils



From tfheen at varnish-software.com  Fri Sep 10 11:59:01 2010
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Fri, 10 Sep 2010 13:59:01 +0200
Subject: Two patches for varnish
In-Reply-To: <7xy6baxcdz.fsf@fsck.linpro.no> (Stig Sandbeck Mathisen's message
	of "Fri, 10 Sep 2010 09:53:44 +0200")
References: <7xy6baxcdz.fsf@fsck.linpro.no>
Message-ID: <87lj797qt6.fsf@qurzaw.linpro.no>

]] Stig Sandbeck Mathisen 

| I've received one patch from Faidon Liambotis for a race condition in
| "make test" causing it to fail at times on the autobuilder network.

Indeed, please just commit these, they look sensible to me.

-- 
Tollef Fog Heen
Varnish Software
t: +47 21 54 41 73



From kristian at varnish-software.com  Fri Sep 10 13:29:47 2010
From: kristian at varnish-software.com (Kristian Lyngstol)
Date: Fri, 10 Sep 2010 15:29:47 +0200
Subject: It's possible to have dinamic backends?
In-Reply-To: <4C8A1C55.5010805@schokola.de>
References: <AANLkTimDcS=EcVxJBQv8bBBjRQHKgGT2wBjFtf1i++8_@mail.gmail.com>
	<4C8A1C55.5010805@schokola.de>
Message-ID: <20100910132946.GK3475@sunrider>

Hi Nils,

On Fri, Sep 10, 2010 at 01:53:57PM +0200, Nils Goroll wrote:
> An alternative is the VSLP director I have written. I haven't published it yet,
> as I have quite a lot of open tickets with patches I proposed, and I don't see
> much value in preparing patches if the core developers don't have enough
> resources to review them. The director is ready, I am using it in production on
> a high traffic site.

we couldn't help but notice a slight frustration in your mail ;)

I notice that you do indeed have some patches that has gone a bit stale in
trac. I suggest you join us on the bug squash, typically on monday 13:00
CET in #varnish-hacking on irc.linpro.no, if you can.

Failing that, you should post them to the -dev mail list and re-post if you
don't hear a reply. I'll skim through the patches in trac by monday, but
I'm not promising anything though.

We usually go through tickets by activity roughly each week. That means
that old and inactive tickets is likely to remain old and inactive. The
best way to get a response is:

1. Join us on IRC, specially #varnish-hacking during bug-washing
2. Keep the ticket alive with updates
3. Post the patches to the varnish-dev list - until you get a reply
4. If all else fail and you still don't get a response, mail phk or myself
   directly.

Hopefully this helps a bit.

- Kristian



From kacperw at online.no  Sat Sep 11 15:38:01 2010
From: kacperw at online.no (Kacper Wysocki)
Date: Sat, 11 Sep 2010 17:38:01 +0200
Subject: It's possible to have dinamic backends?
In-Reply-To: <4C8A1C55.5010805@schokola.de>
References: <AANLkTimDcS=EcVxJBQv8bBBjRQHKgGT2wBjFtf1i++8_@mail.gmail.com>
	<4C8A1C55.5010805@schokola.de>
Message-ID: <AANLkTimNpPHAD5-hbwDgB3K2MxD99YzB8iOP4sgGt+Ma@mail.gmail.com>

2010/9/10 Nils Goroll <slink at schokola.de>:
> Hi Guillermo,
>
>> So, going back to the point, there is any way of configuring the host
>> to use for a request, without it being defined in the backends
>> section? I can generate the IP if needed.
>
> I usually use perl code to generate the VCL, that way can just generate the
> backend selector
>
> ? ? ? ?if (req.http.X-Panel = "1") {
> ? ? ? ? ? ? ? ?set req.backend = server1;
> ? ? ? ?} else if (req.http.X-Panel = "2") {
> ? ? ? ? ? ? ? ?set req.backend = server2;
> ....
>
> An alternative is the VSLP director I have written. I haven't published it yet,
> as I have quite a lot of open tickets with patches I proposed, and I don't see
> much value in preparing patches if the core developers don't have enough
> resources to review them. The director is ready, I am using it in production on
> a high traffic site.

Nils: this VSLP director of yours looks fairly interesting, but maybe
you could post a {patch,github repo,other source} so that people can
look at it instead of reimplementing their own versions?

Guillermo: we accidentally slipped off list, that wasn't intentional
so here we are again. Your approach seems reasonable given the DNS
redirector design. This will let DNS pick the backend and varnish will
cache the response, that might make it a little more manageable.

-K


---------- Forwarded message ----------
From: Guillermo P?rez <bisho at tuenti.com>
Date: 2010/9/10
Subject: Re: It's possible to have dinamic backends?
To: Kacper Wysocki <kacperw at online.no>


> Hi Guillermo,
> the short answer is no but you can use the DNS director for this if
> your backends are on the same subnet. Varnish must have a backend
> object per backend, and the dns director list will spawn them for you.
> Otherwise I'd suggest you can have a script generate and reload the
> VCL for backend definitions.

Yeah, that may work! If I rewrite the host, then the dns director will
make the resolution of the rewrited domain, won't it?

sub vcl_recv {
? ? ? ?C{
? ? ? ? ? ? ? ?if (get_server_for_url) {
? ? ? ? ? ? ? ? ? ? ? ?VRT_SetHdr(sp, HDR_REQ, "\010X-Panel:", sprintf(
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?"%s.imgs.tuenti.int",
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?get_server_for_url(VRT_r_req_url(sp)))
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?);
? ? ? ? ? ? ? ?}
? ? ? ?}C

? ? ? ?set req.backend = dnsbackend;
? ? ? ?set req.http.host = req.http.X-Panel;



director dnsbackend dns {
? ?.list = {
? ? ? ?.port = "80";
? ? ? ?.connection_timeout = 0.4;
? ? ? ?"192.168.0.0"/24;
? ?}
? ?.ttl = 5m;
}

I'lll give it a try,

Thanks a lot!

--
Guille -?????- <bisho at tuenti.com>
:wq


>
> In
> http://lists.varnish-cache.org/pipermail/varnish-dev/2010-March/002420.html
> I have described the basic design.
>
> IIUC, the VSLP director would do what you need. VCL inline code would look
> something like
>
> ? ? ? ?C{
> ? ? ? ? ? ? ? ?if (get_server_for_url) {
> ? ? ? ? ? ? ? ? ? ? ? ?/* pseudo-VCL: set dir.key = get_server_for_url(req.url) */
> ? ? ? ? ? ? ? ? ? ? ? ?VRT_l_dir_key_int(sp, get_server_for_url(VRT_r_req_url(sp)));
> ? ? ? ? ? ? ? ?}
> ? ? ? ?}C
>
> Server selection is then as
>
> ? ? ? ?server = key % nservers
>
> nserver being the number of servers, plus additional logic when the server is
> down etc.
>
>
> Nils
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-dev
>
>



-- 
http://kacper.doesntexist.org
http://windows.dontexist.com
Employ no technique to gain supreme enlightment.
- Mar pa Chos kyi blos gros



From marc.fournier at camptocamp.com  Tue Sep 14 07:33:56 2010
From: marc.fournier at camptocamp.com (Marc Fournier)
Date: Tue, 14 Sep 2010 09:33:56 +0200
Subject: svn access changed?
References: <4C88D0BE.1090802@schokola.de>
	<AANLkTi=81egkLwPsNMbrjBjK_kWtJVC4-W95VX5w9Me7@mail.gmail.com>
Message-ID: <20100914093356.7989a8c9@lonquimay.wrk.lsn.camptocamp.com>

On Thu, 9 Sep 2010 17:34:36 +0200
Per Buer <perbu at varnish-software.com>
wrote:
> [...]
> Sorry, our bad. We're pointing www.varnish-cache.org at a new server
> as of today. I added a rule to proxy /svn back to the old one. Should
> work now.
> 

Maybe I'm dumb, but I'm unable to use the public read-only SVN repo. It
looks like the redirection isn't working as it should:

$ curl -I http://varnish-cache.org/svn/varnish/trunk
HTTP/1.1 302 Moved
Server: Varnish
Retry-After: 0
Location: http://www.varnish-cache.org/svn/varnish/trunk
Date: Tue, 14 Sep 2010 07:30:56 GMT
X-Varnish: 1969895445
Age: 0
Via: 1.1 varnish
Connection: close

Cheers,
Marc





From kristian at varnish-software.com  Tue Sep 14 08:09:20 2010
From: kristian at varnish-software.com (Kristian Lyngstol)
Date: Tue, 14 Sep 2010 10:09:20 +0200
Subject: svn access changed?
In-Reply-To: <20100914093356.7989a8c9@lonquimay.wrk.lsn.camptocamp.com>
References: <4C88D0BE.1090802@schokola.de>
	<AANLkTi=81egkLwPsNMbrjBjK_kWtJVC4-W95VX5w9Me7@mail.gmail.com>
	<20100914093356.7989a8c9@lonquimay.wrk.lsn.camptocamp.com>
Message-ID: <20100914080919.GN3475@sunrider>

On Tue, Sep 14, 2010 at 09:33:56AM +0200, Marc Fournier wrote:
> On Thu, 9 Sep 2010 17:34:36 +0200
> Per Buer <perbu at varnish-software.com>
> wrote:
> > [...]
> > Sorry, our bad. We're pointing www.varnish-cache.org at a new server
> > as of today. I added a rule to proxy /svn back to the old one. Should
> > work now.
> > 
> 
> Maybe I'm dumb, but I'm unable to use the public read-only SVN repo. It
> looks like the redirection isn't working as it should:

Err, it's moved around a bit... twice I see.

http://wwww.varnish-cache.org/svn/trunk

Ie: add the www and remove the "varnish" bit...

- Kristian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: Digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100914/31cda15a/attachment.bin>

From anders at fupp.net  Mon Sep 20 13:49:28 2010
From: anders at fupp.net (Anders Nordby)
Date: Mon, 20 Sep 2010 15:49:28 +0200
Subject: r4480 - trunk/varnish-cache/bin/varnishncsa
In-Reply-To: <20100202123132.GA89144@fupp.net>
References: <D38199BA-78A0-4C7A-A26A-A24D070B63A6@develooper.com>
	<44229.1265050247@critter.freebsd.dk>
	<20100202123132.GA89144@fupp.net>
Message-ID: <20100920134928.GA40437@fupp.net>

Hi,

On Tue, Feb 02, 2010 at 01:31:33PM +0100, Anders Nordby wrote:
>> Well, the code to implement it full was never there, and people were
>> reporting core dumps because of it, so we figured we would make the
>> non-support for -b explicit, until the code could be written.
>> 
>> There are no objections to supporting it, it's just code to be
>> written by somebody... (nudge, nudge, wink, wink...)
> >From a developer point of view, I understand that this feature must be
> improved or removed. But from a user point of view, it has its use, and
> it IS in use. Will this patch be backed out or not? Do I have to use
> patches to see backend traffic in a simple Apache style log?

Never got a reply for this. So some people get a core dump, while others
do not. How about putting a warning in the man page instead of disabling
a desired option?

I still use varnishncsa -b, which is *very* useful for me, to study
backend traffic. Can we get this option back in please?

Bye,

-- 
Anders.



From kristian at varnish-software.com  Tue Sep 21 15:24:40 2010
From: kristian at varnish-software.com (Kristian Lyngstol)
Date: Tue, 21 Sep 2010 17:24:40 +0200
Subject: Operator precedence and #777
Message-ID: <20100921152439.GA23587@sunrider>

Basic problem:

   (!foo ~ "bar")

no longer works in trunk. It used to mean the same thing as:

   (!(foo ~ "bar"))

or

   (foo !~ "bar")

However, this might not be in line with general operator precedence. It
might be more natural - all though perhaps less practical - to have it
mean:

   ((!foo) ~ "bar")

Obviously, that specific example yields code that doesn't actually make
sense. But it's more consistent with what I'd expect.

Slink also pointed out that it might make sense to have a clearly defined
operator precedence list.

My suggestion is: Make sure (!foo ~ "bar") works as it used to in 2.1, but
we should consider either disallowing it (like trunk does right now), or
changing the meaning.

Input?

- Kristian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: Digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100921/ef90e87b/attachment.bin>

From phk at phk.freebsd.dk  Tue Sep 21 15:55:42 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 21 Sep 2010 15:55:42 +0000
Subject: Operator precedence and #777 
In-Reply-To: Your message of "Tue, 21 Sep 2010 17:24:40 +0200."
	<20100921152439.GA23587@sunrider> 
Message-ID: <82819.1285084542@critter.freebsd.dk>

In message <20100921152439.GA23587 at sunrider>, Kristian Lyngstol writes:

>Basic problem:
>
>   (!foo ~ "bar")
>
>no longer works in trunk. It used to mean the same thing as:

The change is unintentional, and I will attempt to fix it.

>Slink also pointed out that it might make sense to have a clearly defined
>operator precedence list.

Yes, I am working from sort of a plan in this respect, and yes
that should go into docs once finished and stable.

The overal outline is that anything boolean has the lowest precedence
which is why (!foo ~ "bar") should be interpreted as (!(foo ~ "bar"))
as it used to be in 2.1

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From slink at schokola.de  Tue Sep 21 17:55:38 2010
From: slink at schokola.de (Nils Goroll)
Date: Tue, 21 Sep 2010 19:55:38 +0200
Subject: Operator precedence and #777
In-Reply-To: <82819.1285084542@critter.freebsd.dk>
References: <82819.1285084542@critter.freebsd.dk>
Message-ID: <4C98F19A.3050804@schokola.de>


> The overal outline is that anything boolean has the lowest precedence
> which is why (!foo ~ "bar") should be interpreted as (!(foo ~ "bar"))
> as it used to be in 2.1

I'm fine with this personally and for the time being, but I'd like to point out
that as VCL evolves as a (domain specific) programming language, I expect people
to report difficulties if operator precedence differs too much from other
languages. The boolean negation operator is a drastic example which has very
high precedence in most languages I know. To me, borrowing operator precedence
from perl would look very natural.

Refs:

http://www.uni-bonn.de/~manfear/javaoperators.php
http://en.wikipedia.org/wiki/Order_of_operations#Programming_languages
http://www.sdsc.edu/~moreland/courses/IntroPerl/docs/manual/pod/perlop.html#SYNOPSIS

Nils



From slink at schokola.de  Wed Sep 22 09:46:02 2010
From: slink at schokola.de (Nils Goroll)
Date: Wed, 22 Sep 2010 11:46:02 +0200
Subject: [patch] getcwd needs size argument on solaris
Message-ID: <4C99D05A.1080906@schokola.de>

Hi,

on (Open)Solaris, test m00000.vtc will fail because of getcwd(NULL, 0) in vtc.c
returning NULL.

from getcwd(3C) :

     If buf is a null pointer, getcwd()  obtains  size  bytes  of
     space using malloc(3C). The pointer returned by getcwd() can
     be used as the argument in a subsequent call to free().

Thanks, Nils
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: vtc_getcwd_solaris.patch
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100922/3f9990b8/attachment.ksh>

From harm.verhagen at gmail.com  Thu Sep 23 08:59:40 2010
From: harm.verhagen at gmail.com (Harm Verhagen)
Date: Thu, 23 Sep 2010 10:59:40 +0200
Subject: caching of request with cookies
Message-ID: <AANLkTinA2qP9aAiPSGh5=SwAy8SdRR-s+RHcDjPRX4Qk@mail.gmail.com>

Hi,

I noticed that varnish out of the box doesn't cache requests that carry a
cookie.

This seems a bit too conservative to me.
According to the http specs the mere existence of a cookie should not
influence cachebility.  In the content of the response depends on the
cookie, the server should tell in the response that the contents are not
cacheble.
The Vary: Cookie header  exists for that reason, no ?

In the current default configuration if you put varnish in front of a
'well-behaving' (as in "adding all the required caching related http
headers")  website.  Most cacheble content does _not_ get cached. Unless you
do some non obvious (to me :) ) configuration of varnish.

What are the thoughts on this list about moving the default varnish
configuration closer to the http specification, regarding caching of request
where the client sends a cookie (and probably leading to problems on
websites that do _not_ use http headers correctly)

That said,
Does anyone has a pointer to a varnish configuration that does the thing I
described above.  I don't mean an example where vanish just strips off
cookies of  images/js/css files  request. (I found plenty examples of that
scenario) But leaving cookies as is, but just look at the responses to
decide if the content should be cached or not. So that varnish can
selectively cache html pages based on their response, where all requests
potentially have cookies).


Mvg,
Harm Verhagen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100923/1cd77317/attachment.html>

From stewsnooze at gmail.com  Thu Sep 23 09:22:55 2010
From: stewsnooze at gmail.com (Stewart Robinson)
Date: Thu, 23 Sep 2010 10:22:55 +0100
Subject: caching of request with cookies
In-Reply-To: <AANLkTinA2qP9aAiPSGh5=SwAy8SdRR-s+RHcDjPRX4Qk@mail.gmail.com>
References: <AANLkTinA2qP9aAiPSGh5=SwAy8SdRR-s+RHcDjPRX4Qk@mail.gmail.com>
Message-ID: <AANLkTimWh1brK3=4n6N-2yM6T1PMs43VGeCU_7szP3-y@mail.gmail.com>

Whilst I don't think Varnish should change the default behaviour
because most sites use cookies to pass around session based
information which evolved after the http spec was written I think you
would probably just add the cookie to your vcl_hash function and

change this in vcl_recv
     if (req.http.Authorization || req.http.Cookie) {
         /* Not cacheable by default */
         return (pass);
    }

to
     if (req.http.Authorization) {
         /* Not cacheable by default */
         return (pass);
    }

Perhaps your vcl_hash could have this appended to it

     if (req.http.Cookie) {
         set req.hash += req.http.Cookie;
     }

I haven't tried this but I don't see why you wouldn't have individual
cache entries for each cookie combo. I think this would flood your
cache if you use the cookie for session info, but if you just keep
simple values in cookies then this should be fine.

Stewart Robinson
fullfatthings.com/


On 23 September 2010 09:59, Harm Verhagen <harm.verhagen at gmail.com> wrote:
> Hi,
>
> I noticed that varnish out of the box doesn't cache requests that carry a
> cookie.
>
> This seems a bit too conservative to me.
> According to the http specs the mere existence of a cookie should not
> influence cachebility.? In the content of the response depends on the
> cookie, the server should tell in the response that the contents are not
> cacheble.
> The Vary: Cookie header? exists for that reason, no ?
>
> In the current default configuration if you put varnish in front of a
> 'well-behaving' (as in "adding all the required caching related http
> headers")? website.? Most cacheble content does _not_ get cached. Unless you
> do some non obvious (to me :) ) configuration of varnish.
>
> What are the thoughts on this list about moving the default varnish
> configuration closer to the http specification, regarding caching of request
> where the client sends a cookie (and probably leading to problems on
> websites that do _not_ use http headers correctly)
>
> That said,
> Does anyone has a pointer to a varnish configuration that does the thing I
> described above.? I don't mean an example where vanish just strips off
> cookies of? images/js/css files? request. (I found plenty examples of that
> scenario) But leaving cookies as is, but just look at the responses to
> decide if the content should be cached or not. So that varnish can
> selectively cache html pages based on their response, where all requests
> potentially have cookies).
>
>
> Mvg,
> Harm Verhagen
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-dev
>



From slink at schokola.de  Thu Sep 23 10:28:14 2010
From: slink at schokola.de (Nils Goroll)
Date: Thu, 23 Sep 2010 12:28:14 +0200
Subject: Proposal/specs for backend conditional requests / aka "GET
	If-Modified-Since" (GET IMS))
Message-ID: <4C9B2BBE.5060309@schokola.de>

Hi All,

we propose a new feature for varnish: Backend conditional requests
(sometimes simplified as "GET If-Modified-Since" (GET IMS)). A working
implementation by Rackspace, Inc. exists.

While both Rackspace and UPLEX do need this feature and would continue
to improve the implementation anyway, we would very much favor a
design and implementation coordinated with the core developers and all
of the Varnish developer community. Our focus is to achieve an
implementation which is fully supported by the core team and will be
integrated into mainline Varnish one day.

We have therefore prepared a "design specification" and would highly
appreciate any constructive feedback, corrections and suggestions for
improvements.

While we are prepared to do all of the required implementation,
documentation and testing (including final tests on highly loaded
production systems), we're open for joint development with other
community members.


Thank you,

Adrian Otto, Rackspace Inc.
Nils Goroll, UPLEX
Geoffrey Simmons, UPLEX


Design Spec Suggestion for Varnish Backend Cache Validation using
Conditional GET
======================================================================

Intro
-----

We need to fulfill the following customer requirement: When a cached
object in Varnish expires in accordance with its TTL, and the object
has not changed on the backend server, the expired object can be
"refreshed" in the Varnish cache to allow its continued use without
re-copying the object over the network from the backend server.

Detail
------

In cases where a backend server can efficiently determine when an
object was last modified, or uses entity tags to provide versioning
capabilities, bandwidth utilization and request latency between the
Varnish server and the backend server can be significantly
improved. This is very important for cached content that is
dynamically generated, or if the object is very large.

RFC 2616 section 13.3 provides a suitable solution for this problem,
but Varnish must be extended to support conditional backend requests
in order to employ the solution.

Currently varnish makes no attempt to generate conditional requests to
the backend servers when content expires in the cache. It always sends
a normal GET request (no If-Modified-Since, If-None-Match, etc. header
is added), which results in a 200 OK response. The body of the
response must be copied over the network repeatedly when TTLs expire.

We intend to extend Varnish to use conditional GET requests when
refreshing expired content in accordance with section 13.3 of RFC
2616. This enables the backend the option to respond with a "304 Not
Modified" response to instruct Varnish to continue to use the object
it already has in cache.

The proposed solution will be based upon an existing implementation by
Adrian Otto, and Dale Chase (c) 2010 Rackspace, US Inc. This work
demonstrates that an RFC-2616 compliant implementation for conditional
back-end requests produces a dramatic improvement in system
efficiency, particularly in cases where cached objects are large, and
expensive to generate on the backend server.

This work will be extended to include relevant VCL support so that
backend requests can be manipulated with separate logic for a refresh
event.

Scope
-----

In a first iteration, we'll be implementing a GET with
If-Modified-Since conditional set to the Last-Modified time and
If-None-match for an ETag, if present, as stored for the cache object;
implementing the full generality of conditional requests would be an
obvious road for future development.

Cache Validation Behavior
-------------------------

If the backend responds to the conditional request with 200 OK, then
the object is updated as usual. If the response is 304 Not Modified,
we update the cache object based on the response headers (primarily
this means updating its TTL, as currently implemented in rfc2616.c).

Additional states, VCL support
------------------------------

We propose to implement cache validation using new states "stale" and
"refresh" into the Varnish state machine.

* cnt_stale() / vcl_stale()

  "stale" parallels "hit" and "miss", to be implemented by adding
  cnt_stale() to cache_center.c and vcl_stale() to VCL.

  An object is stale if it is found in the cache, but its TTL is
  expired. Note that obj.grace can be used to ensure objects will stay
  in the cache for longer than their TTL dictates.

  vcl_stale() is called just after the backend request is prepared
  from req, including all necessary conditionals with respect to the
  existing (stale) cache object. This way, users can modify the
  properly prepared conditional bereq at their wish.

  vcl_stale() provides access to req, bereq and obj. The possible
  return states from vcl_stale() are fetch, refresh, deliver, pass,
  error and restart:

  - fetch: default, same behavior as a miss: Fetch the obj from the
    backend with an unconditional request. All conditional request
    headers will be removed.

    Besides preserving the default behavior, this return value is
    intended to trigger a full backend fetch even if a refresh /
    conditional GET would be available, for instance when

    - backends don't implement conditionals correctly, or

    - the administrator knows that for certain objects there is no
      advantage in sending a conditional request

  - refresh: Continue with the (conditional) GET

    Note: If all conditional request headers are removed in
    vcl_stale(), return(refresh) shall be semantically equivalent to
    return(fetch).

  - deliver / pass / error / restart : obvious

  Besides controlling backend request behavior, the new VCL function
  is intended to allow easy manipulation of request headers only for
  the refresh-case.

* cnt_refresh() / vcl_refresh()

  The state "refresh" parallels "fetch", and does most of what is done
  in fetch, except that the cache object is only updated based on the
  response headers if so indicated.

  In vcl_refresh(), it is possible to access req, bereq, beresp and
  obj (which in this case refers to the cached object, as in
  vcl_hit(), vcl_miss() or vcl_stale()). The possible return states
  for vcl_refresh() are pass, restart, deliver and error (just as for
  vcl_fetch()).

  Since cnt_refresh() will likely duplicate much of what cnt_fetch() does,
  we intend to delegate the parts they have in common into
  subroutines.

  There is also a risk of code duplication for VCL users, if
  vcl_refresh() and vcl_fetch() need to perform common tasks, but this
  can also be alleviated by calling subroutines (vcl_refresh() could
  even call vcl_fetch(), if so desired, and would by default).

Reasons and examples for proposing new VCL procedures
-----------------------------------------------------

* What is vcl_stale() good for?

  Modifying request headers only for the case that a refresh is
  possible:

  One would change backend request headers as in vcl_recv, but only
  in case a refresh is possible. We imagine one would like to
  actively remove conditional statements which are irrelevant, add
  more conditional statements, or tweak the If-Modified-Since.

  Intelligent decisions about when to do a fetch or deliver stale
  content:

  The other major reason for vcl_stale is that the administrator might
  have to handle a buggy backend, which wouldn't deliver correct
  content with conditionals, so a fall back to fetch is needed.

  On the other end, administrators might decide that stale content is
  "fresh enough" and deliver anyway. We imagine a kind of "intelligent
  grace mode" where stale content is delivered without a backend
  refresh, for instance when all backends are down, certain request
  headers are present etc.

* Why vcl_refresh() in addition to vcl_fetch?

  The result of the built in refresh action (header update) might need
  to be amended or corrected, for instance the administrator might
  want to modify the Expires: or Cache-Control: headers.

Updating the in-cache object
----------------------------

The in-cache object must be updated with headers from the 304
response, in particular the cache control headers (Cache-Control,
Expires, Etag ...) must be changed.

The existing Rackspace Inc. implementation copies, for objects smaller
than a configurable threshold, the body data from the existing
object. For larger objects it modifies the in-cache object headers,
which we now believe will lead to inconsistencies when the object is
being read concurrently.

We plan to implement the update of the data in cache by not actually
touching the existing object, but

- by copying any headers not in the 304 response from the existing
  object,
- changing obj.status of the 304 object to the status of the existing
  object and
- letting the new object point to the existing in-cache body data.

To allow multiple cache objects to share body data, we want to add
reference counters to struct storage following the example of the
existing implementation for objects (HSH_Ref(), HSH_Unref() etc). We
will have a new STV_Ref(), STV_Free() will become STV_Unref() and will
be modified to only actually free storage when the reference counter
is zero (after decrementing). STV_alloc() will set the reference
counter to 1 to not break existing code.

We prefer this approach over a read/writer lock on the object because
it should yield better concurrency. Allowing to keep multiple
references to the same struct storage might also turn out to be useful
for other future improvements (ideas like "cache dedupe").

Notes
-----

RFC 2616 generally calls this process "validation", not "refresh". We prefer
the term "refresh" for use within Varnish c and VCL code.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: varnish_state_conditional.pdf
Type: application/pdf
Size: 24981 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100923/cadbe6b8/attachment.pdf>

From harm.verhagen at gmail.com  Thu Sep 23 14:26:48 2010
From: harm.verhagen at gmail.com (Harm Verhagen)
Date: Thu, 23 Sep 2010 16:26:48 +0200
Subject: caching of request with cookies
In-Reply-To: <AANLkTi=KZ3ukeWoDiZoQmynEEZzi6aPsbdstqVse21OF@mail.gmail.com>
References: <AANLkTinA2qP9aAiPSGh5=SwAy8SdRR-s+RHcDjPRX4Qk@mail.gmail.com>
	<AANLkTimWh1brK3=4n6N-2yM6T1PMs43VGeCU_7szP3-y@mail.gmail.com>
	<AANLkTi=KZ3ukeWoDiZoQmynEEZzi6aPsbdstqVse21OF@mail.gmail.com>
Message-ID: <AANLkTi=NPJqp-w_G01YPR4tCLb6UwxvctLzSMn6UFRiO@mail.gmail.com>

On Thu, Sep 23, 2010 at 11:22 AM, Stewart Robinson <stewsnooze at gmail.com>wrote:

>
> change this in vcl_recv
>     if (req.http.Authorization || req.http.Cookie) {
>         /* Not cacheable by default */
>         return (pass);
>    }
>
> to
>     if (req.http.Authorization) {
>         /* Not cacheable by default */
>         return (pass);
>    }
>
> Perhaps your vcl_hash could have this appended to it
>
>     if (req.http.Cookie) {
>         set req.hash += req.http.Cookie;
>     }
>
> I haven't tried this but I don't see why you wouldn't have individual
> cache entries for each cookie combo. I think this would flood your
> cache if you use the cookie for session info, but if you just keep
> simple values in cookies then this should be fine.
>
I'm actually not seeking for a way to cache individual replies per cookie,
that would indeed flood the cache.
I'm looking for a config that does:
    A)  don't cache responses where the response headers say they may not be
cached (read: have a Vary: Cookie header)
    B)  do cache, even when request contains a cookie, when the response
tells it can be cached because it does not have a Vary: Cookie, and caries
valid caching headers (max_age whatever)  (not cached _per_ cookie, but
cached for all requestors.)

So a default config for a website that uses Vary: Cookie response headers to
inform varnish, whether of not responses actually depend on the cookie or
not.

Isn't that how the http specs say it should be done ?  Only the webserver
_knows_ whether the actual response depended on the cookie or not, right ?


Regards,
Harm
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100923/94291bcf/attachment.html>

From aotto at mosso.com  Thu Sep 23 14:48:38 2010
From: aotto at mosso.com (Adrian Otto)
Date: Thu, 23 Sep 2010 07:48:38 -0700
Subject: caching of request with cookies
In-Reply-To: <AANLkTimWh1brK3=4n6N-2yM6T1PMs43VGeCU_7szP3-y@mail.gmail.com>
References: <AANLkTinA2qP9aAiPSGh5=SwAy8SdRR-s+RHcDjPRX4Qk@mail.gmail.com>
	<AANLkTimWh1brK3=4n6N-2yM6T1PMs43VGeCU_7szP3-y@mail.gmail.com>
Message-ID: <7C56D4D2-BAA5-455D-B38E-5F37B63040DD@mosso.com>

Stewart,

I disagree. If the cached content is the same between various users with different cookie values, the cached content should be normalized so that a single cached version can shared by subsequent visitors regardless what the cookie is set to. If you use the approach you described, you will end up with numerous cached copies of the same content in this case. That is useful in the case where content differs for every individual user, but I'd argue that it would be much better to use browser cached for that situation, not a caching proxy. Simply set sensible Cache-Control: max-age=X values. The real benefit of a caching reverse proxy is when numerous users benefit from getting a common (normalized) version of the cached content.

Typically only dynamically generated content needs the cookies in it at all, so if you are only trying to cache static content that does not need any cookies, you could use an approach like the one shown below. Sometimes you want responses with certain response codes to be cached regardless of a cookie value, like this:

vcl_fetch {
        if(obj.status == 404) {
                if (obj.http.Set-Cookie) {
                        remove obj.http.Set-Cookie;
                }
                set obj.ttl = 120s;
                return(deliver);
        }
}

I use the VCL above to cache 404 requests for 2 minutes. I do the same thing for 301, 302, etc. You might also want to do it just for certain file types. This one strips cookies from, and stores a single copy of JPG files:

vcl_fetch {
        if(req.url ~ "*.\(jpg|JPG)$") {
                if (obj.http.Set-Cookie) {
                        remove obj.http.Set-Cookie;
                }
                return(deliver);
        }
}

As long as you have not added the cookie to the hash key, you will get a cache hit no matter what the cookie value is set to. This way you don't need to strip it in vcl_recv. You get a version without a cookie returned when you get a hit on the cache lookup.

Adrian

On Sep 23, 2010, at 2:22 AM, Stewart Robinson wrote:

> Whilst I don't think Varnish should change the default behaviour
> because most sites use cookies to pass around session based
> information which evolved after the http spec was written I think you
> would probably just add the cookie to your vcl_hash function and
> 
> change this in vcl_recv
>     if (req.http.Authorization || req.http.Cookie) {
>         /* Not cacheable by default */
>         return (pass);
>    }
> 
> to
>     if (req.http.Authorization) {
>         /* Not cacheable by default */
>         return (pass);
>    }
> 
> Perhaps your vcl_hash could have this appended to it
> 
>     if (req.http.Cookie) {
>         set req.hash += req.http.Cookie;
>     }
> 
> I haven't tried this but I don't see why you wouldn't have individual
> cache entries for each cookie combo. I think this would flood your
> cache if you use the cookie for session info, but if you just keep
> simple values in cookies then this should be fine.
> 
> Stewart Robinson
> fullfatthings.com/
> 
> 
> On 23 September 2010 09:59, Harm Verhagen <harm.verhagen at gmail.com> wrote:
>> Hi,
>> 
>> I noticed that varnish out of the box doesn't cache requests that carry a
>> cookie.
>> 
>> This seems a bit too conservative to me.
>> According to the http specs the mere existence of a cookie should not
>> influence cachebility.  In the content of the response depends on the
>> cookie, the server should tell in the response that the contents are not
>> cacheble.
>> The Vary: Cookie header  exists for that reason, no ?
>> 
>> In the current default configuration if you put varnish in front of a
>> 'well-behaving' (as in "adding all the required caching related http
>> headers")  website.  Most cacheble content does _not_ get cached. Unless you
>> do some non obvious (to me :) ) configuration of varnish.
>> 
>> What are the thoughts on this list about moving the default varnish
>> configuration closer to the http specification, regarding caching of request
>> where the client sends a cookie (and probably leading to problems on
>> websites that do _not_ use http headers correctly)
>> 
>> That said,
>> Does anyone has a pointer to a varnish configuration that does the thing I
>> described above.  I don't mean an example where vanish just strips off
>> cookies of  images/js/css files  request. (I found plenty examples of that
>> scenario) But leaving cookies as is, but just look at the responses to
>> decide if the content should be cached or not. So that varnish can
>> selectively cache html pages based on their response, where all requests
>> potentially have cookies).
>> 
>> 
>> Mvg,
>> Harm Verhagen
>> 
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at varnish-cache.org
>> http://lists.varnish-cache.org/mailman/listinfo/varnish-dev
>> 
> 
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-dev




From dlachapelle at mrx.ca  Thu Sep 23 18:09:17 2010
From: dlachapelle at mrx.ca (David Lachapelle)
Date: Thu, 23 Sep 2010 14:09:17 -0400
Subject: FetchError
Message-ID: <313EE7E9-8DDC-4D0C-A53E-129C2E5BFB47@mrx.ca>

Hi All,

I've been seeing weird behavior on and off recently on our  varnish install and not sure whats going on, it would seem we're getting random FetchErrors:

> FetchError   c chunked read_error: 0


Which is resulting in a 503 being returned to the end user.  This seems to only happen once, and then on a refresh of the page, it loads correctly.  We do have multiple apache backends, so I thought it might be a particular backend causing problems, so we put a restart in our vcl_error, but that still doesn't seem to help things along.

I've confirmed we have mod_deflate disabled in apache and we're even stripping out the accept encoding in our VCL (sub vcl_recv)

>         remove req.http.Accept-Encoding;


Here's an excerpt from our varnishlog:

>   135 RxRequest    c GET
>   135 RxURL        c xxxxxxx
>   135 RxProtocol   c HTTP/1.1
>   135 RxHeader     c Accept: image/gif, image/jpeg, image/pjpeg, application/x-ms-application, application/vnd.ms-xpsdocument, application/xaml+xml, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-fla
>   135 RxHeader     c Referer: xxxxxxx
>   135 RxHeader     c Accept-Language: en-ca
>   135 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; InfoPath.2; .NET CLR 3.0.30729; .NET4.0C; OfficeLiveConnector.1.5; OfficeLivePatch.1.3)
>   135 RxHeader     c Accept-Encoding: gzip, deflate
>   135 RxHeader     c Host: xxxxxxx
>   135 RxHeader     c Connection: Keep-Alive
>   135 RxHeader     c Cookie: xxxxxxx
>   135 VCL_call     c recv lookup
>   135 VCL_call     c hash hash
>   135 VCL_call     c miss fetch
>   135 Backend      c 146 loadbalancer APP_7
>   135 TTL          c 1763959069 RFC 5 1285264818 0 0 0 0
>   135 VCL_call     c fetch
>   135 TTL          c 1763959069 VCL 31536000 1285264818
>   135 VCL_return   c deliver
>   135 ObjProtocol  c HTTP/1.1
>   135 ObjStatus    c 200
>   135 ObjResponse  c OK
>   135 ObjHeader    c Date: Thu, 23 Sep 2010 18:00:18 GMT
>   135 ObjHeader    c Server: Apache/2.2.14 (Ubuntu)
>   135 ObjHeader    c X-Powered-By: PHP/5.3.2-1ubuntu4.2
>   135 ObjHeader    c Content-Type: text/html; charset=utf-8
>   135 FetchError   c chunked read_error: 0

>   135 VCL_call     c error restart
>   135 VCL_call     c recv lookup
>   135 VCL_call     c error deliver
>   135 Length       c 52
>   135 VCL_call     c deliver deliver
>   135 TxProtocol   c HTTP/1.1
>   135 TxStatus     c 503
>   135 TxResponse   c Service Unavailable
>   135 TxHeader     c Server: Varnish


Anyone have any ideas?

Dave
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3920 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100923/c50573cd/attachment.bin>

From slink at schokola.de  Mon Sep 27 10:23:39 2010
From: slink at schokola.de (Nils Goroll)
Date: Mon, 27 Sep 2010 12:23:39 +0200
Subject: Comment on doc/sphinx/phk/backends.rst : "wildcard'isch scheme"
Message-ID: <4CA070AB.3050301@schokola.de>

Hi Poul-Henning,

you were asking for comments on your idea to have a "wildcard-ish scheme" for
accessing backend-specific information from a VCL.

IMHO, the following problem exists: Wenn a VCL is added, the semantics of
something like b1() changes (when the new VCL defines a different b1 than the
existing VL), and an old but active VCL might not know how to handle more
backends than it defined itself.

For the default case, I think it is important that, within a VCL, only those
backends are visible which were defined within that VCL.

Should backend polling become controllable from a VCL, having some special
notation to access other VCLs' backends might be useful.

Nils



From phk at phk.freebsd.dk  Mon Sep 27 10:29:19 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 27 Sep 2010 10:29:19 +0000
Subject: Comment on doc/sphinx/phk/backends.rst : "wildcard'isch scheme" 
In-Reply-To: Your message of "Mon, 27 Sep 2010 12:23:39 +0200."
	<4CA070AB.3050301@schokola.de> 
Message-ID: <76899.1285583359@critter.freebsd.dk>

In message <4CA070AB.3050301 at schokola.de>, Nils Goroll writes:

>For the default case, I think it is important that, within a VCL, only those
>backends are visible which were defined within that VCL.

Hmm, I may not have written that clearly, right now any given VCL
will only ever see the backends it defines itself.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From slink at schokola.de  Mon Sep 27 10:53:10 2010
From: slink at schokola.de (Nils Goroll)
Date: Mon, 27 Sep 2010 12:53:10 +0200
Subject: Digests and data encoding in Varnish
Message-ID: <4CA07796.9030101@schokola.de>

Hi,

I need some generic way to handle message digests in Varnish, something like

	if (digest(SHA1, req.http.a) == req.http.b)

While it would be easy to implement VCL functions (preferrably in a module) for
such digests always returning a STRING representation with some encoding (base64
or whatever), for the VSLP director (which I briefly introduced here some time
ago and which we will publish when an important rewrite is done), it would be
really handy to also get an INT (or, more specifically, UINT32) representation
for digests.

Also it would be nice if

	if (digest(SHA1, req.http.a) == digest(SHA1, req.http.b))

would need a de-tour via ASCII encoding and also I will need to choose the
encoding as in

	set req.http.foo = encode(BASE64, digest(SHA1, "abc" + "def" + now));

As mentioned, I need this feature, so if no one else wants to work on it, I'd
look after the implementation, but, most importantly, I'd like to ask for the
core developers' advice on how to proceed.

My suggestion is to

* add two types to the vcc/vrt code:
  - DIGEST representing a message digest
  - DATA representing opaque data

* implement the actual functions in two vmods:
  - vmod_digest.c  for the actual message digests and
  - vmod_dataenc.c for the ascii-armor tools

The DIGEST would be represented as a struct with a type selector and some
implementation-private data. DATA would just represent any data, which functions
from vmod_codec.c would convert to and from some string (or other) representation.

Is this a good idea or not? Any comments, suggestions?
Does the core team support this suggestion?

Thanks, Nils



From phk at phk.freebsd.dk  Mon Sep 27 11:00:22 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 27 Sep 2010 11:00:22 +0000
Subject: Digests and data encoding in Varnish 
In-Reply-To: Your message of "Mon, 27 Sep 2010 12:53:10 +0200."
	<4CA07796.9030101@schokola.de> 
Message-ID: <77263.1285585222@critter.freebsd.dk>

In message <4CA07796.9030101 at schokola.de>, Nils Goroll writes:

>While it would be easy to implement VCL functions (preferrably in a module) for
>such digests always returning a STRING representation with some encoding (base64
>or whatever), for the VSLP director (which I briefly introduced here some time
>ago and which we will publish when an important rewrite is done), it would be
>really handy to also get an INT (or, more specifically, UINT32) representation
>for digests.

Compared to the any sensible digest algorithm, the string encoding is
totally nonvisible in the cpu-usage, so I am not convinced that it is
worth the hazzle of adding new datatypes to VRT.

I would simply write a vmod with digest functions somewhat like:

	md5_hex(STRING_LIST input)
	sha25_base64(STRING_LIST input)

and be done with it.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From slink at schokola.de  Mon Sep 27 11:14:00 2010
From: slink at schokola.de (Nils Goroll)
Date: Mon, 27 Sep 2010 13:14:00 +0200
Subject: Digests and data encoding in Varnish
In-Reply-To: <77263.1285585222@critter.freebsd.dk>
References: <77263.1285585222@critter.freebsd.dk>
Message-ID: <4CA07C78.9030107@schokola.de>

> I would simply write a vmod with digest functions somewhat like:
> 
> 	md5_hex(STRING_LIST input)
> 	sha25_base64(STRING_LIST input)
> 
> and be done with it.

Yes, for the time being that will probably be the simplest solution. If it turns
out to be worth it later, we can still make the code more generic.

So I'll write different function permutations returning things like string
encodings and int (cut-off).

Thanks, Nils



From tfheen at varnish-software.com  Mon Sep 27 11:39:21 2010
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Mon, 27 Sep 2010 13:39:21 +0200
Subject: caching of request with cookies
In-Reply-To: <AANLkTinA2qP9aAiPSGh5=SwAy8SdRR-s+RHcDjPRX4Qk@mail.gmail.com>
	(Harm Verhagen's message of "Thu, 23 Sep 2010 10:59:40 +0200")
References: <AANLkTinA2qP9aAiPSGh5=SwAy8SdRR-s+RHcDjPRX4Qk@mail.gmail.com>
Message-ID: <8739sva00m.fsf@qurzaw.linpro.no>

]] Harm Verhagen 

Hi,

| This seems a bit too conservative to me.
| According to the http specs the mere existence of a cookie should not
| influence cachebility.  In the content of the response depends on the
| cookie, the server should tell in the response that the contents are not
| cacheble.
| The Vary: Cookie header  exists for that reason, no ?

In an ideal world, you are right.  Unfortunately, the world is not ideal
and it's far between systems in the wild that does Vary: cookie.  If
your system does, great, just change the VCL. :-)

[...]

| What are the thoughts on this list about moving the default varnish
| configuration closer to the http specification, regarding caching of request
| where the client sends a cookie (and probably leading to problems on
| websites that do _not_ use http headers correctly)

I think changing the current behaviour would be a terribly bad idea.
We've seen people mess this up before, causing significant financial
losses.

Regards,
-- 
Tollef Fog Heen
Varnish Software
t: +47 21 54 41 73



From slink at schokola.de  Mon Sep 27 13:50:22 2010
From: slink at schokola.de (Nils Goroll)
Date: Mon, 27 Sep 2010 15:50:22 +0200
Subject: Proposal/specs for backend conditional requests / aka
	"GET	If-Modified-Since" (GET IMS))
In-Reply-To: <4C9B2BBE.5060309@schokola.de>
References: <4C9B2BBE.5060309@schokola.de>
Message-ID: <4CA0A11E.9060607@schokola.de>

Hi,

I'd like to add a brief update to the following section summarizing my
understanding after talking to phk today, who seems to be really busy and
probably will not find time to respond before the weekend:

> To allow multiple cache objects to share body data, we want to add
> reference counters to struct storage following the example of the
> existing implementation for objects (HSH_Ref(), HSH_Unref() etc).

Though I still believe this should be pretty straight forward for all other
storages, it won't be for -spersistent. After studying the code for an hour or
so, my understanding is the following:

Persistent storage segments the cache (see
http://www.varnish-cache.org/trac/wiki/ArchitecturePersistentStorage) and won't
re-use segments for new objects unless they are completely empty (no live
objects). Right now, this relies on the LRU and TTL based expiry to eventually
clean out segments before running out of space. Having multiple refs to the same
obj in persistent storage (and updating it again and again) would effectively
lead to more and more segments being kept from becoming empty.

I believe what is really needed is additional space management for the
persistent storage. In a first step, when running short of storage, objects
could get nuked from the smallest segment. In a second step, the mechanics to
copy live objects from one segment to another could be implemented. Ideally,
this could be vcl controlled ("should we rather nuke the object or bother
copying it?"). But I see some complications for both, mainly that storage would
need to know which objects are referencing it in order to update those (sounds
wrong).

As long as we don't have any of this, I suggest two alternative temporary solutions:

a) If an object getting refreshed lives in persistent storage, we'll simply copy
it. Actually, the existing Rackspace implementation does this. This is far from
optimal, but won't make much of a difference for small objects and is still much
more efficient than re-fetching the object from backend like today, so we
shouldn't see any performance regression.

For other stevedores, we'll use the reference counter.

b) Add reference counters to persistent storage, too, and simply live with the
cache fragmentation issue. Those using persistent storage would be advised not
to use cache refresh.

At this point, I'd favor a).


Please note that all of this is my personal understanding. I am posting these
thoughts in the hope that my understanding is correct and I'd really appreciate
corrections if it's not.

Thank you, Nils



From slink at schokola.de  Mon Sep 27 17:49:16 2010
From: slink at schokola.de (Nils Goroll)
Date: Mon, 27 Sep 2010 19:49:16 +0200
Subject: [patch] base64_encode
Message-ID: <4CA0D91C.5040301@schokola.de>

Preparing for vmod_digest, could I please have this integrated?

#ifdef TEST_DRIVER block also contains more test cases

Thanks, Nils
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: varnish_base64_encode.patch
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100927/0c692344/attachment.ksh>

From sky at crucially.net  Mon Sep 27 19:27:14 2010
From: sky at crucially.net (Artur Bergman)
Date: Mon, 27 Sep 2010 12:27:14 -0700
Subject: Proposal/specs for backend conditional requests / aka "GET
	If-Modified-Since" (GET IMS))
In-Reply-To: <4CA0A11E.9060607@schokola.de>
References: <4C9B2BBE.5060309@schokola.de> <4CA0A11E.9060607@schokola.de>
Message-ID: <0AAEFA7F-F272-4EA8-8C3B-44D19D34A545@crucially.net>

For persistant storage, just ignore the TTL and throw away the segment  
with the oldest object, refreshed or not.

I am of the opinion that if a method exists to verify the object, LM  
or Etag,  we shouldn't ever expire it. The ttl is just a setting for  
when we should refresh it. Of course, standard LRU should still apply.

I am also less worried about the reader/writer scenario for the  
headers, since by spec you shouldnt' update any headers that aren't  
Expires/Cache-Control (and weirdly enough, Vary)

Artur

On Sep 27, 2010, at 6:50 AM, Nils Goroll wrote:

> Hi,
>
> I'd like to add a brief update to the following section summarizing my
> understanding after talking to phk today, who seems to be really  
> busy and
> probably will not find time to respond before the weekend:
>
>> To allow multiple cache objects to share body data, we want to add
>> reference counters to struct storage following the example of the
>> existing implementation for objects (HSH_Ref(), HSH_Unref() etc).
>
> Though I still believe this should be pretty straight forward for  
> all other
> storages, it won't be for -spersistent. After studying the code for  
> an hour or
> so, my understanding is the following:
>
> Persistent storage segments the cache (see
> http://www.varnish-cache.org/trac/wiki/ 
> ArchitecturePersistentStorage) and won't
> re-use segments for new objects unless they are completely empty (no  
> live
> objects). Right now, this relies on the LRU and TTL based expiry to  
> eventually
> clean out segments before running out of space. Having multiple refs  
> to the same
> obj in persistent storage (and updating it again and again) would  
> effectively
> lead to more and more segments being kept from becoming empty.
>
> I believe what is really needed is additional space management for the
> persistent storage. In a first step, when running short of storage,  
> objects
> could get nuked from the smallest segment. In a second step, the  
> mechanics to
> copy live objects from one segment to another could be implemented.  
> Ideally,
> this could be vcl controlled ("should we rather nuke the object or  
> bother
> copying it?"). But I see some complications for both, mainly that  
> storage would
> need to know which objects are referencing it in order to update  
> those (sounds
> wrong).
>
> As long as we don't have any of this, I suggest two alternative  
> temporary solutions:
>
> a) If an object getting refreshed lives in persistent storage, we'll  
> simply copy
> it. Actually, the existing Rackspace implementation does this. This  
> is far from
> optimal, but won't make much of a difference for small objects and  
> is still much
> more efficient than re-fetching the object from backend like today,  
> so we
> shouldn't see any performance regression.
>
> For other stevedores, we'll use the reference counter.
>
> b) Add reference counters to persistent storage, too, and simply  
> live with the
> cache fragmentation issue. Those using persistent storage would be  
> advised not
> to use cache refresh.
>
> At this point, I'd favor a).
>
>
> Please note that all of this is my personal understanding. I am  
> posting these
> thoughts in the hope that my understanding is correct and I'd really  
> appreciate
> corrections if it's not.
>
> Thank you, Nils
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-dev




From slink at schokola.de  Tue Sep 28 06:35:09 2010
From: slink at schokola.de (Nils Goroll)
Date: Tue, 28 Sep 2010 08:35:09 +0200
Subject: ./lib/libvarnish/crc32.c
Message-ID: <4CA18C9D.4090009@schokola.de>

Hi,

could we get CRC32 back?

For vmod_digest, I'd otherwise like to include Tollef's implementation, but it's
not much code and it could be useful to have it in libvarnish.

Nils



From harm.verhagen at gmail.com  Tue Sep 28 08:28:03 2010
From: harm.verhagen at gmail.com (Harm Verhagen)
Date: Tue, 28 Sep 2010 10:28:03 +0200
Subject: caching of request with cookies
In-Reply-To: <8739sva00m.fsf@qurzaw.linpro.no>
References: <AANLkTinA2qP9aAiPSGh5=SwAy8SdRR-s+RHcDjPRX4Qk@mail.gmail.com>
	<8739sva00m.fsf@qurzaw.linpro.no>
Message-ID: <AANLkTimXr1=-JY2CVheWtCZzNa-kH9Fp26KLYP9S-zZe@mail.gmail.com>

Hi

On Mon, Sep 27, 2010 at 1:39 PM, Tollef Fog Heen <
tfheen at varnish-software.com> wrote:

> ]] Harm Verhagen
>
> Hi,
>
> | This seems a bit too conservative to me.
> | According to the http specs the mere existence of a cookie should not
> | influence cachebility.  In the content of the response depends on the
> | cookie, the server should tell in the response that the contents are not
> | cacheble.
> | The Vary: Cookie header  exists for that reason, no ?
>
> In an ideal world, you are right.  Unfortunately, the world is not ideal
> and it's far between systems in the wild that does Vary: cookie.  If
> your system does, great, just change the VCL. :-)
>

This 'ideal' system is django (out of the box).
I do think a reverse proxy project should also support systems that are
trying to behave ideally.
However most (all?)  of the VCL examples found on the varnish site, and even
VCL examples on forums, are about how to trick varnish into caching when the
system 'misbehaves'.
This is good & usefull, but somehow  the systems that try to follow the http
spec are overseen in the docs.

So i'm actually a) looking for a VCL example for systems that try to behave
ideally.  and b) a request to promote this in the documentation.

This not just a problem with varnish, but other reversy proxies projects
seems to have the same issue. Most configuration examples  & documentation
is geared towards systems that misbehave.


>
> | What are the thoughts on this list about moving the default varnish
> | configuration closer to the http specification, regarding caching of
> request
> | where the client sends a cookie (and probably leading to problems on
> | websites that do _not_ use http headers correctly)
>
> I think changing the current behaviour would be a terribly bad idea.
> We've seen people mess this up before, causing significant financial
> losses.
>
Fair enough.

Regards,
Harm
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-dev/attachments/20100928/d652a259/attachment.html>

From slink at schokola.de  Tue Sep 28 16:03:26 2010
From: slink at schokola.de (Nils Goroll)
Date: Tue, 28 Sep 2010 18:03:26 +0200
Subject: Hold: [patch] base64_encode
In-Reply-To: <4CA0D91C.5040301@schokola.de>
References: <4CA0D91C.5040301@schokola.de>
Message-ID: <4CA211CE.5050909@schokola.de>

I need to go over this again, I need an implementation which will also encode
arbitrary buffers, not just null-terminated strings, also base64_decode
currently doesn't restore the correct buffer length.



From paul.mansfield at taptu.com  Tue Sep 28 17:20:10 2010
From: paul.mansfield at taptu.com (Paul Mansfield)
Date: Tue, 28 Sep 2010 18:20:10 +0100
Subject: Hold: [patch] base64_encode
In-Reply-To: <4CA211CE.5050909@schokola.de>
References: <4CA0D91C.5040301@schokola.de> <4CA211CE.5050909@schokola.de>
Message-ID: <4CA223CA.6080503@taptu.com>

On 28/09/10 17:03, Nils Goroll wrote:
> I need to go over this again, I need an implementation which will also encode
> arbitrary buffers, not just null-terminated strings, also base64_decode
> currently doesn't restore the correct buffer length.


some base65 encoders are badly broken, I seem to remember adapting this
one to my needs; not sure whether there's better ones now as I it was
quite a while ago
http://migbase64.sourceforge.net/



From phk at phk.freebsd.dk  Wed Sep 29 13:01:59 2010
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 29 Sep 2010 13:01:59 +0000
Subject: Spring cleaning
Message-ID: <20440.1285765319@critter.freebsd.dk>


We have held a little bit of a spring cleaning, to dump stuff we have
abandonned.

We are unsure if anybody cares/works/uses these bits, please report
back if you do, or they will be removed at some later date:

	varnish-tools/autobuild
	varnish-tools/fetcher
	varnish-tools/webmin

Thanks in advance.

Also: The people maintaining the python and perl extensions should
start upgrading to the new VSM/VSC stuff in 3.0 soon.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.



From tfheen at varnish-software.com  Thu Sep 30 06:29:45 2010
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Thu, 30 Sep 2010 08:29:45 +0200
Subject: caching of request with cookies
In-Reply-To: <AANLkTimXr1=-JY2CVheWtCZzNa-kH9Fp26KLYP9S-zZe@mail.gmail.com>
	(Harm Verhagen's message of "Tue, 28 Sep 2010 10:28:03 +0200")
References: <AANLkTinA2qP9aAiPSGh5=SwAy8SdRR-s+RHcDjPRX4Qk@mail.gmail.com>
	<8739sva00m.fsf@qurzaw.linpro.no>
	<AANLkTimXr1=-JY2CVheWtCZzNa-kH9Fp26KLYP9S-zZe@mail.gmail.com>
Message-ID: <87zkuzbv6u.fsf@qurzaw.linpro.no>

]] Harm Verhagen 

Hi,

[...]

| I do think a reverse proxy project should also support systems that are
| trying to behave ideally.

Sure, which is why you can write VCL.  We give you a default policy that
is a bit on the conservative side.  The advantage being that it actually
works with more existing solutions.

| However most (all?)  of the VCL examples found on the varnish site, and even
| VCL examples on forums, are about how to trick varnish into caching when the
| system 'misbehaves'.

Sure, this is because most systems do misbehave.  Sad fact of life.

| This is good & usefull, but somehow  the systems that try to follow the http
| spec are overseen in the docs.
| 
| So i'm actually a) looking for a VCL example for systems that try to behave
| ideally.  and b) a request to promote this in the documentation.

For a), take the default VCL, drop the if (req.http.cookie) bit from
vcl_recv.  If Django sends out pages which start sessions with a
s-maxage of 0, you should be able to drop the check for set-cookie in
vcl_fetch too.f

As for b), file bugs in the bug tracker, preferably with patches. :-)

Regards,
-- 
Tollef Fog Heen
Varnish Software
t: +47 21 54 41 73



From oogali at blip.tv  Tue Sep 21 23:30:30 2010
From: oogali at blip.tv (Omachonu Ogali)
Date: Tue, 21 Sep 2010 19:30:30 -0400
Subject: Test failure
Message-ID: <AANLkTikxuwTcEXkokvsVA9KgjP6vNc0bTe127i7hP1he@mail.gmail.com>

I received the following test failure during the build:

---- v1   FAIL CLI response 200 expected 300
#### top  macro def tmpdir=/tmp/vtc.13335.6b8b4567
#### top  macro def bad_ip=10.255.255.255
#    top  TEST ././tests/c00003.vtc starting
#    top  TEST Check that we start if at least one listen address works
##   s1   Starting server
#### s1   macro def s1_addr=127.0.0.1
#### s1   macro def s1_port=33848
#### s1   macro def s1_sock=127.0.0.1:33848
#    s1   Listen on 127.0.0.1:33848
##   s1   Started on 127.0.0.1:33848
##   v1   Launch
###  v1   CMD: cd ../varnishd && ./varnishd -d -d -n
/tmp/vtc.13335.6b8b4567/v1 -p auto_restart=off -p
syslog_cli_traffic=off -a '127.0.0.1:0' -S
/tmp/vtc.13335.6b8b4567/v1/_S -M 127.0.0.1:47953 -P
/tmp/vtc.13335.6b8b4567/v1/varnishd.pid
-sfile,/tmp/vtc.13335.6b8b4567/v1,10M
###  v1   debug| storage_file: filename:
/tmp/vtc.13335.6b8b4567/v1/varnish.v5O0ho size 10 MB.\n
###  v1   debug| Creating new SHMFILE\n
###  v1   debug| Varnish on Linux,2.6.18-194.el5,x86_64,-sfile,-hcritbit\n
###  v1   debug| 200 193     \n
###  v1   debug| -----------------------------\n
###  v1   debug| Varnish HTTP accelerator CLI.\n
###  v1   debug| -----------------------------\n
###  v1   debug| Type 'help' for command list.\n
###  v1   debug| Type 'quit' to close CLI session.\n
###  v1   debug| Type 'start' to launch worker process.\n
###  v1   debug| \n
###  v1   CLI connection fd = 6
###  v1   CLI RX  107
#### v1   CLI RX| vkqhwykqxioyyfcbhxrtjeupeydufenb\n
#### v1   CLI RX| \n
#### v1   CLI RX| Authentication required.\n
#### v1   CLI TX| auth
2cdbd91ae8754abdc65434d020632ce6a50c54e9646160b56274e81b62f72592\n
###  v1   CLI RX  200
#### v1   CLI RX| -----------------------------\n
#### v1   CLI RX| Varnish HTTP accelerator CLI.\n
#### v1   CLI RX| -----------------------------\n
#### v1   CLI RX| Type 'help' for command list.\n
#### v1   CLI RX| Type 'quit' to close CLI session.\n
#### v1   CLI RX| Type 'start' to launch worker process.\n
#### v1   CLI TX| param.set listen_address 10.255.255.255:0
###  v1   CLI RX  200
##   v1   CLI 200 <param.set listen_address 10.255.255.255:0>
#### v1   CLI TX| vcl.inline vcl1 "backend s1 { .host = \"127.0.0.1\";
.port = \"33848\"; }\n"
###  v1   CLI RX  200
#### v1   CLI RX| VCL compiled.
#### v1   CLI TX| vcl.use vcl1
###  v1   CLI RX  200
#### v1   CLI TX| start
###  v1   debug| child (13380) Started\n
###  v1   CLI RX  200
##   v1   CLI 200 <start>
---- v1   FAIL CLI response 200 expected 300
###  v1   debug| Child (13380) said \n
###  v1   debug| Child (13380) said Child starts\n
###  v1   debug| Child (13380) said managed to mmap 10485760 bytes of 10485760\n
#    top  Test timed out
#    top  TEST ././tests/c00003.vtc FAILED


If this helps out at all regarding my environment...

# uname -a
Linux box1.testbench 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010
x86_64 x86_64 x86_64 GNU/Linux

# gcc --version
gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-48)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

# rpm -qa | grep glibc
glibc-headers-2.5-49.el5_5.4
glibc-common-2.5-49.el5_5.4
glibc-devel-2.5-49.el5_5.4
glibc-2.5-49.el5_5.4
glibc-2.5-49.el5_5.4


Let me know if you need any other information from my side.

oo