From CoreyH at ganz.com  Sat Oct  1 13:25:37 2011
From: CoreyH at ganz.com (Corey Hammerton)
Date: Sat, 1 Oct 2011 09:25:37 -0400
Subject: Round-Robin/Random Director Backend Behaviour
Message-ID: <9C8FD2499888F54396801D019DAB849B06591E8A49@TORE2K7V01.ganz.ad>

Hello,

In our environment we plan to put a varnish caching system in place to replace a system that actively synchronizes static content across multiple backend servers. To ensure that the load is equally distributed across all servers we will need to put a director in place (Random director with equal weight across all backends).

If we implement this system and varnish is asked to look for a static asset that is not present in some of the backend servers does it automatically go to the next backend server in the list? If not what VCL code would we have to implement to mitigate this problem?

Thanks.



From bedis9 at gmail.com  Sat Oct  1 22:54:22 2011
From: bedis9 at gmail.com (Baptiste)
Date: Sun, 2 Oct 2011 00:54:22 +0200
Subject: Varnish + stud (ssl) + PROXY protocol
In-Reply-To: <CAJ9Zqn2TxSg75zoh3cxsLzBYHy_vSH3qA=W2zJEmpAuQztSCbw@mail.gmail.com>
References: <CAJ9Zqn2TxSg75zoh3cxsLzBYHy_vSH3qA=W2zJEmpAuQztSCbw@mail.gmail.com>
Message-ID: <CAODHi7pfZHpz9PXeYHq1ErbYxKsRXAVkTZqjvyrNYRWeMy7gOw@mail.gmail.com>

On Wed, Sep 28, 2011 at 1:42 AM, Javier Frias <jfrias at gmail.com> wrote:
> Yeap, I know the varnish ssl question has been beaten to a pulp. But this
> one is slightly different.. ( don't we all say that )
>
> Has anyone used stud instead of stunnel with Varnish? And somewhat related,
> Is there any chance of PROXY support in varnish?
> https://www.varnish-cache.org/trac/ticket/943
>
> I ideally would like to use https://github.com/bumptech/stud instead of
> stunnel as my ssl frontend proxy for a new setup i'm working on which will
> be high TPS all ssl serving. ( Thank to ssl AES-NI and Sandy Bridge awesome
> performance )
>
> Figured I'd check the list to see if anyone had done it before tackling
> this.
>
> -Javier
>

It would be wonderfull to get PROXY protocol in Varnish :)

cheers



From MSchurenko at airg.com  Sun Oct  2 02:35:22 2011
From: MSchurenko at airg.com (Matt Schurenko)
Date: Sat, 1 Oct 2011 19:35:22 -0700
Subject: understanding cache size
Message-ID: <8B5C6F575422414AA91B46C454126B6C02EB282551@EXCHMVS.exchange.airg>

>From what I understand sm_bfree is the amount of bytes that are free in the cache while sm_balloc is the amount of used bytes in the cache (at least with varnish version 2.1.5). What I'm trying to understand is why two servers with the exact same hardware and varnish configuration that are using the same backends can have such a drastic difference with regards to these stats. Both server have 48GB of RAM. On both servers I have configured a cache size of 48 GB like so:

-s file,/tmp/varnish-cache,48G

If I add up sm_bfree and sm_balloc on both servers they equal 48GB:

varnish1:
[root at varnish1 ~]# varnishstat -1 | egrep 'sm_bfree|sm_balloc'
sm_balloc         13053640704          .   bytes allocated
sm_bfree          38485966848          .   bytes free

[root at varnish1 ~]# echo 'scale=3;(13053640704 + 38485966848) / 1024 / 1024 / 1024' | bc
48.000

varnish2:
[root at varnish2 ~]# varnishstat -1 | egrep 'sm_bfree|sm_balloc'
sm_balloc         48657035264          .   bytes allocated
sm_bfree           2882572288          .   bytes free

[root at varnish2 ~]#  echo 'scale=3;(48657035264 + 2882572288) / 1024 / 1024 / 1024' | bc
48.000

Can anyone explain to me why varnish2 uses so much more cache than varnish1? varnish1 has been running for 24 days and it maintains the ratio of bytes allocated to bytes free by frequently lru nuking. (This average to ~ 60 lru nukes per second in MRTG) This server has been quite stable; however I have noticed a lot of disk I/O.

The problem with varnish2 is that it does no lru nuking until the cache is nearly full, at which point the worker thread count maxes out (6000 in my case) and no new clients can connect. Because of this I am only able to run this server for ~ 48 hours before I have to restart varnish.


Thanks,

Matt Schurenko
Systems Administrator

airG(r) Share Your World
Suite 710, 1133 Melville Street
Vancouver, BC  V6E 4E5
P: +1.604.408.2228
F: +1.866.874.8136
E: MSchurenko at airg.com
W: www.airg.com<http://www.airg.com>

airG is one of BC's Top 55 Employers and
Canada's Top Employers for Young People

P Please consider the environment before printing this e-mail.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material communicated under NDA. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111001/ad808f50/attachment.html>

From enno at tty.nl  Mon Oct  3 07:39:43 2011
From: enno at tty.nl (Enno van Amerongen)
Date: Mon, 03 Oct 2011 09:39:43 +0200
Subject: Round-Robin Director Backends
In-Reply-To: <9C8FD2499888F54396801D019DAB849B065993D93A@TORE2K7V01.ganz.ad>
References: <9C8FD2499888F54396801D019DAB849B065993D93A@TORE2K7V01.ganz.ad>
Message-ID: <4E8966BF.7040206@tty.nl>

Not automatically, but you can put the following in your vcl_fetch.
However, if its a genuine 404, 10 backends will be bothered with it.

sub vcl_fetch {

    if (beresp.status == 404) {
        if (req.restarts < 10) {
            return (restart);
        }
    }

    return (deliver);
}



Corey Hammerton wrote:
>
> Hello,
>
>  
>
> I have an environment with multiple backend servers that require most
> images (static content) be replicated out to all servers, this is
> causing unnecessary load on our backend servers. To mitigate this
> problem we plan to put in a Varnish system to grab the assets from the
> backend servers and send them to the client VIA a round-robin director.
>
>  
>
> If Varnish cannot find the asset on the current backend server does it
> automatically go to the next backend in the list? If not what VCL code
> do we need to insert to accomplish this?
>
>  
>
>  
>
> Regards,
>
>  
>
> Corey
>
>  
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc





From perbu at varnish-software.com  Mon Oct  3 08:24:46 2011
From: perbu at varnish-software.com (Per Buer)
Date: Mon, 3 Oct 2011 10:24:46 +0200
Subject: Round-Robin Director Backends
In-Reply-To: <9C8FD2499888F54396801D019DAB849B065993D93A@TORE2K7V01.ganz.ad>
References: <9C8FD2499888F54396801D019DAB849B065993D93A@TORE2K7V01.ganz.ad>
Message-ID: <CAOXZevAAzQNRSObj_HayKGJiSs-Ae1Fi300JU3vPju=kHmvfCw@mail.gmail.com>

Hi,

On Thu, Sep 29, 2011 at 8:52 PM, Corey Hammerton <CoreyH at ganz.com> wrote:


Hello,****
>
> ** **
>
> I have an environment with multiple backend servers that require most
> images (static content) be replicated out to all servers, this is causing
> unnecessary load on our backend servers. To mitigate this problem we plan to
> put in a Varnish system to grab the assets from the backend servers and send
> them to the client VIA a round-robin director. ****
>
> ** **
>
> If Varnish cannot find the asset on the current backend server does it
> automatically go to the next backend in the list? If not what VCL code do we
> need to insert to accomplish this?
>

You should probably have a look at saint mode. It can do what you want.

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111003/c581f907/attachment.html>

From tfheen at varnish-software.com  Mon Oct  3 09:27:23 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Mon, 03 Oct 2011 11:27:23 +0200
Subject: varnishncsa and negative matches on hostname
In-Reply-To: <CADAfL_iJgysbuaxk2YPtRAGmL_5x-uL7Qqxn4URk=pS8kp-ykA@mail.gmail.com>
	(Bryan Stillwell's message of "Mon, 26 Sep 2011 16:53:09 -0600")
References: <CADAfL_iJgysbuaxk2YPtRAGmL_5x-uL7Qqxn4URk=pS8kp-ykA@mail.gmail.com>
Message-ID: <87ipo6tog4.fsf@qurzaw.varnish-software.com>

]] Bryan Stillwell 

Hi,

| /var/log/varnish/other/access.log.  What I believe I need is some kind
| of negated match (perl's ~! or grep's -v option).  Unfortunately I'm
| not seeing anything like that in the documentation.  Does anyone know
| how to make something like this work right now?

-m 'RxHeader:Host: (?!foo)' might work.

see man pcresyntax for more info on PCRE.

Cheers,
-- 
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64



From ljorg6 at gmail.com  Mon Oct  3 12:21:00 2011
From: ljorg6 at gmail.com (=?ISO-8859-1?Q?Lars_J=F8rgensen?=)
Date: Mon, 03 Oct 2011 14:21:00 +0200
Subject: varnishncsa crashing constantly
Message-ID: <4E89A8AC.6090402@gmail.com>

Hi,

Weird thing; since saturday night Varnishncsa has been crashing 
constantly in our setup. We are at varnish-3.0.0, and that version of 
varnishncsa has known to be buggy.

Today I upgraded varnishncsa (only, there was a post about varnish 
instability in the new release) to 3.0.1, but that didn't change anything.

I don't know what I should look for to provide useful information. Below 
is the last log line it posted before dying a few minutes ago:

194.103.189.41 - - [03/Oct/2011:14:14:42 +0200] "GET /da/sso/logout/ 
HTTP/1.1" 302 0 "http://www.kb.dk/da/index.html" "Mozilla/5.0 (Windows 
NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.186 
Safari/535.1"

I run it with these parameters:

DAEMON_OPTS="-a -w $logfile -D -P $pidfile -F '%h %l %u %t \"%m %U %H\" 
%s %b \"%{Referer}i\" \"%{User-agent}i\"'"


-- 
Lars



From jeanmarc.pouchoulon at gmail.com  Mon Oct  3 12:39:39 2011
From: jeanmarc.pouchoulon at gmail.com (jean-marc pouchoulon)
Date: Mon, 3 Oct 2011 14:39:39 +0200
Subject: chunked header nbr syntax error
Message-ID: <CAGBM0LmNFcCLyYmxY=ZmQ3eQ3es0jZNusXPi2wkvb4-TC2AZXw@mail.gmail.com>

Helo varnishlist,

I 've got a problem between calendar application   and varnish 3.0.1

14 ObjResponse  c OK
   14 ObjHeader    c X-Powered-By: Servlet/2.5
   14 ObjHeader    c Server: Sun GlassFish Enterprise Server v2.1
   14 ObjHeader    c X-Powered-By: Servlet/2.5
   14 ObjHeader    c Server: Sun GlassFish Enterprise Server v2.1
   14 ObjHeader    c Date: Mon, 03 Oct 2011 12:16:35 GMT
   14 ObjHeader    c Content-Type: application/javascript;charset=utf-8
   14 FetchError   c chunked header nbr syntax

bug ? ( I tried to increment header size without any success)



jean-marc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111003/f04ca295/attachment.html>

From phk at phk.freebsd.dk  Mon Oct  3 12:52:02 2011
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 03 Oct 2011 12:52:02 +0000
Subject: chunked header nbr syntax error
In-Reply-To: Your message of "Mon, 03 Oct 2011 14:39:39 +0200."
	<CAGBM0LmNFcCLyYmxY=ZmQ3eQ3es0jZNusXPi2wkvb4-TC2AZXw@mail.gmail.com>
Message-ID: <65461.1317646322@critter.freebsd.dk>

In message <CAGBM0LmNFcCLyYmxY=ZmQ3eQ3es0jZNusXPi2wkvb4-TC2AZXw at mail.gmail.com>
, jean-marc pouchoulon writes:

>   14 FetchError   c chunked header nbr syntax

This means that the hex-number giving the chunked length contained
junk.

Try to get a TCP dump if possible ?


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From daniel.carrillo at gmail.com  Mon Oct  3 13:13:09 2011
From: daniel.carrillo at gmail.com (Daniel Carrillo)
Date: Mon, 3 Oct 2011 15:13:09 +0200
Subject: varnishncsa crashing constantly
In-Reply-To: <4E89A8AC.6090402@gmail.com>
References: <4E89A8AC.6090402@gmail.com>
Message-ID: <CAK6GuyR=cqqzg1mOEKAe9U+WnLQ+iEVNnmKc2WjFV0cvZWtMdg@mail.gmail.com>

2011/10/3 Lars J?rgensen <ljorg6 at gmail.com>:
> Hi,
>
> Weird thing; since saturday night Varnishncsa has been crashing constantly
> in our setup. We are at varnish-3.0.0, and that version of varnishncsa has
> known to be buggy.
>
> Today I upgraded varnishncsa (only, there was a post about varnish
> instability in the new release) to 3.0.1, but that didn't change anything.
>
> I don't know what I should look for to provide useful information. Below is
> the last log line it posted before dying a few minutes ago:

Hi, varnishncsa in 3.0.1 is still buggy, several bugs were fixed in
trunk. Particulary:

https://www.varnish-cache.org/trac/ticket/1006

Kind Regards.



From ljorg6 at gmail.com  Mon Oct  3 13:48:44 2011
From: ljorg6 at gmail.com (=?ISO-8859-1?Q?Lars_J=F8rgensen?=)
Date: Mon, 03 Oct 2011 15:48:44 +0200
Subject: varnishncsa crashing constantly
In-Reply-To: <CAK6GuyR=cqqzg1mOEKAe9U+WnLQ+iEVNnmKc2WjFV0cvZWtMdg@mail.gmail.com>
References: <4E89A8AC.6090402@gmail.com>
	<CAK6GuyR=cqqzg1mOEKAe9U+WnLQ+iEVNnmKc2WjFV0cvZWtMdg@mail.gmail.com>
Message-ID: <4E89BD3C.1070905@gmail.com>

On 03-10-2011 15:13, Daniel Carrillo wrote:
> Hi, varnishncsa in 3.0.1 is still buggy, several bugs were fixed in
> trunk. Particulary:
>
> https://www.varnish-cache.org/trac/ticket/1006

Sounds like that might be my way out. How does one get trunk?


-- 
Lars



From daniel.carrillo at gmail.com  Mon Oct  3 14:18:36 2011
From: daniel.carrillo at gmail.com (Daniel Carrillo)
Date: Mon, 3 Oct 2011 16:18:36 +0200
Subject: varnishncsa crashing constantly
In-Reply-To: <4E89BD3C.1070905@gmail.com>
References: <4E89A8AC.6090402@gmail.com>
	<CAK6GuyR=cqqzg1mOEKAe9U+WnLQ+iEVNnmKc2WjFV0cvZWtMdg@mail.gmail.com>
	<4E89BD3C.1070905@gmail.com>
Message-ID: <CAK6GuySp6K4+mXP9ZYvFtJxhLD8jas2QujZQiSkHuhbqA=vRhQ@mail.gmail.com>

2011/10/3 Lars J?rgensen <ljorg6 at gmail.com>:
> On 03-10-2011 15:13, Daniel Carrillo wrote:
>>
>> Hi, varnishncsa in 3.0.1 is still buggy, several bugs were fixed in
>> trunk. Particulary:
>>
>> https://www.varnish-cache.org/trac/ticket/1006
>
> Sounds like that might be my way out. How does one get trunk?

Well, you need to check out de source v?a SVN and recompile it.

In our case, we're using CentOS, so I downloaded the src.rpm from
http://repo.varnish-cache.org/redhat/varnish-3.0/el5/src/, I changed
varnishncsa.c with the current versi?n and I built a new rpm.

Kind regards.



From CoreyH at ganz.com  Mon Oct  3 14:24:30 2011
From: CoreyH at ganz.com (Corey Hammerton)
Date: Mon, 3 Oct 2011 10:24:30 -0400
Subject: Round-Robin Director Backends
In-Reply-To: <CAOXZevAAzQNRSObj_HayKGJiSs-Ae1Fi300JU3vPju=kHmvfCw@mail.gmail.com>
References: <9C8FD2499888F54396801D019DAB849B065993D93A@TORE2K7V01.ganz.ad>
	<CAOXZevAAzQNRSObj_HayKGJiSs-Ae1Fi300JU3vPju=kHmvfCw@mail.gmail.com>
Message-ID: <9C8FD2499888F54396801D019DAB849B06599CB358@TORE2K7V01.ganz.ad>

Are there any plans to make the Varnish directors be able to read the backend response WITHOUT having to put it into VCL code?

For instance, a client requests an asset (a picture) that is present on one server and not on any of the others, if the response from the current backend server is a 404 automatically go to the next backend configured in the director.

From: Per Buer [mailto:perbu at varnish-software.com]
Sent: Monday, October 03, 2011 4:25 AM
To: Corey Hammerton
Cc: varnish-misc at varnish-cache.org; Gordon Cook
Subject: Re: Round-Robin Director Backends

Hi,

On Thu, Sep 29, 2011 at 8:52 PM, Corey Hammerton <CoreyH at ganz.com<mailto:CoreyH at ganz.com>> wrote:


Hello,

I have an environment with multiple backend servers that require most images (static content) be replicated out to all servers, this is causing unnecessary load on our backend servers. To mitigate this problem we plan to put in a Varnish system to grab the assets from the backend servers and send them to the client VIA a round-robin director.

If Varnish cannot find the asset on the current backend server does it automatically go to the next backend in the list? If not what VCL code do we need to insert to accomplish this?

You should probably have a look at saint mode. It can do what you want.

--
[cid:~WRD000.jpg]
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
Varnish makes websites fly!
Whitepapers<http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> | Twitter<https://twitter.com/varnishsoftware>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111003/a6773e75/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ~WRD000.jpg
Type: image/jpeg
Size: 823 bytes
Desc: ~WRD000.jpg
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111003/a6773e75/attachment.jpg>

From perbu at varnish-software.com  Mon Oct  3 15:05:11 2011
From: perbu at varnish-software.com (Per Buer)
Date: Mon, 3 Oct 2011 17:05:11 +0200
Subject: Round-Robin Director Backends
In-Reply-To: <9C8FD2499888F54396801D019DAB849B06599CB358@TORE2K7V01.ganz.ad>
References: <9C8FD2499888F54396801D019DAB849B065993D93A@TORE2K7V01.ganz.ad>
	<CAOXZevAAzQNRSObj_HayKGJiSs-Ae1Fi300JU3vPju=kHmvfCw@mail.gmail.com>
	<9C8FD2499888F54396801D019DAB849B06599CB358@TORE2K7V01.ganz.ad>
Message-ID: <CAOXZevAAMfY-bvz56n_zurzBddpE8eQ-7WeJqAqrxadddEB8GQ@mail.gmail.com>

Hi Corey.

No, there are no such plans. Sorry.

Per.

On Mon, Oct 3, 2011 at 4:24 PM, Corey Hammerton <CoreyH at ganz.com> wrote:

> Are there any plans to make the Varnish directors be able to read the
> backend response WITHOUT having to put it into VCL code?****
>
> ** **
>
> For instance, a client requests an asset (a picture) that is present on one
> server and not on any of the others, if the response from the current
> backend server is a 404 automatically go to the next backend configured in
> the director. ****
>
> ** **
>
> *From:* Per Buer [mailto:perbu at varnish-software.com]
> *Sent:* Monday, October 03, 2011 4:25 AM
> *To:* Corey Hammerton
> *Cc:* varnish-misc at varnish-cache.org; Gordon Cook
> *Subject:* Re: Round-Robin Director Backends****
>
> ** **
>
> Hi,****
>
> ** **
>
> On Thu, Sep 29, 2011 at 8:52 PM, Corey Hammerton <CoreyH at ganz.com> wrote:*
> ***
>
> ** **
>
> ** **
>
> Hello,****
>
>  ****
>
> I have an environment with multiple backend servers that require most
> images (static content) be replicated out to all servers, this is causing
> unnecessary load on our backend servers. To mitigate this problem we plan to
> put in a Varnish system to grab the assets from the backend servers and send
> them to the client VIA a round-robin director. ****
>
>  ****
>
> If Varnish cannot find the asset on the current backend server does it
> automatically go to the next backend in the list? If not what VCL code do we
> need to insert to accomplish this?****
>
> ** **
>
> You should probably have a look at saint mode. It can do what you want.***
> *
>
> ** **
>
> --
> [image: Image removed by sender.]****
>
> Per Buer, CEO
> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
> *Varnish makes websites fly!*****
>
> Whitepapers <http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
> Twitter <https://twitter.com/varnishsoftware> ****
>
> ** **
>



-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111003/f3190273/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ~WRD000.jpg
Type: image/jpeg
Size: 823 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111003/f3190273/attachment.jpg>

From humansky at Princeton.EDU  Tue Oct  4 01:26:59 2011
From: humansky at Princeton.EDU (Henry M. Umansky)
Date: Tue, 4 Oct 2011 01:26:59 +0000
Subject: Varnish and multiple interfaces
Message-ID: <4A5A0CA3-0441-4D19-AA91-D65F1D709A0B@exchange.princeton.edu>

Hello,
I'm running Varnish 2.1.5 on Red Hat Enterprise Linux Server release 6.1. Currently I have two interfaces: eth0 and an alias eth0:0. I need Varnish to bind to eth0:0, which it does perfectly, however, outgoing traffic is going through eth0. Is there anyway to tell Varnish to send outgoing traffic through the same IP address I tell varnish to "listen" to? I guess I can route the traffic accordingly via iptables, but I'd prefer to do it at the application layer if possible.

Any help would be much appreciated, varnish is an amazing product!!!

Henry Umansky
Web Development Services
Princeton University
humansky at princeton.edu
609-258-1674




From david.birdsong at gmail.com  Tue Oct  4 02:02:00 2011
From: david.birdsong at gmail.com (David Birdsong)
Date: Mon, 3 Oct 2011 19:02:00 -0700
Subject: Varnish and multiple interfaces
In-Reply-To: <4A5A0CA3-0441-4D19-AA91-D65F1D709A0B@exchange.princeton.edu>
References: <4A5A0CA3-0441-4D19-AA91-D65F1D709A0B@exchange.princeton.edu>
Message-ID: <CABJfL5iRvCNCXRj-HbDM6zRiJxBygYxVmfYeWMxRgzBxgJ1kYA@mail.gmail.com>

Varnish doesn't send traffic out of interfaces, the OS does. Your
kernel routing table will determine which device is part of a
particular route. For most traffic, the 'default' route is the route
that matches outbound traffic.  You can change your default route to
exit a particular interface--though I'm not sure if an ethernet alias
will work. Try it out.

On Mon, Oct 3, 2011 at 6:26 PM, Henry M. Umansky <humansky at princeton.edu> wrote:
> Hello,
> I'm running Varnish 2.1.5 on Red Hat Enterprise Linux Server release 6.1. Currently I have two interfaces: eth0 and an alias eth0:0. I need Varnish to bind to eth0:0, which it does perfectly, however, outgoing traffic is going through eth0. Is there anyway to tell Varnish to send outgoing traffic through the same IP address I tell varnish to "listen" to? I guess I can route the traffic accordingly via iptables, but I'd prefer to do it at the application layer if possible.
>
> Any help would be much appreciated, varnish is an amazing product!!!
>
> Henry Umansky
> Web Development Services
> Princeton University
> humansky at princeton.edu
> 609-258-1674
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



From humansky at Princeton.EDU  Tue Oct  4 02:31:56 2011
From: humansky at Princeton.EDU (Henry M. Umansky)
Date: Tue, 4 Oct 2011 02:31:56 +0000
Subject: Varnish and multiple interfaces
In-Reply-To: <CABJfL5iRvCNCXRj-HbDM6zRiJxBygYxVmfYeWMxRgzBxgJ1kYA@mail.gmail.com>
References: <4A5A0CA3-0441-4D19-AA91-D65F1D709A0B@exchange.princeton.edu>
	<CABJfL5iRvCNCXRj-HbDM6zRiJxBygYxVmfYeWMxRgzBxgJ1kYA@mail.gmail.com>
Message-ID: <18307DC1-8BE9-46CF-AB37-4F627D73FAE3@exchange.princeton.edu>

That's what I thought too, but I'm also running nginx on the same machine. I set up nginx to listen to eth0:0,  and all traffic to the remote backend sees the IP address of eth0:0. However, when I try to do the same with Varnish, the backends sees the IP address of eth0.

Henry Umansky
Web Development Services
Princeton University
humansky at princeton.edu<mailto:humansky at princeton.edu>
609-258-1674

On Oct 3, 2011, at 10:02 PM, David Birdsong wrote:

Varnish doesn't send traffic out of interfaces, the OS does. Your
kernel routing table will determine which device is part of a
particular route. For most traffic, the 'default' route is the route
that matches outbound traffic.  You can change your default route to
exit a particular interface--though I'm not sure if an ethernet alias
will work. Try it out.

On Mon, Oct 3, 2011 at 6:26 PM, Henry M. Umansky <humansky at princeton.edu<mailto:humansky at princeton.edu>> wrote:
Hello,
I'm running Varnish 2.1.5 on Red Hat Enterprise Linux Server release 6.1. Currently I have two interfaces: eth0 and an alias eth0:0. I need Varnish to bind to eth0:0, which it does perfectly, however, outgoing traffic is going through eth0. Is there anyway to tell Varnish to send outgoing traffic through the same IP address I tell varnish to "listen" to? I guess I can route the traffic accordingly via iptables, but I'd prefer to do it at the application layer if possible.

Any help would be much appreciated, varnish is an amazing product!!!

Henry Umansky
Web Development Services
Princeton University
humansky at princeton.edu<mailto:humansky at princeton.edu>
609-258-1674


_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111004/a4d53f07/attachment.html>

From david.birdsong at gmail.com  Tue Oct  4 02:34:39 2011
From: david.birdsong at gmail.com (David Birdsong)
Date: Mon, 3 Oct 2011 19:34:39 -0700
Subject: Varnish and multiple interfaces
In-Reply-To: <18307DC1-8BE9-46CF-AB37-4F627D73FAE3@exchange.princeton.edu>
References: <4A5A0CA3-0441-4D19-AA91-D65F1D709A0B@exchange.princeton.edu>
	<CABJfL5iRvCNCXRj-HbDM6zRiJxBygYxVmfYeWMxRgzBxgJ1kYA@mail.gmail.com>
	<18307DC1-8BE9-46CF-AB37-4F627D73FAE3@exchange.princeton.edu>
Message-ID: <CABJfL5jQabAWPhTD_rmOcNgOfBh3ytTWhA1Ct2eHrk3fXUaUiw@mail.gmail.com>

Are the remote backends by chance looking at either the layer 4 (ip
level) address OR if present, a value found in an http header such as
X-Forwarded-For?

On Mon, Oct 3, 2011 at 7:31 PM, Henry M. Umansky <humansky at princeton.edu> wrote:
> That's what I thought too, but I'm also running nginx on the same machine. I
> set up nginx to listen to eth0:0, ?and all traffic to the remote backend
> sees the IP address of eth0:0. However, when I try to do the same with
> Varnish, the backends sees the IP address of eth0.
>
> Henry Umansky
> Web Development Services
> Princeton University
> humansky at princeton.edu
> 609-258-1674
>
> On Oct 3, 2011, at 10:02 PM, David Birdsong wrote:
>
> Varnish doesn't send traffic out of interfaces, the OS does. Your
> kernel routing table will determine which device is part of a
> particular route. For most traffic, the 'default' route is the route
> that matches outbound traffic. ?You can change your default route to
> exit a particular interface--though I'm not sure if an ethernet alias
> will work. Try it out.
>
> On Mon, Oct 3, 2011 at 6:26 PM, Henry M. Umansky <humansky at princeton.edu>
> wrote:
>
> Hello,
>
> I'm running Varnish 2.1.5 on Red Hat Enterprise Linux Server release 6.1.
> Currently I have two interfaces: eth0 and an alias eth0:0. I need Varnish to
> bind to eth0:0, which it does perfectly, however, outgoing traffic is going
> through eth0. Is there anyway to tell Varnish to send outgoing traffic
> through the same IP address I tell varnish to "listen" to? I guess I can
> route the traffic accordingly via iptables, but I'd prefer to do it at the
> application layer if possible.
>
> Any help would be much appreciated, varnish is an amazing product!!!
>
> Henry Umansky
>
> Web Development Services
>
> Princeton University
>
> humansky at princeton.edu
>
> 609-258-1674
>
>
> _______________________________________________
>
> varnish-misc mailing list
>
> varnish-misc at varnish-cache.org
>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



From humansky at Princeton.EDU  Tue Oct  4 02:57:07 2011
From: humansky at Princeton.EDU (Henry M. Umansky)
Date: Tue, 4 Oct 2011 02:57:07 +0000
Subject: Varnish and multiple interfaces
In-Reply-To: <CABJfL5jQabAWPhTD_rmOcNgOfBh3ytTWhA1Ct2eHrk3fXUaUiw@mail.gmail.com>
References: <4A5A0CA3-0441-4D19-AA91-D65F1D709A0B@exchange.princeton.edu>
	<CABJfL5iRvCNCXRj-HbDM6zRiJxBygYxVmfYeWMxRgzBxgJ1kYA@mail.gmail.com>
	<18307DC1-8BE9-46CF-AB37-4F627D73FAE3@exchange.princeton.edu>
	<CABJfL5jQabAWPhTD_rmOcNgOfBh3ytTWhA1Ct2eHrk3fXUaUiw@mail.gmail.com>
Message-ID: <186FEE23-3C2E-4752-A97E-E06306D96D2C@exchange.princeton.edu>

According to tcpdump, traffic from varnish is coming from eth0 (incorrect IP) and traffic coming from nginx is coming from eth0:0 (correct IP). Apache logs are also confirming the IP information.

Henry Umansky
Web Development Services
Princeton University
humansky at princeton.edu
609-258-1674

On Oct 3, 2011, at 10:34 PM, David Birdsong wrote:

> Are the remote backends by chance looking at either the layer 4 (ip
> level) address OR if present, a value found in an http header such as
> X-Forwarded-For?
> 
> On Mon, Oct 3, 2011 at 7:31 PM, Henry M. Umansky <humansky at princeton.edu> wrote:
>> That's what I thought too, but I'm also running nginx on the same machine. I
>> set up nginx to listen to eth0:0,  and all traffic to the remote backend
>> sees the IP address of eth0:0. However, when I try to do the same with
>> Varnish, the backends sees the IP address of eth0.
>> 
>> Henry Umansky
>> Web Development Services
>> Princeton University
>> humansky at princeton.edu
>> 609-258-1674
>> 
>> On Oct 3, 2011, at 10:02 PM, David Birdsong wrote:
>> 
>> Varnish doesn't send traffic out of interfaces, the OS does. Your
>> kernel routing table will determine which device is part of a
>> particular route. For most traffic, the 'default' route is the route
>> that matches outbound traffic.  You can change your default route to
>> exit a particular interface--though I'm not sure if an ethernet alias
>> will work. Try it out.
>> 
>> On Mon, Oct 3, 2011 at 6:26 PM, Henry M. Umansky <humansky at princeton.edu>
>> wrote:
>> 
>> Hello,
>> 
>> I'm running Varnish 2.1.5 on Red Hat Enterprise Linux Server release 6.1.
>> Currently I have two interfaces: eth0 and an alias eth0:0. I need Varnish to
>> bind to eth0:0, which it does perfectly, however, outgoing traffic is going
>> through eth0. Is there anyway to tell Varnish to send outgoing traffic
>> through the same IP address I tell varnish to "listen" to? I guess I can
>> route the traffic accordingly via iptables, but I'd prefer to do it at the
>> application layer if possible.
>> 
>> Any help would be much appreciated, varnish is an amazing product!!!
>> 
>> Henry Umansky
>> 
>> Web Development Services
>> 
>> Princeton University
>> 
>> humansky at princeton.edu
>> 
>> 609-258-1674
>> 
>> 
>> _______________________________________________
>> 
>> varnish-misc mailing list
>> 
>> varnish-misc at varnish-cache.org
>> 
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>> 
>> 
>> 
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>> 




From phk at phk.freebsd.dk  Tue Oct  4 05:29:47 2011
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Tue, 04 Oct 2011 05:29:47 +0000
Subject: Varnish and multiple interfaces
In-Reply-To: Your message of "Tue, 04 Oct 2011 02:57:07 GMT."
	<186FEE23-3C2E-4752-A97E-E06306D96D2C@exchange.princeton.edu>
Message-ID: <68577.1317706187@critter.freebsd.dk>

In message <186FEE23-3C2E-4752-A97E-E06306D96D2C at exchange.princeton.edu>, "Henr
y M. Umansky" writes:

>According to tcpdump, traffic from varnish is coming from eth0
>(incorrect IP) and traffic coming from nginx is coming from eth0:0
>(correct IP). Apache logs are also confirming the IP information.

This is a theoretical point I have had in my "notes" file for some
time, you are the first person to raise it in reality.

We don't bind(2) backend connections to any specific IP# in Varnish,
but let the kernel pick whatever it finds most convenient at connect(2)
time.

The kernels I'm aware of will pick the interface IP# which has the
"best route" to the destination, which in your case is eth0 instead
of eth0.0.

The main we have not added a facility to bind backend connections
to a particular IP# is that it is a recipe for reachability problems
and kind of hard to imagine a legit case for wanting to do it in
the first place, so can I get you to describe (possibly in private
email) why you need to do this ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From mattias at nucleus.be  Tue Oct  4 08:01:15 2011
From: mattias at nucleus.be (Mattias Geniar)
Date: Tue, 4 Oct 2011 10:01:15 +0200
Subject: Varnish and multiple interfaces
In-Reply-To: <186FEE23-3C2E-4752-A97E-E06306D96D2C@exchange.princeton.edu>
References: <4A5A0CA3-0441-4D19-AA91-D65F1D709A0B@exchange.princeton.edu><CABJfL5iRvCNCXRj-HbDM6zRiJxBygYxVmfYeWMxRgzBxgJ1kYA@mail.gmail.com><18307DC1-8BE9-46CF-AB37-4F627D73FAE3@exchange.princeton.edu><CABJfL5jQabAWPhTD_rmOcNgOfBh3ytTWhA1Ct2eHrk3fXUaUiw@mail.gmail.com>
	<186FEE23-3C2E-4752-A97E-E06306D96D2C@exchange.princeton.edu>
Message-ID: <18834F5BEC10824891FB8B22AC821A5A01983D25@nucleus-srv01.Nucleus.local>

> According to tcpdump, traffic from varnish is coming from eth0
(incorrect IP)
> and traffic coming from nginx is coming from eth0:0 (correct IP).
Apache logs
> are also confirming the IP information.

That's possible, if both eth0 and eth0:0 are in different subnets. If
your backend for nginx is in the same subnet as eth0:0 but in a
different subnet as eth0, it'll go via the eth0:0 alias.

So my guess would be that your varnish backend is in a different subnet
than your eth0:0 so it takes the default route via eth0?

Regards,
Mattias



From david.birdsong at gmail.com  Tue Oct  4 08:11:07 2011
From: david.birdsong at gmail.com (David Birdsong)
Date: Tue, 4 Oct 2011 01:11:07 -0700
Subject: Varnish and multiple interfaces
In-Reply-To: <18834F5BEC10824891FB8B22AC821A5A01983D25@nucleus-srv01.Nucleus.local>
References: <4A5A0CA3-0441-4D19-AA91-D65F1D709A0B@exchange.princeton.edu>
	<CABJfL5iRvCNCXRj-HbDM6zRiJxBygYxVmfYeWMxRgzBxgJ1kYA@mail.gmail.com>
	<18307DC1-8BE9-46CF-AB37-4F627D73FAE3@exchange.princeton.edu>
	<CABJfL5jQabAWPhTD_rmOcNgOfBh3ytTWhA1Ct2eHrk3fXUaUiw@mail.gmail.com>
	<186FEE23-3C2E-4752-A97E-E06306D96D2C@exchange.princeton.edu>
	<18834F5BEC10824891FB8B22AC821A5A01983D25@nucleus-srv01.Nucleus.local>
Message-ID: <CABJfL5iTnZmBSyF38TyrvGd=yaUOm0hd_1ncsrDi_qffxWSzqA@mail.gmail.com>

Henry sort of implied that the for the purposes of the test, nginx and
varnish were configured similarly in the HTTP chain--that they
theoretically sat in the same ip space and should, therefore, connect
to the backend as from the same source ip.

Right Henry?

On Tue, Oct 4, 2011 at 1:01 AM, Mattias Geniar <mattias at nucleus.be> wrote:
>> According to tcpdump, traffic from varnish is coming from eth0
> (incorrect IP)
>> and traffic coming from nginx is coming from eth0:0 (correct IP).
> Apache logs
>> are also confirming the IP information.
>
> That's possible, if both eth0 and eth0:0 are in different subnets. If
> your backend for nginx is in the same subnet as eth0:0 but in a
> different subnet as eth0, it'll go via the eth0:0 alias.
>
> So my guess would be that your varnish backend is in a different subnet
> than your eth0:0 so it takes the default route via eth0?
>
> Regards,
> Mattias
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



From ljorg6 at gmail.com  Tue Oct  4 10:34:59 2011
From: ljorg6 at gmail.com (=?ISO-8859-1?Q?Lars_J=F8rgensen?=)
Date: Tue, 04 Oct 2011 12:34:59 +0200
Subject: varnishncsa crashing constantly
In-Reply-To: <CAK6GuyR=cqqzg1mOEKAe9U+WnLQ+iEVNnmKc2WjFV0cvZWtMdg@mail.gmail.com>
References: <4E89A8AC.6090402@gmail.com>
	<CAK6GuyR=cqqzg1mOEKAe9U+WnLQ+iEVNnmKc2WjFV0cvZWtMdg@mail.gmail.com>
Message-ID: <4E8AE153.4050103@gmail.com>

On 03-10-2011 15:13, Daniel Carrillo wrote:
> Hi, varnishncsa in 3.0.1 is still buggy, several bugs were fixed in
> trunk.

So I downloaded trunk and compiled it. Then I copied varnishncsa to 
/usr/local/bin, but running it says that it is version 3.0.1. Running it 
from the place it was compiled make it say that it is the trunk version.

I guess some libraries need to be replaced, too. But I don't want to 
interfere too much with a varnish installation that works fine except 
for varnishncsa.

Is there an easy way to test the trunk version of varnishncsa?


-- 
Lars



From daniel.carrillo at gmail.com  Tue Oct  4 10:49:43 2011
From: daniel.carrillo at gmail.com (Daniel Carrillo)
Date: Tue, 4 Oct 2011 12:49:43 +0200
Subject: varnishncsa crashing constantly
In-Reply-To: <4E8AE153.4050103@gmail.com>
References: <4E89A8AC.6090402@gmail.com>
	<CAK6GuyR=cqqzg1mOEKAe9U+WnLQ+iEVNnmKc2WjFV0cvZWtMdg@mail.gmail.com>
	<4E8AE153.4050103@gmail.com>
Message-ID: <CAK6GuyRQck0ctu=hqR6uo070Bn7fXKbL-vekjwOfacp1SOmG-A@mail.gmail.com>

2011/10/4 Lars J?rgensen <ljorg6 at gmail.com>:
> On 03-10-2011 15:13, Daniel Carrillo wrote:
>>
>> Hi, varnishncsa in 3.0.1 is still buggy, several bugs were fixed in
>> trunk.
>
> So I downloaded trunk and compiled it. Then I copied varnishncsa to
> /usr/local/bin, but running it says that it is version 3.0.1. Running it
> from the place it was compiled make it say that it is the trunk version.
>
> I guess some libraries need to be replaced, too. But I don't want to
> interfere too much with a varnish installation that works fine except for
> varnishncsa.
>
> Is there an easy way to test the trunk version of varnishncsa?

In your test environment, If you have compiled it, do an install instead copy.

My recomendation, if you are using some kind of packaging, patch the
src.rpm. deb o whatever and install it in your test environment, do
your tests and copy varnishncsa to production, of course the two
environments must be identical.

If you aren't sure, wait for 3.0.2 :)



From humansky at Princeton.EDU  Tue Oct  4 11:59:14 2011
From: humansky at Princeton.EDU (Henry M. Umansky)
Date: Tue, 4 Oct 2011 11:59:14 +0000
Subject: Varnish and multiple interfaces
In-Reply-To: <CABJfL5iTnZmBSyF38TyrvGd=yaUOm0hd_1ncsrDi_qffxWSzqA@mail.gmail.com>
References: <4A5A0CA3-0441-4D19-AA91-D65F1D709A0B@exchange.princeton.edu>
	<CABJfL5iRvCNCXRj-HbDM6zRiJxBygYxVmfYeWMxRgzBxgJ1kYA@mail.gmail.com>
	<18307DC1-8BE9-46CF-AB37-4F627D73FAE3@exchange.princeton.edu>
	<CABJfL5jQabAWPhTD_rmOcNgOfBh3ytTWhA1Ct2eHrk3fXUaUiw@mail.gmail.com>
	<186FEE23-3C2E-4752-A97E-E06306D96D2C@exchange.princeton.edu>
	<18834F5BEC10824891FB8B22AC821A5A01983D25@nucleus-srv01.Nucleus.local>
	<CABJfL5iTnZmBSyF38TyrvGd=yaUOm0hd_1ncsrDi_qffxWSzqA@mail.gmail.com>
Message-ID: <0DAFC6A0-0BCF-4DF1-AA2A-5C6C270E1889@exchange.princeton.edu>

That is correct, both applications are binding to the same IP from the same machine, just different port addresses.

Henry Umansky
Web Development Services
Princeton University
humansky at princeton.edu
609-258-1674

On Oct 4, 2011, at 4:11 AM, David Birdsong wrote:

> Henry sort of implied that the for the purposes of the test, nginx and
> varnish were configured similarly in the HTTP chain--that they
> theoretically sat in the same ip space and should, therefore, connect
> to the backend as from the same source ip.
> 
> Right Henry?
> 
> On Tue, Oct 4, 2011 at 1:01 AM, Mattias Geniar <mattias at nucleus.be> wrote:
>>> According to tcpdump, traffic from varnish is coming from eth0
>> (incorrect IP)
>>> and traffic coming from nginx is coming from eth0:0 (correct IP).
>> Apache logs
>>> are also confirming the IP information.
>> 
>> That's possible, if both eth0 and eth0:0 are in different subnets. If
>> your backend for nginx is in the same subnet as eth0:0 but in a
>> different subnet as eth0, it'll go via the eth0:0 alias.
>> 
>> So my guess would be that your varnish backend is in a different subnet
>> than your eth0:0 so it takes the default route via eth0?
>> 
>> Regards,
>> Mattias
>> 
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>> 




From humansky at Princeton.EDU  Tue Oct  4 12:10:04 2011
From: humansky at Princeton.EDU (Henry M. Umansky)
Date: Tue, 4 Oct 2011 12:10:04 +0000
Subject: Varnish and multiple interfaces
In-Reply-To: <68577.1317706187@critter.freebsd.dk>
References: <68577.1317706187@critter.freebsd.dk>
Message-ID: <1452E4F9-1332-43A0-946A-9D420C3DB5EC@exchange.princeton.edu>


On Oct 4, 2011, at 1:29 AM, Poul-Henning Kamp wrote:

The main we have not added a facility to bind backend connections
to a particular IP# is that it is a recipe for reachability problems
and kind of hard to imagine a legit case for wanting to do it in
the first place, so can I get you to describe (possibly in private
email) why you need to do this ?


Hello Poul,
Thank you for confirming this, I suspected as much, just wasn't absolutely sure. I'll be happy to explain my use case. I'm trying to create a redundant, highly-available architecture using varnish as the front-end caching server and load-balancer. I have two machines, webproxy01 and webproxy02 that shares a VIP (the aliased interface, eth0:0) via heartbeat, so only one machine will have the appropriate IP at any given time. That being said, I was hoping that all upstream servers will see the IP address of the VIP and not the IP address for the actual machine, that way each machine is interchangeable and abstracted from the stack. If I need to swap out one of the  front-ends with another front-end with a different IP address, I don't want to update all the "trusted gateway" scripts on the backend servers. As long as it is coming from the VIP, it is a trusted gateway. Let me know if this doesn't make sense or if I'm over-architecting the system...which I've been known to do :-)

Henry Umansky
Web Development Services
Princeton University
humansky at princeton.edu<mailto:humansky at princeton.edu>
609-258-1674
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111004/1d857fda/attachment.html>

From mit at stagename.com  Tue Oct  4 16:34:31 2011
From: mit at stagename.com (Mit Rowe)
Date: Tue, 4 Oct 2011 12:34:31 -0400
Subject: varnishncsa -F question
In-Reply-To: <CAO83RbVraPozWwOg=Z6oTeZHga2y7P0hvha1ZsmteaQjyXjngg@mail.gmail.com>
References: <CAFWPQt7oH6kfVsSuJt-=om1x6f-EZTtZ9iyd5FuH1XESX-OF=g@mail.gmail.com>
	<CAO83RbVraPozWwOg=Z6oTeZHga2y7P0hvha1ZsmteaQjyXjngg@mail.gmail.com>
Message-ID: <CAFWPQt7BtsdodaBNfdkPbAMtmoeMRcFCqQsMAbWNHBOhXbVROA@mail.gmail.com>

The trouble comes with me using a start-up script and quotes.

For example, /usr/local/etc/rc.d/varnishncsa

. /etc/rc.subr
name="varnishncsa"
rcvar=`set_rcvar`
command="/usr/local/bin/${name}"
load_rc_config ${name}
: ${varnishncsa_enable:="NO"}
: ${varnishncsa_pidfile:="/var/run/${name}.pid"}
: ${varnishncsa_file:="/var/log/${name}.log"}
: ${varnishncsa_flags:="-P ${varnishncsa_pidfile} -D -a -c -w
${varnishncsa_file}"}
pidfile=${varnishncsa_pidfile}
run_rc_command "$1"


if i stick a -F at the end of varnishncsa_flags that contains "" characters I
get "unterminated quoted string" errors. I've tried combinations of ', ",
\", \' and so forth and haven't yet found the right method.

Same thing happens with ubuntu's startup scripts.

Anyone have a suggestion to resolve this, at least starting with the default
format?

%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"





On Fri, Sep 2, 2011 at 2:26 PM, Jacques <whshub at gmail.com> wrote:

> I think I'm missing something.  We use the following without problem on
> 3.0.1
>
> varnishncsa  -F '%{X-Forwarded-For}i %l %u %t "%r" %s %b "%{Referer}i"
> "%{User-agent}i" "%{Cookie}i" "%{Varnish:hitmiss}x"' -X
> 'pingdom$|\.png$|\.ico$|\.jpg$|\.gif$'
>
>
>
> On Fri, Sep 2, 2011 at 9:22 AM, Mit Rowe <mit at stagename.com> wrote:
>
>> Hello!
>>
>> I am using varnish 3.0.1 and have a question about varnishncsalog
>>
>> I would like to spit out extra parameters in my logs, so i'm attempting to
>> use the -F command line flag.
>>
>> However, my need for spaces seems to be causing some problems.
>>
>> for example...
>>
>> -F %h %t
>>
>> (which i don't expect to work)
>>
>> prints
>>
>> 127.0.0.1<NEWLINE>
>>
>> (and that's it)
>>
>> -F '%h %t'
>>
>> prints
>>
>> '127.0.0.1<NEWLINE>
>>
>> -F "%h %t"
>>
>> prints
>>
>> "127.0.0.1<NEWLINE>
>>
>> and
>>
>> -F %h\ %t
>>
>> prints
>>
>> 127.0.0.1\<NEWLINE>
>>
>> Has anyone had any success specifying a log format containing spaces?
>>
>>
>>
>>
>> --
>> Will 'Mit' Rowe
>> Stagename*
>> *1-866-326-3098
>> mit at stagename.com <josh at stagename.com>
>> www.stagename.com
>> Twitter: @stagename
>>
>> *The information transmitted is intended only for the person or entity to
>> which it is addressed and may contain confidential and/or privileged
>> material. Any review, retransmission, dissemination or other use of this
>> information by persons or entities other than the intended recipient is
>> prohibited. If you received this transmission in error, please contact the
>> sender and delete all material contained herein from your computer.*
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>


-- 
Will 'Mit' Rowe
Stagename*
*1-866-326-3098
mit at stagename.com <josh at stagename.com>
www.stagename.com
Twitter: @stagename

*The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of this
information by persons or entities other than the intended recipient is
prohibited. If you received this transmission in error, please contact the
sender and delete all material contained herein from your computer.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111004/99f5d9c5/attachment.html>

From viapanda at gmail.com  Tue Oct  4 16:44:12 2011
From: viapanda at gmail.com (Olivier)
Date: Tue, 4 Oct 2011 18:44:12 +0200
Subject: varnishncsa -F question
In-Reply-To: <CAFWPQt7BtsdodaBNfdkPbAMtmoeMRcFCqQsMAbWNHBOhXbVROA@mail.gmail.com>
References: <CAFWPQt7oH6kfVsSuJt-=om1x6f-EZTtZ9iyd5FuH1XESX-OF=g@mail.gmail.com>
	<CAO83RbVraPozWwOg=Z6oTeZHga2y7P0hvha1ZsmteaQjyXjngg@mail.gmail.com>
	<CAFWPQt7BtsdodaBNfdkPbAMtmoeMRcFCqQsMAbWNHBOhXbVROA@mail.gmail.com>
Message-ID: <CAOBDNZvrCup9HBMZ+KmDW5KkRpwzj7B26gOnGcS5xEJORzM2bQ@mail.gmail.com>

You may want to read this:
http://stackoverflow.com/questions/1661193/start-stop-daemon-quoted-arguments-misinterpreted

As far as having Ubuntu script behave, you may try something in the line of:
DAEMON_OPTS="-a -c -w ${LOGFILE} -D -P $PIDFILE"
TRIX="-F %t %h \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"
%{ncsa-start}i %{ncsa-end}o"

start_varnishncsa() {
.........
       --chuid $USER --exec ${DAEMON} -- ${DAEMON_OPTS} "${TRIX}" \:



Best,

- Olivier


2011/10/4 Mit Rowe <mit at stagename.com>:
> The trouble comes with me using a start-up script and quotes.
> For example, /usr/local/etc/rc.d/varnishncsa
> . /etc/rc.subr
> name="varnishncsa"
> rcvar=`set_rcvar`
> command="/usr/local/bin/${name}"
> load_rc_config ${name}
> : ${varnishncsa_enable:="NO"}
> : ${varnishncsa_pidfile:="/var/run/${name}.pid"}
> : ${varnishncsa_file:="/var/log/${name}.log"}
> : ${varnishncsa_flags:="-P ${varnishncsa_pidfile} -D -a -c -w
> ${varnishncsa_file}"}
> pidfile=${varnishncsa_pidfile}
> run_rc_command "$1"
>
> if i stick a -F at the end of varnishncsa_flags that contains ""
> characters I get "unterminated quoted string" errors. I've tried
> combinations of ', ", \", \' and so forth and haven't yet found the right
> method.
> Same thing happens with ubuntu's startup scripts.
> Anyone have a suggestion to resolve this, at least starting with the
default
> format?
> %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"
>
>
>
>
> On Fri, Sep 2, 2011 at 2:26 PM, Jacques <whshub at gmail.com> wrote:
>>
>> I think I'm missing something.  We use the following without problem on
>> 3.0.1
>> varnishncsa  -F '%{X-Forwarded-For}i %l %u %t "%r" %s %b "%{Referer}i"
>> "%{User-agent}i" "%{Cookie}i" "%{Varnish:hitmiss}x"' -X
>> 'pingdom$|\.png$|\.ico$|\.jpg$|\.gif$'
>>
>>
>> On Fri, Sep 2, 2011 at 9:22 AM, Mit Rowe <mit at stagename.com> wrote:
>>>
>>> Hello!
>>> I am using varnish 3.0.1 and have a question about varnishncsalog
>>> I would like to spit out extra parameters in my logs, so i'm attempting
>>> to use the -F command line flag.
>>> However, my need for spaces seems to be causing some problems.
>>> for example...
>>> -F %h %t
>>> (which i don't expect to work)
>>> prints
>>> 127.0.0.1<NEWLINE>
>>> (and that's it)
>>> -F '%h %t'
>>> prints
>>> '127.0.0.1<NEWLINE>
>>> -F "%h %t"
>>> prints
>>> "127.0.0.1<NEWLINE>
>>>
>>> and
>>> -F %h\ %t
>>> prints
>>> 127.0.0.1\<NEWLINE>
>>> Has anyone had any success specifying a log format containing spaces?
>>>
>>>
>>>
>>> --
>>> Will 'Mit' Rowe
>>> Stagename
>>> 1-866-326-3098
>>> mit at stagename.com
>>> www.stagename.com
>>> Twitter: @stagename
>>>
>>> The information transmitted is intended only for the person or entity to
>>> which it is addressed and may contain confidential and/or privileged
>>> material. Any review, retransmission, dissemination or other use of this
>>> information by persons or entities other than the intended recipient is
>>> prohibited. If you received this transmission in error, please contact
the
>>> sender and delete all material contained herein from your computer.
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
>
> --
> Will 'Mit' Rowe
> Stagename
> 1-866-326-3098
> mit at stagename.com
> www.stagename.com
> Twitter: @stagename
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of this
> information by persons or entities other than the intended recipient is
> prohibited. If you received this transmission in error, please contact the
> sender and delete all material contained herein from your computer.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111004/1dceddfc/attachment.html>

From mit at stagename.com  Tue Oct  4 18:36:53 2011
From: mit at stagename.com (Mit Rowe)
Date: Tue, 4 Oct 2011 14:36:53 -0400
Subject: varnishncsa -F question
In-Reply-To: <CAOBDNZvrCup9HBMZ+KmDW5KkRpwzj7B26gOnGcS5xEJORzM2bQ@mail.gmail.com>
References: <CAFWPQt7oH6kfVsSuJt-=om1x6f-EZTtZ9iyd5FuH1XESX-OF=g@mail.gmail.com>
	<CAO83RbVraPozWwOg=Z6oTeZHga2y7P0hvha1ZsmteaQjyXjngg@mail.gmail.com>
	<CAFWPQt7BtsdodaBNfdkPbAMtmoeMRcFCqQsMAbWNHBOhXbVROA@mail.gmail.com>
	<CAOBDNZvrCup9HBMZ+KmDW5KkRpwzj7B26gOnGcS5xEJORzM2bQ@mail.gmail.com>
Message-ID: <CAFWPQt7yqszMPq0uLrL+smGFGkYXHepmV9VZzZRoZDbzez4vzw@mail.gmail.com>

Thanks Olivier,

For the FreeBSD startup script i'm now using...

#!/bin/sh
#
# $FreeBSD: ports/www/varnish/files/varnishncsa.in,v 1.4 2010/03/27 00:15:19
dougb Exp $
#

# PROVIDE: varnishncsa
# REQUIRE: DAEMON
# KEYWORD: shutdown

#
# Add the following line to /etc/rc.conf to enable varnishncsa:
#
# varnishncsa_enable="YES"
#
# Configuration variables and their default values:
#
# varnishncsa_pidfile - full path to the PID file.
# default: "/var/run/varnishncsa.pid"
#
# varnishncsa_file - full path to the log file.
# default: "/var/log/varnishncsa.log"
#
# varnishncsa_flags - command line arguments.
# default: "-D -P ${varnishncsa_pidfile} -a -c -w ${varnishncsa_file}"
#
# Add the following line to /etc/newsyslog.conf to rotate the log file
# once a day:
#
# /var/log/varnishncsa.log 640 7 * @T00 JB /var/run/varnishncsa.pid
#
# See varnishncsa(1) for a detailed overview of command-line options.
#

. /etc/rc.subr

name="varnishncsa"
rcvar=`set_rcvar`

command="/usr/local/bin/${name}"

# read configuration and set defaults
load_rc_config ${name}
: ${varnishncsa_enable:="NO"}
: ${varnishncsa_pidfile:="/var/run/${name}.pid"}
: ${varnishncsa_file:="/var/log/varnish/${name}.log"}
: ${varnishncsa_format:='%h %l %u %t \"%r\" %s %b \"%{Referer}i\"
\"%{User-agent}i\"'}
: ${varnishncsa_flags:="-P ${varnishncsa_pidfile} -D -a -c -w
${varnishncsa_file} -F \"${varnishncsa_format}\""}

pidfile=${varnishncsa_pidfile}
run_rc_command "$1"



On Tue, Oct 4, 2011 at 12:44 PM, Olivier <viapanda at gmail.com> wrote:

> You may want to read this:
>
> http://stackoverflow.com/questions/1661193/start-stop-daemon-quoted-arguments-misinterpreted
>
> As far as having Ubuntu script behave, you may try something in the line
> of:
> DAEMON_OPTS="-a -c -w ${LOGFILE} -D -P $PIDFILE"
> TRIX="-F %t %h \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"
> %{ncsa-start}i %{ncsa-end}o"
>
> start_varnishncsa() {
> .........
>        --chuid $USER --exec ${DAEMON} -- ${DAEMON_OPTS} "${TRIX}" \:
>
>
>
> Best,
>
> - Olivier
>
>
> 2011/10/4 Mit Rowe <mit at stagename.com>:
>
> > The trouble comes with me using a start-up script and quotes.
> > For example, /usr/local/etc/rc.d/varnishncsa
> > . /etc/rc.subr
> > name="varnishncsa"
> > rcvar=`set_rcvar`
> > command="/usr/local/bin/${name}"
> > load_rc_config ${name}
> > : ${varnishncsa_enable:="NO"}
> > : ${varnishncsa_pidfile:="/var/run/${name}.pid"}
> > : ${varnishncsa_file:="/var/log/${name}.log"}
> > : ${varnishncsa_flags:="-P ${varnishncsa_pidfile} -D -a -c -w
> > ${varnishncsa_file}"}
> > pidfile=${varnishncsa_pidfile}
> > run_rc_command "$1"
> >
> > if i stick a -F at the end of varnishncsa_flags that contains ""
> > characters I get "unterminated quoted string" errors. I've tried
> > combinations of ', ", \", \' and so forth and haven't yet found the right
> > method.
> > Same thing happens with ubuntu's startup scripts.
> > Anyone have a suggestion to resolve this, at least starting with the
> default
> > format?
> > %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"
> >
> >
> >
> >
> > On Fri, Sep 2, 2011 at 2:26 PM, Jacques <whshub at gmail.com> wrote:
> >>
> >> I think I'm missing something.  We use the following without problem on
> >> 3.0.1
> >> varnishncsa  -F '%{X-Forwarded-For}i %l %u %t "%r" %s %b "%{Referer}i"
> >> "%{User-agent}i" "%{Cookie}i" "%{Varnish:hitmiss}x"' -X
> >> 'pingdom$|\.png$|\.ico$|\.jpg$|\.gif$'
> >>
> >>
> >> On Fri, Sep 2, 2011 at 9:22 AM, Mit Rowe <mit at stagename.com> wrote:
> >>>
> >>> Hello!
> >>> I am using varnish 3.0.1 and have a question about varnishncsalog
> >>> I would like to spit out extra parameters in my logs, so i'm attempting
> >>> to use the -F command line flag.
> >>> However, my need for spaces seems to be causing some problems.
> >>> for example...
> >>> -F %h %t
> >>> (which i don't expect to work)
> >>> prints
> >>> 127.0.0.1<NEWLINE>
> >>> (and that's it)
> >>> -F '%h %t'
> >>> prints
> >>> '127.0.0.1<NEWLINE>
> >>> -F "%h %t"
> >>> prints
> >>> "127.0.0.1<NEWLINE>
> >>>
> >>> and
> >>> -F %h\ %t
> >>> prints
> >>> 127.0.0.1\<NEWLINE>
> >>> Has anyone had any success specifying a log format containing spaces?
> >>>
> >>>
> >>>
> >>> --
> >>> Will 'Mit' Rowe
> >>> Stagename
> >>> 1-866-326-3098
> >>> mit at stagename.com
> >>> www.stagename.com
> >>> Twitter: @stagename
> >>>
> >>> The information transmitted is intended only for the person or entity
> to
> >>> which it is addressed and may contain confidential and/or privileged
> >>> material. Any review, retransmission, dissemination or other use of
> this
> >>> information by persons or entities other than the intended recipient is
> >>> prohibited. If you received this transmission in error, please contact
> the
> >>> sender and delete all material contained herein from your computer.
> >>>
> >>> _______________________________________________
> >>> varnish-misc mailing list
> >>> varnish-misc at varnish-cache.org
> >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >>
> >
> >
> >
> > --
> > Will 'Mit' Rowe
> > Stagename
> > 1-866-326-3098
> > mit at stagename.com
> > www.stagename.com
> > Twitter: @stagename
> >
> > The information transmitted is intended only for the person or entity to
> > which it is addressed and may contain confidential and/or privileged
> > material. Any review, retransmission, dissemination or other use of this
> > information by persons or entities other than the intended recipient is
> > prohibited. If you received this transmission in error, please contact
> the
> > sender and delete all material contained herein from your computer.
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >
>
>


-- 
Will 'Mit' Rowe
Stagename*
*1-866-326-3098
mit at stagename.com <josh at stagename.com>
www.stagename.com
Twitter: @stagename

*The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of this
information by persons or entities other than the intended recipient is
prohibited. If you received this transmission in error, please contact the
sender and delete all material contained herein from your computer.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111004/ee659195/attachment.html>

From brad at teambrad.net  Tue Oct  4 22:26:49 2011
From: brad at teambrad.net (Bradley Falzon)
Date: Wed, 5 Oct 2011 08:56:49 +1030
Subject: Varnish and multiple interfaces
In-Reply-To: <1452E4F9-1332-43A0-946A-9D420C3DB5EC@exchange.princeton.edu>
References: <68577.1317706187@critter.freebsd.dk>
	<1452E4F9-1332-43A0-946A-9D420C3DB5EC@exchange.princeton.edu>
Message-ID: <CABESvY1pWuqaH2_2NTR_3UjLA75+pztumzGyF2PoO0gbaLrD3g@mail.gmail.com>

On Tue, Oct 4, 2011 at 10:40 PM, Henry M. Umansky <humansky at princeton.edu>wrote:

> Let me know if this doesn't make sense or if I'm over-architecting the
> system...which I've been known to do :-)
>

You *could* add another route, tell the kernel to use the eth0:0 IP address
for specific routes:

[root at ellahgrace ~]# ip route
192.168.2.0/24 dev eth1  proto kernel  scope link  src 192.168.2.100

tcpdump of ping request (notice the source IP is 192.168.2.100)
08:36:15.945374 IP 192.168.2.100 > 192.168.2.254: ICMP echo request, id
45145, seq 1, length 64
08:36:15.946212 IP 192.168.2.254 > 192.168.2.100: ICMP echo reply, id 45145,
seq 1, length 64

[root at ellahgrace ~]# ip route add 192.168.2.254/32 dev eth1 proto static
scope link src 192.168.2.101
[root at ellahgrace ~]# ip route
192.168.2.254 dev eth1  proto static  scope link  src 192.168.2.101
192.168.2.0/24 dev eth1  proto kernel  scope link  src 192.168.2.100

Now all packets to 192.168.2.254 should come with the src 192.168.2.101, as
you can see here:

08:44:57.841931 IP 192.168.2.101 > 192.168.2.254: ICMP echo request, id
14171, seq 3, length 64
08:44:57.842533 IP 192.168.2.254 > 192.168.2.101: ICMP echo reply, id 14171,
seq 3, length 64

When the interface belonging to 192.168.2.101 (in my case eth1:1 - yours is
probably eth0:0) goes down, so does the route and future packets will just
come from any available interface. The static route just needs to be added
when your aliased interface gets added, the kernel will take it away when it
is not longer available.

IMHO, I think this will just cause more problems than what it's worth. It's
common, in my experience, to see VIP / Load Balanced setups come from
different interfaces.

IMHO, as for your trusted proxy configurations, just match a subnet, like
192.168.1.0/27 or similar.

-- 
Bradley Falzon
brad at teambrad.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111005/3ec48044/attachment.html>

From ljorg6 at gmail.com  Wed Oct  5 07:21:12 2011
From: ljorg6 at gmail.com (=?ISO-8859-1?Q?Lars_J=F8rgensen?=)
Date: Wed, 05 Oct 2011 09:21:12 +0200
Subject: varnishncsa crashing constantly
In-Reply-To: <CAK6GuyRQck0ctu=hqR6uo070Bn7fXKbL-vekjwOfacp1SOmG-A@mail.gmail.com>
References: <4E89A8AC.6090402@gmail.com>
	<CAK6GuyR=cqqzg1mOEKAe9U+WnLQ+iEVNnmKc2WjFV0cvZWtMdg@mail.gmail.com>
	<4E8AE153.4050103@gmail.com>
	<CAK6GuyRQck0ctu=hqR6uo070Bn7fXKbL-vekjwOfacp1SOmG-A@mail.gmail.com>
Message-ID: <4E8C0568.8060203@gmail.com>

On 04-10-2011 12:49, Daniel Carrillo wrote:
> In your test environment, If you have compiled it, do an install instead copy.

Problem is, my test environment doesn't receive real world traffic thus 
varnishncsa doesn't crash.

> If you aren't sure, wait for 3.0.2 :)

I think I will try and install a full trunk version. A roll-back is 
pretty easy if things behave strangely.


-- 
Lars



From insyte at gmail.com  Wed Oct  5 07:35:35 2011
From: insyte at gmail.com (Ben Beuchler)
Date: Wed, 5 Oct 2011 02:35:35 -0500
Subject: Automatically removing old configs
Message-ID: <CAC29JhXg9ATqZip=Mr_Ca=qfrunUuotg3BkB5yjk_kme-MqNOQ@mail.gmail.com>

Now that I've automated the process of distributing, loading and using
new VCLs, I find that I'm accumulating many inactive configs with 0
references.  I could extend my script (makefile, actually) to discard
older configs as new ones are added, but I thought I'd seen a way to
manage this from within varnishd.  Is there such a thing?

Thanks.

-Ben



From daniel.carrillo at gmail.com  Wed Oct  5 08:38:23 2011
From: daniel.carrillo at gmail.com (Daniel Carrillo)
Date: Wed, 5 Oct 2011 10:38:23 +0200
Subject: varnishncsa crashing constantly
In-Reply-To: <4E8C0568.8060203@gmail.com>
References: <4E89A8AC.6090402@gmail.com>
	<CAK6GuyR=cqqzg1mOEKAe9U+WnLQ+iEVNnmKc2WjFV0cvZWtMdg@mail.gmail.com>
	<4E8AE153.4050103@gmail.com>
	<CAK6GuyRQck0ctu=hqR6uo070Bn7fXKbL-vekjwOfacp1SOmG-A@mail.gmail.com>
	<4E8C0568.8060203@gmail.com>
Message-ID: <CAK6GuySFSmA0e0q7S56bRAcfuR59Pb7ode_9S_cmS6bNvNzVXg@mail.gmail.com>

2011/10/5 Lars J?rgensen <ljorg6 at gmail.com>:
> On 04-10-2011 12:49, Daniel Carrillo wrote:
>>
>> In your test environment, If you have compiled it, do an install instead
>> copy.
>
> Problem is, my test environment doesn't receive real world traffic thus
> varnishncsa doesn't crash.

If your crash is because ticket 1006, you can trigger it:

telnet localhost 80
GET / HTTP/1.1
ImAJunkHeader
(enter)
(enter)



From tphuocthien at gmail.com  Wed Oct  5 18:12:43 2011
From: tphuocthien at gmail.com (=?UTF-8?B?VGhp4buHbiBUcuG6p24gUGjGsOG7m2M=?=)
Date: Thu, 6 Oct 2011 01:12:43 +0700
Subject: How to run Varnish as Windows Service
In-Reply-To: <CAA67PhQynx05zP1L1gVoCf5QgC91__SNqz3LLqEo45xhtFvGDg@mail.gmail.com>
References: <CAA67PhQynx05zP1L1gVoCf5QgC91__SNqz3LLqEo45xhtFvGDg@mail.gmail.com>
Message-ID: <CAA67PhTwfPq0J7NSRrTo8n8KxvV6Phhyn1N5guA5K8NJO3HHDg@mail.gmail.com>

Hi everybody,

I uses Varnish Web Cache in Windows Server 2008. Could anybody tell me: how
to run Varnish as Windows Service ?
Because this is a server,it should run automatically when Windows is
startup.

I research multi way to do. Download some wrapper software and intergate
Varnish into them. But not success.

Thanks a lot,
Denny Jimmy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111006/c2962218/attachment.html>

From paulalexander.polo at gmail.com  Wed Oct  5 18:50:04 2011
From: paulalexander.polo at gmail.com (Paul-Alexander Polo)
Date: Wed, 5 Oct 2011 14:50:04 -0400
Subject: Varnish vs aiCache
Message-ID: <CAAswRkg34kR1cQF5NAscC9YDemnFH8BfpAr+F=YwNt8ewkXiSg@mail.gmail.com>

Hello,



it is very difficult to find information on how Varnish compares to aiCache.

The only information we could find is that aiCache support SSL, POST caching
and would "crush" Varnish when the origin is not local ??

Also, aiCache is not open-source ;(

Is anybody have experience with aiCache and Varnish and could make a good
and honest comparaison ?

What are the best and cons of both ?

Would you use Varnish to setup your own CDN (for example on Amazon)  when
the origin is a few thousand of kilometers away?



Thank you everyone,



Paul-Alexander Polo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111005/74e59d33/attachment.html>

From ljorg6 at gmail.com  Thu Oct  6 10:56:32 2011
From: ljorg6 at gmail.com (=?ISO-8859-1?Q?Lars_J=F8rgensen?=)
Date: Thu, 06 Oct 2011 12:56:32 +0200
Subject: varnishncsa crashing constantly
In-Reply-To: <4E8C0568.8060203@gmail.com>
References: <4E89A8AC.6090402@gmail.com>
	<CAK6GuyR=cqqzg1mOEKAe9U+WnLQ+iEVNnmKc2WjFV0cvZWtMdg@mail.gmail.com>
	<4E8AE153.4050103@gmail.com>
	<CAK6GuyRQck0ctu=hqR6uo070Bn7fXKbL-vekjwOfacp1SOmG-A@mail.gmail.com>
	<4E8C0568.8060203@gmail.com>
Message-ID: <4E8D8960.90309@gmail.com>

On 05-10-2011 09:21, Lars J?rgensen wrote:
> I think I will try and install a full trunk version. A roll-back is
> pretty easy if things behave strangely.

Okay, tried installing trunk. Varnishncsa still crashes (8 times in the 
last hour or so). What do I need to do to provide useful information for 
a bug report?


-- 
Lars



From daniel.carrillo at gmail.com  Thu Oct  6 11:05:57 2011
From: daniel.carrillo at gmail.com (Daniel Carrillo)
Date: Thu, 6 Oct 2011 13:05:57 +0200
Subject: varnishncsa crashing constantly
In-Reply-To: <4E8D8960.90309@gmail.com>
References: <4E89A8AC.6090402@gmail.com>
	<CAK6GuyR=cqqzg1mOEKAe9U+WnLQ+iEVNnmKc2WjFV0cvZWtMdg@mail.gmail.com>
	<4E8AE153.4050103@gmail.com>
	<CAK6GuyRQck0ctu=hqR6uo070Bn7fXKbL-vekjwOfacp1SOmG-A@mail.gmail.com>
	<4E8C0568.8060203@gmail.com> <4E8D8960.90309@gmail.com>
Message-ID: <CAK6GuySO9TMsLnLfZSfFspKG-+BqJbW80W4BCpSU2afYpomubg@mail.gmail.com>

2011/10/6 Lars J?rgensen <ljorg6 at gmail.com>:
> On 05-10-2011 09:21, Lars J?rgensen wrote:
>>
>> I think I will try and install a full trunk version. A roll-back is
>> pretty easy if things behave strangely.
>
> Okay, tried installing trunk. Varnishncsa still crashes (8 times in the last
> hour or so). What do I need to do to provide useful information for a bug
> report?

Check this https://www.varnish-cache.org/trac/wiki/DebuggingVarnish

You could create a new ticket if you are sure that your problem is not
already in the bug track.

Hope you could find de problem.



From janneolausson at gmail.com  Thu Oct  6 12:14:54 2011
From: janneolausson at gmail.com (Jan Olausson)
Date: Thu, 6 Oct 2011 14:14:54 +0200
Subject: Varnish closes conenction to haproxy
Message-ID: <E2BBA08D-1AE2-4E83-987D-7CD4209326F6@gmail.com>

Hi!

I have a problem with varnish closes the connection to haproxy infront of it.
Haproxy reports a 502 error after exactly 60 seconds after it begun handling the request.

If I skip both haproxy and varnish and let nginx handle the request directly, everything works although it takes 4-5 minutes to complete the request due to the complexity of the work that has to be done.

Anyone here have a suggestion what I shall try?
Varnish version is: varnish-3.0.0 revision 3bd5997

Best regards
Jan Olausson
Sweden





From stewsnooze at gmail.com  Thu Oct  6 12:25:37 2011
From: stewsnooze at gmail.com (Stewart Robinson)
Date: Thu, 6 Oct 2011 13:25:37 +0100
Subject: Varnish closes conenction to haproxy
In-Reply-To: <E2BBA08D-1AE2-4E83-987D-7CD4209326F6@gmail.com>
References: <E2BBA08D-1AE2-4E83-987D-7CD4209326F6@gmail.com>
Message-ID: <CAES+E52CVh3Qf2ZbU31bKrODOTpHbFGMHBDcbHrY_ayxYbthxA@mail.gmail.com>

Hi,

Have you specified long timeouts in your backend declaration?

backend www {
  .host = "www.example.com";
  .port = "http";
  .connect_timeout = 1s;
  .first_byte_timeout = 5s;
  .between_bytes_timeout = 2s;
}


Stew

On 6 October 2011 13:14, Jan Olausson <janneolausson at gmail.com> wrote:

> Hi!
>
> I have a problem with varnish closes the connection to haproxy infront of
> it.
> Haproxy reports a 502 error after exactly 60 seconds after it begun
> handling the request.
>
> If I skip both haproxy and varnish and let nginx handle the request
> directly, everything works although it takes 4-5 minutes to complete the
> request due to the complexity of the work that has to be done.
>
> Anyone here have a suggestion what I shall try?
> Varnish version is: varnish-3.0.0 revision 3bd5997
>
> Best regards
> Jan Olausson
> Sweden
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111006/cfe04416/attachment.html>

From janneolausson at gmail.com  Thu Oct  6 12:36:15 2011
From: janneolausson at gmail.com (Jan Olausson)
Date: Thu, 6 Oct 2011 14:36:15 +0200
Subject: Varnish closes conenction to haproxy
In-Reply-To: <CAES+E52CVh3Qf2ZbU31bKrODOTpHbFGMHBDcbHrY_ayxYbthxA@mail.gmail.com>
References: <E2BBA08D-1AE2-4E83-987D-7CD4209326F6@gmail.com>
	<CAES+E52CVh3Qf2ZbU31bKrODOTpHbFGMHBDcbHrY_ayxYbthxA@mail.gmail.com>
Message-ID: <652562DB-727D-4769-AB96-F7403124F799@gmail.com>

Hi,

Silly me not to mention it but yes. The values I have are:

    .connect_timeout = 1s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 5s;

Best regards
Jan


6 okt 2011 kl. 14.25 skrev Stewart Robinson:

> Hi,
> 
> Have you specified long timeouts in your backend declaration?
> 
> backend www {
>   .host = "www.example.com";
>   .port = "http";
>   .connect_timeout = 1s;
>   .first_byte_timeout = 5s;
>   .between_bytes_timeout = 2s;
> }
> 
> Stew
> 
> On 6 October 2011 13:14, Jan Olausson <janneolausson at gmail.com> wrote:
> Hi!
> 
> I have a problem with varnish closes the connection to haproxy infront of it.
> Haproxy reports a 502 error after exactly 60 seconds after it begun handling the request.
> 
> If I skip both haproxy and varnish and let nginx handle the request directly, everything works although it takes 4-5 minutes to complete the request due to the complexity of the work that has to be done.
> 
> Anyone here have a suggestion what I shall try?
> Varnish version is: varnish-3.0.0 revision 3bd5997
> 
> Best regards
> Jan Olausson
> Sweden
> 
> 
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111006/04c19b99/attachment.html>

From listas at kurtkraut.net  Thu Oct  6 21:59:45 2011
From: listas at kurtkraut.net (Kurt Kraut)
Date: Thu, 6 Oct 2011 18:59:45 -0300
Subject: Varnish drops connections and make several HTTP 206
Message-ID: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>

Hi,


I've already reported this on Varnish 3.0 and I waited to test it on Varnish
3.0.1 and the behaviour is the same. The majority of HTTP GET of files
larger than 1mb are constantly interrupted and restarted with HTTP 206. Here
is a sample collected from varnishncsa:

177.26.0.252 - - [06/Oct/2011:18:40:50 -0300] "GET
http://video.kurtkraut.net/static/user/16/16855/0ae200b1a77438c17b1624d3987dc782.mp4HTTP/1.1"
206 2 "-" "AppleCoreMedia/1.0.0.8H7 (iPhone; U; CPU OS 4_3_2 like
Mac OS X; pt_br)"
177.26.0.252 - - [06/Oct/2011:18:40:50 -0300] "GET
http://video.kurtkraut.net/static/user/16/16855/0ae200b1a77438c17b1624d3987dc782.mp4HTTP/1.1"
206 2 "-" "AppleCoreMedia/1.0.0.8H7 (iPhone; U; CPU OS 4_3_2 like
Mac OS X; pt_br)"
177.26.0.252 - - [06/Oct/2011:18:40:50 -0300] "GET
http://video.kurtkraut.net/static/user/16/16855/0ae200b1a77438c17b1624d3987dc782.mp4HTTP/1.1"
206 2 "-" "AppleCoreMedia/1.0.0.8H7 (iPhone; U; CPU OS 4_3_2 like
Mac OS X; pt_br)"
177.26.0.252 - - [06/Oct/2011:18:40:50 -0300] "GET
http://video.kurtkraut.net/static/user/16/16855/0ae200b1a77438c17b1624d3987dc782.mp4HTTP/1.1"
206 - "-" "AppleCoreMedia/1.0.0.8H7 (iPhone; U; CPU OS 4_3_2 like
Mac OS X; pt_br)"

And this goes on for the next 14 lines, until all data was transfered. Full
log except in http://pastebin.com/7r6gQsia

This was an iPhone watching a MP4 video, but this also happens with curl,
wget, Firefox, anything I've tested. If I point the FQDN straight to the
backend it doesn't happen. Also, different backends (Apache, nginx were
tested) the result is the same. I have three varnish 3.0.1 servers on
different servers, on different datacenters, with different CentOS installs
and they all behave the same.

I belive this is a bug, so my question is:

1) Is it a known issue?
2) If not, what further details would be helpful to make a bug report?
3) Does anyone suggest a workaround for this?


Thanks in advance,

Kurt Kraut
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111006/38a49af4/attachment.html>

From mit at stagename.com  Fri Oct  7 00:35:59 2011
From: mit at stagename.com (Mit Rowe)
Date: Thu, 6 Oct 2011 20:35:59 -0400
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
Message-ID: <CAFWPQt4_GQLaBBWNSMn972Zp-jxQOs-t3FucSQU_eWSPF0c2+A@mail.gmail.com>

There is a _possibility_ that this is legitimate traffic and not a varnish
issue.

"206" responses happen in requests for video content when a client (like a
media player) specifically requests that a "chunk" be delivered by sending a
"Range:" header; the server sends a 206 status and several extra header
fields in its response to acknowledge its ability to recognize the request,
ability and willingness to send just a piece of the file rather than the
whole file, and to indicate that the content it is giving represents only a
partial piece of the file requested.

For example, a media player is playing a video that exists at a URL and the
user fast-forwards to a certain point on the timeline, the player can stop
the current play, requests a chunk it calculates is at that position in the
timeline, and resume from that point forward. It could also theoretically
use the same mechanism to populate it's internal cache.

There are references here:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html (search for "206
Partial Content"), here (
http://en.wikipedia.org/wiki/Http_status_codes#2xx_Success) and here:
http://labs.apache.org/webarch/http/draft-fielding-http/p5-range.html#status.206

Hope this helps,
-Mit



 On Thu, Oct 6, 2011 at 5:59 PM, Kurt Kraut <listas at kurtkraut.net> wrote:

> Hi,
>
>
> I've already reported this on Varnish 3.0 and I waited to test it on
> Varnish 3.0.1 and the behaviour is the same. The majority of HTTP GET of
> files larger than 1mb are constantly interrupted and restarted with HTTP
> 206. Here is a sample collected from varnishncsa:
>
> 177.26.0.252 - - [06/Oct/2011:18:40:50 -0300] "GET
> http://video.kurtkraut.net/static/user/16/16855/0ae200b1a77438c17b1624d3987dc782.mp4HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.8H7 (iPhone; U; CPU OS 4_3_2 like
> Mac OS X; pt_br)"
> 177.26.0.252 - - [06/Oct/2011:18:40:50 -0300] "GET
> http://video.kurtkraut.net/static/user/16/16855/0ae200b1a77438c17b1624d3987dc782.mp4HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.8H7 (iPhone; U; CPU OS 4_3_2 like
> Mac OS X; pt_br)"
> 177.26.0.252 - - [06/Oct/2011:18:40:50 -0300] "GET
> http://video.kurtkraut.net/static/user/16/16855/0ae200b1a77438c17b1624d3987dc782.mp4HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.8H7 (iPhone; U; CPU OS 4_3_2 like
> Mac OS X; pt_br)"
> 177.26.0.252 - - [06/Oct/2011:18:40:50 -0300] "GET
> http://video.kurtkraut.net/static/user/16/16855/0ae200b1a77438c17b1624d3987dc782.mp4HTTP/1.1" 206 - "-" "AppleCoreMedia/1.0.0.8H7 (iPhone; U; CPU OS 4_3_2 like
> Mac OS X; pt_br)"
>
> And this goes on for the next 14 lines, until all data was transfered. Full
> log except in http://pastebin.com/7r6gQsia
>
> This was an iPhone watching a MP4 video, but this also happens with curl,
> wget, Firefox, anything I've tested. If I point the FQDN straight to the
> backend it doesn't happen. Also, different backends (Apache, nginx were
> tested) the result is the same. I have three varnish 3.0.1 servers on
> different servers, on different datacenters, with different CentOS installs
> and they all behave the same.
>
> I belive this is a bug, so my question is:
>
> 1) Is it a known issue?
> 2) If not, what further details would be helpful to make a bug report?
> 3) Does anyone suggest a workaround for this?
>
>
> Thanks in advance,
>
> Kurt Kraut
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



-- 
Will 'Mit' Rowe
Stagename*
*1-866-326-3098
mit at stagename.com <josh at stagename.com>
www.stagename.com
Twitter: @stagename
Facebook: facebook.com/stagename
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111006/073419d2/attachment.html>

From tfheen at varnish-software.com  Fri Oct  7 06:54:30 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Fri, 07 Oct 2011 08:54:30 +0200
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	(Kurt Kraut's message of "Thu, 6 Oct 2011 18:59:45 -0300")
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
Message-ID: <87vcs12swp.fsf@qurzaw.varnish-software.com>

]] Kurt Kraut 

| I've already reported this on Varnish 3.0 and I waited to test it on Varnish
| 3.0.1 and the behaviour is the same. The majority of HTTP GET of files
| larger than 1mb are constantly interrupted and restarted with HTTP 206. Here
| is a sample collected from varnishncsa:

206 is ?Partial Content? which we send if we get a request with Range
headers.

| This was an iPhone watching a MP4 video, but this also happens with curl,
| wget, Firefox, anything I've tested.

Can you please provide varnishlog + command line from a run with curl
that shows the problem?


[...]

| 3) Does anyone suggest a workaround for this?

sub vcl_recv {
  unset req.http.Range;
}

that'll increase the amount of bandwidth clients are consuming, though.

Cheers,
-- 
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64



From elatino17 at hotmail.com  Thu Oct  6 21:55:44 2011
From: elatino17 at hotmail.com (NitO MaNia)
Date: Thu, 6 Oct 2011 21:55:44 +0000
Subject: question about varnish
Message-ID: <SNT143-W159E3F270B85F3122EDF42BDF90@phx.gbl>


is there any way to use Varnish in Windows 2008 server 64  bit? i already install Cygwin but it doesnt work with this version of windows, so i cant setup varnish , tks. 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111006/67a1a65f/attachment.html>

From tfheen at varnish-software.com  Fri Oct  7 07:15:49 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Fri, 07 Oct 2011 09:15:49 +0200
Subject: question about varnish
In-Reply-To: <SNT143-W159E3F270B85F3122EDF42BDF90@phx.gbl> (NitO MaNia's
	message of "Thu, 6 Oct 2011 21:55:44 +0000")
References: <SNT143-W159E3F270B85F3122EDF42BDF90@phx.gbl>
Message-ID: <87mxdd2rx6.fsf@qurzaw.varnish-software.com>

]] NitO MaNia 

| is there any way to use Varnish in Windows 2008 server 64  bit? i
| already install Cygwin but it doesnt work with this version of
| windows, so i cant setup varnish , tks.

No, we don't support Windows.

-- 
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64



From listas at kurtkraut.net  Fri Oct  7 18:17:48 2011
From: listas at kurtkraut.net (Kurt Kraut)
Date: Fri, 7 Oct 2011 15:17:48 -0300
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <87vcs12swp.fsf@qurzaw.varnish-software.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
Message-ID: <CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>

Hi,

What I'm reporting is not true HTTP 206 requests for resuming downloads or
genuine HTTP clients requesting for these. The logs I provide below of wget
and cURL shows that Varnish is deliberately dropping connections, forcing
the HTTP client to attempt to resume the download. So the HTTP is a
consequence of a issue generated by Varnish. This started to affect me on
version 3.0 and currently in 3.0.1

Bellow is a URL containing a full output log for the command wget --verbose
http://varnish.kurtkraut.net/1GBtest.zip

Pastebin: http://pastebin.com/hfYRhKq7

Also, the output of cURL: http://pastebin.com/CuMr3mbp

You may reproduce the issue trying to download
http://varnish.kurtkraut.net/1GBtest.zip - I will leave this URL online for
the next weeks.

What do you think? Is it a bug?

Thanks,


Kurt Kraut

2011/10/7 Tollef Fog Heen <tfheen at varnish-software.com>

> ]] Kurt Kraut
>
> | I've already reported this on Varnish 3.0 and I waited to test it on
> Varnish
> | 3.0.1 and the behaviour is the same. The majority of HTTP GET of files
> | larger than 1mb are constantly interrupted and restarted with HTTP 206.
> Here
> | is a sample collected from varnishncsa:
>
> 206 is ?Partial Content? which we send if we get a request with Range
> headers.
>
> | This was an iPhone watching a MP4 video, but this also happens with curl,
> | wget, Firefox, anything I've tested.
>
> Can you please provide varnishlog + command line from a run with curl
> that shows the problem?
>
>
> [...]
>
> | 3) Does anyone suggest a workaround for this?
>
> sub vcl_recv {
>  unset req.http.Range;
> }
>
> that'll increase the amount of bandwidth clients are consuming, though.
>
> Cheers,
> --
> Tollef Fog Heen
> Varnish Software
> t: +47 21 98 92 64
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111007/61bd0d47/attachment.html>

From perbu at varnish-software.com  Fri Oct  7 18:54:48 2011
From: perbu at varnish-software.com (Per Buer)
Date: Fri, 7 Oct 2011 20:54:48 +0200
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
Message-ID: <CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>

Hi Kurt,


On Fri, Oct 7, 2011 at 8:17 PM, Kurt Kraut <listas at kurtkraut.net> wrote:

> Hi,
> (..)
>
> What do you think? Is it a bug?
>

Okey. This is one of the reasons top-posting is bad. Tollef asked you for
varnishlog entry, which is the only sane input we can have in order to
actually tell you what is going on  without it being total speculation.

You ignored that. Why? If you need help actually getting sane  varnishlog
output please tell so.

I'm _guessing_ you are running into timeouts. But without a log entry,
that's only a guess.

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111007/6344927a/attachment.html>

From kacperw at gmail.com  Fri Oct  7 21:34:48 2011
From: kacperw at gmail.com (Kacper Wysocki)
Date: Fri, 7 Oct 2011 17:34:48 -0400
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
	<CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
Message-ID: <CABaBnj5nxc-fASeSKWhT_vbZ4=C0AhvbDUpb9ZdtCyx_6HnhUQ@mail.gmail.com>

On Fri, Oct 7, 2011 at 2:54 PM, Per Buer <perbu at varnish-software.com> wrote:
> On Fri, Oct 7, 2011 at 8:17 PM, Kurt Kraut <listas at kurtkraut.net> wrote:
>> What do you think? Is it a bug?
>
> Okey. This is one of the reasons top-posting is bad. Tollef asked you for varnishlog entry, which is the only sane input we can have in order to actually tell you what is going on ?without it being total speculation.

I also noticed similar behavior with large files with Varnish 3.0.1.
Today, when I tried to do large files.. Here's a wget run.

2011-10-07 17:18:59 (442 KB/s) - Connection closed at byte 28062437. Retrying.
2011-10-07 17:20:03 (486 KB/s) - Connection closed at byte 59268331. Retrying.
2011-10-07 17:21:07 (500 KB/s) - Connection closed at byte 90762224. Retrying.

varnishlog craps out with the following message:
   11 Debug        c Write error, retval = 28062720, len = 107637870,
errno = Success

Indeed, wget does do Range: requests when the connection dies, but a
different machine (OSX with curl and firefox) would end the download
thinking the file was done.

> I'm _guessing_ you are running into timeouts. But without a log entry, that's only a guess.

Could be timeouts, but I'm not sure why they would happen in this case.

Here's the pure unbridled unadulterated vlog
http://u.delta9.pl/k/drop/v-drops.vlog

oh on a side note it seems maybe the 3.0 varnishlog format isn't
compatible with the 2.x one?

--
http://comotion.delta9.pl
http://u.delta9.pl
http://kacper.doesntexist.org
Too much order is its own chaos.
Employ no technique to gain supreme enlightment.



From perbu at varnish-software.com  Sat Oct  8 08:23:59 2011
From: perbu at varnish-software.com (Per Buer)
Date: Sat, 8 Oct 2011 10:23:59 +0200
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CABaBnj5nxc-fASeSKWhT_vbZ4=C0AhvbDUpb9ZdtCyx_6HnhUQ@mail.gmail.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
	<CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
	<CABaBnj5nxc-fASeSKWhT_vbZ4=C0AhvbDUpb9ZdtCyx_6HnhUQ@mail.gmail.com>
Message-ID: <CAOXZevAc6zU1JDkRD3=s=cRFg+=v-FU+jczPBGoUngBci+AjRA@mail.gmail.com>

Hi Kacper,

On Fri, Oct 7, 2011 at 11:34 PM, Kacper Wysocki <kacperw at gmail.com> wrote:

>
> I also noticed similar behavior with large files with Varnish 3.0.1.
> Today, when I tried to do large files.. Here's a wget run.
>
> 2011-10-07 17:18:59 (442 KB/s) - Connection closed at byte 28062437.
> Retrying.
> 2011-10-07 17:20:03 (486 KB/s) - Connection closed at byte 59268331.
> Retrying.
> 2011-10-07 17:21:07 (500 KB/s) - Connection closed at byte 90762224.
> Retrying.
>


Notice that there is 60 seconds between each fetch + some time to reconnect.
60s is the default for send_timeout since 3.0. It used to be 600 seconds.


> > I'm _guessing_ you are running into timeouts. But without a log entry,
> that's only a guess.
>
> Could be timeouts, but I'm not sure why they would happen in this case.
>
> Here's the pure unbridled unadulterated vlog
> http://u.delta9.pl/k/drop/v-drops.vlog


404.

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111008/5124e28e/attachment.html>

From jdzstz at gmail.com  Sat Oct  8 10:59:31 2011
From: jdzstz at gmail.com (jdzstz - gmail dot com)
Date: Sat, 8 Oct 2011 12:59:31 +0200
Subject: question about varnish
In-Reply-To: <SNT143-W159E3F270B85F3122EDF42BDF90@phx.gbl>
References: <SNT143-W159E3F270B85F3122EDF42BDF90@phx.gbl>
Message-ID: <CA+wAnvyPc0_SNPexo1SXbT0AhN=WzQRNaYmi+_u+hDU43Y01-Q@mail.gmail.com>

Varnish relies on cygwin to run on windows plataform, so if cygwin is not
working ok, you must fix it before.
You have more information about cygwin at: http://www.cygwin.com/
<http://www.cygwin.com/%20>

They say that cygwin works on windows 2008 server 64 bits, but running in 32
bits compatibility mode:

http://www.cygwin.com/faq/ <http://www.cygwin.com/faq/%20>

*What versions of Windows are supported?

Cygwin can be expected to run on all modern 32 bit versions of Windows,
except Windows CE and Windows 95/98/Me. This includes, as of the time of
writing this, Windows NT4 SP4 or later, Windows 2000, Windows XP, Windows
Server 2003, Windows Vista, Windows Server 2008, Windows 7, as well as the
WOW64 32 bit environment on released 64 bit versions of Windows
(XP/2003/Vista/2008/7/2008 R2). As far as we know no one is working on a
native 64 bit version of Cygwin. Since Cygwin is a community-supported free
software project, patches to provide support for other versions would be
thoughtfully considered. Paid support contracts or enhancements are
available through Red Hat. For information about getting a Red Hat support
contract, see http://cygwin.com/license.html.

Keep in mind that Cygwin can only do as much as the underlying OS supports.
Because of this, Cygwin will behave differently, and exhibit different
limitations, on the various versions of Windows.
*


But i have googled a bit and found people with some issues with that
operating system:

   - http://old.nabble.com/Windows-2008-64-bit-install-td30813342.html
   -
   http://old.nabble.com/Bash-startup-problems-on-Windows-2008-server-td18110908.html
   -
   http://old.nabble.com/Windows-Server-2008-64-bit-setup.exe-bash-problem-tt28907612.html


you can ask for help at cygwin mailing list: http://cygwin.com/lists.html
(i am subscribed to some of them)

Regards,

Jorge

2011/10/6 NitO MaNia <elatino17 at hotmail.com>:
> is there any way to use Varnish in Windows 2008 server 64  bit? i already
> install Cygwin but it doesnt work with this version of windows, so i cant
> setup varnish , tks.
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111008/1ad117bf/attachment.html>

From simon at darkmere.gen.nz  Sun Oct  9 02:27:29 2011
From: simon at darkmere.gen.nz (Simon Lyall)
Date: Sun, 9 Oct 2011 15:27:29 +1300 (NZDT)
Subject: Varnish vs aiCache
In-Reply-To: <CAAswRkg34kR1cQF5NAscC9YDemnFH8BfpAr+F=YwNt8ewkXiSg@mail.gmail.com>
References: <CAAswRkg34kR1cQF5NAscC9YDemnFH8BfpAr+F=YwNt8ewkXiSg@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1110091506350.13572@green.darkmere.gen.nz>

On Wed, 5 Oct 2011, Paul-Alexander Polo wrote:
> Would you use Varnish to setup your own CDN (for example on Amazon) ?when
> the origin is a few thousand of kilometers away?

Origin: 	Auckland, New Zealand
Varnish Box:	Strasbourg, France
Distance:	18339 km

Would and do.

-- 
Simon Lyall  |  Very Busy  |  Web: http://www.darkmere.gen.nz/
"To stay awake all night adds a day to your life" - Stilgar | eMT.

From bedis9 at gmail.com  Sun Oct  9 13:10:42 2011
From: bedis9 at gmail.com (Baptiste)
Date: Sun, 9 Oct 2011 15:10:42 +0200
Subject: Varnish closes conenction to haproxy
In-Reply-To: <652562DB-727D-4769-AB96-F7403124F799@gmail.com>
References: <E2BBA08D-1AE2-4E83-987D-7CD4209326F6@gmail.com>
	<CAES+E52CVh3Qf2ZbU31bKrODOTpHbFGMHBDcbHrY_ayxYbthxA@mail.gmail.com>
	<652562DB-727D-4769-AB96-F7403124F799@gmail.com>
Message-ID: <CAODHi7rxycBxWESayZpCcvqGRTON2Z7E90g=5L4j3+UoGdd+fQ@mail.gmail.com>

Hi

And what's are your HAProxy timeouts?
By the way, are you sure the client is still waiting for the response?

cheers

On Thu, Oct 6, 2011 at 2:36 PM, Jan Olausson <janneolausson at gmail.com> wrote:
> Hi,
> Silly me not to mention it but yes. The values I have are:
> ? ? .connect_timeout = 1s;
> ? ? .first_byte_timeout = 600s;
> ? ? .between_bytes_timeout = 5s;
> Best regards
> Jan
>
> 6 okt 2011 kl. 14.25 skrev Stewart Robinson:
>
> Hi,
> Have you specified long timeouts in your backend declaration?
>
> backend www {
>   .host = "www.example.com";
>   .port = "http";
>   .connect_timeout = 1s;
>   .first_byte_timeout = 5s;
>   .between_bytes_timeout = 2s;
> }
>
> Stew
>
> On 6 October 2011 13:14, Jan Olausson <janneolausson at gmail.com> wrote:
>>
>> Hi!
>>
>> I have a problem with varnish closes the connection to haproxy infront of
>> it.
>> Haproxy reports a 502 error after exactly 60 seconds after it begun
>> handling the request.
>>
>> If I skip both haproxy and varnish and let nginx handle the request
>> directly, everything works although it takes 4-5 minutes to complete the
>> request due to the complexity of the work that has to be done.
>>
>> Anyone here have a suggestion what I shall try?
>> Varnish version is: varnish-3.0.0 revision 3bd5997
>>
>> Best regards
>> Jan Olausson
>> Sweden
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



From listas at kurtkraut.net  Sun Oct  9 15:44:59 2011
From: listas at kurtkraut.net (Kurt Kraut)
Date: Sun, 9 Oct 2011 12:44:59 -0300
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
	<CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
Message-ID: <CAPbn28=FoabrtWum4dspFpMmjBWQgB-UT=Bq=RHhv7Sek5M6QQ@mail.gmail.com>

Hi,


Sorry for the lack of varnishlog. My mistake. Here goes the varnishlog, the
correspondent wget log and other stuff that might be useful:

Varnishlog: http://pastebin.com/SVZRqL2k
Wget log downloading through varnish: http://pastebin.com/uv4kp97m
Wget log downloading straight through backend:
http://pastebin.com/5usCHpE0(note that there are no interruptions)
My default.vcl file: http://pastebin.com/idiBvh6f

If you need any further details I will be glad to provide them.


Thanks in advance,


Kurt Kraut

2011/10/7 Per Buer <perbu at varnish-software.com>

> Hi Kurt,
>
>
> On Fri, Oct 7, 2011 at 8:17 PM, Kurt Kraut <listas at kurtkraut.net> wrote:
>
>> Hi,
>> (..)
>>
>>
>> What do you think? Is it a bug?
>>
>
> Okey. This is one of the reasons top-posting is bad. Tollef asked you for
> varnishlog entry, which is the only sane input we can have in order to
> actually tell you what is going on  without it being total speculation.
>
> You ignored that. Why? If you need help actually getting sane  varnishlog
> output please tell so.
>
> I'm _guessing_ you are running into timeouts. But without a log entry,
> that's only a guess.
>
> --
> Per Buer, CEO
> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
> *Varnish makes websites fly!*
> Whitepapers <http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
> Twitter <https://twitter.com/varnishsoftware>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111009/04e2c0c8/attachment.html>

From Jeff.Stroomer at disney.com  Sun Oct  9 21:03:06 2011
From: Jeff.Stroomer at disney.com (Stroomer, Jeff)
Date: Sun, 9 Oct 2011 14:03:06 -0700
Subject: varnish won't come up cleanly after reboot
Message-ID: <4F86EEFDF44D2D41951D491B61B7D433248A0E4A61@SM-CALA-VXMB06B.swna.wdpr.disney.com>

Varnish folks,

I am having trouble getting varnish to come up cleanly after a reboot.  But if I ssh into the machine, and do a /etc/init.d/varnish restart by hand, then varnish starts working correctly.  Here are the details.

I'm running varnish 3.0.1 on a RedHat 5.5 VM.  Following a reboot, I see the following:

$ ps -eaf | grep varnish
root      3255     1  0 13:26 ?        00:00:00 /usr/sbin/varnishd -P /var/run/varnish.pid -a :6081 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,1G

The key thing here is that I see the manager process, but not the child.  After I do an /etc/init.d/varnish restart by hand, I see both manager and child:

$ ps -eaf | grep varnish
root      3525     1  0 13:31 ?        00:00:00 /usr/sbin/varnishd -P /var/run/varnish.pid -a :6081 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,1G
varnish   3526  3525  0 13:31 ?        00:00:00 /usr/sbin/varnishd -P /var/run/varnish.pid -a :6081 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,1G

And sure enough, when I look in /var/log/messages following my reboot, I see this:

Oct  9 20:26:09 duckster varnishd[3255]: child (3257) Started
Oct  9 20:26:09 duckster varnishd[3255]: Pushing vcls failed: dlopen(./vcl.1GLakAnf.so): ./vcl.1GLakAnf.so: cannot open shared object file: Permission denied
Oct  9 20:26:09 duckster varnishd[3255]: Child (3257) said Child starts
Oct  9 20:26:09 duckster varnishd[3255]: Child (3257) said SMF.s0 mmap'ed 1073741824 bytes of 1073741824
Oct  9 20:26:09 duckster varnishd[3255]: Child (3257) said Child dies
Oct  9 20:26:09 duckster varnishd[3255]: Child (3257) died

I checked /tmp, and it has plenty of space, and permissions are 0777.

Any ideas?

Jeff

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111009/13c21eaf/attachment.html>

From TFigueiro at au.westfield.com  Sun Oct  9 21:42:10 2011
From: TFigueiro at au.westfield.com (Thiago Figueiro)
Date: Sun, 9 Oct 2011 21:42:10 +0000
Subject: Varnish vs aiCache
In-Reply-To: <alpine.DEB.2.00.1110091506350.13572@green.darkmere.gen.nz>
References: <CAAswRkg34kR1cQF5NAscC9YDemnFH8BfpAr+F=YwNt8ewkXiSg@mail.gmail.com>
	<alpine.DEB.2.00.1110091506350.13572@green.darkmere.gen.nz>
Message-ID: <22F96A574165AA4482FF290947347774207882F0@AUBDC00-MBX01P.au.ad.westfield.com>

> On Wed, 5 Oct 2011, Paul-Alexander Polo wrote:
>> Would you use Varnish to setup your own CDN (for example on Amazon)  
>> when the origin is a few thousand of kilometers away?

Simon Lyall wrote:
> Origin: 	Auckland, New Zealand
> Varnish Box:	Strasbourg, France
> Distance:	18339 km
>
> Would and do.? 

I second that. 15 and 17 thousand km in a "straight" line.

Origin (Varnish 0): Sydney, AU
Varnish 1: Chicago, US (Rackspace ORD1)
Varnish 2: London, UK (Rackspace LON3)

We had London fetch content from Chicago and, that node being sick, London would fetch from Sydney directly.

This setup has since been replaced by a brand-name CDN but it served us very well for 2 years.  In fact, it worked so well that we didn't see a performance improvement (latency) when moving to a CDN (we still went ahead due to other reasons).


Thiago Figueiro
Technical Operations Manager

Westfield Digital Business
W www.westfield.com.au
E tfigueiro at au.westfield.com


______________________________________________________
    CONFIDENTIALITY NOTICE    
This electronic mail message, including any and/or all attachments, is for the sole use of the intended recipient(s), and may contain confidential and/or privileged information, pertaining to business conducted under the direction and supervision of the sending organization. All electronic mail messages, which may have been established as expressed views and/or opinions (stated either within the electronic mail message or any of its attachments), are left to the sole responsibility of that of the sender, and are not necessarily attributed to the sending organization. Unauthorized interception, review, use, disclosure or distribution of any such information contained within this electronic mail message and/or its attachment(s), is (are) strictly prohibited. If you are not the intended recipient, please contact the sender by replying to this electronic mail message, along with the destruction all copies of the original electronic mail message (along with any attachments).
______________________________________________________



From kacperw at gmail.com  Mon Oct 10 01:01:51 2011
From: kacperw at gmail.com (Kacper Wysocki)
Date: Sun, 9 Oct 2011 21:01:51 -0400
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CAOXZevAc6zU1JDkRD3=s=cRFg+=v-FU+jczPBGoUngBci+AjRA@mail.gmail.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
	<CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
	<CABaBnj5nxc-fASeSKWhT_vbZ4=C0AhvbDUpb9ZdtCyx_6HnhUQ@mail.gmail.com>
	<CAOXZevAc6zU1JDkRD3=s=cRFg+=v-FU+jczPBGoUngBci+AjRA@mail.gmail.com>
Message-ID: <CABaBnj4HT6jQrvj1rS=ZLUyvCgP2=SYRYgJAxf4Ei9dWuuvmoA@mail.gmail.com>

On Sat, Oct 8, 2011 at 4:23 AM, Per Buer <perbu at varnish-software.com> wrote:
> On Fri, Oct 7, 2011 at 11:34 PM, Kacper Wysocki <kacperw at gmail.com> wrote:
>>
>> I also noticed similar behavior with large files with Varnish 3.0.1.
>> Today, when I tried to do large files.. Here's a wget run.
>>
>> 2011-10-07 17:18:59 (442 KB/s) - Connection closed at byte 28062437. Retrying.
>> 2011-10-07 17:20:03 (486 KB/s) - Connection closed at byte 59268331. Retrying.
>> 2011-10-07 17:21:07 (500 KB/s) - Connection closed at byte 90762224. Retrying.
>
>
> Notice that there is 60 seconds between each fetch + some time to reconnect. 60s is the default for send_timeout since 3.0. It used to be 600 seconds.

There were bytes in transit thoughout the session.

>> > I'm _guessing_ you are running into timeouts. But without a log entry, that's only a guess.
>>
>> Could be timeouts, but I'm not sure why they would happen in this case.
>>
>> Here's the pure unbridled unadulterated vlog
> 404.

grr. serves me right for moving it

http://u.delta9.pl/k/drop/log/v-drops.vlog



From kacperw at gmail.com  Mon Oct 10 02:39:34 2011
From: kacperw at gmail.com (Kacper Wysocki)
Date: Sun, 9 Oct 2011 22:39:34 -0400
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CAPbn28=FoabrtWum4dspFpMmjBWQgB-UT=Bq=RHhv7Sek5M6QQ@mail.gmail.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
	<CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
	<CAPbn28=FoabrtWum4dspFpMmjBWQgB-UT=Bq=RHhv7Sek5M6QQ@mail.gmail.com>
Message-ID: <CABaBnj6JFNTb+t02B2+m64o9Hm_PfbJLQttmSwNQW5nRe0EmmA@mail.gmail.com>

On Sun, Oct 9, 2011 at 11:44 AM, Kurt Kraut <listas at kurtkraut.net> wrote:
> Sorry for the lack of varnishlog. My mistake. Here goes the varnishlog, the correspondent wget log and other stuff that might be useful:
>
> Varnishlog: http://pastebin.com/SVZRqL2k
> Wget log downloading through varnish: http://pastebin.com/uv4kp97m
> Wget log downloading straight through backend: http://pastebin.com/5usCHpE0 (note that there are no interruptions)
> My default.vcl file: http://pastebin.com/idiBvh6f
>
> If you need any further details I will be glad to provide them.

Apart from containing error 503's from your backend going down (?)
there is this line from the requests you refer to:
   13 Debug        c Write error, retval = 25318280, len = 104857948,
errno = Success

looks very much like mine..

setting send_timeout lower does make this problem more frequent for me
and setting this higher makes it less frequent, but why exactly are we
hitting the send timeout here? doesnt make sense to me.

-k



From perbu at varnish-software.com  Mon Oct 10 08:34:43 2011
From: perbu at varnish-software.com (Per Buer)
Date: Mon, 10 Oct 2011 10:34:43 +0200
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CABaBnj4HT6jQrvj1rS=ZLUyvCgP2=SYRYgJAxf4Ei9dWuuvmoA@mail.gmail.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
	<CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
	<CABaBnj5nxc-fASeSKWhT_vbZ4=C0AhvbDUpb9ZdtCyx_6HnhUQ@mail.gmail.com>
	<CAOXZevAc6zU1JDkRD3=s=cRFg+=v-FU+jczPBGoUngBci+AjRA@mail.gmail.com>
	<CABaBnj4HT6jQrvj1rS=ZLUyvCgP2=SYRYgJAxf4Ei9dWuuvmoA@mail.gmail.com>
Message-ID: <CAOXZevATABk0mGuV5tL9Vmmq-5r2cTSiUS+VkHNXE2WYb+yNmg@mail.gmail.com>

On Mon, Oct 10, 2011 at 3:01 AM, Kacper Wysocki <kacperw at gmail.com> wrote:

> On Sat, Oct 8, 2011 at 4:23 AM, Per Buer <perbu at varnish-software.com>
> wrote:
> > On Fri, Oct 7, 2011 at 11:34 PM, Kacper Wysocki <kacperw at gmail.com>
> wrote:
> >>
> >> I also noticed similar behavior with large files with Varnish 3.0.1.
> >> Today, when I tried to do large files.. Here's a wget run.
> >>
> >> 2011-10-07 17:18:59 (442 KB/s) - Connection closed at byte 28062437.
> Retrying.
> >> 2011-10-07 17:20:03 (486 KB/s) - Connection closed at byte 59268331.
> Retrying.
> >> 2011-10-07 17:21:07 (500 KB/s) - Connection closed at byte 90762224.
> Retrying.
> >
> >
> > Notice that there is 60 seconds between each fetch + some time to
> reconnect. 60s is the default for send_timeout since 3.0. It used to be 600
> seconds.
>
> There were bytes in transit thoughout the session.
>

Yes. We don't care. :-)

No, but seriously, this is still hitting the timeout. PHK changed the
default back in February after some discussions in the user group. I think
Tollef is working on making the timeout actually not hit unless you have no
data going over the connection. Hopefully we'll expect a patch within the
week.

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111010/30a31c12/attachment.html>

From Harri.Paivaniemi at tieto.com  Mon Oct 10 10:10:02 2011
From: Harri.Paivaniemi at tieto.com (Harri.Paivaniemi at tieto.com)
Date: Mon, 10 Oct 2011 13:10:02 +0300
Subject: Stupid has a stupid fetch- problem
Message-ID: <F5A4C3A7AC59A74FB0A0D35F1E8BDB462C9056C213@EXMB03.eu.tieto.com>

Hi,

I'm just so stupid with this.

Can somebody see, why on the earth with this vcl_fetch EVERY request goes righ into the last configured apache_farm?

If I test this with correct url's, I can't get any traffic to match to this:

"^/easearch/json/something/") {

or this:

} elseif (req.url ~ "^/easearch") {

... and all goes, no matter what I ask, to this:

} elseif (req.url ~ "^/bar/.*\.(gif|jpg|jpeg|png|bmp|htm|html|xml|css|js|swf|flv)$") {

... and to apache_farm, like it was te default or something.

How come?

Thanks.

--------------------------------


sub vcl_fetch {

#1. www.foo.fi and foo.fi vhosts

if ((req.http.Host ~ "www\.foo.fi") || req.http.Host ~ "foo\.fi") {

#Set default site header
set req.http.Host = "foo.fi";

#easearch rules

#check error status first

if (beresp.status > 399) {
#no cache if status is 400 or bigger
set beresp.ttl = 0s;
return (hit_for_pass);
}

if (req.url ~ "^/easearch/json/something/") {
set beresp.ttl = 60m;
remove beresp.http.Server;
set req.backend = tom_jerry_8080;
unset beresp.http.set-cookie;
set beresp.grace = 300s;
return (deliver);

} elseif (req.url ~ "^/easearch") {
set beresp.ttl = 10m;
remove beresp.http.Server;
set req.backend = tom_jerry_8080;
unset beresp.http.set-cookie;
set beresp.grace = 300s;
return (deliver);

} elseif (req.url ~ "^/bar/.*\.(gif|jpg|jpeg|png|bmp|htm|html|xml|css|js|swf|flv)$") {
set req.backend = apache_farm_80;
unset beresp.http.set-cookie;
set beresp.ttl = 3m;
set beresp.grace = 300s;
return (deliver);
 
} else {
#no rules anymore for foo.fi vhost
error 200 "No rules configured for vhost foo.fi...";
}

} else {
#this was the last vhost congifured in this instance...
error 200 "Not permitted";

}
}




From tfheen at varnish-software.com  Mon Oct 10 13:00:15 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Mon, 10 Oct 2011 15:00:15 +0200
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CABaBnj6JFNTb+t02B2+m64o9Hm_PfbJLQttmSwNQW5nRe0EmmA@mail.gmail.com>
	(Kacper Wysocki's message of "Sun, 9 Oct 2011 22:39:34 -0400")
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
	<CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
	<CAPbn28=FoabrtWum4dspFpMmjBWQgB-UT=Bq=RHhv7Sek5M6QQ@mail.gmail.com>
	<CABaBnj6JFNTb+t02B2+m64o9Hm_PfbJLQttmSwNQW5nRe0EmmA@mail.gmail.com>
Message-ID: <87pqi5ypb4.fsf@qurzaw.varnish-software.com>

]] Kacper Wysocki 

| setting send_timeout lower does make this problem more frequent for me
| and setting this higher makes it less frequent, but why exactly are we
| hitting the send timeout here? doesnt make sense to me.

Because our implementations of TCP timeouts is not completely correct or
complete.  You could try https://gist.github.com/1275251 and see if that
makes for more happiness.

Regards,
-- 
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64



From mattias at nucleus.be  Mon Oct 10 13:16:17 2011
From: mattias at nucleus.be (Mattias Geniar)
Date: Mon, 10 Oct 2011 15:16:17 +0200
Subject: varnish won't come up cleanly after reboot
In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D433248A0E4A61@SM-CALA-VXMB06B.swna.wdpr.disney.com>
References: <4F86EEFDF44D2D41951D491B61B7D433248A0E4A61@SM-CALA-VXMB06B.swna.wdpr.disney.com>
Message-ID: <18834F5BEC10824891FB8B22AC821A5A01983ED0@nucleus-srv01.Nucleus.local>

> Oct? 9 20:26:09 duckster varnishd[3255]: child (3257) Started
> Oct? 9 20:26:09 duckster varnishd[3255]: Pushing vcls failed: dlopen(./vcl.1GLakAnf.so): ./vcl.1GLakAnf.so: cannot open shared object file: Permission denied 
> Oct? 9 20:26:09 duckster varnishd[3255]: Child (3257) said Child starts
> Oct? 9 20:26:09 duckster varnishd[3255]: Child (3257) said Child dies
> Oct? 9 20:26:09 duckster varnishd[3255]: Child (3257) died
> 
> I checked /tmp, and it has plenty of space, and permissions are 0777.
> 
> Any ideas?

Could you list all start-up programs? It may be that Varnish is trying to start too early and that is simply needs to have its startup order changed.
# ls -alh /etc/rc.d/rc3.d/

Regards,
Mattias



From spydergst_97 at yahoo.com  Mon Oct 10 17:11:24 2011
From: spydergst_97 at yahoo.com (M.H.)
Date: Mon, 10 Oct 2011 10:11:24 -0700 (PDT)
Subject: IP pass thru
Message-ID: <1318266684.5014.YahooMailNeo@web111706.mail.gq1.yahoo.com>

Is it possible to let clients with certain IP's pass thru to the backend and not cache using varnish?



From roberto.fernandezcrisial at gmail.com  Mon Oct 10 17:19:12 2011
From: roberto.fernandezcrisial at gmail.com (=?utf-8?B?Um9iZXJ0byBPLiBGZXJuw6FuZGV6IENyaXNpYWw=?=)
Date: Mon, 10 Oct 2011 17:19:12 +0000
Subject: IP pass thru
In-Reply-To: <1318266684.5014.YahooMailNeo@web111706.mail.gq1.yahoo.com>
References: <1318266684.5014.YahooMailNeo@web111706.mail.gq1.yahoo.com>
Message-ID: <755908677-1318267154-cardhu_decombobulator_blackberry.rim.net-1105551874-@b27.c27.bise6.blackberry>

M.H.,

Yes, you can:

if (client.ip == IP)
{
return(pass);
}

Roberto O. Fern?ndez Crisial
@rofc
-----Original Message-----
From: "M.H." <spydergst_97 at yahoo.com>
Sender: varnish-misc-bounces at varnish-cache.org
Date: Mon, 10 Oct 2011 10:11:24 
To: varnish-misc at varnish-cache.org<varnish-misc at varnish-cache.org>
Reply-To: "M.H." <spydergst_97 at yahoo.com>
Subject: IP pass thru

Is it possible to let clients with certain IP's pass thru to the backend and not cache using varnish?

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

From kacperw at gmail.com  Mon Oct 10 21:36:58 2011
From: kacperw at gmail.com (Kacper Wysocki)
Date: Mon, 10 Oct 2011 17:36:58 -0400
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <87pqi5ypb4.fsf@qurzaw.varnish-software.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
	<CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
	<CAPbn28=FoabrtWum4dspFpMmjBWQgB-UT=Bq=RHhv7Sek5M6QQ@mail.gmail.com>
	<CABaBnj6JFNTb+t02B2+m64o9Hm_PfbJLQttmSwNQW5nRe0EmmA@mail.gmail.com>
	<87pqi5ypb4.fsf@qurzaw.varnish-software.com>
Message-ID: <CABaBnj5k9Lvy6b1oS21Q4sEY9mefoJGVUjwqmEuT+bzDjNYAUg@mail.gmail.com>

On Mon, Oct 10, 2011 at 9:00 AM, Tollef Fog Heen
<tfheen at varnish-software.com> wrote:
> ]] Kacper Wysocki
>
> | setting send_timeout lower does make this problem more frequent for me
> | and setting this higher makes it less frequent, but why exactly are we
> | hitting the send timeout here? doesnt make sense to me.

> Because our implementations of TCP timeouts is not completely correct or
> complete. ?You could try https://gist.github.com/1275251 and see if that
> makes for more happiness.

Oh, who knew. So if you set the timeout to 10s then it will kill
transfers at 10 seconds?

I could try patching the version string to 3.0.2 or do you mean there
is some change in 3.0.2 that attempts to make more happiness?
:-P none of this is very intuitive. So far I've bumped the timeout and
so all I can pray is that
 1) clients have enough bandwidth to complete any file they are
interested in within the timeout, and
 2) noone thinks of slowl0ris'ing me considering my whacky timeout value

:-P 0K



From tfheen at varnish-software.com  Tue Oct 11 05:31:31 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Tue, 11 Oct 2011 07:31:31 +0200
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CABaBnj5k9Lvy6b1oS21Q4sEY9mefoJGVUjwqmEuT+bzDjNYAUg@mail.gmail.com>
	(Kacper Wysocki's message of "Mon, 10 Oct 2011 17:36:58 -0400")
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
	<CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
	<CAPbn28=FoabrtWum4dspFpMmjBWQgB-UT=Bq=RHhv7Sek5M6QQ@mail.gmail.com>
	<CABaBnj6JFNTb+t02B2+m64o9Hm_PfbJLQttmSwNQW5nRe0EmmA@mail.gmail.com>
	<87pqi5ypb4.fsf@qurzaw.varnish-software.com>
	<CABaBnj5k9Lvy6b1oS21Q4sEY9mefoJGVUjwqmEuT+bzDjNYAUg@mail.gmail.com>
Message-ID: <87ehykytzg.fsf@qurzaw.varnish-software.com>

]] Kacper Wysocki 

| On Mon, Oct 10, 2011 at 9:00 AM, Tollef Fog Heen
| <tfheen at varnish-software.com> wrote:
| > ]] Kacper Wysocki
| >
| > | setting send_timeout lower does make this problem more frequent for me
| > | and setting this higher makes it less frequent, but why exactly are we
| > | hitting the send timeout here? doesnt make sense to me.
| 
| > Because our implementations of TCP timeouts is not completely correct or
| > complete. ?You could try https://gist.github.com/1275251 and see if that
| > makes for more happiness.
| 
| Oh, who knew. So if you set the timeout to 10s then it will kill
| transfers at 10 seconds?

Yes.  This is actually documented, if a bit surprising.

| I could try patching the version string to 3.0.2 or do you mean there
| is some change in 3.0.2 that attempts to make more happiness?

Oh, that was the wrong patch, try https://gist.github.com/1277365 instead.

-- 
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64



From tfheen at varnish-software.com  Tue Oct 11 06:08:08 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Tue, 11 Oct 2011 08:08:08 +0200
Subject: /var/lib/varnish/varnish.*
In-Reply-To: <CAJxePN+6qm5hGu-DtV1LziTiZKdun1KZ6EYi6ZJHAogRb4XD-Q@mail.gmail.com>
	(alexus@gmail.com's message of "Mon, 29 Aug 2011 11:18:02 -0400")
References: <CAJxePN+6qm5hGu-DtV1LziTiZKdun1KZ6EYi6ZJHAogRb4XD-Q@mail.gmail.com>
Message-ID: <8762jwysaf.fsf@qurzaw.varnish-software.com>

]] alexus 

| I dont know if something is wrong or if it suppose to be like this,
| but if I leave varnish by itself for a long time I get something like
| this:

[...]

| is this a bug or a feature?

Bug.  Fixed in git master and will be fixed in 3.0.2 which should be out
shortly.

-- 
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64



From perbu at varnish-software.com  Tue Oct 11 07:33:11 2011
From: perbu at varnish-software.com (Per Buer)
Date: Tue, 11 Oct 2011 09:33:11 +0200
Subject: IP pass thru
In-Reply-To: <1318266684.5014.YahooMailNeo@web111706.mail.gq1.yahoo.com>
References: <1318266684.5014.YahooMailNeo@web111706.mail.gq1.yahoo.com>
Message-ID: <CAOXZevAnqeYDa_pH0r1o5z7JsSz1YmwLGsL6Q9A__wjhFYtANQ@mail.gmail.com>

Hi.
On Mon, Oct 10, 2011 at 7:11 PM, M.H. <spydergst_97 at yahoo.com> wrote:
>
> Is it possible to let clients with certain IP's pass thru to the backend and not cache using varnish?

Yes. However, what you probably want is to use an ACL rather than
matching directly against IP addresses. It's probably faster as well
(not that it would be possible to measure the difference).

acl passem {
        "192.168.55.0/24";
}


sub vcl_recv {
  (..)
  if (!(client.ip ~ passem)) {
    return(pass);
  }
(..)

}






--
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
Varnish makes websites fly!
Whitepapers?| Video?| Twitter



From paulo at aliancaproject.com  Tue Oct 11 13:28:43 2011
From: paulo at aliancaproject.com (Paulo Paracatu)
Date: Tue, 11 Oct 2011 10:28:43 -0300
Subject: Varnish serving content from wrong backend
Message-ID: <CABQXmR0dBoT=TpbwBSGEXpEMieqWrxGw+Tf6peEzpzT-3PWPWQ@mail.gmail.com>

Hello,
It seems there is something broken in my Varnish configuration files.
I have two Varnish servers running and 14 backend servers. The first backend
loaded in the VCLs is the 's105', which host several adult (including
hentai) sites, and after it it loads the other 13 backends, hosting several
wordpress blogs and invision board forums.

The problem here is: many users are reporting adult content being loaded
instead of their regular content.
Here are some screens I was able to get from the users:
http://img441.imageshack.us/img441/18/dotaa1111.jpg
It should load a regular invision board topic, hosted at s162 (5? backend
loaded), instead it loaded the page from a hentai site hosted at s105.
Here it happens again: http://img810.imageshack.us/img810/7326/img1dbr.png
And again: http://img12.imageshack.us/img12/9215/img2dbr.png
And again: http://i.imgur.com/8HFrD.jpg (I don't think this one is from
s105..., but who knows).

Anyone here ever had anything similar to this?

Thanks,
Paulo.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111011/9441f9da/attachment.html>

From varnish at mm.quex.org  Tue Oct 11 14:41:50 2011
From: varnish at mm.quex.org (Michael Alger)
Date: Tue, 11 Oct 2011 22:41:50 +0800
Subject: Varnish serving content from wrong backend
In-Reply-To: <CABQXmR0dBoT=TpbwBSGEXpEMieqWrxGw+Tf6peEzpzT-3PWPWQ@mail.gmail.com>
References: <CABQXmR0dBoT=TpbwBSGEXpEMieqWrxGw+Tf6peEzpzT-3PWPWQ@mail.gmail.com>
Message-ID: <20111011144150.GA7775@grum.quex.org>

On Tue, Oct 11, 2011 at 10:28:43AM -0300, Paulo Paracatu wrote:
> It seems there is something broken in my Varnish configuration
> files.  I have two Varnish servers running and 14 backend servers.
> The first backend loaded in the VCLs is the 's105', which host
> several adult (including hentai) sites, and after it it loads the
> other 13 backends, hosting several wordpress blogs and invision
> board forums.

The first backend Varnish loads will be the default backend, i.e.
where requests are sent if you don't explicitly set req.backend in
vcl_recv. It sounds like you might not be correctly setting your
backend when processing requests.

Can you provide your VCL, or at least a representative sample of how
you're mapping your sites to backends?

If you're relying on host headers, are you properly handling all
possibilities? e.g. some browsers might send the port number so you'll
get something like "Host: www.some.site:80" which maybe you're not
handling (expecting only "www.some.site") and therefore it's going
through to the default backend.



From paulo at aliancaproject.com  Tue Oct 11 14:54:45 2011
From: paulo at aliancaproject.com (Paulo Paracatu)
Date: Tue, 11 Oct 2011 11:54:45 -0300
Subject: Varnish serving content from wrong backend
In-Reply-To: <20111011144150.GA7775@grum.quex.org>
References: <CABQXmR0dBoT=TpbwBSGEXpEMieqWrxGw+Tf6peEzpzT-3PWPWQ@mail.gmail.com>
	<20111011144150.GA7775@grum.quex.org>
Message-ID: <CABQXmR25yQFEz3N5Y76rZhVKemu9G4Z_-VM6fBc1Ooc6OO=fUw@mail.gmail.com>

>
> Can you provide your VCL, or at least a representative sample of how
> you're mapping your sites to backends?
>

Sure, here is it: http://pastebin.com/uvFKH4ub


>
> If you're relying on host headers, are you properly handling all
> possibilities? e.g. some browsers might send the port number so you'll
> get something like "Host: www.some.site:80" which maybe you're not
> handling (expecting only "www.some.site") and therefore it's going
> through to the default backend.
>

Oh... Makes sense. As you can see in my VCL example, I'm not expecting the
":80".
Do you think this may be the problem, so?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111011/aca153d2/attachment.html>

From varnish at mm.quex.org  Tue Oct 11 15:12:46 2011
From: varnish at mm.quex.org (Michael Alger)
Date: Tue, 11 Oct 2011 23:12:46 +0800
Subject: Varnish serving content from wrong backend
In-Reply-To: <CABQXmR25yQFEz3N5Y76rZhVKemu9G4Z_-VM6fBc1Ooc6OO=fUw@mail.gmail.com>
References: <CABQXmR0dBoT=TpbwBSGEXpEMieqWrxGw+Tf6peEzpzT-3PWPWQ@mail.gmail.com>
	<20111011144150.GA7775@grum.quex.org>
	<CABQXmR25yQFEz3N5Y76rZhVKemu9G4Z_-VM6fBc1Ooc6OO=fUw@mail.gmail.com>
Message-ID: <20111011151246.GA8420@grum.quex.org>

On Tue, Oct 11, 2011 at 11:54:45AM -0300, Paulo Paracatu wrote:
> > Can you provide your VCL, or at least a representative sample of
> > how you're mapping your sites to backends?
> 
> Sure, here is it: http://pastebin.com/uvFKH4ub
> 
> > If you're relying on host headers, are you properly handling all
> > possibilities? e.g. some browsers might send the port number so
> > you'll get something like "Host: www.some.site:80" which maybe
> > you're not handling (expecting only "www.some.site") and therefore
> > it's going through to the default backend.
> 
> Oh... Makes sense. As you can see in my VCL example, I'm not
> expecting the ":80".  Do you think this may be the problem, so?

It's a possibility... I don't know offhand which browsers include the
port in the host header for standard ports. It might be a good idea to
use a regsub to delete a trailing :\d+ from the host header on every
request just to be sure (assuming the port isn't important to you).

Also I had another thought that you should check your vcl_hash to make
sure it's including any information you or your backends use to decide
which content to serve. For example if it wasn't incorporating the
host header into the hash (only the request path) then the first
resource with a particular path would be cached by Varnish, and then
served for any request with the same path, regardless of host.

Do your backends make use of the X-Sub or X-Host header you add to the
request? If so, you'd want to make sure they're incorporated into the
object's hash. If you're only using them temporarily to build
redirects etc. then that's occurring before the cache lookup, so
shouldn't be a concern.

If you're not already, it might also be worthwhile checking the
request actually includes the host header and displaying an error
message or something if it's missing. Probably unlikely to be the case
with any graphical browser in use today, but it never hurts to be
thorough.

Lastly, if you can't find a way to reproduce this on demand, then
getting logs from it may be difficult. It might be worthwhile to set
up a "dummy" backend as the first one Varnish loads, which should
never actually be used by any of your sites. That way if it does
somehow get used, you'll know the request somehow failed to get
assigned to a backend, and you can log the request (on the dummy
backend, even) - maybe something about it will stick out as the
obvious culprit. Plus, it'd avoid accidentally serving certain images
to unsuspecting users.



From paulo at aliancaproject.com  Tue Oct 11 16:58:55 2011
From: paulo at aliancaproject.com (Paulo Paracatu)
Date: Tue, 11 Oct 2011 13:58:55 -0300
Subject: Varnish serving content from wrong backend
In-Reply-To: <20111011151246.GA8420@grum.quex.org>
References: <CABQXmR0dBoT=TpbwBSGEXpEMieqWrxGw+Tf6peEzpzT-3PWPWQ@mail.gmail.com>
	<20111011144150.GA7775@grum.quex.org>
	<CABQXmR25yQFEz3N5Y76rZhVKemu9G4Z_-VM6fBc1Ooc6OO=fUw@mail.gmail.com>
	<20111011151246.GA8420@grum.quex.org>
Message-ID: <CABQXmR0c7r9iVpfdngm-kyuFDnPtCZ306Mg86E+xSCCiLYe3Bg@mail.gmail.com>

>
> It's a possibility... I don't know offhand which browsers include the
> port in the host header for standard ports. It might be a good idea to
> use a regsub to delete a trailing :\d+ from the host header on every
> request just to be sure (assuming the port isn't important to you).
>

Good idea. I'll do it today. Thanks.


>
> Also I had another thought that you should check your vcl_hash to make
> sure it's including any information you or your backends use to decide
> which content to serve. For example if it wasn't incorporating the
> host header into the hash (only the request path) then the first
> resource with a particular path would be cached by Varnish, and then
> served for any request with the same path, regardless of host.
>

I'm using the default vcl_hash. I didn't really read about vcl_hash yet.
*sub vcl_hash {*
*    hash_data(req.url);*
*    if (req.http.host) {*
*        hash_data(req.http.host);*
*    } else {*
*        hash_data(server.ip);*
*    }*
*    return (hash);*
*}*



>
> Do your backends make use of the X-Sub or X-Host header you add to the
> request? If so, you'd want to make sure they're incorporated into the
> object's hash. If you're only using them temporarily to build
> redirects etc. then that's occurring before the cache lookup, so
> shouldn't be a concern.


> If you're not already, it might also be worthwhile checking the
> request actually includes the host header and displaying an error
> message or something if it's missing. Probably unlikely to be the case
> with any graphical browser in use today, but it never hurts to be
> thorough.
>
> Lastly, if you can't find a way to reproduce this on demand, then
> getting logs from it may be difficult. It might be worthwhile to set
> up a "dummy" backend as the first one Varnish loads, which should
> never actually be used by any of your sites. That way if it does
> somehow get used, you'll know the request somehow failed to get
> assigned to a backend, and you can log the request (on the dummy
> backend, even) - maybe something about it will stick out as the
> obvious culprit. Plus, it'd avoid accidentally serving certain images
> to unsuspecting users.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111011/b6573b1c/attachment.html>

From listas at kurtkraut.net  Tue Oct 11 19:33:49 2011
From: listas at kurtkraut.net (Kurt Kraut)
Date: Tue, 11 Oct 2011 16:33:49 -0300
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CAPbn28=FoabrtWum4dspFpMmjBWQgB-UT=Bq=RHhv7Sek5M6QQ@mail.gmail.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
	<CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
	<CAPbn28=FoabrtWum4dspFpMmjBWQgB-UT=Bq=RHhv7Sek5M6QQ@mail.gmail.com>
Message-ID: <CAPbn28mRNsM1uit8E57EG8yBH9jKx2T9nCY6EO1Ts1N2j-bA-g@mail.gmail.com>

Hi,


Does anyone have a clue on the issue I've reported? I've provided the
varnishlog during the time varnish keeps stopping the download.

Thanks in advance,


Kurt Kraut

2011/10/9 Kurt Kraut <listas at kurtkraut.net>

> Hi,
>
>
> Sorry for the lack of varnishlog. My mistake. Here goes the varnishlog, the
> correspondent wget log and other stuff that might be useful:
>
> Varnishlog: http://pastebin.com/SVZRqL2k
> Wget log downloading through varnish: http://pastebin.com/uv4kp97m
> Wget log downloading straight through backend:
> http://pastebin.com/5usCHpE0 (note that there are no interruptions)
> My default.vcl file: http://pastebin.com/idiBvh6f
>
> If you need any further details I will be glad to provide them.
>
>
>
> Thanks in advance,
>
>
> Kurt Kraut
>
> 2011/10/7 Per Buer <perbu at varnish-software.com>
>
>> Hi Kurt,
>>
>>
>> On Fri, Oct 7, 2011 at 8:17 PM, Kurt Kraut <listas at kurtkraut.net> wrote:
>>
>>> Hi,
>>> (..)
>>>
>>>
>>> What do you think? Is it a bug?
>>>
>>
>> Okey. This is one of the reasons top-posting is bad. Tollef asked you for
>> varnishlog entry, which is the only sane input we can have in order to
>> actually tell you what is going on  without it being total speculation.
>>
>> You ignored that. Why? If you need help actually getting sane  varnishlog
>> output please tell so.
>>
>> I'm _guessing_ you are running into timeouts. But without a log entry,
>> that's only a guess.
>>
>> --
>> Per Buer, CEO
>> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
>> *Varnish makes websites fly!*
>> Whitepapers <http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
>> Twitter <https://twitter.com/varnishsoftware>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111011/94fc6eb8/attachment.html>

From apj at mutt.dk  Tue Oct 11 19:40:17 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Tue, 11 Oct 2011 21:40:17 +0200
Subject: Varnish drops connections and make several HTTP 206
In-Reply-To: <CAPbn28mRNsM1uit8E57EG8yBH9jKx2T9nCY6EO1Ts1N2j-bA-g@mail.gmail.com>
References: <CAPbn28=MwCbeAb11T=jBUnO_tdHM_NhzzV_98-KOXsw4=1uUTA@mail.gmail.com>
	<87vcs12swp.fsf@qurzaw.varnish-software.com>
	<CAPbn28mzxuxGeHDF2h+avvukfzL0+Uhy_U8UdqbkQ60+_61Aqg@mail.gmail.com>
	<CAOXZevB70rqTNZA6RaUimM7b_GaGZnBDGQjRJtT9o_fSG3qOOA@mail.gmail.com>
	<CAPbn28=FoabrtWum4dspFpMmjBWQgB-UT=Bq=RHhv7Sek5M6QQ@mail.gmail.com>
	<CAPbn28mRNsM1uit8E57EG8yBH9jKx2T9nCY6EO1Ts1N2j-bA-g@mail.gmail.com>
Message-ID: <20111011194017.GG13866@nerd.dk>

On Tue, Oct 11, 2011 at 04:33:49PM -0300, Kurt Kraut wrote:
> 
> Does anyone have a clue on the issue I've reported? I've provided the
> varnishlog during the time varnish keeps stopping the download.

Any reason to think it's not what Tollef already addressed elsewhere in this
thread?

-- 
Andreas



From damon at huddler-inc.com  Tue Oct 11 21:36:43 2011
From: damon at huddler-inc.com (Damon Snyder)
Date: Tue, 11 Oct 2011 14:36:43 -0700
Subject: varnish won't come up cleanly after reboot
In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D433248A0E4A61@SM-CALA-VXMB06B.swna.wdpr.disney.com>
References: <4F86EEFDF44D2D41951D491B61B7D433248A0E4A61@SM-CALA-VXMB06B.swna.wdpr.disney.com>
Message-ID: <CACkQbugbFSZxTSr4j-j52cOL+GjOss0RP+4FjOvvZJ5OcUCDdQ@mail.gmail.com>

Make sure that varnishd has access to read and write to the working dir
(usually /var/lib/varnish or the directory you specify at the command line
with -n).

Damon

On Sun, Oct 9, 2011 at 2:03 PM, Stroomer, Jeff <Jeff.Stroomer at disney.com>wrote:

> Varnish folks,****
>
> ** **
>
> I am having trouble getting varnish to come up cleanly after a reboot.  But
> if I ssh into the machine, and do a /etc/init.d/varnish restart by hand,
> then varnish starts working correctly.  Here are the details.****
>
> ** **
>
> I?m running varnish 3.0.1 on a RedHat 5.5 VM.  Following a reboot, I see
> the following:****
>
> ** **
>
> $ ps -eaf | grep varnish****
>
> root      3255     1  0 13:26 ?        00:00:00 /usr/sbin/varnishd -P
> /var/run/varnish.pid -a :6081 -f /etc/varnish/default.vcl -T
> 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S
> /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,1G****
>
> ** **
>
> The key thing here is that I see the manager process, but not the child.
> After I do an /etc/init.d/varnish restart by hand, I see both manager and
> child:****
>
> ** **
>
> $ ps -eaf | grep varnish****
>
> root      3525     1  0 13:31 ?        00:00:00 /usr/sbin/varnishd -P
> /var/run/varnish.pid -a :6081 -f /etc/varnish/default.vcl -T
> 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S
> /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,1G****
>
> varnish   3526  3525  0 13:31 ?        00:00:00 /usr/sbin/varnishd -P
> /var/run/varnish.pid -a :6081 -f /etc/varnish/default.vcl -T
> 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S
> /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,1G****
>
> ** **
>
> And sure enough, when I look in /var/log/messages following my reboot, I
> see this:****
>
> ** **
>
> Oct  9 20:26:09 duckster varnishd[3255]: child (3257) Started****
>
> Oct  9 20:26:09 duckster varnishd[3255]: Pushing vcls failed: dlopen(./
> vcl.1GLakAnf.so): ./vcl.1GLakAnf.so: cannot open shared object file:
> Permission denied ****
>
> Oct  9 20:26:09 duckster varnishd[3255]: Child (3257) said Child starts***
> *
>
> Oct  9 20:26:09 duckster varnishd[3255]: Child (3257) said SMF.s0 mmap'ed
> 1073741824 bytes of 1073741824****
>
> Oct  9 20:26:09 duckster varnishd[3255]: Child (3257) said Child dies****
>
> Oct  9 20:26:09 duckster varnishd[3255]: Child (3257) died****
>
> ** **
>
> I checked /tmp, and it has plenty of space, and permissions are 0777.****
>
> ** **
>
> Any ideas?****
>
> ** **
>
> Jeff****
>
> ** **
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111011/935e7d81/attachment.html>

From Jeff.Stroomer at disney.com  Wed Oct 12 04:01:40 2011
From: Jeff.Stroomer at disney.com (Stroomer, Jeff)
Date: Tue, 11 Oct 2011 21:01:40 -0700
Subject: req.url in vcl_recv
In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D4332489A5273C@SM-CALA-VXMB06B.swna.wdpr.disney.com>
References: <4F86EEFDF44D2D41951D491B61B7D4332489A5273C@SM-CALA-VXMB06B.swna.wdpr.disney.com>
Message-ID: <4F86EEFDF44D2D41951D491B61B7D433248A24F45E@SM-CALA-VXMB06B.swna.wdpr.disney.com>

Varnish folks,

Suppose the incoming URL in Varnish is http://xyz.com/a/b/c.  What should I expect to see as the value for req.url inside vcl_recv?  I've seen plenty of example vcl code that seems to say req.url will contain /a/b/c.  But  it looks like  I'm actually getting the full URL, i.e., http://xyz.com/a/b/c.   Is this correct?  If so, why to all the examples seems to say the http://xyz.c/om/ prefix will be stripped off?  If it  makes a difference, I'm running Varnish 3.0.1 on Ubuntu 11.04.

Thanks,

Jeff

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111011/7628a948/attachment.html>

From varnish at mm.quex.org  Wed Oct 12 09:01:15 2011
From: varnish at mm.quex.org (Michael Alger)
Date: Wed, 12 Oct 2011 17:01:15 +0800
Subject: req.url in vcl_recv
In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D433248A24F45E@SM-CALA-VXMB06B.swna.wdpr.disney.com>
References: <4F86EEFDF44D2D41951D491B61B7D4332489A5273C@SM-CALA-VXMB06B.swna.wdpr.disney.com>
	<4F86EEFDF44D2D41951D491B61B7D433248A24F45E@SM-CALA-VXMB06B.swna.wdpr.disney.com>
Message-ID: <20111012090115.GA12862@grum.quex.org>

On Tue, Oct 11, 2011 at 09:01:40PM -0700, Stroomer, Jeff wrote:
> Suppose the incoming URL in Varnish is http://xyz.com/a/b/c.  What
> should I expect to see as the value for req.url inside vcl_recv?
> I've seen plenty of example vcl code that seems to say req.url will
> contain /a/b/c.  But  it looks like  I'm actually getting the full
> URL, i.e., http://xyz.com/a/b/c.   Is this correct?  If so, why to
> all the examples seems to say the http://xyz.c/om/ prefix will be
> stripped off?  If it  makes a difference, I'm running Varnish 3.0.1
> on Ubuntu 11.04.

req.url will contain whatever the requesting client sent in the
request after the method, so if the browser sent an absolute URI like
  GET http://xyz.com/a/b/c HTTP/1.1
then req.url will be "http://xyz.com/a/b/c".

If the browser sent just the path, like:
  GET /a/b/c HTTP/1.1
then req.url will be "/a/b/c".

Clients will normally only send the full URL if they think they're
talking to a proxy server (i.e. they've been configured to use a proxy
server) as that's the RFC-mandated behaviour. Many web servers will
correctly handle both forms, but when speaking to an origin server the
client is supposed to just send the request path, not the full URI
along with protocol specification.

What browser are you using? Do you have any other proxies / load
balancers in front of Varnish that can be configured to act as if
they're talking to a backend rather than a proxy? As far as the HTTP
protocol is concerned, Varnish is more akin to an origin server than a
proxy, and anything talking to it should be treating it as such.



From lampe at hauke-lampe.de  Wed Oct 12 10:21:04 2011
From: lampe at hauke-lampe.de (Hauke Lampe)
Date: Wed, 12 Oct 2011 12:21:04 +0200
Subject: req.url in vcl_recv
In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D433248A24F45E@SM-CALA-VXMB06B.swna.wdpr.disney.com>
References: <4F86EEFDF44D2D41951D491B61B7D4332489A5273C@SM-CALA-VXMB06B.swna.wdpr.disney.com>
	<4F86EEFDF44D2D41951D491B61B7D433248A24F45E@SM-CALA-VXMB06B.swna.wdpr.disney.com>
Message-ID: <4E956A10.5040806@hauke-lampe.de>

Hi Jeff.

On 12.10.2011 06:01, Stroomer, Jeff wrote:

> I've seen plenty of example vcl code that seems to say req.url will
contain /a/b/c.
> But it looks like I'm actually getting the full URL, i.e.,
> http://xyz.com/a/b/c.

If you see that in each and every request, I'd suspect another reverse
proxy or load balancer at your end.

I see this request format, too, occasionally, but I haven't tried to
figure out which browsers/bot actually cause it.

I use this code in vcl_recv() to normalize the URL:

# fix "GET http://host/path" request
# extract Host: header and rewrite URL
if (req.url ~ "(?i)^https?://") {
	set req.http.Host = regsub(req.url, "(?i)^https?://([^/]*).*", "\1");
	set req.url = regsub(req.url, "(?i)^https?://[^/]*/?(.*)$", "/\1");
}

( http://cfg.openchaos.org/varnish/vcl/3.x/common/normalize_http.vcl )


Hauke.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111012/a6978033/attachment.bin>

From sta at netimage.dk  Wed Oct 12 14:20:20 2011
From: sta at netimage.dk (=?ISO-8859-1?Q?S=F8ren_Thing_Andersen?=)
Date: Wed, 12 Oct 2011 16:20:20 +0200
Subject: Modifying ESI-requests based on original request's beresp
Message-ID: <4E95A224.90900@netimage.dk>

Hi.

I have a login-page with an ESI-block.

The login page issues a new session cookie so the original request's 
beresp contains a Set-Cookie header.
But the ESI-request to the server contains the OLD value of the cookie - 
or none at all if no cookie was present in the original request.

Is there a way to access the "parent" request in VCL when handling an 
ESI-request, so I can access the original beresp.http.Set-Cookie 
variable before making the ESI-request?

If not, can I do it with inline C?

Best regards
     S?ren Thing.



From deger at cobercafe.net  Wed Oct 12 16:52:25 2011
From: deger at cobercafe.net (Domnick Eger)
Date: Wed, 12 Oct 2011 11:52:25 -0500
Subject: Nginx & Varnish NFS
Message-ID: <196E6AD53BB39D458E3EC5D7F5109D4E1684E8E666@DFW1MBX07.mex07a.mlsrvr.com>

Hi Group,

We are starting to use Varnish, and noticed when some content that lives on NFS mounts don't actually works when Varnish is enabled. Anyone else have this issue ?

Thanks,

Domnick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111012/ce1908c9/attachment.html>

From jlevine at iwin.com  Wed Oct 12 21:31:36 2011
From: jlevine at iwin.com (Joshua Levine)
Date: Wed, 12 Oct 2011 14:31:36 -0700
Subject: Nginx & Varnish NFS
In-Reply-To: <196E6AD53BB39D458E3EC5D7F5109D4E1684E8E666@DFW1MBX07.mex07a.mlsrvr.com>
References: <196E6AD53BB39D458E3EC5D7F5109D4E1684E8E666@DFW1MBX07.mex07a.mlsrvr.com>
Message-ID: <2D221DD1-9F1A-4221-8D5C-FAB7603CDEF5@iwin.com>



On Oct 12, 2011, at 9:52 AM, Domnick Eger wrote:

> Hi Group,
>  
> We are starting to use Varnish, and noticed when some content that lives on NFS mounts don?t actually works when Varnish is enabled. Anyone else have this issue ?
>  
> Thanks,
>  
> Domnick.
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

I have not noticed the issue you report, however for NFS I note that we have nginx between varnish and an apache backend for some of our content, (routed by varnish). This allows apache to serve as the central management for most/all virtual hosting, rewrites, and expires headers, however for large files on NFS, it lets nginx manage local disk caching saving the majority of the NFS overhead.

Joshua
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111012/af7a9c0d/attachment.html>

From jhl at contentfleet.com  Thu Oct 13 07:42:58 2011
From: jhl at contentfleet.com (Jan-Hendrik Lendholt)
Date: Thu, 13 Oct 2011 09:42:58 +0200
Subject: Avoid ReWrite Rule
Message-ID: <4E969682.3080401@contentfleet.com>

Hi there,

we've got a backend sever (apache2, php5.2) and want to put varnish in 
front of it.

I installed varnish parallel to the running set-up.

Now apache runs on 80, I want varnish to run on 8080.

I set:

backend default {
      .host = "127.0.0.1";
      .port = "80";
}

when I call domain:8080, I just get a "301 moved permanently" and I get 
redirected to domain:80.

I saw some rewrite rules that forward all requests, not being on 
domain:80 or domain:80/some/content/here always to domain:80 in order to 
always show the landing page.

Is there a possibility to avoid that?

Thanks in advance!

Jan



From rtshilston at gmail.com  Thu Oct 13 09:31:47 2011
From: rtshilston at gmail.com (Rob S)
Date: Thu, 13 Oct 2011 10:31:47 +0100
Subject: OT: SSL handshake issues
Message-ID: <CAKF-8eBmxnvytr-V4RDc25xu6KgXWX017ApdL0UEyOnCsSj3xA@mail.gmail.com>

Hi,

Sorry that this is definitely off-topic, but I've got a strange
problem and I'm hoping someone on the list might be able to point me
in the right direction.  We've a web browser / server combination
which, comparatively in frequently (around 1 in 20 requests) when
sending the "Client Hello" SSL handshake is receiving an "Encrypted
Alert" response, rather than a "Server Hello" response.  I've got a
packet trace I can share demonstrating this, and budget to pay for
advice.  What we're trying to establish is if there's a server-side
fault (and if so, why it's intermittent) or if it's an invalid
client-side request.  We've not got the expertise in house, and are
looking to understand the situation quite rapidly, hence trying to ask
the community.

Many thanks,



Rob



From dan at retrobadger.net  Thu Oct 13 10:23:41 2011
From: dan at retrobadger.net (Dan)
Date: Thu, 13 Oct 2011 11:23:41 +0100
Subject: a single varnish, multiple sites, and a probe for each
Message-ID: <4E96BC2D.80501@retrobadger.net>

I am trying to setup varnish to allow for a healthcheck on each site 
even though multiple sites are on the same server. This would mean that 
if one sites probe.php failed, the other sites wouldn't go down (unless 
their probes also failed). I have tried to get started on this, but am 
finding it hard to get a reliable solution, I am thinking that if each 
site has it's own directer, with the director pointing to the same 
backend, then this should get the director to only fail if it's site is 
down

This assumption is of course not working, so I am trying to work out 
why, and whether there are any docs on this.  I have posted this onto 
the forum, but was advised that this may be a better place to post.  I 
have attached a link to the forum where there is also some code snippets

Link to forum post: https://www.varnish-cache.org/forum/topic/166

thankyou,
Dan





From thiago at cmoraes.com  Thu Oct 13 17:20:48 2011
From: thiago at cmoraes.com (Thiago Moraes)
Date: Thu, 13 Oct 2011 14:20:48 -0300
Subject: Caching a low quantity of huge files with Varnish
Message-ID: <CAOPp8Z9zv0UPkqGzUu6n5zevpVLN83LNjjpabBmcdFFLeWSbzg@mail.gmail.com>

I posted this on serverfault.com and haven't received any answers.
Would appreciate any help here:

I have one server that provides access to some hundreds of files. All
of them are really big (some have more than 10GB). These files won't
change and are read only, but I'll need to provide access to them via
a WAN.

I want to make the access faster by using a reverse proxy server
running near my users. For example:

-User X wants to access something on server A.
-User X access a reverse proxy server on his LAN which causes a cache
miss. The file is downloaded to this proxy server.
-The next time user X wants the same file, he doesn't need to go get
it in my main server.

I know that this situation is trivial, and I won't have a lot of
traffic, but my problem is more related to the file sizes themselves.
I read something about Squid having problems caching files larger than
2GB. Does Varnish face the same kind of problem?

Thank you for any thoughts on that.

Thiago Moraes - EnC 07 - UFSCar



From l at lrowe.co.uk  Thu Oct 13 17:53:58 2011
From: l at lrowe.co.uk (Laurence Rowe)
Date: Thu, 13 Oct 2011 18:53:58 +0100
Subject: Caching a low quantity of huge files with Varnish
In-Reply-To: <CAOPp8Z9zv0UPkqGzUu6n5zevpVLN83LNjjpabBmcdFFLeWSbzg@mail.gmail.com>
References: <CAOPp8Z9zv0UPkqGzUu6n5zevpVLN83LNjjpabBmcdFFLeWSbzg@mail.gmail.com>
Message-ID: <CAOycyLSMZonqw5UxyZrbo5rOrQectd=f58nKN21dJOi4cuA4RQ@mail.gmail.com>

On 13 October 2011 18:20, Thiago Moraes <thiago at cmoraes.com> wrote:
> I posted this on serverfault.com and haven't received any answers.
> Would appreciate any help here:
>
> I have one server that provides access to some hundreds of files. All
> of them are really big (some have more than 10GB). These files won't
> change and are read only, but I'll need to provide access to them via
> a WAN.
>
> I want to make the access faster by using a reverse proxy server
> running near my users. For example:
>
> -User X wants to access something on server A.
> -User X access a reverse proxy server on his LAN which causes a cache
> miss. The file is downloaded to this proxy server.
> -The next time user X wants the same file, he doesn't need to go get
> it in my main server.
>
> I know that this situation is trivial, and I won't have a lot of
> traffic, but my problem is more related to the file sizes themselves.
> I read something about Squid having problems caching files larger than
> 2GB. Does Varnish face the same kind of problem?
>
> Thank you for any thoughts on that.

I don't know of any specific size limits, but you will almost
certainly need the streaming support currently being developed at
https://github.com/mbgrydeland/varnish-cache-streaming. Without
streaming support the Varnish proxy will wait to download the entire
10GB file before returning any bytes downstream.

Laurence



From varnish at mm.quex.org  Fri Oct 14 08:06:17 2011
From: varnish at mm.quex.org (Michael Alger)
Date: Fri, 14 Oct 2011 16:06:17 +0800
Subject: a single varnish, multiple sites, and a probe for each
In-Reply-To: <4E96BC2D.80501@retrobadger.net>
References: <4E96BC2D.80501@retrobadger.net>
Message-ID: <20111014080617.GA27562@grum.quex.org>

On Thu, Oct 13, 2011 at 11:23:41AM +0100, Dan wrote:
> I am trying to setup varnish to allow for a healthcheck on each site
> even though multiple sites are on the same server. This would mean
> that if one sites probe.php failed, the other sites wouldn't go down
> (unless their probes also failed). I have tried to get started on
> this, but am finding it hard to get a reliable solution, I am
> thinking that if each site has it's own directer, with the director
> pointing to the same backend, then this should get the director to
> only fail if it's site is down

The probe is associated with the backend, not the director. Looking at
the configuration you posted on the forum, how do you imagine Varnish
would be able to determine that you have X number of sites all of
which have a probe.php and need to be monitored independently?

> This assumption is of course not working, so I am trying to work out
> why, and whether there are any docs on this.  I have posted this
> onto the forum, but was advised that this may be a better place to
> post.  I have attached a link to the forum where there is also some
> code snippets
> 
> Link to forum post: https://www.varnish-cache.org/forum/topic/166

Directors are used to group multiple backends for the same site
together for failover and load balancing purposes. If you only have
the one backend server for a particular site, you don't need to use
directors at all.

You want to use a separate backend definition for every site you host.
Each backend has its own health status which is updated using the
probe configured for that backend.

For your purposes, you probably want to use the alternate method of
specifying the probe request, which lets you control all the headers
sent in the probe. It'll look something like this:

  .probe = {
    .request =
      "GET /probe.php HTTP/1.0"
      "Host: www.virtualhost.example.com"
      "Connection: close";
    .interval = 5s;
    .timeout = 1 s;
  }

In this case, I'm assuming your backend does "name-based" virtual
hosting so you need to be able to send the appropriate Host: header in
order to check the correct site.

Just in case it's not clear: you can have multiple Varnish backends
configured for a single real server. They don't even need to be
different, although if they're identical they'll both get sick/healthy
at the same time so it won't serve much purpose. But, you might probe
a different path in order to test different parts of the site's
functionality.

As an example, if you have a complex site hosting a store and a web
forum on the same server, you might set up separate backends so you
can probe the store and forum separately, rather than e.g. having a
special probe page that checks both, or not checking the forum at all.
That way if the forum falls over, your store can still be available,
because your VCL directs the request to a different Varnish backend.

So, you're on the right track, but you want to be using separate
backends for each site, not separate directors. Directors should only
be used if the backends they reference are functionally identical.



From kristian at varnish-software.com  Fri Oct 14 09:03:17 2011
From: kristian at varnish-software.com (Kristian Lyngstol)
Date: Fri, 14 Oct 2011 11:03:17 +0200
Subject: Caching a low quantity of huge files with Varnish
In-Reply-To: <CAOPp8Z9zv0UPkqGzUu6n5zevpVLN83LNjjpabBmcdFFLeWSbzg@mail.gmail.com>
References: <CAOPp8Z9zv0UPkqGzUu6n5zevpVLN83LNjjpabBmcdFFLeWSbzg@mail.gmail.com>
Message-ID: <20111014090317.GB4766@freud.kly.no>

Greetings,

On Thu, Oct 13, 2011 at 02:20:48PM -0300, Thiago Moraes wrote:
> I have one server that provides access to some hundreds of files. All
> of them are really big (some have more than 10GB). These files won't
> change and are read only, but I'll need to provide access to them via
> a WAN.
> 
> I want to make the access faster by using a reverse proxy server
> running near my users. For example:
> 
> -User X wants to access something on server A.
> -User X access a reverse proxy server on his LAN which causes a cache
> miss. The file is downloaded to this proxy server.
> -The next time user X wants the same file, he doesn't need to go get
> it in my main server.

My first thought is why you need a reverse proxy and not a regular one.

> I read something about Squid having problems caching files larger than
> 2GB. Does Varnish face the same kind of problem?

Short answer: No.

From Varnish 3, we do regular tests to see that files bigger than 4GB
are dealt with sensibly. 2/4GB being the barrier, and beyond that, the
logical limit is your memory manager/hardware/network/client, etc. 10GB
/shouldn't/ be a big problem.

But keep in mind:

- You need a 64bit system.
- Use the latest Varnish version. 3.0.1 or even 3.0.2 when that
  finishes. 2.1 is not recommended.
- If your users will access these files concurrently, you will really
  want to try out Martin's streaming branch, as that will ensure that
  multiple clients get the content as it arrives at the cache. Without
  Martin's branch, the first user that gets a cache miss will get the
  content "streamed" while Varnish downloads it, while other users have
  to wait for the download to finish before they get any data.
- If your data is compressible, make sure you use gzip. Even if your
  clients don't support it (e.g: scripts?), Varnish does. If it isn't
  compressible, consider disabling it.
- Make sure the cache is big enough, or that evictions are
  well-controlled. If you're mixing small and large files, you might end
  up evicting hundreds of small files to fit room for a single large
  one.

With all of that said, I'd like to emphasis that support for large
files is relatively new. I expect there's room for optimizations.

The less used features are also the less optimized features. As a
consequence of the lack of feedback.

I would greatly appreciate any feedback you have if/when you test this.

- Kristian

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: Digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111014/c886b9cc/attachment.bin>

From thiago at cmoraes.com  Sat Oct 15 18:50:22 2011
From: thiago at cmoraes.com (Thiago Moraes)
Date: Sat, 15 Oct 2011 15:50:22 -0300
Subject: Caching a low quantity of huge files with Varnish
In-Reply-To: <20111014090317.GB4766@freud.kly.no>
References: <CAOPp8Z9zv0UPkqGzUu6n5zevpVLN83LNjjpabBmcdFFLeWSbzg@mail.gmail.com>
	<20111014090317.GB4766@freud.kly.no>
Message-ID: <CAOPp8Z_SczLhz5syTc9-ba58HYe_ufdshBzkSnTCMS31phP3Ng@mail.gmail.com>

Hey Kristian,

Thanks for your answer, it was really clarifying.

2011/10/14 Kristian Lyngstol <kristian at varnish-software.com>

> Greetings,
>
> On Thu, Oct 13, 2011 at 02:20:48PM -0300, Thiago Moraes wrote:
> > I have one server that provides access to some hundreds of files. All
> > of them are really big (some have more than 10GB). These files won't
> > change and are read only, but I'll need to provide access to them via
> > a WAN.
> >
> > I want to make the access faster by using a reverse proxy server
> > running near my users. For example:
> >
> > -User X wants to access something on server A.
> > -User X access a reverse proxy server on his LAN which causes a cache
> > miss. The file is downloaded to this proxy server.
> > -The next time user X wants the same file, he doesn't need to go get
> > it in my main server.
>
> My first thought is why you need a reverse proxy and not a regular one.
>

I'm not really experienced in this kind of work so the choice was made based
on what I read about cache servers in the past weeks. As the server would
only cache data from my application, it appeared to me that the reverse
proxy alternative would be less "intrusive". I don't have the power to
change the network topology from my users so I don't want to redirect all
their traffic through my proxy. Probably, there already exists a proxy
server in their networks. Am I missing something with this approach? Is it
necessary?


>
> > I read something about Squid having problems caching files larger than
> > 2GB. Does Varnish face the same kind of problem?
>
> Short answer: No.
>
> From Varnish 3, we do regular tests to see that files bigger than 4GB
> are dealt with sensibly. 2/4GB being the barrier, and beyond that, the
> logical limit is your memory manager/hardware/network/client, etc. 10GB
> /shouldn't/ be a big problem.
>
> But keep in mind:
>
> - You need a 64bit system.
> - Use the latest Varnish version. 3.0.1 or even 3.0.2 when that
>  finishes. 2.1 is not recommended.
>

As the project is new I'll use the current stable version, probably. Are
there any advantages for me to use 3.0.2 right now? How's the stability of
the developer's version? Would you recommendo to start already using it?


> - If your users will access these files concurrently, you will really
>  want to try out Martin's streaming branch, as that will ensure that
>  multiple clients get the content as it arrives at the cache. Without
>  Martin's branch, the first user that gets a cache miss will get the
>  content "streamed" while Varnish downloads it, while other users have
>  to wait for the download to finish before they get any data.

- If your data is compressible, make sure you use gzip. Even if your
>  clients don't support it (e.g: scripts?), Varnish does. If it isn't
>  compressible, consider disabling it.
>

I don't really know how to answer that, but I'll ask you something about
this anyway=p.
My application is a server with a REST interface to a bunch of scientific
data files. The interface allows you to slice the data and to get only a
subset of it transparently. As the server was developed by someone else, I
don't know how it handles data compression, but as it is possible to access
data through some really ugly applications, I believe it's not possible to
use compression due to the clients.

How would varnish compression work? It would send compressed data to the
cache server and them uncompress it before sending to the clients?


> - Make sure the cache is big enough, or that evictions are
>  well-controlled. If you're mixing small and large files, you might end
>  up evicting hundreds of small files to fit room for a single large
>  one.
>

I'll have mostly large files. There are some little ones with just a few MB,
but they're the exception in my case.


>
> With all of that said, I'd like to emphasis that support for large
> files is relatively new. I expect there's room for optimizations.
>
> The less used features are also the less optimized features. As a
> consequence of the lack of feedback.
>
> I would greatly appreciate any feedback you have if/when you test this.
>
> - Kristian
>
>
I'll provide some feedback when I get it. I'm learning how to configure
varnish right now and figuring out some way of testing if my requisites are
fulfilled. By the way, would you recommend some way to test the server? Are
there any automated tool to do this kind of test?

Thank you very much,

Thiago Moraes - EnC07 - UFSCar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111015/654ae27e/attachment.html>

From jeroen.ooms at stat.ucla.edu  Tue Oct 18 20:09:28 2011
From: jeroen.ooms at stat.ucla.edu (Jeroen Ooms)
Date: Tue, 18 Oct 2011 13:09:28 -0700
Subject: beresp.cacheable on Varnish 3.
Message-ID: <CABFfbXsnduAL2-=dH-f4jVS57rgWok42eyyz-W7ek+Gz0hJMog@mail.gmail.com>

I am using varnish among other things to cache a wordpress site. I just
upgraded to Ubuntu 11.10, and thereby also the default varnish is updated
from 2.1. to 2.2. However I'm getting an error now: Unknown variable
'beresp.cacheable'.

Basically the problem is that I want to enforce caching of all content,
except for stuff under /wp-login and /wp-admin. Furthermore it should also
cache HTTP 400 responses. Below the code that I grabbed from somewhere I
don't remember, that did the job in Varnish 2.1. How should I update this
now beresp.cacheable is no longer available?

Thanks,

sub vcl_fetch {
   if (!(req.url ~ "wp-(login|admin)")) {
      unset beresp.http.set-cookie;
      set beresp.ttl = 12h;
      set beresp.cacheable = true;
      set beresp.http.cache-control = "public";
   }
   if ( beresp.status == 400 ) {
      set beresp.ttl = 1h;
      set beresp.cacheable = true;
      set beresp.http.cache-control = "public";
   }
   if (!beresp.cacheable) {
      return (pass);
   }
   if (beresp.http.Set-Cookie) {
      return (pass);
   }
   return (deliver);
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111018/704cb335/attachment.html>

From jeroen.ooms at stat.ucla.edu  Tue Oct 18 20:12:47 2011
From: jeroen.ooms at stat.ucla.edu (Jeroen Ooms)
Date: Tue, 18 Oct 2011 13:12:47 -0700
Subject: beresp.cacheable on Varnish 3.
In-Reply-To: <CABFfbXsnduAL2-=dH-f4jVS57rgWok42eyyz-W7ek+Gz0hJMog@mail.gmail.com>
References: <CABFfbXsnduAL2-=dH-f4jVS57rgWok42eyyz-W7ek+Gz0hJMog@mail.gmail.com>
Message-ID: <CABFfbXt3=299KTH1nvMuYezBcEdq=+kqbd5XVV7M9iGt1NKhaQ@mail.gmail.com>

Apologies, the varnish I am now running is not version 2.2 but version 3.


On Tue, Oct 18, 2011 at 1:09 PM, Jeroen Ooms <jeroen.ooms at stat.ucla.edu>wrote:

> I am using varnish among other things to cache a wordpress site. I just
> upgraded to Ubuntu 11.10, and thereby also the default varnish is updated
> from 2.1. to 2.2. However I'm getting an error now: Unknown variable
> 'beresp.cacheable'.
>
> Basically the problem is that I want to enforce caching of all content,
> except for stuff under /wp-login and /wp-admin. Furthermore it should also
> cache HTTP 400 responses. Below the code that I grabbed from somewhere I
> don't remember, that did the job in Varnish 2.1. How should I update this
> now beresp.cacheable is no longer available?
>
> Thanks,
>
> sub vcl_fetch {
>    if (!(req.url ~ "wp-(login|admin)")) {
>       unset beresp.http.set-cookie;
>       set beresp.ttl = 12h;
>       set beresp.cacheable = true;
>       set beresp.http.cache-control = "public";
>    }
>    if ( beresp.status == 400 ) {
>       set beresp.ttl = 1h;
>       set beresp.cacheable = true;
>       set beresp.http.cache-control = "public";
>    }
>    if (!beresp.cacheable) {
>       return (pass);
>    }
>    if (beresp.http.Set-Cookie) {
>       return (pass);
>    }
>    return (deliver);
> }
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111018/93ae3511/attachment.html>

From bedis9 at gmail.com  Wed Oct 19 04:52:06 2011
From: bedis9 at gmail.com (Baptiste)
Date: Wed, 19 Oct 2011 06:52:06 +0200
Subject: beresp.cacheable on Varnish 3.
In-Reply-To: <CABFfbXt3=299KTH1nvMuYezBcEdq=+kqbd5XVV7M9iGt1NKhaQ@mail.gmail.com>
References: <CABFfbXsnduAL2-=dH-f4jVS57rgWok42eyyz-W7ek+Gz0hJMog@mail.gmail.com>
	<CABFfbXt3=299KTH1nvMuYezBcEdq=+kqbd5XVV7M9iGt1NKhaQ@mail.gmail.com>
Message-ID: <CAODHi7qaYNXyZF5_FOBG=fO2BggYV7kjMUeE7e5nR_nvtSHGkg@mail.gmail.com>

On Tue, Oct 18, 2011 at 10:12 PM, Jeroen Ooms <jeroen.ooms at stat.ucla.edu> wrote:
> Apologies, the varnish I am now running is not version 2.2 but version 3.
>
> On Tue, Oct 18, 2011 at 1:09 PM, Jeroen Ooms <jeroen.ooms at stat.ucla.edu>
> wrote:
>>
>> I am using varnish among other things to cache a wordpress site. I just
>> upgraded to Ubuntu 11.10, and thereby also the default varnish is updated
>> from 2.1. to 2.2. However I'm getting an error now: Unknown variable
>> 'beresp.cacheable'.
>> Basically the problem is that I want to enforce caching of all content,
>> except for stuff under /wp-login and /wp-admin. Furthermore it should also
>> cache HTTP 400 responses.?Below the code that I grabbed from somewhere I
>> don't remember, that did the job in Varnish 2.1. How should I update this
>> now beresp.cacheable is no longer available?
>> Thanks,
>> sub vcl_fetch {
>> ? ?if (!(req.url ~ "wp-(login|admin)")) {
>> ? ? ? unset beresp.http.set-cookie;
>> ? ? ??set beresp.ttl = 12h;
>> ? ? ??set beresp.cacheable = true;
>> ? ? ??set beresp.http.cache-control = "public";
>> ? ?}
>> ? ?if ( beresp.status == 400 ) {
>> ? ?? ?set beresp.ttl = 1h;
>> ? ?? ?set beresp.cacheable = true;
>> ? ?? ?set beresp.http.cache-control = "public";
>> ? ?}
>> ? ?if (!beresp.cacheable) {
>> ? ?? ?return (pass);
>> ? ?}
>> ? ?if (beresp.http.Set-Cookie) {
>> ? ?? ?return (pass);
>> ? ?}
>> ? ?return (deliver);
>> }
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

Hi,

This is what I do for Wordpress in vcl_fetch on varcnish 3 (afair, it
worked in 2.1):

        set beresp.grace = 1d;

        # wordpress
        if (req.http.host ~ "XXXXXXXX" && !(req.url ~ "wp-(login|admin)")) {
                unset beresp.http.set-cookie;
                set beresp.cacheable = true;
        }

        if (beresp.cacheable && !beresp.http.cache-control) {
                set beresp.ttl = 600s;
                set beresp.http.cache-control = "max-age=600";
        }


and 404 are cached.

Cheers



From Harri.Paivaniemi at tieto.com  Wed Oct 19 05:01:50 2011
From: Harri.Paivaniemi at tieto.com (Harri.Paivaniemi at tieto.com)
Date: Wed, 19 Oct 2011 08:01:50 +0300
Subject: beresp.cacheable on Varnish 3.
In-Reply-To: <CAODHi7qaYNXyZF5_FOBG=fO2BggYV7kjMUeE7e5nR_nvtSHGkg@mail.gmail.com>
References: <CABFfbXsnduAL2-=dH-f4jVS57rgWok42eyyz-W7ek+Gz0hJMog@mail.gmail.com>
	<CABFfbXt3=299KTH1nvMuYezBcEdq=+kqbd5XVV7M9iGt1NKhaQ@mail.gmail.com>,
	<CAODHi7qaYNXyZF5_FOBG=fO2BggYV7kjMUeE7e5nR_nvtSHGkg@mail.gmail.com>
Message-ID: <F5A4C3A7AC59A74FB0A0D35F1E8BDB462C9056C2A4@EXMB03.eu.tieto.com>

Hi,

At least I remember that in vcl 3, besp.cacheable is gone and should use "set beresp.ttl = 0s;" to say it's not cacheable etc.

-hjp

________________________________________
From: varnish-misc-bounces at varnish-cache.org [varnish-misc-bounces at varnish-cache.org] On Behalf Of Baptiste [bedis9 at gmail.com]
Sent: 19 October 2011 7:52
To: Jeroen Ooms
Cc: varnish-misc at varnish-cache.org
Subject: Re: beresp.cacheable on Varnish 3.

On Tue, Oct 18, 2011 at 10:12 PM, Jeroen Ooms <jeroen.ooms at stat.ucla.edu> wrote:

> Apologies, the varnish I am now running is not version 2.2 but version 3.
>
> On Tue, Oct 18, 2011 at 1:09 PM, Jeroen Ooms <jeroen.ooms at stat.ucla.edu>
> wrote:
>>
>> I am using varnish among other things to cache a wordpress site. I just
>> upgraded to Ubuntu 11.10, and thereby also the default varnish is updated
>> from 2.1. to 2.2. However I'm getting an error now: Unknown variable
>> 'beresp.cacheable'.
>> Basically the problem is that I want to enforce caching of all content,
>> except for stuff under /wp-login and /wp-admin. Furthermore it should also
>> cache HTTP 400 responses. Below the code that I grabbed from somewhere I
>> don't remember, that did the job in Varnish 2.1. How should I update this
>> now beresp.cacheable is no longer available?
>> Thanks,
>> sub vcl_fetch {
>>    if (!(req.url ~ "wp-(login|admin)")) {
>>       unset beresp.http.set-cookie;
>>       set beresp.ttl = 12h;
>>       set beresp.cacheable = true;
>>       set beresp.http.cache-control = "public";
>>    }
>>    if ( beresp.status == 400 ) {
>>       set beresp.ttl = 1h;
>>       set beresp.cacheable = true;
>>       set beresp.http.cache-control = "public";
>>    }
>>    if (!beresp.cacheable) {
>>       return (pass);
>>    }
>>    if (beresp.http.Set-Cookie) {
>>       return (pass);
>>    }
>>    return (deliver);
>> }
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

Hi,

This is what I do for Wordpress in vcl_fetch on varcnish 3 (afair, it
worked in 2.1):

        set beresp.grace = 1d;

        # wordpress
        if (req.http.host ~ "XXXXXXXX" && !(req.url ~ "wp-(login|admin)")) {
                unset beresp.http.set-cookie;
                set beresp.cacheable = true;
        }

        if (beresp.cacheable && !beresp.http.cache-control) {
                set beresp.ttl = 600s;
                set beresp.http.cache-control = "max-age=600";
        }


and 404 are cached.

Cheers

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From bedis9 at gmail.com  Wed Oct 19 07:26:12 2011
From: bedis9 at gmail.com (Baptiste)
Date: Wed, 19 Oct 2011 09:26:12 +0200
Subject: beresp.cacheable on Varnish 3.
In-Reply-To: <F5A4C3A7AC59A74FB0A0D35F1E8BDB462C9056C2A4@EXMB03.eu.tieto.com>
References: <CABFfbXsnduAL2-=dH-f4jVS57rgWok42eyyz-W7ek+Gz0hJMog@mail.gmail.com>
	<CABFfbXt3=299KTH1nvMuYezBcEdq=+kqbd5XVV7M9iGt1NKhaQ@mail.gmail.com>
	<CAODHi7qaYNXyZF5_FOBG=fO2BggYV7kjMUeE7e5nR_nvtSHGkg@mail.gmail.com>
	<F5A4C3A7AC59A74FB0A0D35F1E8BDB462C9056C2A4@EXMB03.eu.tieto.com>
Message-ID: <CAODHi7ogSGJgRPFPre=oErvQMZs36TVa5dBETByS350U=2nHew@mail.gmail.com>

ah, right, I thought I upgraded my varnish, but obviously I'm still
running "varnish-2.1.5 SVN".
I'll upgrade it very soon and share with you the impact on the configuration.

cheers


On Wed, Oct 19, 2011 at 7:01 AM,  <Harri.Paivaniemi at tieto.com> wrote:
> Hi,
>
> At least I remember that in vcl 3, besp.cacheable is gone and should use "set beresp.ttl = 0s;" to say it's not cacheable etc.
>
> -hjp
>
> ________________________________________
> From: varnish-misc-bounces at varnish-cache.org [varnish-misc-bounces at varnish-cache.org] On Behalf Of Baptiste [bedis9 at gmail.com]
> Sent: 19 October 2011 7:52
> To: Jeroen Ooms
> Cc: varnish-misc at varnish-cache.org
> Subject: Re: beresp.cacheable on Varnish 3.
>
> On Tue, Oct 18, 2011 at 10:12 PM, Jeroen Ooms <jeroen.ooms at stat.ucla.edu> wrote:
>
>> Apologies, the varnish I am now running is not version 2.2 but version 3.
>>
>> On Tue, Oct 18, 2011 at 1:09 PM, Jeroen Ooms <jeroen.ooms at stat.ucla.edu>
>> wrote:
>>>
>>> I am using varnish among other things to cache a wordpress site. I just
>>> upgraded to Ubuntu 11.10, and thereby also the default varnish is updated
>>> from 2.1. to 2.2. However I'm getting an error now: Unknown variable
>>> 'beresp.cacheable'.
>>> Basically the problem is that I want to enforce caching of all content,
>>> except for stuff under /wp-login and /wp-admin. Furthermore it should also
>>> cache HTTP 400 responses. Below the code that I grabbed from somewhere I
>>> don't remember, that did the job in Varnish 2.1. How should I update this
>>> now beresp.cacheable is no longer available?
>>> Thanks,
>>> sub vcl_fetch {
>>> ? ?if (!(req.url ~ "wp-(login|admin)")) {
>>> ? ? ? unset beresp.http.set-cookie;
>>> ? ? ? set beresp.ttl = 12h;
>>> ? ? ? set beresp.cacheable = true;
>>> ? ? ? set beresp.http.cache-control = "public";
>>> ? ?}
>>> ? ?if ( beresp.status == 400 ) {
>>> ? ? ? set beresp.ttl = 1h;
>>> ? ? ? set beresp.cacheable = true;
>>> ? ? ? set beresp.http.cache-control = "public";
>>> ? ?}
>>> ? ?if (!beresp.cacheable) {
>>> ? ? ? return (pass);
>>> ? ?}
>>> ? ?if (beresp.http.Set-Cookie) {
>>> ? ? ? return (pass);
>>> ? ?}
>>> ? ?return (deliver);
>>> }
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
> Hi,
>
> This is what I do for Wordpress in vcl_fetch on varcnish 3 (afair, it
> worked in 2.1):
>
> ? ? ? ?set beresp.grace = 1d;
>
> ? ? ? ?# wordpress
> ? ? ? ?if (req.http.host ~ "XXXXXXXX" && !(req.url ~ "wp-(login|admin)")) {
> ? ? ? ? ? ? ? ?unset beresp.http.set-cookie;
> ? ? ? ? ? ? ? ?set beresp.cacheable = true;
> ? ? ? ?}
>
> ? ? ? ?if (beresp.cacheable && !beresp.http.cache-control) {
> ? ? ? ? ? ? ? ?set beresp.ttl = 600s;
> ? ? ? ? ? ? ? ?set beresp.http.cache-control = "max-age=600";
> ? ? ? ?}
>
>
> and 404 are cached.
>
> Cheers
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



From aja at danskespil.dk  Wed Oct 19 08:43:50 2011
From: aja at danskespil.dk (Allan Jacobsen)
Date: Wed, 19 Oct 2011 08:43:50 +0000
Subject: Problem with custom error page
Message-ID: <C44D4F7FBB8AAA418EB0EA8E249125C7279E9C@WILLIAMS2.danskespil.lokal>

Hi

I am making my own 503 error page, at most things work fine, but I would like to tell the viewer which of the 2 backends it is that gives the 503, and i can't seem to find the right variable for this. I have tried req.backend which gives (null) and some others, but no luck.


Best regards/MVH
Allan Jacobsen
Infrastruktur, IT-teknik, Danske Spil A/S
Korsdalsvej 135, 2605 Br?ndby




From audun at ytterdal.net  Wed Oct 19 08:57:42 2011
From: audun at ytterdal.net (Audun Ytterdal)
Date: Wed, 19 Oct 2011 10:57:42 +0200
Subject: Problem with custom error page
In-Reply-To: <C44D4F7FBB8AAA418EB0EA8E249125C7279E9C@WILLIAMS2.danskespil.lokal>
References: <C44D4F7FBB8AAA418EB0EA8E249125C7279E9C@WILLIAMS2.danskespil.lokal>
Message-ID: <CAKBjT_7vD5qjv4LMKy4mJZuXUseY4SLuPGTVvTsJkNkuq8HTfA@mail.gmail.com>

On Wed, Oct 19, 2011 at 10:43 AM, Allan Jacobsen <aja at danskespil.dk> wrote:
> Hi
>
> I am making my own 503 error page, at most things work fine, but I would like to tell the viewer which of the 2 backends it is that gives the 503, and i can't seem to find the right variable for this. I have tried req.backend which gives (null) and some others, but no luck.

req.backend should be ok. I'm using
                                        <li>URL: http://"} +
req.http.host + req.url + {"</li>
                                        <li>Referer: "} +
req.http.referer + {"</li>
                                        <li>Request: "} + req.request
+ {"</li>
                                        <li>Time: "} +
obj.http.x-vg-utc + {"</li>
                                        <li>Client: "} + client.ip +
{"</li>
                                        <li>User-Agent: "} +
req.http.user-agent + {"</li>
                                        <li>X-VG-Device: "} +
req.http.X-VG-Device + {"</li>
                                        <li>Cache: "} + server.ip +
{"</li>
                                        <li>Backend: "} + req.backend
+ {"</li>
In my error-page

Varnish defaults backend is the first backend you list in your config.
If you don't set it explict it will use the default and the
req.backend will probably be "null" so my first backend is

backend unassigned {
        # Wildbill
	.host = "10.84.200.5";
        .port = "80";
}

And set it all backends expliclty in the vcl_recv.

-- 
Audun Ytterdal
http://audun.ytterdal.net



From audun at ytterdal.net  Wed Oct 19 09:25:13 2011
From: audun at ytterdal.net (Audun Ytterdal)
Date: Wed, 19 Oct 2011 11:25:13 +0200
Subject: varnishlog and logging errors
Message-ID: <CAKBjT_64JXeh9uxVQ7VdwWcfZE70XLH607_gBXFU8Lo1YXpqBw@mail.gmail.com>

Hi list

I'm trying to figure out the best way of logging errors in varnish to
a logfile. The scenario am looking at is when a client get some HTTP
5XX message i want to know as much as possible about that error. The
cause of the error could be many things

1) Backend times out on connect
2) Backend times out on time to first byte
3) Backend times out on time between bytes
4) Backend failes with 503 from apache/php
5) Backend closed the connection prematurly
6) Backend did something funky
7) Client did something funky
8) Varnish vcl is configured to send a error because of some reason
(detect attacks)

And so on.

My alternatives are varnishlog and varnishncsa.

varnishlog -c  -m "TxStatus:^5" works, but vil only log the
varnish<->client data

varnishlog -b  -m "RxStatus:^5" log varnsh<->server data, but not client

removing -b and -c gives all data, but then you are flooded with extra
data that I have to filter

varnishlog   -m "TxStatus:404" |egrep -v '(SessionClose|StatSess|ExpKill)'

But here I can't log TxStatus or RxStatus and it seems to log alot of
TxStatus: 200 for some reason.

Has anyone looked into such type of logging using varnishlog or varnishnsca?

When varnish failes, is it a way to figure out if it was 1) 2) or 3)
for example?





-- 
Audun Ytterdal
http://audun.ytterdal.net



From mbrouwer at songteksten.nl  Wed Oct 19 15:29:02 2011
From: mbrouwer at songteksten.nl (Maikel Brouwer - Songteksten.nl)
Date: Wed, 19 Oct 2011 17:29:02 +0200
Subject: Multi country cache problem
Message-ID: <1319038142.2178.10.camel@Maikel-Laptop-Werk>

Hi,

I've got the following problem. Maybe someone has a solution for it. My
site is visited by two countries (Belgium and Netherlands). I have two
different advertisement agencies to display ads for the specific
countries. This means that every page how something like this:

if ($country == 'be') {
  //display stuff
} else {
  //display other stuff)
}

The problem is that when someone from 'be' visits the page when it's not
yet cached the code in the 'be' block will be cached. When someone from
'nl' visits the page it will see the 'be' block.

I have three of those ad blocks on every page, with 500k+ pages (dynamic
content).

Thanks,
Maikel






From hugo.cisneiros at gmail.com  Wed Oct 19 17:15:43 2011
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Wed, 19 Oct 2011 15:15:43 -0200
Subject: Multi country cache problem
In-Reply-To: <1319038142.2178.10.camel@Maikel-Laptop-Werk>
References: <1319038142.2178.10.camel@Maikel-Laptop-Werk>
Message-ID: <CA+KACLkniB+rJfYpVs6iTEaSdGA2xvEOEo-H0NJQV+b_VQ2nPA@mail.gmail.com>

On Wed, Oct 19, 2011 at 1:29 PM, Maikel Brouwer - Songteksten.nl
<mbrouwer at songteksten.nl> wrote:
> I've got the following problem. Maybe someone has a solution for it. My
> site is visited by two countries (Belgium and Netherlands). I have two
> different advertisement agencies to display ads for the specific
> countries. This means that every page how something like this:
>
> if ($country == 'be') {
> ?//display stuff
> } else {
> ?//display other stuff)
> }
>
> The problem is that when someone from 'be' visits the page when it's not
> yet cached the code in the 'be' block will be cached. When someone from
> 'nl' visits the page it will see the 'be' block.

You can modify the vcl_hash to cache content based on client's ip.

For example, you can create an acl with the IP ranges from two countries:

acl country1 {
   "192.168.0.0"/24;
}

acl country2 {
   "192.168.1.0"/24;
}

Then on vcl_hash:

sub vcl_hash {
   hash_data(req.url);

   if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

   if (client.ip ~ country1) {
      hash_data("country1");
   } elseif (client.ip ~ country2) {
      hash_data("country2");
   }
}

This would create TWO cache entries (it will duplicate your cache
size), one for country1 and one for country2. Actually, if the user
isn't from country1 and country2, it'll create another cache entry for
the rest (else). You can change this logic as you wish. The hash_data
is the key to create multiple cache entries depending on any usual
variable on the vcl.

--
[]'s
Hugo
www.devin.com.br



From hugo.cisneiros at gmail.com  Wed Oct 19 17:25:58 2011
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Wed, 19 Oct 2011 15:25:58 -0200
Subject: varnishlog and logging errors
In-Reply-To: <CAKBjT_64JXeh9uxVQ7VdwWcfZE70XLH607_gBXFU8Lo1YXpqBw@mail.gmail.com>
References: <CAKBjT_64JXeh9uxVQ7VdwWcfZE70XLH607_gBXFU8Lo1YXpqBw@mail.gmail.com>
Message-ID: <CA+KACLkwGf39gkn8j1Wmr=aX61XkctkP6YD28UbYCCi8z++OnQ@mail.gmail.com>

On Wed, Oct 19, 2011 at 7:25 AM, Audun Ytterdal <audun at ytterdal.net> wrote:
> Hi list
>
> I'm trying to figure out the best way of logging errors in varnish to
> a logfile. The scenario am looking at is when a client get some HTTP
> 5XX message i want to know as much as possible about that error. The
> cause of the error could be many things
[...]
> My alternatives are varnishlog and varnishncsa.

I use syslog to do it. When a request goes to vcl_error, I log it,
including the status of the object for example. It's a bit more
difficult to do (since it uses C variables, not VCL language ones),
but works very fine.

For example:

In the beginning of default.vcl:

C{
        #include <syslog.h>
}C

Then on vcl_error:

if (obj.status >= 500) {
C{
                syslog(LOG_INFO, "Error (%s) (%s) (%s)",
VRT_r_req_url(sp), VRT_r_obj_response(sp), VRT_r_req_xid(sp));
}C
}

References:

https://www.varnish-cache.org/trac/wiki/VCLExampleSyslog

-- 
[]'s
Hugo
www.devin.com.br



From hugo.cisneiros at gmail.com  Wed Oct 19 17:30:36 2011
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Wed, 19 Oct 2011 15:30:36 -0200
Subject: beresp.cacheable on Varnish 3.
In-Reply-To: <CABFfbXsnduAL2-=dH-f4jVS57rgWok42eyyz-W7ek+Gz0hJMog@mail.gmail.com>
References: <CABFfbXsnduAL2-=dH-f4jVS57rgWok42eyyz-W7ek+Gz0hJMog@mail.gmail.com>
Message-ID: <CA+KACLm_cj2V5ZCtQxsrHXZPYA-A1FJauUQje_nx7dFYx80qPg@mail.gmail.com>

On Tue, Oct 18, 2011 at 6:09 PM, Jeroen Ooms <jeroen.ooms at stat.ucla.edu> wrote:
> I am using varnish among other things to cache a wordpress site. I just
> upgraded to Ubuntu 11.10, and thereby also the default varnish is updated
> from 2.1. to 2.2. However I'm getting an error now: Unknown variable
> 'beresp.cacheable'.

For a more detailed listing of changes from 2.x to 3, see:

https://www.varnish-cache.org/trac/browser/doc/changes.rst
Section: Changes from 2.1.5 to 3.0 beta 1 -> VCL

Many things changed.

Here I had to modify these items:

- return(pass) on vcl_fetch is now return(hit_for_pass)
- For concatenating strings, you must now put a "+" between them.
- obj.cacheable and beresp.cacheable are gone. Must change to *.ttl = 0s
- purge() function is now named ban()

-- 
[]'s
Hugo
www.devin.com.br



From lampe at hauke-lampe.de  Wed Oct 19 17:39:55 2011
From: lampe at hauke-lampe.de (Hauke Lampe)
Date: Wed, 19 Oct 2011 19:39:55 +0200
Subject: Multi country cache problem
In-Reply-To: <1319038142.2178.10.camel@Maikel-Laptop-Werk>
References: <1319038142.2178.10.camel@Maikel-Laptop-Werk>
Message-ID: <4E9F0B6B.6010008@hauke-lampe.de>

On 19.10.2011 17:29, Maikel Brouwer - Songteksten.nl wrote:

> if ($country == 'be') {
>   //display stuff
> } else {
>   //display other stuff)
> }
> 
> The problem is that when someone from 'be' visits the page when it's not
> yet cached the code in the 'be' block will be cached. When someone from
> 'nl' visits the page it will see the 'be' block.

I'd probably resolve the GeoIP country in VCL and set an HTTP header:
| set req.http.X-Country-Code = geoip.client_country_code();

The webserver then would have to add a "Vary: X-Country-Code" header to
country-specific pages. E.g. in apache:
| Header merge Vary X-Country-Code

varnish would cache one copy of each object for every country code, so
you might want to reduce the additional header to "be-or-not-be".

A better option could be to use ESI and cache the HTML code for pages
and ads separately. I haven't tested ESI with varnish yet, so I cannot
give an example here.


Hauke.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111019/97df77d5/attachment.bin>

From lampe at hauke-lampe.de  Wed Oct 19 17:49:39 2011
From: lampe at hauke-lampe.de (Hauke Lampe)
Date: Wed, 19 Oct 2011 19:49:39 +0200
Subject: varnishlog and logging errors
In-Reply-To: <CA+KACLkwGf39gkn8j1Wmr=aX61XkctkP6YD28UbYCCi8z++OnQ@mail.gmail.com>
References: <CAKBjT_64JXeh9uxVQ7VdwWcfZE70XLH607_gBXFU8Lo1YXpqBw@mail.gmail.com>
	<CA+KACLkwGf39gkn8j1Wmr=aX61XkctkP6YD28UbYCCi8z++OnQ@mail.gmail.com>
Message-ID: <4E9F0DB3.40801@hauke-lampe.de>

On 19.10.2011 19:25, Hugo Cisneiros (Eitch) wrote:

> I use syslog to do it. When a request goes to vcl_error, I log it,
> including the status of the object for example. It's a bit more
> difficult to do (since it uses C variables, not VCL language ones),
> but works very fine.

varnish 3.x implements a syslog function via the std vmod.

|import std;
|
|vcl_error {
|  if (obj.status >= 500) {
|    std.syslog(180, "Error (" + req.url + ") (" + obj.response + ") ("
+ req.xid + ")");
|  }
|}

I couldn't find the LOG_* constants defined in the VCL scope, so I use
integer literals here.

References:
https://www.varnish-cache.org/docs/trunk/reference/vmod_std.html



Hauke.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111019/2295e985/attachment.bin>

From hugo.cisneiros at gmail.com  Wed Oct 19 17:59:22 2011
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Wed, 19 Oct 2011 15:59:22 -0200
Subject: varnishlog and logging errors
In-Reply-To: <4E9F0DB3.40801@hauke-lampe.de>
References: <CAKBjT_64JXeh9uxVQ7VdwWcfZE70XLH607_gBXFU8Lo1YXpqBw@mail.gmail.com>
	<CA+KACLkwGf39gkn8j1Wmr=aX61XkctkP6YD28UbYCCi8z++OnQ@mail.gmail.com>
	<4E9F0DB3.40801@hauke-lampe.de>
Message-ID: <CA+KACLnrei0dTUXPrJLPWVx0-YebZcyxT+SFWwfOTz9U1sZSGA@mail.gmail.com>

On Wed, Oct 19, 2011 at 3:49 PM, Hauke Lampe <lampe at hauke-lampe.de> wrote:
> On 19.10.2011 19:25, Hugo Cisneiros (Eitch) wrote:
>
>> I use syslog to do it. When a request goes to vcl_error, I log it,
>> including the status of the object for example. It's a bit more
>> difficult to do (since it uses C variables, not VCL language ones),
>> but works very fine.
>
> varnish 3.x implements a syslog function via the std vmod.

That's very good to know! I will test it ASAP!

Thanks,

-- 
[]'s
Hugo
www.devin.com.br



From insyte at gmail.com  Thu Oct 20 03:36:43 2011
From: insyte at gmail.com (Ben Beuchler)
Date: Wed, 19 Oct 2011 22:36:43 -0500
Subject: Frequent varnish crashes
Message-ID: <CAC29JhVEuusrhk4zRLvbRGc9C=XMG5GYMyN+sWo5T_xn9RJ9pA@mail.gmail.com>

I'm running varnish 3.0.1 on Ubuntu 10.04, 64 bit.

Varnish had been running quite reliably when we only had one of our
sites passing through it.  Tonight I added more, about doubling the
traffic to about 40 - 50 requests / second.  Now the backend is
crashing every 1 - 4 minutes, reporting "Assert error in
vfp_testgzip_end(), cache_gzip.c line 656: Condition((vg) != NULL) not
true."  Searching for that error lead me to this bug:
https://www.varnish-cache.org/trac/ticket/1014

I tried applying the patch linked to as the fix, but that resulted in
a varnishd that would crash nearly every second.

Any suggestions?

-Ben


Output of panic.show:
http://paste.ubuntu.com/713808/

Output from "varnishstat -1":
http://paste.ubuntu.com/713831/

Main section of my VCL:
http://paste.ubuntu.com/713825/

Backend include:
http://paste.ubuntu.com/713826/

And the purge_acl include is a simple acl with only two IPs.



From mbrouwer at songteksten.nl  Thu Oct 20 21:39:31 2011
From: mbrouwer at songteksten.nl (Maikel Brouwer - Songteksten.nl)
Date: Thu, 20 Oct 2011 23:39:31 +0200
Subject: Multi country cache problem
In-Reply-To: <4E9F0B6B.6010008@hauke-lampe.de>
References: <1319038142.2178.10.camel@Maikel-Laptop-Werk>
	<4E9F0B6B.6010008@hauke-lampe.de>
Message-ID: <1319146771.2011.4.camel@Maikel-Laptop-Werk>

On Wed, 2011-10-19 at 19:39 +0200, Hauke Lampe wrote:
> On 19.10.2011 17:29, Maikel Brouwer - Songteksten.nl wrote:
> 
> > if ($country == 'be') {
> >   //display stuff
> > } else {
> >   //display other stuff)
> > }
> > 
> > The problem is that when someone from 'be' visits the page when it's not
> > yet cached the code in the 'be' block will be cached. When someone from
> > 'nl' visits the page it will see the 'be' block.
> 
> I'd probably resolve the GeoIP country in VCL and set an HTTP header:
> | set req.http.X-Country-Code = geoip.client_country_code();
> 
> The webserver then would have to add a "Vary: X-Country-Code" header to
> country-specific pages. E.g. in apache:
> | Header merge Vary X-Country-Code
> 
> varnish would cache one copy of each object for every country code, so
> you might want to reduce the additional header to "be-or-not-be".
> 
> A better option could be to use ESI and cache the HTML code for pages
> and ads separately. I haven't tested ESI with varnish yet, so I cannot
> give an example here.
> 
> 
> Hauke.

Thanks for the advice. 

But, when I'm using the geoip plugin from varnish it gives me the
following error:

(https://www.varnish-cache.org/trac/wiki/GeoipUsingInlineC)

Starting HTTP accelerator: varnishd failed!
Message from C-compiler:
./vcl.giASc5vH.c: In function ?VGC_function_vcl_hash?:
./vcl.giASc5vH.c:569: warning: implicit declaration of function
?VRT_l_req_hash?
Message from dlopen:
Compiled VCL program failed to load:
  ./vcl.giASc5vH.so: undefined symbol: VRT_l_req_hash
Running dlopen failed, exit 1

VCL compilation failed

Maybe it is not working anymore in v3? I'm using the example VCL that
was provided and installed Maxmind GeoIP C API without problems.

Thanks,

Maikel




From l at lrowe.co.uk  Thu Oct 20 22:06:46 2011
From: l at lrowe.co.uk (Laurence Rowe)
Date: Thu, 20 Oct 2011 23:06:46 +0100
Subject: Multi country cache problem
In-Reply-To: <1319146771.2011.4.camel@Maikel-Laptop-Werk>
References: <1319038142.2178.10.camel@Maikel-Laptop-Werk>
	<4E9F0B6B.6010008@hauke-lampe.de>
	<1319146771.2011.4.camel@Maikel-Laptop-Werk>
Message-ID: <CAOycyLSG46GZbC-zCqHwR8dRyb+rKU6auzCfUWBmUO5iS4jN+Q@mail.gmail.com>

On 20 October 2011 22:39, Maikel Brouwer - Songteksten.nl
<mbrouwer at songteksten.nl> wrote:
> Thanks for the advice.
>
> But, when I'm using the geoip plugin from varnish it gives me the
> following error:
>
> (https://www.varnish-cache.org/trac/wiki/GeoipUsingInlineC)
>
> Starting HTTP accelerator: varnishd failed!
> Message from C-compiler:
> ./vcl.giASc5vH.c: In function ?VGC_function_vcl_hash?:
> ./vcl.giASc5vH.c:569: warning: implicit declaration of function
> ?VRT_l_req_hash?
> Message from dlopen:
> Compiled VCL program failed to load:
> ?./vcl.giASc5vH.so: undefined symbol: VRT_l_req_hash
> Running dlopen failed, exit 1
>
> VCL compilation failed
>
> Maybe it is not working anymore in v3? I'm using the example VCL that
> was provided and installed Maxmind GeoIP C API without problems.

Maybe try: https://github.com/leed25d/geoip-vmod

Laurence



From sahmed1020 at gmail.com  Fri Oct 21 14:08:43 2011
From: sahmed1020 at gmail.com (S Ahmed)
Date: Fri, 21 Oct 2011 10:08:43 -0400
Subject: mark html with a flag to say it is served via varnish?
Message-ID: <CAG2rwuP5Vs_JOx=DubGHqk1r4oHrShL7mieR2Fpz0kAwJXzR7g@mail.gmail.com>

WHen a page gets cached in varnish, is it possible to have some sort of html
flag so by looking at the html you can see when it was cached in varnish?

e.g.


<!-- varnish cached at 20111021 5:44pm -->

I'm using ruby on rails.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111021/031a5625/attachment.html>

From shib4u at gmail.com  Fri Oct 21 14:21:39 2011
From: shib4u at gmail.com (Shibashish)
Date: Fri, 21 Oct 2011 19:51:39 +0530
Subject: mark html with a flag to say it is served via varnish?
In-Reply-To: <CAG2rwuP5Vs_JOx=DubGHqk1r4oHrShL7mieR2Fpz0kAwJXzR7g@mail.gmail.com>
References: <CAG2rwuP5Vs_JOx=DubGHqk1r4oHrShL7mieR2Fpz0kAwJXzR7g@mail.gmail.com>
Message-ID: <CAOV_DHj0TRua3XZo4t2oLLX3qJ5ZnzVnXLnTSP=Uxgqw0A=0sQ@mail.gmail.com>

On Fri, Oct 21, 2011 at 7:38 PM, S Ahmed <sahmed1020 at gmail.com> wrote:

> WHen a page gets cached in varnish, is it possible to have some sort of
> html flag so by looking at the html you can see when it was cached in
> varnish?
>
> e.g.
>
>
> <!-- varnish cached at 20111021 5:44pm -->
>
> I'm using ruby on rails.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>


sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
}

Check the header of the elements (html or jpg or gif, etc) on your website,
u will see something as...

X-Cache HIT
or
X-Cache MISS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111021/74c6d915/attachment.html>

From audun at ytterdal.net  Fri Oct 21 16:17:29 2011
From: audun at ytterdal.net (Audun Ytterdal)
Date: Fri, 21 Oct 2011 18:17:29 +0200
Subject: varnishlog and logging errors
In-Reply-To: <CA+KACLnrei0dTUXPrJLPWVx0-YebZcyxT+SFWwfOTz9U1sZSGA@mail.gmail.com>
References: <CAKBjT_64JXeh9uxVQ7VdwWcfZE70XLH607_gBXFU8Lo1YXpqBw@mail.gmail.com>
	<CA+KACLkwGf39gkn8j1Wmr=aX61XkctkP6YD28UbYCCi8z++OnQ@mail.gmail.com>
	<4E9F0DB3.40801@hauke-lampe.de>
	<CA+KACLnrei0dTUXPrJLPWVx0-YebZcyxT+SFWwfOTz9U1sZSGA@mail.gmail.com>
Message-ID: <CAKBjT_5rmeoct=aw96xcNVoUZXaraF6Xm+bRiXmRU-POrusA7Q@mail.gmail.com>

That helps a bit. But you don't get all the info from varnishlog.
Andnot _why_ it failed.
On Wed, Oct 19, 2011 at 7:59 PM, Hugo Cisneiros (Eitch)
<hugo.cisneiros at gmail.com> wrote:
> On Wed, Oct 19, 2011 at 3:49 PM, Hauke Lampe <lampe at hauke-lampe.de> wrote:
>> On 19.10.2011 19:25, Hugo Cisneiros (Eitch) wrote:
>>
>>> I use syslog to do it. When a request goes to vcl_error, I log it,
>>> including the status of the object for example. It's a bit more
>>> difficult to do (since it uses C variables, not VCL language ones),
>>> but works very fine.
>>
>> varnish 3.x implements a syslog function via the std vmod.
>
> That's very good to know! I will test it ASAP!
>
> Thanks,
>
> --
> []'s
> Hugo
> www.devin.com.br
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



-- 
Audun Ytterdal
http://audun.ytterdal.net



From jwellband at gmail.com  Sun Oct 23 17:40:57 2011
From: jwellband at gmail.com (Jason W.)
Date: Sun, 23 Oct 2011 13:40:57 -0400
Subject: Mass redirects/backend selection with Varnish?
Message-ID: <CAFTZbZUy4R9v4PDvYQE+oM+=P+gYhriBBbWdQOHcJ48-nvT+oQ@mail.gmail.com>

We're looking to move from squid as a reverse proxy to using varnish.
However, I'm not able to come up with a drop-in replacement for
backend selection and 301 redirects.

Currently, we have squid using squirm[1] as a redirector. For every
request coming to squid, a list of squirm patterns (regexes) is
consulted and a rewritten URL is constructed. This URL can be either a
301 redirect (URL prefaced with "301:") or a backend URL. If it's a
301, squid removes the 301: and serves up the redirect. If it's a
backend URL, squid rewrites the URL internally (the new URL is what
gets stored in the cache but the client never sees it) and fetches it
from the backend. We don't configure squid itself to use a single
origin. If the URL isn't matched by a squirm pattern, it's not
successfully served by squid.

Given a single squid instance (we have 7 currently), there can be
anywhere from 50 to 1000 patterns. They are stored in a
space-delimited text file with a regex that is matched on the URL and
a rewritten URL, some of which use backreferences from the matched
regex.

Obviously, we could do this with a giant list of if statements using
req.url and/or req.host. This doesn't strike me as ideal. Our thinking
is to abstract the selection of backend URL and/or whether to 301
redirect out of VCL. We've considered writing a custom VMOD to handle
this (either implement the squirm functionality or use squirm in the
same way that squid does), but I wanted to get the community's take
before we reinvent the wheel or do something crazy.

Is this something that is feasible with varnish or should I move this
functionality elsewhere in the stack? Any ideas are welcome.

Thanks much!

[1] http://squirm.foote.com.au/

-- 
HTH, YMMV, HANW :)

Jason

The path to enlightenment is /usr/bin/enlightenment.



From carrot at carrotis.com  Mon Oct 24 05:33:25 2011
From: carrot at carrotis.com (Calvin Park)
Date: Mon, 24 Oct 2011 14:33:25 +0900
Subject: response time in varnishncsa 3.0.1
Message-ID: <CAAOWZC+VXHsB-jKKB=OCK_=NsdDoUzeLTgZDWKV2zPfsEVFp+g@mail.gmail.com>

Hi Varnish Users

I want to check between users and varnishd using varnishncsa.
(response time each user)
But there is no option in Logformat.
How to do this ?
Thanks



From a.kabenin at ngenix.net  Mon Oct 24 08:00:21 2011
From: a.kabenin at ngenix.net (Alexander Kabenin)
Date: Mon, 24 Oct 2011 12:00:21 +0400
Subject: Hits + Misses <> client requests
Message-ID: <CA+XVOJi8Qt9nQCCubG_tdx6XycgejOwumg8gWH9bFp7e-sDqMg@mail.gmail.com>

Dear list,

I was trying to figure out a hit ratio of two varnish caches
configured as a chain:

Customer -> Varnish A -> Varnish B -> Backend.

My idea was to use Customer requests number from Varnish A and Cache
Misses of Varnish B.
(This scheme is simplified from my real installation, I use several
Varnish A's and Varnish B's).

I get figures from varnishstat. And I've discovered some weird numbers:

client_req             104764         0.04 Client requests received
cache_hit                7392         0.00 Cache hits
cache_hitpass               0         0.00 Cache hits for pass
cache_miss              13444         0.01 Cache misses

I was sure that the sum of cache_hit + cache_hitpass + cache_miss
should be equal to client_req. But my stats are far from that.
Furthermore, my client_req is not equal to number of records in the
NCSA log (client_req is always greater).

What am I missing?

Thanks in advance!

--
Regards,
Alexander Kabenin



From apj at mutt.dk  Mon Oct 24 09:04:41 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Mon, 24 Oct 2011 11:04:41 +0200
Subject: Hits + Misses <> client requests
In-Reply-To: <CA+XVOJi8Qt9nQCCubG_tdx6XycgejOwumg8gWH9bFp7e-sDqMg@mail.gmail.com>
References: <CA+XVOJi8Qt9nQCCubG_tdx6XycgejOwumg8gWH9bFp7e-sDqMg@mail.gmail.com>
Message-ID: <20111024090441.GN13866@nerd.dk>

On Mon, Oct 24, 2011 at 12:00:21PM +0400, Alexander Kabenin wrote:
> 
> I was sure that the sum of cache_hit + cache_hitpass + cache_miss
> should be equal to client_req. But my stats are far from that.
> Furthermore, my client_req is not equal to number of records in the
> NCSA log (client_req is always greater).

An unparseable request will give neither of the three, and in addition, it
won't be logged by the current varnishncsa.
I've recently added defaults to varnishncsa in trunk so it should log all
requests including those varnishd was not able to parse.

-- 
Andreas



From varnish at mm.quex.org  Mon Oct 24 14:05:43 2011
From: varnish at mm.quex.org (Michael Alger)
Date: Mon, 24 Oct 2011 22:05:43 +0800
Subject: Mass redirects/backend selection with Varnish?
In-Reply-To: <CAFTZbZUy4R9v4PDvYQE+oM+=P+gYhriBBbWdQOHcJ48-nvT+oQ@mail.gmail.com>
References: <CAFTZbZUy4R9v4PDvYQE+oM+=P+gYhriBBbWdQOHcJ48-nvT+oQ@mail.gmail.com>
Message-ID: <20111024140543.GA29473@grum.quex.org>

On Sun, Oct 23, 2011 at 01:40:57PM -0400, Jason W. wrote:
> We're looking to move from squid as a reverse proxy to using varnish.
> However, I'm not able to come up with a drop-in replacement for
> backend selection and 301 redirects.
> 
> Currently, we have squid using squirm[1] as a redirector. For every
> request coming to squid, a list of squirm patterns (regexes) is
> consulted and a rewritten URL is constructed. This URL can be either a
> 301 redirect (URL prefaced with "301:") or a backend URL. If it's a
> 301, squid removes the 301: and serves up the redirect. If it's a
> backend URL, squid rewrites the URL internally (the new URL is what
> gets stored in the cache but the client never sees it) and fetches it
> from the backend. We don't configure squid itself to use a single
> origin. If the URL isn't matched by a squirm pattern, it's not
> successfully served by squid.
> 
> Given a single squid instance (we have 7 currently), there can be
> anywhere from 50 to 1000 patterns. They are stored in a
> space-delimited text file with a regex that is matched on the URL and
> a rewritten URL, some of which use backreferences from the matched
> regex.
> 
> Obviously, we could do this with a giant list of if statements using
> req.url and/or req.host. This doesn't strike me as ideal.

I had the same thought when we migrated from squid + jesred to Varnish;
we had several thousand patterns across a few sites. I did simplify
things a little by implementing virtual-hosting type behaviour within
Varnish, so it only had to process redirects for the particular site the
request was actually for. If you can make a similar optimisation you
might find the amount of processing per request drops considerably.

While the if/elsif ladder looks a bit ugly and like a lot of work, it's
actually pretty much exactly what squirm is already doing. So I think
you'll find the performance to be about the same; possibly a bit faster
since if you implement it within VCL you won't have the overhead of
communicating over a pipe.

The only issue you'd have with doing it in Varnish is if your backend
hosts are pretty much arbitrary; Varnish needs each origin to be
explicitly defined. This requires a slight change to the logic, in that
you need to set req.backend appropriately, in addition to req.host
and/or req.url. But, it's not really complex.

> Our thinking is to abstract the selection of backend URL and/or
> whether to 301 redirect out of VCL. We've considered writing a custom
> VMOD to handle this (either implement the squirm functionality or use
> squirm in the same way that squid does), but I wanted to get the
> community's take before we reinvent the wheel or do something crazy.
> 
> Is this something that is feasible with varnish or should I move this
> functionality elsewhere in the stack? Any ideas are welcome.

I do think that abstracting it out of the VCL so you don't have to
actually manage the if/elsif ladder directly is probably a good idea.
It'd certainly be workable but you've probably got better things to do
with your time. I guess it depends how frequently you make changes or
additions to your redirections.

When I moved to Varnish, I took the opportunity to place all the
redirects and rewrites into our "DNS management system", which is just
an in-house hodge-podge of Python and perl. The redirects are specified
in a similar format to squirm/jesred, and I have a script that parses
them and spits out appropriate VCL. That file is then rsynced to each of
the proxies, and included from the appropriate site's configuration.

If you are happy editing it directly, then there's no real issue. One
nice thing about Varnish is you can tell it to load a new config while
it's running, and if it can't compile it, it'll just tell you to rack
off and keep running the existing one. So even if you break the config
the site keeps running without a hiccup. jesred liked to just stop doing
any redirects if I broke its pattern file, and the comment about "Dodo
mode" makes me think squirm may well do the same thing.

So in summary: try not to fret about the ugliness of a massive if/elsif
ladder. That's what squirm is doing, anyway. It might be a good time to
decide if directly editing the pattern file is how you want to be
managing all those redirects, and if that's what's really bugging you,
implement a better solution for that. The VCL itself isn't really a
problem, Varnish seems quite happy to load massive configurations.

I don't think I directly answered your question, so in case you didn't
infer an answer: I personally don't think you'd benefit from doing your
own custom squirm-like (or other) handler if performance is your concern.
I think you'd be better off doing a quick hackish mass-conversion of as
many patterns as you can and seeing how Varnish performs. My hunch is
that'll alleviate any concerns you've got.



From sam at musicmatter.co.uk  Mon Oct 24 14:55:39 2011
From: sam at musicmatter.co.uk (Sam Davey)
Date: Mon, 24 Oct 2011 15:55:39 +0100
Subject: Filling the varnish cache
Message-ID: <CANTi4ARTJj=87DT6+zerwd4M+KjowPq1UwmXB+ihHc1C=yaUow@mail.gmail.com>

Hi,

I am running varnish-2.1.5 which is performing great.  However I want
to ensure all visitors experience the speed benefits of the cache.

Currently the first visitor to a particular URL will experience a full
page load with all the apache, php, database overheads associated with
a none cached request.  And with the version of varnish I am running
the cache is always purges if the service is restarted.

I am currently using the scrapy web crawling framework which works
quite well bit it isn't a good fit for my production environment
(CentOS5 and python2.6 don't go easily) and I would prefer to build a
solution into my application so that purging and crawling can be
triggered intelligently.  The web application is php which I realise
is not the most efficient language to spider with, however I'm sure I
could l knock something up using the CURL library.

So my question is:

What tried and tested methods do other people use to fill their varnish cache?

Kind Regards,

Sam
Music Matter is a trading style of Matter Retail Ltd, Unit A116 Tustin Way, Longridge Road, Preston, PR2 5LX, United Kingdom.

Registered in England & Wales, Company number 06920634.

This email and any files transmitted with it are confidential. If you have received this email in error please notify the sender and then delete it immediately. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Matter Retail Ltd. The recipient should check this email and any attachments for the presence of viruses. Matter Retail Ltd. accepts no liability for any damage caused by any virus transmitted by this email.

Save resources: Think before you print.




From tobias.eichelbroenner at lamp-solutions.de  Mon Oct 24 15:13:10 2011
From: tobias.eichelbroenner at lamp-solutions.de (=?ISO-8859-1?Q?Tobias_Eichelbr=F6nner?=)
Date: Mon, 24 Oct 2011 17:13:10 +0200
Subject: Filling the varnish cache
In-Reply-To: <CANTi4ARTJj=87DT6+zerwd4M+KjowPq1UwmXB+ihHc1C=yaUow@mail.gmail.com>
References: <CANTi4ARTJj=87DT6+zerwd4M+KjowPq1UwmXB+ihHc1C=yaUow@mail.gmail.com>
Message-ID: <4EA58086.5000803@lamp-solutions.de>

Hallo
> What tried and tested methods do other people use to fill their varnish cache?
I use wget with -r -l4 to get a page hit on all my pages

Sincerely





From jwellband at gmail.com  Tue Oct 25 00:20:13 2011
From: jwellband at gmail.com (Jason W.)
Date: Mon, 24 Oct 2011 20:20:13 -0400
Subject: Mass redirects/backend selection with Varnish?
In-Reply-To: <20111024140543.GA29473@grum.quex.org>
References: <CAFTZbZUy4R9v4PDvYQE+oM+=P+gYhriBBbWdQOHcJ48-nvT+oQ@mail.gmail.com>
	<20111024140543.GA29473@grum.quex.org>
Message-ID: <CAFTZbZUkBR+GB6VzjjQ04gOibMgXn-pZVcZnuSxnkYED8wn3MQ@mail.gmail.com>

Michael (and list),

First, thanks much for the response. It's much appreciated.

> I had the same thought when we migrated from squid + jesred to Varnish;
> we had several thousand patterns across a few sites. I did simplify
> things a little by implementing virtual-hosting type behaviour within
> Varnish, so it only had to process redirects for the particular site the
> request was actually for. If you can make a similar optimisation you
> might find the amount of processing per request drops considerably.

Not sure I follow here - sounds like you mean putting different sites
on diffing IPs and/or different varnishes? We do that with squid to
match each app cluster but there is no squid-centric reason for this.
We've 13 major sites that we'd front with varnish initially, and if
that works well, then we'd use it for more, so possibly 100 disparate
sites - each with more than one backend.

> While the if/elsif ladder looks a bit ugly and like a lot of work, it's
> actually pretty much exactly what squirm is already doing. So I think
> you'll find the performance to be about the same; possibly a bit faster
> since if you implement it within VCL you won't have the overhead of
> communicating over a pipe.

Guess I wasn't clear :)  It's more of a configuration query since
we're used to defining behavior in one place (the patterns) versus two
(patterns and a preconfigured backend).

> The only issue you'd have with doing it in Varnish is if your backend
> hosts are pretty much arbitrary; Varnish needs each origin to be
> explicitly defined. This requires a slight change to the logic, in that
> you need to set req.backend appropriately, in addition to req.host
> and/or req.url. But, it's not really complex.

That was my initial concern. After reading your email, I realized I
could define all the backends we'd need with deterministic names and
possibly write some VCL to select the backend from the "rewritten"
hostname. I will have to play with this and see if it works as I think
it does.

Luckily, we don't rewrite the Host header in squid, so that's one less
thing varnish has to do :)

Some of our patterns don't specify a hostname component; e.g.
http://([^/]+)/images(/.*) ==> http://static.\1/images\2. I know about
Varnish's regex substitution, so I'm hopeful that I can do the
rewrites in VCL.

> I do think that abstracting it out of the VCL so you don't have to
> actually manage the if/elsif ladder directly is probably a good idea.
> It'd certainly be workable but you've probably got better things to do
> with your time. I guess it depends how frequently you make changes or
> additions to your redirections.

Changes are made by the dev team a few times a week and rolled out
weekly via cfengine grabbing the latest from their VCS, pushing the
patterns out to the caching boxes and poking squid/squirm.

> When I moved to Varnish, I took the opportunity to place all the
> redirects and rewrites into our "DNS management system", which is just
> an in-house hodge-podge of Python and perl. The redirects are specified
> in a similar format to squirm/jesred, and I have a script that parses
> them and spits out appropriate VCL. That file is then rsynced to each of
> the proxies, and included from the appropriate site's configuration.

Heh - I've used this idea elsewhere but never thought of writing
something to generate VCL. Thanks for the reminder ;)

> If you are happy editing it directly, then there's no real issue. One
> nice thing about Varnish is you can tell it to load a new config while
> it's running, and if it can't compile it, it'll just tell you to rack
> off and keep running the existing one. So even if you break the config
> the site keeps running without a hiccup. jesred liked to just stop doing
> any redirects if I broke its pattern file, and the comment about "Dodo
> mode" makes me think squirm may well do the same thing.

Heh - this happened once or twice, then the dev team wrote a test
harness that calls squirm with a bunch of URLs and ensures that the
expected rewrites are output. We may have to substitute with curls
against a non-prod varnish (assuming the config compiles).

> I don't think I directly answered your question, so in case you didn't
> infer an answer: I personally don't think you'd benefit from doing your
> own custom squirm-like (or other) handler if performance is your concern.
> I think you'd be better off doing a quick hackish mass-conversion of as
> many patterns as you can and seeing how Varnish performs. My hunch is
> that'll alleviate any concerns you've got.

Thanks much for the ideas and for telling me that someone else was
(ab)using squid redirecters ;)

-- 
HTH, YMMV, HANW :)

Jason

The path to enlightenment is /usr/bin/enlightenment.



From perbu at varnish-software.com  Tue Oct 25 07:29:18 2011
From: perbu at varnish-software.com (Per Buer)
Date: Tue, 25 Oct 2011 09:29:18 +0200
Subject: Filling the varnish cache
In-Reply-To: <CANTi4ARTJj=87DT6+zerwd4M+KjowPq1UwmXB+ihHc1C=yaUow@mail.gmail.com>
References: <CANTi4ARTJj=87DT6+zerwd4M+KjowPq1UwmXB+ihHc1C=yaUow@mail.gmail.com>
Message-ID: <CAOXZevA8KkLTAF9UXxxuVwBK-JcscgaFKON4PDJiKGazcST8gA@mail.gmail.com>

Hi,

On Mon, Oct 24, 2011 at 4:55 PM, Sam Davey <sam at musicmatter.co.uk> wrote:
>
> What tried and tested methods do other people use to fill their varnish cache?

It depends a bit on the usage pattern, but generally I can say that by
far the most efficient way to fill the cache is to have the users do
it. Since we do request coalescing the load on the backend servers
won't be that big while your cache is filling up. A lot of news and
portal sites achieve a high hitrate after only a couple of seconds.

So, unless your website has some very special properties, I would
recommend that you make your users "warm" the cache.

However, if you have a load balancer in front of Varnish and the load
balancer does health probes, you might consider making it poll the
front page. The front page is often heavy to generate and making sure
it is in cache before letting the hoards in might be  a good idea.

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
Varnish makes websites fly!
Whitepapers?| Video?| Twitter



From a.kabenin at ngenix.net  Wed Oct 26 15:14:13 2011
From: a.kabenin at ngenix.net (Alexander Kabenin)
Date: Wed, 26 Oct 2011 19:14:13 +0400
Subject: Hits + Misses <> client requests
In-Reply-To: <20111024090441.GN13866@nerd.dk>
References: <CA+XVOJi8Qt9nQCCubG_tdx6XycgejOwumg8gWH9bFp7e-sDqMg@mail.gmail.com>
	<20111024090441.GN13866@nerd.dk>
Message-ID: <CA+XVOJgDsK0Nd5ZkmWUdfsw07B0ogO9qY6MowPM0CBBC9wHwXw@mail.gmail.com>

Thank you for your reply Andreas.

But it is hardly the case, I cannot imagine that number of unparseable
request to occur.
And my logged requests number is far from su, of cache_hit +
cache_hitpass + cache_miss.

--
Alexander


On Mon, Oct 24, 2011 at 13:04, Andreas Plesner Jacobsen <apj at mutt.dk> wrote:
> On Mon, Oct 24, 2011 at 12:00:21PM +0400, Alexander Kabenin wrote:
>>
>> I was sure that the sum of cache_hit + cache_hitpass + cache_miss
>> should be equal to client_req. But my stats are far from that.
>> Furthermore, my client_req is not equal to number of records in the
>> NCSA log (client_req is always greater).
>
> An unparseable request will give neither of the three, and in addition, it
> won't be logged by the current varnishncsa.
> I've recently added defaults to varnishncsa in trunk so it should log all
> requests including those varnishd was not able to parse.
>
> --
> Andreas
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



From apj at mutt.dk  Wed Oct 26 15:33:33 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Wed, 26 Oct 2011 17:33:33 +0200
Subject: Hits + Misses <> client requests
In-Reply-To: <CA+XVOJgDsK0Nd5ZkmWUdfsw07B0ogO9qY6MowPM0CBBC9wHwXw@mail.gmail.com>
References: <CA+XVOJi8Qt9nQCCubG_tdx6XycgejOwumg8gWH9bFp7e-sDqMg@mail.gmail.com>
	<20111024090441.GN13866@nerd.dk>
	<CA+XVOJgDsK0Nd5ZkmWUdfsw07B0ogO9qY6MowPM0CBBC9wHwXw@mail.gmail.com>
Message-ID: <20111026153333.GP13866@nerd.dk>

On Wed, Oct 26, 2011 at 07:14:13PM +0400, Alexander Kabenin wrote:
> Thank you for your reply Andreas.
> 
> But it is hardly the case, I cannot imagine that number of unparseable
> request to occur.

I'm not saying it's the only case, I was just presenting one of the reasons
that the numbers do not necessarily add up.

return(restart) is another that I can think of on the top of my head.

But the main point still stands: You're trying to correlate numbers that cannot
necessarily be correlated.

-- 
Andreas



From bstillwell at photobucket.com  Thu Oct 27 04:56:51 2011
From: bstillwell at photobucket.com (Bryan Stillwell)
Date: Wed, 26 Oct 2011 22:56:51 -0600
Subject: Sporadic 503 errors with 3.0.2
Message-ID: <CADAfL_iLh1Xy3wkxPY7EvzZYZEOxEQoTaefiQo3dgu+Nbz1+fw@mail.gmail.com>

I'm seeing sporadic 503 errors that I'm wondering if someone could
help me with.  If I read this right, there appears to be a problem
with the backend:

  151 VCL_call     c recv lookup
  151 VCL_call     c hash
  151 Hash         c /2le1ert_th.jpg
  151 Hash         c s7.tinypic.com
  151 VCL_return   c hash
  151 VCL_call     c miss fetch
  151 FetchError   c no backend connection
  151 VCL_call     c error deliver
  151 VCL_call     c deliver deliver
  151 TxProtocol   c HTTP/1.1
  151 TxStatus     c 503
  151 TxResponse   c Service Unavailable
  151 TxHeader     c Server: Varnish
  151 TxHeader     c Content-Type: text/html; charset=utf-8
  151 TxHeader     c Retry-After: 5
  151 TxHeader     c Content-Length: 419
  151 TxHeader     c Accept-Ranges: bytes
  151 TxHeader     c Date: Thu, 27 Oct 2011 04:17:57 GMT
  151 TxHeader     c X-Varnish: 1163340533
  151 TxHeader     c Age: 1
  151 TxHeader     c Via: 1.1 varnish
  151 TxHeader     c Connection: close
  151 TxHeader     c X-Cache: MISS

I currently have the backend configured to use a single IP, which
points to a pair of load balancers that distribute it to numerous
webservers.  It works great 99.99% of the time, but that other 0.01% I
get 503s.

Is there any way to get more details on what the failure with the
backend is?  Was there a failure connecting to it?  Bad response?
Timeout?

I almost think this is a varnish issue (seen on both 3.0.1 and 3.0.2),
because I'm seeing the problem on all my varnish boxes, in multiple
data centers, pointing to different backends.  The only other thing I
can thing of is there might be an issue with having the load balancers
in between varnish and the backends.  I plan on testing removing them
from the equation by just referring to all the webservers individually
soon, but I haven't gotten to that yet.

Any help would be appreciated.

Thanks,
Bryan



From apj at mutt.dk  Thu Oct 27 08:48:22 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Thu, 27 Oct 2011 10:48:22 +0200
Subject: Sporadic 503 errors with 3.0.2
In-Reply-To: <CADAfL_iLh1Xy3wkxPY7EvzZYZEOxEQoTaefiQo3dgu+Nbz1+fw@mail.gmail.com>
References: <CADAfL_iLh1Xy3wkxPY7EvzZYZEOxEQoTaefiQo3dgu+Nbz1+fw@mail.gmail.com>
Message-ID: <20111027084822.GQ13866@nerd.dk>

On Wed, Oct 26, 2011 at 10:56:51PM -0600, Bryan Stillwell wrote:

> I'm seeing sporadic 503 errors that I'm wondering if someone could
> help me with.  If I read this right, there appears to be a problem
> with the backend:

You're right:

>   151 FetchError   c no backend connection
> 
> Is there any way to get more details on what the failure with the
> backend is?  Was there a failure connecting to it?  Bad response?
> Timeout?

Check these counters:

backend_unhealthy, backend_busy, backend_fail

-- 
Andreas



From bstillwell at photobucket.com  Thu Oct 27 19:06:26 2011
From: bstillwell at photobucket.com (Bryan Stillwell)
Date: Thu, 27 Oct 2011 13:06:26 -0600
Subject: Sporadic 503 errors with 3.0.2
In-Reply-To: <20111027084822.GQ13866@nerd.dk>
References: <CADAfL_iLh1Xy3wkxPY7EvzZYZEOxEQoTaefiQo3dgu+Nbz1+fw@mail.gmail.com>
	<20111027084822.GQ13866@nerd.dk>
Message-ID: <CADAfL_hybyL1JgcE-8rKeC6sN9A=MvQ-FAOV0ccP2ub8smezSg@mail.gmail.com>

On Thu, Oct 27, 2011 at 2:48 AM, Andreas Plesner Jacobsen <apj at mutt.dk> wrote:
> On Wed, Oct 26, 2011 at 10:56:51PM -0600, Bryan Stillwell wrote:
>>
>> Is there any way to get more details on what the failure with the
>> backend is? ?Was there a failure connecting to it? ?Bad response?
>> Timeout?
>
> Check these counters:
>
> backend_unhealthy, backend_busy, backend_fail

backend_conn ? ? ? ? ?12413302 ? ? ? ?66.44 Backend conn. success
backend_unhealthy ? ? ? ? ? ?0 ? ? ? ? 0.00 Backend conn. not attempted
backend_busy ? ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 Backend conn. too many
backend_fail ? ? ? ? ? ? ?4917 ? ? ? ? 0.03 Backend conn. failures

It looks like backend_fail numbers line up with what I'm seeing.  Any
suggestions on figuring out what is failing?

Thanks,
Bryan



From tfheen at varnish-software.com  Fri Oct 28 06:41:13 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Fri, 28 Oct 2011 08:41:13 +0200
Subject: Sporadic 503 errors with 3.0.2
In-Reply-To: <CADAfL_hybyL1JgcE-8rKeC6sN9A=MvQ-FAOV0ccP2ub8smezSg@mail.gmail.com>
	(Bryan Stillwell's message of "Thu, 27 Oct 2011 13:06:26 -0600")
References: <CADAfL_iLh1Xy3wkxPY7EvzZYZEOxEQoTaefiQo3dgu+Nbz1+fw@mail.gmail.com>
	<20111027084822.GQ13866@nerd.dk>
	<CADAfL_hybyL1JgcE-8rKeC6sN9A=MvQ-FAOV0ccP2ub8smezSg@mail.gmail.com>
Message-ID: <87y5w54nzq.fsf@qurzaw.varnish-software.com>

]] Bryan Stillwell 

| It looks like backend_fail numbers line up with what I'm seeing.  Any
| suggestions on figuring out what is failing?

My first guess would be you hitting connect_timeout, try increasing that
and see if the rate changes?

-- 
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64



From mhettwer at team.mobile.de  Fri Oct 28 10:16:39 2011
From: mhettwer at team.mobile.de (Hettwer, Marian)
Date: Fri, 28 Oct 2011 11:16:39 +0100
Subject: Sporadic 503 errors with 3.0.2
In-Reply-To: <87y5w54nzq.fsf@qurzaw.varnish-software.com>
Message-ID: <CAD04D35.17A0A%mhettwer@team.mobile.de>

On 28.10.11 08:41, "Tollef Fog Heen" <tfheen at varnish-software.com> wrote:


>]] Bryan Stillwell
>
>| It looks like backend_fail numbers line up with what I'm seeing.  Any
>| suggestions on figuring out what is failing?
>
>My first guess would be you hitting connect_timeout, try increasing that
>and see if the rate changes?

Also check between_bytes_timeout. In 2.1.5 it defaults to 60 seconds. It
wasn't enough for some requests on my platforms.
One might think that 60 seconds in between bytes is bad enough for a web
application, right? ;)
No worries, those requests where made by machines, not humans...

Anyway, as Tollef also suggested, check your timeout settings on both ends
(varnish and backend).

Cheers,
Marian




From bstillwell at photobucket.com  Mon Oct 31 03:40:21 2011
From: bstillwell at photobucket.com (Bryan Stillwell)
Date: Sun, 30 Oct 2011 21:40:21 -0600
Subject: Sporadic 503 errors with 3.0.2
In-Reply-To: <87y5w54nzq.fsf@qurzaw.varnish-software.com>
References: <CADAfL_iLh1Xy3wkxPY7EvzZYZEOxEQoTaefiQo3dgu+Nbz1+fw@mail.gmail.com>
	<20111027084822.GQ13866@nerd.dk>
	<CADAfL_hybyL1JgcE-8rKeC6sN9A=MvQ-FAOV0ccP2ub8smezSg@mail.gmail.com>
	<87y5w54nzq.fsf@qurzaw.varnish-software.com>
Message-ID: <CADAfL_jaJBbaRMf_R_XuNHrzLEh_0dN_rXuZp3mAFm9hKO=oqQ@mail.gmail.com>

On Fri, Oct 28, 2011 at 12:41 AM, Tollef Fog Heen
<tfheen at varnish-software.com> wrote:
> ]] Bryan Stillwell
>
> | It looks like backend_fail numbers line up with what I'm seeing. ?Any
> | suggestions on figuring out what is failing?
>
> My first guess would be you hitting connect_timeout, try increasing that
> and see if the rate changes?

Thanks for the response, but unfortunately that setting didn't have
any noticeable effect on the number of backend failures.

However, it did get me re-evaluating whether or not the problem
actually existed between varnish and the web servers.  Out of the
switches, networking cables, and load balancers, I decided to try
removing the load balancers today (since I was planning on doing that
anyways).  Once I did, the number of backend failures and 503
responses dropped to zero!  :-)

Bryan



From Harri.Paivaniemi at tieto.com  Mon Oct 31 04:37:38 2011
From: Harri.Paivaniemi at tieto.com (Harri.Paivaniemi at tieto.com)
Date: Mon, 31 Oct 2011 06:37:38 +0200
Subject: Sporadic 503 errors with 3.0.2
In-Reply-To: <CADAfL_jaJBbaRMf_R_XuNHrzLEh_0dN_rXuZp3mAFm9hKO=oqQ@mail.gmail.com>
References: <CADAfL_iLh1Xy3wkxPY7EvzZYZEOxEQoTaefiQo3dgu+Nbz1+fw@mail.gmail.com>
	<20111027084822.GQ13866@nerd.dk>
	<CADAfL_hybyL1JgcE-8rKeC6sN9A=MvQ-FAOV0ccP2ub8smezSg@mail.gmail.com>
	<87y5w54nzq.fsf@qurzaw.varnish-software.com>,
	<CADAfL_jaJBbaRMf_R_XuNHrzLEh_0dN_rXuZp3mAFm9hKO=oqQ@mail.gmail.com>
Message-ID: <F5A4C3A7AC59A74FB0A0D35F1E8BDB462C93F9CA44@EXMB03.eu.tieto.com>

Hi,

Stupid guestion: Are you sure that the backed is healthy from the LB's point of view when this 503 appears? I mean if the LB is struggling with the probe sometimes...

You could also try to change to lb-director in loadbalancer to something else (I guess you are using RR) and ensure that you have no stickyness in this backend farm configured.

-hjp


________________________________________
From: varnish-misc-bounces at varnish-cache.org [varnish-misc-bounces at varnish-cache.org] On Behalf Of Bryan Stillwell [bstillwell at photobucket.com]
Sent: 31 October 2011 5:40
To: varnish-misc at varnish-cache.org
Subject: Re: Sporadic 503 errors with 3.0.2

On Fri, Oct 28, 2011 at 12:41 AM, Tollef Fog Heen
<tfheen at varnish-software.com> wrote:
> ]] Bryan Stillwell
>
> | It looks like backend_fail numbers line up with what I'm seeing.  Any
> | suggestions on figuring out what is failing?
>
> My first guess would be you hitting connect_timeout, try increasing that
> and see if the rate changes?

Thanks for the response, but unfortunately that setting didn't have
any noticeable effect on the number of backend failures.

However, it did get me re-evaluating whether or not the problem
actually existed between varnish and the web servers.  Out of the
switches, networking cables, and load balancers, I decided to try
removing the load balancers today (since I was planning on doing that
anyways).  Once I did, the number of backend failures and 503
responses dropped to zero!  :-)

Bryan

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From bstillwell at photobucket.com  Mon Oct 31 04:54:51 2011
From: bstillwell at photobucket.com (Bryan Stillwell)
Date: Sun, 30 Oct 2011 22:54:51 -0600
Subject: Sporadic 503 errors with 3.0.2
In-Reply-To: <F5A4C3A7AC59A74FB0A0D35F1E8BDB462C93F9CA44@EXMB03.eu.tieto.com>
References: <CADAfL_iLh1Xy3wkxPY7EvzZYZEOxEQoTaefiQo3dgu+Nbz1+fw@mail.gmail.com>
	<20111027084822.GQ13866@nerd.dk>
	<CADAfL_hybyL1JgcE-8rKeC6sN9A=MvQ-FAOV0ccP2ub8smezSg@mail.gmail.com>
	<87y5w54nzq.fsf@qurzaw.varnish-software.com>
	<CADAfL_jaJBbaRMf_R_XuNHrzLEh_0dN_rXuZp3mAFm9hKO=oqQ@mail.gmail.com>
	<F5A4C3A7AC59A74FB0A0D35F1E8BDB462C93F9CA44@EXMB03.eu.tieto.com>
Message-ID: <CADAfL_j-QmzHO6rQo7RESvZRLL5ZXOE5p8Bcrs5nT4T0PH+20Q@mail.gmail.com>

On Sun, Oct 30, 2011 at 10:37 PM,  <Harri.Paivaniemi at tieto.com> wrote:
> Stupid guestion: Are you sure that the backed is healthy from the LB's point of view when this 503 appears? I mean if the LB is struggling with the probe sometimes...
>
> You could also try to change to lb-director in loadbalancer to something else (I guess you are using RR) and ensure that you have no stickyness in this backend farm configured.

Once I got rid of the hardware load balancers between varnish and the
web servers the problem went away.  We had originally designed it that
way so that we could disable varnish (squid previously) and the
webservers the same way.  The load balancers were really old hardware,
so it's not much of a surprise that they were to blame.

Bryan