From thiago at cmoraes.com  Wed Nov  2 00:15:11 2011
From: thiago at cmoraes.com (Thiago Moraes)
Date: Tue, 1 Nov 2011 22:15:11 -0200
Subject: Slicing cached data in a Varnish server
Message-ID: <CAOPp8Z940EkO80BM58k9MYO6b-AbBUZ30XBG41ivJKt4SFqD-A@mail.gmail.com>

I'm currently studying how to deploy varnish over a backend which responds
to REST queries such as:

http://myserver.com/myFile?range[0:50]

Suppose that the maximum range from myFile is [0:100]. In case I cache the
entire file, is there any way I can slice my data in my Varnish server so
that I can handle requests to subsets of myFile without getting cache
misses? Unfortunately every file I have is HUGE (I emailed this same
mailing list sometime ago), above a GB, so it's no possible to cache
multiple copies for each request.

Is it even possible to call some external software from varnish to slice
the cached data before delivering it? I have a program capable of doing
this slice, but I don't know how to call it in the context of a request
inside varnish. I'm almost sure that I'll have to use inline C, but are
there any documentation on this theme?


thanks,
Thiago Moraes - EnC 07 - UFSCar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111101/071bdc61/attachment.html>

From perbu at varnish-software.com  Wed Nov  2 08:44:50 2011
From: perbu at varnish-software.com (Per Buer)
Date: Wed, 2 Nov 2011 09:44:50 +0100
Subject: Slicing cached data in a Varnish server
In-Reply-To: <CAOPp8Z940EkO80BM58k9MYO6b-AbBUZ30XBG41ivJKt4SFqD-A@mail.gmail.com>
References: <CAOPp8Z940EkO80BM58k9MYO6b-AbBUZ30XBG41ivJKt4SFqD-A@mail.gmail.com>
Message-ID: <CAOXZevDYkpZUMOwu=t9jEWnMU80koUY0d6KRN8RECzXn1RFJEw@mail.gmail.com>

On Wed, Nov 2, 2011 at 1:15 AM, Thiago Moraes <thiago at cmoraes.com> wrote:

> I'm currently studying how to deploy varnish over a backend which responds
> to REST queries such as:
>
> http://myserver.com/myFile?range[0:50]
>
> Suppose that the maximum range from myFile is [0:100]. In case I cache the
> entire file, is there any way I can slice my data in my Varnish server so
> that I can handle requests to subsets of myFile without getting cache
> misses?


Can't you rewrite the request so it'll use the Range: header? That way
you'll only keep one copy. You'll have some latency on the first request as
Varnish will fetch the whole thing.

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/80b4b596/attachment.html>

From connor.walls at skillpages.com  Wed Nov  2 09:53:48 2011
From: connor.walls at skillpages.com (Connor Walls)
Date: Wed, 2 Nov 2011 09:53:48 +0000
Subject: Question about varnishstat and hitrate
Message-ID: <B9FE204AB6292842A9B015B87DD4CFA727969FEC0B@SBS-SERVER.weedle.local>

Hi,

I had a question about the numbers generated in varnishstat. Looking at the current hitrate average for one of our boxes is giving an average of 0.4833 for the last 1000 requests, and the hitrate is in and around this number on all of our varnish boxes. We are also recording the output of varnishlog to a file, including a "X-Cache" header, as per this article: https://www.varnish-cache.org/trac/wiki/VCLExampleHitMissHeader. Now, what I'm curious about is whenever I look at the responses that have been logged, <5% of them are being served from the cache, which is at odds with the 48% that varnishstat is reporting. Is the hitrate average only taking into account pages that are passed, and not pages that are fetched? Why is there such a discrepancy between the two values?

Here's a sample output from varnishstat, if it helps explain things better.

4+18:04:37                                           domU-12-31-39-10-74-A8
Hitrate ratio:       10      100     1000
Hitrate avg:     0.3646   0.4249   0.4833

     1528600         3.96         3.72 Client connections accepted
    10346326        23.77        25.19 Client requests received
     5244804         0.99        12.77 Cache hits
        9509         0.00         0.02 Cache hits for pass
      147772         0.00         0.36 Cache misses
       22490         0.00         0.05 Backend conn. success
     4731723        21.79        11.52 Backend conn. reuses
       12459         0.00         0.03 Backend conn. was closed
     4744236        19.81        11.55 Backend conn. recycles
        9505         0.00         0.02 Fetch head
     4727908        20.80        11.51 Fetch with Length
        1751         0.00         0.00 Fetch chunked
         422          .            .   N struct sess_mem
          96          .            .   N struct sess
        4104          .            .   N struct object
        4148          .            .   N struct objectcore
        4115          .            .   N struct objecthead
          48          .            .   N struct vbe_conn
          53          .            .   N worker threads
        7788         0.00         0.02 N worker threads created
           0         0.00         0.00 N queued work requests
       32999         0.00         0.08 N overflowed work requests
           1          .            .   N backends
      136098          .            .   N expired objects
      397731          .            .   N LRU moved objects
     8861590        17.82        21.58 Objects sent with write
     1528600         3.96         3.72 Total Sessions
    10346326        23.77        25.19 Total Requests
     4600511        20.80        11.20 Total pass
     4744331        20.80        11.55 Total fetch
  5613543692     14549.86     13669.00 Total header bytes
50230527274     64650.96    122311.52 Total body bytes
      398864         1.98         0.97 Session Closed
           1         0.00         0.00 Session Pipeline
     9989106        21.79        24.32 Session Linger
     4717398        17.82        11.49 Session herd
   731094771      2065.68      1780.22 SHM records
    29558086        95.06        71.97 SHM writes
       14131         0.00         0.03 SHM MTX contention
         353         0.00         0.00 SHM cycles through buffer
     3518804        16.83         8.57 SMA allocator requests
        8184          .            .   SMA outstanding allocations
    46354320          .            .   SMA outstanding bytes
12051521527          .            .   SMA bytes allocated
12005167207          .            .   SMA bytes free
      357190         1.98         0.87 SMS allocator requests
           0          .            .   SMS outstanding allocations
           0          .            .   SMS outstanding bytes
   135529138          .            .   SMS bytes allocated
   135529138          .            .   SMS bytes freed
     4744578        20.80        11.55 Backend requests made
           1         0.00         0.00 N vcl total
           1         0.00         0.00 N vcl available
      240460          .            .   N total active purges
      353239         1.98         0.86 N new purges added
      112779         0.99         0.27 N old purges deleted
      510753         0.99         1.24 N objects tested
    32284561       412.94        78.61 N regexps tested against
      155686         0.99         0.38 N duplicate purges removed


Thanks for any help,
Connor Walls

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/c088b993/attachment.html>

From mehul at onlygizmos.com  Wed Nov  2 10:02:38 2011
From: mehul at onlygizmos.com (Mehul Ved)
Date: Wed, 2 Nov 2011 15:32:38 +0530
Subject: Varnish in front of apache with name and ip based virtualhosts
Message-ID: <CAJC7t5Ng4t0zziXmR4h+beSvoz3+Z4c8zvkfcqRSqu8_9FN7dA@mail.gmail.com>

Hi,
   I have been reading up on various sites, documents but am not able to
figure out how to set up varnish to work with my setup.
Here's how is my apache configured:
1. Apache runs on port 80 currently and listens on 3 IP addresses:
x.x.x.91, x.x.x.92 and x.x.x.93
2. Only one site on runs x.x.x.92 and x.x.x.93, each. But, multiple
websites are configured on x.x.x.91.

As of now, I am running varnish on a higher port with only default backend
configured in default.vcl

backend default {
  .host = "127.0.0.1";
  .port = "80";
}


-- 
With Regards
Mehul Ved
9757062714
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/cf20fb2a/attachment.html>

From stig at zedge.net  Wed Nov  2 10:14:31 2011
From: stig at zedge.net (Stig Bakken)
Date: Wed, 2 Nov 2011 11:14:31 +0100
Subject: Question about varnishstat and hitrate
In-Reply-To: <B9FE204AB6292842A9B015B87DD4CFA727969FEC0B@SBS-SERVER.weedle.local>
References: <B9FE204AB6292842A9B015B87DD4CFA727969FEC0B@SBS-SERVER.weedle.local>
Message-ID: <CAAQKwWi53D_Yrrn0uPfc3XMfJaNwA8+WObjVYr-_STk-LUjR7Q@mail.gmail.com>

Varnish's cache hit rate is a percentage of hits among requests that went
through vcl_lookup, not all requests.

 - Stig

On Wed, Nov 2, 2011 at 10:53 AM, Connor Walls
<connor.walls at skillpages.com>wrote:

> Hi,****
>
> ** **
>
> I had a question about the numbers generated in varnishstat. Looking at
> the current hitrate average for one of our boxes is giving an average of
> 0.4833 for the last 1000 requests, and the hitrate is in and around this
> number on all of our varnish boxes. We are also recording the output of
> varnishlog to a file, including a ?X-Cache? header, as per this article:
> https://www.varnish-cache.org/trac/wiki/VCLExampleHitMissHeader. Now,
> what I?m curious about is whenever I look at the responses that have been
> logged, <5% of them are being served from the cache, which is at odds with
> the 48% that varnishstat is reporting. Is the hitrate average only taking
> into account pages that are passed, and not pages that are fetched? Why is
> there such a discrepancy between the two values?****
>
> ** **
>
> Here?s a sample output from varnishstat, if it helps explain things better.
> ****
>
> ** **
>
> 4+18:04:37                                           domU-12-31-39-10-74-A8
> ****
>
> Hitrate ratio:       10      100     1000****
>
> Hitrate avg:     0.3646   0.4249   0.4833****
>
> ** **
>
>      1528600         3.96         3.72 Client connections accepted****
>
>     10346326        23.77        25.19 Client requests received****
>
>      5244804         0.99        12.77 Cache hits****
>
>         9509         0.00         0.02 Cache hits for pass****
>
>       147772         0.00         0.36 Cache misses****
>
>        22490         0.00         0.05 Backend conn. success****
>
>      4731723        21.79        11.52 Backend conn. reuses****
>
>        12459         0.00         0.03 Backend conn. was closed****
>
>      4744236        19.81        11.55 Backend conn. recycles****
>
>         9505         0.00         0.02 Fetch head****
>
>      4727908        20.80        11.51 Fetch with Length****
>
>         1751         0.00         0.00 Fetch chunked****
>
>          422          .            .   N struct sess_mem****
>
>           96          .            .   N struct sess****
>
>         4104          .            .   N struct object****
>
>         4148          .            .   N struct objectcore****
>
>         4115          .            .   N struct objecthead****
>
>           48          .            .   N struct vbe_conn****
>
>           53          .            .   N worker threads****
>
>         7788         0.00         0.02 N worker threads created****
>
>            0         0.00         0.00 N queued work requests****
>
>        32999         0.00         0.08 N overflowed work requests****
>
>            1          .            .   N backends****
>
>       136098          .            .   N expired objects****
>
>       397731          .            .   N LRU moved objects****
>
>      8861590        17.82        21.58 Objects sent with write****
>
>      1528600         3.96         3.72 Total Sessions****
>
>     10346326        23.77        25.19 Total Requests****
>
>      4600511        20.80        11.20 Total pass****
>
>      4744331        20.80        11.55 Total fetch****
>
>   5613543692     14549.86     13669.00 Total header bytes****
>
> 50230527274     64650.96    122311.52 Total body bytes****
>
>       398864         1.98         0.97 Session Closed****
>
>            1         0.00         0.00 Session Pipeline****
>
>      9989106        21.79        24.32 Session Linger****
>
>      4717398        17.82        11.49 Session herd****
>
>    731094771      2065.68      1780.22 SHM records****
>
>     29558086        95.06        71.97 SHM writes****
>
>        14131         0.00         0.03 SHM MTX contention****
>
>          353         0.00         0.00 SHM cycles through buffer****
>
>      3518804        16.83         8.57 SMA allocator requests****
>
>         8184          .            .   SMA outstanding allocations****
>
>     46354320          .            .   SMA outstanding bytes****
>
> 12051521527          .            .   SMA bytes allocated****
>
> 12005167207          .            .   SMA bytes free****
>
>       357190         1.98         0.87 SMS allocator requests****
>
>            0          .            .   SMS outstanding allocations****
>
>            0          .            .   SMS outstanding bytes****
>
>    135529138          .            .   SMS bytes allocated****
>
>    135529138          .            .   SMS bytes freed****
>
>      4744578        20.80        11.55 Backend requests made****
>
>            1         0.00         0.00 N vcl total****
>
>            1         0.00         0.00 N vcl available****
>
>       240460          .            .   N total active purges****
>
>       353239         1.98         0.86 N new purges added****
>
>       112779         0.99         0.27 N old purges deleted****
>
>       510753         0.99         1.24 N objects tested****
>
>     32284561       412.94        78.61 N regexps tested against****
>
>       155686         0.99         0.38 N duplicate purges removed****
>
> ** **
>
> ** **
>
> Thanks for any help,****
>
> Connor Walls ****
>
> ** **
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



-- 
Stig Bakken
CTO, Zedge.net - free your phone!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/e18299eb/attachment.html>

From connor.walls at skillpages.com  Wed Nov  2 10:15:21 2011
From: connor.walls at skillpages.com (Connor Walls)
Date: Wed, 2 Nov 2011 10:15:21 +0000
Subject: Question about varnishstat and hitrate
In-Reply-To: <CAAQKwWi53D_Yrrn0uPfc3XMfJaNwA8+WObjVYr-_STk-LUjR7Q@mail.gmail.com>
References: <B9FE204AB6292842A9B015B87DD4CFA727969FEC0B@SBS-SERVER.weedle.local>
	<CAAQKwWi53D_Yrrn0uPfc3XMfJaNwA8+WObjVYr-_STk-LUjR7Q@mail.gmail.com>
Message-ID: <B9FE204AB6292842A9B015B87DD4CFA727969FEC1D@SBS-SERVER.weedle.local>

Great, thanks very much!

-Connor

From: Stig Bakken [mailto:stig at zedge.net]
Sent: 02 November 2011 10:15
To: Connor Walls
Cc: varnish-misc at varnish-cache.org
Subject: Re: Question about varnishstat and hitrate

Varnish's cache hit rate is a percentage of hits among requests that went through vcl_lookup, not all requests.

 - Stig

On Wed, Nov 2, 2011 at 10:53 AM, Connor Walls <connor.walls at skillpages.com<mailto:connor.walls at skillpages.com>> wrote:
Hi,

I had a question about the numbers generated in varnishstat. Looking at the current hitrate average for one of our boxes is giving an average of 0.4833 for the last 1000 requests, and the hitrate is in and around this number on all of our varnish boxes. We are also recording the output of varnishlog to a file, including a ?X-Cache? header, as per this article: https://www.varnish-cache.org/trac/wiki/VCLExampleHitMissHeader. Now, what I?m curious about is whenever I look at the responses that have been logged, <5% of them are being served from the cache, which is at odds with the 48% that varnishstat is reporting. Is the hitrate average only taking into account pages that are passed, and not pages that are fetched? Why is there such a discrepancy between the two values?

Here?s a sample output from varnishstat, if it helps explain things better.

4+18:04:37                                           domU-12-31-39-10-74-A8
Hitrate ratio:       10      100     1000
Hitrate avg:     0.3646   0.4249   0.4833

     1528600         3.96         3.72 Client connections accepted
    10346326        23.77        25.19 Client requests received
     5244804         0.99        12.77 Cache hits
        9509         0.00         0.02 Cache hits for pass
      147772         0.00         0.36 Cache misses
       22490         0.00         0.05 Backend conn. success
     4731723        21.79        11.52 Backend conn. reuses
       12459         0.00         0.03 Backend conn. was closed
     4744236        19.81        11.55 Backend conn. recycles
        9505         0.00         0.02 Fetch head
     4727908        20.80        11.51 Fetch with Length
        1751         0.00         0.00 Fetch chunked
         422          .            .   N struct sess_mem
          96          .            .   N struct sess
        4104          .            .   N struct object
        4148          .            .   N struct objectcore
        4115          .            .   N struct objecthead
          48          .            .   N struct vbe_conn
          53          .            .   N worker threads
        7788         0.00         0.02 N worker threads created
           0         0.00         0.00 N queued work requests
       32999         0.00         0.08 N overflowed work requests
           1          .            .   N backends
      136098          .            .   N expired objects
      397731          .            .   N LRU moved objects
     8861590        17.82        21.58 Objects sent with write
     1528600         3.96         3.72 Total Sessions
    10346326        23.77        25.19 Total Requests
     4600511        20.80        11.20 Total pass
     4744331        20.80        11.55 Total fetch
  5613543692<tel:5613543692>     14549.86     13669.00 Total header bytes
50230527274     64650.96    122311.52 Total body bytes
      398864         1.98         0.97 Session Closed
           1         0.00         0.00 Session Pipeline
     9989106        21.79        24.32 Session Linger
     4717398        17.82        11.49 Session herd
   731094771      2065.68      1780.22 SHM records
    29558086        95.06        71.97 SHM writes
       14131         0.00         0.03 SHM MTX contention
         353         0.00         0.00 SHM cycles through buffer
     3518804        16.83         8.57 SMA allocator requests
        8184          .            .   SMA outstanding allocations
    46354320          .            .   SMA outstanding bytes
12051521527          .            .   SMA bytes allocated
12005167207          .            .   SMA bytes free
      357190         1.98         0.87 SMS allocator requests
           0          .            .   SMS outstanding allocations
           0          .            .   SMS outstanding bytes
   135529138          .            .   SMS bytes allocated
   135529138          .            .   SMS bytes freed
     4744578        20.80        11.55 Backend requests made
           1         0.00         0.00 N vcl total
           1         0.00         0.00 N vcl available
      240460          .            .   N total active purges
      353239         1.98         0.86 N new purges added
      112779         0.99         0.27 N old purges deleted
      510753         0.99         1.24 N objects tested
    32284561       412.94        78.61 N regexps tested against
      155686         0.99         0.38 N duplicate purges removed


Thanks for any help,
Connor Walls


_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



--
Stig Bakken
CTO, Zedge.net - free your phone!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/24ec1fac/attachment.html>

From stewsnooze at gmail.com  Wed Nov  2 10:19:25 2011
From: stewsnooze at gmail.com (Stewart Robinson)
Date: Wed, 2 Nov 2011 10:19:25 +0000
Subject: Varnish in front of apache with name and ip based virtualhosts
In-Reply-To: <CAJC7t5Ng4t0zziXmR4h+beSvoz3+Z4c8zvkfcqRSqu8_9FN7dA@mail.gmail.com>
References: <CAJC7t5Ng4t0zziXmR4h+beSvoz3+Z4c8zvkfcqRSqu8_9FN7dA@mail.gmail.com>
Message-ID: <-1045190813558317428@unknownmsgid>

Make your backend definition point at the 3 ip addresses you are using
or additionally make apache listen on 127...

Stewart Robinson
@stewsnooze


On 2 Nov 2011, at 10:03, Mehul Ved <mehul at onlygizmos.com> wrote:

> Hi,
>    I have been reading up on various sites, documents but am not able to figure out how to set up varnish to work with my setup.
> Here's how is my apache configured:
> 1. Apache runs on port 80 currently and listens on 3 IP addresses: x.x.x.91, x.x.x.92 and x.x.x.93
> 2. Only one site on runs x.x.x.92 and x.x.x.93, each. But, multiple websites are configured on x.x.x.91.
>
> As of now, I am running varnish on a higher port with only default backend configured in default.vcl
>
> backend default {
>   .host = "127.0.0.1";
>   .port = "80";
> }
>
>
> --
> With Regards
> Mehul Ved
> 9757062714
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From apj at mutt.dk  Wed Nov  2 10:59:28 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Wed, 2 Nov 2011 11:59:28 +0100
Subject: Varnish in front of apache with name and ip based virtualhosts
In-Reply-To: <-1045190813558317428@unknownmsgid>
References: <CAJC7t5Ng4t0zziXmR4h+beSvoz3+Z4c8zvkfcqRSqu8_9FN7dA@mail.gmail.com>
	<-1045190813558317428@unknownmsgid>
Message-ID: <20111102105928.GT13866@nerd.dk>

On Wed, Nov 02, 2011 at 10:19:25AM +0000, Stewart Robinson wrote:

> Make your backend definition point at the 3 ip addresses you are using
> or additionally make apache listen on 127...

And use server.ip to select the backend in varnish

-- 
Andreas



From phk at phk.freebsd.dk  Wed Nov  2 11:09:27 2011
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 02 Nov 2011 11:09:27 +0000
Subject: Soliciting input on the ban lurker
Message-ID: <97985.1320232167@critter.freebsd.dk>


I have been looking at ticket 1045 and the ban-lurker and I think
I need some thoughtfull input before I proceed.

The ban-lurker we have now, will try test the objects which sit
at the oldest bans and check them against all bans.

This can give three results:

  1) Som ban matches, object is killed.

  2) We cannot tell test certain bans, because they require req.*

  3) All bans tested, none match, object moved to top of ban-list.

Eventually, this may empty the last ban on the list so it can be
removed.

The advantage to this strategy is that the ban lurker works one
object through the ban-list at a time, and cleans up the ban-list
from the tail end, where presumbly, we find the objects which are
not going to "shake loose" due to traffic.

The disadvantage of this strategy is that we get stuck on bans
requiring req.*, those seem to be very common, and the ban-lurker
does surprisingly much duplicate work when that happens.


Another strategy could go like this:

Find the last ban on the list that does not require req.*, test
it against all objects further down the ban-list, then mark the
ban "GONE" because no object will need to be tested against it again.

The advantage of this strategy is that the "non-req" bans will
all be dealt with by the ban-lurker and that no duplicate work
is done at all:  All tests are productive.

The disadvantage is that it works the other way with respect to
memory pressure:  It takes one ban against many objects, objects
that we have not used for a long time and which therefore introduces
memory pressure.  Worst case is where you add a non-req ban, and
the lurker will test it against every single object in your cache.

It is not obvious to me that either of these strategies are optimal
and I am getting to the point where I wonder if maybe we need to
offer a choice of strategy ?

Some input and operational experience would be most welcome...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.



From mehul at onlygizmos.com  Wed Nov  2 11:40:17 2011
From: mehul at onlygizmos.com (Mehul Ved)
Date: Wed, 2 Nov 2011 17:10:17 +0530
Subject: Varnish in front of apache with name and ip based virtualhosts
In-Reply-To: <20111102105928.GT13866@nerd.dk>
References: <CAJC7t5Ng4t0zziXmR4h+beSvoz3+Z4c8zvkfcqRSqu8_9FN7dA@mail.gmail.com>
	<-1045190813558317428@unknownmsgid>
	<20111102105928.GT13866@nerd.dk>
Message-ID: <CAJC7t5MQYXB37X3cxDapZM5QfD631LGG=1-4ZCjgLeKdbc4PZw@mail.gmail.com>

Thank you Stewart and Andreas for your responses.

On Wed, Nov 2, 2011 at 4:29 PM, Andreas Plesner Jacobsen <apj at mutt.dk>wrote:

> On Wed, Nov 02, 2011 at 10:19:25AM +0000, Stewart Robinson wrote:
>
> > Make your backend definition point at the 3 ip addresses you are using
> > or additionally make apache listen on 127...
>
> And use server.ip to select the backend in varnish
>

I created 3 backends. Backend is selected on basis of server.ip as defined
in vcl_recv.
It seems to be working as of now. Will confirm again once I am go live with
varnish.


-- 
With Regards
Mehul Ved
9757062714
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/df4cddca/attachment.html>

From apj at mutt.dk  Wed Nov  2 12:00:23 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Wed, 2 Nov 2011 13:00:23 +0100
Subject: Soliciting input on the ban lurker
In-Reply-To: <97985.1320232167@critter.freebsd.dk>
References: <97985.1320232167@critter.freebsd.dk>
Message-ID: <20111102120022.GU13866@nerd.dk>

On Wed, Nov 02, 2011 at 11:09:27AM +0000, Poul-Henning Kamp wrote:
> 
> The disadvantage of this strategy is that we get stuck on bans
> requiring req.*, those seem to be very common, and the ban-lurker
> does surprisingly much duplicate work when that happens.

In my experience, users either

* know about the ban lurker (uncommon), and write 100% ban-lurker friendly bans
or
* don't know about the ban lurker, and write 100% req-based ban. (In part due
  to the presence of ban.url and ban_url)

Which is why I don't understand why the ban lurker was enabled by default,
since it takes extra effort to write the ban lurker friendly bans, and enabling
the ban lurker, when you start doing that.

This makes me think that optimizing for the case where lurker friendly and
unfriendly bans are mixed is not where the effort should be spent.

-- 
Andreas



From apj at mutt.dk  Wed Nov  2 12:06:10 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Wed, 2 Nov 2011 13:06:10 +0100
Subject: Soliciting input on the ban lurker
In-Reply-To: <20111102120022.GU13866@nerd.dk>
References: <97985.1320232167@critter.freebsd.dk>
	<20111102120022.GU13866@nerd.dk>
Message-ID: <20111102120610.GV13866@nerd.dk>

On Wed, Nov 02, 2011 at 01:00:23PM +0100, Andreas Plesner Jacobsen wrote:
> 
> Which is why I don't understand why the ban lurker was enabled by default,
> since it takes extra effort to write the ban lurker friendly bans, and enabling
> the ban lurker, when you start doing that.

Should have been "and enabling the ban lurker is easy, when you start doing that".

-- 
Andreas



From mit at stagename.com  Wed Nov  2 19:03:35 2011
From: mit at stagename.com (Mit Rowe)
Date: Wed, 2 Nov 2011 15:03:35 -0400
Subject: unique request ids
Message-ID: <CAFWPQt5JArJijQqxnHz5AKQtfm5o-zhyciCotPjmH3sbP3GVNw@mail.gmail.com>

Hello!

For many reasons (QA, performance analytics, etc) we log all of our proxy
traffic into a database.

We do this with varnish by collecting a text log via varnishncsa -F and a
CSV formatted string, and regularly run a batch process that inserts this
CSV data into a database.

I was wondering... does varnish internally generate a reliably random
unique value for each request (similar to mod_unique_id in apache or a
uuid/guid), and if so is there a way that i can write it out to my logs,
perhaps via an insertion into a request header and using the %{HEADER}i
macro in varnishncsa?

-Mit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/a810fc73/attachment.html>

From straightflush at gmail.com  Wed Nov  2 19:54:09 2011
From: straightflush at gmail.com (AD)
Date: Wed, 2 Nov 2011 15:54:09 -0400
Subject: unique request ids
In-Reply-To: <CAFWPQt5JArJijQqxnHz5AKQtfm5o-zhyciCotPjmH3sbP3GVNw@mail.gmail.com>
References: <CAFWPQt5JArJijQqxnHz5AKQtfm5o-zhyciCotPjmH3sbP3GVNw@mail.gmail.com>
Message-ID: <CANZWjw56Wx2ccyaW+DC0DPAYAzr8YPQ3Snsg9=zaLead=EFgVw@mail.gmail.com>

we do something similar.  My suggestion,

 md5(logentry + currentTimeInMilliseconds) as your key as you loop through
the logfile



On Wed, Nov 2, 2011 at 3:03 PM, Mit Rowe <mit at stagename.com> wrote:

> Hello!
>
> For many reasons (QA, performance analytics, etc) we log all of our proxy
> traffic into a database.
>
> We do this with varnish by collecting a text log via varnishncsa -F and a
> CSV formatted string, and regularly run a batch process that inserts this
> CSV data into a database.
>
> I was wondering... does varnish internally generate a reliably random
> unique value for each request (similar to mod_unique_id in apache or a
> uuid/guid), and if so is there a way that i can write it out to my logs,
> perhaps via an insertion into a request header and using the %{HEADER}i
> macro in varnishncsa?
>
> -Mit
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/e08bd138/attachment.html>

From mit at stagename.com  Wed Nov  2 20:25:04 2011
From: mit at stagename.com (Mit Rowe)
Date: Wed, 2 Nov 2011 16:25:04 -0400
Subject: unique request ids
In-Reply-To: <CANZWjw56Wx2ccyaW+DC0DPAYAzr8YPQ3Snsg9=zaLead=EFgVw@mail.gmail.com>
References: <CAFWPQt5JArJijQqxnHz5AKQtfm5o-zhyciCotPjmH3sbP3GVNw@mail.gmail.com>
	<CANZWjw56Wx2ccyaW+DC0DPAYAzr8YPQ3Snsg9=zaLead=EFgVw@mail.gmail.com>
Message-ID: <CAFWPQt5Ldtn47i8AzMzho+EaBSUK4J0tMsKkGQtLUkg-ewW8_w@mail.gmail.com>

for our purposes, it would have to be done at the point of log writing
(primarily, but not limited to, the need to ensure that if for whatever
reason we have to re-process a log file that we can quickly identify lines
already written to the db via a unique index).



On Wed, Nov 2, 2011 at 3:54 PM, AD <straightflush at gmail.com> wrote:

> we do something similar.  My suggestion,
>
>  md5(logentry + currentTimeInMilliseconds) as your key as you loop through
> the logfile
>
>
>
> On Wed, Nov 2, 2011 at 3:03 PM, Mit Rowe <mit at stagename.com> wrote:
>
>> Hello!
>>
>> For many reasons (QA, performance analytics, etc) we log all of our proxy
>> traffic into a database.
>>
>> We do this with varnish by collecting a text log via varnishncsa -F and a
>> CSV formatted string, and regularly run a batch process that inserts this
>> CSV data into a database.
>>
>> I was wondering... does varnish internally generate a reliably random
>> unique value for each request (similar to mod_unique_id in apache or a
>> uuid/guid), and if so is there a way that i can write it out to my logs,
>> perhaps via an insertion into a request header and using the %{HEADER}i
>> macro in varnishncsa?
>>
>> -Mit
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>


-- 
Will 'Mit' Rowe
Stagename*
*1-866-326-3098
mit at stagename.com <josh at stagename.com>
www.stagename.com
Twitter: @stagename

*The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of this
information by persons or entities other than the intended recipient is
prohibited. If you received this transmission in error, please contact the
sender and delete all material contained herein from your computer.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/7ea99a1e/attachment.html>

From perbu at varnish-software.com  Wed Nov  2 20:27:30 2011
From: perbu at varnish-software.com (Per Buer)
Date: Wed, 2 Nov 2011 21:27:30 +0100
Subject: unique request ids
In-Reply-To: <CAFWPQt5JArJijQqxnHz5AKQtfm5o-zhyciCotPjmH3sbP3GVNw@mail.gmail.com>
References: <CAFWPQt5JArJijQqxnHz5AKQtfm5o-zhyciCotPjmH3sbP3GVNw@mail.gmail.com>
Message-ID: <CAOXZevBCRj26=_24jaRiga3hEKjXFFXKnKGuRwvYXN2dF1-+ag@mail.gmail.com>

Hi Mit,

On Wed, Nov 2, 2011 at 8:03 PM, Mit Rowe <mit at stagename.com> wrote:

>
> I was wondering... does varnish internally generate a reliably random
> unique value for each request (similar to mod_unique_id in apache or a
> uuid/guid), and if so is there a way that i can write it out to my logs,
> perhaps via an insertion into a request header and using the %{HEADER}i
> macro in varnishncsa?
>

There is a request ID, in Varnishlog you'll see it as the last number in
the ReqStart lines ie.

    1932 ReqStart     c 127.0.0.1 54471 1236808814

You could probably extract it from the X-Varnish header, it will be the
first of one or two numbers in that header.

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/ba53371f/attachment.html>

From mit at stagename.com  Wed Nov  2 20:43:20 2011
From: mit at stagename.com (Mit Rowe)
Date: Wed, 2 Nov 2011 16:43:20 -0400
Subject: unique request ids
In-Reply-To: <CAOXZevBCRj26=_24jaRiga3hEKjXFFXKnKGuRwvYXN2dF1-+ag@mail.gmail.com>
References: <CAFWPQt5JArJijQqxnHz5AKQtfm5o-zhyciCotPjmH3sbP3GVNw@mail.gmail.com>
	<CAOXZevBCRj26=_24jaRiga3hEKjXFFXKnKGuRwvYXN2dF1-+ag@mail.gmail.com>
Message-ID: <CAFWPQt5J35oDhuh8tXXco0UjZ+vYGGMas-MJDL5uU0j1+ePEcg@mail.gmail.com>

Hi Per,

That's looks promising, i could save the X-Varnish header in the logs and
during the import strip off the second number.

Is that a sequential number or a random number?
What is its range?





On Wed, Nov 2, 2011 at 4:27 PM, Per Buer <perbu at varnish-software.com> wrote:

> Hi Mit,
>
> On Wed, Nov 2, 2011 at 8:03 PM, Mit Rowe <mit at stagename.com> wrote:
>
>>
>> I was wondering... does varnish internally generate a reliably random
>> unique value for each request (similar to mod_unique_id in apache or a
>> uuid/guid), and if so is there a way that i can write it out to my logs,
>> perhaps via an insertion into a request header and using the %{HEADER}i
>> macro in varnishncsa?
>>
>
> There is a request ID, in Varnishlog you'll see it as the last number in
> the ReqStart lines ie.
>
>     1932 ReqStart     c 127.0.0.1 54471 1236808814
>
> You could probably extract it from the X-Varnish header, it will be the
> first of one or two numbers in that header.
>
> --
> Per Buer, CEO
> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
> *Varnish makes websites fly!*
> Whitepapers <http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
> Twitter <https://twitter.com/varnishsoftware>
>
>
>


-- 
Will 'Mit' Rowe
Stagename*
*1-866-326-3098
mit at stagename.com <josh at stagename.com>
www.stagename.com
Twitter: @stagename

*The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of this
information by persons or entities other than the intended recipient is
prohibited. If you received this transmission in error, please contact the
sender and delete all material contained herein from your computer.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/d578171f/attachment.html>

From perbu at varnish-software.com  Wed Nov  2 21:05:39 2011
From: perbu at varnish-software.com (Per Buer)
Date: Wed, 2 Nov 2011 22:05:39 +0100
Subject: unique request ids
In-Reply-To: <CAFWPQt5J35oDhuh8tXXco0UjZ+vYGGMas-MJDL5uU0j1+ePEcg@mail.gmail.com>
References: <CAFWPQt5JArJijQqxnHz5AKQtfm5o-zhyciCotPjmH3sbP3GVNw@mail.gmail.com>
	<CAOXZevBCRj26=_24jaRiga3hEKjXFFXKnKGuRwvYXN2dF1-+ag@mail.gmail.com>
	<CAFWPQt5J35oDhuh8tXXco0UjZ+vYGGMas-MJDL5uU0j1+ePEcg@mail.gmail.com>
Message-ID: <CAOXZevAe3bxOGejx_qV-=NMGCQ1HBr-GF7Sk4Ff4zzOrYnzOOg@mail.gmail.com>

On Wed, Nov 2, 2011 at 9:43 PM, Mit Rowe <mit at stagename.com> wrote:

> Hi Per,
>
> That's looks promising, i could save the X-Varnish header in the logs and
> during the import strip off the second number.
>
> Is that a sequential number or a random number?
>

sequential.


> What is its range?
>

I'm guessing it is a 64bit int that would be unique for that instance of
Varnish. i.e. all bets are off if you restart or are running Varnish on
multiple servers. You might want to have a look at the source - look for
xid. Not sure where it is actually constructed.


-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111102/acff2546/attachment.html>

From ask at develooper.com  Wed Nov  2 22:33:14 2011
From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Wed, 2 Nov 2011 15:33:14 -0700
Subject: FAQ/Compression out of date
Message-ID: <C1314696-97E1-43E6-ADEB-9EC145D62717@develooper.com>

https://www.varnish-cache.org/trac/wiki/FAQ/Compression looks like it could use some qualified review to be updated to 3.0.


 - ask


From kelvin1111111 at gmail.com  Thu Nov  3 03:39:10 2011
From: kelvin1111111 at gmail.com (Kelvin Loke)
Date: Thu, 3 Nov 2011 11:39:10 +0800
Subject: Cache Wildcard Domains
Message-ID: <CAAct57yH0o-vW-n6Hdq44X=EjVQrTHPHo6gF1SyEU3LkXPrEoQ@mail.gmail.com>

Hi,

I am wondering if this can be done:

Visitor can access to our website using many different domains:

Eg.
www.123.com
www.abc.com
www.567.com
www.xyz.com

All these domains point to the same website same contents.

To better utilize the cache, can I keep one cache of
http://xxx.xxx.xxx/photo.png, instead of 4 cache for 4 different
domains?

Thanks!

Regards,
Kelvin Loke



From pprocacci at datapipe.com  Thu Nov  3 04:48:07 2011
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Wed, 2 Nov 2011 23:48:07 -0500
Subject: Cache Wildcard Domains
In-Reply-To: <CAAct57yH0o-vW-n6Hdq44X=EjVQrTHPHo6gF1SyEU3LkXPrEoQ@mail.gmail.com>
References: <CAAct57yH0o-vW-n6Hdq44X=EjVQrTHPHo6gF1SyEU3LkXPrEoQ@mail.gmail.com>
Message-ID: <20111103044807.GS47636@nat.myhome>

If you don't do any virtual hosting, you can modify vcl_hash to not hash based on the host parameter and just hash on the url:

############################
sub vcl_hash {
   hash_data(req.url);
   return (hash);
}
############################

Otherwise, you'd need to normalize the host head I think.

~Paul


On Thu, Nov 03, 2011 at 11:39:10AM +0800, Kelvin Loke wrote:
> Hi,
>
> I am wondering if this can be done:
>
> Visitor can access to our website using many different domains:
>
> Eg.
> www.123.com
> www.abc.com
> www.567.com
> www.xyz.com
>
> All these domains point to the same website same contents.
>
> To better utilize the cache, can I keep one cache of
> http://xxx.xxx.xxx/photo.png, instead of 4 cache for 4 different
> domains?
>
> Thanks!
>
> Regards,
> Kelvin Loke
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.



From perbu at varnish-software.com  Thu Nov  3 09:27:51 2011
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 3 Nov 2011 10:27:51 +0100
Subject: FAQ/Compression out of date
In-Reply-To: <C1314696-97E1-43E6-ADEB-9EC145D62717@develooper.com>
References: <C1314696-97E1-43E6-ADEB-9EC145D62717@develooper.com>
Message-ID: <CAOXZevArxk9B2vaa91jhG-7d5SSLUFzV5U03y3FyaQk68OTeqg@mail.gmail.com>

Hi Ask,

On Wed, Nov 2, 2011 at 11:33 PM, Ask Bj?rn Hansen <ask at develooper.com>wrote:

> https://www.varnish-cache.org/trac/wiki/FAQ/Compression looks like it
> could use some qualified review to be updated to 3.0.
>

Thanks for the heads up. I've pushed some docs (
https://www.varnish-cache.org/docs/trunk/tutorial/compression.html ) which
should remedy the issue.


Per.

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111103/f75a2373/attachment.html>

From tfheen at varnish-software.com  Thu Nov  3 12:34:22 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Thu, 03 Nov 2011 13:34:22 +0100
Subject: unique request ids
In-Reply-To: <CAOXZevAe3bxOGejx_qV-=NMGCQ1HBr-GF7Sk4Ff4zzOrYnzOOg@mail.gmail.com>
	(Per Buer's message of "Wed, 2 Nov 2011 22:05:39 +0100")
References: <CAFWPQt5JArJijQqxnHz5AKQtfm5o-zhyciCotPjmH3sbP3GVNw@mail.gmail.com>
	<CAOXZevBCRj26=_24jaRiga3hEKjXFFXKnKGuRwvYXN2dF1-+ag@mail.gmail.com>
	<CAFWPQt5J35oDhuh8tXXco0UjZ+vYGGMas-MJDL5uU0j1+ePEcg@mail.gmail.com>
	<CAOXZevAe3bxOGejx_qV-=NMGCQ1HBr-GF7Sk4Ff4zzOrYnzOOg@mail.gmail.com>
Message-ID: <87zkgd5qr5.fsf@qurzaw.varnish-software.com>

]] Per Buer 

| > What is its range?
| 
| I'm guessing it is a 64bit int that would be unique for that instance of
| Varnish. i.e. all bets are off if you restart or are running Varnish on
| multiple servers. You might want to have a look at the source - look for
| xid. Not sure where it is actually constructed.

It's seeded as a random number and then increased (albeit unlocked) so
you can end up with duplicates.

It's also a 32 bit number, so it will eventually wrap (about 10 days
wrap time if you average 5kr/s).

Cheers,
-- 
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64



From rlsmile at gmail.com  Wed Nov  2 11:13:02 2011
From: rlsmile at gmail.com (Richard Lane)
Date: Wed, 2 Nov 2011 06:13:02 -0500
Subject: Varnish in front of apache with name and ip based virtualhosts
Message-ID: <E22B5766-2375-484B-A0DA-3E85049B2F3D@gmail.com>

We have setup varnish on port 80 in front of 4 separate physical servers running apache on port 81. Varnish points to these apache backend servers.
Each apache server host the same multiple sites on each node. Are you looking to split request for certain sites to a particular apache/backend server?
I can take one backend sever out of service and varnish will not send request to that server. 





From thierry.magnien at sfr.com  Thu Nov  3 16:14:50 2011
From: thierry.magnien at sfr.com (MAGNIEN, Thierry)
Date: Thu, 3 Nov 2011 17:14:50 +0100
Subject: Backend DNS lookup refresh
Message-ID: <4A029B1A60B8E340A50D654D2F130DAA2FF1D518B2@EXCV001.encara.local.ads>

Hi all,

We are dealing with a great number of backends in our VCL file and we have problems with some backends changing IP address (for example when hosted on services such as Amazon S3, that switch IP address without telling anyone).

We would like to be able to refresh the DNS resolution of backends when needed and after a quick look at the source code, it seems there are several ways to handle this.

I would like to get your feeling about what would be the best solution before trying to write a patch, so here are my thoughts, just as it comes out of my mind ;-)

DNS resolution of backends is done when the VCL is loaded. IP address is stored in the compiled VCL as a sockaddr struct.

The first way to solve the problem would be to implement a "vcl reload" CLI function. In that case, this needs some tricks to be able to recompile a VCL which has the same name of the one used.

A 2nd way would be to make backend IP addresses resolved when needed (at first backend request and then cached for a "TTL" time), and not at VCL loading. However, this breaks most of the the optimization brought by the sockaddr structure.

A point discussed quickly with Per Buer on IRC was to have probes checking the IP addresses of the backends and update when an IP change is detected.

All this makes me think that it's not so easy as it could seem, and that's why I would accept any advice/idea/comment about this.

Regards,
Thierry




From mehul at onlygizmos.com  Fri Nov  4 07:57:23 2011
From: mehul at onlygizmos.com (Mehul Ved)
Date: Fri, 4 Nov 2011 13:27:23 +0530
Subject: Varnish in front of apache with name and ip based virtualhosts
In-Reply-To: <E22B5766-2375-484B-A0DA-3E85049B2F3D@gmail.com>
References: <E22B5766-2375-484B-A0DA-3E85049B2F3D@gmail.com>
Message-ID: <CAJC7t5MM6n3b_bReenf8c+2t4yEim5-aiatN5aEoLTexLDUXeA@mail.gmail.com>

On Wed, Nov 2, 2011 at 4:43 PM, Richard Lane <rlsmile at gmail.com> wrote:

Hi Richard,

We have setup varnish on port 80 in front of 4 separate physical servers
> running apache on port 81. Varnish points to these apache backend servers.
> Each apache server host the same multiple sites on each node. Are you
> looking to split request for certain sites to a particular apache/backend
> server?
> I can take one backend sever out of service and varnish will not send
> request to that server.


My setup is much simpler. Just one physical server with multiple IP
addresses. The server.ip with multiple backends worked perfectly.


-- 
With Regards
Mehul Ved
9757062714
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111104/117f0a92/attachment.html>

From AGresens at Scholastic.com  Fri Nov  4 18:44:22 2011
From: AGresens at Scholastic.com (Gresens, August)
Date: Fri, 4 Nov 2011 14:44:22 -0400
Subject: Intermittent 503's from Varnish
Message-ID: <8D333BF67F814C4D9803C99961BE5E7005E04092@corpex07.corp.scholasticinc.local>

Hello

 

We're having a very nasty production issue. We're getting intermittent
503 from Varnish in front of a healthy backend.

 

This seems to happen on cache-miss, but we're not seeing requests make
it to the backend. Varnish immediately returns the 503. After the first
503, the page can usually be loaded by requesting it a few more times.
This seems to indicate that there is some condition within Varnish that
prevents it from hitting the backend that clears up quickly.

 

Has anyone seen this issue before? Is there anything we can try?

 

This is our startup config:

 

DAEMON_OPTS="-a :80 \

-T localhost:6082 \

-f /etc/varnish/default.vcl \

-S /etc/varnish/secret \

-s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \

-p http_headers=128 \

-p sess_workspace=262144 \

-p connect_timeout=60 \

-p max_restarts=6 \

-p log_hashstring=on \

-p cli_timeout=60 \

-t 0"

 

We are on version 2.0.5

 

Thanks,

 

August

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111104/12576140/attachment.html>

From rtshilston at gmail.com  Fri Nov  4 18:52:39 2011
From: rtshilston at gmail.com (Rob S)
Date: Fri, 4 Nov 2011 18:52:39 +0000
Subject: Intermittent 503's from Varnish
In-Reply-To: <8D333BF67F814C4D9803C99961BE5E7005E04092@corpex07.corp.scholasticinc.local>
References: <8D333BF67F814C4D9803C99961BE5E7005E04092@corpex07.corp.scholasticinc.local>
Message-ID: <CAKF-8eCDT68o0DzvF6juFzMGR7w+Eh=p57BkDmCyvCv1MOxsPA@mail.gmail.com>

Can you send the varnishlog for one of these failing requests? ?Is
this a POST or a GET request? ?Are you using default VCL, or can you
send the VCL you are using?

Rob
On Fri, Nov 4, 2011 at 6:44 PM, Gresens, August <AGresens at scholastic.com> wrote:
> Hello
>
>
>
> We?re having a very nasty production issue. We?re getting intermittent 503
> from Varnish in front of a healthy backend.
>
>
>
> This seems to happen on cache-miss, but we?re not seeing requests make it to
> the backend. Varnish immediately returns the 503. After the first 503, the
> page can usually be loaded by requesting it a few more times. This seems to
> indicate that there is some condition within Varnish that prevents it from
> hitting the backend that clears up quickly.
>
>
>
> Has anyone seen this issue before? Is there anything we can try?
>
>
>
> This is our startup config:
>
>
>
> DAEMON_OPTS="-a :80 \
>
> -T localhost:6082 \
>
> -f /etc/varnish/default.vcl \
>
> -S /etc/varnish/secret \
>
> -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \
>
> -p http_headers=128 \
>
> -p sess_workspace=262144 \
>
> -p connect_timeout=60 \
>
> -p max_restarts=6 \
>
> -p log_hashstring=on \
>
> -p cli_timeout=60 \
>
> -t 0"
>
>
>
> We are on version 2.0.5
>
>
>
> Thanks,
>
>
>
> August
>
>
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



From leiwang at rhapsody.com  Fri Nov  4 19:39:58 2011
From: leiwang at rhapsody.com (Lei Wang)
Date: Fri, 4 Nov 2011 12:39:58 -0700
Subject: Intermittent 503's from Varnish
In-Reply-To: <CAKF-8eCDT68o0DzvF6juFzMGR7w+Eh=p57BkDmCyvCv1MOxsPA@mail.gmail.com>
References: <8D333BF67F814C4D9803C99961BE5E7005E04092@corpex07.corp.scholasticinc.local>
	<CAKF-8eCDT68o0DzvF6juFzMGR7w+Eh=p57BkDmCyvCv1MOxsPA@mail.gmail.com>
Message-ID: <CAD_kORfhnC3tDwCm--yWgk_w3A0UA1Lp=WGtSf-wNjtZQ1O6WQ@mail.gmail.com>

I had the same problem. Please help!

/usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f
/etc/varnish/varnish80.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u
varnish -g varnish -i varnish81 -S /etc/varnish/secret -p
thread_pool_add_delay 2 -p thread_pools 1 -p thread_pool_min 800 -p
thread_pool_max 2000 -p session_linger 50 -p sess_workspace 4096 -n
varnish80 -s malloc,6.5G

configuration related to this problem:
backend eadrm {
  .host = "drm-ea.xxxxxxxx.com";
  .port = "80";
  .connect_timeout = 40s;
  .first_byte_timeout = 80s;
  .between_bytes_timeout = 40s;
}

  }  elsif (req.http.host ~ "drm-ea") {
     unset beresp.http.set-cookie;
     set beresp.ttl = 86400s;
     log "Adjusting EA-DRM TTL to 86400 seconds";

Thanks,

Lei
On Fri, Nov 4, 2011 at 11:52 AM, Rob S <rtshilston at gmail.com> wrote:

> Can you send the varnishlog for one of these failing requests?  Is
> this a POST or a GET request?  Are you using default VCL, or can you
> send the VCL you are using?
>
> Rob
> On Fri, Nov 4, 2011 at 6:44 PM, Gresens, August <AGresens at scholastic.com>
> wrote:
> > Hello
> >
> >
> >
> > We?re having a very nasty production issue. We?re getting intermittent
> 503
> > from Varnish in front of a healthy backend.
> >
> >
> >
> > This seems to happen on cache-miss, but we?re not seeing requests make
> it to
> > the backend. Varnish immediately returns the 503. After the first 503,
> the
> > page can usually be loaded by requesting it a few more times. This seems
> to
> > indicate that there is some condition within Varnish that prevents it
> from
> > hitting the backend that clears up quickly.
> >
> >
> >
> > Has anyone seen this issue before? Is there anything we can try?
> >
> >
> >
> > This is our startup config:
> >
> >
> >
> > DAEMON_OPTS="-a :80 \
> >
> > -T localhost:6082 \
> >
> > -f /etc/varnish/default.vcl \
> >
> > -S /etc/varnish/secret \
> >
> > -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \
> >
> > -p http_headers=128 \
> >
> > -p sess_workspace=262144 \
> >
> > -p connect_timeout=60 \
> >
> > -p max_restarts=6 \
> >
> > -p log_hashstring=on \
> >
> > -p cli_timeout=60 \
> >
> > -t 0"
> >
> >
> >
> > We are on version 2.0.5
> >
> >
> >
> > Thanks,
> >
> >
> >
> > August
> >
> >
> >
> >
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111104/86b7568d/attachment.html>

From AGresens at Scholastic.com  Fri Nov  4 19:48:11 2011
From: AGresens at Scholastic.com (Gresens, August)
Date: Fri, 4 Nov 2011 15:48:11 -0400
Subject: Intermittent 503's from Varnish
In-Reply-To: <CAD_kORfhnC3tDwCm--yWgk_w3A0UA1Lp=WGtSf-wNjtZQ1O6WQ@mail.gmail.com>
Message-ID: <8D333BF67F814C4D9803C99961BE5E7005E04168@corpex07.corp.scholasticinc.local>

We were able to resolve this. Our Varnish instance is pointing a load
balanced  pool of backend servers (a vip on a hardware load balancer),
so to Varnish there is only one backend. We had a situation in which
certain images were returning 500 errors from the backend. Because saint
mode was enabled, it invalidated the backend (the only back end) after a
certain number of failures within a particular window. We disabled saint
mode and this fixed this issue. Best, August

 

________________________________

From: Lei Wang [mailto:leiwang at rhapsody.com] 
Sent: Friday, November 04, 2011 3:40 PM
To: Rob S
Cc: Gresens, August; varnish-misc at varnish-cache.org
Subject: Re: Intermittent 503's from Varnish

 

I had the same problem. Please help!

/usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f
/etc/varnish/varnish80.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u
varnish -g varnish -i varnish81 -S /etc/varnish/secret -p
thread_pool_add_delay 2 -p thread_pools 1 -p thread_pool_min 800 -p
thread_pool_max 2000 -p session_linger 50 -p sess_workspace 4096 -n
varnish80 -s malloc,6.5G

configuration related to this problem:
backend eadrm {
  .host = "drm-ea.xxxxxxxx.com";
  .port = "80";
  .connect_timeout = 40s;
  .first_byte_timeout = 80s;
  .between_bytes_timeout = 40s;
}

  }  elsif (req.http.host ~ "drm-ea") {
     unset beresp.http.set-cookie;
     set beresp.ttl = 86400s;
     log "Adjusting EA-DRM TTL to 86400 seconds";

Thanks,

Lei

On Fri, Nov 4, 2011 at 11:52 AM, Rob S <rtshilston at gmail.com> wrote:

Can you send the varnishlog for one of these failing requests?  Is
this a POST or a GET request?  Are you using default VCL, or can you
send the VCL you are using?

Rob

On Fri, Nov 4, 2011 at 6:44 PM, Gresens, August
<AGresens at scholastic.com> wrote:
> Hello
>
>
>
> We're having a very nasty production issue. We're getting intermittent
503
> from Varnish in front of a healthy backend.
>
>
>
> This seems to happen on cache-miss, but we're not seeing requests make
it to
> the backend. Varnish immediately returns the 503. After the first 503,
the
> page can usually be loaded by requesting it a few more times. This
seems to
> indicate that there is some condition within Varnish that prevents it
from
> hitting the backend that clears up quickly.
>
>
>
> Has anyone seen this issue before? Is there anything we can try?
>
>
>
> This is our startup config:
>
>
>
> DAEMON_OPTS="-a :80 \
>
> -T localhost:6082 \
>
> -f /etc/varnish/default.vcl \
>
> -S /etc/varnish/secret \
>
> -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \
>
> -p http_headers=128 \
>
> -p sess_workspace=262144 \
>
> -p connect_timeout=60 \
>
> -p max_restarts=6 \
>
> -p log_hashstring=on \
>
> -p cli_timeout=60 \
>
> -t 0"
>
>
>
> We are on version 2.0.5
>
>
>
> Thanks,
>
>
>
> August
>
>
>
>
>

> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111104/72dc389c/attachment.html>

From leiwang at rhapsody.com  Fri Nov  4 20:26:50 2011
From: leiwang at rhapsody.com (Lei Wang)
Date: Fri, 4 Nov 2011 13:26:50 -0700
Subject: Intermittent 503's from Varnish
In-Reply-To: <8D333BF67F814C4D9803C99961BE5E7005E04168@corpex07.corp.scholasticinc.local>
References: <CAD_kORfhnC3tDwCm--yWgk_w3A0UA1Lp=WGtSf-wNjtZQ1O6WQ@mail.gmail.com>
	<8D333BF67F814C4D9803C99961BE5E7005E04168@corpex07.corp.scholasticinc.local>
Message-ID: <CAD_kORcxrk3JrTANuP8_FGXKt6rWmhHcxQXnPA2wOLWjcEt0Ww@mail.gmail.com>

Hi August,

Thank for the info. I will definitely try that today.

Have a nice day.

Lei

On Fri, Nov 4, 2011 at 12:48 PM, Gresens, August <AGresens at scholastic.com>wrote:

>  We were able to resolve this. Our Varnish instance is pointing a load
> balanced  pool of backend servers (a vip on a hardware load balancer), so
> to Varnish there is only one backend. We had a situation in which certain
> images were returning 500 errors from the backend. Because saint mode was
> enabled, it invalidated the backend (the only back end) after a certain
> number of failures within a particular window. We disabled saint mode and
> this fixed this issue. Best, August****
>
> ** **
>  ------------------------------
>
> *From:* Lei Wang [mailto:leiwang at rhapsody.com]
> *Sent:* Friday, November 04, 2011 3:40 PM
> *To:* Rob S
> *Cc:* Gresens, August; varnish-misc at varnish-cache.org
> *Subject:* Re: Intermittent 503's from Varnish****
>
> ** **
>
> I had the same problem. Please help!
>
> /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f
> /etc/varnish/varnish80.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u
> varnish -g varnish -i varnish81 -S /etc/varnish/secret -p
> thread_pool_add_delay 2 -p thread_pools 1 -p thread_pool_min 800 -p
> thread_pool_max 2000 -p session_linger 50 -p sess_workspace 4096 -n
> varnish80 -s malloc,6.5G
>
> configuration related to this problem:
> backend eadrm {
>   .host = "drm-ea.xxxxxxxx.com";
>   .port = "80";
>   .connect_timeout = 40s;
>   .first_byte_timeout = 80s;
>   .between_bytes_timeout = 40s;
> }
>
>   }  elsif (req.http.host ~ "drm-ea") {
>      unset beresp.http.set-cookie;
>      set beresp.ttl = 86400s;
>      log "Adjusting EA-DRM TTL to 86400 seconds";
>
> Thanks,
>
> Lei****
>
> On Fri, Nov 4, 2011 at 11:52 AM, Rob S <rtshilston at gmail.com> wrote:****
>
> Can you send the varnishlog for one of these failing requests?  Is
> this a POST or a GET request?  Are you using default VCL, or can you
> send the VCL you are using?
>
> Rob****
>
> On Fri, Nov 4, 2011 at 6:44 PM, Gresens, August <AGresens at scholastic.com>
> wrote:
> > Hello
> >
> >
> >
> > We?re having a very nasty production issue. We?re getting intermittent
> 503
> > from Varnish in front of a healthy backend.
> >
> >
> >
> > This seems to happen on cache-miss, but we?re not seeing requests make
> it to
> > the backend. Varnish immediately returns the 503. After the first 503,
> the
> > page can usually be loaded by requesting it a few more times. This seems
> to
> > indicate that there is some condition within Varnish that prevents it
> from
> > hitting the backend that clears up quickly.
> >
> >
> >
> > Has anyone seen this issue before? Is there anything we can try?
> >
> >
> >
> > This is our startup config:
> >
> >
> >
> > DAEMON_OPTS="-a :80 \
> >
> > -T localhost:6082 \
> >
> > -f /etc/varnish/default.vcl \
> >
> > -S /etc/varnish/secret \
> >
> > -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \
> >
> > -p http_headers=128 \
> >
> > -p sess_workspace=262144 \
> >
> > -p connect_timeout=60 \
> >
> > -p max_restarts=6 \
> >
> > -p log_hashstring=on \
> >
> > -p cli_timeout=60 \
> >
> > -t 0"
> >
> >
> >
> > We are on version 2.0.5
> >
> >
> >
> > Thanks,
> >
> >
> >
> > August
> >
> >
> >
> >
> >****
>
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc****
>
> ** **
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111104/48abd99e/attachment.html>

From straightflush at gmail.com  Fri Nov  4 20:33:51 2011
From: straightflush at gmail.com (AD)
Date: Fri, 4 Nov 2011 16:33:51 -0400
Subject: Intermittent 503's from Varnish
In-Reply-To: <CAD_kORcxrk3JrTANuP8_FGXKt6rWmhHcxQXnPA2wOLWjcEt0Ww@mail.gmail.com>
References: <CAD_kORfhnC3tDwCm--yWgk_w3A0UA1Lp=WGtSf-wNjtZQ1O6WQ@mail.gmail.com>
	<8D333BF67F814C4D9803C99961BE5E7005E04168@corpex07.corp.scholasticinc.local>
	<CAD_kORcxrk3JrTANuP8_FGXKt6rWmhHcxQXnPA2wOLWjcEt0Ww@mail.gmail.com>
Message-ID: <CANZWjw4jviRFXhLftWu2FwBcpPKZ2yBi1Cwrb4xQCHHTbRVF4w@mail.gmail.com>

Shouldnt saint mode return a 200 OK (stale cache item) if the backend fails
and you are within the "Saint Window" for that object?  I dont think
turning off saint mode is going to correct this issue.

On Fri, Nov 4, 2011 at 4:26 PM, Lei Wang <leiwang at rhapsody.com> wrote:

> Hi August,
>
> Thank for the info. I will definitely try that today.
>
> Have a nice day.
>
> Lei
>
>
> On Fri, Nov 4, 2011 at 12:48 PM, Gresens, August <AGresens at scholastic.com>wrote:
>
>>  We were able to resolve this. Our Varnish instance is pointing a load
>> balanced  pool of backend servers (a vip on a hardware load balancer), so
>> to Varnish there is only one backend. We had a situation in which certain
>> images were returning 500 errors from the backend. Because saint mode was
>> enabled, it invalidated the backend (the only back end) after a certain
>> number of failures within a particular window. We disabled saint mode and
>> this fixed this issue. Best, August****
>>
>> ** **
>>  ------------------------------
>>
>> *From:* Lei Wang [mailto:leiwang at rhapsody.com]
>> *Sent:* Friday, November 04, 2011 3:40 PM
>> *To:* Rob S
>> *Cc:* Gresens, August; varnish-misc at varnish-cache.org
>> *Subject:* Re: Intermittent 503's from Varnish****
>>
>> ** **
>>
>> I had the same problem. Please help!
>>
>> /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f
>> /etc/varnish/varnish80.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u
>> varnish -g varnish -i varnish81 -S /etc/varnish/secret -p
>> thread_pool_add_delay 2 -p thread_pools 1 -p thread_pool_min 800 -p
>> thread_pool_max 2000 -p session_linger 50 -p sess_workspace 4096 -n
>> varnish80 -s malloc,6.5G
>>
>> configuration related to this problem:
>> backend eadrm {
>>   .host = "drm-ea.xxxxxxxx.com";
>>   .port = "80";
>>   .connect_timeout = 40s;
>>   .first_byte_timeout = 80s;
>>   .between_bytes_timeout = 40s;
>> }
>>
>>   }  elsif (req.http.host ~ "drm-ea") {
>>      unset beresp.http.set-cookie;
>>      set beresp.ttl = 86400s;
>>      log "Adjusting EA-DRM TTL to 86400 seconds";
>>
>> Thanks,
>>
>> Lei****
>>
>> On Fri, Nov 4, 2011 at 11:52 AM, Rob S <rtshilston at gmail.com> wrote:****
>>
>> Can you send the varnishlog for one of these failing requests?  Is
>> this a POST or a GET request?  Are you using default VCL, or can you
>> send the VCL you are using?
>>
>> Rob****
>>
>> On Fri, Nov 4, 2011 at 6:44 PM, Gresens, August <AGresens at scholastic.com>
>> wrote:
>> > Hello
>> >
>> >
>> >
>> > We?re having a very nasty production issue. We?re getting intermittent
>> 503
>> > from Varnish in front of a healthy backend.
>> >
>> >
>> >
>> > This seems to happen on cache-miss, but we?re not seeing requests make
>> it to
>> > the backend. Varnish immediately returns the 503. After the first 503,
>> the
>> > page can usually be loaded by requesting it a few more times. This
>> seems to
>> > indicate that there is some condition within Varnish that prevents it
>> from
>> > hitting the backend that clears up quickly.
>> >
>> >
>> >
>> > Has anyone seen this issue before? Is there anything we can try?
>> >
>> >
>> >
>> > This is our startup config:
>> >
>> >
>> >
>> > DAEMON_OPTS="-a :80 \
>> >
>> > -T localhost:6082 \
>> >
>> > -f /etc/varnish/default.vcl \
>> >
>> > -S /etc/varnish/secret \
>> >
>> > -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \
>> >
>> > -p http_headers=128 \
>> >
>> > -p sess_workspace=262144 \
>> >
>> > -p connect_timeout=60 \
>> >
>> > -p max_restarts=6 \
>> >
>> > -p log_hashstring=on \
>> >
>> > -p cli_timeout=60 \
>> >
>> > -t 0"
>> >
>> >
>> >
>> > We are on version 2.0.5
>> >
>> >
>> >
>> > Thanks,
>> >
>> >
>> >
>> > August
>> >
>> >
>> >
>> >
>> >****
>>
>> > _______________________________________________
>> > varnish-misc mailing list
>> > varnish-misc at varnish-cache.org
>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>> >
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc****
>>
>> ** **
>>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111104/8fe01fe8/attachment.html>

From graham.lyons at bbc.co.uk  Mon Nov  7 09:06:01 2011
From: graham.lyons at bbc.co.uk (Graham Lyons)
Date: Mon, 7 Nov 2011 09:06:01 +0000
Subject: varnish-misc Digest, Vol 68, Issue 9
In-Reply-To: <mailman.3.1320490802.330.varnish-misc@varnish-cache.org>
References: <mailman.3.1320490802.330.varnish-misc@varnish-cache.org>
Message-ID: <2DE3D00F-77F5-4751-9D8E-4BD9DFADC4F3@bbc.co.uk>

We did some experimentation with saint mode before deciding not to turn it on because we found that we'd be opening ourselves up to exactly this issue.

Once the saint mode threshold is reached for a backend then that backend won't be contacted for anything at all and Varnish will return a 503. It can't return the stale content if it's not in the cache.

On 5 Nov 2011, at 11:00, varnish-misc-request at varnish-cache.org wrote:

> Send varnish-misc mailing list submissions to
> 	varnish-misc at varnish-cache.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> or, via email, send a message with subject or body 'help' to
> 	varnish-misc-request at varnish-cache.org
> 
> You can reach the person managing the list at
> 	varnish-misc-owner at varnish-cache.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of varnish-misc digest..."
> 
> 
> Today's Topics:
> 
>   1. Re: Intermittent 503's from Varnish (Lei Wang)
>   2. Re: Intermittent 503's from Varnish (AD)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Fri, 4 Nov 2011 13:26:50 -0700
> From: Lei Wang <leiwang at rhapsody.com>
> To: "Gresens, August" <AGresens at scholastic.com>
> Cc: varnish-misc at varnish-cache.org
> Subject: Re: Intermittent 503's from Varnish
> Message-ID:
> 	<CAD_kORcxrk3JrTANuP8_FGXKt6rWmhHcxQXnPA2wOLWjcEt0Ww at mail.gmail.com>
> Content-Type: text/plain; charset="windows-1252"
> 
> Hi August,
> 
> Thank for the info. I will definitely try that today.
> 
> Have a nice day.
> 
> Lei
> 
> On Fri, Nov 4, 2011 at 12:48 PM, Gresens, August <AGresens at scholastic.com>wrote:
> 
>> We were able to resolve this. Our Varnish instance is pointing a load
>> balanced  pool of backend servers (a vip on a hardware load balancer), so
>> to Varnish there is only one backend. We had a situation in which certain
>> images were returning 500 errors from the backend. Because saint mode was
>> enabled, it invalidated the backend (the only back end) after a certain
>> number of failures within a particular window. We disabled saint mode and
>> this fixed this issue. Best, August****
>> 
>> ** **
>> ------------------------------
>> 
>> *From:* Lei Wang [mailto:leiwang at rhapsody.com]
>> *Sent:* Friday, November 04, 2011 3:40 PM
>> *To:* Rob S
>> *Cc:* Gresens, August; varnish-misc at varnish-cache.org
>> *Subject:* Re: Intermittent 503's from Varnish****
>> 
>> ** **
>> 
>> I had the same problem. Please help!
>> 
>> /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f
>> /etc/varnish/varnish80.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u
>> varnish -g varnish -i varnish81 -S /etc/varnish/secret -p
>> thread_pool_add_delay 2 -p thread_pools 1 -p thread_pool_min 800 -p
>> thread_pool_max 2000 -p session_linger 50 -p sess_workspace 4096 -n
>> varnish80 -s malloc,6.5G
>> 
>> configuration related to this problem:
>> backend eadrm {
>>  .host = "drm-ea.xxxxxxxx.com";
>>  .port = "80";
>>  .connect_timeout = 40s;
>>  .first_byte_timeout = 80s;
>>  .between_bytes_timeout = 40s;
>> }
>> 
>>  }  elsif (req.http.host ~ "drm-ea") {
>>     unset beresp.http.set-cookie;
>>     set beresp.ttl = 86400s;
>>     log "Adjusting EA-DRM TTL to 86400 seconds";
>> 
>> Thanks,
>> 
>> Lei****
>> 
>> On Fri, Nov 4, 2011 at 11:52 AM, Rob S <rtshilston at gmail.com> wrote:****
>> 
>> Can you send the varnishlog for one of these failing requests?  Is
>> this a POST or a GET request?  Are you using default VCL, or can you
>> send the VCL you are using?
>> 
>> Rob****
>> 
>> On Fri, Nov 4, 2011 at 6:44 PM, Gresens, August <AGresens at scholastic.com>
>> wrote:
>>> Hello
>>> 
>>> 
>>> 
>>> We?re having a very nasty production issue. We?re getting intermittent
>> 503
>>> from Varnish in front of a healthy backend.
>>> 
>>> 
>>> 
>>> This seems to happen on cache-miss, but we?re not seeing requests make
>> it to
>>> the backend. Varnish immediately returns the 503. After the first 503,
>> the
>>> page can usually be loaded by requesting it a few more times. This seems
>> to
>>> indicate that there is some condition within Varnish that prevents it
>> from
>>> hitting the backend that clears up quickly.
>>> 
>>> 
>>> 
>>> Has anyone seen this issue before? Is there anything we can try?
>>> 
>>> 
>>> 
>>> This is our startup config:
>>> 
>>> 
>>> 
>>> DAEMON_OPTS="-a :80 \
>>> 
>>> -T localhost:6082 \
>>> 
>>> -f /etc/varnish/default.vcl \
>>> 
>>> -S /etc/varnish/secret \
>>> 
>>> -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \
>>> 
>>> -p http_headers=128 \
>>> 
>>> -p sess_workspace=262144 \
>>> 
>>> -p connect_timeout=60 \
>>> 
>>> -p max_restarts=6 \
>>> 
>>> -p log_hashstring=on \
>>> 
>>> -p cli_timeout=60 \
>>> 
>>> -t 0"
>>> 
>>> 
>>> 
>>> We are on version 2.0.5
>>> 
>>> 
>>> 
>>> Thanks,
>>> 
>>> 
>>> 
>>> August
>>> 
>>> 
>>> 
>>> 
>>> ****
>> 
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>> 
>> 
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc****
>> 
>> ** **
>> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20111104/48abd99e/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 2
> Date: Fri, 4 Nov 2011 16:33:51 -0400
> From: AD <straightflush at gmail.com>
> To: Lei Wang <leiwang at rhapsody.com>
> Cc: varnish-misc at varnish-cache.org
> Subject: Re: Intermittent 503's from Varnish
> Message-ID:
> 	<CANZWjw4jviRFXhLftWu2FwBcpPKZ2yBi1Cwrb4xQCHHTbRVF4w at mail.gmail.com>
> Content-Type: text/plain; charset="windows-1252"
> 
> Shouldnt saint mode return a 200 OK (stale cache item) if the backend fails
> and you are within the "Saint Window" for that object?  I dont think
> turning off saint mode is going to correct this issue.
> 
> On Fri, Nov 4, 2011 at 4:26 PM, Lei Wang <leiwang at rhapsody.com> wrote:
> 
>> Hi August,
>> 
>> Thank for the info. I will definitely try that today.
>> 
>> Have a nice day.
>> 
>> Lei
>> 
>> 
>> On Fri, Nov 4, 2011 at 12:48 PM, Gresens, August <AGresens at scholastic.com>wrote:
>> 
>>> We were able to resolve this. Our Varnish instance is pointing a load
>>> balanced  pool of backend servers (a vip on a hardware load balancer), so
>>> to Varnish there is only one backend. We had a situation in which certain
>>> images were returning 500 errors from the backend. Because saint mode was
>>> enabled, it invalidated the backend (the only back end) after a certain
>>> number of failures within a particular window. We disabled saint mode and
>>> this fixed this issue. Best, August****
>>> 
>>> ** **
>>> ------------------------------
>>> 
>>> *From:* Lei Wang [mailto:leiwang at rhapsody.com]
>>> *Sent:* Friday, November 04, 2011 3:40 PM
>>> *To:* Rob S
>>> *Cc:* Gresens, August; varnish-misc at varnish-cache.org
>>> *Subject:* Re: Intermittent 503's from Varnish****
>>> 
>>> ** **
>>> 
>>> I had the same problem. Please help!
>>> 
>>> /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f
>>> /etc/varnish/varnish80.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u
>>> varnish -g varnish -i varnish81 -S /etc/varnish/secret -p
>>> thread_pool_add_delay 2 -p thread_pools 1 -p thread_pool_min 800 -p
>>> thread_pool_max 2000 -p session_linger 50 -p sess_workspace 4096 -n
>>> varnish80 -s malloc,6.5G
>>> 
>>> configuration related to this problem:
>>> backend eadrm {
>>>  .host = "drm-ea.xxxxxxxx.com";
>>>  .port = "80";
>>>  .connect_timeout = 40s;
>>>  .first_byte_timeout = 80s;
>>>  .between_bytes_timeout = 40s;
>>> }
>>> 
>>>  }  elsif (req.http.host ~ "drm-ea") {
>>>     unset beresp.http.set-cookie;
>>>     set beresp.ttl = 86400s;
>>>     log "Adjusting EA-DRM TTL to 86400 seconds";
>>> 
>>> Thanks,
>>> 
>>> Lei****
>>> 
>>> On Fri, Nov 4, 2011 at 11:52 AM, Rob S <rtshilston at gmail.com> wrote:****
>>> 
>>> Can you send the varnishlog for one of these failing requests?  Is
>>> this a POST or a GET request?  Are you using default VCL, or can you
>>> send the VCL you are using?
>>> 
>>> Rob****
>>> 
>>> On Fri, Nov 4, 2011 at 6:44 PM, Gresens, August <AGresens at scholastic.com>
>>> wrote:
>>>> Hello
>>>> 
>>>> 
>>>> 
>>>> We?re having a very nasty production issue. We?re getting intermittent
>>> 503
>>>> from Varnish in front of a healthy backend.
>>>> 
>>>> 
>>>> 
>>>> This seems to happen on cache-miss, but we?re not seeing requests make
>>> it to
>>>> the backend. Varnish immediately returns the 503. After the first 503,
>>> the
>>>> page can usually be loaded by requesting it a few more times. This
>>> seems to
>>>> indicate that there is some condition within Varnish that prevents it
>>> from
>>>> hitting the backend that clears up quickly.
>>>> 
>>>> 
>>>> 
>>>> Has anyone seen this issue before? Is there anything we can try?
>>>> 
>>>> 
>>>> 
>>>> This is our startup config:
>>>> 
>>>> 
>>>> 
>>>> DAEMON_OPTS="-a :80 \
>>>> 
>>>> -T localhost:6082 \
>>>> 
>>>> -f /etc/varnish/default.vcl \
>>>> 
>>>> -S /etc/varnish/secret \
>>>> 
>>>> -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \
>>>> 
>>>> -p http_headers=128 \
>>>> 
>>>> -p sess_workspace=262144 \
>>>> 
>>>> -p connect_timeout=60 \
>>>> 
>>>> -p max_restarts=6 \
>>>> 
>>>> -p log_hashstring=on \
>>>> 
>>>> -p cli_timeout=60 \
>>>> 
>>>> -t 0"
>>>> 
>>>> 
>>>> 
>>>> We are on version 2.0.5
>>>> 
>>>> 
>>>> 
>>>> Thanks,
>>>> 
>>>> 
>>>> 
>>>> August
>>>> 
>>>> 
>>>> 
>>>> 
>>>> ****
>>> 
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc at varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>> 
>>> 
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc****
>>> 
>>> ** **
>>> 
>> 
>> 
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20111104/8fe01fe8/attachment-0001.html>
> 
> ------------------------------
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> 
> End of varnish-misc Digest, Vol 68, Issue 9
> *******************************************


http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
					



From marc.fournier at camptocamp.com  Sun Nov  6 10:56:05 2011
From: marc.fournier at camptocamp.com (Marc Fournier)
Date: Sun, 06 Nov 2011 11:56:05 +0100
Subject: Backend DNS lookup refresh
In-Reply-To: <4A029B1A60B8E340A50D654D2F130DAA2FF1D518B2@EXCV001.encara.local.ads>
References: <4A029B1A60B8E340A50D654D2F130DAA2FF1D518B2@EXCV001.encara.local.ads>
Message-ID: <1320575076-sup-1624@lonquimay.wrk.lsn.camptocamp.com>

Hello Thierry,

Excerpts from MAGNIEN, Thierry's message of Thu Nov 03 17:14:50 +0100 2011:
> 
> We are dealing with a great number of backends in our VCL file and we
> have problems with some backends changing IP address (for example when
> hosted on services such as Amazon S3, that switch IP address without
> telling anyone).

A couple of years ago, we worked around the specific case of S3 by putting
a simple nginx proxy between S3 and varnish.

Amazon S3 emits HTTP redirects when DNS changes occur[?]. Unfortunately
varnish doesn't handle them either. But nginx does, as well as backend DNS
changes.

It doesn't really answer your question, but this somewhat ugly workaround
has been working very reliably for us since then. It would be nicer if
varnish could deal with these 2 cases, though.

Cheers,
Marc

[?] http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?Redirects.html



From steve.webster at lovefilm.com  Mon Nov  7 12:42:49 2011
From: steve.webster at lovefilm.com (Steve Webster)
Date: Mon, 7 Nov 2011 12:42:49 +0000
Subject: Intermittent 503's from Varnish
In-Reply-To: <CANZWjw4jviRFXhLftWu2FwBcpPKZ2yBi1Cwrb4xQCHHTbRVF4w@mail.gmail.com>
References: <CAD_kORfhnC3tDwCm--yWgk_w3A0UA1Lp=WGtSf-wNjtZQ1O6WQ@mail.gmail.com>
	<8D333BF67F814C4D9803C99961BE5E7005E04168@corpex07.corp.scholasticinc.local>
	<CAD_kORcxrk3JrTANuP8_FGXKt6rWmhHcxQXnPA2wOLWjcEt0Ww@mail.gmail.com>
	<CANZWjw4jviRFXhLftWu2FwBcpPKZ2yBi1Cwrb4xQCHHTbRVF4w@mail.gmail.com>
Message-ID: <15335923-6245-4A1B-8A01-513404788994@lovefilm.com>

On 4 Nov 2011, at 20:33, AD wrote:

> Shouldnt saint mode return a 200 OK (stale cache item) if the backend fails and you are within the "Saint Window" for that object?  I dont think turning off saint mode is going to correct this issue.

>From what I understand, what you are describing is "grace mode".

Saint mode is the "my backend is in trouble, please stop sending requests to it" protection, whereas grace mode allows Varnish to serve up stale content for a predetermined amount of time. Grace mode is useful for serving stale content when your backend is sick, but it's also useful when Varnish has already made a backend request for that data and is awaiting a response (i.e. similar to Squid 2.7's stale-while-revalidate).

See:

https://www.varnish-cache.org/docs/3.0/tutorial/handling_misbehaving_servers.html

Cheers,

Steve

-- 
Steve Webster
Web Architect
LOVEFiLM

(Apologies for the overly long and pointless corporate footer that follows.)

-----------------------------------------------------------------------------------------------------------------------------------------
LOVEFiLM UK Limited is a company registered in England and Wales. 
Registered Number: 06528297. 
Registered Office: No.9, 6 Portal Way, London W3 6RU, United Kingdom.

This e-mail is confidential to the ordinary user of the e-mail address to which it was addressed. If you have received it in error, 
please delete it from your system and notify the sender immediately.

This email message has been delivered safely and archived online by Mimecast.
For more information please visit http://www.mimecast.co.uk 
-----------------------------------------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111107/1dbb62ed/attachment.html>

From apj at mutt.dk  Mon Nov  7 12:59:21 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Mon, 7 Nov 2011 13:59:21 +0100
Subject: Intermittent 503's from Varnish
In-Reply-To: <CAD_kORfhnC3tDwCm--yWgk_w3A0UA1Lp=WGtSf-wNjtZQ1O6WQ@mail.gmail.com>
References: <8D333BF67F814C4D9803C99961BE5E7005E04092@corpex07.corp.scholasticinc.local>
	<CAKF-8eCDT68o0DzvF6juFzMGR7w+Eh=p57BkDmCyvCv1MOxsPA@mail.gmail.com>
	<CAD_kORfhnC3tDwCm--yWgk_w3A0UA1Lp=WGtSf-wNjtZQ1O6WQ@mail.gmail.com>
Message-ID: <20111107125920.GX13866@nerd.dk>

On Fri, Nov 04, 2011 at 12:39:58PM -0700, Lei Wang wrote:

> I had the same problem. Please help!

503 is not a particular problem, it's a blanket "something went wrong" error.
We need to see varnishlog for those requests to get any idea about what is
wrong.

-- 
Andreas



From straightflush at gmail.com  Mon Nov  7 13:31:14 2011
From: straightflush at gmail.com (AD)
Date: Mon, 7 Nov 2011 08:31:14 -0500
Subject: Intermittent 503's from Varnish
In-Reply-To: <20111107125920.GX13866@nerd.dk>
References: <8D333BF67F814C4D9803C99961BE5E7005E04092@corpex07.corp.scholasticinc.local>
	<CAKF-8eCDT68o0DzvF6juFzMGR7w+Eh=p57BkDmCyvCv1MOxsPA@mail.gmail.com>
	<CAD_kORfhnC3tDwCm--yWgk_w3A0UA1Lp=WGtSf-wNjtZQ1O6WQ@mail.gmail.com>
	<20111107125920.GX13866@nerd.dk>
Message-ID: <CANZWjw6RAeYvhGsW_pfpGWozuuYcCATML0ELEj_f4+CQx65VqA@mail.gmail.com>

Hmm Steve, not sure, here is the snippet from that wiki link.

*Saint mode enables you to discard a certain page from one backend server
and either try another server or serve stale content from cache
*
*
*
*When we set beresp.saintmode to 10 seconds Varnish will not ask that server
for URL for 10 seconds. A blacklist, more or less. Also a restart is
performed so if you have other backends capable of serving that content
Varnish will try those. When you are out of backends Varnish will serve the
content from its stale cache.*




On Mon, Nov 7, 2011 at 7:59 AM, Andreas Plesner Jacobsen <apj at mutt.dk>wrote:

> On Fri, Nov 04, 2011 at 12:39:58PM -0700, Lei Wang wrote:
>
> > I had the same problem. Please help!
>
> 503 is not a particular problem, it's a blanket "something went wrong"
> error.
> We need to see varnishlog for those requests to get any idea about what is
> wrong.
>
> --
> Andreas
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111107/c6043707/attachment.html>

From steve.webster at lovefilm.com  Mon Nov  7 14:51:31 2011
From: steve.webster at lovefilm.com (Steve Webster)
Date: Mon, 7 Nov 2011 14:51:31 +0000
Subject: Intermittent 503's from Varnish
In-Reply-To: <CANZWjw6RAeYvhGsW_pfpGWozuuYcCATML0ELEj_f4+CQx65VqA@mail.gmail.com>
References: <8D333BF67F814C4D9803C99961BE5E7005E04092@corpex07.corp.scholasticinc.local>
	<CAKF-8eCDT68o0DzvF6juFzMGR7w+Eh=p57BkDmCyvCv1MOxsPA@mail.gmail.com>
	<CAD_kORfhnC3tDwCm--yWgk_w3A0UA1Lp=WGtSf-wNjtZQ1O6WQ@mail.gmail.com>
	<20111107125920.GX13866@nerd.dk>
	<CANZWjw6RAeYvhGsW_pfpGWozuuYcCATML0ELEj_f4+CQx65VqA@mail.gmail.com>
Message-ID: <02828CD6-2EBB-421F-813F-33C557A0F438@lovefilm.com>

On 7 Nov 2011, at 13:31, AD wrote:

> Hmm Steve, not sure, here is the snippet from that wiki link.
> 
> Saint mode enables you to discard a certain page from one backend server and either try another server or serve stale content from cache
> 
> When we set beresp.saintmode to 10 seconds Varnish will not ask that server for URL for 10 seconds. A blacklist, more or less. Also a restart is performed so if you have other backends capable of serving that content Varnish will try those. When you are out of backends Varnish will serve the content from its stale cache.

If you set saintmode without also setting grace on both the cache object and the request, Varnish will simply end up returning 503 errors when you're out of working backends.

Cheers,

Steve

-- 
Steve Webster
Web Architect
LOVEFiLM

(Apologies for the overly long and pointless corporate footer that follows.)

-----------------------------------------------------------------------------------------------------------------------------------------
LOVEFiLM UK Limited is a company registered in England and Wales. 
Registered Number: 06528297. 
Registered Office: No.9, 6 Portal Way, London W3 6RU, United Kingdom.

This e-mail is confidential to the ordinary user of the e-mail address to which it was addressed. If you have received it in error, 
please delete it from your system and notify the sender immediately.

This email message has been delivered safely and archived online by Mimecast.
For more information please visit http://www.mimecast.co.uk 
-----------------------------------------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111107/db571364/attachment.html>

From straightflush at gmail.com  Mon Nov  7 15:52:14 2011
From: straightflush at gmail.com (AD)
Date: Mon, 7 Nov 2011 10:52:14 -0500
Subject: Intermittent 503's from Varnish
In-Reply-To: <02828CD6-2EBB-421F-813F-33C557A0F438@lovefilm.com>
References: <8D333BF67F814C4D9803C99961BE5E7005E04092@corpex07.corp.scholasticinc.local>
	<CAKF-8eCDT68o0DzvF6juFzMGR7w+Eh=p57BkDmCyvCv1MOxsPA@mail.gmail.com>
	<CAD_kORfhnC3tDwCm--yWgk_w3A0UA1Lp=WGtSf-wNjtZQ1O6WQ@mail.gmail.com>
	<20111107125920.GX13866@nerd.dk>
	<CANZWjw6RAeYvhGsW_pfpGWozuuYcCATML0ELEj_f4+CQx65VqA@mail.gmail.com>
	<02828CD6-2EBB-421F-813F-33C557A0F438@lovefilm.com>
Message-ID: <CANZWjw7E4YSNCsCMooKA559NZLWfNH5r2q2o4xf_T551L6KPxA@mail.gmail.com>

Hmm that seems strange.  So if you dont set grace mode, saint mode just
offers you the ability to use an alternate backend and take one out of
rotation?  Grace mode seems to be a different use case where it will serve
stale while multiple requests are requesting the same object to prevent
request queueing.

Otherwise cant you achieve the same result with a director + healthchecks?



On Mon, Nov 7, 2011 at 9:51 AM, Steve Webster <steve.webster at lovefilm.com>wrote:

> On 7 Nov 2011, at 13:31, AD wrote:
>
> > Hmm Steve, not sure, here is the snippet from that wiki link.
> >
> > Saint mode enables you to discard a certain page from one backend server
> and either try another server or serve stale content from cache
> >
> > When we set beresp.saintmode to 10 seconds Varnish will not ask that
> server for URL for 10 seconds. A blacklist, more or less. Also a restart is
> performed so if you have other backends capable of serving that content
> Varnish will try those. When you are out of backends Varnish will serve the
> content from its stale cache.
>
> If you set saintmode without also setting grace on both the cache object
> and the request, Varnish will simply end up returning 503 errors when
> you're out of working backends.
>
>
> Cheers,
>
> Steve
>
> --
> Steve Webster
> Web Architect
> LOVEFiLM
>
> (Apologies for the overly long and pointless corporate footer that
> follows.)
>
>
>  ------------------------------
>
> LOVEFiLM UK Limited is a company registered in England and Wales.
> Registered Number: 06528297.
> Registered Office: No.9, 6 Portal Way, London W3 6RU, United Kingdom.
>
> This e-mail is confidential to the ordinary user of the e-mail address to
> which it was
> addressed. If you have received it in error, please delete it from your
> system and notify
> the sender immediately.
>
> This email message has been delivered safely and archived online by
> Mimecast.
> For more information please visit http://www.mimecast.co.uk
> ------------------------------
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111107/9895a416/attachment.html>

From steve.webster at lovefilm.com  Mon Nov  7 16:17:59 2011
From: steve.webster at lovefilm.com (Steve Webster)
Date: Mon, 7 Nov 2011 16:17:59 +0000
Subject: Intermittent 503's from Varnish
In-Reply-To: <CANZWjw7E4YSNCsCMooKA559NZLWfNH5r2q2o4xf_T551L6KPxA@mail.gmail.com>
References: <8D333BF67F814C4D9803C99961BE5E7005E04092@corpex07.corp.scholasticinc.local>
	<CAKF-8eCDT68o0DzvF6juFzMGR7w+Eh=p57BkDmCyvCv1MOxsPA@mail.gmail.com>
	<CAD_kORfhnC3tDwCm--yWgk_w3A0UA1Lp=WGtSf-wNjtZQ1O6WQ@mail.gmail.com>
	<20111107125920.GX13866@nerd.dk>
	<CANZWjw6RAeYvhGsW_pfpGWozuuYcCATML0ELEj_f4+CQx65VqA@mail.gmail.com>
	<02828CD6-2EBB-421F-813F-33C557A0F438@lovefilm.com>
	<CANZWjw7E4YSNCsCMooKA559NZLWfNH5r2q2o4xf_T551L6KPxA@mail.gmail.com>
Message-ID: <0931B377-FCA5-4596-A64F-83A9FA51463F@lovefilm.com>

On 7 Nov 2011, at 15:52, AD wrote:

> Hmm that seems strange.  So if you dont set grace mode, saint mode just offers you the ability to use an alternate backend and take one out of rotation?  Grace mode seems to be a different use case where it will serve stale while multiple requests are requesting the same object to prevent request queueing.
> 
> Otherwise cant you achieve the same result with a director + healthchecks?

As I understand it, Saint mode is more granular than the health check mechanism in that it blacklists a given URL on a given backend for a period of time that you specify. Health checks affect the backend as a whole.

Cheers,

Steve

-- 
Steve Webster
Web Architect
LOVEFiLM

(Apologies for the overly long and pointless corporate footer that follows.)

-----------------------------------------------------------------------------------------------------------------------------------------
LOVEFiLM UK Limited is a company registered in England and Wales. 
Registered Number: 06528297. 
Registered Office: No.9, 6 Portal Way, London W3 6RU, United Kingdom.

This e-mail is confidential to the ordinary user of the e-mail address to which it was addressed. If you have received it in error, 
please delete it from your system and notify the sender immediately.

This email message has been delivered safely and archived online by Mimecast.
For more information please visit http://www.mimecast.co.uk 
-----------------------------------------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111107/a7a259d0/attachment.html>

From sahmed1020 at gmail.com  Tue Nov  8 20:22:39 2011
From: sahmed1020 at gmail.com (S Ahmed)
Date: Tue, 8 Nov 2011 15:22:39 -0500
Subject: things to consider with caching an ecommerce website
Message-ID: <CAG2rwuNvSLD3n=RzEFyPf+DY0R-GtqhVyhGLigqhtQsRHmjN7g@mail.gmail.com>

What are the things you have to consider with using varnish on say an
ecommerce website?

Pages that display the product, assuming no customization for logged in
users, means I can serve cached pages to both logged in and anonymous users.

But if I want to display the logged in users name on the page, that means
the page cannot be cached correct?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111108/f0c3c6f8/attachment.html>

From l at lrowe.co.uk  Tue Nov  8 20:42:14 2011
From: l at lrowe.co.uk (Laurence Rowe)
Date: Tue, 8 Nov 2011 12:42:14 -0800
Subject: things to consider with caching an ecommerce website
In-Reply-To: <CAG2rwuNvSLD3n=RzEFyPf+DY0R-GtqhVyhGLigqhtQsRHmjN7g@mail.gmail.com>
References: <CAG2rwuNvSLD3n=RzEFyPf+DY0R-GtqhVyhGLigqhtQsRHmjN7g@mail.gmail.com>
Message-ID: <CAOycyLTFc8qLqFNQoHLdWyKsEQyTOnNg6Yt5aixx4OOdwrADyQ@mail.gmail.com>

On 8 November 2011 12:22, S Ahmed <sahmed1020 at gmail.com> wrote:
> What are the things you have to consider with using varnish on say an
> ecommerce website?
> Pages that display the product, assuming no customization for logged in
> users, means I can serve cached pages to both logged in and anonymous users.
> But if I want to display the logged in users name on the page, that means
> the page cannot be cached correct?

You can show the logged in user's name on a cached page using either
ESI or some simple javascript (set some user data on a cookie on
login, display this in appropriate place with javascript.)

Laurence



From ask at develooper.com  Wed Nov  9 06:42:30 2011
From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Tue, 8 Nov 2011 22:42:30 -0800
Subject: How do I filter with varnishlog?
Message-ID: <6B425904-16C8-4AD7-974C-FD4E198EE05D@develooper.com>

I expected

	varnishlog -m ReqStart:76.172.28.40

to only log requests from 76.172.28.40; but that doesn't seem to be the case.  It's logging less than without the -m parameter (mostly TxHeader and RxHeader lines), but not in a way that makes sense.

(Using 3.0.2).



 - ask
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4798 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111108/5636f002/attachment.bin>

From tfheen at varnish-software.com  Wed Nov  9 07:49:47 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Wed, 09 Nov 2011 08:49:47 +0100
Subject: How do I filter with varnishlog?
In-Reply-To: <6B425904-16C8-4AD7-974C-FD4E198EE05D@develooper.com> ("Ask
	=?utf-8?Q?Bj=C3=B8rn?=
	Hansen"'s message of "Tue, 8 Nov 2011 22:42:30 -0800")
References: <6B425904-16C8-4AD7-974C-FD4E198EE05D@develooper.com>
Message-ID: <878vnpvip0.fsf@qurzaw.varnish-software.com>

]] Ask Bj?rn Hansen 

| I expected
| 
| 	varnishlog -m ReqStart:76.172.28.40
| 
| to only log requests from 76.172.28.40; but that doesn't seem to be the case.  It's logging less than without the -m parameter (mostly TxHeader and RxHeader lines), but not in a way that makes sense.

You might want to add -c in there too, to filter out a bit of noise.

-- 
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64



From ask at perl.org  Wed Nov  9 06:39:46 2011
From: ask at perl.org (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Tue, 8 Nov 2011 22:39:46 -0800
Subject: docs: -b and -c flag in varnishlog
Message-ID: <96E2D7F2-9E5A-4BA5-9425-9B31AE63A910@perl.org>

-b and -c are still documented in varnishlog and on https://www.varnish-cache.org/docs/trunk/tutorial/logging.html

(I thought those flags were gone permanently)


 - ask


From sam at musicmatter.co.uk  Thu Nov 10 12:26:20 2011
From: sam at musicmatter.co.uk (Sam Davey)
Date: Thu, 10 Nov 2011 12:26:20 +0000
Subject: Varnish - Apache - Port 81 Trailing Slash Rewrite
Message-ID: <CANTi4ATuFE67yZpc+7e8mtpkzAKDHT3FhBY4anEJ2qpvc+_wwA@mail.gmail.com>

Hi,

I'm running varnish-2.1.5 on port 80 with an apache backend on port 81.

I'm having a strange problem with my configuration.

Basically I have added a new folder called 'test' in my webroot so I
can access via www.exampledomain.com/test/.  This directory only
contains a simple html file.

If I access the url with the trailing slash everything works fine.

However if I access the url without the trailing slash my page hangs
and eventually the address bar displays
http://www.exampledomain.com:81/test/

It is hanging because I don't let port 81 through my firewall.

I do have some rewrite rules in apache but they pass through
RewriteCond %{REQUEST_FILENAME} !-d so they should not run.

As such I am confused.  What is causing it to try and rewrite or
redirect to http://www.exampledomain.com:81/test/?

How can I configure it correctly? (BTW I am using Plesk 9.5.2 if that
makes a difference)


Obviously I could add the following to my vcl_recv

if (req.http.host ~ "^(www\.)?exampledomain\.co.uk" && req.url~ "^/test") {
set req.url = req.url "/";
}

But that makes the content look like its coming from /test instead of
/test/ and as such in certain circumstances I could get a duplicate
content penalty.  Also I must be correct in thinking that this sort of
handling should take place in the backend, not in varnish.

Kind regards,

Sam
Music Matter is a trading style of Matter Retail Ltd, Unit A116 Tustin Way, Longridge Road, Preston, PR2 5LX, United Kingdom.

Registered in England & Wales, Company number 06920634.

This email and any files transmitted with it are confidential. If you have received this email in error please notify the sender and then delete it immediately. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Matter Retail Ltd. The recipient should check this email and any attachments for the presence of viruses. Matter Retail Ltd. accepts no liability for any damage caused by any virus transmitted by this email.

Save resources: Think before you print.




From varnish at mm.quex.org  Thu Nov 10 13:09:20 2011
From: varnish at mm.quex.org (Michael Alger)
Date: Thu, 10 Nov 2011 21:09:20 +0800
Subject: Varnish - Apache - Port 81 Trailing Slash Rewrite
In-Reply-To: <CANTi4ATuFE67yZpc+7e8mtpkzAKDHT3FhBY4anEJ2qpvc+_wwA@mail.gmail.com>
References: <CANTi4ATuFE67yZpc+7e8mtpkzAKDHT3FhBY4anEJ2qpvc+_wwA@mail.gmail.com>
Message-ID: <20111110130920.GA23182@grum.quex.org>

On Thu, Nov 10, 2011 at 12:26:20PM +0000, Sam Davey wrote:
> I'm running varnish-2.1.5 on port 80 with an apache backend on port 81.
> 
> I'm having a strange problem with my configuration.
> 
> Basically I have added a new folder called 'test' in my webroot so
> I can access via www.exampledomain.com/test/.  This directory only
> contains a simple html file.
> 
> If I access the url with the trailing slash everything works fine.
> 
> However if I access the url without the trailing slash my page hangs
> and eventually the address bar displays
> http://www.exampledomain.com:81/test/
> 
> It is hanging because I don't let port 81 through my firewall.
> 
> I do have some rewrite rules in apache but they pass through
> RewriteCond %{REQUEST_FILENAME} !-d so they should not run.
> 
> As such I am confused.  What is causing it to try and rewrite or
> redirect to http://www.exampledomain.com:81/test/?

It's standard behaviour of Apache and most web servers. The reason
is partly what you've identified -- /test and /test/ are different
URLs but end up being the same thing, so it redirects to the "one
true location" of the resource. It's also so that relative links
within that location still work. Imagine your test document embeds
an image using <img src="image.png">. If the browser thinks it's
accessing /test/ then it will fetch /test/image.png. If it thinks
it's accessing /test then it'll fetch /image.png.

> How can I configure it correctly? (BTW I am using Plesk 9.5.2 if
> that makes a difference)

Generally it's best to make your web server listen & respond to the
same host/port that clients are accessing it. So, configure it as if
Varnish wasn't there, and then slot Varnish in front of it somehow.

One way to do this is to have Apache bind to a loopback address on
port 80 (e.g. 127.0.0.1, but you can use 127.anything). That way
Apache will still feel like it's living on port 80 and when it does
its stuff there's no issue.

You might be able to override Apache's notion of the port somehow;
my backends are on different servers to the proxies though so I've
never needed to deal with this.

> But that makes the content look like its coming from /test instead
> of /test/ and as such in certain circumstances I could get a
> duplicate content penalty.  Also I must be correct in thinking
> that this sort of handling should take place in the backend, not
> in varnish.

Yes, I'd definitely sort it out in the backend.



From kpettijohn at tarot.com  Thu Nov 10 17:16:17 2011
From: kpettijohn at tarot.com (Kevin Pettijohn)
Date: Thu, 10 Nov 2011 17:16:17 +0000
Subject: PUT requests with Varnish and Chrome
Message-ID: <CAE146E1.808A%kpettijohn@tarot.com>

Hello all,

 
I have been trying to wrap my head around a Varnish/Chrome issue regarding
http PUT requests and would appreciate any ideas/insight.  The problem is
when we use a PUT request through Chrome (15.0.874.106) the request
reaches Varnish (2.1.5) from the client as a PUT but when varnish sends
the request to the backend it has turned into a GET request.  We have only
been able to recreate this issue in Chrome. In order to get around this
issue for Chrome we are using ?pipe? which as expected uses PUT between
both the client and the server.

Varnishlog:
  
16 SessionOpen  c 10.3.10.124 60714 :80
   16 ReqStart     c 10.3.10.124 60714 1424074046
   16 RxRequest    c PUT
   16 RxURL        c /fun-and-games/cosmic-profile/get_cosmic_profile
   16 RxProtocol   c HTTP/1.1
   16 RxHeader     c Host: stage.blah.com
   16 RxHeader     c Connection: keep-alive
   16 RxHeader     c Content-Length: 257
   16 RxHeader     c Origin: http://stage.blah.com
   16 RxHeader     c X-Requested-With: XMLHttpRequest
   16 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X
10_6_8) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106
Safari/535.2
   16 RxHeader     c Content-Type: application/x-www-form-urlencoded
   16 RxHeader     c Accept: */*
   16 RxHeader     c Referer:
http://stage.blah.com/free-online-games/cosmic-profile/
   16 RxHeader     c Accept-Encoding: gzip,deflate,sdch
   16 RxHeader     c Accept-Language: en-US,en;q=0.8
   16 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
   16 RxHeader     c Cookie: csession=fbph0fapbes3rin0kh7iphhj76;
__utma=70084306.1318547882.1320866763.1320878236.1320885100.4;
__utmc=70084306; 
__utmz=70084306.1320866763.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none
)
   16 VCL_call     c recv
   16 VCL_return   c lookup
   16 VCL_call     c hash
   16 VCL_return   c hash
   16 VCL_call     c miss
   16 VCL_return   c fetch
   16 Backend      c 17 t4dev t4dev
   17 TxRequest    - GET
   17 TxURL        - /fun-and-games/cosmic-profile/get_cosmic_profile
   17 TxProtocol   - HTTP/1.1
   17 TxHeader     - Host: stage.blah.com
   17 TxHeader     - Origin: http://stage.blah.com
   17 TxHeader     - X-Requested-With: XMLHttpRequest
   17 TxHeader     - User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X
10_6_8) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106
Safari/535.2
   17 TxHeader     - Content-Type: application/x-www-form-urlencoded
   17 TxHeader     - Accept: */*
   17 TxHeader     - Referer:
http://stage.blah.com/free-online-games/cosmic-profile/
   17 TxHeader     - Accept-Encoding: gzip,deflate,sdch
   17 TxHeader     - Accept-Language: en-US,en;q=0.8
   17 TxHeader     - Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
   17 TxHeader     - Cookie: csession=fbph0fapbes3rin0kh7iphhj76;
__utma=70084306.1318547882.1320866763.1320878236.1320885100.4;
__utmc=70084306; 
__utmz=70084306.1320866763.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none
)
   17 TxHeader     - X-Varnish: 1424074046
   17 RxProtocol   - HTTP/1.1
   17 RxStatus     - 200
   17 RxResponse   - OK
   17 RxHeader     - Content-Type: application/json
   17 RxHeader     - Connection: keep-alive
   17 RxHeader     - Transfer-Encoding: chunked
   16 TTL          c 1424074046 RFC 120 1320888101 0 0 0 0
   16 VCL_call     c fetch
   16 VCL_return   c deliver
   16 ObjProtocol  c HTTP/1.1
   16 ObjStatus    c 200
   16 ObjResponse  c OK
   16 ObjHeader    c Content-Type: application/json
   17 Fetch_Body   - 3 0 1
   17 Length       - 2
   17 BackendReuse - t4dev
   16 VCL_call     c deliver


The PUT request:

 
 $.ajax('/fun-and-games/cosmic-profile/get_cosmic_profile', {
  data: {
    name: 'Kevin',
    birth_month: 'October',
    birth_day: '05',
    birth_year: '1970',
    birth_hour: '07',
    birth_minute: '12',
    birth_ampm: 'PM',
    gender: 'male',
    birth_cityID: '1391126',
    picture_url: 
'http://avatars.blah.com/profile_d4cca4e1-91d9-4f5e-9699-3519cb15ce3f.jpg',
    dontknow: false,
    persist: true
  }, type: "PUT",
  success: function(data) {
    console.log('SUCCESS', data);
  },
  error: function(data) {
    console.log('ERROR', data);
  }
});

Any ideas would be much appreciated!

Thanks,
Kevin




From csdiedrich at gmail.com  Thu Nov 10 17:25:11 2011
From: csdiedrich at gmail.com (Cristiano Diedrich)
Date: Thu, 10 Nov 2011 14:25:11 -0300
Subject: Doubt with configuration
Message-ID: <CAHoPciH==FTOGHRT55Ey5ag93tapYfeiHJhU2vn+3Gxu_boCKA@mail.gmail.com>

Hi,

Sorry for my english ...

My question is:
I have several backend web server (apache) and vanish in front of them to
carry out the contention of cache pages.Like all sites link to the server
vanish when there is a fault with him all the sites were down, right?
How can this issue be addressed? using the CARP with replication through
drbd?
How can I work in an environment like this?

This environment is not in production, I'm just planning how I can improve
my web server environment.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111110/c2f59702/attachment.html>

From insyte at gmail.com  Thu Nov 10 20:00:22 2011
From: insyte at gmail.com (Ben Beuchler)
Date: Thu, 10 Nov 2011 14:00:22 -0600
Subject: Doubt with configuration
In-Reply-To: <CAHoPciH==FTOGHRT55Ey5ag93tapYfeiHJhU2vn+3Gxu_boCKA@mail.gmail.com>
References: <CAHoPciH==FTOGHRT55Ey5ag93tapYfeiHJhU2vn+3Gxu_boCKA@mail.gmail.com>
Message-ID: <CAC29JhXZUryQG3=xsL03gQMcdoZ+FUwDDOY26dPga9=33sN8xA@mail.gmail.com>

> I have several?backend?web server?(apache)?and?vanish?in front of them?to
> carry out the?contention?of?cache?pages.Like all?sites link
> to?the?server?vanish?when?there is a fault?with him?all the
> sites?were?down,?right?

If you're using a single varnish server, then when it goes down your
sites are down.

> How can?this issue?be addressed??using the?CARP?with
> replication?through?drbd?

There are a number of options.  CARP would work, but when you fail
over to the second varnish server it will have an empty cache.

We use a load balancer in front of a pair of varnish servers, so they
are both in service with warm caches at all times.  The load balancer
handles failover.  This is a pretty common configuration.

-Ben



From guyb at spc.int  Thu Nov 10 21:33:07 2011
From: guyb at spc.int (Guy Brodie)
Date: Thu, 10 Nov 2011 21:33:07 +0000
Subject: No subject
Message-ID: <17DEC0FA12D68446AE105B6A3C7A2D7703DC6D@NOUEXCH01.corp.spc.int>

Hi Everyone!


I have a curious case of unusual backend selection.


To summarise,  I have two backend servers servicing content for two 'subsites',  eg:


www.company.com/subsite1 ->  serviced by backend 'server1'

www.company.com/subsite2 ->  serviced by backend 'server2'


My problem is with subsite 2.


It works:   when in my browser i directly enter the url "www.company.com/subsite2".

It doesn't work:   when i restart the browser and browse to www.company.com/subsite1 (handled by server1),  *then* click a link to subsite2 .   At this point,  packet captures show the request still going to server1, despite the vcl clearly defining subsite2 should be handled by server2 (and as confirmed by subsite2 working when no previous browsing history has occurred on the company site).


Furthermore,  when the request for subsite2 is incorrectly sent to server1,  the website generates a redirect. This redirect is to the correct URL that should go to server2,  but is again sent to server1, and a redirect loop between the browser and server1 occurs because varnish insists on sending the request to server1 instead of server2.


Curiously,  the varnishlog has no record of this looping behaviour at all!    I  vaguely remember reading that a malformed request does not make it into the varnishlog output?


So I'm left wondering about what is the difference between hitting the subsite2 URL directly versus navigating to it via subsite1.   Is this behaviour tied to:

* cookie based CMS being used on subsite1 - does varnish use cookies to determine back-end re-use or something?  I didn't think so...

* http tcp session re-use - the browser is re-using a single TCP session when this happens,  but I thought varnish inspects every request regardless - i.e. it doesn't blindly pass an existing TCP session from the browser through to the backend server it had previously selected?

* perhaps a malformed request or something?



Our environment is a reasonably simple one I think.  We basically use regex on URL and host to determine which backend to use. We use minimal caching functions - just for static objects , and even that is a token effort.  Varnish serves primarily as a 'URL router' for us.      We are running varnishd (varnish-3.0.0 revision 3bd5997)


So I'm kind of stuck!     I realise I haven't posted any config.  At this stage,  I'm hoping someone can help with confirming my understanding of varnish behaviour regards cookies, session re-use and malformed request handling as above.  But any ideas on how to start debugging this one will be greatly appreciated!


Thankyou!


Guy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111110/867f6678/attachment.html>

From guyb at spc.int  Thu Nov 10 21:35:05 2011
From: guyb at spc.int (Guy Brodie)
Date: Thu, 10 Nov 2011 21:35:05 +0000
Subject: varnish incorrect back-end selection problem
In-Reply-To: <17DEC0FA12D68446AE105B6A3C7A2D7703DC6D@NOUEXCH01.corp.spc.int>
References: <17DEC0FA12D68446AE105B6A3C7A2D7703DC6D@NOUEXCH01.corp.spc.int>
Message-ID: <17DEC0FA12D68446AE105B6A3C7A2D7703DC81@NOUEXCH01.corp.spc.int>

Hi Everyone!

(sorry - re-post with subject line this time :-) )


I have a curious case of unusual backend selection.


To summarise,  I have two backend servers servicing content for two 'subsites',  eg:


www.company.com/subsite1 ->  serviced by backend 'server1'

www.company.com/subsite2 ->  serviced by backend 'server2'


My problem is with subsite 2.


It works:   when in my browser i directly enter the url "www.company.com/subsite2".

It doesn't work:   when i restart the browser and browse to www.company.com/subsite1 (handled by server1),  *then* click a link to subsite2 .   At this point,  packet captures show the request still going to server1, despite the vcl clearly defining subsite2 should be handled by server2 (and as confirmed by subsite2 working when no previous browsing history has occurred on the company site).


Furthermore,  when the request for subsite2 is incorrectly sent to server1,  the website generates a redirect. This redirect is to the correct URL that should go to server2,  but is again sent to server1, and a redirect loop between the browser and server1 occurs because varnish insists on sending the request to server1 instead of server2.


Curiously,  the varnishlog has no record of this looping behaviour at all!    I  vaguely remember reading that a malformed request does not make it into the varnishlog output?


So I'm left wondering about what is the difference between hitting the subsite2 URL directly versus navigating to it via subsite1.   Is this behaviour tied to:

* cookie based CMS being used on subsite1 - does varnish use cookies to determine back-end re-use or something?  I didn't think so...

* http tcp session re-use - the browser is re-using a single TCP session when this happens,  but I thought varnish inspects every request regardless - i.e. it doesn't blindly pass an existing TCP session from the browser through to the backend server it had previously selected?

* perhaps a malformed request or something?



Our environment is a reasonably simple one I think.  We basically use regex on URL and host to determine which backend to use. We use minimal caching functions - just for static objects , and even that is a token effort.  Varnish serves primarily as a 'URL router' for us.      We are running varnishd (varnish-3.0.0 revision 3bd5997)


So I'm kind of stuck!     I realise I haven't posted any config.  At this stage,  I'm hoping someone can help with confirming my understanding of varnish behaviour regards cookies, session re-use and malformed request handling as above.  But any ideas on how to start debugging this one will be greatly appreciated!


Thankyou!


Guy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111110/83c75779/attachment.html>

From apj at mutt.dk  Thu Nov 10 21:40:51 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Thu, 10 Nov 2011 22:40:51 +0100
Subject: varnish incorrect back-end selection problem
In-Reply-To: <17DEC0FA12D68446AE105B6A3C7A2D7703DC81@NOUEXCH01.corp.spc.int>
References: <17DEC0FA12D68446AE105B6A3C7A2D7703DC6D@NOUEXCH01.corp.spc.int>
	<17DEC0FA12D68446AE105B6A3C7A2D7703DC81@NOUEXCH01.corp.spc.int>
Message-ID: <20111110214051.GZ13866@nerd.dk>

On Thu, Nov 10, 2011 at 09:35:05PM +0000, Guy Brodie wrote:
> 
> * http tcp session re-use - the browser is re-using a single TCP session when
> this happens,  but I thought varnish inspects every request regardless - i.e.
> it doesn't blindly pass an existing TCP session from the browser through to
> the backend server it had previously selected?

This can happen if you pipe.

https://www.varnish-cache.org/trac/wiki/VCLExamplePipe

-- 
Andreas



From guyb at spc.int  Thu Nov 10 22:23:51 2011
From: guyb at spc.int (Guy Brodie)
Date: Thu, 10 Nov 2011 22:23:51 +0000
Subject: varnish incorrect back-end selection problem
In-Reply-To: <20111110214051.GZ13866@nerd.dk>
References: <17DEC0FA12D68446AE105B6A3C7A2D7703DC6D@NOUEXCH01.corp.spc.int><17DEC0FA12D68446AE105B6A3C7A2D7703DC81@NOUEXCH01.corp.spc.int>
	<20111110214051.GZ13866@nerd.dk>
Message-ID: <17DEC0FA12D68446AE105B6A3C7A2D7703DD10@NOUEXCH01.corp.spc.int>

Aha... thats it!  - Thankyou Andreas!

Yes we have some pipe employed as work arounds for issues with range requests, If-Modified-Since / e-tags, and some large download/streaming requests.

I put the suggested change in, and it has worked for the back-end selection problem just fine.  I'm off to regression test the other reasons we had the pipe command in there, although it looks like a pretty straight-forward fix.

Thanks again,

Guy. 

-----Original Message-----
From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Andreas Plesner Jacobsen
Sent: Friday, 11 November 2011 8:41 AM
To: varnish-misc at varnish-cache.org
Subject: Re: varnish incorrect back-end selection problem

On Thu, Nov 10, 2011 at 09:35:05PM +0000, Guy Brodie wrote:
> 
> * http tcp session re-use - the browser is re-using a single TCP session when
> this happens,  but I thought varnish inspects every request regardless - i.e.
> it doesn't blindly pass an existing TCP session from the browser through to
> the backend server it had previously selected?

This can happen if you pipe.

https://www.varnish-cache.org/trac/wiki/VCLExamplePipe

-- 
Andreas

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

From kennyds at ac-systems.com  Thu Nov 10 09:33:44 2011
From: kennyds at ac-systems.com (Kenny Deckers)
Date: Thu, 10 Nov 2011 10:33:44 +0100
Subject: Error when compiling varnish  "You need rst2man installed to make
	dist"
Message-ID: <4EBB9A78.30209@ac-systems.com>

Hey

I just cloned the latest varnish git-repository to compile from source 
because I want to "compile Varnish from source for other reasons" as 
mentioned on the documentation site.

All the dependencies listed on the website have been installed:

  * autotools-dev
  * automake1.9
  * libtool
  * autoconf
  * libncurses-dev
  * groff-base
  * libpcre3-dev
  * pkg-config
  * Python Docutils
  * Sphinx

I ran the following commands: |
|

|./autogen.sh| -> works fine

|./configure| -> works fine too

|make| -> fails with the following error:

|creating libvmod_std.la
(cd .libs&&  rm -f libvmod_std.la&&  ln -s ../libvmod_std.la libvmod_std.la)
========================================
*You need rst2man installed to make dist*
========================================
make[3]: *** [vmod_std.3] Error 1
make[3]: Leaving directory `/tmp/varnish-cache/lib/libvmod_std'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/tmp/varnish-cache/lib'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/varnish-cache'
make: *** [all] Error 2
|

I have tried installing rst2man but it cannot be found since it's a 
virtual package?:

|apt-get install rst2man
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package rst2man
|

Any help would be appreciated!

I'm running Debian 6.

Kind regards,
Kenny Deckers

-- 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111110/e7d892da/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: business_card.jpg
Type: image/jpeg
Size: 25873 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111110/e7d892da/attachment.jpg>

From tfheen at varnish-software.com  Fri Nov 11 08:38:54 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Fri, 11 Nov 2011 09:38:54 +0100
Subject: Error when compiling varnish "You need rst2man installed to make
	dist"
In-Reply-To: <4EBB9A78.30209@ac-systems.com> (Kenny Deckers's message of "Thu, 
	10 Nov 2011 10:33:44 +0100")
References: <4EBB9A78.30209@ac-systems.com>
Message-ID: <87mxc3yrxd.fsf@qurzaw.varnish-software.com>

]] Kenny Deckers 

| I have tried installing rst2man but it cannot be found since it's a virtual package?:

It comes from python-docutils.

-- 
Tollef Fog Heen
Varnish Software
t: +47 21 98 92 64



From jeroen.ooms at stat.ucla.edu  Sat Nov 12 05:55:57 2011
From: jeroen.ooms at stat.ucla.edu (Jeroen Ooms)
Date: Fri, 11 Nov 2011 21:55:57 -0800
Subject: ubuntu package help
Message-ID: <CABFfbXux6fXh9BEutVqc_+oj2ie6ezQVNHoioew74NLu=K0dNQ@mail.gmail.com>

I installed varnish on Ubuntu 11.10 Oneiric though the default repositories
(apt-get install varnish). The Ubuntu package does not just supply
/usr/sbin/varnishd and /etc/varnish/default.vcl; it also creates some
startup entries (/etc/init.d/varnish, varnishlog, varnishncsa) by default.
I would like to use this package as it was meant, but it is not clear to me
how I am supposed to use it and I don't think there is any documentation.

Some questions:

- Does the package assume port 80 is available (hence does it conflict with
Apache2) ? I installed both but port 80 seems to be solely used for Apache.
- How am I supposed to include my own VCL script? Should I just overwrite
/etc/varnish/default.vcl ?
- Is there any place to 'configure' the varnishd startup options, or should
I hack those into the startup scripts ?

Thanks,

Jeroen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111111/f3bb3a5f/attachment.html>

From varnish at mm.quex.org  Sat Nov 12 06:26:57 2011
From: varnish at mm.quex.org (Michael Alger)
Date: Sat, 12 Nov 2011 14:26:57 +0800
Subject: ubuntu package help
In-Reply-To: <CABFfbXux6fXh9BEutVqc_+oj2ie6ezQVNHoioew74NLu=K0dNQ@mail.gmail.com>
References: <CABFfbXux6fXh9BEutVqc_+oj2ie6ezQVNHoioew74NLu=K0dNQ@mail.gmail.com>
Message-ID: <20111112062657.GA31858@grum.quex.org>

On Fri, Nov 11, 2011 at 09:55:57PM -0800, Jeroen Ooms wrote:
> I installed varnish on Ubuntu 11.10 Oneiric though the default
> repositories (apt-get install varnish). The Ubuntu package does
> not just supply /usr/sbin/varnishd and /etc/varnish/default.vcl;
> it also creates some startup entries (/etc/init.d/varnish,
> varnishlog, varnishncsa) by default.  I would like to use this
> package as it was meant, but it is not clear to me how I am
> supposed to use it and I don't think there is any documentation.

I use Debian myself, but Ubuntu is a Debian derivative so probably
follows mostly the same conventions. In that case, you can probably
find documentation in /usr/share/doc/varnish.

> - Does the package assume port 80 is available (hence does it
> conflict with Apache2) ? I installed both but port 80 seems to be
> solely used for Apache.
> - How am I supposed to include my own VCL script? Should I just
> overwrite /etc/varnish/default.vcl ?
> - Is there any place to 'configure' the varnishd startup options,
> or should I hack those into the startup scripts ?

See if there's an /etc/default/varnish file. If it is, that'd
probably be where you specify the startup options, which would
answer all of the above questions.

If not, have a look at what the init.d script does. I'm pretty sure
it's going to be sourcing most of its configuration from some other
location.



From cosimo at streppone.it  Sat Nov 12 07:47:38 2011
From: cosimo at streppone.it (Cosimo Streppone)
Date: Sat, 12 Nov 2011 08:47:38 +0100
Subject: ubuntu package help
In-Reply-To: <CABFfbXux6fXh9BEutVqc_+oj2ie6ezQVNHoioew74NLu=K0dNQ@mail.gmail.com>
References: <CABFfbXux6fXh9BEutVqc_+oj2ie6ezQVNHoioew74NLu=K0dNQ@mail.gmail.com>
Message-ID: <op.v4to9o0cs5ttvb@cn01>

On Sat, 12 Nov 2011 06:55:57 +0100, Jeroen Ooms  
<jeroen.ooms at stat.ucla.edu> wrote:

> I installed varnish on Ubuntu 11.10 Oneiric though the default  
> repositories (apt-get install varnish).
> [...]
> I would like to use this package as it was meant, but it is not clear to  
> me how I am supposed to use it and I don't think there is any  
> documentation.

/etc/default/varnish binds everything together.

If you read puppet stuff, this should give you
some useful info:

https://github.com/cosimo/puppet-modules/blob/master/varnish/templates/debian-defaults.erb

-- 
Cosimo



From kennyds at ac-systems.com  Mon Nov 14 08:42:59 2011
From: kennyds at ac-systems.com (Kenny Deckers)
Date: Mon, 14 Nov 2011 09:42:59 +0100
Subject: Multiple Set-Cookie Headers
Message-ID: <4EC0D493.80709@ac-systems.com>

Hi,

I'm having the following issue with "multiple set-cookie headers":

Below you can see a curl request to a website that is running varnish:

root at magento-development:/# curl -I http://XXXXXXXXX

[1] 5332
root at magento-development:/# HTTP/1.1 200 OK
Date: Thu, 10 Nov 2011 14:49:56 GMT
Server: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny13 with Suhosin-Patch
X-Powered-By: PHP/5.2.6-1+lenny13
*Set-Cookie: store=scfr; expires=Fri, 09-Nov-2012 14:49:56 GMT; path=/; domain=XXXXXXXXX; httponly
Set-Cookie: frontend=9ac04aa3912eb78eb79f98dd531f7ba6; expires=Thu, 10 Nov 2011 15:49:57 GMT; path=/;*domain=XXXXXXXXX; HttpOnly
Expires: Thu, 10 Nov 2011 16:49:57 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
*Set-Cookie: EXTERNAL_NO_CACHE=1; expires=Thu, 10-Nov-2011 15:49:57 GMT; path=/; domain=XXXXXXXXX; httponly*
X-Cache-Debug: 1
Vary: Accept-Encoding,User-Agent
Content-Type: text/html; charset=UTF-8

As you can see there are multiple Set-Cookie headers can't be used since a limitation of Varnish is that it will only be able to "analyze" the first line. I tried this header-modification vmod:

https://github.com/KristianLyng/libvmod-header

which did not solve the issue.

Any help here?

Kind regards,
Kenny

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111114/d2d148c4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: business_card.jpg
Type: image/jpeg
Size: 25873 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111114/d2d148c4/attachment.jpg>

From kristian at varnish-software.com  Mon Nov 14 10:58:09 2011
From: kristian at varnish-software.com (Kristian Lyngstol)
Date: Mon, 14 Nov 2011 11:58:09 +0100
Subject: Multiple Set-Cookie Headers
In-Reply-To: <4EC0D493.80709@ac-systems.com>
References: <4EC0D493.80709@ac-systems.com>
Message-ID: <20111114105809.GB5602@freud.kly.no>

Hi,

On Mon, Nov 14, 2011 at 09:42:59AM +0100, Kenny Deckers wrote:
> I'm having the following issue with "multiple set-cookie headers":

(...)

What issue? You are not saying anything about what you wan to
accomplish.

Your web server sends multiple set-cookie headers. I assume you entered
the stars yourself.

What are you trying to accomplish, and what have you actually tried with
the header vmod?

- Kristian



From roberto.fernandezcrisial at gmail.com  Mon Nov 14 12:51:31 2011
From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=)
Date: Mon, 14 Nov 2011 09:51:31 -0300
Subject: Multiple Set-Cookie Headers
In-Reply-To: <20111114105809.GB5602@freud.kly.no>
References: <4EC0D493.80709@ac-systems.com>
	<20111114105809.GB5602@freud.kly.no>
Message-ID: <CAAyZgShq79yZGXfhXtPxmXWGd3RuzEbyOgJf_Y-FJyo8j08N+w@mail.gmail.com>

Hi,

I think Kenny's issue is that Varnish could be caching Set-Cookie response
and send cached object to all users with those Set-Cookie headers (like any
session).

I think Kenny should pass any "Set-Cookie: EXTERNAL_NO_CACHE=1" response.

Roberto O. Fern?ndez Crisial
@rofc*
*
On Mon, Nov 14, 2011 at 7:58 AM, Kristian Lyngstol <
kristian at varnish-software.com> wrote:

> Hi,
>
> On Mon, Nov 14, 2011 at 09:42:59AM +0100, Kenny Deckers wrote:
> > I'm having the following issue with "multiple set-cookie headers":
>
> (...)
>
> What issue? You are not saying anything about what you wan to
> accomplish.
>
> Your web server sends multiple set-cookie headers. I assume you entered
> the stars yourself.
>
> What are you trying to accomplish, and what have you actually tried with
> the header vmod?
>
> - Kristian
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111114/d5be266c/attachment.html>

From arjen at thenextweb.com  Mon Nov 14 17:51:21 2011
From: arjen at thenextweb.com (Arjen Schat)
Date: Mon, 14 Nov 2011 18:51:21 +0100
Subject: Stuck on max 1024 threads
Message-ID: <D6A66FFE-4E34-4EA4-B3A4-96D17DD5A562@thenextweb.com>

Hi,

Varnish can't get over a 1024 threads.
thread_pool_max = 1500

All kernel settings are above 1024
Ulimit is above 1024

Do you have to restart the whole machine to make certain settings take effect?

I've simulated high traffic by settings thread_pool_min above 1024, at that moment the varnish won't handle any request.

Thank you,
Arjen


From cosimo at streppone.it  Mon Nov 14 18:03:37 2011
From: cosimo at streppone.it (Cosimo Streppone)
Date: Mon, 14 Nov 2011 19:03:37 +0100
Subject: Stuck on max 1024 threads
In-Reply-To: <D6A66FFE-4E34-4EA4-B3A4-96D17DD5A562@thenextweb.com>
References: <D6A66FFE-4E34-4EA4-B3A4-96D17DD5A562@thenextweb.com>
Message-ID: <op.v4x64btns5ttvb@localhost.localdomain>

On Mon, 14 Nov 2011 18:51:21 +0100, Arjen Schat <arjen at thenextweb.com>  
wrote:

> Varnish can't get over a 1024 threads.
> thread_pool_max = 1500

It can. Trust me.

> All kernel settings are above 1024
> Ulimit is above 1024

Can you post your startup command,
or /etc/default/varnish if you have it?

-- 
Cosimo



From simon at darkmere.gen.nz  Mon Nov 14 21:50:37 2011
From: simon at darkmere.gen.nz (Simon Lyall)
Date: Tue, 15 Nov 2011 10:50:37 +1300 (NZDT)
Subject: collectd 4.10 and Varnish
Message-ID: <alpine.DEB.2.00.1111151043110.31241@green.darkmere.gen.nz>


We're running collectd using version 4.10 from EPEL on RHEL/Centos.

There is a varnish plugin recently released with collectd but it requires 
version 5.10 of collectd which is fairly recently released and doesn't 
have rpms (I'd prefer not to build my own).

1. I was wondering if anybody has a backported varnish plugin for collectd
    4.10 or any other solution? I came across this post in the archives:

  https://www.varnish-cache.org/lists/pipermail/varnish-misc/2010-January/003736.html

   but I've not tried it yet. Any experience?


2. I was also thinking of using the collectd:tail plugin to pick out
    interesting stuff in the varnishncsa log. Things like hits on different
    types of pages on my site ( frontpage, section pages, article pages,
    etc ). Anyone else done something similar?


-- 
Simon Lyall  |  Very Busy  |  Web: http://www.darkmere.gen.nz/
"To stay awake all night adds a day to your life" - Stilgar | eMT.




From cosimo at streppone.it  Tue Nov 15 08:51:27 2011
From: cosimo at streppone.it (Cosimo Streppone)
Date: Tue, 15 Nov 2011 09:51:27 +0100
Subject: Stuck on max 1024 threads
In-Reply-To: <7FB4E6DE-D006-4DA3-8791-6BFE0201A6B9@arjenschat.com>
References: <D6A66FFE-4E34-4EA4-B3A4-96D17DD5A562@thenextweb.com>
	<op.v4x64btns5ttvb@localhost.localdomain>
	<7FB4E6DE-D006-4DA3-8791-6BFE0201A6B9@arjenschat.com>
Message-ID: <op.v4zb71ivs5ttvb@localhost.localdomain>

On Mon, 14 Nov 2011 19:38:44 +0100, Arjen Schat <arjen at thenextweb.com>  
wrote:

> This is the config used for startup.

Please send emails to the list.

> # Maximum number of open files (for ulimit -n)
> NFILES=131072
>
> [...]
>
> DAEMON_OPTS="-a 219.201.78.31:80 \
>              -T 168.20.2.106:6082 \
>              -f /etc/varnish/default.vcl \
>              -p listen_depth=4096 \
>              -p session_linger=150 \
>              -p thread_pool_add_delay=5 \
>              -p thread_pools=4 \
>              -p thread_pool_max=1500 \
>              -p thread_pool_min=50 \
>              -u varnish -g varnish \
>              -S /etc/varnish/secret \
>              -s malloc,6G"

How much memory do you have in your system?
An idea would be to check syslog for varnishd messages.

Also check varnishstat for the "N worker threads limited" counter.
That gives you an idea of how many and how quickly varnish tries
to create threads but doesn't succeed, IINM.

-- 
Cosimo



From arjen at thenextweb.com  Tue Nov 15 09:28:46 2011
From: arjen at thenextweb.com (Arjen Schat)
Date: Tue, 15 Nov 2011 10:28:46 +0100
Subject: Stuck on max 1024 threads
In-Reply-To: <op.v4zb71ivs5ttvb@localhost.localdomain>
References: <D6A66FFE-4E34-4EA4-B3A4-96D17DD5A562@thenextweb.com>
	<op.v4x64btns5ttvb@localhost.localdomain>
	<7FB4E6DE-D006-4DA3-8791-6BFE0201A6B9@arjenschat.com>
	<op.v4zb71ivs5ttvb@localhost.localdomain>
Message-ID: <2022A73D-BE26-40BC-AA3B-06A97DDA4044@arjenschat.com>

The machine has 16G

Munin complains like:  N worker threads not created is 4.33
Even on sustained high traffic, for half an hour for instance, it will always max out on 1024 threads. The munin graphs always flat out at this point.




On 15 nov. 2011, at 09:51, Cosimo Streppone wrote:

> On Mon, 14 Nov 2011 19:38:44 +0100, Arjen Schat <arjen at thenextweb.com> wrote:
> 
>> This is the config used for startup.
> 
> Please send emails to the list.
> 
>> # Maximum number of open files (for ulimit -n)
>> NFILES=131072
>> 
>> [...]
>> 
>> DAEMON_OPTS="-a 219.201.78.31:80 \
>>             -T 168.20.2.106:6082 \
>>             -f /etc/varnish/default.vcl \
>>             -p listen_depth=4096 \
>>             -p session_linger=150 \
>>             -p thread_pool_add_delay=5 \
>>             -p thread_pools=4 \
>>             -p thread_pool_max=1500 \
>>             -p thread_pool_min=50 \
>>             -u varnish -g varnish \
>>             -S /etc/varnish/secret \
>>             -s malloc,6G"
> 
> How much memory do you have in your system?
> An idea would be to check syslog for varnishd messages.
> 
> Also check varnishstat for the "N worker threads limited" counter.
> That gives you an idea of how many and how quickly varnish tries
> to create threads but doesn't succeed, IINM.
> 
> -- 
> Cosimo
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc




From thierry.magnien at sfr.com  Tue Nov 15 10:50:41 2011
From: thierry.magnien at sfr.com (MAGNIEN, Thierry)
Date: Tue, 15 Nov 2011 11:50:41 +0100
Subject: [PATCH proposal] RE: Backend DNS lookup refresh
Message-ID: <4A029B1A60B8E340A50D654D2F130DAA2FF45759CB@EXCV001.encara.local.ads>

Hi,

Here is a first version of a patch to add the "vcl.reload" function to CLI, in order to refresh DNS lookups without having to send a new VCL or restart varnish.

It is far from perfect but may be used as a starting point. It applies on master and of course all comments are welcome.

Regards,
Thierry

-----Message d'origine-----
De?: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] De la part de MAGNIEN, Thierry
Envoy??: jeudi 3 novembre 2011 17:15
??: 'varnish-misc at varnish-cache.org' (varnish-misc at varnish-cache.org)
Objet?: Backend DNS lookup refresh

Hi all,

We are dealing with a great number of backends in our VCL file and we have problems with some backends changing IP address (for example when hosted on services such as Amazon S3, that switch IP address without telling anyone).

We would like to be able to refresh the DNS resolution of backends when needed and after a quick look at the source code, it seems there are several ways to handle this.

I would like to get your feeling about what would be the best solution before trying to write a patch, so here are my thoughts, just as it comes out of my mind ;-)

DNS resolution of backends is done when the VCL is loaded. IP address is stored in the compiled VCL as a sockaddr struct.

The first way to solve the problem would be to implement a "vcl reload" CLI function. In that case, this needs some tricks to be able to recompile a VCL which has the same name of the one used.

A 2nd way would be to make backend IP addresses resolved when needed (at first backend request and then cached for a "TTL" time), and not at VCL loading. However, this breaks most of the the optimization brought by the sockaddr structure.

A point discussed quickly with Per Buer on IRC was to have probes checking the IP addresses of the backends and update when an IP change is detected.

All this makes me think that it's not so easy as it could seem, and that's why I would accept any advice/idea/comment about this.

Regards,
Thierry


_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vcl.reload.patch
Type: application/octet-stream
Size: 5198 bytes
Desc: vcl.reload.patch
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111115/adf138d9/attachment.obj>

From kennyds at ac-systems.com  Mon Nov 14 11:19:52 2011
From: kennyds at ac-systems.com (Kenny Deckers)
Date: Mon, 14 Nov 2011 12:19:52 +0100
Subject: Multiple Set-Cookie Headers
In-Reply-To: <4EC0F663.7070609@ac-systems.com>
References: <4EC0F663.7070609@ac-systems.com>
Message-ID: <4EC0F958.5090803@ac-systems.com>

  Hi,

Sorry that I did not elaborate this enough. Let me get into details:

We are running a magento web shop with a varnish layer in front of it on 
a separate server. At some point, the applications has to tell Varnish 
that it cannot use cached content, this happens by the setting the 
following header:

|Set-Cookie: EXTERNAL_NO_CACHE=1; expires=Thu, 10-Nov-2011 15:49:57 GMT; path=/; domain=XXXXXXXXX; httponly

*In the default.vcl the following happens the following in the sub vcl_fetch method:*

if (beresp.http.Set-Cookie ~ "NO_CACHE=") {
   return (pass)
}

Since there are multiple set-cookie headers, varnish will only be able to analyze the first Set-Cookie header.
This means that if-condition above fails and that it will not return a "pass" causing the application to serve cached content while it shouldn't.

I have compiled|the header vmod|  and imported it at the top of default.vcl. (import header;)

And I replaced the following:|
|
if (beresp.http.Set-Cookie ~ "NO_CACHE=") {|

with:

header.copy(beresp.http.Set-Cookie, beresp.http.X-Old-Cookie);
|if (beresp.http.X-Old-Cookie ~ "NO_CACHE=") {|

This still did not fix the issue.

Thanks for responding so quickly, if there is anything else we need to let you know, please tell us.

Awaiting your reply.

Kind regards,
Kenny
|
|




On 11/14/2011 11:58 AM, Kristian Lyngstol wrote:
> Hi,
>
> On Mon, Nov 14, 2011 at 09:42:59AM +0100, Kenny Deckers wrote:
>> I'm having the following issue with "multiple set-cookie headers":
> (...)
>
> What issue? You are not saying anything about what you wan to
> accomplish.
>
> Your web server sends multiple set-cookie headers. I assume you entered
> the stars yourself.
>
> What are you trying to accomplish, and what have you actually tried with
> the header vmod?
>
> - Kristian
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visithttp://www.messagelabs.com/email
> ______________________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111114/98f38c02/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 25873 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111114/98f38c02/attachment.jpe>

From kennyds at ac-systems.com  Mon Nov 14 13:19:39 2011
From: kennyds at ac-systems.com (Kenny Deckers)
Date: Mon, 14 Nov 2011 14:19:39 +0100
Subject: Multiple Set-Cookie Headers
In-Reply-To: <CAAyZgShq79yZGXfhXtPxmXWGd3RuzEbyOgJf_Y-FJyo8j08N+w@mail.gmail.com>
References: <4EC0D493.80709@ac-systems.com>
	<20111114105809.GB5602@freud.kly.no>
	<CAAyZgShq79yZGXfhXtPxmXWGd3RuzEbyOgJf_Y-FJyo8j08N+w@mail.gmail.com>
Message-ID: <4EC1156B.9000201@ac-systems.com>

Hi,

Yes that is correct. if the "Set-Cookie: EXTERNAL_NO_CACHE=1" header is 
set the response should be "passed".
Since Varnish can only analyze the first "Set-Cookie" header and the 
"EXTERNAL_NO_CACHE=1" is present in one of the following headers it will 
not see it and I can't have it 'pass' the response.

Regards,
Kenny


On 11/14/2011 01:51 PM, Roberto O. Fern?ndez Crisial wrote:
> Hi,
>
> I think Kenny's issue is that Varnish could be caching Set-Cookie 
> response and send cached object to all users with those Set-Cookie 
> headers (like any session).
>
> I think Kenny should pass any "Set-Cookie: EXTERNAL_NO_CACHE=1" response.
>
> Roberto O. Fern?ndez Crisial
> @rofc*
> *
> On Mon, Nov 14, 2011 at 7:58 AM, Kristian Lyngstol 
> <kristian at varnish-software.com <mailto:kristian at varnish-software.com>> 
> wrote:
>
>     Hi,
>
>     On Mon, Nov 14, 2011 at 09:42:59AM +0100, Kenny Deckers wrote:
>     > I'm having the following issue with "multiple set-cookie headers":
>
>     (...)
>
>     What issue? You are not saying anything about what you wan to
>     accomplish.
>
>     Your web server sends multiple set-cookie headers. I assume you
>     entered
>     the stars yourself.
>
>     What are you trying to accomplish, and what have you actually
>     tried with
>     the header vmod?
>
>     - Kristian
>
>     _______________________________________________
>     varnish-misc mailing list
>     varnish-misc at varnish-cache.org <mailto:varnish-misc at varnish-cache.org>
>     https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111114/ecea9896/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: business_card.jpg
Type: image/jpeg
Size: 25873 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111114/ecea9896/attachment.jpg>

From j.vanarragon at lukkien.com  Wed Nov 16 09:09:03 2011
From: j.vanarragon at lukkien.com (Jaap van Arragon)
Date: Wed, 16 Nov 2011 10:09:03 +0100
Subject: Multiple Set-Cookie Headers
In-Reply-To: <4EC1156B.9000201@ac-systems.com>
Message-ID: <CAE93C3F.58572%j.vanarragon@lukkien.com>

Hi Kenny,

We?ve had the same problem. The only solution we came up, because we didn?t
had a lot of time, was to adjust the application ;-(

So I am very curious to your thread and the solution off it.

Regards,
Jaap 


On 11/14/11 2:19 PM, "Kenny Deckers" <kennyds at ac-systems.com> wrote:

>    Hi,
>  
>  Yes that is correct. if the "Set-Cookie: EXTERNAL_NO_CACHE=1" header is set
> the response should be "passed".
>  Since Varnish can only analyze the first "Set-Cookie" header and the
> "EXTERNAL_NO_CACHE=1" is present in one of the following headers it will not
> see it and I can't have it 'pass' the response.
>  
>  Regards,
>  Kenny
>  
> 
>  
>  
>  On 11/14/2011 01:51 PM, Roberto O. Fern?ndez Crisial wrote:
>>  
>> Hi,
>>  
>>  
>>  
>> I think Kenny's issue is that Varnish could be caching Set-Cookie response
>> and send cached object to all users with those Set-Cookie headers (like any
>> session).
>>  
>>  
>>  
>> I think Kenny should pass any "Set-Cookie: EXTERNAL_NO_CACHE=1" response.
>>  
>>  Roberto O. Fern?ndez Crisial
>>  @rofc
>>  
>>  
>>  On Mon, Nov 14, 2011 at 7:58 AM, Kristian Lyngstol
>> <kristian at varnish-software.com> wrote:
>>  
>>>  Hi,
>>>  
>>> 
>>>  On Mon, Nov 14, 2011 at 09:42:59AM +0100, Kenny Deckers wrote:
>>>>  > I'm having the following issue with "multiple set-cookie headers":
>>>  
>>>  
>>>  (...)
>>>  
>>>  What issue? You are not saying anything about what you wan to
>>>  accomplish.
>>>  
>>>  Your web server sends multiple set-cookie headers. I assume you entered
>>>  the stars yourself.
>>>  
>>>  What are you trying to accomplish, and what have you actually tried with
>>>  the header vmod?
>>>  
>>>  - Kristian
>>>  
>>>  _______________________________________________
>>>  varnish-misc mailing list
>>>  varnish-misc at varnish-cache.org
>>>  https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>  
>>  
>>  
>>  
>>  
>> ______________________________________________________________________
>>  This email has been scanned by the MessageLabs Email Security System.
>>  For more information please visit http://www.messagelabs.com/email
>> ______________________________________________________________________
>>  
>  
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/567ed8e6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.jpg
Type: image/jpeg
Size: 25873 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/567ed8e6/attachment.jpg>

From apj at mutt.dk  Wed Nov 16 09:19:07 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Wed, 16 Nov 2011 10:19:07 +0100
Subject: Multiple Set-Cookie Headers
In-Reply-To: <4EC0F958.5090803@ac-systems.com>
References: <4EC0F663.7070609@ac-systems.com> <4EC0F958.5090803@ac-systems.com>
Message-ID: <20111116091907.GC13866@nerd.dk>

On Mon, Nov 14, 2011 at 12:19:52PM +0100, Kenny Deckers wrote:

> |if (beresp.http.X-Old-Cookie ~ "NO_CACHE=") {|

if(header.get(beresp.http.Set-Cookie, "NO_CACHE=")) {

-- 
Andreas



From marc.fournier at camptocamp.com  Wed Nov 16 11:31:47 2011
From: marc.fournier at camptocamp.com (Marc Fournier)
Date: Wed, 16 Nov 2011 12:31:47 +0100
Subject: collectd 4.10 and Varnish
In-Reply-To: <alpine.DEB.2.00.1111151043110.31241@green.darkmere.gen.nz>
References: <alpine.DEB.2.00.1111151043110.31241@green.darkmere.gen.nz>
Message-ID: <1321442578-sup-7101@lonquimay.wrk.lsn.camptocamp.com>

Excerpts from Simon Lyall's message of Mon Nov 14 22:50:37 +0100 2011:
> 
> We're running collectd using version 4.10 from EPEL on RHEL/Centos.
> 
> There is a varnish plugin recently released with collectd but it requires 
> version 5.10 of collectd which is fairly recently released and doesn't 
> have rpms (I'd prefer not to build my own).
> 
> 1. I was wondering if anybody has a backported varnish plugin for collectd
>     4.10 or any other solution? I came across this post in the archives:

I have done exactly this a few months ago. I just pushed the patch to:
https://github.com/mfournier/collectd/tree/varnish-on-4.10.2

Not sure if the plugin is up to date with the one found in 5.x though...

Cheers,
Marc



From rlane at ahbelo.com  Wed Nov 16 12:06:01 2011
From: rlane at ahbelo.com (Lane, Richard)
Date: Wed, 16 Nov 2011 12:06:01 +0000
Subject: Cookie value compare, integer
Message-ID: <CAE90349.4B913%rlane@ahbelo.com>

Is anyone comparing cookie values that are integers?

I am getting the value of a cookie and then trying to compare it to another integer value but am getting a compile error. Can I compare integers stored in the req object? Should I be going about this a different way?

For example:

--sitename.vcl

sitename_recv{

    set req.http.compareVal=10;
    set req.http.cookieVal=5;

   call cookieValCompare;

}

sub cookieValCompare{

     if( req.http.cookieVal < req.http.compareVal)
    {
       #do something extra
    }
}


If I try to compile via the console I get:

Message from VCC-compiler:
Expected ')' got '<'
(program line 109), at
(/etc/varnish/sitename.vcl Line 11 Pos 28)
      if( req.http.cookieVal < req.http.compareVal )
---------------------------------#-----------------------------
Running VCC-compiler failed, exit 1VCL compilation failed
Command failed with error code 106
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/b91b0b69/attachment.html>

From apj at mutt.dk  Wed Nov 16 12:23:15 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Wed, 16 Nov 2011 13:23:15 +0100
Subject: Cookie value compare, integer
In-Reply-To: <CAE90349.4B913%rlane@ahbelo.com>
References: <CAE90349.4B913%rlane@ahbelo.com>
Message-ID: <20111116122315.GD13866@nerd.dk>

On Wed, Nov 16, 2011 at 12:06:01PM +0000, Lane, Richard wrote:
> 
> I am getting the value of a cookie and then trying to compare it to another
> integer value but am getting a compile error. Can I compare integers stored
> in the req object? Should I be going about this a different way?

Use the conversion method in vmod_std:
https://www.varnish-cache.org/docs/trunk/reference/vmod_std.html#integer

-- 
Andreas



From ask at develooper.com  Wed Nov 16 18:07:58 2011
From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Wed, 16 Nov 2011 10:07:58 -0800
Subject: objects vs object heads
Message-ID: <FEA4A92D-5B15-4C30-B6F9-B9E94DD78D19@develooper.com>

On a new/test installation (on 3.0.2; the one with the gzip crash hence the frequent restarts) I enabled persistent storage on one of the two nodes.

Before enabling persistent storage "objects" and "object heads" were basically the same number (see first graph below from the other server).  After enabling persistent storage (graph from the other server below) "objects" is a tiny number and "object heads" seems similar to before.

Is that expected?

Ask


 



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/b51ed2aa/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: varnish_objects-day.png
Type: image/png
Size: 27852 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/b51ed2aa/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: varnish_objects-day.png
Type: image/png
Size: 29307 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/b51ed2aa/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4798 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/b51ed2aa/attachment.bin>

From rlane at ahbelo.com  Wed Nov 16 18:53:05 2011
From: rlane at ahbelo.com (Lane, Richard)
Date: Wed, 16 Nov 2011 18:53:05 +0000
Subject: Cookie value compare, integer
In-Reply-To: <CAE90349.4B913%rlane@ahbelo.com>
Message-ID: <CAE962B1.4B95A%rlane@ahbelo.com>

I am currently on Varnish 2.1.5. Looks like that vmod_std module was added in 3.x version. I was hoping to avoid getting into writing C code for this. Any other ideas?

>
> I am getting the value of a cookie and then trying to compare it to another
> integer value but am getting a compile error. Can I compare integers stored
> in the req object? Should I be going about this a different way?

Use the conversion method in vmod_std:
https://www.varnish-cache.org/docs/trunk/reference/vmod_std.html#integer

--
Andreas


On 11/16/11 6:06 AM, "Lane, Richard" <rlane at ahbelo.com> wrote:

Is anyone comparing cookie values that are integers?

I am getting the value of a cookie and then trying to compare it to another integer value but am getting a compile error. Can I compare integers stored in the req object? Should I be going about this a different way?

For example:

--sitename.vcl

sitename_recv{

    set req.http.compareVal=10;
    set req.http.cookieVal=5;

   call cookieValCompare;

}

sub cookieValCompare{

     if( req.http.cookieVal < req.http.compareVal)
    {
       #do something extra
    }
}


If I try to compile via the console I get:

Message from VCC-compiler:
Expected ')' got '<'
(program line 109), at
(/etc/varnish/sitename.vcl Line 11 Pos 28)
      if( req.http.cookieVal < req.http.compareVal )
---------------------------------#-----------------------------
Running VCC-compiler failed, exit 1VCL compilation failed
Command failed with error code 106

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/75fe6b99/attachment.html>

From ask at develooper.com  Thu Nov 17 05:28:43 2011
From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=)
Date: Wed, 16 Nov 2011 21:28:43 -0800
Subject: Is persistent varnish storage ready for use? (was: objects vs object
	heads)
In-Reply-To: <FEA4A92D-5B15-4C30-B6F9-B9E94DD78D19@develooper.com>
References: <FEA4A92D-5B15-4C30-B6F9-B9E94DD78D19@develooper.com>
Message-ID: <8ACC438B-B38C-4525-84D9-BA8B65AE5891@develooper.com>


On Nov 16, 2011, at 10:07, Ask Bj?rn Hansen wrote:

> On a new/test installation (on 3.0.2; the one with the gzip crash hence the frequent restarts) I enabled persistent storage on one of the two nodes.

Uh, our systems guy who's much smarter than me noticed that it was crashing every few minutes (log below).  I think what happened was that Varnish got confused because our "persistent" file was the same as the old "file" file?

This problem went away when we removed the file and let Varnish create a new one -- but apparently it just crashed in some other way a little while later.   (I wasn't in the office today and only understood that it didn't work and they were going to switch back).

Anyway ? is persistent storage considered ready for use?  It's listed as "experimental" in the man page, but I was hoping that was overly cautious since the feature has been around for a while now.  :-)


 - ask


Nov 16 21:23:52 varnish2 varnishd[4978]: child (30252) Started
Nov 16 21:23:52 varnish2 varnishd[4978]: Child (30252) said Child starts
Nov 16 21:23:52 varnish2 varnishd[4978]: Child (30252) said Dropped 10 segments to make free_reserve
Nov 16 21:23:53 varnish2 varnishd[4978]: Child (30252) said Silo completely loaded
Nov 16 21:26:57 varnish2 varnishd[4978]: Child (30252) said Out of space in persistent silo
Nov 16 21:26:57 varnish2 varnishd[4978]: Child (30252) said Committing suicide, restart will make space
Nov 16 21:26:57 varnish2 varnishd[4978]: Child (30252) ended
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4798 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/8d84549c/attachment.bin>

From arjen at thenextweb.com  Thu Nov 17 08:30:17 2011
From: arjen at thenextweb.com (Arjen Schat)
Date: Thu, 17 Nov 2011 09:30:17 +0100
Subject: Stuck on max 1024 threads
In-Reply-To: <7FB4E6DE-D006-4DA3-8791-6BFE0201A6B9@arjenschat.com>
References: <D6A66FFE-4E34-4EA4-B3A4-96D17DD5A562@thenextweb.com>
	<op.v4x64btns5ttvb@localhost.localdomain>
	<7FB4E6DE-D006-4DA3-8791-6BFE0201A6B9@arjenschat.com>
Message-ID: <4D6BF7F2-F6E1-487B-A908-D923DD75CEA2@arjenschat.com>

Does anyone have information in this subject?

Greatly appreciated,

Arjen

On 14 nov. 2011, at 19:38, Arjen Schat wrote:

> Hi,
> 
> This is the config used for startup.
> 
> # Maximum number of open files (for ulimit -n)
> NFILES=131072
> 
> # Locked shared memory (for ulimit -l)
> # Default log size is 82MB + header
> MEMLOCK=82000
> 
> 
> DAEMON_OPTS="-a 219.201.78.31:80 \
>             -T 168.20.2.106:6082 \
>             -f /etc/varnish/default.vcl \
>             -p listen_depth=4096 \
>             -p session_linger=150 \
>             -p thread_pool_add_delay=5 \
>             -p thread_pools=4 \
>             -p thread_pool_max=1500 \
>             -p thread_pool_min=50 \
>             -u varnish -g varnish \
>             -S /etc/varnish/secret \
>             -s malloc,6G"
> 
> 
> Thank you,
> Arjen
> 
> On 14 nov. 2011, at 19:03, Cosimo Streppone wrote:
> 
>> On Mon, 14 Nov 2011 18:51:21 +0100, Arjen Schat <arjen at thenextweb.com> wrote:
>> 
>>> Varnish can't get over a 1024 threads.
>>> thread_pool_max = 1500
>> 
>> It can. Trust me.
>> 
>>> All kernel settings are above 1024
>>> Ulimit is above 1024
>> 
>> Can you post your startup command,
>> or /etc/default/varnish if you have it?
>> 
>> -- 
>> Cosimo
>> 
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> 




From apj at mutt.dk  Thu Nov 17 09:15:06 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Thu, 17 Nov 2011 10:15:06 +0100
Subject: Stuck on max 1024 threads
In-Reply-To: <4D6BF7F2-F6E1-487B-A908-D923DD75CEA2@arjenschat.com>
References: <D6A66FFE-4E34-4EA4-B3A4-96D17DD5A562@thenextweb.com>
	<op.v4x64btns5ttvb@localhost.localdomain>
	<7FB4E6DE-D006-4DA3-8791-6BFE0201A6B9@arjenschat.com>
	<4D6BF7F2-F6E1-487B-A908-D923DD75CEA2@arjenschat.com>
Message-ID: <20111117091506.GF13866@nerd.dk>

On Thu, Nov 17, 2011 at 09:30:17AM +0100, Arjen Schat wrote:

> Does anyone have information in this subject?

what does "cat /proc/`pidof -s varnishd`/limits" have to say?

Also, please quote properly on the list.

-- 
Andreas



From martin at 925.dk  Thu Nov 17 09:30:55 2011
From: martin at 925.dk (Martin Koch Andersen)
Date: Thu, 17 Nov 2011 10:30:55 +0100
Subject: ESI ban question
Message-ID: <7FB92104-7DFF-4C13-A8C9-75E7D9E1CAE1@925.dk>

Hello,

If you have /a.html containing:

<h1>a.html</h1>
<esi:include src="/b.html" />

And I then do a BAN of /a.html - it seems that /b.html is banned as well?
This to me is strange, as they should be considered two different cache entities, no?

I would not expect a fetch of /b.html when banning only /a.html.
Am using varnish-3.0.1.

Kind regards,

-- 
Martin Koch Andersen
http://925.dk


From apj at mutt.dk  Thu Nov 17 09:42:19 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Thu, 17 Nov 2011 10:42:19 +0100
Subject: ESI ban question
In-Reply-To: <7FB92104-7DFF-4C13-A8C9-75E7D9E1CAE1@925.dk>
References: <7FB92104-7DFF-4C13-A8C9-75E7D9E1CAE1@925.dk>
Message-ID: <20111117094219.GG13866@nerd.dk>

On Thu, Nov 17, 2011 at 10:30:55AM +0100, Martin Koch Andersen wrote:
> 
> If you have /a.html containing:
> 
> <h1>a.html</h1>
> <esi:include src="/b.html" />
> 
> And I then do a BAN of /a.html - it seems that /b.html is banned as well?

No:

apj at marvin:~$ cat /var/www/a.html 
<h1>a.html</h1>
<esi:include src="/b.html" />
apj at marvin:~$ cat /var/www/b.html 
2
apj at marvin:~$ varnishadm ban.url a.html
CLI connected to 127.0.0.1 6082

apj at marvin:~$ GET localhost/a.html
<h1>a.html</h1>
1
apj at marvin:~$ varnishadm ban.url b.html
CLI connected to 127.0.0.1 6082

apj at marvin:~$ GET localhost/a.html
<h1>a.html</h1>
2

(b.html used to contain "1")

-- 
Andreas



From martin at 925.dk  Thu Nov 17 10:11:20 2011
From: martin at 925.dk (Martin Koch Andersen)
Date: Thu, 17 Nov 2011 11:11:20 +0100
Subject: ESI ban question
In-Reply-To: <20111117094219.GG13866@nerd.dk>
References: <7FB92104-7DFF-4C13-A8C9-75E7D9E1CAE1@925.dk>
	<20111117094219.GG13866@nerd.dk>
Message-ID: <F104B30D-801C-40EA-AA13-EF13DC7ECEC0@925.dk>


Den 17/11/2011 kl. 10.42 skrev Andreas Plesner Jacobsen:
> On Thu, Nov 17, 2011 at 10:30:55AM +0100, Martin Koch Andersen wrote:
>> 
>> If you have /a.html containing:
>> 
>> <h1>a.html</h1>
>> <esi:include src="/b.html" />
>> 
>> And I then do a BAN of /a.html - it seems that /b.html is banned as well?
> 
> No:

Thanks. Well thats all good then. Thats the behavior I would expect.

I must be doing something else wrong.

-- 
Martin Koch Andersen
http://925.dk



From arjen at thenextweb.com  Thu Nov 17 13:41:58 2011
From: arjen at thenextweb.com (Arjen Schat)
Date: Thu, 17 Nov 2011 14:41:58 +0100
Subject: Stuck on max 1024 threads
In-Reply-To: <20111117091506.GF13866@nerd.dk>
References: <D6A66FFE-4E34-4EA4-B3A4-96D17DD5A562@thenextweb.com>
	<op.v4x64btns5ttvb@localhost.localdomain>
	<7FB4E6DE-D006-4DA3-8791-6BFE0201A6B9@arjenschat.com>
	<4D6BF7F2-F6E1-487B-A908-D923DD75CEA2@arjenschat.com>
	<20111117091506.GF13866@nerd.dk>
Message-ID: <F63D5691-AADE-4055-99E5-64ED759ACCDC@arjenschat.com>

On 17 nov. 2011, at 10:15, Andreas Plesner Jacobsen wrote:

> what does "cat /proc/`pidof -s varnishd`/limits" have to say?

Max processes             1024                 123864               processes 

That was the hint was looking for. With the extra command "ulimit -u 4096" in /etc/rc.d/init.d/varnish it works. Varnish should now be able to handle 4000+ simultaneous connections.

Thank you,
Arjen


From Jeff.Stroomer at disney.com  Thu Nov 17 21:10:52 2011
From: Jeff.Stroomer at disney.com (Stroomer, Jeff)
Date: Thu, 17 Nov 2011 13:10:52 -0800
Subject: non-sudo access to varnishlog
Message-ID: <4F86EEFDF44D2D41951D491B61B7D433249CA0374A@SM-CALA-VXMB06B.swna.wdpr.disney.com>

Everyone,

We're getting close to deploying the live version of our website.  On the live site, I don't have sudo rights, but nevertheless I need to be able to log in and debug problems.  Unfortunately, when I try to run varnishlog without sudo I run into the following:

jstroomer at duckman ~
$ varnishlog
Cannot open /var/lib/varnish/duckman/_.vsm: Permission denied

I've tried playing with the permissions on /etc/varnish/secret and /var/lib/varnish/duckman/_.vsm, but so far haven't found a way to solve my problem.  Any advice?

Jeff

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111117/6d56fb60/attachment.html>

From lampe at hauke-lampe.de  Thu Nov 17 22:47:59 2011
From: lampe at hauke-lampe.de (Hauke Lampe)
Date: Thu, 17 Nov 2011 23:47:59 +0100
Subject: non-sudo access to varnishlog
In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D433249CA0374A@SM-CALA-VXMB06B.swna.wdpr.disney.com>
References: <4F86EEFDF44D2D41951D491B61B7D433249CA0374A@SM-CALA-VXMB06B.swna.wdpr.disney.com>
Message-ID: <4EC58F1F.4040605@hauke-lampe.de>

Hi Jeff.

On 17.11.2011 22:10, Stroomer, Jeff wrote:

> On the live site, I don't have sudo rights, but nevertheless I need to
be able to log in and debug problems.  Unfortunately, when I try to run
varnishlog without sudo I run into the following:
> 
> jstroomer at duckman ~
> $ varnishlog
> Cannot open /var/lib/varnish/duckman/_.vsm: Permission denied

I'd try to convince the admin to give you sudo rights to run varnishlog
as the varnishlog user or group.

If you can't use sudo, however, ask the server admin to add your user to
the varnishlog group and set the varnish directories permissions so that
the group can access them:

dr-xr-x--- 2 varnish varnishlog  /var/lib/varnish
dr-xr-x--- 2 varnish varnishlog  /var/lib/varnish/$yourhost

("r" isn't actually necessary, but it probably helps debugging problems
if you can list the directory contents, too)

packbart at alberich:~$ id
uid=1000(packbart) gid=[...] 113(varnishlog)
packbart at alberich:~$ varnishlog
    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1321569446 1.0


Hauke

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111117/155e74d0/attachment.bin>

From kennyds at ac-systems.com  Wed Nov 16 09:15:50 2011
From: kennyds at ac-systems.com (Kenny Deckers)
Date: Wed, 16 Nov 2011 10:15:50 +0100
Subject: Multiple Set-Cookie Headers
In-Reply-To: <CAE93C3F.58572%j.vanarragon@lukkien.com>
References: <CAE93C3F.58572%j.vanarragon@lukkien.com>
Message-ID: <4EC37F46.1000201@ac-systems.com>

Hi,

- I installed Varnish 3.0.1 from source
- I installed libvmod-header (https://github.com/KristianLyng/libvmod-header
- I modified the default.vcl vcl_fetch method:
     beresp.http.Set-Cookie ~ "NO_CACHE="
     replaced by
     header.get(beresp.http.Set-Cookie, "NO_CACHE=")

If there is anything else please let me know. If you want us to have a 
look at it please let me know so I can get back to you.

Regards,/
Kenny
/


On 11/16/2011 10:09 AM, Jaap van Arragon wrote:
> Hi Kenny,
>
> We've had the same problem. The only solution we came up, because we 
> didn't had a lot of time, was to adjust the application ;-(
>
> So I am very curious to your thread and the solution off it.
>
> Regards,
> Jaap
>
>
> On 11/14/11 2:19 PM, "Kenny Deckers" <kennyds at ac-systems.com> wrote:
>
>       Hi,
>
>      Yes that is correct. if the "Set-Cookie: EXTERNAL_NO_CACHE=1"
>     header is set the response should be "passed".
>      Since Varnish can only analyze the first "Set-Cookie" header and
>     the "EXTERNAL_NO_CACHE=1" is present in one of the following
>     headers it will not see it and I can't have it 'pass' the response.
>
>      Regards,
>      Kenny
>
>
>
>
>      On 11/14/2011 01:51 PM, Roberto O. Fern?ndez Crisial wrote:
>
>
>         Hi,
>
>
>
>         I think Kenny's issue is that Varnish could be caching
>         Set-Cookie response and send cached object to all users with
>         those Set-Cookie headers (like any session).
>
>
>
>         I think Kenny should pass any "Set-Cookie:
>         EXTERNAL_NO_CACHE=1" response.
>
>          Roberto O. Fern?ndez Crisial
>          @rofc
>         *
>         *
>          On Mon, Nov 14, 2011 at 7:58 AM, Kristian Lyngstol
>         <kristian at varnish-software.com> wrote:
>
>             Hi,
>
>
>              On Mon, Nov 14, 2011 at 09:42:59AM +0100, Kenny Deckers
>             wrote:
>             > I'm having the following issue with "multiple set-cookie
>             headers":
>
>
>              (...)
>
>              What issue? You are not saying anything about what you wan to
>              accomplish.
>
>              Your web server sends multiple set-cookie headers. I
>             assume you entered
>              the stars yourself.
>
>              What are you trying to accomplish, and what have you
>             actually tried with
>              the header vmod?
>
>              - Kristian
>
>              _______________________________________________
>              varnish-misc mailing list
>             varnish-misc at varnish-cache.org
>             https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
>
>
>         ______________________________________________________________________
>          This email has been scanned by the MessageLabs Email Security
>         System.
>          For more information please visit
>         http://www.messagelabs.com/email
>         ______________________________________________________________________
>
>
>     ------------------------------------------------------------------------
>     _______________________________________________
>     varnish-misc mailing list
>     varnish-misc at varnish-cache.org
>     https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/69368b7b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: business_card.jpg
Type: image/jpeg
Size: 25873 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/69368b7b/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 25873 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111116/69368b7b/attachment.jpe>

From christian at moc.net  Fri Nov 18 14:18:38 2011
From: christian at moc.net (Christian Jul Jensen)
Date: Fri, 18 Nov 2011 15:18:38 +0100
Subject: Manipulating ESI request
Message-ID: <4EC6693E.8040100@moc.net>

Hi

I am prototyping a setup of using Varnish as the hub for a 
Service-Oriented setup.

The idea is to be able to run stand-alone applications within the 
context of a cms transparently to the end-user.

I'm trying to use ESI for that. Which works, but I would like to 
manipulate the request for the esi depending on the request.

I imagined something like

	set beresp.do_esi = true;
	set beresp.url = beresp.url + somePartOf(req.url);

in vcl_fetch, but I cant make it work.

Is this not possible, or do I just not have a good enough understanding 
of the call sequence in varnish?

thanks

-- 
Christian Jul Jensen

MOC - http://moc.net
@christianjul




From apj at mutt.dk  Fri Nov 18 14:25:53 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Fri, 18 Nov 2011 15:25:53 +0100
Subject: Manipulating ESI request
In-Reply-To: <4EC6693E.8040100@moc.net>
References: <4EC6693E.8040100@moc.net>
Message-ID: <20111118142553.GA3214@nerd.dk>

On Fri, Nov 18, 2011 at 03:18:38PM +0100, Christian Jul Jensen wrote:
> 
> I'm trying to use ESI for that. Which works, but I would like to
> manipulate the request for the esi depending on the request.
> 
> I imagined something like
> 
> 	set beresp.do_esi = true;
> 	set beresp.url = beresp.url + somePartOf(req.url);

That's too late. beresp is not available until after you've fetched from the
backend. So yes, you're rewriting it, but it has no effect.

-- 
Andreas



From christian at moc.net  Fri Nov 18 14:41:32 2011
From: christian at moc.net (Christian Jul Jensen)
Date: Fri, 18 Nov 2011 15:41:32 +0100
Subject: Manipulating ESI request
In-Reply-To: <4EC66C47.6090309@moc.net>
References: <4EC6693E.8040100@moc.net> <20111118142553.GA3214@nerd.dk>
	<4EC66C47.6090309@moc.net>
Message-ID: <4EC66E9C.9070209@moc.net>

On 18. nov. 15.25, Andreas Plesner Jacobsen wrote:
> That's too late. beresp is not available until after you've fetched
> from the
> backend. So yes, you're rewriting it, but it has no effect.

Thanks, I thought so. Any suggestion on how to achieve this? Or is it 
just not possible to manipulate the ESI request based on parameters on 
the triggering request?

-- 
Christian Jul Jensen

MOC - http://moc.net
@christianjul



From apj at mutt.dk  Fri Nov 18 14:49:29 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Fri, 18 Nov 2011 15:49:29 +0100
Subject: Manipulating ESI request
In-Reply-To: <4EC66E9C.9070209@moc.net>
References: <4EC6693E.8040100@moc.net> <20111118142553.GA3214@nerd.dk>
	<4EC66C47.6090309@moc.net> <4EC66E9C.9070209@moc.net>
Message-ID: <20111118144929.GC3214@nerd.dk>

On Fri, Nov 18, 2011 at 03:41:32PM +0100, Christian Jul Jensen wrote:

> >That's too late. beresp is not available until after you've fetched
> >from the
> >backend. So yes, you're rewriting it, but it has no effect.
> 
> Thanks, I thought so. Any suggestion on how to achieve this? Or is
> it just not possible to manipulate the ESI request based on
> parameters on the triggering request?

I can't think of a way. The ESI requests are considered isolated requests, and
vcl_fetch may not have run just before fetching them (if the outer object is
cached, but the ESId object isn't). There may also be more than one ESI
request, so how would you separate them?

I think you need to rethink your strategy.

-- 
Andreas



From travis.crowder at spechal.com  Fri Nov 18 20:36:08 2011
From: travis.crowder at spechal.com (Travis Crowder)
Date: Fri, 18 Nov 2011 14:36:08 -0600
Subject: Can Varnish Set Cookies?
Message-ID: <4EC6C1B8.8030508@spechal.com>

Sorry if this is on the ineternet somewhere, but I have yet to find it.

Can Varnish set a cookie?

In example, if a user on a mobile device goes to xyz.com, Varnish sees a 
mobile user agent and redirects them to m.xyz.com via the vcl_error 
section.  Before or after the redirect, can Varnish set a cookie without 
the application doing so?

Thanks,
Travis



From roberto.fernandezcrisial at gmail.com  Fri Nov 18 20:48:56 2011
From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=)
Date: Fri, 18 Nov 2011 17:48:56 -0300
Subject: Can Varnish Set Cookies?
In-Reply-To: <4EC6C1B8.8030508@spechal.com>
References: <4EC6C1B8.8030508@spechal.com>
Message-ID: <CAAyZgSg2D3W7XYY7JhfM1WaxLSNCLWGFXhZHO1ipj9n8vOt7JA@mail.gmail.com>

Travis,

Have you tried with..

set req.http.Set-Cookie = "something";
set beresp.http.Set-Cookie = "something";

?

Regards,
Roberto O. Fern?ndez Crisial
@rofc


On Fri, Nov 18, 2011 at 5:36 PM, Travis Crowder
<travis.crowder at spechal.com>wrote:

> Sorry if this is on the ineternet somewhere, but I have yet to find it.
>
> Can Varnish set a cookie?
>
> In example, if a user on a mobile device goes to xyz.com, Varnish sees a
> mobile user agent and redirects them to m.xyz.com via the vcl_error
> section.  Before or after the redirect, can Varnish set a cookie without
> the application doing so?
>
> Thanks,
> Travis
>
> ______________________________**_________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc<https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111118/f9674d79/attachment.html>

From jhayter at manta.com  Fri Nov 18 20:55:40 2011
From: jhayter at manta.com (Jim Hayter)
Date: Fri, 18 Nov 2011 20:55:40 +0000
Subject: Can Varnish Set Cookies?
In-Reply-To: <4EC6C1B8.8030508@spechal.com>
References: <4EC6C1B8.8030508@spechal.com>
Message-ID: <AB0CF49818D2624A82954E14751BE3D42464504B@cmhvm005>

Travis wrote:
 > Sorry if this is on the ineternet somewhere, but I have yet to find it.

 > Can Varnish set a cookie?

> In example, if a user on a mobile device goes to xyz.com, Varnish sees a 
> mobile user agent and redirects them to m.xyz.com via the vcl_error 
> section.  Before or after the redirect, can Varnish set a cookie without 
> the application doing so?

> Thanks,
> Travis

I'm running varnish 2.0.5 but would not expect the syntax to have changed much.
The obj part may no longer be what is used, I believe the current versions use beresp.

I use the code below in vcl_fetch.
   If (test for setting cookie) {
	set obj.http.Set-Cookie = "use_mobile_dev=yes; domain=.<xxxxxxxx>.com; path=/";
   }

This works for the limited case where I need it.  I don't remember the impact if any 
Set-Cookie headers already exist.

Jim



From travis.crowder at spechal.com  Sat Nov 19 05:33:18 2011
From: travis.crowder at spechal.com (Travis Crowder)
Date: Fri, 18 Nov 2011 23:33:18 -0600
Subject: Can Varnish Set Cookies?
In-Reply-To: <AB0CF49818D2624A82954E14751BE3D42464504B@cmhvm005>
References: <4EC6C1B8.8030508@spechal.com>
	<AB0CF49818D2624A82954E14751BE3D42464504B@cmhvm005>
Message-ID: <4EC73F9E.1040203@spechal.com>

On 11/18/2011 2:55 PM, Jim Hayter wrote:
> Travis wrote:
>   >  Sorry if this is on the ineternet somewhere, but I have yet to find it.
>
>   >  Can Varnish set a cookie?
>
>> In example, if a user on a mobile device goes to xyz.com, Varnish sees a
>> mobile user agent and redirects them to m.xyz.com via the vcl_error
>> section.  Before or after the redirect, can Varnish set a cookie without
>> the application doing so?
>> Thanks,
>> Travis
> I'm running varnish 2.0.5 but would not expect the syntax to have changed much.
> The obj part may no longer be what is used, I believe the current versions use beresp.
>
> I use the code below in vcl_fetch.
>     If (test for setting cookie) {
> 	set obj.http.Set-Cookie = "use_mobile_dev=yes; domain=.<xxxxxxxx>.com; path=/";
>     }
>
> This works for the limited case where I need it.  I don't remember the impact if any
> Set-Cookie headers already exist.
>
> Jim

Thank you!

I am using this for mobile detection in case it can help anyone else.  
It checks for either device=mobile or device=desktop in the query 
string.  It works with = or / to be diverse in case my developers decide 
to change it to use device/desktop or device/mobile in the future.

vcl_fetch(){
         # if they are accessing device/desktop, pass
         if(req.url ~ "device(=|/)desktop") {
                 if(req.http.Cookie ~ "view_type=mobile"){
                         error 306 req.http.Host;
                 }
                 return(pass);
         } elseif (req.url ~ "device(=|/)mobile" && req.http.Cookie ~ 
"view_type=desktop") {
                 error 305 req.http.host;
         }
         if(req.http.Cookie !~ "view_type" && req.http.Host !~ "^m\.") {
                 if (
                   req.http.User-Agent ~ "iPad" ||
                   req.http.User-Agent ~ "iP(hone|od)" ||
                   req.http.User-Agent ~ "Android" ||
                   req.http.User-Agent ~ "SymbianOS" ||
                   req.http.User-Agent ~ "^BlackBerry" ||
                   req.http.User-Agent ~ "^SonyEricsson" ||
                   req.http.User-Agent ~ "^Nokia" ||
                   req.http.User-Agent ~ "^SAMSUNG" ||
                   req.http.User-Agent ~ "^LG") {
                         # Varnish can't really do redirects, so we will 
error out
                         # and have the error handler do the redirect
                         error 305 req.http.host;
                 }
         }
}

vcl_error() {
         if (obj.status == 305){
                 set obj.http.Set-Cookie = "view_type=mobile; expires: 
Session; path=/; domain=." obj.response;
                 set obj.http.Location = "http://m." obj.response;
                 set obj.status = 302;
                 return(deliver);
         }
         if (obj.status == 306) {
                 set obj.http.Set-Cookie = "view_type=desktop; expires: 
Session; path=/; domain=." obj.response;
                 set obj.http.Location = "http://" obj.response;
                 set obj.status = 302;
         }

}



From travis.crowder at spechal.com  Sat Nov 19 05:35:52 2011
From: travis.crowder at spechal.com (Travis Crowder)
Date: Fri, 18 Nov 2011 23:35:52 -0600
Subject: Can Varnish Set Cookies?
In-Reply-To: <4EC73F9E.1040203@spechal.com>
References: <4EC6C1B8.8030508@spechal.com>
	<AB0CF49818D2624A82954E14751BE3D42464504B@cmhvm005>
	<4EC73F9E.1040203@spechal.com>
Message-ID: <4EC74038.8060206@spechal.com>

Sorry, vcl_fetch() should be vcl_recv() ... I copy/pasted only the 
relevant parts and typed out the scopes. :\

On 11/18/2011 11:33 PM, Travis Crowder wrote:
> On 11/18/2011 2:55 PM, Jim Hayter wrote:
>> Travis wrote:
>> >  Sorry if this is on the ineternet somewhere, but I have yet to 
>> find it.
>>
>> >  Can Varnish set a cookie?
>>
>>> In example, if a user on a mobile device goes to xyz.com, Varnish 
>>> sees a
>>> mobile user agent and redirects them to m.xyz.com via the vcl_error
>>> section.  Before or after the redirect, can Varnish set a cookie 
>>> without
>>> the application doing so?
>>> Thanks,
>>> Travis
>> I'm running varnish 2.0.5 but would not expect the syntax to have 
>> changed much.
>> The obj part may no longer be what is used, I believe the current 
>> versions use beresp.
>>
>> I use the code below in vcl_fetch.
>>     If (test for setting cookie) {
>>     set obj.http.Set-Cookie = "use_mobile_dev=yes; 
>> domain=.<xxxxxxxx>.com; path=/";
>>     }
>>
>> This works for the limited case where I need it.  I don't remember 
>> the impact if any
>> Set-Cookie headers already exist.
>>
>> Jim
>
> Thank you!
>
> I am using this for mobile detection in case it can help anyone else.  
> It checks for either device=mobile or device=desktop in the query 
> string.  It works with = or / to be diverse in case my developers 
> decide to change it to use device/desktop or device/mobile in the future.
>
> vcl_fetch(){
>         # if they are accessing device/desktop, pass
>         if(req.url ~ "device(=|/)desktop") {
>                 if(req.http.Cookie ~ "view_type=mobile"){
>                         error 306 req.http.Host;
>                 }
>                 return(pass);
>         } elseif (req.url ~ "device(=|/)mobile" && req.http.Cookie ~ 
> "view_type=desktop") {
>                 error 305 req.http.host;
>         }
>         if(req.http.Cookie !~ "view_type" && req.http.Host !~ "^m\.") {
>                 if (
>                   req.http.User-Agent ~ "iPad" ||
>                   req.http.User-Agent ~ "iP(hone|od)" ||
>                   req.http.User-Agent ~ "Android" ||
>                   req.http.User-Agent ~ "SymbianOS" ||
>                   req.http.User-Agent ~ "^BlackBerry" ||
>                   req.http.User-Agent ~ "^SonyEricsson" ||
>                   req.http.User-Agent ~ "^Nokia" ||
>                   req.http.User-Agent ~ "^SAMSUNG" ||
>                   req.http.User-Agent ~ "^LG") {
>                         # Varnish can't really do redirects, so we 
> will error out
>                         # and have the error handler do the redirect
>                         error 305 req.http.host;
>                 }
>         }
> }
>
> vcl_error() {
>         if (obj.status == 305){
>                 set obj.http.Set-Cookie = "view_type=mobile; expires: 
> Session; path=/; domain=." obj.response;
>                 set obj.http.Location = "http://m." obj.response;
>                 set obj.status = 302;
>                 return(deliver);
>         }
>         if (obj.status == 306) {
>                 set obj.http.Set-Cookie = "view_type=desktop; expires: 
> Session; path=/; domain=." obj.response;
>                 set obj.http.Location = "http://" obj.response;
>                 set obj.status = 302;
>         }
>
> }
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From ruben at varnish-software.com  Sat Nov 19 11:33:54 2011
From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=)
Date: Sat, 19 Nov 2011 12:33:54 +0100
Subject: Can Varnish Set Cookies?
In-Reply-To: <AB0CF49818D2624A82954E14751BE3D42464504B@cmhvm005>
References: <4EC6C1B8.8030508@spechal.com>
	<AB0CF49818D2624A82954E14751BE3D42464504B@cmhvm005>
Message-ID: <CAEfnkDaf5n67s1Z0LXngdH6fCGDMnU31ZSiJGvUvEjz0A0Zw_A@mail.gmail.com>

On 11/18/11, Jim Hayter <jhayter at manta.com> wrote:
>
> I'm running varnish 2.0.5 but would not expect the syntax to have changed
> much.
>

* Varnish 2.0.x is no longer supported. Upgrade to the latest stable
release is always recommended:
 https://www.varnish-cache.org/releases/

* There have been at least two major syntax change in Varnish that
entered 2.1 and also there are some changes in 3.0.
2.0 to 2.1 changes: http://www.dmsp.de/static/varnish/changes-2.1.0.html
2.1 to 3.0 changes:
https://www.varnish-cache.org/trac/browser/doc/changes.rst?rev=7b841df818ec169bd7b176e2fd36acbc3cbfb537


> Jim


-- 
Rub?n Romero, Varnish Software
Phone: +47 21 98 92 62 / Mobile: +47 959 64 088 / Skype: ruben_varnish /
GTalk:  <http://goo.gl/c4D6e>
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI>
 | Twitter <https://twitter.com/varnishsoftware> |
LinkedIn<http://www.linkedin.com/company/varnish-software>



From ivan.martinez at masterion.com  Sun Nov 20 18:05:55 2011
From: ivan.martinez at masterion.com (Ivan Martinez)
Date: Sun, 20 Nov 2011 19:05:55 +0100
Subject: Varnish not in port 80 when starting as service
Message-ID: <82fcd8a1dd8c2541209d9ccd0ee4dfd3@mail.masterion.com>

Hello all,
I just installed Varnish 3.0.2 in a CentOS 6 system. When I manually 
start Varnish as follows:

[root at masterion init.d]# varnishd -f /etc/varnish/default.vcl

Varnish is up and running in port 80. However, when I do it as a 
service:

[root at masterion init.d]# pkill varnishd
[root at masterion init.d]# /etc/init.d/varnish start
Starting Varnish Cache:                                    [  OK  ]
[root at masterion init.d]# /etc/init.d/varnish status
varnishd (pid  10648) is running...
[root at masterion init.d]#

Apparently Varnish is running but there is nothing in port 80. What am 
I doing wrong?.
Thank you,
Ivan



From jogalara at gmail.com  Sun Nov 20 18:10:19 2011
From: jogalara at gmail.com (Jo Galara)
Date: Sun, 20 Nov 2011 19:10:19 +0100
Subject: Caching/Headers question
Message-ID: <4EC9428B.8080909@gmail.com>

Hi!

I got the following setup (on a Debian Squeeze 64-bit machine): Internet
<-> Varnish <-> Apache

What I want to achieve is that Varnish forwards all requests from the
Internet to the Apache backend (if its not in the cache), uses the
headers from the backend to choose whether or not and how long to cache
the file and then send the file to the browser with the _same_ headers
it got from the from the backend.

How do I achieve that?

-- 
Regards,

Jo Galara


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111120/5bea8844/attachment.bin>

From straightflush at gmail.com  Sun Nov 20 20:32:22 2011
From: straightflush at gmail.com (AD)
Date: Sun, 20 Nov 2011 15:32:22 -0500
Subject: Varnish not in port 80 when starting as service
In-Reply-To: <82fcd8a1dd8c2541209d9ccd0ee4dfd3@mail.masterion.com>
References: <82fcd8a1dd8c2541209d9ccd0ee4dfd3@mail.masterion.com>
Message-ID: <CANZWjw7i4Qm+BmKDvTVtK2tTdSFDx+kZZNhwtDj5AVaA7h9Zew@mail.gmail.com>

is there a parameter VARNISH_LISTEN_PORT in your startup script ?  This is
in /etc/default/varnish on Ubunut, not sure what it would be on CentOS



On Sun, Nov 20, 2011 at 1:05 PM, Ivan Martinez
<ivan.martinez at masterion.com>wrote:

> Hello all,
> I just installed Varnish 3.0.2 in a CentOS 6 system. When I manually start
> Varnish as follows:
>
> [root at masterion init.d]# varnishd -f /etc/varnish/default.vcl
>
> Varnish is up and running in port 80. However, when I do it as a service:
>
> [root at masterion init.d]# pkill varnishd
> [root at masterion init.d]# /etc/init.d/varnish start
> Starting Varnish Cache:                                    [  OK  ]
> [root at masterion init.d]# /etc/init.d/varnish status
> varnishd (pid  10648) is running...
> [root at masterion init.d]#
>
> Apparently Varnish is running but there is nothing in port 80. What am I
> doing wrong?.
> Thank you,
> Ivan
>
> ______________________________**_________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc<https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111120/b0c053a2/attachment.html>

From ivan.martinez at masterion.com  Sun Nov 20 21:53:46 2011
From: ivan.martinez at masterion.com (Ivan Martinez)
Date: Sun, 20 Nov 2011 22:53:46 +0100
Subject: Varnish not in port 80 when starting as service
In-Reply-To: <CANZWjw7i4Qm+BmKDvTVtK2tTdSFDx+kZZNhwtDj5AVaA7h9Zew@mail.gmail.com>
References: <82fcd8a1dd8c2541209d9ccd0ee4dfd3@mail.masterion.com>
	<CANZWjw7i4Qm+BmKDvTVtK2tTdSFDx+kZZNhwtDj5AVaA7h9Zew@mail.gmail.com>
Message-ID: <98423140bd66a273712c4d14089029e1@mail.masterion.com>

  

Oh, I see, the parameter is in "/etc/sysconfig/varnish". That
configuration file is only read when Varnish is run as a system service,
that's what makes the difference. Thank you. 

Ivan 

On Sun, 20 Nov
2011 15:32:22 -0500, AD wrote: 

> is there a parameter
VARNISH_LISTEN_PORT in your startup script ? This is in
/etc/default/varnish on Ubunut, not sure what it would be on CentOS 
>

> On Sun, Nov 20, 2011 at 1:05 PM, Ivan Martinez wrote:
> 
>> Hello
all,
>> I just installed Varnish 3.0.2 in a CentOS 6 system. When I
manually start Varnish as follows:
>> 
>> [root at masterion init.d]#
varnishd -f /etc/varnish/default.vcl
>> 
>> Varnish is up and running in
port 80. However, when I do it as a service:
>> 
>> [root at masterion
init.d]# pkill varnishd
>> [root at masterion init.d]# /etc/init.d/varnish
start
>> Starting Varnish Cache: [ OK ]
>> [root at masterion init.d]#
/etc/init.d/varnish status
>> varnishd (pid 10648) is running...
>>
[root at masterion init.d]#
>> 
>> Apparently Varnish is running but there
is nothing in port 80. What am I doing wrong?.
>> Thank you,
>> Ivan
>>

>> _______________________________________________
>> varnish-misc
mailing list
>> varnish-misc at varnish-cache.org [1]
>>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc [2]

 


Links:
------
[1] mailto:varnish-misc at varnish-cache.org
[2]
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
[3]
mailto:ivan.martinez at masterion.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111120/52b8b46c/attachment.html>

From hugo.cisneiros at gmail.com  Mon Nov 21 14:54:44 2011
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Mon, 21 Nov 2011 12:54:44 -0200
Subject: Caching/Headers question
In-Reply-To: <4EC9428B.8080909@gmail.com>
References: <4EC9428B.8080909@gmail.com>
Message-ID: <CA+KACL=5v7t_Qt9foVhwwFTRNKkAgAfg6H=cyZoM8_C+CiWBTQ@mail.gmail.com>

On Sun, Nov 20, 2011 at 4:10 PM, Jo Galara <jogalara at gmail.com> wrote:

> I got the following setup (on a Debian Squeeze 64-bit machine): Internet
> <-> Varnish <-> Apache
>
> What I want to achieve is that Varnish forwards all requests from the
> Internet to the Apache backend (if its not in the cache), uses the
> headers from the backend to choose whether or not and how long to cache
> the file and then send the file to the browser with the _same_ headers
> it got from the from the backend.
>

In vcl_fetch you have all the headers available with beresp.http.<header>.
You can set the TTL based on a header. For example:

if (beresp.http.X-Cache-Time) {
  beresp.ttl = beresp.http.X-Cache-Time;
  return (deliver);
} else
  # default cache time - 30 minutes
  beresp.ttl = 30m;
}

This way, backend must return the X-Cache-Time with something like "30s",
"1m" or anything varnish understands (no cache at all would be "0s").

If you don't unset the header on vcl_deliver, varnish will deliver the
exact headers received on beresp.

-- 
[]'s
Hugo
www.devin.com.br
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111121/e8dc15f7/attachment.html>

From jogalara at gmail.com  Mon Nov 21 18:04:00 2011
From: jogalara at gmail.com (Jo Galara)
Date: Mon, 21 Nov 2011 19:04:00 +0100
Subject: Caching/Headers question
In-Reply-To: <CA+KACL=5v7t_Qt9foVhwwFTRNKkAgAfg6H=cyZoM8_C+CiWBTQ@mail.gmail.com>
References: <4EC9428B.8080909@gmail.com>
	<CA+KACL=5v7t_Qt9foVhwwFTRNKkAgAfg6H=cyZoM8_C+CiWBTQ@mail.gmail.com>
Message-ID: <4ECA9290.5050106@gmail.com>

By the way: Is there a way to "preload" files which ran out of TTL?

Let's say I download file example.bin which is cached for 24h (Apache
backend sends Expires header) and download it the next time 48h later.
Varnish would have to redownload the file once a Client (Browser)
requests that file again, right? Is there a way so Varnish automatically
downloads files from the Apache backend as soon as their files Expire?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111121/b47aee77/attachment.bin>

From hugo.cisneiros at gmail.com  Mon Nov 21 18:11:22 2011
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Mon, 21 Nov 2011 16:11:22 -0200
Subject: Caching/Headers question
In-Reply-To: <4ECA9290.5050106@gmail.com>
References: <4EC9428B.8080909@gmail.com>
	<CA+KACL=5v7t_Qt9foVhwwFTRNKkAgAfg6H=cyZoM8_C+CiWBTQ@mail.gmail.com>
	<4ECA9290.5050106@gmail.com>
Message-ID: <CA+KACLnOx91ef+R5wuJTnvVEVnqpyZOqFQERKdoPg9ycsazQPg@mail.gmail.com>

On Mon, Nov 21, 2011 at 4:04 PM, Jo Galara <jogalara at gmail.com> wrote:
>
> By the way: Is there a way to "preload" files which ran out of TTL?
>
> Let's say I download file example.bin which is cached for 24h (Apache
> backend sends Expires header) and download it the next time 48h later.
> Varnish would have to redownload the file once a Client (Browser)
> requests that file again, right? Is there a way so Varnish automatically
> downloads files from the Apache backend as soon as their files Expire?

Not that I know. Usually we create a text file with the URLs and
access them with a script to put in the cache. This is called 'warming
the cache'. Can be used to refresh the cache too, but it's complicated
since you have to know each ttl time for the URL.

Other way is to cache like forever and let the back-end notify varnish
to ban the object from the cache. It's a very efficient method for not
having varnish download the file from back-end unnecessarily :)

--
[]'s
Hugo
www.devin.com.br



From jeroen.ooms at stat.ucla.edu  Tue Nov 22 23:12:31 2011
From: jeroen.ooms at stat.ucla.edu (Jeroen Ooms)
Date: Tue, 22 Nov 2011 15:12:31 -0800
Subject: varnish + wp + ec2 micro
Message-ID: <CABFfbXs+kM58WqMh68oWOp1-m40vS5CZ4sb8Cc59nM9nz5AqxA@mail.gmail.com>

I am looking for the best solution to host a wordpress site as cheap as
possible with a reasonable performance. The good thing is that the site is
only updated once in a while and is not very large.

I was thinking about spinning up an EC2 micro instance with ubuntu server,
installing mysql, apache2, php, wordpress and varnish. Does anyone have
experiences with a similar setup? EC2 micro instance has only 613M memory,
should this be sufficient to run ubuntu server with apache2, mysql and
varnish?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111122/c5895c57/attachment.html>

From russ at vshift.com  Tue Nov 22 23:55:12 2011
From: russ at vshift.com (Ruslan Sivak)
Date: Tue, 22 Nov 2011 18:55:12 -0500
Subject: varnish + wp + ec2 micro
In-Reply-To: <CABFfbXs+kM58WqMh68oWOp1-m40vS5CZ4sb8Cc59nM9nz5AqxA@mail.gmail.com>
References: <CABFfbXs+kM58WqMh68oWOp1-m40vS5CZ4sb8Cc59nM9nz5AqxA@mail.gmail.com>
Message-ID: <CACXVBpcjhf6rNFT7gmvPOLYi12SS=v+eu1v+h9=yO6tRuA7_Ew@mail.gmail.com>

Personally, I would use nginx.  I've had decent performance in testing of
nginx + php-fpm + drupal + memcache.

With varnish, performance would probably be even better.

Russ

On Nov 22, 2011 6:12 PM, "Jeroen Ooms" <jeroen.ooms at stat.ucla.edu> wrote:

> I am looking for the best solution to host a wordpress site as cheap as
> possible with a reasonable performance. The good thing is that the site is
> only updated once in a while and is not very large.
>
> I was thinking about spinning up an EC2 micro instance with ubuntu server,
> installing mysql, apache2, php, wordpress and varnish. Does anyone have
> experiences with a similar setup? EC2 micro instance has only 613M memory,
> should this be sufficient to run ubuntu server with apache2, mysql and
> varnish?
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111122/1a0bad57/attachment.html>

From j.vanarragon at lukkien.com  Wed Nov 23 11:32:00 2011
From: j.vanarragon at lukkien.com (Jaap van Arragon)
Date: Wed, 23 Nov 2011 12:32:00 +0100
Subject: Return Lookup
Message-ID: <CAF29840.59375%j.vanarragon@lukkien.com>

Hello,

Hopefully a short question:

We have the following code in our VCL file:

sub vcl_recv {
    
    # Purge through http
    if (req.request == "PURGE") {
        if (!client.ip ~ purge) {
            error 405 "Not allowed.";
        }
        ban("req.url =" + req.url + "&& req.http.host =" + req.http.host);
        error 200 "Purged.";
    }

    if (req.http.cookie) {
        unset req.http.cookie;
        return (lookup);
    }
    
    if (req.http.host ~ "example.url.com") {
        set req.backend = live;
    }
    else {
        error 404 "Unkown Virtual host";
    }

When our request ( example.url.com/asdja?adn23 ) comes in we see that the
backend is selected which it first came across. We?ve could pinpoint it to
the unset.req.http.cookie part. If we move the unset cookie part under the
req.http.host ~ "example.url.com" part it all goes fine.

Can it be that the return(lookup); in the unset req.http.cookie part skips
the req.http.host part? And thus the first backend is selected?

Thank you.

Regards,
Jaap 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111123/c3e7703f/attachment.html>

From hugo.cisneiros at gmail.com  Wed Nov 23 11:49:01 2011
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Wed, 23 Nov 2011 09:49:01 -0200
Subject: Return Lookup
In-Reply-To: <CAF29840.59375%j.vanarragon@lukkien.com>
References: <CAF29840.59375%j.vanarragon@lukkien.com>
Message-ID: <CA+KACLnEh5FmCbhLfZ3n8qZ4nBRDbzCBA4rWQrneU6SZPPSnKQ@mail.gmail.com>

On Wed, Nov 23, 2011 at 9:32 AM, Jaap van Arragon
<j.vanarragon at lukkien.com> wrote:
> Hello,
>
> Hopefully a short question:
>
> We have the following code in our VCL file:
>
> sub vcl_recv {
>
> ????# Purge through http
> ????if (req.request == "PURGE") {
> ????????if (!client.ip ~ purge) {
> ????????????error 405 "Not allowed.";
> ????????}
> ????????ban("req.url =" + req.url + "&& req.http.host =" + req.http.host);
> ????????error 200 "Purged.";
> ????}
>
> ????if (req.http.cookie) {
> ????????unset req.http.cookie;
> ????????return (lookup);
> ????}
>
> ????if (req.http.host ~ "example.url.com") {
> ????????set req.backend = live;
> ????}
> ????else {
> ????????error 404 "Unkown Virtual host";
> ????}
>
> When our request ( example.url.com/asdja?adn23 ) comes in we see that the
> backend is selected which it first came across. We?ve could pinpoint it to
> the unset.req.http.cookie part. If we move the unset cookie part under the
> req.http.host ~ "example.url.com" part it all goes fine.
>
> Can it be that the return(lookup); in the unset req.http.cookie part skips
> the req.http.host part? And thus the first backend is selected?

Yes. The 'return(lookup)' after the 'unset req.http.cookie' will skip
all the rest of vcl_fetch. It will lookup in the cache and go to the
vcl_hit or vcl_pass depending if the object is found or not.

You must place return(lookup) only after all required rules are done.
For example, I think this would work:

if (req.http.cookie) {
  if (req.http.host ~ "example.url.com") {
    set req.backend = live;
??}
??else {
????error 404 "Unkown Virtual host";
??}

  unset req.http.cookie;
??return (lookup);
}

Hope it will help.

-- 
[]'s
Hugo
www.devin.com.br



From j.vanarragon at lukkien.com  Wed Nov 23 12:38:28 2011
From: j.vanarragon at lukkien.com (Jaap van Arragon)
Date: Wed, 23 Nov 2011 13:38:28 +0100
Subject: Return Lookup
In-Reply-To: <CA+KACLnEh5FmCbhLfZ3n8qZ4nBRDbzCBA4rWQrneU6SZPPSnKQ@mail.gmail.com>
Message-ID: <CAF2A7D4.59384%j.vanarragon@lukkien.com>

Hello Hugo,

Thank you for your answer.

It was most helpful!!

Regards,
Jaap


On 11/23/11 12:49 PM, "Hugo Cisneiros (Eitch)" <hugo.cisneiros at gmail.com>
wrote:

> On Wed, Nov 23, 2011 at 9:32 AM, Jaap van Arragon
> <j.vanarragon at lukkien.com> wrote:
>> Hello,
>> 
>> Hopefully a short question:
>> 
>> We have the following code in our VCL file:
>> 
>> sub vcl_recv {
>> 
>> ????# Purge through http
>> ????if (req.request == "PURGE") {
>> ????????if (!client.ip ~ purge) {
>> ????????????error 405 "Not allowed.";
>> ????????}
>> ????????ban("req.url =" + req.url + "&& req.http.host =" + req.http.host);
>> ????????error 200 "Purged.";
>> ????}
>> 
>> ????if (req.http.cookie) {
>> ????????unset req.http.cookie;
>> ????????return (lookup);
>> ????}
>> 
>> ????if (req.http.host ~ "example.url.com") {
>> ????????set req.backend = live;
>> ????}
>> ????else {
>> ????????error 404 "Unkown Virtual host";
>> ????}
>> 
>> When our request ( example.url.com/asdja?adn23 ) comes in we see that the
>> backend is selected which it first came across. We?ve could pinpoint it to
>> the unset.req.http.cookie part. If we move the unset cookie part under the
>> req.http.host ~ "example.url.com" part it all goes fine.
>> 
>> Can it be that the return(lookup); in the unset req.http.cookie part skips
>> the req.http.host part? And thus the first backend is selected?
> 
> Yes. The 'return(lookup)' after the 'unset req.http.cookie' will skip
> all the rest of vcl_fetch. It will lookup in the cache and go to the
> vcl_hit or vcl_pass depending if the object is found or not.
> 
> You must place return(lookup) only after all required rules are done.
> For example, I think this would work:
> 
> if (req.http.cookie) {
>   if (req.http.host ~ "example.url.com") {
>     set req.backend = live;
> ??}
> ??else {
> ????error 404 "Unkown Virtual host";
> ??}
> 
>   unset req.http.cookie;
> ??return (lookup);
> }
> 
> Hope it will help.





From jogalara at gmail.com  Wed Nov 23 18:17:41 2011
From: jogalara at gmail.com (Jo Galara)
Date: Wed, 23 Nov 2011 19:17:41 +0100
Subject: Caching/Headers question
In-Reply-To: <CA+KACLnOx91ef+R5wuJTnvVEVnqpyZOqFQERKdoPg9ycsazQPg@mail.gmail.com>
References: <4EC9428B.8080909@gmail.com>	<CA+KACL=5v7t_Qt9foVhwwFTRNKkAgAfg6H=cyZoM8_C+CiWBTQ@mail.gmail.com>	<4ECA9290.5050106@gmail.com>
	<CA+KACLnOx91ef+R5wuJTnvVEVnqpyZOqFQERKdoPg9ycsazQPg@mail.gmail.com>
Message-ID: <4ECD38C5.4090503@gmail.com>

On 11/21/2011 07:11 PM, Hugo Cisneiros (Eitch) wrote:

> Other way is to cache like forever and let the back-end notify varnish
> to ban the object from the cache. It's a very efficient method for not
> having varnish download the file from back-end unnecessarily :)

How can I do that?

-- 
Regards,

Jo Galara


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111123/958e5b77/attachment.bin>

From hugo.cisneiros at gmail.com  Wed Nov 23 18:31:07 2011
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Wed, 23 Nov 2011 16:31:07 -0200
Subject: Caching/Headers question
In-Reply-To: <CA+KACLnB28FV6tRyycbCg7ZmYWQMptKcEFfFs2cymqCdOk5-qg@mail.gmail.com>
References: <4EC9428B.8080909@gmail.com>
	<CA+KACL=5v7t_Qt9foVhwwFTRNKkAgAfg6H=cyZoM8_C+CiWBTQ@mail.gmail.com>
	<4ECA9290.5050106@gmail.com>
	<CA+KACLnOx91ef+R5wuJTnvVEVnqpyZOqFQERKdoPg9ycsazQPg@mail.gmail.com>
	<4ECD38C5.4090503@gmail.com>
	<CA+KACLnB28FV6tRyycbCg7ZmYWQMptKcEFfFs2cymqCdOk5-qg@mail.gmail.com>
Message-ID: <CA+KACLnb2jhpZ_r3M4cjhgWxFe+v0Mj6+Qgwx6gfvviqRHGU+A@mail.gmail.com>

On Wed, Nov 23, 2011 at 4:17 PM, Jo Galara <jogalara at gmail.com> wrote:
> On 11/21/2011 07:11 PM, Hugo Cisneiros (Eitch) wrote:
>
>> Other way is to cache like forever and let the back-end notify varnish
>> to ban the object from the cache. It's a very efficient method for not
>> having varnish download the file from back-end unnecessarily :)
>
> How can I do that?

You can begin reading this:

https://www.varnish-cache.org/docs/trunk/tutorial/purging.html

In my case the ban list works very well.

--
[]'s
Hugo
www.devin.com.br



From rob at bigfish.co.uk  Wed Nov 23 18:37:43 2011
From: rob at bigfish.co.uk (Rob Miller)
Date: Wed, 23 Nov 2011 18:37:43 +0000
Subject: Caching/Headers question
In-Reply-To: <4ECD38C5.4090503@gmail.com>
References: <4EC9428B.8080909@gmail.com>	<CA+KACL=5v7t_Qt9foVhwwFTRNKkAgAfg6H=cyZoM8_C+CiWBTQ@mail.gmail.com>	<4ECA9290.5050106@gmail.com>
	<CA+KACLnOx91ef+R5wuJTnvVEVnqpyZOqFQERKdoPg9ycsazQPg@mail.gmail.com>
	<4ECD38C5.4090503@gmail.com>
Message-ID: <0505B629-1EA6-45C6-A60F-0BFC30E25FA0@bigfish.co.uk>

On 23 Nov 2011, at 18:17, Jo Galara wrote:

> On 11/21/2011 07:11 PM, Hugo Cisneiros (Eitch) wrote:
> 
>> Other way is to cache like forever and let the back-end notify varnish
>> to ban the object from the cache. It's a very efficient method for not
>> having varnish download the file from back-end unnecessarily :)
> 
> How can I do that?



When faced with the same issue, I wrote a Ruby script that includes functionality for purging and refreshing an entire domain:

https://github.com/robmiller/Varnish-Toolkit

It supports things like purging a page along with all its external assets, purging a whole domain, and purging and then spidering a whole domain. 

We often use it when we make a change to, say, the header or footer of a website that has a very slow backend (as some of our Magento ecommerce sites do); that way, our spider gets all of the slow requests and our end users get none.

Using this, we don't generally worry about TTLs; we cache everything for a really long time, then purge when we know we need to (either automatically, when a new piece of content goes live, or manually when we make a change to a stylesheet or template file).

You probably shouldn't use it unless you write lurker-friendly bans,[1] though.

Rob

[1]: http://kristianlyng.wordpress.com/2010/07/28/smart-bans-with-varnish/

--

Rob Miller
Big Fish? | 11 Chelsea Wharf | 15 Lots Road | London | SW10 0QJ
 
Office number: +44 (0)20 7795 0075
Direct dial: +44 (0)20 7376 6799
 
e-mail rob at bigfish.co.uk
http://www.bigfish.co.uk

If you're not a stalker, why not follow us on Twitter? @bigfishlondon




From mark at silverstatetech.com  Wed Nov 23 19:14:47 2011
From: mark at silverstatetech.com (Mark Gavin)
Date: Wed, 23 Nov 2011 19:14:47 +0000
Subject: Cannot Start Varnish with Multiple Backends
Message-ID: <62518C333241F34CA6A0247A847DC1A416470BC1@MBX024-W1-CA-7.EXCH024.DOMAIN.LOCAL>

I am having so many problems getting started with Varnish to complete my two-part project. Part one is to configure Varnish as a load balancer, part two is to setup caching, but this mail is for part one only (getting Varnish running as a load balancer). Here is what I have

Dedicated Varnish Server
Dual Xeon processors, 32GB RAM, Red Hat EL6 64-bit, Varnish 2.1.5 installed from Yum using RHEL6 repository

Two Dedicated Web Servers
Dual Xeon processors, 16GB RAM, Red Hat EL5 64-bit, Apache 2.2.3 installed with Yum

Network

All servers behind hardware firewall, all servers communicate over internal subnet, all servers can ping each other, all servers have OS firewalls turned off, no restrictions between servers over the LAN.



I installed Varnish using the Red Hat EPEL repository with "yum install varnish", made a copy of the default VCL file, renamed the default file to forum.vcl and entered the following (note everything above and below this section is default and commented out from the original default file)



backend server9 {

  .host = "192.168.1.118";

  .port = "80";

  .probe = {

                .url = "/";

                .interval = 5s;

               .timeout = 1 s;

                .window = 5;

                .threshold = 3;

  }

}



backend server10 {

  .host = "192.168.1.5";

  .port = "80";

  .probe = {

                .url = "/";

                .interval = 5s;

                .timeout = 1 s;

                .window = 5;

                .threshold = 3;

  }

}



director baz round-robin {

        {

                .backend = server9;

        }

        {

                .backend = server10;

        }

}



sub vcl_recv {

   if (req.http.host ~ "^(www.)?I put my domain name here$") {

       set req.backend = baz;

   }

}



I then executed the following commands

pkill varnish

varnishd -f /etc/varnish/forum.vcl -s malloc,10G -a 0.0.0.0:80

service varnish start



Alternate commands for above

pkill varnish

varnishd -a :80 -b 192.168.1.117:80 -b 192.168.1.5:80

service varnish start



and receive "Starting varnish HTTP accelerator: [FAILED]"



What I have tried

1.       Commenting out each of the backends (one at a time)

2.       Commenting out the   if (req.http.host ~ "^(www.)?I put my domain name here$") {

3.       Removing and reinstalling

4.       Starting Varnish with the default backend (that works, so I assume Varnish is installed properly)



Basically it seems like when I define a backend that is another server, varnish will not start. I have read through many pages of documentation examples and I do not see where I am going wrong with this setup or I missed something. I am so frustrated at this point and would be so very grateful for any help you can offer.



Thank you,



Mark





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111123/7db90c5e/attachment.html>

From hugo.cisneiros at gmail.com  Wed Nov 23 19:20:16 2011
From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch))
Date: Wed, 23 Nov 2011 17:20:16 -0200
Subject: Cannot Start Varnish with Multiple Backends
In-Reply-To: <62518C333241F34CA6A0247A847DC1A416470BC1@MBX024-W1-CA-7.EXCH024.DOMAIN.LOCAL>
References: <62518C333241F34CA6A0247A847DC1A416470BC1@MBX024-W1-CA-7.EXCH024.DOMAIN.LOCAL>
Message-ID: <CA+KACLnpxHX5bDEZfp=V7svphdX1fXsNiRPLkUgdGFmm0d4KgQ@mail.gmail.com>

On Wed, Nov 23, 2011 at 5:14 PM, Mark Gavin <mark at silverstatetech.com> wrote:
> I am having so many problems getting started with Varnish to complete my
> two-part project. Part one is to configure Varnish as a load balancer, part
> two is to setup caching, but this mail is for part one only (getting Varnish
> running as a load balancer). Here is what I have
[...]

> I then executed the following commands
>
> pkill varnish
>
> varnishd -f /etc/varnish/forum.vcl -s malloc,10G -a 0.0.0.0:80
>
> service varnish start

I don't know if this is it, but if you execute varnishd before service
varnish start, the service start will fail as varnishd is already
binded to ip/port and is already running.

The proper way on RHEL/CentOS is to edit /etc/sysconfig/varnish for
daemon options, and use service varnish start *only*.

On Debian/Ubuntu, it's /etc/default/varnish.

I don't see any errors on your VCL, and if you have any errors, it'll
show if you execute varnishd manually (instead of using service).

Hope it helps.

-- 
[]'s
Hugo
www.devin.com.br



From rob at bigfish.co.uk  Wed Nov 23 19:20:31 2011
From: rob at bigfish.co.uk (Rob Miller)
Date: Wed, 23 Nov 2011 19:20:31 +0000
Subject: Cannot Start Varnish with Multiple Backends
In-Reply-To: <62518C333241F34CA6A0247A847DC1A416470BC1@MBX024-W1-CA-7.EXCH024.DOMAIN.LOCAL>
References: <62518C333241F34CA6A0247A847DC1A416470BC1@MBX024-W1-CA-7.EXCH024.DOMAIN.LOCAL>
Message-ID: <01916B52-D6E0-4201-88B4-11996A0DA616@bigfish.co.uk>


On 23 Nov 2011, at 19:14, Mark Gavin wrote:

> 
> Basically it seems like when I define a backend that is another server, varnish will not start. I have read through many pages of documentation examples and I do not see where I am going wrong with this setup or I missed something. I am so frustrated at this point and would be so very grateful for any help you can offer.

You need to set a default backend, in case your `if` doesn't match:

# Default backend for all hosts
set req.backend = baz;

if ( req.http.host == "example.com" ) {
	# Backend for just this host
	set req.backend = server10;
}

(Presumably you're going to actually switch between two backends ? or else your `if` isn't necessary!)

Rob


--

Rob Miller
Big Fish? | 11 Chelsea Wharf | 15 Lots Road | London | SW10 0QJ
 
Office number: +44 (0)20 7795 0075
Direct dial: +44 (0)20 7376 6799
 
e-mail rob at bigfish.co.uk
http://www.bigfish.co.uk

If you're not a stalker, why not follow us on Twitter? @bigfishlondon




From martin at 925.dk  Wed Nov 23 19:21:38 2011
From: martin at 925.dk (Martin Koch Andersen)
Date: Wed, 23 Nov 2011 20:21:38 +0100
Subject: file,/var/tmp,50% - but no file in /var/tmp
Message-ID: <B9D3B3DF-62BB-4FE9-9533-EA3C38A67163@925.dk>

Hello,

I use this -s parameter "file,/var/tmp,50%" for Vanish.

With Varnish 3.0.1 a (memory mapped) file was thus created in /var/tmp, when Varnish was started.
With Varnish 3.0.2 no such file is created. And I don't see Varnish having any open file descriptors to anything in /var/tmp.

I use FreeBSD, and the -s parameter is passed using "varnishd_storage" from rc.conf.

Wonder if the parameter is being ignored, or if something has changed with regards to the file creation?

Kind regards,

-- 
Martin Koch Andersen
http://925.dk


From jogalara at gmail.com  Wed Nov 23 21:30:03 2011
From: jogalara at gmail.com (Jo Galara)
Date: Wed, 23 Nov 2011 22:30:03 +0100
Subject: Backend Keep-Alive
Message-ID: <4ECD65DB.6010104@gmail.com>

I'm using Apache2 as backend. Should I enable Keep-Alives in Apache? Are
the default timeouts ok or should I change them?

-- 
Regards,

Jo Galara


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111123/384b4498/attachment.bin>

From traian.bratucu at eea.europa.eu  Thu Nov 24 08:09:35 2011
From: traian.bratucu at eea.europa.eu (Traian Bratucu)
Date: Thu, 24 Nov 2011 09:09:35 +0100
Subject: Backend Keep-Alive
In-Reply-To: <4ECD65DB.6010104@gmail.com>
References: <4ECD65DB.6010104@gmail.com>
Message-ID: <E7097BFB283E614C96AE10D5307726E301E0DE3685A4@SLOTHMAIL.eea.eu.int>

> -----Original Message-----
> From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-
> bounces at varnish-cache.org] On Behalf Of Jo Galara
> Sent: Wednesday, November 23, 2011 10:30 PM
> To: varnish-misc at varnish-cache.org
> Subject: Backend Keep-Alive
> 
> I'm using Apache2 as backend. Should I enable Keep-Alives in Apache? Are
> the default timeouts ok or should I change them?
> 
> --
> Regards,
> 
> Jo Galara
> 

If Varnish is on the same machine or on the same LAN, I would say to disable apache keep-alive since there is no real benefit to it. 
Regarding the timeouts, if you are referring to Apache, then probably yes, in Varnish you may want to increase the send timeout variable (/etc/sysconfig/varnish or /etc/default/varnish) since it can mess with large files.

Traian

From r at roze.lv  Thu Nov 24 16:34:22 2011
From: r at roze.lv (Reinis Rozitis)
Date: Thu, 24 Nov 2011 18:34:22 +0200
Subject: Incomplete files in 3.0.x
Message-ID: <1F0FF7738A574F299EC53CB1161CB383@DD21>

Hello,
I have been using 2.1.5 for quite some time in a heavy environment and everything works as expected.

Now I'm trying to upgrade to 3.0.2 (just because of persistent storage possibility on a ssd disk) but in my tests I experience 
strange behaviour of Varnish - as it randomly caches incomplete image files - which display like half size on browsers.

It does happen on randomly pattern - when the storage file is deleted or in malloc case just restarted out of 40-50 requests I will 
sure get one file which is corrupted, but it doesn't reproduce on the same files.
So I have no idea where to start digging or am I doing something wrong - as in the config/startup options are not valid anymore for 
3.0.x ?




The config and startup is pretty simple and taken out of 2.1.x instances which have been working fine for a year with the minor 
changes of beresp.cachable replaced with beresp.ttl and hash_data


varnishd -f default.vcl -s persistent,/tmp/varnish,10m -h classic,500009 -p connect_timeout=60 -p thread_pools=8 -p 
thread_pool_min=100 -p thread_pool_max=5000 -thread_pool_add_delay=2 -p cli_timeout=50 -p session_linger=120 -p lru_interval=20

(-s persistent,/tmp/varnish,10m / -s malloc,10m - are just for testing purposes the original 2.1 instances run with 64Gb malloc)




backend default {
        .host = "10.0.10.1";
        .port = "8080";
        .connect_timeout = 60s;
        .first_byte_timeout = 60s;
        .between_bytes_timeout = 60s;
}

sub vcl_recv {

   set req.grace = 30s;

   unset req.http.Cookie;
   unset req.http.Accept-Encoding;
   unset req.http.X-Forwarded-For;
   unset req.http.Cache-Control;

   set req.url = regsub(req.url, "\?.*", "");

}

sub vcl_hash {
    hash_data(req.url);
    return (hash);
}

sub vcl_fetch {

    remove beresp.http.Server;
    set beresp.grace = 30s;

    if (beresp.ttl <= 0s) {
        return (hit_for_pass);
    }

    if (beresp.ttl < 5m) {
             set beresp.ttl = 5m;
    }

    return (deliver);
}

sub vcl_deliver {
        remove resp.http.Via;

        if (obj.hits > 0) {
                set resp.http.X-Cache = "HIT";
        }

        return (deliver);
}

sub vcl_error {
        if (obj.status == 503 && req.restarts < 5) {
                return (restart);
        }
}



wbr
rr 




From me at ibotty.net  Thu Nov 24 18:59:35 2011
From: me at ibotty.net (me at ibotty.net)
Date: Thu, 24 Nov 2011 19:59:35 +0100
Subject: warmup with curl not working
Message-ID: <4ECE9417.1080102@ibotty.net>

hi,

i am pretty new to varnish but managed to get it to mostly work in a
pretty short timeframe. i am amazed.

the only problem so far is, that my warmup script is not working.

i warm the cache with
 curl -sSH 'X-Refresh: DOIT' localhost:80/test/

and when accessed again with
 curl localhost:80/test/
it gets the cached copy.

but with firefox, it does not hit the cache the first time (it works the
next time).

a shortened example of my vcl follows (ask, if there is anything
missing). i added return (lookup) after req.hash_always_miss=true
because it did always go to cached copy when i did not return at this point.
if this is not how one should build a vcl please enlighten me .

sub vcl_recv {

  # force a cache refresh for the warmup- and regenerate-script
  # e.g.: curl -H 'X-Refresh: DOIT' $page
  # https://www.varnish-cache.org/trac/wiki/VCLExampleHashAlwaysMiss
  if (req.http.X-REFRESH) {
    set req.hash_always_miss = true;
    return (lookup);
  }

  if (req.url ~ "^test") {
    unset req.http.cookie;
    set req.grace = 12h;
    return (lookup);
  }
}

sub vcl_fetch {
  set beresp.grace = 24h; # should be the same as max grace in recv

  if (beresp.http.content-type ~ "text") {
    set beresp.do_gzip = true;
  }

  if (req.url ~ "^test") {
    unset beresp.http.set-cookie;
    set beresp.ttl = 14h;
    return (deliver);
  }
}

sub vcl_hit {
  if (req.request == "PURGE") {
    purge;
    error 200 "Purged";
  }
}

sub vcl_miss {
  if (req.request == "PURGE") {
    error 400 "Not in cache.";
  }
}

the backend sets a vary: accept-encoding header, but it seems not to
make any difference (tested with curl and -H 'Accept-Encoding: gzip,
deflate' and variations).

varnish is 3.0.2-1~1squeeze1 from the debian repo.

thank all of you in advance,
 tobias florek



From cosimo at streppone.it  Thu Nov 24 19:18:55 2011
From: cosimo at streppone.it (Cosimo Streppone)
Date: Thu, 24 Nov 2011 20:18:55 +0100
Subject: warmup with curl not working
In-Reply-To: <4ECE9417.1080102@ibotty.net>
References: <4ECE9417.1080102@ibotty.net>
Message-ID: <op.v5gs9t0hs5ttvb@cn01>

On Thu, 24 Nov 2011 19:59:35 +0100, <me at ibotty.net> wrote:

> the only problem so far is, that my warmup script is not working.
>
> i warm the cache with
>  curl -sSH 'X-Refresh: DOIT' localhost:80/test/
>
> and when accessed again with
>  curl localhost:80/test/
> it gets the cached copy.
>
> but with firefox, it does not hit the cache the first time (it works the
> next time).

Here comes the standard answer :)

1) Restart varnish
2) Run "varnishlog" and keep it running, recording the output
3) Make your request with curl. Repeat it.
4) Make your request with firefox. Repeat it.

Output of "varnishlog" will usually reveal the truth.
If not to you, to someone on this list :)

Cheers,

-- 
Cosimo



From apj at mutt.dk  Thu Nov 24 19:27:35 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Thu, 24 Nov 2011 20:27:35 +0100
Subject: warmup with curl not working
In-Reply-To: <4ECE9417.1080102@ibotty.net>
References: <4ECE9417.1080102@ibotty.net>
Message-ID: <20111124192735.GI3214@nerd.dk>

On Thu, Nov 24, 2011 at 07:59:35PM +0100, me at ibotty.net wrote:
> 
>   if (req.url ~ "^test") {

This won't match. req.url will usually start with a /

So you're hitting the default VCL, and if you have a cookie, that means
a pass.

-- 
Andreas



From me at ibotty.net  Thu Nov 24 19:43:48 2011
From: me at ibotty.net (me at ibotty.net)
Date: Thu, 24 Nov 2011 20:43:48 +0100
Subject: warmup with curl not working
In-Reply-To: <op.v5gs9t0hs5ttvb@cn01>
References: <4ECE9417.1080102@ibotty.net> <op.v5gs9t0hs5ttvb@cn01>
Message-ID: <4ECE9E74.1050605@ibotty.net>

hi,

> Here comes the standard answer :)

thanks for your nice hint ;). i forgot to attach the log output.

the missing / before test was a typo (in my cutting down my file). i get
pretty good hitration (around .9), so this is not the problem. also, i
test for /en/ and not /test/...



    0 CLI          - Rd vcl.load "boot" ./vcl.a_DPMqHA.so
    0 CLI          - Wr 200 36 Loaded "./vcl.a_DPMqHA.so" as "boot"
    0 CLI          - Rd vcl.use "boot"
    0 CLI          - Wr 200 0
    0 CLI          - Rd start
    0 Debug        - "Acceptor is epoll"
    0 CLI          - Wr 200 0
    0 WorkThread   - 0x7f77e4cfeab0 start
    0 WorkThread   - 0x7f77e44fdab0 start
    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1322163341 1.0
   14 BackendOpen  b default 127.0.0.1 52776 127.0.0.1 81
   14 TxRequest    b GET
   14 TxURL        b /en/
   14 TxProtocol   b HTTP/1.1
   14 TxHeader     b User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu)
libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
   14 TxHeader     b Host: localhost
   14 TxHeader     b Accept: */*
   14 TxHeader     b X-Refresh: DOIT
   14 TxHeader     b X-Varnish: 547875415
   14 TxHeader     b Accept-Encoding: gzip
   14 RxProtocol   b HTTP/1.1
   14 RxStatus     b 200
   14 RxResponse   b OK
   14 RxHeader     b Date: Thu, 24 Nov 2011 19:35:38 GMT
   14 RxHeader     b Server: Apache/2.2.16 (Debian)
   14 RxHeader     b X-Powered-By: PHP/5.3.3-7+squeeze3
   14 RxHeader     b Set-Cookie: language=0; path=/
   14 RxHeader     b Set-Cookie: language=1; path=/
   14 RxHeader     b Vary: Accept-Encoding
   14 RxHeader     b Content-Encoding: gzip
   14 RxHeader     b Content-Length: 8842
   14 RxHeader     b Content-Type: text/html; charset=UTF-8
   14 Fetch_Body   b 4(length) cls 0 mklen 1
   14 Length       b 8842
   14 BackendReuse b default
   12 SessionOpen  c 127.0.0.1 33137 :80
   12 ReqStart     c 127.0.0.1 33137 547875415
   12 RxRequest    c GET
   12 RxURL        c /en/
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu)
libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
   12 RxHeader     c Host: localhost
   12 RxHeader     c Accept: */*
   12 RxHeader     c X-Refresh: DOIT
   12 VCL_call     c recv lookup
   12 VCL_call     c hash
   12 Hash         c /en/
   12 Hash         c localhost
   12 VCL_return   c hash
   12 VCL_call     c miss fetch
   12 Backend      c 14 default default
   12 TTL          c 547875415 RFC 120 -1 -1 1322163342 0 1322163338 0 0
   12 VCL_call     c fetch
   12 TTL          c 547875415 VCL 123 86400 -1 1322163339 -3
   12 TTL          c 547875415 VCL 50403 86400 -1 1322163339 -3
   12 VCL_return   c deliver
   12 ObjProtocol  c HTTP/1.1
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Thu, 24 Nov 2011 19:35:38 GMT
   12 ObjHeader    c Server: Apache/2.2.16 (Debian)
   12 ObjHeader    c X-Powered-By: PHP/5.3.3-7+squeeze3
   12 ObjHeader    c Vary: Accept-Encoding
   12 ObjHeader    c Content-Encoding: gzip
   12 ObjHeader    c Content-Type: text/html; charset=UTF-8
   12 Gzip         c u F - 8842 80878 80 80 70670
   12 VCL_call     c deliver deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache/2.2.16 (Debian)
   12 TxHeader     c X-Powered-By: PHP/5.3.3-7+squeeze3
   12 TxHeader     c Vary: Accept-Encoding
   12 TxHeader     c Content-Type: text/html; charset=UTF-8
   12 TxHeader     c Transfer-Encoding: chunked
   12 TxHeader     c Date: Thu, 24 Nov 2011 19:35:42 GMT
   12 TxHeader     c X-Varnish: 547875415
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 Gzip         c U D - 8842 80878 80 80 70670
   12 Length       c 65536
   12 ReqEnd       c 547875415 1322163338.951378584 1322163342.057184219
0.000226021 3.105214834 0.000590801
   12 SessionClose c EOF
   12 StatSess     c 127.0.0.1 33137 3 1 1 0 0 1 287 65536
   12 SessionOpen  c 127.0.0.1 33139 :80
   12 ReqStart     c 127.0.0.1 33139 547875416
   12 RxRequest    c GET
   12 RxURL        c /en/
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu)
libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
   12 RxHeader     c Host: localhost
   12 RxHeader     c Accept: */*
   12 VCL_call     c recv lookup
   12 VCL_call     c hash
   12 Hash         c /en/
   12 Hash         c localhost
   12 VCL_return   c hash
   12 Hit          c 547875415
   12 VCL_call     c hit deliver
   12 VCL_call     c deliver deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache/2.2.16 (Debian)
   12 TxHeader     c X-Powered-By: PHP/5.3.3-7+squeeze3
   12 TxHeader     c Vary: Accept-Encoding
   12 TxHeader     c Content-Type: text/html; charset=UTF-8
   12 TxHeader     c Transfer-Encoding: chunked
   12 TxHeader     c Date: Thu, 24 Nov 2011 19:35:42 GMT
   12 TxHeader     c X-Varnish: 547875416 547875415
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 Gzip         c U D - 8842 80878 80 80 70670
   12 Length       c 65536
   12 ReqEnd       c 547875416 1322163342.062689304 1322163342.063267231
0.000050068 0.000052691 0.000525236
   12 SessionClose c EOF
   12 StatSess     c 127.0.0.1 33139 0 1 1 0 0 0 297 65536
    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1322163344 1.0
   14 TxRequest    b GET
   14 TxURL        b /en/
   14 TxProtocol   b HTTP/1.1
   14 TxHeader     b Host: test.anost.net
   14 TxHeader     b User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0)
Gecko/20100101 Firefox/8.0
   14 TxHeader     b Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
   14 TxHeader     b Accept-Language: en-us,en;q=0.5
   14 TxHeader     b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
   14 TxHeader     b DNT: 1
   14 TxHeader     b Authorization: Basic
ZGV2LXVzZXI6VzRjSDFUSWJZQXJTbEZMZE1YYW8zNXFXdkFhUFcyRE0=
   14 TxHeader     b X-Varnish: 547875417
   14 TxHeader     b Accept-Encoding: gzip
   14 RxProtocol   b HTTP/1.1
   14 RxStatus     b 200
   14 RxResponse   b OK
   14 RxHeader     b Date: Thu, 24 Nov 2011 19:35:42 GMT
   14 RxHeader     b Server: Apache/2.2.16 (Debian)
   14 RxHeader     b X-Powered-By: PHP/5.3.3-7+squeeze3
   14 RxHeader     b Set-Cookie: language=1; path=/
   14 RxHeader     b Set-Cookie: language=1; path=/
   14 RxHeader     b Vary: Accept-Encoding
   14 RxHeader     b Content-Encoding: gzip
   14 RxHeader     b Content-Length: 8800
   14 RxHeader     b Content-Type: text/html; charset=UTF-8
   14 Fetch_Body   b 4(length) cls 0 mklen 1
   14 Length       b 8800
   14 BackendReuse b default
   12 SessionOpen  c 91.65.242.21 60794 :80
   12 ReqStart     c 91.65.242.21 60794 547875417
   12 RxRequest    c GET
   12 RxURL        c /en/
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Host: test.anost.net
   12 RxHeader     c User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0)
Gecko/20100101 Firefox/8.0
   12 RxHeader     c Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
   12 RxHeader     c Accept-Language: en-us,en;q=0.5
   12 RxHeader     c Accept-Encoding: gzip, deflate
   12 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
   12 RxHeader     c DNT: 1
   12 RxHeader     c Connection: keep-alive
   12 RxHeader     c Authorization: Basic
ZGV2LXVzZXI6VzRjSDFUSWJZQXJTbEZMZE1YYW8zNXFXdkFhUFcyRE0=
   12 VCL_call     c recv lookup
   12 VCL_call     c hash
   12 Hash         c /en/
   12 Hash         c test.anost.net
   12 VCL_return   c hash
   12 VCL_call     c miss fetch
   12 Backend      c 14 default default
   12 TTL          c 547875417 RFC 120 -1 -1 1322163346 0 1322163342 0 0
   12 VCL_call     c fetch
   12 TTL          c 547875417 VCL 123 86400 -1 1322163343 -3
   12 TTL          c 547875417 VCL 50403 86400 -1 1322163343 -3
   12 VCL_return   c deliver
   12 ObjProtocol  c HTTP/1.1
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Thu, 24 Nov 2011 19:35:42 GMT
   12 ObjHeader    c Server: Apache/2.2.16 (Debian)
   12 ObjHeader    c X-Powered-By: PHP/5.3.3-7+squeeze3
   12 ObjHeader    c Vary: Accept-Encoding
   12 ObjHeader    c Content-Encoding: gzip
   12 ObjHeader    c Content-Type: text/html; charset=UTF-8
   12 Gzip         c u F - 8800 80869 80 80 70329
   12 VCL_call     c deliver deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache/2.2.16 (Debian)
   12 TxHeader     c X-Powered-By: PHP/5.3.3-7+squeeze3
   12 TxHeader     c Vary: Accept-Encoding
   12 TxHeader     c Content-Encoding: gzip
   12 TxHeader     c Content-Type: text/html; charset=UTF-8
   12 TxHeader     c Content-Length: 8800
   12 TxHeader     c Accept-Ranges: bytes
   12 TxHeader     c Date: Thu, 24 Nov 2011 19:35:46 GMT
   12 TxHeader     c X-Varnish: 547875417
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 Length       c 8800
   12 ReqEnd       c 547875417 1322163342.933949709 1322163346.017340183
0.000137091 3.083226681 0.000163794
   12 Debug        c herding
   12 Interrupted  c ReqStart

[...] from there on other requests (to different paths).


thanks for your help.
 tobias florek



From apj at mutt.dk  Thu Nov 24 19:54:55 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Thu, 24 Nov 2011 20:54:55 +0100
Subject: warmup with curl not working
In-Reply-To: <4ECE9E74.1050605@ibotty.net>
References: <4ECE9417.1080102@ibotty.net> <op.v5gs9t0hs5ttvb@cn01>
	<4ECE9E74.1050605@ibotty.net>
Message-ID: <20111124195455.GJ3214@nerd.dk>

On Thu, Nov 24, 2011 at 08:43:48PM +0100, me at ibotty.net wrote:
> 
> > Here comes the standard answer :)
> 
> thanks for your nice hint ;). i forgot to attach the log output.

   12 RxHeader     c Host: localhost
   12 RxHeader     c Host: test.anost.net

Different hosts, different hash.

https://www.varnish-cache.org/docs/trunk/tutorial/increasing_your_hitrate.html#normalizing-your-namespace

-- 
Andreas



From me at ibotty.net  Thu Nov 24 21:33:25 2011
From: me at ibotty.net (me at ibotty.net)
Date: Thu, 24 Nov 2011 22:33:25 +0100
Subject: warmup with curl not working
In-Reply-To: <20111124195455.GJ3214@nerd.dk>
References: <4ECE9417.1080102@ibotty.net> <op.v5gs9t0hs5ttvb@cn01>
	<4ECE9E74.1050605@ibotty.net> <20111124195455.GJ3214@nerd.dk>
Message-ID: <4ECEB825.50509@ibotty.net>

hi,

thank you for your hint. that got it working :).

regards,
 tobias florek



From tfheen at varnish-software.com  Fri Nov 25 07:06:42 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Fri, 25 Nov 2011 08:06:42 +0100
Subject: file,/var/tmp,50% - but no file in /var/tmp
In-Reply-To: <B9D3B3DF-62BB-4FE9-9533-EA3C38A67163@925.dk> (Martin Koch
	Andersen's message of "Wed, 23 Nov 2011 20:21:38 +0100")
References: <B9D3B3DF-62BB-4FE9-9533-EA3C38A67163@925.dk>
Message-ID: <87lir4vfy5.fsf@qurzaw.varnish-software.com>

]] Martin Koch Andersen 

Hi,

| With Varnish 3.0.1 a (memory mapped) file was thus created in /var/tmp, when Varnish was started.
| With Varnish 3.0.2 no such file is created. And I don't see Varnish having any open file descriptors to anything in /var/tmp.

3.0.2 is fixed to unlink the file on startup to automatically free the
space when Varnish stops.  Unlinked files do not have a directory
associated with them, just a file descriptor.

Regards,
-- 
Tollef Fog Heen
Technical lead, Varnish Software
t: +47 21 98 92 64



From tfheen at varnish-software.com  Fri Nov 25 07:07:48 2011
From: tfheen at varnish-software.com (Tollef Fog Heen)
Date: Fri, 25 Nov 2011 08:07:48 +0100
Subject: Backend Keep-Alive
In-Reply-To: <4ECD65DB.6010104@gmail.com> (Jo Galara's message of "Wed, 23 Nov
	2011 22:30:03 +0100")
References: <4ECD65DB.6010104@gmail.com>
Message-ID: <87hb1svfwb.fsf@qurzaw.varnish-software.com>

]] Jo Galara 

Hi,

| I'm using Apache2 as backend. Should I enable Keep-Alives in Apache?

Yes.

| Are the default timeouts ok or should I change them?

The should be ok, but increasing them won't hurt.  Varnish keeps backend
connections open and reuses them.

Regards,
-- 
Tollef Fog Heen
Technical lead, Varnish Software
t: +47 21 98 92 64



From mattias at nucleus.be  Fri Nov 25 09:09:42 2011
From: mattias at nucleus.be (Mattias Geniar)
Date: Fri, 25 Nov 2011 10:09:42 +0100
Subject: Varnish as a daemon fails on CentOS 6
Message-ID: <18834F5BEC10824891FB8B22AC821A5A01C6D440@nucleus-srv01.Nucleus.local>

Hi,

I'm having a peculiar problem for which I don't see the solution
straight away, perhaps someone can point me in the right direction.

I'm trying to get Varnish 3.0.2 compiled from source to work on a CentOS
6 x64 running on an OpenVZ container (the openvz kernel could be messing
with me here).

If I manually start Varnish in the foreground, it works:
> # varnishd -F -f /usr/local/etc/varnish/default.vcl
> socket(): Address family not supported by protocol
> child (5214) Started
> Child (5214) said Child starts
> Child (5214) said SMF.s0 mmap'ed 104857600 bytes of 104857600
> ^CManager got SIGINT => CTRL+C killed
> Stopping Child

If I try via the service, it fails:
> # service varnish start
> Starting Varnish Cache:                                    [FAILED]

When I strace it, I don't see any immediate problem:
> # strace -e trace=file service varnish start
> execve("/sbin/service", ["service", "varnish", "start"], [/* 18 vars
*/]) = 0
> access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
directory)
> open("/etc/ld.so.cache", O_RDONLY)      = 3
> open("/lib64/libtinfo.so.5", O_RDONLY)  = 3
> open("/lib64/libdl.so.2", O_RDONLY)     = 3
> open("/lib64/libc.so.6", O_RDONLY)      = 3
> open("/dev/tty", O_RDWR|O_NONBLOCK)     = 3
> open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
> open("/proc/meminfo", O_RDONLY)         = 3
> stat("/root", {st_mode=S_IFDIR|0550, st_size=4096, ...}) = 0
> stat(".", {st_mode=S_IFDIR|0550, st_size=4096, ...}) = 0
> open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3
> open("/sbin/service", O_RDONLY)         = 3
> open("/etc/init.d/functions", O_RDONLY) = 3
> --- SIGCHLD (Child exited) @ 0 (0) ---
> stat("/etc/sysconfig/i18n", {st_mode=S_IFREG|0644, st_size=47, ...}) =
0
> open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
> open("/etc/profile.d/lang.sh", O_RDONLY) = 3
> stat("/root/.i18n", 0x7fffa8819680)     = -1 ENOENT (No such file or
directory)
> stat("/etc/sysconfig/init", {st_mode=S_IFREG|0644, st_size=1153, ...})
= 0
> open("/etc/sysconfig/init", O_RDONLY)   = 3
> open("/usr/share/locale/locale.alias", O_RDONLY) = 3
> open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/initscripts.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
> open("/usr/share/locale/en_US.utf8/LC_MESSAGES/initscripts.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
> open("/usr/share/locale/en_US/LC_MESSAGES/initscripts.mo", O_RDONLY) =
-1 ENOENT (No such file or directory)
> open("/usr/share/locale/en.UTF-8/LC_MESSAGES/initscripts.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
> open("/usr/share/locale/en.utf8/LC_MESSAGES/initscripts.mo", O_RDONLY)
= -1 ENOENT (No such file or directory)
> open("/usr/share/locale/en/LC_MESSAGES/initscripts.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
> --- SIGCHLD (Child exited) @ 0 (0) ---
> --- SIGCHLD (Child exited) @ 0 (0) ---
> chdir("/")                              = 0
> stat("/etc/init.d/varnish", {st_mode=S_IFREG|0755, st_size=2851, ...})
= 0
> stat(".", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
> stat("/sbin/env", 0x7fffa881a600)       = -1 ENOENT (No such file or
directory)
> stat("/usr/sbin/env", 0x7fffa881a600)   = -1 ENOENT (No such file or
directory)
> stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
> stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
> access("/bin/env", X_OK)                = 0
> stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
> access("/bin/env", R_OK)                = 0
> stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
> stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
> access("/bin/env", X_OK)                = 0
> stat("/bin/env", {st_mode=S_IFREG|0755, st_size=23832, ...}) = 0
> vaccess("/bin/env", R_OK)                = 0
> Starting Varnish Cache:                                    [FAILED]
> --- SIGCHLD (Child exited) @ 0 (0) ---

It fails right after the /bin/env call, which leads me to believe it is
probably environment-related, but I don't see any indication on where to
look.

The init.d script I used is in attachment, that one works fine on all
CentOS 5 x64 machines.

Regards,
Mattias
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: init_d_varnish.txt
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111125/2bdc9c3f/attachment.txt>

From martin at 925.dk  Fri Nov 25 11:24:47 2011
From: martin at 925.dk (Martin Koch Andersen)
Date: Fri, 25 Nov 2011 12:24:47 +0100
Subject: file,/var/tmp,50% - but no file in /var/tmp
In-Reply-To: <87lir4vfy5.fsf@qurzaw.varnish-software.com>
References: <B9D3B3DF-62BB-4FE9-9533-EA3C38A67163@925.dk>
	<87lir4vfy5.fsf@qurzaw.varnish-software.com>
Message-ID: <B623E1D3-39EF-450F-9DA7-AB7582DD65AE@925.dk>


Den 25/11/2011 kl. 08.06 skrev Tollef Fog Heen:
> | With Varnish 3.0.1 a (memory mapped) file was thus created in /var/tmp, when Varnish was started.
> | With Varnish 3.0.2 no such file is created. And I don't see Varnish having any open file descriptors to anything in /var/tmp.
> 
> 3.0.2 is fixed to unlink the file on startup to automatically free the
> space when Varnish stops.  Unlinked files do not have a directory
> associated with them, just a file descriptor.

Ok thanks. This is not really obvious from the changelog [1].

[1] https://www.varnish-cache.org/releases/varnish-cache-3.0.2

-- 
Martin Koch Andersen
http://925.dk


From apj at mutt.dk  Sun Nov 27 09:56:25 2011
From: apj at mutt.dk (Andreas Plesner Jacobsen)
Date: Sun, 27 Nov 2011 10:56:25 +0100
Subject: Soliciting input on the ban lurker
In-Reply-To: <CANZWjw7OmxSETtnn_EQhCkWLqqa2wqaSYV=F31BcTX4Xf5riEQ@mail.gmail.com>
References: <97985.1320232167@critter.freebsd.dk>
	<20111102120022.GU13866@nerd.dk> <20111102120610.GV13866@nerd.dk>
	<CANZWjw7OmxSETtnn_EQhCkWLqqa2wqaSYV=F31BcTX4Xf5riEQ@mail.gmail.com>
Message-ID: <20111127095624.GK3214@nerd.dk>

On Sat, Nov 26, 2011 at 03:06:21PM -0500, Adam Denenberg wrote:

> does anyone have thoughts on this.  The statement "don't know about
> the ban lurker,
> and write 100% req-based ban. (In part due
>  to the presence of ban.url and ban_url)" to me means people will be
> issuing "ban" commands and not realizing they wont work b/c the ban needs
> to happen on the obj and not the req.

They'll work just fine, they just won't be processed by the ban lurker.

> Are there any strategies to this other than some of the workarounds posted
> (setting headers and and then storing the req.url in obj.url in vcl_recv)

No. (And there's no obj.url, but we get the picture)

-- 
Andreas



From mattias at nucleus.be  Sun Nov 27 23:12:43 2011
From: mattias at nucleus.be (Mattias Geniar)
Date: Mon, 28 Nov 2011 00:12:43 +0100
Subject: Varnish as a daemon fails on CentOS 6
In-Reply-To: <D150525FBF81FA4BBD34C4E4EE350D28075C6E1C@nucleus-srv01.Nucleus.local>
References: <D150525FBF81FA4BBD34C4E4EE350D28075C6E1C@nucleus-srv01.Nucleus.local>
Message-ID: <18834F5BEC10824891FB8B22AC821A5A01C6D464@nucleus-srv01.Nucleus.local>

> It fails right after the /bin/env call, which leads me 
> to believe it is probably environment-related, but I 
> don't see any indication on where to look.

Time to solve my own problem: turned out to be a missing directive in
/etc/sysconfig/varnish that I overlooked. Strange that the strace didn't
show that file being read. Foreground mode doesn't read it, daemonized
mode does.

Solved it is!

Regards,
Mattias



From simon at darkmere.gen.nz  Sun Nov 27 23:19:13 2011
From: simon at darkmere.gen.nz (Simon Lyall)
Date: Mon, 28 Nov 2011 12:19:13 +1300 (NZDT)
Subject: collectd 4.10 and Varnish
In-Reply-To: <1321442578-sup-7101@lonquimay.wrk.lsn.camptocamp.com>
References: <alpine.DEB.2.00.1111151043110.31241@green.darkmere.gen.nz>
	<1321442578-sup-7101@lonquimay.wrk.lsn.camptocamp.com>
Message-ID: <alpine.DEB.2.00.1111281212430.22198@green.darkmere.gen.nz>

On Wed, 16 Nov 2011, Marc Fournier wrote:
> Excerpts from Simon Lyall's message of Mon Nov 14 22:50:37 +0100 2011:
>>
>> We're running collectd using version 4.10 from EPEL on RHEL/Centos.
>>
>> There is a varnish plugin recently released with collectd but it requires
>> version 5.10 of collectd which is fairly recently released and doesn't
>> have rpms (I'd prefer not to build my own).
>>
>> 1. I was wondering if anybody has a backported varnish plugin for collectd
>>     4.10 or any other solution? I came across this post in the archives:
>
> I have done exactly this a few months ago. I just pushed the patch to:
> https://github.com/mfournier/collectd/tree/varnish-on-4.10.2
>
> Not sure if the plugin is up to date with the one found in 5.x though...

Thanks Marc, I didn't have any luck getting the plugin to work (it 
worked enough to complain when I did syntax errors but didn't output 
any data and there are some problems with running the python plugin so I 
eneded up writing a little script in bash and running it via the collectd 
exec plugin.

I've attached it in case anyone wants it. Seems to work and should be 
compatable with the 5.x collectd plugin (see evil 1-liner in the script).

-- 
Simon Lyall  |  Very Busy  |  Web: http://www.darkmere.gen.nz/
"To stay awake all night adds a day to your life" - Stilgar | eMT.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: varnish-collectd.sh
Type: application/x-sh
Size: 9538 bytes
Desc: 
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111128/8d7ede78/attachment.sh>

From jogalara at gmail.com  Mon Nov 28 03:37:18 2011
From: jogalara at gmail.com (Jo Galara)
Date: Mon, 28 Nov 2011 04:37:18 +0100
Subject: Backend Keep-Alive
In-Reply-To: <87hb1svfwb.fsf@qurzaw.varnish-software.com>
References: <4ECD65DB.6010104@gmail.com>
	<87hb1svfwb.fsf@qurzaw.varnish-software.com>
Message-ID: <4ED301EE.8070407@gmail.com>

Hi,

this confirmed what I thought. Thanks.

... never trust the EU guys! (no offence)


On 11/25/2011 08:07 AM, Tollef Fog Heen wrote:
> ]] Jo Galara 
>
> Hi,
>
> | I'm using Apache2 as backend. Should I enable Keep-Alives in Apache?
>
> Yes.
>
> | Are the default timeouts ok or should I change them?
>
> The should be ok, but increasing them won't hurt.  Varnish keeps backend
> connections open and reuses them.
>
> Regards,
-- 
Regards,

Jo Galara


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20111128/44478326/attachment.bin>

From rtshilston at gmail.com  Wed Nov 30 19:17:26 2011
From: rtshilston at gmail.com (Rob S)
Date: Wed, 30 Nov 2011 19:17:26 +0000
Subject: Client reflected response back to varnish, and got into a nasty loop
Message-ID: <CAKF-8eAU0GeqQqdigRQE9fgu5RbxbSdXKz=TzUm9OYL28Eq=tw@mail.gmail.com>

Hi,

We had a strange incident happy on a varnish node today, which
resulted in a sustained extra 60 reqs/sec for around 4.5 hrs of
identical sequential requests with the *request* being "HTTP/1.1 302
Found".  The particular IP address had been browsing normally, until
they got the following response:


<snip>
  491 TxProtocol   c HTTP/1.1
  491 TxStatus     c 200
  491 TxResponse   c OK
  491 TxHeader     c Content-Type: text/javascript;charset=UTF-8
  491 TxHeader     c Cache-Control: no-cache, no-store, max-age=0,
must-revalidate, private, no-transform
  491 TxHeader     c Expires: Fri, 29 Apr 2011 05:00:00 GMT
  491 TxHeader     c Content-Length: 2715
  491 TxHeader     c Date: Wed, 30 Nov 2011 12:04:53 GMT
  491 TxHeader     c X-Varnish: 747013197
  491 TxHeader     c Age: 0
  491 TxHeader     c Via: 1.1 varnish
  491 TxHeader     c Connection: keep-alive
  491 Length       c 2715
  491 ReqEnd       c 747013197 1322654693.630128860
1322654693.636650801 0.030966997 0.006455421 0.000066519


This looks absolutely fine - we sent them a short javascript snippet
that shouldn't be cached.  This is a request which happens thousands
of times an hour with no issue.  What happened next is strange:  The
response we'd just sent them (above) came back to varnish as a fresh
request:

  491 ReqStart     c 86.166.aaa.bbb 56853 747013203
  491 RxRequest    c HTTP/1.1
  491 RxURL        c 200
  491 RxProtocol   c OK
  491 RxHeader     c Content-Type: text/javascript;charset=UTF-8
  491 RxHeader     c Cache-Control: no-cache, no-store, max-age=0,
must-revalidate, private, no-transform
  491 RxHeader     c Expires: Fri, 29 Apr 2011 05:00:00 GMT
  491 RxHeader     c Content-Length: 2715
  491 RxHeader     c Date: Wed, 30 Nov 2011 12:04:53 GMT
  491 RxHeader     c X-Varnish: 747013197
  491 RxHeader     c Age: 0
  491 RxHeader     c Via: 1.1 varnish
  491 RxHeader     c Connection: keep-alive
  491 RxHeader     c X-Antivirus: avast! 4
  491 RxHeader     c X-Antivirus-Status: Clean


Look how we *received* an X-Varnish header, with the same ID as the
previous response.  But, it must have gone via the client, as an
additional header was added - the X-Antivirus.  This then was a miss
in Varnish, and fell through to our backends, which responds with a
redirect if requested with an invalid hostname:

  491 VCL_call     c recv pass
  491 VCL_call     c hash hash
  491 VCL_call     c pass pass
  491 Backend      c 823 primarypool web01
  491 TTL          c 747013203 RFC 120 1322654693 0 0 0 0
  491 VCL_call     c fetch
  491 TTL          c 747013203 VCL 0 1322654694
  491 VCL_return   c deliver
  491 ObjProtocol  c HTTP/1.1
  491 ObjStatus    c 302
  491 ObjResponse  c Found
  491 ObjHeader    c Date: Wed, 30 Nov 2011 11:04:54 GMT
  491 ObjHeader    c Location: http://www.mainsite.com
  491 ObjHeader    c Content-Length: 201
  491 ObjHeader    c Content-Type: text/html; charset=iso-8859-1
  491 ObjHeader    c Cache-Control: max-age=0, public
  491 ObjHeader    c X-Varnish-Status: u
  491 VCL_call     c deliver deliver
  491 TxProtocol   c HTTP/1.1
  491 TxStatus     c 302
  491 TxResponse   c Found
  491 TxHeader     c Location: http://www.mainsite.com
  491 TxHeader     c Content-Type: text/html; charset=iso-8859-1
  491 TxHeader     c Cache-Control: max-age=0, public
  491 TxHeader     c X-Varnish-Status: u
  491 TxHeader     c Content-Length: 201
  491 TxHeader     c Date: Wed, 30 Nov 2011 12:04:53 GMT
  491 TxHeader     c X-Varnish: 747013203
  491 TxHeader     c Age: 0
  491 TxHeader     c Via: 1.1 varnish
  491 TxHeader     c Connection: keep-alive
  491 Length       c 201
  491 ReqEnd       c 747013203 1322654693.676038265
1322654693.701058388 0.039387465 0.024901628 0.000118494

And then this pattern continued for over 4 hrs.


  491 ReqStart     c 86.166.aaa.bbb 56853 747013208
  491 RxRequest    c HTTP/1.1
  491 RxURL        c 302
  491 RxProtocol   c Found
  491 RxHeader     c Location: http://www.mainsite.com
  491 RxHeader     c Content-Type: text/html; charset=iso-8859-1
  491 RxHeader     c Cache-Control: max-age=0, public
  491 RxHeader     c X-Varnish-Status: u
  491 RxHeader     c Content-Length: 201
  491 RxHeader     c Date: Wed, 30 Nov 2011 12:04:53 GMT
  491 RxHeader     c X-Varnish: 747013203
  491 RxHeader     c Age: 0
  491 RxHeader     c Via: 1.1 varnish
  491 RxHeader     c Connection: keep-alive
  491 VCL_call     c recv pass
  491 VCL_call     c hash hash
  491 VCL_call     c pass pass
  491 Backend      c 115 primarypool web02
  491 TTL          c 747013208 RFC 120 1322654693 0 0 0 0
  491 VCL_call     c fetch
  491 TTL          c 747013208 VCL 0 1322654694
  491 VCL_return   c deliver
  491 ObjProtocol  c HTTP/1.1
  491 ObjStatus    c 302
  491 ObjResponse  c Found
  491 ObjHeader    c Date: Wed, 30 Nov 2011 11:04:54 GMT
  491 ObjHeader    c Location: http://www.mainsite.com
  491 ObjHeader    c Content-Length: 201
  491 ObjHeader    c Content-Type: text/html; charset=iso-8859-1
  491 ObjHeader    c Cache-Control: max-age=0, public
  491 ObjHeader    c X-Varnish-Status: u
  491 VCL_call     c deliver deliver
  491 TxProtocol   c HTTP/1.1
  491 TxStatus     c 302
  491 TxResponse   c Found
  491 TxHeader     c Location: http://www.mainsite.com
  491 TxHeader     c Content-Type: text/html; charset=iso-8859-1
  491 TxHeader     c Cache-Control: max-age=0, public
  491 TxHeader     c X-Varnish-Status: u
  491 TxHeader     c Content-Length: 201
  491 TxHeader     c Date: Wed, 30 Nov 2011 12:04:53 GMT
  491 TxHeader     c X-Varnish: 747013208
  491 TxHeader     c Age: 0
  491 TxHeader     c Via: 1.1 varnish
  491 TxHeader     c Connection: keep-alive
  491 Length       c 201
  491 ReqEnd       c 747013208 1322654693.774824381
1322654693.775967836 0.073765993 0.001085758 0.000057697

  491 ReqStart     c 86.166.aaa.bbb 56853 747013209
  491 RxRequest    c HTTP/1.1
  491 RxURL        c 302
  491 RxProtocol   c Found
  491 RxHeader     c Location: http://www.mainsite.com
  491 RxHeader     c Content-Type: text/html; charset=iso-8859-1
  491 RxHeader     c Cache-Control: max-age=0, public
  491 RxHeader     c X-Varnish-Status: u
  491 RxHeader     c Content-Length: 201
  491 RxHeader     c Date: Wed, 30 Nov 2011 12:04:53 GMT
  491 RxHeader     c X-Varnish: 747013208
  491 RxHeader     c Age: 0
  491 RxHeader     c Via: 1.1 varnish
  491 RxHeader     c Connection: keep-alive
  491 VCL_call     c recv pass
  491 VCL_call     c hash hash
  491 VCL_call     c pass pass
  491 Backend      c 115 primarypool web03
  491 TTL          c 747013209 RFC 120 1322654693 0 0 0 0
  491 VCL_call     c fetch
  491 TTL          c 747013209 VCL 0 1322654694
  491 VCL_return   c deliver
  491 ObjProtocol  c HTTP/1.1
  491 ObjStatus    c 302
  491 ObjResponse  c Found
  491 ObjHeader    c Date: Wed, 30 Nov 2011 11:04:55 GMT
  491 ObjHeader    c Location: http://www.mainsite.com
  491 ObjHeader    c Content-Length: 201
  491 ObjHeader    c Content-Type: text/html; charset=iso-8859-1
  491 ObjHeader    c Cache-Control: max-age=0, public
  491 ObjHeader    c X-Varnish-Status: u
  491 VCL_call     c deliver deliver
  491 TxProtocol   c HTTP/1.1
  491 TxStatus     c 302
  491 TxResponse   c Found
  491 TxHeader     c Location: http://www.mainsite.com
  491 TxHeader     c Content-Type: text/html; charset=iso-8859-1
  491 TxHeader     c Cache-Control: max-age=0, public
  491 TxHeader     c X-Varnish-Status: u
  491 TxHeader     c Content-Length: 201
  491 TxHeader     c Date: Wed, 30 Nov 2011 12:04:53 GMT
  491 TxHeader     c X-Varnish: 747013209
  491 TxHeader     c Age: 0
  491 TxHeader     c Via: 1.1 varnish
  491 TxHeader     c Connection: keep-alive
  491 Length       c 201
  491 ReqEnd       c 747013209 1322654693.791796923
1322654693.793078899 0.015829086 0.001143694 0.000138283




Has anyone seen anything like this before?  How would you reduce the
likelihood of this circular (almost) DoS attach in Varnish?  Should we
drop inbound requests if they contain an RxHeader of X-Varnish?  Our
vcl_recv is fairly complex, but ultimately boils down to
return(lookup) if it doesn't match any earlier criteria, so that's the
path being hit.

We didn't see a significant change in network bandwidth usage, and at
60 requests per second, I wonder if this tight loop happened
internally within Varnish, rather than going all the way to the end IP
address.  60 sequential requests per second is one every 16ms.  The IP
address is an ADSL connection in the UK, so it *could* be 16ms away,
but that's at the very low end of what's the likely ping time.  But,
on the other hand, 16ms is a long time for Varnish to be in a loop in
the datacentre, given that the backend web servers are on the same
gigabit LAN as Varnish and our backends are fast for this request.

We're running varnish-2.1.5-1.el6.x86_64 on CentOS 6.0 (2.6.32-71.el6.x86_64)


Thanks,


Rob



From cosimo at streppone.it  Wed Nov 30 22:10:20 2011
From: cosimo at streppone.it (Cosimo Streppone)
Date: Wed, 30 Nov 2011 23:10:20 +0100
Subject: Varnish DAEMON_OPTS Options Errors and Debian startup script
Message-ID: <op.v5r47iejs5ttvb@cn01>

This thread about starting up varnish with a custom cc_command:

https://www.varnish-cache.org/lists/pipermail/varnish-misc/2011-May/thread.html#6114

was never "closed".

I think I found the cause of the problem.

Due to the way $DAEMON_OPTS is included in the start-stop-daemon command,
and because of bash vs dash differences, if your default shell is dash
the command is not expanded correctly.

Copying and pasting exactly the same 'start-stop-daemon' command
in a bash shell works just fine.

http://stackoverflow.com/questions/5906603/varnish-daemon-opts-options-errors/8333333#8333333

If other people could confirm that this workaround works for them,
maybe we could even file a bug in Debian.

-- 
Cosimo