From geoff at uplex.de  Mon Jun  1 09:40:22 2015
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 01 Jun 2015 11:40:22 +0200
Subject: Workspace exhaustion with Varnish 4 under load
In-Reply-To: <55675D93.60801@uplex.de>
References: <55675D93.60801@uplex.de>
Message-ID: <556C2886.9000907@uplex.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 05/28/2015 08:25 PM, Geoff Simmons wrote:
> 
> We're getting our first experience with testing Varnish 4 under
> load (v4.0.3), and we're having severe problems with workspace
> exhaustion

To follow this up, in case someone finds it in an archive search
someday -- it turns out that this was a problem entirely of our own
making, due to a bug in our VCL.

We had VCL code to replace req.url in vcl_recv() under certain
circumstances, but it should have been done only at ESI level 0, and
it was happening at every ESI level. The result was massive ESI
expansion in both depth and breadth that was enough to explode
workspaces. The trac ticket (#1743) explains in a little more detail
how that happened.

After we fixed the VCL bug so that the req.url replacement only
happens at ESI level 0, the tests ran successfully. So let it be said
that our load tests do *not* indicate a problem with workspace
management on the part of varnishd.

The ticket will be closed, thanks again to phk for helping out.


Best,
Geoff
- -- 
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBCAAGBQJVbCh6AAoJEOUwvh9pJNURKhcP/Ry2rWqhgfdo1178VWQhLv6L
hfS0KAd1wqx0dniHCB92Ln6bfv4sLVITpKo6fBd3Kt2M6up12IAKtx+D9DT+YVVu
qglmivtYZKHCHi75LUb2iIR+dPb7I1Uj4+jl0vv9AxDvicgggakWqNAQrRvdVVfc
f69xtC0Bc2LJxizRiBB83RkdhGqDMzwcrIlJhoS5M7VDvxdAx15McQY4qLjw9H6F
w5vyI3njqyjsthMZFVHXNKCHG7PDD4aody2Wcv/yy0DkH3gSmqFPVXqqztOliZk3
I6T62ouAntadc73Nt5LG9x82xPrb2wO5/lFU7QrKUtKzWrCQntjG3ven4D97tWOZ
2/S2BvA1PbUU4YIkwkZozM2nyM02Xco1JtUz4Wf9LspGjimA9ij2z4XQBSzNrFR/
JPIqWXgmzFPCV9nOjrGBFPJIig7nSoi6heLLA7DwGlEUwnYSEyrwDejf7XIhBmj9
fx8rYbDGQ+X6M0cdKPe6dh1V/M46qjifHcb89m5q5GvXJMFP9KHED7MDqXEbMnij
0wuKvXuz7AUqJoOD4VoZNvck1ZTLr6rmeP/mDibaFJI9/J8NLWhR2hHjyIoRYMBp
Zaj5l/ElhXQSgvDcx7YP1EaY/bWm+DEdTQGN+q/anE95DpgGPAYzIAAx1q9woc5h
gUIm4DrtlHkjp4r+1wWj
=+nEV
-----END PGP SIGNATURE-----



From akelly at snafu.de  Tue Jun  9 16:28:30 2015
From: akelly at snafu.de (Andrew Kelly)
Date: Tue, 09 Jun 2015 18:28:30 +0200
Subject: X-Frame-Options
Message-ID: <1433867310.2865.2.camel@h2114982.stratoserver.net>

Hi,

is it possible to either set or unset the X-Frame-Options meta header
based on the IP address of the visiting client?

Many thanks in advance,

Andy




From andrew.langhorn at digital.cabinet-office.gov.uk  Tue Jun  9 16:42:53 2015
From: andrew.langhorn at digital.cabinet-office.gov.uk (Andrew Langhorn)
Date: Tue, 9 Jun 2015 17:42:53 +0100
Subject: X-Frame-Options
In-Reply-To: <1433867310.2865.2.camel@h2114982.stratoserver.net>
References: <1433867310.2865.2.camel@h2114982.stratoserver.net>
Message-ID: <CANCa_WbTOFNN4Yov1yvTmJt7ARH9BhiOLj+MJfJ8qdm+0fORkA@mail.gmail.com>

I work on a stack which does something similar, and I blogged about it a
while ago: http://ajlanghorn.com/2015/05/varnish-includes/

In your case, you should be able to do copy a good chunk of that, but
change the return to the lookup function for beresp.http.X-Frame-Options =
"value" or something.

Let me know if you'd like me to expand on that.

On 9 June 2015 at 17:28, Andrew Kelly <akelly at snafu.de> wrote:

> Hi,
>
> is it possible to either set or unset the X-Frame-Options meta header
> based on the IP address of the visiting client?
>
> Many thanks in advance,
>
> Andy
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



-- 
Andrew Langhorn
Web Operations
Government Digital Service

e: andrew.langhorn at digital.cabinet-office.gov.uk
t: +44 (0)7810 737375
a: 6th Floor, Aviation House, 125 Kingsway, London, WC2B 6NH
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150609/7491dc0d/attachment.html>

From lyle.mantooth at primasupply.com  Thu Jun 18 20:27:40 2015
From: lyle.mantooth at primasupply.com (Lyle Mantooth)
Date: Thu, 18 Jun 2015 16:27:40 -0400
Subject: Ban strategy
Message-ID: <558329BC.2040106@primasupply.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hello list,

I've noticed that Varnish tends to use more and more CPU as time goes
on, and everything I've read points to a large ban list being
responsible. That sounds reasonable, but I don't think our ban list is
particularly large, even at the busiest of times. On the other hand,
it's hardly ever empty because our content team is constantly making
changes throughout our catalog.

So I come to you, Varnish experts, to see if there's a better way to
construct our ban regexes. Currently, our CMS collects the URLs that a
product might appear on and bans them all at once like so:

```
ban obj.http.x-url ~
^/node/123$|^/product/title$|^/category/77$|^/catalog/title$
ban obj.http.x-url ~
^/node/124$|^/product/other$|^/category/77$|^/catalog/title$
```

It is often the case that several products in a category will be
updated more or less together, so the category pages would be matched
by several bans in the list. Which is easier on the ban lurker:
combined bans like this or more, but simpler, bans that are easier to
match? Even when that would mean more duplicate bans to be issued for
category pages?

- -- 
- --Lyle Mantooth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQGcBAEBCgAGBQJVgym4AAoJEG21Lq4SOleJafAL/2P9FVzBGnneWp2P7GqXOiFV
sF8e+NZHGni/OHvG761PR50UhxNhlxALkdVJVX/6clengUtQ5pf8BLozJNpP7Jjt
qvujx2DpNbLR5ecRuzAcbIdkavAOq2l5Y2wh+1qjlusuf2ofVvPs3Zt/BHQuXsL1
EHvE1h5gxsyJAcOsbPlRkdb1SAz6PV36wH1toHMk9XXubzXR5W94yzGjLf64fZZ4
IkowCP2+og1asSJSdFEQ5lSyDfI9//QRbibQ2MsTP5t8PsFgQRcw0v3eyTOBDGz6
TCQPbbSRZ0MQsFwDV64yipGtktKQUSsA/lKSAyZDc8XqSRV/Sts1aq+B88wVdy12
+MWJLk4yCn6qvqX+9b7shqYmrMJ42wFHiEg2kfkMS/pIk9ooTDK8+Mi9Gq7AZRKD
iIwQJk55kXKjfArdsKKZIul3MFpLxmKj43UpZq9n8NJ6IxMXUpGrAKiuWbsy4zYG
CJJaENNjTWi+Y9MG4GwtySzzl7KFkKmSeoo1aC7CmA==
=HnFb
-----END PGP SIGNATURE-----



From pbrown at nbmedia.com  Tue Jun 23 22:11:06 2015
From: pbrown at nbmedia.com (Peter Brown)
Date: Tue, 23 Jun 2015 22:11:06 +0000
Subject: Looking for Expert Varnish Consultant (Drupal/WP/DNN)
Message-ID: <116A5A222B28B84E92C959C5866857C1483C5994@NBNYMAIL2.nbmedia.com>

Dear Varnish Gurus,

Two years ago, I found a wonderfully skilled Varnish consultant through this list.

He's still working with us, but has some limitations on time, so I'm now looking for
a second person, who can be a backup and fill in some potential scheduling blanks.

Here are the specs:

- looking for an expert-level Varnish guru with strong experience scaling Varnish

- we're located on the East Coast of the USA, and would prefer to be able to speak on the phone
            . our working hours are 10am - 6pm EST, with phone calls up to 8pm EST
            . since this is part time, it's okay if you can only talk between 6pm and 8pm.

            (thus, we're leaning toward a consultant in the US, but if you're not in the US,
            the above time-frames are important.)

- ongoing Varnish consulting - infrequent, but possibly time sensitive when it occurs

- work can be done in the evenings or weekends

- strong communication skills in clear English are a must.

- detail oriented, with good documentation skills (i.e. to document your work)

- we're a medium to large magazine publisher that uses Drupal and WordPress (on Linux)
            and DotNetNuke (DNN) (on Windows)

            => We currently use Varnish with both Drupal and WordPress, but would like to
            integrate the purging mechanisms with DotNetNuke, as well.

- we have an active/passive Varnish server sitting in front of multiple web nodes
            (so we're not running Varnish on the web nodes, which means that we can
            theoretically use it for DNN on Windows)

- you must have strong experience integrating Varnish with one or more of the CMS's above
            (especially Drupal -- that's our primary Varnish CMS)

            => Thus, strong Drupal experience is important.

Please email me off list with your details.

Thanks!

Peter Brown

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150623/414764b8/attachment.html>

From varnish at tengu.ch  Wed Jun 24 08:15:20 2015
From: varnish at tengu.ch (=?UTF-8?B?Q8OpZHJpYyBKZWFubmVyZXQ=?=)
Date: Wed, 24 Jun 2015 10:15:20 +0200
Subject: "hash" director and app cookie
Message-ID: <558A6718.7050102@tengu.ch>

Hello,

I have some troubles for a varnish4 vcl. Here's the facts:

- one round_robin director for stateless accesses (no authentication)
- one hash director for stateful accesses (authentication, cookie? blah)
- both directors have different backend servers

The round_robin part is, of course, the easiest. It handles queries as
expected, all is green.

But the hash one is a bit more complicated: I guess I'll need to use
client.identity or something like that, but in order to ensure it really
does some load-balancing, I'll need some "proofs" the service is indeed
load-balanced and redirecting to the right instances.

How may I do that? using a simple "curl" doesn't show any
load-balancing, as I'm hitting always the same instance. I'm pretty sure
this is due to client.identity ? how may I fake it?
I didn't find any documentation regarding this object, and reading
source-code in order to know a bit more isn't really the easiest way to
understand what's done.

Thanks for you help!

Cheers,

C.



From dridi at varni.sh  Wed Jun 24 08:32:27 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 24 Jun 2015 10:32:27 +0200
Subject: "hash" director and app cookie
In-Reply-To: <558A6718.7050102@tengu.ch>
References: <558A6718.7050102@tengu.ch>
Message-ID: <CABoVN9AnWejXxSP9op4opOQXUCZB9VnKth1SUhyUzOKVLANbRQ@mail.gmail.com>

On Wed, Jun 24, 2015 at 10:15 AM, C?dric Jeanneret <varnish at tengu.ch> wrote:
> How may I do that? using a simple "curl" doesn't show any
> load-balancing, as I'm hitting always the same instance. I'm pretty sure
> this is due to client.identity ? how may I fake it?

Hi,

Have a look at varnishtest for faking it, you can make your backends
reply with a hardcoded header and check that on the client side.

> I didn't find any documentation regarding this object, and reading
> source-code in order to know a bit more isn't really the easiest way to
> understand what's done.

See the documentation at the bottom:
https://www.varnish-cache.org/docs/4.0/reference/vmod_directors.generated.html#backend-hash-backend-string-list

The hash director doesn't rely on client.identity anymore, but on
whatever you want to use to route the traffic.

Cheers,
Dridi

> Thanks for you help!
>
> Cheers,
>
> C.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From varnish at tengu.ch  Wed Jun 24 10:54:47 2015
From: varnish at tengu.ch (=?UTF-8?B?Q8OpZHJpYyBKZWFubmVyZXQ=?=)
Date: Wed, 24 Jun 2015 12:54:47 +0200
Subject: "hash" director and app cookie
In-Reply-To: <CABoVN9AnWejXxSP9op4opOQXUCZB9VnKth1SUhyUzOKVLANbRQ@mail.gmail.com>
References: <558A6718.7050102@tengu.ch>
 <CABoVN9AnWejXxSP9op4opOQXUCZB9VnKth1SUhyUzOKVLANbRQ@mail.gmail.com>
Message-ID: <558A8C77.602@tengu.ch>



On 06/24/2015 10:32 AM, Dridi Boukelmoune wrote:
> On Wed, Jun 24, 2015 at 10:15 AM, C?dric Jeanneret <varnish at tengu.ch> wrote:
>> How may I do that? using a simple "curl" doesn't show any
>> load-balancing, as I'm hitting always the same instance. I'm pretty sure
>> this is due to client.identity ? how may I fake it?
> 
> Hi,
> 
> Have a look at varnishtest for faking it, you can make your backends
> reply with a hardcoded header and check that on the client side.
> 
>> I didn't find any documentation regarding this object, and reading
>> source-code in order to know a bit more isn't really the easiest way to
>> understand what's done.
> 
> See the documentation at the bottom:
> https://www.varnish-cache.org/docs/4.0/reference/vmod_directors.generated.html#backend-hash-backend-string-list
> 
> The hash director doesn't rely on client.identity anymore, but on
> whatever you want to use to route the traffic.
> 
> Cheers,
> Dridi
> 
>> Thanks for you help!
>>
>> Cheers,
>>
>> C.

Hello Dridi,

Thanks, I think I was just not awake, I'm pretty sure I'll be able to do
what I need, I was focused on the application cookie instead of just
thinking "the POST for login will be redirected to the statefull
backend, as well as the other so long there is a cookie matching a
certain pattern, hence we're using the hash director from the very
beginning, hence no issue at all".

This should do the trick hopefully, and I'll see how to fake the "thing
that identify my query" in order to ensure it goes on different hosts.

Cheers,

C.



From freja.borginger at portsit.se  Thu Jun 25 10:00:22 2015
From: freja.borginger at portsit.se (Freja Borginger)
Date: Thu, 25 Jun 2015 10:00:22 +0000
Subject: mass-redirect similar to how "tinyurl"-sites works
Message-ID: <af4b1b4dc4064ca6b4fcc54305ffb3b1@jon-int-exh13.ports.local>

Hi!

I'm going to setup a mass-redirect server and I'm wondering which, if any, of my plans are going to work. The redirects are similar to those "tinyurl"-like websites where a short unique string leads to a full URL.

This is what I know I'm faced with:

Over 1,500,000 different redirects.
300 visitors per second.
70 redirects added per hour.

Plan A:
One if-section for each URL to redirect.
Will 1,500,000 if-sections-lookup slow down Varnish?
Will compiling via varnishadm be slow? If so, will it be faster to split the configurations in several .vcl and only compile the changed ones if possible?

Plan B:
Let varnish cache 301 redirects forever. Let aria2 and visitors populate the cache when varnish is started.
Populating will probably take a long time. I'm hoping the backend of squid+squirm will handle the 300 lookups/s from visitors otherwise visitors will use another backend which is already populated.
Anyway, will the lookup in cache be faster than plan A where lookup is done in if-sections?

Plan C:
Use vmod-sql to issue queries to a database. I think this will be way too slow because I don't think I can cache those. Sadly I didn't find much documentation on this vmod.

I saw there was a similar question asked before. From that one I gathered that the if-sections-lookup will be faster than non-cached redirects from a squid+squirm backend, hence why I'd cache those responses in plan B.

Freja Borginger
IT

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150625/46d066d1/attachment.html>

From bluethundr at gmail.com  Mon Jun 29 00:57:26 2015
From: bluethundr at gmail.com (Tim Dunphy)
Date: Sun, 28 Jun 2015 20:57:26 -0400
Subject: cache pages with apache auth
Message-ID: <CAOZy0ekORpidAYNJ1V4gE9UAg70PSzNZtVUiexy1JTSOZo0EqQ@mail.gmail.com>

Hey guys,

 Ok so I have a website that uses apache basic authentication that needs to
be cached.

 When apache auth is in the config, I get a 503 error and this is what I
see in the varnishlog:

    0 Backend_health - web2 Still sick 4--X-R- 0 3 5 0.014946 0.000000
HTTP/1.1 401 Unauthorized
    0 Backend_health - web1 Still sick 4--X-R- 0 3 5 0.014766 0.000000
HTTP/1.1 404 Not Found

As soon as I remove apache auth from the site, it starts working from
behind varnish.

So I tried using this tutorial to cache the site with authentication in
place:

http://blog.tenya.me/blog/2011/12/14/varnish-http-authentication/

Here's my vcl file:

probe healthcheck {

  .url = "/healthcheck.php";

  .timeout = 5s;

  .interval = 2s;

  .window = 5;

  .threshold = 3;

}


backend web1 {

  .host = ?10?10.10.25?;

  .port = "80";

  .probe = healthcheck;

  .connect_timeout = 30s;

  .first_byte_timeout = 30s;

  .between_bytes_timeout = 30s;

  .max_connections = 70;

}


backend web2 {

  .host = ?10.10.10.26;

  .port = "80";

  .probe = healthcheck;

  .connect_timeout = 30s;

  .first_byte_timeout = 30s;

  .between_bytes_timeout = 30s;

  .max_connections = 70;

}



director www client {

  { .backend = web1 ; .weight = 2;  }

  { .backend = web2 ; .weight = 2;  }

 }



sub vcl_recv {



    set req.backend = www;


    unset req.http.cookie;


    if (! req.http.Authorization ~ "Basic someBase64hash?)

    {

        error 401 "Restricted";

     }


    if (req.backend.healthy) {

       set req.grace = 30s;

      } else {

        set req.grace = 4h;

    }


    return (lookup);

}


sub vcl_fetch {



   if ( req.url ~ "^/index.php$" || req.url ~
"^/cometchat/cometchat_receive.php$") {

      set beresp.ttl = 3600s;

   }


    set beresp.grace = 4h;


    return (deliver);

}


sub vcl_error {


  if (obj.status == 401) {

  set obj.http.Content-Type = "text/html; charset=utf-8";

  set obj.http.WWW-Authenticate = "Basic realm=Secured";

  synthetic {"


 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"

 "http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">


 <HTML>

 <HEAD>

 <TITLE>Error</TITLE>

 <META HTTP-EQUIV='Content-Type' CONTENT='text/html;'>

 </HEAD>

 <BODY><H1>401 Unauthorized (varnish)</H1></BODY>

 </HTML>

 "};

  return (deliver);

  }



}


sub vcl_deliver {

     if (obj.hits> 0) {

      set resp.http.X-Cache = "HIT";

     } else {

        set resp.http.X-Cache = "MISS";

     }

 }

I'd appreciate any tips on how to get this to work you may have!


Thanks,

Tim




-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20150628/25e23c7d/attachment.html>

From tobias.eichelbroenner at lamp-solutions.de  Mon Jun 29 07:27:20 2015
From: tobias.eichelbroenner at lamp-solutions.de (=?windows-1252?Q?Tobias_Eichelbr=F6nner?=)
Date: Mon, 29 Jun 2015 09:27:20 +0200
Subject: cache pages with apache auth
In-Reply-To: <CAOZy0ekORpidAYNJ1V4gE9UAg70PSzNZtVUiexy1JTSOZo0EqQ@mail.gmail.com>
References: <CAOZy0ekORpidAYNJ1V4gE9UAg70PSzNZtVUiexy1JTSOZo0EqQ@mail.gmail.com>
Message-ID: <5590F358.9060209@lamp-solutions.de>

Hi Tim,

> Backend_health - web2 Still sick 4--X-R- 0 3 5 0.014946 0.000000
> HTTP/1.1 401 Unauthorized

seems to me your healthcheck on
 .url = "/healthcheck.php";
does not send any authorization to your backend, so the probing fails.
The most simple solution is the disable authorization for
healthcheck.php in you Webserver.

Keep in mind that if more then one user access your restricted area they
probably get the cached contend from the other user delivered. You could
put authorization header into the hash in give every user a different
password.

Sincerely,

Tobias

-- 
LAMP solutions GmbH
Gostenhofer Hauptstrasse 35
90443 Nuernberg

Amtsgericht Nuernberg: HRB 22366
Geschaeftsfuehrer: Heiko Schubert

Es gelten unsere allgemeinen Geschaeftsbedingungen.
http://www.lamp-solutions.de/agbs/

Telefon      : 0911 / 376 516 0
Fax          : 0911 / 376 516 11
E-Mail       : support at lamp-solutions.de
Web          : www.lamp-solutions.de
Facebook     : http://www.facebook.com/LAMPsolutions
Twitter      : http://twitter.com/#!/lampsolutions



From varnish at tengu.ch  Mon Jun 29 09:26:57 2015
From: varnish at tengu.ch (=?UTF-8?B?Q8OpZHJpYyBKZWFubmVyZXQ=?=)
Date: Mon, 29 Jun 2015 11:26:57 +0200
Subject: Log backend name/ip
Message-ID: <55910F61.5000903@tengu.ch>

Hello,

I have a specific varnishncsa format, and I'm wanting to log the backend
used, especially when the request hits some director (either round_robin
or hash).

I think there would be two places in the VCL in order to log this
information:
* in vcl_backend_response
* in vcl_backend_error

Is this correct? If so, I seem to be unable to get the backend using, in
both cases, beresp.backend.ip

Here's how I do my stuff:

sub vcl_backend_response {
 std.log("backend_used:" + beresp.backend.ip);
 [? some other things?]
}
sub vcl_backend_error {
 std.log("backend_used:" + beresp.backend.ip);
 [? some other things?]
}

Then I fetch the entry using
"varnish.backend_name":"%{VCL_Log:backend_used}x"

in my varnishncsa formatted string (output as a JSON, there are many
other fields in there, of course).

Is this correct? Did I missed anything in order to ensure I actually get
the backend IP?

Thanks!

Cheers,

C.