From subharaj.manna at gmail.com  Mon Jan  4 06:56:17 2016
From: subharaj.manna at gmail.com (Debraj Manna)
Date: Mon, 4 Jan 2016 12:26:17 +0530
Subject: Recommended hardware for Varnish 4.1
Message-ID: <CAF6DVKN2=YKG=wfWXp8=qknXX=d+HvvoA02-zdSX2sCfJRYnNA@mail.gmail.com>

What is the recommended hardware for varnish (RAM, CPU cores, disk, etc)?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160104/0f845815/attachment.html>

From perbu at varnish-software.com  Mon Jan  4 10:24:49 2016
From: perbu at varnish-software.com (Per Buer)
Date: Mon, 4 Jan 2016 11:24:49 +0100
Subject: Recommended hardware for Varnish 4.1
In-Reply-To: <CAF6DVKN2=YKG=wfWXp8=qknXX=d+HvvoA02-zdSX2sCfJRYnNA@mail.gmail.com>
References: <CAF6DVKN2=YKG=wfWXp8=qknXX=d+HvvoA02-zdSX2sCfJRYnNA@mail.gmail.com>
Message-ID: <CAOXZevBkx4P6O58e6magxY_GnhR2qZ-hH2J=qfpUbQbHW-VKHQ@mail.gmail.com>

On Mon, Jan 4, 2016 at 7:56 AM, Debraj Manna <subharaj.manna at gmail.com>
wrote:

> What is the recommended hardware for varnish (RAM, CPU cores, disk, etc)?
>

Hi Debraj.

Lots of memory (depending on the amount of content you'll need to cache).
You don't need much CPU. Typically some sort of "green" CPU with low speed
is fine.

Per.


-- 
*Per Buer*
CTO | Varnish Software AS
Cell: +47 95839117
We Make Websites Fly!
www.varnish-software.com
<http://info.varnish-software.com/signature>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160104/a45625e8/attachment.html>

From subharaj.manna at gmail.com  Mon Jan  4 12:36:14 2016
From: subharaj.manna at gmail.com (Debraj Manna)
Date: Mon, 4 Jan 2016 18:06:14 +0530
Subject: Recommended hardware for Varnish 4.1
In-Reply-To: <CAOXZevBkx4P6O58e6magxY_GnhR2qZ-hH2J=qfpUbQbHW-VKHQ@mail.gmail.com>
References: <CAF6DVKN2=YKG=wfWXp8=qknXX=d+HvvoA02-zdSX2sCfJRYnNA@mail.gmail.com>
 <CAOXZevBkx4P6O58e6magxY_GnhR2qZ-hH2J=qfpUbQbHW-VKHQ@mail.gmail.com>
Message-ID: <CAF6DVKM1H+o4ryD2HRW8M=_dcpuLtPEXtX=fRG7qCO3CuZCuow@mail.gmail.com>

Thanks Per.

If I have a 64 GB RAM is there any recommendation on at max how much should
we add to malloc storage and how much should we left for other processes
and varnish.

On Mon, Jan 4, 2016 at 3:54 PM, Per Buer <perbu at varnish-software.com> wrote:

>
>
> On Mon, Jan 4, 2016 at 7:56 AM, Debraj Manna <subharaj.manna at gmail.com>
> wrote:
>
>> What is the recommended hardware for varnish (RAM, CPU cores, disk, etc)?
>>
>
> Hi Debraj.
>
> Lots of memory (depending on the amount of content you'll need to cache).
> You don't need much CPU. Typically some sort of "green" CPU with low speed
> is fine.
>
> Per.
>
>
> --
> *Per Buer*
> CTO | Varnish Software AS
> Cell: +47 95839117
> We Make Websites Fly!
> www.varnish-software.com
> <http://info.varnish-software.com/signature>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160104/73334865/attachment.html>

From sweeks at imvu.com  Tue Jan  5 01:16:50 2016
From: sweeks at imvu.com (Stephen Weeks)
Date: Mon, 4 Jan 2016 17:16:50 -0800
Subject: Dramatic memory use increase on varnish upgrade
Message-ID: <CAOkvGXvrW65DVY5xvUw70tMo7ZA6WYKVi6BJ4A49ZQO5FpEg7g@mail.gmail.com>

We have several varnishd instances that have been running fine for
years at constant memory use.  We've recently decided to upgrade our
linux distribution.  After some troubleshooting, we were unable to
find a configuration of Varnish on the new distro version that did not
have dramatically higher memory usage.  After that, we tried upgrading
to a modern, supported version of Varnish, but were also unable to
find a configuration that doesn't use dramatically more memory.  We
have tried the following configurations:

* Varnish 3.0.3, Ubuntu 10.04, Linux 3.2, 24G malloc storage, glibc
malloc, constant memory use at 29G rss
* Varnish 3.0.5, Ubuntu 14.04, Linux 3.19, 24G malloc storage,
jemalloc, memory usage grows until it passes 64G and starts swapping
* Varnish 4.1.0, Ubuntu 14.04, Linux 3.19, 24G malloc storage,
jemalloc, memory usage grows until it passes 64G and starts swapping
* Varnish 4.1.0, Ubuntu 14.04, Linux 3.19, 24G malloc storage, glibc
malloc, memory usage grows faster than with jemalloc until it passes
64G and starts swapping

I've reviewed varnishstat output, but I've been unable to account for
the increased memory usage.  I asked the IRC channel for support a few
weeks ago, and nobody else there was able to account for this in
varnishstat, and the only idea that we were able to come up with was
to try glibc malloc with Varnish 4.1, which as you can see above
turned out to be slightly worse.  I'm really out of ideas here; any
help in trying to find a configuration that has constrained memory
usage on a more-modern platform would be highly appreciated.

Details on the functional Varnish 3 instances with a steady 29G memory usage:

Command line: /usr/sbin/varnishd -P /var/run/varnishd.pid -a :3000 -T
localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p
sess_timeout 60 -p thread_pool_min 500 -p thread_pool_max 2000 -p
nuke_limit 1000 -s malloc,24G
varnishstat -1: http://sprunge.us/iKUC
vcl config: http://sprunge.us/LOUE
libraries linked: http://sprunge.us/FOSL

Details on the Varnish 3 instances with much higher memory usage:

Command line: /usr/sbin/varnishd -P /var/run/varnishd.pid -a :3000 -T
localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p
sess_timeout=60 -p thread_pool_min=500 -p thread_pool_max=2000 -p
nuke_limit=1000 -s malloc,24G
varnishstat -1: http://sprunge.us/KGIa
vcl config: Same config as above, http://sprunge.us/LOUE

Details on the varnish 4.1 instances with much higher memory usage:

Command line: /usr/sbin/varnishd -P /run/varnishd.pid -a :3000 -T
localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p
timeout_idle=60 -p thread_pool_min=500 -p thread_pool_max=2000 -p
nuke_limit=1000 -s malloc,24G
varnishstat -1: http://sprunge.us/gCOG
vcl config: http://sprunge.us/LQRM
libraries linked: http://sprunge.us/WCbj

Graph of memory usage for two varnish instances.  AF002245 is Varnish
3.0.3, AF002256 is Varnish 4.1.0: http://i.imgur.com/Fxsms48.png

The closest I've been able to find on Google is people talking about
accidentally using too much Transient space, but in varnishstat I see
<800k used in transient:

SMA.Transient.g_bytes                   766728          .   Bytes outstanding

They both have comparable numbers of objects cached:

n_object                419597          .   N struct object
MAIN.n_object                420371          .   object structs made

I'm pretty sure I've got to have something bad in my config, because
I'm seeing the same behaviour on both 3.0.5 and 4.1.0, and a memory
leak this bad for so long seems very unlikely.  I've considered trying
to narrow this down by porting forward exactly the same 3.0.3 version
to a slightly more modern platform, or trying to backport 4.1.0 back
to the ancient Ubuntu 10.04, but I'm not really sure what I'd do
differently if either of those worked, as I would rather not stay on
such an old platform if I can avoid it, in either case.

Are there any other metrics I can gather here?  Can anyone give me any
recommendations on how I can configure Varnish to constrain its memory
usage?  Any ideas about what I'm missing?



From feld at feld.me  Tue Jan  5 02:24:47 2016
From: feld at feld.me (Mark Felder)
Date: Mon, 04 Jan 2016 20:24:47 -0600
Subject: Dramatic memory use increase on varnish upgrade
In-Reply-To: <CAOkvGXvrW65DVY5xvUw70tMo7ZA6WYKVi6BJ4A49ZQO5FpEg7g@mail.gmail.com>
References: <CAOkvGXvrW65DVY5xvUw70tMo7ZA6WYKVi6BJ4A49ZQO5FpEg7g@mail.gmail.com>
Message-ID: <1451960687.591928.482984930.637F5C47@webmail.messagingengine.com>



On Mon, Jan 4, 2016, at 19:16, Stephen Weeks wrote:
> We have several varnishd instances that have been running fine for
> years at constant memory use.  We've recently decided to upgrade our
> linux distribution.  After some troubleshooting, we were unable to
> find a configuration of Varnish on the new distro version that did not
> have dramatically higher memory usage.  

I suspect Linux kernel's memory management changes are your problem, not
Varnish.

What is the version of the kernel on the instances that work well for
you?


-- 
  Mark Felder
  feld at feld.me



From sweeks at imvu.com  Tue Jan  5 02:28:04 2016
From: sweeks at imvu.com (Stephen Weeks)
Date: Mon, 4 Jan 2016 18:28:04 -0800
Subject: Dramatic memory use increase on varnish upgrade
In-Reply-To: <1451960687.591928.482984930.637F5C47@webmail.messagingengine.com>
References: <CAOkvGXvrW65DVY5xvUw70tMo7ZA6WYKVi6BJ4A49ZQO5FpEg7g@mail.gmail.com>
 <1451960687.591928.482984930.637F5C47@webmail.messagingengine.com>
Message-ID: <CAOkvGXttzZ_7NLUaLYMHXD3NRwOcHTkxG4LquU6QVcx8DuiFDw@mail.gmail.com>

On Jan 4, 2016 6:24 PM, "Mark Felder" <feld at feld.me> wrote:
>
>
>
> On Mon, Jan 4, 2016, at 19:16, Stephen Weeks wrote:
> > We have several varnishd instances that have been running fine for
> > years at constant memory use.  We've recently decided to upgrade our
> > linux distribution.  After some troubleshooting, we were unable to
> > find a configuration of Varnish on the new distro version that did not
> > have dramatically higher memory usage.
>
> I suspect Linux kernel's memory management changes are your problem, not
> Varnish.
>
> What is the version of the kernel on the instances that work well for
> you?

The instances with constant memory use are on Linux 3.2. All the
misbehaving instances have been on Linux 3.19.

Any ideas how to mitigate this or any Linux kernel configuration I could
change? Any ideas what kernel behavior changes could be responsible for
this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160104/e3b957b3/attachment.html>

From feld at feld.me  Tue Jan  5 02:44:18 2016
From: feld at feld.me (Mark Felder)
Date: Mon, 04 Jan 2016 20:44:18 -0600
Subject: Dramatic memory use increase on varnish upgrade
In-Reply-To: <1451960687.591928.482984930.637F5C47@webmail.messagingengine.com>
References: <CAOkvGXvrW65DVY5xvUw70tMo7ZA6WYKVi6BJ4A49ZQO5FpEg7g@mail.gmail.com>
 <1451960687.591928.482984930.637F5C47@webmail.messagingengine.com>
Message-ID: <1451961858.596504.482992810.40E152A9@webmail.messagingengine.com>



On Mon, Jan 4, 2016, at 20:24, Mark Felder wrote:
> 
> 
> On Mon, Jan 4, 2016, at 19:16, Stephen Weeks wrote:
> > We have several varnishd instances that have been running fine for
> > years at constant memory use.  We've recently decided to upgrade our
> > linux distribution.  After some troubleshooting, we were unable to
> > find a configuration of Varnish on the new distro version that did not
> > have dramatically higher memory usage.  
> 
> I suspect Linux kernel's memory management changes are your problem, not
> Varnish.
> 
> What is the version of the kernel on the instances that work well for
> you?
> 
> 

I don't have any expertise on kernels that new. My Linux knowledge ends
around 2.6.32 :-)

I'd probably look at Linux commits in git under mm/ which is where all
the memory management code is at and read through until I saw something
significant enough like some new knobs or a major overhaul that might be
worth trying to back out and test with a custom kernel.

If you have the time and the hardware I'd also try installing FreeBSD
and testing your varnish config just to see another data point on memory
usage. If it's considerably better, see if you can get a Linux kernel
developer's attention and point out that 3.2 and a different OS both
fare much, much better for your workload. They might have enough
expertise to track down a regression.

-- 
  Mark Felder
  feld at feld.me



From krishna.ku at flipkart.com  Tue Jan  5 03:49:15 2016
From: krishna.ku at flipkart.com (Krishna Kumar (Engineering))
Date: Tue, 5 Jan 2016 09:19:15 +0530
Subject: Dramatic memory use increase on varnish upgrade
In-Reply-To: <1451961858.596504.482992810.40E152A9@webmail.messagingengine.com>
References: <CAOkvGXvrW65DVY5xvUw70tMo7ZA6WYKVi6BJ4A49ZQO5FpEg7g@mail.gmail.com>
 <1451960687.591928.482984930.637F5C47@webmail.messagingengine.com>
 <1451961858.596504.482992810.40E152A9@webmail.messagingengine.com>
Message-ID: <CACycfVZ9W-i9jX0KphbyS0M7rkSB9PTCf1GBdiAi6ausGROAtg@mail.gmail.com>

How about compare the sysctl settings from the two systems?

On Tue, Jan 5, 2016 at 8:14 AM, Mark Felder <feld at feld.me> wrote:

>
>
> On Mon, Jan 4, 2016, at 20:24, Mark Felder wrote:
> >
> >
> > On Mon, Jan 4, 2016, at 19:16, Stephen Weeks wrote:
> > > We have several varnishd instances that have been running fine for
> > > years at constant memory use.  We've recently decided to upgrade our
> > > linux distribution.  After some troubleshooting, we were unable to
> > > find a configuration of Varnish on the new distro version that did not
> > > have dramatically higher memory usage.
> >
> > I suspect Linux kernel's memory management changes are your problem, not
> > Varnish.
> >
> > What is the version of the kernel on the instances that work well for
> > you?
> >
> >
>
> I don't have any expertise on kernels that new. My Linux knowledge ends
> around 2.6.32 :-)
>
> I'd probably look at Linux commits in git under mm/ which is where all
> the memory management code is at and read through until I saw something
> significant enough like some new knobs or a major overhaul that might be
> worth trying to back out and test with a custom kernel.
>
> If you have the time and the hardware I'd also try installing FreeBSD
> and testing your varnish config just to see another data point on memory
> usage. If it's considerably better, see if you can get a Linux kernel
> developer's attention and point out that 3.2 and a different OS both
> fare much, much better for your workload. They might have enough
> expertise to track down a regression.
>
> --
>   Mark Felder
>   feld at feld.me
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160105/af29d087/attachment.html>

From sweeks at imvu.com  Tue Jan  5 05:44:29 2016
From: sweeks at imvu.com (Stephen Weeks)
Date: Mon, 4 Jan 2016 21:44:29 -0800
Subject: Dramatic memory use increase on varnish upgrade
In-Reply-To: <CACycfVZ9W-i9jX0KphbyS0M7rkSB9PTCf1GBdiAi6ausGROAtg@mail.gmail.com>
References: <CAOkvGXvrW65DVY5xvUw70tMo7ZA6WYKVi6BJ4A49ZQO5FpEg7g@mail.gmail.com>
 <1451960687.591928.482984930.637F5C47@webmail.messagingengine.com>
 <1451961858.596504.482992810.40E152A9@webmail.messagingengine.com>
 <CACycfVZ9W-i9jX0KphbyS0M7rkSB9PTCf1GBdiAi6ausGROAtg@mail.gmail.com>
Message-ID: <CAOkvGXu7NsAheVek3uzk2vrY6Sh3cQ8=JYX25RSfK295e2UYHQ@mail.gmail.com>

We change very few settings in sysctl, and they're all the same
changes from default on both hosts, but here's what everything's
currently set to.  Maybe some defaults have changed.  I read through
the diff, but nothing jumped out at me as plausibly relevant:

sysctl on a 3.2 kernel host with constant memory use: http://sprunge.us/ThQV
sysctl on a 3.19 kernel host with excessively-growing memory use:
http://sprunge.us/EIcB
Diff of the two reports: http://sprunge.us/BfMY

All of our deployments so far are on Linux, and nobody here has any
experience with FreeBSD, so adding a new platform to support would be
a pretty big investment, but it might be worth investigating if it
would help get a kernel developer's attention.

On Mon, Jan 4, 2016 at 7:49 PM, Krishna Kumar (Engineering)
<krishna.ku at flipkart.com> wrote:
> How about compare the sysctl settings from the two systems?
>
> On Tue, Jan 5, 2016 at 8:14 AM, Mark Felder <feld at feld.me> wrote:
>>
>>
>>
>> On Mon, Jan 4, 2016, at 20:24, Mark Felder wrote:
>> >
>> >
>> > On Mon, Jan 4, 2016, at 19:16, Stephen Weeks wrote:
>> > > We have several varnishd instances that have been running fine for
>> > > years at constant memory use.  We've recently decided to upgrade our
>> > > linux distribution.  After some troubleshooting, we were unable to
>> > > find a configuration of Varnish on the new distro version that did not
>> > > have dramatically higher memory usage.
>> >
>> > I suspect Linux kernel's memory management changes are your problem, not
>> > Varnish.
>> >
>> > What is the version of the kernel on the instances that work well for
>> > you?
>> >
>> >
>>
>> I don't have any expertise on kernels that new. My Linux knowledge ends
>> around 2.6.32 :-)
>>
>> I'd probably look at Linux commits in git under mm/ which is where all
>> the memory management code is at and read through until I saw something
>> significant enough like some new knobs or a major overhaul that might be
>> worth trying to back out and test with a custom kernel.
>>
>> If you have the time and the hardware I'd also try installing FreeBSD
>> and testing your varnish config just to see another data point on memory
>> usage. If it's considerably better, see if you can get a Linux kernel
>> developer's attention and point out that 3.2 and a different OS both
>> fare much, much better for your workload. They might have enough
>> expertise to track down a regression.
>>
>> --
>>   Mark Felder
>>   feld at feld.me
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>



From subharaj.manna at gmail.com  Wed Jan  6 05:41:39 2016
From: subharaj.manna at gmail.com (Debraj Manna)
Date: Wed, 6 Jan 2016 11:11:39 +0530
Subject: Varnish Log Doubts
Message-ID: <CAF6DVKP1ZAJDcGtgyvBO_B3b-eU6mPQRC3aH+0bZwv0s+JDGMw@mail.gmail.com>

Hi,

After going through this article
<https://www.varnish-cache.org/docs/trunk/users-guide/troubleshooting.html>
I am having the following doubts. Can some one help me to clarify this:-



   1. As per the varnish ebook varnishncsa -displays Varnish access logs &
   varnishlog is used to access request-specific data. Can some let me know
   what logs then go in /var/log/messages & /var/log/syslog ?
   2. When varnish crashes due to panic where will varnish dump the
   logs/traces? I know panic.show shows the panic messages in cli. Is there
   a way I can view the panic messages in some files?
   3. When varnish crashes due to segfault where will varnish puts the
   messages and core dump?
   4. How can I dump the 503 / Guru Meditation errors in a file?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160106/4802ff88/attachment.html>

From goncalo.silva at muzzley.com  Wed Jan  6 10:43:53 2016
From: goncalo.silva at muzzley.com (=?UTF-8?Q?Gon=C3=A7alo_Silva?=)
Date: Wed, 6 Jan 2016 10:43:53 +0000
Subject: chunked trailer server-authorization
Message-ID: <CADOexLqnVBWy2vu0ea6-QDaGkJKuEgFGHnw7XMk6u-_N9jifEQ@mail.gmail.com>

Hello,

I'm having problems getting varnish processing this kind of requests.
If i do the request direct to the backend i have no problem and this is the
result:

------
root at Reverse:~# curl --raw -v -X "GET" -H 'Authorization: Hawk id="xxxx",
ts="xxxxx", nonce="xxxxx", mac="akA+pAge015LbAq+fq3YPG9PM0EgSl3/xxxxxx="'
-H 'Host: myhostbackend.com'
http://myhostbackend.com:4050/profiles/89a12688-8653-4f76-bb18-xxxxx
* Hostname was NOT found in DNS cache
*   Trying myhostbackend.com...
* Connected to myhostbackend.com (xx.xx.xx.xx) port 4050 (#0)
> GET /profiles/89a12688-8653-4f76-bb18-xxxxxxx HTTP/1.1
> User-Agent: curl/7.35.0
> Accept: */*
> Authorization: Hawk id="xxxx", ts="xxxxx", nonce="xxxxx",
mac="akA+pAge015LbAq+fq3YPG9PM0EgSl3/xxxxxx="
> Host: myhostbackend.com
>
< HTTP/1.1 200 OK
< media-type: application/json;v=v0
< content-type: application/json; charset=utf-8
< cache-control: no-cache
< trailer: server-authorization
< transfer-encoding: chunked
< Date: Tue, 05 Jan 2016 17:17:07 GMT
< Connection: keep-alive
<
228
{"id":"xxxxxxxx","uuid":"89a12688-8653-xxxxxxx-xxxx","kind":"device","name":"apagar","provider":"apagarrr","resourceUrl":"
http://xxx.com/channels","authorizationUrl":"http://xxx.com/authorization
","subscriptionUrl":"http://xxx.com/subscriptions","photoUrl":"
http://xxx.com/images/profile.jpg
","requiredCapabilityVersion":1,"accessList":[],"auth":{"inbound":{},"outbound":{}},"globalManagerAccess":false,"openOauthInBrowser":false}
0
server-authorization: Hawk
mac="RuMz8Rm1DAq3Uuq1HSz/IjCidjn/R73+LnX6pntIKX4=",
hash="vrcvy0YWrI4CO5qMmGP6GLgGy3VexSFvjXdJGeYzBOQ="

* Connection #0 to host xxx.xxx.xxx.xxx left intact

-----

if i do the request via varnish, i'm not getting all the response, and
don't how why but the http header "trailer: server-authorization" doesn't
shows up. I can see "FetchError chunked tail no NL" on the varnish log, but
not sure what it means in this situation.

----
root at Reverse:~# curl --raw -v -X "GET" -H 'Authorization: Hawk id="xxxx",
ts="xxxxx", nonce="xxxxx", mac="akA+pAge015LbAq+fq3YPG9PM0EgSl3/xxxxxx="'
-H 'Host: myhostbackend.com'
http://127.0.0.1:6081/profiles/89a12688-8653-4f76-bb18-xxxxx
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 6081 (#0)
> GET /profiles/89a12688-8653-4f76-bb18-xxxxxxx HTTP/1.1
> User-Agent: curl/7.35.0
> Accept: */*
> Authorization: Hawk id="xxxx", ts="xxxxx", nonce="xxxxx",
mac="akA+pAge015LbAq+fq3YPG9PM0EgSl3/xxxxxx="
> Host: myhostbackend.com

 HTTP/1.1 200 OK
< media-type: application/json;v=v0
< content-type: application/json; charset=utf-8
< cache-control: no-cache
< Date: Tue, 05 Jan 2016 17:14:54 GMT
< X-Varnish: 32773
< Age: 0
< Via: 1.1 varnish-v4
< Accept-Ranges: bytes
< Transfer-Encoding: chunked
< Connection: keep-alive

<
00228
{"id":"xxxxxxxx","uuid":"89a12688-8653-xxxxxxx-xxxx","kind":"device","name":"apagar","provider":"apagarrr","resourceUrl":"
http://xxx.com/channels","authorizationUrl":"http://xxx.com/authorization
","subscriptionUrl":"http://xxx.com/subscriptions","photoUrl":"
http://xxx.com/images/profile.jpg
","requiredCapabilityVersion":1,"accessList":[],"auth":{"inbound":{},"outbound":{}},"globalManagerAccess":false,"openOauthInBrowser":false}
* transfer closed with outstanding read data remaining
* Closing connection 0
--------

The log from varnish is:

*   << BeReq    >> 32774
-   Begin          bereq 32773 pass
-   Timestamp      Start: 1452014094.207771 0.000000 0.000000
-   BereqMethod    GET
-   BereqURL       /profiles/89a12688-8653-4f76-bb18-3bfe2b829400
-   BereqProtocol  HTTP/1.1
-   BereqHeader    User-Agent: curl/7.35.0
-   BereqHeader    Accept: */*
-   BereqHeader    Authorization: Hawk idAuthorization: Hawk id="xxxx",
ts="xxxxx", nonce="xxxxx", mac="akA+pAge015LbAq+fq3YPG9PM0EgSl3/xxxxxx="
-   BereqHeader    Host: myhostbackend.com
-   BereqHeader    X-Forwarded-For: 127.0.0.1
-   BereqHeader    X-Varnish: 32774
-   VCL_call       BACKEND_FETCH
-   VCL_return     fetch
-   BackendOpen    18 boot.apiserver xxx.xxx.xxx.xxx 4050 xxx.xxx.xxx.xxx
34845
-   Timestamp      Bereq: 1452014094.208532 0.000761 0.000761
-   Timestamp      Beresp: 1452014094.220400 0.012629 0.011868
-   BerespProtocol HTTP/1.1
-   BerespStatus   200
-   BerespReason   OK
-   BerespHeader   media-type: application/json;v=v0
-   BerespHeader   content-type: application/json; charset=utf-8
-   BerespHeader   cache-control: no-cache
-   BerespHeader   trailer: server-authorization
-   BerespHeader   transfer-encoding: chunked
-   BerespHeader   Date: Tue, 05 Jan 2016 17:14:54 GMT
-   BerespHeader   Connection: keep-alive
-   TTL            RFC 120 10 -1 1452014094 1452014094 1452014094 0 0
-   VCL_call       BACKEND_RESPONSE
-   TTL            VCL 120 10 0 1452014094
-   VCL_return     deliver
-   Storage        malloc Transient
-   ObjProtocol    HTTP/1.1
-   ObjStatus      200
-   ObjReason      OK
-   ObjHeader      media-type: application/json;v=v0
-   ObjHeader      content-type: application/json; charset=utf-8
-   ObjHeader      cache-control: no-cache
-   ObjHeader      Date: Tue, 05 Jan 2016 17:14:54 GMT
-   Fetch_Body     2 chunked stream
-   FetchError     chunked tail no NL
-   BackendClose   18 boot.apiserver
-   BereqAcct      310 0 310 246 552 798
-   End


If i put the varnish configuration using pipe for this requests, everything
works, but of course i can't cache them.
Can anyone understands what is going on here and if this is a configuration
problem or something more deeper?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160106/8b4ce79c/attachment.html>

From bertrand.caplet at chunkz.net  Wed Jan  6 18:54:03 2016
From: bertrand.caplet at chunkz.net (Bertrand Caplet)
Date: Wed, 6 Jan 2016 19:54:03 +0100
Subject: Varnish on ubuntu, malloc configuration
Message-ID: <568D62CB.1000804@chunkz.net>

Hi all,
I have installed varnish a while ago on a ubuntu server but now I need
varnish to use less memory.
I did configure in /etc/default/varnish the DAEMON_OPTS to :
"-a :6081 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -p thread_pools= 8 \
             -p thread_pool_max=50 \
             -s malloc,128m"
but varnish is still configured with malloc,256m.
I also tried to change directly configuration in /etc/init.d/varnish but
same problem.

Do you have a clue ?

Thanks in advance,

-- 
CHUNKZ.NET - script kiddie and computer technician
Bertrand Caplet, Flers (FR)
Feel free to send encrypted/signed messages
Key ID: 6E494EB9
GPG FP: CB1B 664A 9165 98F8 459F 0807 CA35 B76F 6E49 4EB9


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160106/13a50977/attachment.bin>

From guery.b at gmail.com  Wed Jan  6 20:49:07 2016
From: guery.b at gmail.com (=?UTF-8?Q?Boris_Gu=c3=a9ry?=)
Date: Wed, 6 Jan 2016 21:49:07 +0100
Subject: Varnish on ubuntu, malloc configuration
In-Reply-To: <568D62CB.1000804@chunkz.net>
References: <568D62CB.1000804@chunkz.net>
Message-ID: <568D7DC3.8010705@gmail.com>

Try adding `set -x` to your `/etc/init.d/varnish` and restart it to
have a better clue of where values are from and if there are any
sourced files.

Most likely `DAEMON_OPTS` are overridden somewhere.

Le 06/01/2016 19:54, Bertrand Caplet a ?crit :
> Hi all, I have installed varnish a while ago on a ubuntu server but
> now I need varnish to use less memory. I did configure in
> /etc/default/varnish the DAEMON_OPTS to : "-a :6081 \ -T
> localhost:6082 \ -f /etc/varnish/default.vcl \ -S
> /etc/varnish/secret \ -p thread_pools= 8 \ -p thread_pool_max=50 \ 
> -s malloc,128m" but varnish is still configured with malloc,256m. I
> also tried to change directly configuration in /etc/init.d/varnish
> but same problem.
> 
> Do you have a clue ?
> 
> Thanks in advance,
> 
> 
> 
> _______________________________________________ varnish-misc
> mailing list varnish-misc at varnish-cache.org 
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> 



-- 
Boris Gu?ry

twitter: @borisguery
mobile:  +33 6 86 83 03 12
skype:   borisguery
pgp:     0x034C6265



From japrice at gmail.com  Thu Jan  7 02:05:52 2016
From: japrice at gmail.com (Jason Price)
Date: Wed, 6 Jan 2016 21:05:52 -0500
Subject: Varnish Log Doubts
In-Reply-To: <CAF6DVKP1ZAJDcGtgyvBO_B3b-eU6mPQRC3aH+0bZwv0s+JDGMw@mail.gmail.com>
References: <CAF6DVKP1ZAJDcGtgyvBO_B3b-eU6mPQRC3aH+0bZwv0s+JDGMw@mail.gmail.com>
Message-ID: <CAChvjRDxE4zCRTwgT8S8LiK9cYHVHW0zrojWPQ4qFeZwjzueZg@mail.gmail.com>

1. Any messages generated with std.syslog(<reasonable_integer>, "...") will
go to /var/log/messages.  There are a few in default.vcl IIRC.
2. You have to look for them in varnishadm panic.show.  It leaves a fair
bit of data in /var/log/messages too.
3.  It won't dump core. The watchdog will restart it.  See also #2
4.  Guru meditations: Look at the default.vcl for the vcl_error function.
Anything funneled through there can trigger a guru meditation.  Clone the
function to your vcl, end it with a 'return(deliver)', and add your logging
there.  Varnish originated 503's come through here too.

-Jason

On Wed, Jan 6, 2016 at 12:41 AM, Debraj Manna <subharaj.manna at gmail.com>
wrote:

> Hi,
>
> After going through this article
> <https://www.varnish-cache.org/docs/trunk/users-guide/troubleshooting.html>
> I am having the following doubts. Can some one help me to clarify this:-
>
>
>
>    1. As per the varnish ebook varnishncsa -displays Varnish access logs
>    & varnishlog is used to access request-specific data. Can some let me
>    know what logs then go in /var/log/messages & /var/log/syslog ?
>    2. When varnish crashes due to panic where will varnish dump the
>    logs/traces? I know panic.show shows the panic messages in cli. Is
>    there a way I can view the panic messages in some files?
>    3. When varnish crashes due to segfault where will varnish puts the
>    messages and core dump?
>    4. How can I dump the 503 / Guru Meditation errors in a file?
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160106/b8db295e/attachment.html>

From perbu at varnish-software.com  Thu Jan  7 08:23:11 2016
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 7 Jan 2016 09:23:11 +0100
Subject: Varnish Log Doubts
In-Reply-To: <CAF6DVKP1ZAJDcGtgyvBO_B3b-eU6mPQRC3aH+0bZwv0s+JDGMw@mail.gmail.com>
References: <CAF6DVKP1ZAJDcGtgyvBO_B3b-eU6mPQRC3aH+0bZwv0s+JDGMw@mail.gmail.com>
Message-ID: <CAOXZevBFDpL_tXcW-+GJPEHVVqd4iv8ON2KtwC1FTgSZxfOrpw@mail.gmail.com>

On Wed, Jan 6, 2016 at 6:41 AM, Debraj Manna <subharaj.manna at gmail.com>
wrote:

> Hi,
>
> After going through this article
> <https://www.varnish-cache.org/docs/trunk/users-guide/troubleshooting.html>
> I am having the following doubts. Can some one help me to clarify this:-
>
>  4. How can I dump the 503 / Guru Meditation errors in a file?
>


What part of the documentation was unclear? Do you know how to redirect to
a file?

-- 
*Per Buer*
CTO | Varnish Software AS
Cell: +47 95839117
We Make Websites Fly!
www.varnish-software.com
<http://info.varnish-software.com/signature>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160107/cab4a385/attachment.html>

From subharaj.manna at gmail.com  Thu Jan  7 08:54:55 2016
From: subharaj.manna at gmail.com (Debraj Manna)
Date: Thu, 7 Jan 2016 14:24:55 +0530
Subject: Varnish Log Doubts
In-Reply-To: <CAOXZevBFDpL_tXcW-+GJPEHVVqd4iv8ON2KtwC1FTgSZxfOrpw@mail.gmail.com>
References: <CAF6DVKP1ZAJDcGtgyvBO_B3b-eU6mPQRC3aH+0bZwv0s+JDGMw@mail.gmail.com>
 <CAOXZevBFDpL_tXcW-+GJPEHVVqd4iv8ON2KtwC1FTgSZxfOrpw@mail.gmail.com>
Message-ID: <CAF6DVKN1y_Bjx-OUHtmGswH4X7cq+JqCQz1b8A4GrMCnMe=syQ@mail.gmail.com>

By redirecting to file you mean enabling VARNISHNCSA_ENABLED &
VARNISHLOG_ENABLED in /var/default/varnishncsa & /var/default/varnishlog
respectively.

As Jason suggested right now using std.syslog I am adding some custom log
messages to /var/log/messages.

sub vcl_backend_error {
   std.syslog(180, "error: Backend "  + bereq.backend + " Url " + bereq.url
+ " Status " + beresp.status + " Reason " + beresp.reason);
}

Is it possible to redirect the custom logs to some other file?

On Thu, Jan 7, 2016 at 1:53 PM, Per Buer <perbu at varnish-software.com> wrote:

> On Wed, Jan 6, 2016 at 6:41 AM, Debraj Manna <subharaj.manna at gmail.com>
> wrote:
>
>> Hi,
>>
>> After going through this article
>> <https://www.varnish-cache.org/docs/trunk/users-guide/troubleshooting.html>
>> I am having the following doubts. Can some one help me to clarify this:-
>>
>>  4. How can I dump the 503 / Guru Meditation errors in a file?
>>
>
>
> What part of the documentation was unclear? Do you know how to redirect to
> a file?
>
> --
> *Per Buer*
> CTO | Varnish Software AS
> Cell: +47 95839117
> We Make Websites Fly!
> www.varnish-software.com
> <http://info.varnish-software.com/signature>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160107/83a34774/attachment.html>

From perbu at varnish-software.com  Thu Jan  7 08:56:39 2016
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 7 Jan 2016 09:56:39 +0100
Subject: Varnish Log Doubts
In-Reply-To: <CAF6DVKN1y_Bjx-OUHtmGswH4X7cq+JqCQz1b8A4GrMCnMe=syQ@mail.gmail.com>
References: <CAF6DVKP1ZAJDcGtgyvBO_B3b-eU6mPQRC3aH+0bZwv0s+JDGMw@mail.gmail.com>
 <CAOXZevBFDpL_tXcW-+GJPEHVVqd4iv8ON2KtwC1FTgSZxfOrpw@mail.gmail.com>
 <CAF6DVKN1y_Bjx-OUHtmGswH4X7cq+JqCQz1b8A4GrMCnMe=syQ@mail.gmail.com>
Message-ID: <CAOXZevCGFQMDPx_uUE89YP0twethupBrifwvoqpxhF7WjgDeXA@mail.gmail.com>

No, by redirecting to a file I mean using the < and > operators in the
shell. To redirect output from the the varnishlog command (the one listed
in the docs) to a file.

On Thu, Jan 7, 2016 at 9:54 AM, Debraj Manna <subharaj.manna at gmail.com>
wrote:

> By redirecting to file you mean enabling VARNISHNCSA_ENABLED &
> VARNISHLOG_ENABLED in /var/default/varnishncsa & /var/default/varnishlog
> respectively.
>
> As Jason suggested right now using std.syslog I am adding some custom log
> messages to /var/log/messages.
>
> sub vcl_backend_error {
>    std.syslog(180, "error: Backend "  + bereq.backend + " Url " +
> bereq.url + " Status " + beresp.status + " Reason " + beresp.reason);
> }
>
> Is it possible to redirect the custom logs to some other file?
>
> On Thu, Jan 7, 2016 at 1:53 PM, Per Buer <perbu at varnish-software.com>
> wrote:
>
>> On Wed, Jan 6, 2016 at 6:41 AM, Debraj Manna <subharaj.manna at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> After going through this article
>>> <https://www.varnish-cache.org/docs/trunk/users-guide/troubleshooting.html>
>>> I am having the following doubts. Can some one help me to clarify this:-
>>>
>>>  4. How can I dump the 503 / Guru Meditation errors in a file?
>>>
>>
>>
>> What part of the documentation was unclear? Do you know how to redirect
>> to a file?
>>
>> --
>> *Per Buer*
>> CTO | Varnish Software AS
>> Cell: +47 95839117
>> We Make Websites Fly!
>> www.varnish-software.com
>> <http://info.varnish-software.com/signature>
>>
>
>


-- 
*Per Buer*
CTO | Varnish Software AS
Cell: +47 95839117
We Make Websites Fly!
www.varnish-software.com
<http://info.varnish-software.com/signature>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160107/45946ee4/attachment.html>

From subharaj.manna at gmail.com  Thu Jan  7 13:01:37 2016
From: subharaj.manna at gmail.com (Debraj Manna)
Date: Thu, 7 Jan 2016 18:31:37 +0530
Subject: Varnish 4 Not Obeying no-store
Message-ID: <CAF6DVKPta4xw=ig1trAZNwzEY2HDqKrZEGVCHmkcPMzjhobbjA@mail.gmail.com>

Hi,

I have a response in which no-store is set. But varnish is still caching
the response for default 2 mins.

root at SPk-D-0534:/var/log# curl -i "localhost:8080"
HTTP/1.1 200 OK
Cache-Control: no-store
Date: Thu, 07 Jan 2016 12:59:38 GMT
Content-Length: 22
Content-Type: text/plain; charset=utf-8
X-Varnish: 32777 32775
Age: 5
Via: 1.1 varnish-v4
Accept-Ranges: bytes
Connection: keep-alive

Hello World No Cached


By vcl_backend_response looks something like below:-

sub vcl_backend_response {
    if (bereq.url ~ "^/rde_server/") {
       set beresp.ttl = 300s;
       return(deliver);
    }
    if(beresp.http.Cache-Control ~ "no-store|no-cache") {
       return(deliver);
    }
    unset beresp.http.Cache-Control;
}

Can some one let me know what is going wrong or I have to do some special
handling in vcl so that varnish does not cache the response with no-store
set in Cache-Control?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160107/b55e60a2/attachment.html>

From perbu at varnish-software.com  Thu Jan  7 13:24:27 2016
From: perbu at varnish-software.com (Per Buer)
Date: Thu, 7 Jan 2016 14:24:27 +0100
Subject: Varnish 4 Not Obeying no-store
In-Reply-To: <CAF6DVKPta4xw=ig1trAZNwzEY2HDqKrZEGVCHmkcPMzjhobbjA@mail.gmail.com>
References: <CAF6DVKPta4xw=ig1trAZNwzEY2HDqKrZEGVCHmkcPMzjhobbjA@mail.gmail.com>
Message-ID: <CAOXZevCTpS7yyitO24AR-wdL_++WJaCTspGLYJeSia3at7Lz-w@mail.gmail.com>

On Thu, Jan 7, 2016 at 2:01 PM, Debraj Manna <subharaj.manna at gmail.com>
wrote:

> Hi,
>
> I have a response in which no-store is set. But varnish is still caching
> the response for default 2 mins.
> (..)
> By vcl_backend_response looks something like below:-
>
> sub vcl_backend_response {
>     if (bereq.url ~ "^/rde_server/") {
>        set beresp.ttl = 300s;
>        return(deliver);
>     }
>     if(beresp.http.Cache-Control ~ "no-store|no-cache") {
>        return(deliver);
>     }
>     unset beresp.http.Cache-Control;
> }
>
> Can some one let me know what is going wrong or I have to do some special
> handling in vcl so that varnish does not cache the response with no-store
> set in Cache-Control?
>

You are using the wrong action. return(deliver) doesn't mark the object as
uncacheable. It only marks it as OK for delivery to the client.

Do something like this:

if(beresp.http.Cache-Control ~ "no-store|no-cache") {
   set beresp.ttl = 120s;
   set beresp.uncacheable = true;
   (..)


This will create a hit-for-pass object to stop serialised access to the URL
and

-- 
*Per Buer*
CTO | Varnish Software AS
Cell: +47 95839117
We Make Websites Fly!
www.varnish-software.com
<http://info.varnish-software.com/signature>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160107/c30f4760/attachment.html>

From t.honacker at googlemail.com  Thu Jan  7 13:28:54 2016
From: t.honacker at googlemail.com (Tobias Honacker)
Date: Thu, 7 Jan 2016 14:28:54 +0100
Subject: Varnish 4 Not Obeying no-store
In-Reply-To: <CAF6DVKPta4xw=ig1trAZNwzEY2HDqKrZEGVCHmkcPMzjhobbjA@mail.gmail.com>
References: <CAF6DVKPta4xw=ig1trAZNwzEY2HDqKrZEGVCHmkcPMzjhobbjA@mail.gmail.com>
Message-ID: <CADepKeA_j6iq=imVWHR3cknq3dCj780FAEZHkWJHkuPtQNLzmQ@mail.gmail.com>

You have to add beresp.uncacheable. Try this:

--snip--

    if(beresp.http.Cache-Control ~ "no-store|no-cache") {
         set beresp.uncacheable = true;
        set beresp.ttl = 120s;
       return(deliver);
    }

--snap--

2016-01-07 14:01 GMT+01:00 Debraj Manna <subharaj.manna at gmail.com>:

> Hi,
>
> I have a response in which no-store is set. But varnish is still caching
> the response for default 2 mins.
>
> root at SPk-D-0534:/var/log# curl -i "localhost:8080"
> HTTP/1.1 200 OK
> Cache-Control: no-store
> Date: Thu, 07 Jan 2016 12:59:38 GMT
> Content-Length: 22
> Content-Type: text/plain; charset=utf-8
> X-Varnish: 32777 32775
> Age: 5
> Via: 1.1 varnish-v4
> Accept-Ranges: bytes
> Connection: keep-alive
>
> Hello World No Cached
>
>
> By vcl_backend_response looks something like below:-
>
> sub vcl_backend_response {
>     if (bereq.url ~ "^/rde_server/") {
>        set beresp.ttl = 300s;
>        return(deliver);
>     }
>     if(beresp.http.Cache-Control ~ "no-store|no-cache") {
>        return(deliver);
>     }
>     unset beresp.http.Cache-Control;
> }
>
> Can some one let me know what is going wrong or I have to do some special
> handling in vcl so that varnish does not cache the response with no-store
> set in Cache-Control?
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160107/38def971/attachment.html>

From uxbod at splatnix.net  Fri Jan  8 08:46:25 2016
From: uxbod at splatnix.net (Phil Daws)
Date: Fri, 8 Jan 2016 08:46:25 +0000 (GMT)
Subject: Varnish 4.1 to HTTP/S backend
Message-ID: <169017286.3892311.1452242785141.JavaMail.zimbra@innovot.com>

Hello:

I read that Varnish 4.1 is now able to pull from a HTTP/S backend but could not find any examples of how to do that; is that correct ?

Thanks, Phil





From guillaume at varnish-software.com  Fri Jan  8 08:57:37 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 8 Jan 2016 09:57:37 +0100
Subject: Varnish 4.1 to HTTP/S backend
In-Reply-To: <169017286.3892311.1452242785141.JavaMail.zimbra@innovot.com>
References: <169017286.3892311.1452242785141.JavaMail.zimbra@innovot.com>
Message-ID: <CAJ6ZYQx2WE7P3T2QNTJLBYTehpAC3HJzZr-kFyV7TgT53AZzuQ@mail.gmail.com>

Hi Phil,

It's not :-)

Using the PROXY protocol, clients can talk using HTTP/S to your varnish
box, but HTTP/S to backend is a Varnish Cache Plus exclusive for now.

-- 
Guillaume Quintard

On Fri, Jan 8, 2016 at 9:46 AM, Phil Daws <uxbod at splatnix.net> wrote:

> Hello:
>
> I read that Varnish 4.1 is now able to pull from a HTTP/S backend but
> could not find any examples of how to do that; is that correct ?
>
> Thanks, Phil
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160108/eefd664f/attachment.html>

From perbu at varnish-software.com  Fri Jan  8 09:06:10 2016
From: perbu at varnish-software.com (Per Buer)
Date: Fri, 8 Jan 2016 10:06:10 +0100
Subject: Varnish 4.1 to HTTP/S backend
In-Reply-To: <169017286.3892311.1452242785141.JavaMail.zimbra@innovot.com>
References: <169017286.3892311.1452242785141.JavaMail.zimbra@innovot.com>
Message-ID: <CAOXZevCv06Aq12m_jazLzdkc2XXDq414pis9-fFb3CKnE5h5bw@mail.gmail.com>

Hi Phil.

On Fri, Jan 8, 2016 at 9:46 AM, Phil Daws <uxbod at splatnix.net> wrote:

I read that Varnish 4.1 is now able to pull from a HTTP/S backend but could
> not find any examples of how to do that; is that correct ?
>

Unfortunately not. It should be possible to write a https VMOD now, it
might be quite cumbersome to write (the internal APIs are not all in place)
but nobody has picked up the glove (yet). If I recall correctly you might
not be able to reuse the internal HTTP client inside varnishd when writing
the VMOD which means you'll have to use an external https capable library
and redo a lot of the backend semantics. But I haven't looked into this in
a while so take all this with a grain of salt. If somebody know more I'm
sure I'll be corrected or confirmed.

Cheers,

Per.

-- 
*Per Buer*
CTO | Varnish Software AS
Cell: +47 95839117
We Make Websites Fly!
www.varnish-software.com
<http://info.varnish-software.com/signature>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160108/8c1a927c/attachment.html>

From dridi at varni.sh  Fri Jan  8 09:54:15 2016
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 8 Jan 2016 10:54:15 +0100
Subject: Varnish 4.1 to HTTP/S backend
In-Reply-To: <CAOXZevCv06Aq12m_jazLzdkc2XXDq414pis9-fFb3CKnE5h5bw@mail.gmail.com>
References: <169017286.3892311.1452242785141.JavaMail.zimbra@innovot.com>
 <CAOXZevCv06Aq12m_jazLzdkc2XXDq414pis9-fFb3CKnE5h5bw@mail.gmail.com>
Message-ID: <CABoVN9CqXuKq8=MAxb7KS6yE5YAaP=gEmiA3i-h-ZAVh5Jf6Hg@mail.gmail.com>

> Unfortunately not. It should be possible to write a https VMOD now, it might be quite cumbersome to write (the internal APIs are not all in place) but nobody has picked up the glove (yet). If I recall correctly you might not be able to reuse the internal HTTP client inside varnishd when writing the VMOD which means you'll have to use an external https capable library and redo a lot of the backend semantics. But I haven't looked into this in a while so take all this with a grain of salt. If somebody know more I'm sure I'll be corrected or confirmed.

Sounds accurate to me. It can be done but it's a lot of work partly
because we can't reuse the native HTTP backends.



From sahmed1020 at gmail.com  Fri Jan  8 22:21:21 2016
From: sahmed1020 at gmail.com (S Ahmed)
Date: Fri, 8 Jan 2016 17:21:21 -0500
Subject: invalidating cache via http
Message-ID: <CAG2rwuPTV+hTaeBUHcnqYVS7qPTAanR75YYJuGcU_3dU1HDj9g@mail.gmail.com>

Hello,

I recall reading that you can invalidate a given cache via an http request
to varnish, is this correct?

Given the above, would it be practical if I was to invalid at the rate of
100 requests per second?


Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160108/f4daf469/attachment.html>

From pprocacci at datapipe.com  Fri Jan  8 23:04:23 2016
From: pprocacci at datapipe.com (Paul A. Procacci)
Date: Fri, 8 Jan 2016 18:04:23 -0500
Subject: invalidating cache via http
In-Reply-To: <CAG2rwuPTV+hTaeBUHcnqYVS7qPTAanR75YYJuGcU_3dU1HDj9g@mail.gmail.com>
References: <CAG2rwuPTV+hTaeBUHcnqYVS7qPTAanR75YYJuGcU_3dU1HDj9g@mail.gmail.com>
Message-ID: <20160108230423.GA55258@workvm.myhome>


On Fri, Jan 08, 2016 at 05:21:21PM -0500, S Ahmed wrote:
>    Hello,
>    I recall reading that you can invalidate a given cache via an http
>    request to varnish, is this correct?
>    Given the above, would it be practical if I was to invalid at the rate
>    of 100 requests per second?
>    Thanks!

You can PURGE the cache as frequently as you like via http requests.
100 times a second almost begs the question though....why cache in the first
place?

~Paul



From infos at opendoc.net  Mon Jan 11 16:08:50 2016
From: infos at opendoc.net (Alexandre)
Date: Mon, 11 Jan 2016 17:08:50 +0100
Subject: Varnish and memory limitation
Message-ID: <5693D392.2050407@opendoc.net>

Hello everyone,

I have a varnish 3 I have to migrate Varnish in February. Currently, I 
need to use more than 64 GB of memory. Is it possible to use more than 
64 GB of memory with Varnish ?

Best regards

Alexandre.



From japrice at gmail.com  Tue Jan 12 13:31:09 2016
From: japrice at gmail.com (Jason Price)
Date: Tue, 12 Jan 2016 08:31:09 -0500
Subject: Varnish and memory limitation
In-Reply-To: <5693D392.2050407@opendoc.net>
References: <5693D392.2050407@opendoc.net>
Message-ID: <CAChvjRCxx50kNgSKbjb27Wh5S+_arzCV7FKx_ZXtgREbvc+SaA@mail.gmail.com>

Yes.  More than 64 gb malloc size works fine.  I was using 150gb with no
issue.

Fair warning through:  varnish will use more ram than the malloc size.
That really is just the total size of all cached objects.  There's about 1k
of overhead per object to deal with.  So the Resident Set Size will
probably be higher than your malloc size.

(Also: if you have very short lived objects (10 seconds by default) that
doesn't accrue into the 'malloc' size either... they stack in an unbounded
'transient' storage by default.  I got bitten by caching errors for 5
seconds with this default)

-Jason

On Mon, Jan 11, 2016 at 11:08 AM, Alexandre <infos at opendoc.net> wrote:

> Hello everyone,
>
> I have a varnish 3 I have to migrate Varnish in February. Currently, I
> need to use more than 64 GB of memory. Is it possible to use more than 64
> GB of memory with Varnish ?
>
> Best regards
>
> Alexandre.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160112/17d76fd2/attachment.html>

From infos at opendoc.net  Tue Jan 12 14:27:08 2016
From: infos at opendoc.net (Alexandre)
Date: Tue, 12 Jan 2016 15:27:08 +0100
Subject: Varnish and memory limitation
In-Reply-To: <CAChvjRCxx50kNgSKbjb27Wh5S+_arzCV7FKx_ZXtgREbvc+SaA@mail.gmail.com>
References: <5693D392.2050407@opendoc.net>
 <CAChvjRCxx50kNgSKbjb27Wh5S+_arzCV7FKx_ZXtgREbvc+SaA@mail.gmail.com>
Message-ID: <56950D3C.2000907@opendoc.net>

Thank you very much for this precious information. I will use 64 GB malloc.

Regards,
Alexandre.

On 12/01/16 14:31, Jason Price wrote:
> Yes.  More than 64 gb malloc size works fine.  I was using 150gb with no
> issue.
>
> Fair warning through:  varnish will use more ram than the malloc size.
> That really is just the total size of all cached objects.  There's about
> 1k of overhead per object to deal with.  So the Resident Set Size will
> probably be higher than your malloc size.
>
> (Also: if you have very short lived objects (10 seconds by default) that
> doesn't accrue into the 'malloc' size either... they stack in an
> unbounded 'transient' storage by default.  I got bitten by caching
> errors for 5 seconds with this default)



From maxence at bothorel.net  Tue Jan 12 15:18:59 2016
From: maxence at bothorel.net (maxence at bothorel.net)
Date: Tue, 12 Jan 2016 15:18:59 +0000
Subject: Varnish3 and Munin : Hit rate exceeds 100%
Message-ID: <f306adccffc23aa8face174f4560e944@mail.beastie.eu>

Hello every one,
I'm working with multiple Varnish 3 on Debian Jessie, and I have an issue with Munin.
The Hit Rate graph exceeds 100%.

This is extract from 'varnishstat -1' :
client_req 2977215 34.73 Client requests received
cache_hit 5095361 59.44 Cache hits
cache_hitpass 64796 0.76 Cache hits for pass
cache_miss 834384 9.73 Cache misses

We can see there are more cache_hit than client_req. I wonder if this problem is due to ESI request, which is not use in the graph.
Is there a way to include ESI requests in the graph ?

You can see a graph here : https://lut.im/v4u71OPh4C/q5xNkDneiftmsH2b.png (https://lut.im/v4u71OPh4C/q5xNkDneiftmsH2b.png)

Thank you,

Maxence.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160112/0ac71bd9/attachment.html>

From geoff at uplex.de  Tue Jan 12 15:33:26 2016
From: geoff at uplex.de (Geoff Simmons)
Date: Tue, 12 Jan 2016 16:33:26 +0100
Subject: Varnish3 and Munin : Hit rate exceeds 100%
In-Reply-To: <f306adccffc23aa8face174f4560e944@mail.beastie.eu>
References: <f306adccffc23aa8face174f4560e944@mail.beastie.eu>
Message-ID: <56951CC6.1070603@uplex.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 01/12/2016 04:18 PM, maxence at bothorel.net wrote:
> 
> client_req              2977215        34.73 Client requests
> received cache_hit              5095361        59.44 Cache hits 
> cache_hitpass          64796         0.76 Cache hits for pass 
> cache_miss            834384         9.73 Cache misses

hitrate = cache_hit / (cache_hit + cache_miss) = 86%

client_req doesn't enter into the equation.

> We can see there are more cache_hit than client_req. I wonder if
> this problem is due to ESI request, which is not use in the graph.

Yes, client_req only counts "incoming" requests, while client_hit,
_hitpass and _miss also include results from ESIs.

> Is there a way to include ESI requests in the graph ?

Since both cache_hit and cache_miss count hits and misses from ESIs,
the hitrate equation will in fact account for the ESIs as well.

> You can see a graph here :
> https://lut.im/v4u71OPh4C/q5xNkDneiftmsH2b.png

I get an SSL error from that link.


HTH,
Geoff
- -- 
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJWlRzGAAoJEOUwvh9pJNURAqgP/0ReHPhvYJUuTSBNdsJYW54J
eMOzOMySNBTSCWwkjSt3bM6X8zCMZFS3BuhKdFH+ZmpDC0OcxjONQ5AqIfuqMCMu
CqTLZST46rCgVO6H81h83HGIzbKUWDAJftlpHjupmCPqyF0hcZftJK6B0H/b8xUw
unh0KBuPVAjSrWPrrVtmprSZAmjSR3ZxFLC6F7BJCe45Tz+4tckiVxwpYj0koz8a
U0QZ4CM1hvzTusUJb9LTN2rfgoWh1p71w8RsRJJ7uUU5a2PusTnrhtuwfkfqAba3
zzMQs2EMkVs89lNSVDHA0Nc28ibzshEEzCwb6zmi2ciyqb4otBDVXcNioVDY8QHW
mQJH8CxBAmebw67rgHbbtlsrAxhZUdS8ITDKxnOGHHeoqRt7WKLYzu+nJTYNdvOo
TyD1RFosT+wJutzSWxu+IMxTcBDFOUcdmd3QeuMmVouxy2hl/lHqp+1CzD/g34Ds
t3dxrISTOHX8ayDkXRmu8c7qKA2zpJwfPqrsBn7erypSq3KMEMp6lUwZWfOMHQxd
wGe9HNZ/hh8Ho/j/XXia4rWxdyIcaAuonJMVupIHvM5ogDPlVkUr4/dvrVc3JWzD
2c9BzVmOJdI/EIHoNJDYspM3EE2WZ+qnZ326Cb2CnsvOO6j9uyU/YtG620Jsk7OF
SbMuzHfmoqmHBMFQ3YPg
=bDAI
-----END PGP SIGNATURE-----



From jayanth.ka at directi.com  Wed Jan 13 09:49:52 2016
From: jayanth.ka at directi.com (Jayanth Kalyanasundaram)
Date: Wed, 13 Jan 2016 15:19:52 +0530
Subject: Query regarding "http first read error: EOF" error in varnish
Message-ID: <CAPJM_-fjh1Bavsrf4kPXFPjuSk6wQ1upn5BPsRr4C0za+OGH1w@mail.gmail.com>

Hello,

We are currently using varnish 4.1.0 as a caching layer on top of a few of
our production servers, and from time to time we see a few 503 errors in
the varnish log with the error "-   FetchError     http first read error:
EOF". The complete log for this is shared here: http://pastebin.com/JSqywXDa

The issue doesn't seem to be a connection problem with the backend because
a simple curl call of the same URL from the varnish box seems to work fine.
So we'd like to know what could be the reason behind the error mentioned
above and how we can fix it.

Thanks and Regards,
Jayanth
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160113/a3f56133/attachment.html>

From maxence at bothorel.net  Wed Jan 13 10:29:28 2016
From: maxence at bothorel.net (maxence at bothorel.net)
Date: Wed, 13 Jan 2016 10:29:28 +0000
Subject: Varnish3 and Munin : Hit rate exceeds 100%
In-Reply-To: <56951CC6.1070603@uplex.de>
References: <56951CC6.1070603@uplex.de>
 <f306adccffc23aa8face174f4560e944@mail.beastie.eu>
Message-ID: <54b28bc10c8a041daa93d4c0b2373c67@mail.beastie.eu>

12 janvier 2016 16:33 "Geoff Simmons" <geoff at uplex.de> a ?crit:
> hitrate = cache_hit / (cache_hit + cache_miss) = 86%
> 
> client_req doesn't enter into the equation.
>

Well, you're right. That is how our Nagios plugin do the maths.
It appears Munin use client_req :
https://github.com/munin-monitoring/munin/blob/devel/plugins/node.d/varnish_ 

>> You can see a graph here :
>> https://lut.im/v4u71OPh4C/q5xNkDneiftmsH2b.png
> 
> I get an SSL error from that link.

Works fine on my firefox. lut.im is a french website to upload image.

Thank you,

Maxence.



From geoff at uplex.de  Wed Jan 13 11:00:37 2016
From: geoff at uplex.de (Geoff Simmons)
Date: Wed, 13 Jan 2016 12:00:37 +0100
Subject: Query regarding "http first read error: EOF" error in varnish
In-Reply-To: <CAPJM_-fjh1Bavsrf4kPXFPjuSk6wQ1upn5BPsRr4C0za+OGH1w@mail.gmail.com>
References: <CAPJM_-fjh1Bavsrf4kPXFPjuSk6wQ1upn5BPsRr4C0za+OGH1w@mail.gmail.com>
Message-ID: <56962E55.7090904@uplex.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 01/13/2016 10:49 AM, Jayanth Kalyanasundaram wrote:
> Hello,
> 
> We are currently using varnish 4.1.0 as a caching layer on top of a
> few of our production servers, and from time to time we see a few
> 503 errors in the varnish log with the error "-   FetchError
> http first read error: EOF". The complete log for this is shared
> here: http://pastebin.com/JSqywXDa

"http first read error: EOF" means that there was a socket read error
while attempting to fetch the first chunk of the response header from
the connection to your backend, usually a first byte timeout or a
closed connection.

- -   Timestamp      Bereq: 1452675822.047840 0.015508 0.015508
- -   FetchError     http first read error: EOF
- -   BackendClose   232 reload_2016-01-12T07:28:50.cp_12
- -   Timestamp      Beresp: 1452675876.038544 54.006212 53.990704

This says that about 54 seconds elapsed after Varnish sent the backend
request until it gave up waiting for the response. That could very
well be a first byte timeout, especially since the waiting time is so
close to the round number 54 (although 54 seconds is an unusual value
for first_byte_timeout).

> The issue doesn't seem to be a connection problem with the backend 
> because a simple curl call of the same URL from the varnish box
> seems to work fine. So we'd like to know what could be the reason
> behind the error mentioned above and how we can fix it.

Does Varnish get the FetchError every time, or only sometimes? It
could have been that your curl tests were "just lucky".


HTH,
Geoff
- -- 
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJWli5KAAoJEOUwvh9pJNUR9MkQAInpekQYVjtixJuTkXBxVM6m
J397M1D3b89BiXe8bNPiaTMG/+VpsCK8maXoDAgiZ1NYfRaEYqH9gRGkTQu6x9pT
A3tyP4yu/gClfDGQp4UikGlYLvUnXtJL8KQTjRNcb/WHlWnGRCHfrHfh6xRJfkJm
4cSVJRfiq+hJvEF+4TVZdA5IbTK73Q8r9Xvk4d2pAyG3smeCqJQMo2Zq5b1a2yGu
H1kvQPMDxh3pgkw7trWtpi0b7XznG0hTZ5kT1GdTJ0xZYWXLTcZ5TJ0UX1ZqAz/a
ywmzq3KQpBr3MvSNN17m1HKsw7vbTtAeNMm8bXMf7DfWSxCmbcMAWZKlzlwsSW0B
n+7h3SrL5s0JqQeDM8VP9RkALv/HOrDfg60wvY0HCjc/X7EPvqs0T2K8Eq+WZlqx
ChgMpfgYaOtfD0cUXg1t383LJOoUCy2JuG9ceoqsOuep7B08QfsQdQEh9S+6zkaJ
HIStUOVb7SDAfo0zvq43aICFRgz6wTCuEirdfmFM5Ru5rba/9Oog1NiumGlg7HCU
6d+G0hRYW3wBpgXxgLVdn6gHXnsfj1mkZCEwBUC1L8/5nMB12hfzyWo75a8JqhOT
RZKAXm9sSpOulHclneLlzXqM88BnRqKl/UTIbW7rHQPmvq5fbxW36GgUYM42xm9E
pr8SR/RSaQMyAtJLYpVI
=cnn7
-----END PGP SIGNATURE-----



From jayanth.ka at directi.com  Thu Jan 14 05:29:18 2016
From: jayanth.ka at directi.com (Jayanth Kalyanasundaram)
Date: Thu, 14 Jan 2016 10:59:18 +0530
Subject: Query regarding "http first read error: EOF" error in varnish
In-Reply-To: <56962E55.7090904@uplex.de>
References: <CAPJM_-fjh1Bavsrf4kPXFPjuSk6wQ1upn5BPsRr4C0za+OGH1w@mail.gmail.com>
 <56962E55.7090904@uplex.de>
Message-ID: <CAPJM_-eziaXhSP9CCTV-ZPuqisuLUjx1RBZfidiM+G8QBJVXTQ@mail.gmail.com>

On Wed, Jan 13, 2016 at 4:30 PM, Geoff Simmons <geoff at uplex.de> wrote:
>
>
> - -   Timestamp      Bereq: 1452675822.047840 0.015508 0.015508
> - -   FetchError     http first read error: EOF
> - -   BackendClose   232 reload_2016-01-12T07:28:50.cp_12
> - -   Timestamp      Beresp: 1452675876.038544 54.006212 53.990704
>
> This says that about 54 seconds elapsed after Varnish sent the backend
> request until it gave up waiting for the response. That could very
> well be a first byte timeout, especially since the waiting time is so
> close to the round number 54 (although 54 seconds is an unusual value
> for first_byte_timeout).
>
> > The issue doesn't seem to be a connection problem with the backend
> > because a simple curl call of the same URL from the varnish box
> > seems to work fine. So we'd like to know what could be the reason
> > behind the error mentioned above and how we can fix it.
>
> Does Varnish get the FetchError every time, or only sometimes? It
> could have been that your curl tests were "just lucky".
>
 Hey, like you said the issue only happens sometimes. But the
first_byte_timeout for all the backends is set to 300 seconds.

[root at varnish ~]# sudo varnishadm "param.show first_byte_timeout"
first_byte_timeout
        Value is: 300.000 [seconds]
        Default is: 60.000
        Minimum is: 0.000

Also the time difference between the backend request and backend response
seems to be different each time this issue occurs. For instance, in this
case http://pastebin.com/eS2Ldnbv the difference is just 0.6 seconds. So I
don't think it can be an issue with timeouts.

Would this also happen if the backend server gives an empty response to the
request ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160114/59561f78/attachment.html>

From jayanth.ka at directi.com  Thu Jan 14 05:37:52 2016
From: jayanth.ka at directi.com (Jayanth Kalyanasundaram)
Date: Thu, 14 Jan 2016 11:07:52 +0530
Subject: Query regarding "http first read error: EOF" error in varnish
In-Reply-To: <CAPJM_-eziaXhSP9CCTV-ZPuqisuLUjx1RBZfidiM+G8QBJVXTQ@mail.gmail.com>
References: <CAPJM_-fjh1Bavsrf4kPXFPjuSk6wQ1upn5BPsRr4C0za+OGH1w@mail.gmail.com>
 <56962E55.7090904@uplex.de>
 <CAPJM_-eziaXhSP9CCTV-ZPuqisuLUjx1RBZfidiM+G8QBJVXTQ@mail.gmail.com>
Message-ID: <CAPJM_-dmHRXT8-Aq58FaimeCP53E3jW=m7yP9vr7H6XQcVXi2g@mail.gmail.com>

>
> Also the time difference between the backend request and backend response
> seems to be different each time this issue occurs. For instance, in this
> case http://pastebin.com/eS2Ldnbv the difference is just 0.6 seconds. So
> I don't think it can be an issue with timeouts.
>
> sorry wrong pastebin. This is the correct one:
http://pastebin.com/ZFLgQX90 where the time difference is only 27 seconds
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160114/b7f7b69d/attachment.html>

From geoff at uplex.de  Thu Jan 14 06:53:10 2016
From: geoff at uplex.de (Geoff Simmons)
Date: Thu, 14 Jan 2016 07:53:10 +0100
Subject: Query regarding "http first read error: EOF" error in varnish
In-Reply-To: <CAPJM_-eziaXhSP9CCTV-ZPuqisuLUjx1RBZfidiM+G8QBJVXTQ@mail.gmail.com>
References: <CAPJM_-fjh1Bavsrf4kPXFPjuSk6wQ1upn5BPsRr4C0za+OGH1w@mail.gmail.com>
 <56962E55.7090904@uplex.de>
 <CAPJM_-eziaXhSP9CCTV-ZPuqisuLUjx1RBZfidiM+G8QBJVXTQ@mail.gmail.com>
Message-ID: <569745D6.70606@uplex.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 1/14/16 6:29 AM, Jayanth Kalyanasundaram wrote:
> 
> Hey, like you said the issue only happens sometimes. But the 
> first_byte_timeout for all the backends is set to 300 seconds.
> 
> [root at varnish ~]# sudo varnishadm "param.show first_byte_timeout" 
> first_byte_timeout Value is: 300.000 [seconds] Default is: 60.000 
> Minimum is: 0.000
> 
> Also the time difference between the backend request and backend 
> response seems to be different each time this issue occurs. For 
> instance, in this case http://pastebin.com/eS2Ldnbv the difference
> is just 0.6 seconds. So I don't think it can be an issue with
> timeouts.
> 
> Would this also happen if the backend server gives an empty
> response to the request ?

Yes, that would also happen, or if the connection is prematurely
closed. Since your backend apparently responds with times ranging from
0.6s to 27s to 54s, and sometimes either drops the connection or sends
an empty response, it looks like your problem is the backend and not
Varnish.

If I may add, "set the Varnish timeout really high" (as in 5 minutes)
is a common reaction from backend developers, but is usually not the
way to solve a such a problem. With that timeout, your worker threads
could be idle for up to five minutes, and you could rapidly exhaust
your thread pool. There's no way around it -- the backend needs to be
fixed.


Best,
Geoff
- -- 
UPLEX Systemoptimierung
Scheffelstra?e 32
22301 Hamburg
http://uplex.de/
Mob: +49-176-63690917
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)

iQIcBAEBCAAGBQJWl0XVAAoJEOUwvh9pJNURH5gP/AuFxAcddrCoSLjbsGF/7D8C
zmuyaL3fY89LFwSavTKq1IVU8vPedKE2P+MAO6wlfOPIOV9EAFOOJ04kXVd4RvhQ
+RCO4xCIX8wKrpla+mF2fD9f0zW5S8AaYm1B935XfLyibsjNtDT3aAY7KE0mqTSO
MH6/HaEH7WiFOe2Ahp9YPQeqmrixj8Cf0w99sort9ZxTb6Seh8rZxcg0nwFJnFFB
mKWa1f6pC6mlocc4NkykQGm6sn5ywWff+NyVCj+tqKRJj+SEtVDemcgd6VPItmRa
BB9ixLBjIvCM2881surpLo4GPqTqGZqS8FYlWKj7jfBmrJaQjYuUGOeTB0Z8l+Dr
7HLBWn9OYx2dgQ2IIl4kinKjfSQBqCPW+kGOyfHpNrzdUJuNHUAmOmB1tdtl3heM
H3H4ysDfHnXNsPNYLNIsIclQOFdqG11yt4YrGR36y+3+oywOqAFvs89YLv8ohiPJ
cl6r/PzqocdliDbWAF/hsSSwd8KFGORQmIESH/qbAHMjlFdwfdt3zQBjS1hdnQrj
n9lY0t+pwbnnxgVwot6EMNOQUAP3gkKVUF8e02lqdlG4sm73IuTG8ooxe6+vonEX
Eyd2HzBASdOtDQ2abj1I6scHrzP1jdg+1SmlsiAkx+DJfpNftUVp5RI/jK67q9Ln
MrdlQWfYryv1yOcvICF0
=LALc
-----END PGP SIGNATURE-----



From A.Hongens at netmatch.nl  Thu Jan 14 19:55:23 2016
From: A.Hongens at netmatch.nl (=?iso-8859-1?Q?Angelo_H=F6ngens?=)
Date: Thu, 14 Jan 2016 19:55:23 +0000
Subject: Varnish caching and the X-Requested-With header
In-Reply-To: <6A7ABA19243F1E4EADD8BB1563CDDCCB8A728C3C@TIL-EXCH-05.netmatch.local>
References: <6A7ABA19243F1E4EADD8BB1563CDDCCB8A728C3C@TIL-EXCH-05.netmatch.local>
Message-ID: <6A7ABA19243F1E4EADD8BB1563CDDCCB8A728C6B@TIL-EXCH-05.netmatch.local>

Hey, I'm trying to debug varnish not caching some request. I see the uncacheable object has the request header X-Requested-With: XMLHttpRequest, and I guess that somehow make Varnish not cache the request. But I cannot find this anywhere in the documentation. Is this expected behavior? In which module (vcl_recv, vcl_deliver) can I undo this behavior?

-- 
?
With kind regards,

Angelo H?ngens
Systems Administrator




From nospam at nickswoodart.com  Fri Jan 15 15:46:47 2016
From: nospam at nickswoodart.com (Nick)
Date: Fri, 15 Jan 2016 10:46:47 -0500
Subject: Conditional backend requests with beresp.ttl=0.
Message-ID: <56991467.4090608@nickswoodart.com>

Hi,

I'm wondering if there is a way to configure varnish to send conditional 
requests to the backend when "Cache-Control: no-cache" has been returned 
in a response with an "Etag".

I've tried setting beresp.ttl = 0s and beresp.keep = 1d, but the keep 
time seems to be ignored and the backend isn't getting an 
"If-None-Match" header.

I'm using Varnish 4.0.3, but I've noticed that setting ttl=0s seems to 
cause the keep time to be ignored.  I think the grace time might also be 
ignored, though I haven't verified that yet.

The only way I've been able to force behaviour similar to what I'm 
looking for is to set ttl=1s, but there is a use case where it isn't 
desirable to wait that extra second.

I've also tried setting max-age=0.

Any help would be appreciated!

-- 
Nick




From h at prechler.com  Fri Jan 15 19:48:48 2016
From: h at prechler.com (Hubert Prechler)
Date: Fri, 15 Jan 2016 20:48:48 +0100
Subject: Unable to link OpenSSL
Message-ID: <56994D20.3080909@prechler.com>

Hi,

we're trying to link OpenSSL via cc_command paramter but it doesn't work. What are we missing?

##
_ at lima /etc/systemd/system # service varnish start
Job for varnish.service failed. See 'systemctl status varnish.service' and 'journalctl -xn' for details.
_ at lima /etc/systemd/system # systemctl status -l varnish.service
? varnish.service - Varnish Cache, a high-performance HTTP accelerator
    Loaded: loaded (/etc/systemd/system/varnish.service; disabled)
    Active: failed (Result: exit-code) since Fri 2016-01-15 16:26:31 CET; 14s ago
   Process: 17994 ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/magento.vcl 
-S /etc/varnish/secret -p feature=+esi_disable_xml_check,+esi_ignore_other_elements -p 
vsl_reclen=4084 -p vcc_allow_inline_c=on -p cc_command=exec cc -fpic -shared -Wl,-x -o %o %s 
-lcrypto -lssl -s malloc,5G (code=exited, status=2)

Jan 15 16:26:31 lima varnishd[17994]: /usr/bin/ld: warning: Cannot create .eh_frame_hdr section, 
--eh-frame-hdr ignored.
Jan 15 16:26:31 lima varnishd[17994]: /usr/bin/ld: error in /bin/sh(.eh_frame); no .eh_frame_hdr 
table will be created.
Jan 15 16:26:31 lima varnishd[17994]: Message from dlopen:
Jan 15 16:26:31 lima varnishd[17994]: Could not load compiled VCL.
Jan 15 16:26:31 lima varnishd[17994]: dlopen(vcl_boot/vgc.so) = vcl_boot/vgc.so: symbol lxstat, 
version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference
Jan 15 16:26:31 lima varnishd[17994]: Running dlopen failed, exited with 1
Jan 15 16:26:31 lima varnishd[17994]: VCL compilation failed
Jan 15 16:26:31 lima systemd[1]: varnish.service: control process exited, code=exited status=2
Jan 15 16:26:31 lima systemd[1]: Failed to start Varnish Cache, a high-performance HTTP accelerator.
Jan 15 16:26:31 lima systemd[1]: Unit varnish.service entered failed state.


##
_ at lima /etc/systemd/system # varnishd -V
varnishd (varnish-4.1.0 revision 3041728)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS
_ at lima /etc/systemd/system # uname -a
Linux lima 3.16.0-4-amd64 # 1 SMP Debian 3.16.7-ckt20-1+deb8u2 (2016-01-02) x86_64 GNU/Linux
_ at lima /etc/systemd/system # dpkg -s libc6 | grep ^Version
Version: 2.19-18+deb8u1
_ at lima /etc/systemd/system # dpkg -s openssl | grep ^Version
Version: 1.0.1k-3+deb8u2
_ at lima /etc/systemd/system # dpkg -s libssl-dev | grep ^Version
Version: 1.0.1k-3+deb8u2


## VCL inline snippet:
C{
     #include <string.h>
     #include <stdio.h>
     #include <stdlib.h>
     #include <openssl/md5.h>

     /**
      * create md5 hash of string and return it
      */
     char *generate_formkey(char *string) {
         // generate md5
         unsigned char result[MD5_DIGEST_LENGTH];
         MD5((const unsigned char *)string, strlen(string), result);

         // convert to chars
         static char md5string[MD5_DIGEST_LENGTH + 1];
         const char *hex = "0123456789ABCDEF";
         unsigned char *pin = result;
         char *pout = md5string;

         for(; pin < result + sizeof(result); pout+=2, pin++) {
             pout[0] = hex[(*pin>>4) & 0xF];
             pout[1] = hex[ *pin     & 0xF];
         }
         pout[-1] = 0;

      // return md5
         return md5string;
     }
}C

Thank you,
Hubert



From james.stanton at acquia.com  Mon Jan 18 16:50:19 2016
From: james.stanton at acquia.com (James Stanton)
Date: Mon, 18 Jan 2016 11:50:19 -0500
Subject: Setting param from within VCL
Message-ID: <CA+ojfmZ6+LEkEoJXSSzxCHW6Y-sW4BzRwd2ycJcqYqsJ5da-5w@mail.gmail.com>

How could I set a varnishd param (esi_syntax) from within my VCL? I tried
some embedded C from my vcl_init function to call mcf_findpar but it
wouldn't link . Is this even the right approach? What can I call from
embedded C in my VCL?

Thanks,
Jim

-- 

James Stanton

Technical Account Manager | Acquia <http://www.acquia.com/>

*O: *617-588-7711

E: james.stanton at acquia.com

A: 53 State St., Boston, MA 02109


Acquia Named a *Leader* in new Gartner WCM Magic Quadrant
<https://www.acquia.com/gartner-magic-quadrant>

Start Building Drupal Sites on Acquia Cloud for Free!
<http://acquia.com/free>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160118/afada4f0/attachment.html>

From japrice at gmail.com  Tue Jan 19 20:31:57 2016
From: japrice at gmail.com (Jason Price)
Date: Tue, 19 Jan 2016 15:31:57 -0500
Subject: Setting param from within VCL
In-Reply-To: <CA+ojfmZ6+LEkEoJXSSzxCHW6Y-sW4BzRwd2ycJcqYqsJ5da-5w@mail.gmail.com>
References: <CA+ojfmZ6+LEkEoJXSSzxCHW6Y-sW4BzRwd2ycJcqYqsJ5da-5w@mail.gmail.com>
Message-ID: <CAChvjRAPhvkU0n+13cVHdF3mQmQOVzgkvzFXfeXkAEqNio+onQ@mail.gmail.com>

you won't be able to change parameters from within the VCL/Embeded C code.
You have to change those on daemon startup or through varnishadm

-Jason

On Mon, Jan 18, 2016 at 11:50 AM, James Stanton <james.stanton at acquia.com>
wrote:

> How could I set a varnishd param (esi_syntax) from within my VCL? I tried
> some embedded C from my vcl_init function to call mcf_findpar but it
> wouldn't link . Is this even the right approach? What can I call from
> embedded C in my VCL?
>
> Thanks,
> Jim
>
> --
>
> James Stanton
>
> Technical Account Manager | Acquia <http://www.acquia.com/>
>
> *O: *617-588-7711
>
> E: james.stanton at acquia.com
>
> A: 53 State St., Boston, MA 02109
>
>
> Acquia Named a *Leader* in new Gartner WCM Magic Quadrant
> <https://www.acquia.com/gartner-magic-quadrant>
>
> Start Building Drupal Sites on Acquia Cloud for Free!
> <http://acquia.com/free>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160119/bb04ca25/attachment.html>

From guillaume at varnish-software.com  Thu Jan 21 12:32:37 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Thu, 21 Jan 2016 13:32:37 +0100
Subject: Varnish caching and the X-Requested-With header
In-Reply-To: <6A7ABA19243F1E4EADD8BB1563CDDCCB8A728C6B@TIL-EXCH-05.netmatch.local>
References: <6A7ABA19243F1E4EADD8BB1563CDDCCB8A728C3C@TIL-EXCH-05.netmatch.local>
 <6A7ABA19243F1E4EADD8BB1563CDDCCB8A728C6B@TIL-EXCH-05.netmatch.local>
Message-ID: <CAJ6ZYQx-abnuMDzc_-UiMVG4-Qn2vzKE_SeoeW2x5iYwb8ykHw@mail.gmail.com>

Hi Angelo,

Your first step should be to look at the varnishlog, find the relevant
request, and compare with what your vcl (and the builtin one) to understand
the behaviour.

-- 
Guillaume Quintard

On Thu, Jan 14, 2016 at 8:55 PM, Angelo H?ngens <A.Hongens at netmatch.nl>
wrote:

> Hey, I'm trying to debug varnish not caching some request. I see the
> uncacheable object has the request header X-Requested-With: XMLHttpRequest,
> and I guess that somehow make Varnish not cache the request. But I cannot
> find this anywhere in the documentation. Is this expected behavior? In
> which module (vcl_recv, vcl_deliver) can I undo this behavior?
>
> --
>
> With kind regards,
>
> Angelo H?ngens
> Systems Administrator
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160121/7c82ead4/attachment.html>

From lcruzero at gmail.com  Thu Jan 21 19:45:45 2016
From: lcruzero at gmail.com (L Cruzero)
Date: Thu, 21 Jan 2016 14:45:45 -0500
Subject: Intermittent http first read error: EOF
Message-ID: <CAO+Gocc0znDwGnADGDX4wteWJCxTsp-Dae_Ny=j+Xu+R722Dnw@mail.gmail.com>

Hi, I'm occasionally seeing this error " http first read error: EOF" in
varnishlog for content that exist, and not exceeding "first_byte_timeout"
TTL
I was considering issuing a restart < 4 on 503's with a URL condition match
since this is happening pretty rarely, I'm seeing the error just 2-3 times
while also getting a 200 for the same html asset 3,000+ times within a
couple of mins of logging.


varnish-4.0.3


# varnishadm "param.show first_byte_timeout"

first_byte_timeout

        Value is: 60.000 [seconds] (default)

        Default is: 60.000

        Minimum is: 0.000

# varnishadm "param.show thread_pool_timeout"

thread_pool_timeout

        Value is: 120.000 [seconds]

        Default is: 300.000

        Minimum is: 10.000


VCL code used to define and use backend where 503 errors are being
generated..

backend wwwdot {

        .host = "web-prod-ssf.domain.ly";

        .port = "80";

        }


if (req.http.host ~ "^(origin-www|www)") {

        set req.backend_hint = wwwdot;

        return(pass);

}

<< BeReq    >> 575646365

-   Begin          bereq 575646364 pass

-   Timestamp      Start: 1453229310.040839 0.000000 0.000000

-   BereqMethod    GET

-   BereqURL       /toprail-domain.html

-   BereqProtocol  HTTP/1.1

-   BereqHeader    DNT: 1

-   BereqHeader    Cookie: bknx_fa=1453197478872; bknx_ss=1453229297581;
CPN_crispkey=;
CPN_geo=eyJpcF9hZGRyZXNzIjoiNzAuMTk2LjEzMi4yMyIsImlwX3R5cGUiOiJNYXBwZWQiLCJOZXR3b3JrIjp7ImNvbm5lY3Rpb25fdHlwZSI6Im1vYmlsZSB3aXJlbGVzcyIsImxpbmVfc3BlZWQiOiJsb3ciLCJpcF9yb3V0aW5nX3R5cGUiO

-   BereqHeader    Accept: */*

-   BereqHeader    User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like
Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13C75
Safari/601.1

-   BereqHeader    Accept-Language: en-us

-   BereqHeader    Referer:
http://www.domain.com/outdoors/index.ssf/2014/02/wild_boar_attacks_slidell_man.html

-   BereqHeader    X-Client-Dest-Addr: 69.164.6.110

-   BereqHeader    True-Client-IP: 70.196.132.44

-   BereqHeader    X-Via: 1.1 sw.cds943.dal.llnw.net:8000 (EdgePrism/4.3.1.0),
1.1 cds1158.dal.llnw.net:80 (EdgePrism/4.3.1.0), 1.1 cds1079.lga.llnw.net:80
(EdgePrism/4.3.1.0)

-   BereqHeader    Host: www.domain.com

-   BereqHeader    Accept-Encoding: identity

-   BereqHeader    X-Forwarded-For: 70.196.132.44, 69.164.7.89,
69.164.43.169, 69.164.48.181, 10.51.13.254

-   BereqHeader    X-Varnish: 575646365

-   VCL_call       BACKEND_FETCH

-   VCL_return     fetch

-   BackendOpen    86 wwwdot(69.4.99.100,,80) 10.51.13.97 56085

-   Backend        86 wwwdot wwwdot(69.2.99.10,,80)

-   Timestamp      Bereq: 1453229310.041475 0.000635 0.000635

-   FetchError     http first read error: EOF

-   BackendClose   86 wwwdot(69.4.99.100,,80)

-   Timestamp      Beresp: 1453229310.041717 0.000877 0.000242

-   Timestamp      Error: 1453229310.041720 0.000881 0.000003

-   BerespProtocol HTTP/1.1

-   BerespStatus   503

-   BerespReason   Service Unavailable

-   BerespReason   Backend fetch failed

-   BerespHeader   Date: Tue, 19 Jan 2016 18:48:30 GMT

-   BerespHeader   Server: Varnish

-   VCL_call       BACKEND_ERROR

-   BerespHeader   Content-Type: text/html; charset=utf-8

-   BerespHeader   Retry-After: 5

-   VCL_return     deliver

-   Storage        malloc Transient

-   ObjProtocol    HTTP/1.1

-   ObjStatus      503

-   ObjReason      Backend fetch failed

-   ObjHeader      Date: Tue, 19 Jan 2016 18:48:30 GMT

-   ObjHeader      Server: Varnish

-   ObjHeader      Content-Type: text/html; charset=utf-8

-   ObjHeader      Retry-After: 5

-   Length         286

-   BereqAcct      5960 0 5960 0 0 0

-   End


any suggestions and or ideas on solving this issue would be much
appreciated.

Thanks

-LC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160121/e9263aaf/attachment.html>

From subharaj.manna at gmail.com  Fri Jan 22 12:07:48 2016
From: subharaj.manna at gmail.com (Debraj Manna)
Date: Fri, 22 Jan 2016 17:37:48 +0530
Subject: Difference between n_object, n_objectcore & n_objecthead
Message-ID: <CAF6DVKOjYtPoEYkqm1e-ZvNSxKip2TXwMpat+w3wNmjkZnhfRA@mail.gmail.com>

Hi,


   - What is the difference between n_object, n_objectcore & n_objecthead?
   - When the objects get expired (or the value of MAIN.n_expired changes)
   will the value of these counters reduce?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160122/00b3aa55/attachment.html>

From japrice at gmail.com  Fri Jan 22 13:55:33 2016
From: japrice at gmail.com (Jason Price)
Date: Fri, 22 Jan 2016 08:55:33 -0500
Subject: Intermittent http first read error: EOF
In-Reply-To: <CAO+Gocc0znDwGnADGDX4wteWJCxTsp-Dae_Ny=j+Xu+R722Dnw@mail.gmail.com>
References: <CAO+Gocc0znDwGnADGDX4wteWJCxTsp-Dae_Ny=j+Xu+R722Dnw@mail.gmail.com>
Message-ID: <CAChvjRBs9Cr1sLHRY03993k5Xvj6YJmhHE9VvMujb7AHHkMxFQ@mail.gmail.com>

Is there any chance the web app behind varnish is doing this?

I hate to suggest this, but capturing the problem with tcpdump and finding
it in wireshark may be the best way of proving which side has the problem.
Capture the pcap file with full timestamps, and correlate the times with
varnish log, and follow the TCP connection.

-Jason

On Thu, Jan 21, 2016 at 2:45 PM, L Cruzero <lcruzero at gmail.com> wrote:

> Hi, I'm occasionally seeing this error " http first read error: EOF" in
> varnishlog for content that exist, and not exceeding "first_byte_timeout"
> TTL
> I was considering issuing a restart < 4 on 503's with a URL condition
> match since this is happening pretty rarely, I'm seeing the error just 2-3
> times while also getting a 200 for the same html asset 3,000+ times within
> a couple of mins of logging.
>
>
> varnish-4.0.3
>
>
> # varnishadm "param.show first_byte_timeout"
>
> first_byte_timeout
>
>         Value is: 60.000 [seconds] (default)
>
>         Default is: 60.000
>
>         Minimum is: 0.000
>
> # varnishadm "param.show thread_pool_timeout"
>
> thread_pool_timeout
>
>         Value is: 120.000 [seconds]
>
>         Default is: 300.000
>
>         Minimum is: 10.000
>
>
> VCL code used to define and use backend where 503 errors are being
> generated..
>
> backend wwwdot {
>
>         .host = "web-prod-ssf.domain.ly";
>
>         .port = "80";
>
>         }
>
>
> if (req.http.host ~ "^(origin-www|www)") {
>
>         set req.backend_hint = wwwdot;
>
>         return(pass);
>
> }
>
> << BeReq    >> 575646365
>
> -   Begin          bereq 575646364 pass
>
> -   Timestamp      Start: 1453229310.040839 0.000000 0.000000
>
> -   BereqMethod    GET
>
> -   BereqURL       /toprail-domain.html
>
> -   BereqProtocol  HTTP/1.1
>
> -   BereqHeader    DNT: 1
>
> -   BereqHeader    Cookie: bknx_fa=1453197478872; bknx_ss=1453229297581;
> CPN_crispkey=;
> CPN_geo=eyJpcF9hZGRyZXNzIjoiNzAuMTk2LjEzMi4yMyIsImlwX3R5cGUiOiJNYXBwZWQiLCJOZXR3b3JrIjp7ImNvbm5lY3Rpb25fdHlwZSI6Im1vYmlsZSB3aXJlbGVzcyIsImxpbmVfc3BlZWQiOiJsb3ciLCJpcF9yb3V0aW5nX3R5cGUiO
>
> -   BereqHeader    Accept: */*
>
> -   BereqHeader    User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like
> Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13C75
> Safari/601.1
>
> -   BereqHeader    Accept-Language: en-us
>
> -   BereqHeader    Referer:
> http://www.domain.com/outdoors/index.ssf/2014/02/wild_boar_attacks_slidell_man.html
>
> -   BereqHeader    X-Client-Dest-Addr: 69.164.6.110
>
> -   BereqHeader    True-Client-IP: 70.196.132.44
>
> -   BereqHeader    X-Via: 1.1 sw.cds943.dal.llnw.net:8000 (EdgePrism/
> 4.3.1.0), 1.1 cds1158.dal.llnw.net:80 (EdgePrism/4.3.1.0), 1.1
> cds1079.lga.llnw.net:80 (EdgePrism/4.3.1.0)
>
> -   BereqHeader    Host: www.domain.com
>
> -   BereqHeader    Accept-Encoding: identity
>
> -   BereqHeader    X-Forwarded-For: 70.196.132.44, 69.164.7.89,
> 69.164.43.169, 69.164.48.181, 10.51.13.254
>
> -   BereqHeader    X-Varnish: 575646365
>
> -   VCL_call       BACKEND_FETCH
>
> -   VCL_return     fetch
>
> -   BackendOpen    86 wwwdot(69.4.99.100,,80) 10.51.13.97 56085
>
> -   Backend        86 wwwdot wwwdot(69.2.99.10,,80)
>
> -   Timestamp      Bereq: 1453229310.041475 0.000635 0.000635
>
> -   FetchError     http first read error: EOF
>
> -   BackendClose   86 wwwdot(69.4.99.100,,80)
>
> -   Timestamp      Beresp: 1453229310.041717 0.000877 0.000242
>
> -   Timestamp      Error: 1453229310.041720 0.000881 0.000003
>
> -   BerespProtocol HTTP/1.1
>
> -   BerespStatus   503
>
> -   BerespReason   Service Unavailable
>
> -   BerespReason   Backend fetch failed
>
> -   BerespHeader   Date: Tue, 19 Jan 2016 18:48:30 GMT
>
> -   BerespHeader   Server: Varnish
>
> -   VCL_call       BACKEND_ERROR
>
> -   BerespHeader   Content-Type: text/html; charset=utf-8
>
> -   BerespHeader   Retry-After: 5
>
> -   VCL_return     deliver
>
> -   Storage        malloc Transient
>
> -   ObjProtocol    HTTP/1.1
>
> -   ObjStatus      503
>
> -   ObjReason      Backend fetch failed
>
> -   ObjHeader      Date: Tue, 19 Jan 2016 18:48:30 GMT
>
> -   ObjHeader      Server: Varnish
>
> -   ObjHeader      Content-Type: text/html; charset=utf-8
>
> -   ObjHeader      Retry-After: 5
>
> -   Length         286
>
> -   BereqAcct      5960 0 5960 0 0 0
>
> -   End
>
>
> any suggestions and or ideas on solving this issue would be much
> appreciated.
>
> Thanks
>
> -LC
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160122/a1a02d33/attachment.html>

From raphael.malie at gmail.com  Fri Jan 22 14:32:23 2016
From: raphael.malie at gmail.com (=?UTF-8?B?UmFwaGHDq2wgTWFsacOp?=)
Date: Fri, 22 Jan 2016 15:32:23 +0100
Subject: Conditional requests / if-modified-since not working
Message-ID: <CAOBnEp2CrdVxtsX+ZJ46A6XZWAZ2rHauSjL7mqDid3Pj0urxvQ@mail.gmail.com>

Hello,

I did a fresh install of Varnish 4.1 to test conditional requests with a
last-modified header sent from the backend, here are my backend headers:

Cache-Control: must-revalidate, proxy-revalidate, public
Last-Modified: Fri, 22 Jan 2016 14:27:50 GMT

On the next query, Varnish should try to contact the backend with a
if-modified-since header, and the backend shoud send back a "304 not
modified" answer if nothing changed.

But Varnish never tries to contact the backend, he is always caching my
page with the default ttl (120s).

Thank you for any help,
Rapha?l
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160122/da201a67/attachment.html>

From lcruzero at gmail.com  Fri Jan 22 14:38:21 2016
From: lcruzero at gmail.com (L Cruzero)
Date: Fri, 22 Jan 2016 09:38:21 -0500
Subject: Intermittent http first read error: EOF
In-Reply-To: <CAChvjRBs9Cr1sLHRY03993k5Xvj6YJmhHE9VvMujb7AHHkMxFQ@mail.gmail.com>
References: <CAO+Gocc0znDwGnADGDX4wteWJCxTsp-Dae_Ny=j+Xu+R722Dnw@mail.gmail.com>
 <CAChvjRBs9Cr1sLHRY03993k5Xvj6YJmhHE9VvMujb7AHHkMxFQ@mail.gmail.com>
Message-ID: <CAO+Gocfhh7JrHo4d-5ps_uWqzs0nPHk+YGZvgBRfi8P4TP6u4A@mail.gmail.com>

thanks Jason, I've not found any 400+ or 500+ errors in any of web app
server logs that correlate to varnishlog timestamps. also, I should have
included this bit of info. the backend that varnish connects to is a
Netscaler LB, both thread_pool_timeout in varnish and tcp persistent
connection timeout in LB are set to 120secs. pointing varnish directly to a
web app server to get a good tcpdump output wont be as straight forward as
it sounds, due to content switching policies being applied at the NS to
server various page components from different web app endpoints. I'm
considering upgrading to 4.1, which list "backend connection timeout" as
one of the changes.

https://www.varnish-cache.org/docs/trunk/whats-new/changes.html


thanks.
-LC


On Fri, Jan 22, 2016 at 8:55 AM, Jason Price <japrice at gmail.com> wrote:

> Is there any chance the web app behind varnish is doing this?
>
> I hate to suggest this, but capturing the problem with tcpdump and finding
> it in wireshark may be the best way of proving which side has the problem.
> Capture the pcap file with full timestamps, and correlate the times with
> varnish log, and follow the TCP connection.
>
> -Jason
>
> On Thu, Jan 21, 2016 at 2:45 PM, L Cruzero <lcruzero at gmail.com> wrote:
>
>> Hi, I'm occasionally seeing this error " http first read error: EOF" in
>> varnishlog for content that exist, and not exceeding "first_byte_timeout"
>> TTL
>> I was considering issuing a restart < 4 on 503's with a URL condition
>> match since this is happening pretty rarely, I'm seeing the error just 2-3
>> times while also getting a 200 for the same html asset 3,000+ times within
>> a couple of mins of logging.
>>
>>
>> varnish-4.0.3
>>
>>
>> # varnishadm "param.show first_byte_timeout"
>>
>> first_byte_timeout
>>
>>         Value is: 60.000 [seconds] (default)
>>
>>         Default is: 60.000
>>
>>         Minimum is: 0.000
>>
>> # varnishadm "param.show thread_pool_timeout"
>>
>> thread_pool_timeout
>>
>>         Value is: 120.000 [seconds]
>>
>>         Default is: 300.000
>>
>>         Minimum is: 10.000
>>
>>
>> VCL code used to define and use backend where 503 errors are being
>> generated..
>>
>> backend wwwdot {
>>
>>         .host = "web-prod-ssf.domain.ly";
>>
>>         .port = "80";
>>
>>         }
>>
>>
>> if (req.http.host ~ "^(origin-www|www)") {
>>
>>         set req.backend_hint = wwwdot;
>>
>>         return(pass);
>>
>> }
>>
>> << BeReq    >> 575646365
>>
>> -   Begin          bereq 575646364 pass
>>
>> -   Timestamp      Start: 1453229310.040839 0.000000 0.000000
>>
>> -   BereqMethod    GET
>>
>> -   BereqURL       /toprail-domain.html
>>
>> -   BereqProtocol  HTTP/1.1
>>
>> -   BereqHeader    DNT: 1
>>
>> -   BereqHeader    Cookie: bknx_fa=1453197478872; bknx_ss=1453229297581;
>> CPN_crispkey=;
>> CPN_geo=eyJpcF9hZGRyZXNzIjoiNzAuMTk2LjEzMi4yMyIsImlwX3R5cGUiOiJNYXBwZWQiLCJOZXR3b3JrIjp7ImNvbm5lY3Rpb25fdHlwZSI6Im1vYmlsZSB3aXJlbGVzcyIsImxpbmVfc3BlZWQiOiJsb3ciLCJpcF9yb3V0aW5nX3R5cGUiO
>>
>> -   BereqHeader    Accept: */*
>>
>> -   BereqHeader    User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 9_2
>> like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0
>> Mobile/13C75 Safari/601.1
>>
>> -   BereqHeader    Accept-Language: en-us
>>
>> -   BereqHeader    Referer:
>> http://www.domain.com/outdoors/index.ssf/2014/02/wild_boar_attacks_slidell_man.html
>>
>> -   BereqHeader    X-Client-Dest-Addr: 69.164.6.110
>>
>> -   BereqHeader    True-Client-IP: 70.196.132.44
>>
>> -   BereqHeader    X-Via: 1.1 sw.cds943.dal.llnw.net:8000 (EdgePrism/
>> 4.3.1.0), 1.1 cds1158.dal.llnw.net:80 (EdgePrism/4.3.1.0), 1.1
>> cds1079.lga.llnw.net:80 (EdgePrism/4.3.1.0)
>>
>> -   BereqHeader    Host: www.domain.com
>>
>> -   BereqHeader    Accept-Encoding: identity
>>
>> -   BereqHeader    X-Forwarded-For: 70.196.132.44, 69.164.7.89,
>> 69.164.43.169, 69.164.48.181, 10.51.13.254
>>
>> -   BereqHeader    X-Varnish: 575646365
>>
>> -   VCL_call       BACKEND_FETCH
>>
>> -   VCL_return     fetch
>>
>> -   BackendOpen    86 wwwdot(69.4.99.100,,80) 10.51.13.97 56085
>>
>> -   Backend        86 wwwdot wwwdot(69.2.99.10,,80)
>>
>> -   Timestamp      Bereq: 1453229310.041475 0.000635 0.000635
>>
>> -   FetchError     http first read error: EOF
>>
>> -   BackendClose   86 wwwdot(69.4.99.100,,80)
>>
>> -   Timestamp      Beresp: 1453229310.041717 0.000877 0.000242
>>
>> -   Timestamp      Error: 1453229310.041720 0.000881 0.000003
>>
>> -   BerespProtocol HTTP/1.1
>>
>> -   BerespStatus   503
>>
>> -   BerespReason   Service Unavailable
>>
>> -   BerespReason   Backend fetch failed
>>
>> -   BerespHeader   Date: Tue, 19 Jan 2016 18:48:30 GMT
>>
>> -   BerespHeader   Server: Varnish
>>
>> -   VCL_call       BACKEND_ERROR
>>
>> -   BerespHeader   Content-Type: text/html; charset=utf-8
>>
>> -   BerespHeader   Retry-After: 5
>>
>> -   VCL_return     deliver
>>
>> -   Storage        malloc Transient
>>
>> -   ObjProtocol    HTTP/1.1
>>
>> -   ObjStatus      503
>>
>> -   ObjReason      Backend fetch failed
>>
>> -   ObjHeader      Date: Tue, 19 Jan 2016 18:48:30 GMT
>>
>> -   ObjHeader      Server: Varnish
>>
>> -   ObjHeader      Content-Type: text/html; charset=utf-8
>>
>> -   ObjHeader      Retry-After: 5
>>
>> -   Length         286
>>
>> -   BereqAcct      5960 0 5960 0 0 0
>>
>> -   End
>>
>>
>> any suggestions and or ideas on solving this issue would be much
>> appreciated.
>>
>> Thanks
>>
>> -LC
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160122/76883060/attachment.html>

From japrice at gmail.com  Fri Jan 22 18:54:45 2016
From: japrice at gmail.com (Jason Price)
Date: Fri, 22 Jan 2016 13:54:45 -0500
Subject: Intermittent http first read error: EOF
In-Reply-To: <CAO+Gocfhh7JrHo4d-5ps_uWqzs0nPHk+YGZvgBRfi8P4TP6u4A@mail.gmail.com>
References: <CAO+Gocc0znDwGnADGDX4wteWJCxTsp-Dae_Ny=j+Xu+R722Dnw@mail.gmail.com>
 <CAChvjRBs9Cr1sLHRY03993k5Xvj6YJmhHE9VvMujb7AHHkMxFQ@mail.gmail.com>
 <CAO+Gocfhh7JrHo4d-5ps_uWqzs0nPHk+YGZvgBRfi8P4TP6u4A@mail.gmail.com>
Message-ID: <CAChvjRDEG+OkoECbeO6FMPbo31agmO2dDxJHnRmMMD+3xDDnEQ@mail.gmail.com>

That backend timeout feature does sound promising... Nescalers do reap
connections.

To be completely clear, I was arguing that you should run tcpdump on the
varnish server itself, listening for all traffic to it's direct backend.

You could do that... That'd give you a better idea of what's happening on
that server.  To augment it, you'd also need to run 'nstcpdump.sh' on the
netscaler and ideally on the backend itself.  Then you'd need to line up
the HTTP transaction across all of them.  It'd won't be much fun, but
you'll see exactly where the problem is, and know who to yell at to fix it.

-Jason

On Fri, Jan 22, 2016 at 9:38 AM, L Cruzero <lcruzero at gmail.com> wrote:

> thanks Jason, I've not found any 400+ or 500+ errors in any of web app
> server logs that correlate to varnishlog timestamps. also, I should have
> included this bit of info. the backend that varnish connects to is a
> Netscaler LB, both thread_pool_timeout in varnish and tcp persistent
> connection timeout in LB are set to 120secs. pointing varnish directly to a
> web app server to get a good tcpdump output wont be as straight forward as
> it sounds, due to content switching policies being applied at the NS to
> server various page components from different web app endpoints. I'm
> considering upgrading to 4.1, which list "backend connection timeout" as
> one of the changes.
>
> https://www.varnish-cache.org/docs/trunk/whats-new/changes.html
>
>
> thanks.
> -LC
>
>
> On Fri, Jan 22, 2016 at 8:55 AM, Jason Price <japrice at gmail.com> wrote:
>
>> Is there any chance the web app behind varnish is doing this?
>>
>> I hate to suggest this, but capturing the problem with tcpdump and
>> finding it in wireshark may be the best way of proving which side has the
>> problem.  Capture the pcap file with full timestamps, and correlate the
>> times with varnish log, and follow the TCP connection.
>>
>> -Jason
>>
>> On Thu, Jan 21, 2016 at 2:45 PM, L Cruzero <lcruzero at gmail.com> wrote:
>>
>>> Hi, I'm occasionally seeing this error " http first read error: EOF" in
>>> varnishlog for content that exist, and not exceeding "first_byte_timeout"
>>> TTL
>>> I was considering issuing a restart < 4 on 503's with a URL condition
>>> match since this is happening pretty rarely, I'm seeing the error just 2-3
>>> times while also getting a 200 for the same html asset 3,000+ times within
>>> a couple of mins of logging.
>>>
>>>
>>> varnish-4.0.3
>>>
>>>
>>> # varnishadm "param.show first_byte_timeout"
>>>
>>> first_byte_timeout
>>>
>>>         Value is: 60.000 [seconds] (default)
>>>
>>>         Default is: 60.000
>>>
>>>         Minimum is: 0.000
>>>
>>> # varnishadm "param.show thread_pool_timeout"
>>>
>>> thread_pool_timeout
>>>
>>>         Value is: 120.000 [seconds]
>>>
>>>         Default is: 300.000
>>>
>>>         Minimum is: 10.000
>>>
>>>
>>> VCL code used to define and use backend where 503 errors are being
>>> generated..
>>>
>>> backend wwwdot {
>>>
>>>         .host = "web-prod-ssf.domain.ly";
>>>
>>>         .port = "80";
>>>
>>>         }
>>>
>>>
>>> if (req.http.host ~ "^(origin-www|www)") {
>>>
>>>         set req.backend_hint = wwwdot;
>>>
>>>         return(pass);
>>>
>>> }
>>>
>>> << BeReq    >> 575646365
>>>
>>> -   Begin          bereq 575646364 pass
>>>
>>> -   Timestamp      Start: 1453229310.040839 0.000000 0.000000
>>>
>>> -   BereqMethod    GET
>>>
>>> -   BereqURL       /toprail-domain.html
>>>
>>> -   BereqProtocol  HTTP/1.1
>>>
>>> -   BereqHeader    DNT: 1
>>>
>>> -   BereqHeader    Cookie: bknx_fa=1453197478872; bknx_ss=1453229297581;
>>> CPN_crispkey=;
>>> CPN_geo=eyJpcF9hZGRyZXNzIjoiNzAuMTk2LjEzMi4yMyIsImlwX3R5cGUiOiJNYXBwZWQiLCJOZXR3b3JrIjp7ImNvbm5lY3Rpb25fdHlwZSI6Im1vYmlsZSB3aXJlbGVzcyIsImxpbmVfc3BlZWQiOiJsb3ciLCJpcF9yb3V0aW5nX3R5cGUiO
>>>
>>> -   BereqHeader    Accept: */*
>>>
>>> -   BereqHeader    User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 9_2
>>> like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0
>>> Mobile/13C75 Safari/601.1
>>>
>>> -   BereqHeader    Accept-Language: en-us
>>>
>>> -   BereqHeader    Referer:
>>> http://www.domain.com/outdoors/index.ssf/2014/02/wild_boar_attacks_slidell_man.html
>>>
>>> -   BereqHeader    X-Client-Dest-Addr: 69.164.6.110
>>>
>>> -   BereqHeader    True-Client-IP: 70.196.132.44
>>>
>>> -   BereqHeader    X-Via: 1.1 sw.cds943.dal.llnw.net:8000 (EdgePrism/
>>> 4.3.1.0), 1.1 cds1158.dal.llnw.net:80 (EdgePrism/4.3.1.0), 1.1
>>> cds1079.lga.llnw.net:80 (EdgePrism/4.3.1.0)
>>>
>>> -   BereqHeader    Host: www.domain.com
>>>
>>> -   BereqHeader    Accept-Encoding: identity
>>>
>>> -   BereqHeader    X-Forwarded-For: 70.196.132.44, 69.164.7.89,
>>> 69.164.43.169, 69.164.48.181, 10.51.13.254
>>>
>>> -   BereqHeader    X-Varnish: 575646365
>>>
>>> -   VCL_call       BACKEND_FETCH
>>>
>>> -   VCL_return     fetch
>>>
>>> -   BackendOpen    86 wwwdot(69.4.99.100,,80) 10.51.13.97 56085
>>>
>>> -   Backend        86 wwwdot wwwdot(69.2.99.10,,80)
>>>
>>> -   Timestamp      Bereq: 1453229310.041475 0.000635 0.000635
>>>
>>> -   FetchError     http first read error: EOF
>>>
>>> -   BackendClose   86 wwwdot(69.4.99.100,,80)
>>>
>>> -   Timestamp      Beresp: 1453229310.041717 0.000877 0.000242
>>>
>>> -   Timestamp      Error: 1453229310.041720 0.000881 0.000003
>>>
>>> -   BerespProtocol HTTP/1.1
>>>
>>> -   BerespStatus   503
>>>
>>> -   BerespReason   Service Unavailable
>>>
>>> -   BerespReason   Backend fetch failed
>>>
>>> -   BerespHeader   Date: Tue, 19 Jan 2016 18:48:30 GMT
>>>
>>> -   BerespHeader   Server: Varnish
>>>
>>> -   VCL_call       BACKEND_ERROR
>>>
>>> -   BerespHeader   Content-Type: text/html; charset=utf-8
>>>
>>> -   BerespHeader   Retry-After: 5
>>>
>>> -   VCL_return     deliver
>>>
>>> -   Storage        malloc Transient
>>>
>>> -   ObjProtocol    HTTP/1.1
>>>
>>> -   ObjStatus      503
>>>
>>> -   ObjReason      Backend fetch failed
>>>
>>> -   ObjHeader      Date: Tue, 19 Jan 2016 18:48:30 GMT
>>>
>>> -   ObjHeader      Server: Varnish
>>>
>>> -   ObjHeader      Content-Type: text/html; charset=utf-8
>>>
>>> -   ObjHeader      Retry-After: 5
>>>
>>> -   Length         286
>>>
>>> -   BereqAcct      5960 0 5960 0 0 0
>>>
>>> -   End
>>>
>>>
>>> any suggestions and or ideas on solving this issue would be much
>>> appreciated.
>>>
>>> Thanks
>>>
>>> -LC
>>>
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160122/6ac04eaa/attachment.html>

From subharaj.manna at gmail.com  Fri Jan 22 19:41:38 2016
From: subharaj.manna at gmail.com (Debraj Manna)
Date: Sat, 23 Jan 2016 01:11:38 +0530
Subject: Difference between n_object, n_objectcore & n_objecthead
In-Reply-To: <CAF6DVKOjYtPoEYkqm1e-ZvNSxKip2TXwMpat+w3wNmjkZnhfRA@mail.gmail.com>
References: <CAF6DVKOjYtPoEYkqm1e-ZvNSxKip2TXwMpat+w3wNmjkZnhfRA@mail.gmail.com>
Message-ID: <CAF6DVKMq6fB4aRb__tv0jDmTwhuz-Rg3Y5Gk9-7y+EFO24swWQ@mail.gmail.com>

Can someone point me to some doc where these stats are explained?
On Jan 22, 2016 5:37 PM, "Debraj Manna" <subharaj.manna at gmail.com> wrote:

> Hi,
>
>
>    - What is the difference between n_object, n_objectcore & n_objecthead?
>    - When the objects get expired (or the value of MAIN.n_expired changes)
>    will the value of these counters reduce?
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160123/619fde16/attachment.html>

From guillaume at varnish-software.com  Fri Jan 22 20:50:12 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 22 Jan 2016 21:50:12 +0100
Subject: Conditional requests / if-modified-since not working
In-Reply-To: <CAOBnEp2CrdVxtsX+ZJ46A6XZWAZ2rHauSjL7mqDid3Pj0urxvQ@mail.gmail.com>
References: <CAOBnEp2CrdVxtsX+ZJ46A6XZWAZ2rHauSjL7mqDid3Pj0urxvQ@mail.gmail.com>
Message-ID: <CAJ6ZYQzgROF3FB-rLSWvN6NRDn7+1O02ggh+haSB1zYsg+pFJQ@mail.gmail.com>

Hi Rapha?l,

Varnish ignores the must revalidate header, if you wish to emulate the
behaviour, you can use something like this in your vcl:

sub vcl_backend_response {
    if (beresp.http.cache-control ~ "must-revalidate") {
        set beresp.ttl = 1s;
        set beresp.grace = 0s;
        set beresp.keep = 1w;
    }
}

This way, you'd only cache for 1 second (don't set it to 0, or all the
requests for this object would be done sequentially), but will keep the
object for a week, revalidating it each time it is requested and its ttl is
expired.

-- 
Guillaume Quintard

On Fri, Jan 22, 2016 at 3:32 PM, Rapha?l Mali? <raphael.malie at gmail.com>
wrote:

> Hello,
>
> I did a fresh install of Varnish 4.1 to test conditional requests with a
> last-modified header sent from the backend, here are my backend headers:
>
> Cache-Control: must-revalidate, proxy-revalidate, public
> Last-Modified: Fri, 22 Jan 2016 14:27:50 GMT
>
> On the next query, Varnish should try to contact the backend with a
> if-modified-since header, and the backend shoud send back a "304 not
> modified" answer if nothing changed.
>
> But Varnish never tries to contact the backend, he is always caching my
> page with the default ttl (120s).
>
> Thank you for any help,
> Rapha?l
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160122/95c24be0/attachment.html>

From raphael.malie at gmail.com  Fri Jan 22 23:14:40 2016
From: raphael.malie at gmail.com (=?UTF-8?B?UmFwaGHDq2wgTWFsacOp?=)
Date: Sat, 23 Jan 2016 00:14:40 +0100
Subject: Conditional requests / if-modified-since not working
In-Reply-To: <CAJ6ZYQzgROF3FB-rLSWvN6NRDn7+1O02ggh+haSB1zYsg+pFJQ@mail.gmail.com>
References: <CAOBnEp2CrdVxtsX+ZJ46A6XZWAZ2rHauSjL7mqDid3Pj0urxvQ@mail.gmail.com>
 <CAJ6ZYQzgROF3FB-rLSWvN6NRDn7+1O02ggh+haSB1zYsg+pFJQ@mail.gmail.com>
Message-ID: <CAOBnEp3r-XiaNyB4mbDQxP7WwPT55jNQexfEijZo4CHfsE3Urw@mail.gmail.com>

Hi Guillaume,

It works, finally!

Many thanks,
Rapha?l

2016-01-22 21:50 GMT+01:00 Guillaume Quintard <
guillaume at varnish-software.com>:

> Hi Rapha?l,
>
> Varnish ignores the must revalidate header, if you wish to emulate the
> behaviour, you can use something like this in your vcl:
>
> sub vcl_backend_response {
>     if (beresp.http.cache-control ~ "must-revalidate") {
>         set beresp.ttl = 1s;
>         set beresp.grace = 0s;
>         set beresp.keep = 1w;
>     }
> }
>
> This way, you'd only cache for 1 second (don't set it to 0, or all the
> requests for this object would be done sequentially), but will keep the
> object for a week, revalidating it each time it is requested and its ttl is
> expired.
>
> --
> Guillaume Quintard
>
> On Fri, Jan 22, 2016 at 3:32 PM, Rapha?l Mali? <raphael.malie at gmail.com>
> wrote:
>
>> Hello,
>>
>> I did a fresh install of Varnish 4.1 to test conditional requests with a
>> last-modified header sent from the backend, here are my backend headers:
>>
>> Cache-Control: must-revalidate, proxy-revalidate, public
>> Last-Modified: Fri, 22 Jan 2016 14:27:50 GMT
>>
>> On the next query, Varnish should try to contact the backend with a
>> if-modified-since header, and the backend shoud send back a "304 not
>> modified" answer if nothing changed.
>>
>> But Varnish never tries to contact the backend, he is always caching my
>> page with the default ttl (120s).
>>
>> Thank you for any help,
>> Rapha?l
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160123/ee840119/attachment.html>

From guillaume at varnish-software.com  Fri Jan 22 23:39:23 2016
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Sat, 23 Jan 2016 00:39:23 +0100
Subject: Conditional requests / if-modified-since not working
In-Reply-To: <CAOBnEp3r-XiaNyB4mbDQxP7WwPT55jNQexfEijZo4CHfsE3Urw@mail.gmail.com>
References: <CAOBnEp2CrdVxtsX+ZJ46A6XZWAZ2rHauSjL7mqDid3Pj0urxvQ@mail.gmail.com>
 <CAJ6ZYQzgROF3FB-rLSWvN6NRDn7+1O02ggh+haSB1zYsg+pFJQ@mail.gmail.com>
 <CAOBnEp3r-XiaNyB4mbDQxP7WwPT55jNQexfEijZo4CHfsE3Urw@mail.gmail.com>
Message-ID: <CAJ6ZYQz4r4tQAfemr4Auxp=NwXc99aEzu+92zR2YTS=Z5aJzXw@mail.gmail.com>

You are welcome. Have fun.

-- 
Guillaume Quintard

On Sat, Jan 23, 2016 at 12:14 AM, Rapha?l Mali? <raphael.malie at gmail.com>
wrote:

> Hi Guillaume,
>
> It works, finally!
>
> Many thanks,
> Rapha?l
>
> 2016-01-22 21:50 GMT+01:00 Guillaume Quintard <
> guillaume at varnish-software.com>:
>
>> Hi Rapha?l,
>>
>> Varnish ignores the must revalidate header, if you wish to emulate the
>> behaviour, you can use something like this in your vcl:
>>
>> sub vcl_backend_response {
>>     if (beresp.http.cache-control ~ "must-revalidate") {
>>         set beresp.ttl = 1s;
>>         set beresp.grace = 0s;
>>         set beresp.keep = 1w;
>>     }
>> }
>>
>> This way, you'd only cache for 1 second (don't set it to 0, or all the
>> requests for this object would be done sequentially), but will keep the
>> object for a week, revalidating it each time it is requested and its ttl is
>> expired.
>>
>> --
>> Guillaume Quintard
>>
>> On Fri, Jan 22, 2016 at 3:32 PM, Rapha?l Mali? <raphael.malie at gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> I did a fresh install of Varnish 4.1 to test conditional requests with a
>>> last-modified header sent from the backend, here are my backend headers:
>>>
>>> Cache-Control: must-revalidate, proxy-revalidate, public
>>> Last-Modified: Fri, 22 Jan 2016 14:27:50 GMT
>>>
>>> On the next query, Varnish should try to contact the backend with a
>>> if-modified-since header, and the backend shoud send back a "304 not
>>> modified" answer if nothing changed.
>>>
>>> But Varnish never tries to contact the backend, he is always caching my
>>> page with the default ttl (120s).
>>>
>>> Thank you for any help,
>>> Rapha?l
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160123/46ceb869/attachment.html>

From lkarsten at varnish-software.com  Mon Jan 25 12:16:16 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Mon, 25 Jan 2016 13:16:16 +0100
Subject: Varnish Cache 4.1.1-beta2 needs testing
Message-ID: <20160125121615.GA4895@immer.varnish-software.com>

Hi all.

Last week we released and uploaded Varnish Cache 4.1.1-beta2.

Normally I announce releases on varnish-announce@ when they
are done, but this time around I think it is cruical that we
get it tested with load before that.

So, if someone has a bit of effort to spare and can try it out with some
load, it would be great.

Source release can be found in the normal location:

    https://repo.varnish-cache.org/source/varnish-4.1.1-beta2.tar.gz


Package files for EL6/EL7/Debian/Ubuntu are here:

    https://filebin.varnish-software.com/pkg-vc411beta2

(only for the beta, will be on repo.varnish-cache.org when 4.1.1-final
is out)


-- 
Lasse Karstensen
Release manager



From lkarsten at varnish-software.com  Mon Jan 25 12:23:54 2016
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Mon, 25 Jan 2016 13:23:54 +0100
Subject: Varnish Cache 4.1.1-beta2 needs testing
In-Reply-To: <20160125121615.GA4895@immer.varnish-software.com>
References: <20160125121615.GA4895@immer.varnish-software.com>
Message-ID: <20160125122353.GB4895@immer.varnish-software.com>

On Mon, Jan 25, 2016 at 01:16:16PM +0100, Lasse Karstensen wrote:
[..]
> Source release can be found in the normal location:
> 
>     https://repo.varnish-cache.org/source/varnish-4.1.1-beta2.tar.gz
>
> Package files for EL6/EL7/Debian/Ubuntu are here:
>     https://filebin.varnish-software.com/pkg-vc411beta2

I forgot to add the link to the changes file:

    https://github.com/varnish/Varnish-Cache/blob/4.1/doc/changes.rst


-- 
Lasse Karstensen
Varnish Software AS



From Tom.Anheyer at Berlinonline.de  Mon Jan 25 13:37:48 2016
From: Tom.Anheyer at Berlinonline.de (Tom Anheyer)
Date: Mon, 25 Jan 2016 14:37:48 +0100
Subject: Varnish Cache 4.1.1-beta2 needs testing
In-Reply-To: <20160125122353.GB4895@immer.varnish-software.com>
References: <20160125121615.GA4895@immer.varnish-software.com>
 <20160125122353.GB4895@immer.varnish-software.com>
Message-ID: <56A6252C.9060001@Berlinonline.de>

Small typo in changes:

Zero Content-Length is no longer sent on 304 responses.
  -> Zero Content-Length is no longer sent on 204 responses.

tom

Am 25.01.2016 um 13:23 schrieb Lasse Karstensen:
> On Mon, Jan 25, 2016 at 01:16:16PM +0100, Lasse Karstensen wrote:
> [..]
>> Source release can be found in the normal location:
>>
>>      https://repo.varnish-cache.org/source/varnish-4.1.1-beta2.tar.gz
>>
>> Package files for EL6/EL7/Debian/Ubuntu are here:
>>      https://filebin.varnish-software.com/pkg-vc411beta2
> I forgot to add the link to the changes file:
>
>      https://github.com/varnish/Varnish-Cache/blob/4.1/doc/changes.rst
>
>


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4266 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160125/c3727464/attachment.bin>

From fgsch at lodoss.net  Mon Jan 25 17:55:07 2016
From: fgsch at lodoss.net (Federico Schwindt)
Date: Mon, 25 Jan 2016 17:55:07 +0000
Subject: Varnish Cache 4.1.1-beta2 needs testing
Message-ID: <CAJV_h0aANU1Jq6aj9KOxVoyqyDCXJ+NGyz3Y7PJ0JAhJ60qfGA@mail.gmail.com>

> Small typo in changes:
>
> Zero Content-Length is no longer sent on 304 responses.
>   -> Zero Content-Length is no longer sent on 204 responses.

That's not a typo, it should say 304, not 204.

The zero C-L for 204 responses was introduced as a side effect of 271e1c52
but this was between 4.1.0 and 4.1.1 so never made into a release.

f.-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160125/32f1b508/attachment.html>

From georgi.int at gmail.com  Tue Jan 26 10:24:16 2016
From: georgi.int at gmail.com (georgi.int at gmail.com)
Date: Tue, 26 Jan 2016 12:24:16 +0200
Subject: junk after gzip encoding
Message-ID: <56A74950.3030700@gmail.com>

Hello,
Can you please tell me how can I fix this error from the varnishlog and 
whyt it's caused, because everything I read in the net couldn't help me. 
Sometimes on some websites the most pages works fine, but only certain 
css files give 503 with this error.

Thank you in advance.



From rainer at ultra-secure.de  Tue Jan 26 16:05:39 2016
From: rainer at ultra-secure.de (rainer at ultra-secure.de)
Date: Tue, 26 Jan 2016 17:05:39 +0100
Subject: Why doesn't varnish cache these requests?
Message-ID: <e4d4687f1f7fd821392addece9833ba5@ultra-secure.de>

Hi,


I have typo3 setup with nging + php-fpm.

The following request to the backend (using curl):

> GET /the_url/ HTTP/1.1
> Host: the_server
> User-Agent: curl/7.46.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 26 Jan 2016 15:30:39 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 31195
< Connection: keep-alive
< Content-Language: de
< Expires: Thu, 25 Feb 2016 15:07:05 GMT
< ETag: "c8b13cf27951a887cd9e54f307456e2e"
< Cache-Control: max-age=2590586
< Pragma: public
< TYPO3-Pid: 6
< TYPO3-Sitename: 6a48956158576b1c9fcab0ff657e8ef8e337b750
<

does not seem to get cached.
I also use the "static file cache" module and as such, the page is 
served directly by nginx.
Note that varnish is on a different server than nginx.


I have the following vcl, taken directly from the example in the varnish 
typo3 plugin:

/*
  * snowflake Varnish Configuration for TYPO3
  *
  * (c) 2013 snowflake productions gmbh <varnish at snowflake.ch>
  * All rights reserved
  *
  * This script is part of the TYPO3 project. The TYPO3 project is
  * free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
  * the Free Software Foundation; either version 2 of the License, or
  * (at your option) any later version.
  *
  * The GNU General Public License can be found at
  * http://www.gnu.org/copyleft/gpl.html.
  *
  * This script is distributed in the hope that it will be useful,
  * but WITHOUT ANY WARRANTY; without even the implied warranty of
  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  * GNU General Public License for more details.
  *
  * This copyright notice MUST APPEAR in all copies of the script!
  */

vcl 4.0;

/*
  * Declare the default Backend Server
  */

backend default {
	.host = "10.6.2.100";
	.port = "80";
}


/*
  * BAN ACL
  * Clients in this lists are allowed to issue BAN commands
  */

acl ban {
	"127.0.0.1";
}


/*
  * vcl_recv
  */

sub vcl_recv {
	# Catch BAN Command
	if (req.method == "BAN" && client.ip ~ ban) {

		if(req.http.Varnish-Ban-All == "1" && 
req.http.Varnish-Ban-TYPO3-Sitename) {
			ban("req.url ~ /" + " && obj.http.TYPO3-Sitename == " + 
req.http.Varnish-Ban-TYPO3-Sitename);
			return (synth(200, "Banned all on site "+ 
req.http.Varnish-Ban-TYPO3-Sitename)) ;
		} else if(req.http.Varnish-Ban-All == "1") {
			ban("req.url ~ /");
			return (synth(200, "Banned all"));
		}

		if(req.http.Varnish-Ban-TYPO3-Pid && 
req.http.Varnish-Ban-TYPO3-Sitename) {
			ban("obj.http.TYPO3-Pid == " + req.http.Varnish-Ban-TYPO3-Pid + " && 
obj.http.TYPO3-Sitename == " + req.http.Varnish-Ban-TYPO3-Sitename);
			return (synth(202, "Banned TYPO3 pid " + 
req.http.Varnish-Ban-TYPO3-Pid + " on site " + 
req.http.Varnish-Ban-TYPO3-Sitename));
		} else if(req.http.Varnish-Ban-TYPO3-Pid) {
			ban("obj.http.TYPO3-Pid == " + req.http.Varnish-Ban-TYPO3-Pid);
			return (synth(200, "Banned TYPO3 pid "+ 
req.http.Varnish-Ban-TYPO3-Pid)) ;
		}

	}

	# Set X-Forwarded-For Header
	if (req.restarts == 0) {

		if (req.http.x-forwarded-for) {
			set req.http.X-Forwarded-For =
			req.http.X-Forwarded-For + ", " + client.ip;
		} else {
			set req.http.X-Forwarded-For = client.ip;
		}

	}

	# Pipe unknown Methods
	if (req.method != "GET" &&
		req.method != "HEAD" &&
		req.method != "PUT" &&
		req.method != "POST" &&
		req.method != "TRACE" &&
		req.method != "OPTIONS" &&
		req.method != "DELETE") {
		return (pipe);
	}

	# Cache only GET or HEAD Requests
	if (req.method != "GET" && req.method != "HEAD") {
		return (pass);
	}

	# do not cache TYPO3 BE User requests
	if (req.http.Cookie ~ "be_typo_user" || req.url ~ "^/typo3/") {
		return (pass);
	}

	# do not cache Authorized content
	if (req.http.Authorization) {
		return (pass);
	}

	# do not cache Cookie based content
	if (req.http.Cookie) {
		return (pass);
	}

	# Lookup everything else in the Cache
	return (hash);
}


/*
  * vcl_backend_response
  */

sub vcl_backend_response {
	# Cache only GET or HEAD Requests
	if (bereq.method != "GET" && bereq.method != "HEAD") {
		# set beresp.ttl = 120s;
		set beresp.uncacheable = true;
		return (deliver);
	}

	# Cache static files
	if (bereq.url ~ 
"^[^?]*\.(css|js|htc|txt|swf|flv|pdf|gif|jpe?g|png|ico|woff|ttf|eot|otf|xml|md5|json)($|\?)") 
{
		return (deliver);
	}

	# Cache static Pages
	if (beresp.http.TYPO3-Pid && beresp.http.Pragma == "public") {
		unset beresp.http.Set-Cookie;
		return (deliver);
	}

	# do not cache everything else
	# set beresp.ttl = 120s;
	set beresp.uncacheable = true;
	return (deliver);
}


/*
  * vcl_deliver
  */

sub vcl_deliver {
	# Expires Header set by TYPO3 are used to define Varnish caching only
	# therefore do not send them to the Client
	if (resp.http.TYPO3-Pid && resp.http.Pragma == "public") {
		unset resp.http.expires;
		unset resp.http.pragma;
		unset resp.http.cache-control;
	}

	# smart Ban related
	unset resp.http.TYPO3-Pid;
	unset resp.http.TYPO3-Sitename;
	# show cache hits
         if (obj.hits > 0) {
                 set resp.http.X-Cache = "HIT";
         } else {
                 set resp.http.X-Cache = "MISS";
         }

	return (deliver);
}


It does cache jpgs etc.

It's varnish 4.1 on FreeBSD 10.1 (amd64).



Rainer





From hosting at snowflake.ch  Tue Jan 26 16:35:18 2016
From: hosting at snowflake.ch (Andri Steiner | snowflake Hosting)
Date: Tue, 26 Jan 2016 17:35:18 +0100
Subject: Why doesn't varnish cache these requests?
In-Reply-To: <e4d4687f1f7fd821392addece9833ba5@ultra-secure.de>
References: <e4d4687f1f7fd821392addece9833ba5@ultra-secure.de>
Message-ID: <56A7A046.1080507@snowflake.ch>

Hi Rainer,

>> GET /the_url/ HTTP/1.1
>> Host: the_server
>> User-Agent: curl/7.46.0
>> Accept: */*
>>
> < HTTP/1.1 200 OK
> < Server: nginx
> < Date: Tue, 26 Jan 2016 15:30:39 GMT
> < Content-Type: text/html; charset=utf-8
> < Content-Length: 31195
> < Connection: keep-alive
> < Content-Language: de
> < Expires: Thu, 25 Feb 2016 15:07:05 GMT
> < ETag: "c8b13cf27951a887cd9e54f307456e2e"
> < Cache-Control: max-age=2590586
> < Pragma: public
> < TYPO3-Pid: 6
> < TYPO3-Sitename: 6a48956158576b1c9fcab0ff657e8ef8e337b750

I cannot see anything wrong with this response, the VCL checks on
beresp.http.TYPO3-Pid && beresp.http.Pragma == "public" which is both
fulfilled.

Can you share a varnishlog output of such a request?


> Note that varnish is on a different server than nginx.

Does not matter as it seems you have TYPO3 configured properly because
otherwise the TYPO3-* headers would not be there at all. BTW, you could
add your backend server to the "ban" ACL, without this the TYPO3 Backend
wont be allowed to purge the Varnish cache.

Cheers,
Andri



From rainer at ultra-secure.de  Wed Jan 27 00:22:10 2016
From: rainer at ultra-secure.de (Rainer Duffner)
Date: Wed, 27 Jan 2016 01:22:10 +0100
Subject: Why doesn't varnish cache these requests?
In-Reply-To: <56A7A046.1080507@snowflake.ch>
References: <e4d4687f1f7fd821392addece9833ba5@ultra-secure.de>
 <56A7A046.1080507@snowflake.ch>
Message-ID: <A2B6B289-3A15-47E6-A78A-95E1A27A40D1@ultra-secure.de>


> Am 26.01.2016 um 17:35 schrieb Andri Steiner | snowflake Hosting <hosting at snowflake.ch>:
> 
> Does not matter as it seems you have TYPO3 configured properly because
> otherwise the TYPO3-* headers would not be there at all. BTW, you could
> add your backend server to the "ban" ACL, without this the TYPO3 Backend
> wont be allowed to purge the Varnish cache.


Yes, I did that now.

The site is still in development and all the typo3 stuff is done by someone else (i.e. 3rd party).
Maybe they fixed it in the meantime.  Though, it should have come from nginx anyway.

Because I see that I get a HIT now, using curl.

Which is a bit puzzling.

Though, when I use FF + Firebug, I still get a MISS. Somehow, somewhere, it?s still cached in the browser :-(
In Opera, I do see the HITs.

Still, I?ve got a loading-time for the larger images that looks a bit too much.

Site has 49 elements, 3.1MB.
Though, when I check your company?s start-page with the various tools out there, it shows similar behavior with larger images.

The site takes <2s to load on my 20/2 DSL.
Loading times vary depending on the location. I get 5-6s from various European locations via webpagetest.org <http://webpagetest.org/>, and about 10-12 seconds via Asia and 5-6 seconds from the US.



Rainer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160127/52b21e33/attachment.html>

From lee at leetrout.com  Wed Jan 27 06:29:59 2016
From: lee at leetrout.com (Lee Trout)
Date: Wed, 27 Jan 2016 01:29:59 -0500
Subject: Trouble with caching POSTs (using 4.1 & libvmod-bodyaccess)
Message-ID: <CAMN4RjKVEZWKsfgWYNgs6O1JNXb__AnfrkVZ1t1p73qWykV5Nw@mail.gmail.com>

Hey everyone,

I've been stumped tonight trying to cache POST requests based on body
content. I first tried 4.0.3 with libvmod-bodyaccess and I was getting
400's from my upstream server from what appeared to be a missing content
length header (or maybe even missing body). I saw another email on this
list from last year where PHK thought there was a bug that had been fixed.

So I moved on to 4.1 and rebuilt libvmod-bodyaccess but unfortunately now
I'm just seeing a crash so I'm sure I've done something silly.

Any help would be greatly appreciated - VCL and varnish log below...

Thanks!

VCL

# cat /etc/varnish/default.vcl
vcl 4.0;
import bodyaccess;
import std;
# Requires libvmod-bodyaccess
# See
http://info.varnish-software.com/blog/introducing-libvmod-bodyaccess-reqbody-access-functions
# Check this file once installed with
# sudo /usr/sbin/varnishd -C -f /etc/varnish/default.vcl

backend default {
    .host = "10.1.1.109";
    .port = "80";
}


# Hash requests based on the request body (model input)
sub vcl_hash {
  if (req.method == "POST") {
    bodyaccess.hash_req_body();
  }
    return(lookup);
}

# On cache things that look like model api calls
sub vcl_recv {
    if (req.url ~ "/([-0-9a-zA-Z]+)/models/([-0-9a-zA-Z]+)/" && req.method
== "POST") {
        # Set the method as a header so we can reuse it for the fetch
        set req.http.x-meth = req.method;
        # Buffer the request body
        std.cache_req_body(10KB);
        return (hash);
    }
    return (pass);
}

sub vcl_backend_fetch {
  # Reuse our method we kept as a header in recv
  set bereq.method = bereq.http.x-meth;
  return(fetch);
}

sub vcl_backend_response {
  # Cache everything for 5 minutes
  set beresp.ttl = 5m;
}

sub vcl_deliver {
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }
}


and varnish log


*              << Session  >>   1
-            1 Begin          c sess 0 HTTP/1
-            1 SessOpen       c 50.142.106.39 46903 :80 10.1.1.145 80
1453875715.569886 9
-            1 Link           c req 2 rxreq
-            1 VSL            - flush
-            1 End            - synth

*              << Request  >>   2
-            2 Begin          c req 1 rxreq
-            2 Timestamp      c Start: 1453875715.569958 0.000000 0.000000
-            2 Timestamp      c Req: 1453875715.569958 0.000000 0.000000
-            2 ReqStart       c 50.142.106.39 46903
-            2 ReqMethod      c POST
-            2 ReqURL         c /lee/models/HelloWorld/
-            2 ReqProtocol    c HTTP/1.1
-            2 ReqHeader      c Authorization: Basic
bGVlOmU0YjFjMTAyZTI1YmYyNjk2Njk1NzRkMjE2MDlhMGZm
-            2 ReqHeader      c Host: 54.213.164.12
-            2 ReqHeader      c Connection: close
-            2 ReqHeader      c User-Agent: Paw/2.2.9 (Macintosh; OS
X/10.11.3) GCDHTTPRequest
-            2 ReqHeader      c Content-Length: 21
-            2 ReqHeader      c X-Forwarded-For: 50.142.106.39
-            2 VCL_call       c RECV
-            2 ReqHeader      c x-meth: POST
-            2 Storage        c malloc Transient
-            2 Timestamp      c ReqBody: 1453875715.570038 0.000079 0.000079
-            2 Debug          c "VRT_CacheReqBody(10240): 21"
-            2 VCL_return     c hash
-            2 VCL_call       c HASH
-            2 VSL            c flush
-            2 End            c synth

Log abandoned
Log reacquired
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160127/c1dfc5f1/attachment.html>

From apj at mutt.dk  Wed Jan 27 07:09:19 2016
From: apj at mutt.dk (Andreas Plesner)
Date: Wed, 27 Jan 2016 08:09:19 +0100
Subject: Trouble with caching POSTs (using 4.1 & libvmod-bodyaccess)
In-Reply-To: <CAMN4RjKVEZWKsfgWYNgs6O1JNXb__AnfrkVZ1t1p73qWykV5Nw@mail.gmail.com>
References: <CAMN4RjKVEZWKsfgWYNgs6O1JNXb__AnfrkVZ1t1p73qWykV5Nw@mail.gmail.com>
Message-ID: <20160127070919.GB2656@nerd.dk>

On Wed, Jan 27, 2016 at 01:29:59AM -0500, Lee Trout wrote:
> 
> So I moved on to 4.1 and rebuilt libvmod-bodyaccess but unfortunately now
> I'm just seeing a crash so I'm sure I've done something silly.

Did you use the 4.1 branch of the vmod?

-- 
Andreas



From ericksson.faustino at gmail.com  Wed Jan 27 11:44:49 2016
From: ericksson.faustino at gmail.com (Erick Faustino)
Date: Wed, 27 Jan 2016 09:44:49 -0200
Subject: Varnish 503 responses
Message-ID: <CAEx-NVawzpDkgV=zLfFhGAh8KLVxukySeOAp5xBCX=J6B-_xKg@mail.gmail.com>

Hi, everybody!

We are using varnish, to cache our website. We are using varnish inside
AWS, so we have an ELB with 2 instances (autoscaling) with varnish + nginx.
Everything is fine, until the backend application has some fails (5xx
responses) or timeouts then varnish has much more 5xx responses than the
backend.

 For example: If backend respond 300 reqs with 5xx http status, varnish
delivers some stale content (thats ok and expected due our vcl config), but
the 5xx responses is ten times or more than backend, like 5000.
I think that the responses from varnish, should be very similar with the
backend, right? Why this happens? Where can I look to clarify this?
(varnishstat, varnishlog, varnishncsa log and etc). I would appreciate any
help. If you need more information about my config, feel free to ask.

Best regards,

Erick Faustino
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160127/7814916f/attachment.html>

From lee at leetrout.com  Wed Jan 27 13:36:00 2016
From: lee at leetrout.com (Lee Trout)
Date: Wed, 27 Jan 2016 08:36:00 -0500
Subject: Trouble with caching POSTs (using 4.1 & libvmod-bodyaccess)
In-Reply-To: <20160127070919.GB2656@nerd.dk>
References: <CAMN4RjKVEZWKsfgWYNgs6O1JNXb__AnfrkVZ1t1p73qWykV5Nw@mail.gmail.com>
 <20160127070919.GB2656@nerd.dk>
Message-ID: <CAMN4Rj+wr=YkwVchVU-BkMGQf+eiZ_-sW8pSBJUcsEGfM-O99g@mail.gmail.com>

Yes I did. I checked out 4.1 and did make clean and then reconfigured and
ran make again and make install again. No issues.

On Wednesday, January 27, 2016, Andreas Plesner <apj at mutt.dk> wrote:

> On Wed, Jan 27, 2016 at 01:29:59AM -0500, Lee Trout wrote:
> >
> > So I moved on to 4.1 and rebuilt libvmod-bodyaccess but unfortunately now
> > I'm just seeing a crash so I'm sure I've done something silly.
>
> Did you use the 4.1 branch of the vmod?
>
> --
> Andreas
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org <javascript:;>
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>


-- 
Sorry I fat thumbed this on an iPhone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160127/43ae2553/attachment.html>

From arianna.aondio at varnish-software.com  Wed Jan 27 14:46:52 2016
From: arianna.aondio at varnish-software.com (Arianna Aondio)
Date: Wed, 27 Jan 2016 15:46:52 +0100
Subject: varnish-misc Digest, Vol 118, Issue 23
In-Reply-To: <mailman.5.1453892401.23567.varnish-misc@varnish-cache.org>
References: <mailman.5.1453892401.23567.varnish-misc@varnish-cache.org>
Message-ID: <CAL6a+5hQZocM1zMfnOSCtx3kmtg=q3wNe99R56uvQ1yiP83Nqw@mail.gmail.com>

> > So I moved on to 4.1 and rebuilt libvmod-bodyaccess but unfortunately now
> > I'm just seeing a crash so I'm sure I've done something silly.
>
> Did you use the 4.1 branch of the vmod?

Hi Lee, can you please open a github issue for this?
Also can you check if it triggers any panic?
Thanks




-- 
Arianna Aondio
Field Engineer | Varnish Software Group
Mobile: +47 98062619
We Make Websites Fly
www.varnish-software.com



From lee at leetrout.com  Wed Jan 27 15:25:20 2016
From: lee at leetrout.com (Lee Trout)
Date: Wed, 27 Jan 2016 10:25:20 -0500
Subject: varnish-misc Digest, Vol 118, Issue 23
In-Reply-To: <CAL6a+5hQZocM1zMfnOSCtx3kmtg=q3wNe99R56uvQ1yiP83Nqw@mail.gmail.com>
References: <mailman.5.1453892401.23567.varnish-misc@varnish-cache.org>
 <CAL6a+5hQZocM1zMfnOSCtx3kmtg=q3wNe99R56uvQ1yiP83Nqw@mail.gmail.com>
Message-ID: <CAMN4Rj+wVZoWzBC4xizN8xwVj8o1FCjm_RK9DV-J+DAeSq20Og@mail.gmail.com>

Sorry, I should have ran debug mode last night- too tired to think clearly
:)

The info for the panic is

Assert error in VRT_count(), cache/cache_vcl.c line 362:
  Condition((ctx->vcl)->magic == 0x214188f2) not true.

and I opened an issue https://github.com/aondio/libvmod-bodyaccess/issues/7

Thanks!

On Wed, Jan 27, 2016 at 9:46 AM, Arianna Aondio <
arianna.aondio at varnish-software.com> wrote:

> > > So I moved on to 4.1 and rebuilt libvmod-bodyaccess but unfortunately
> now
> > > I'm just seeing a crash so I'm sure I've done something silly.
> >
> > Did you use the 4.1 branch of the vmod?
>
> Hi Lee, can you please open a github issue for this?
> Also can you check if it triggers any panic?
> Thanks
>
>
>
>
> --
> Arianna Aondio
> Field Engineer | Varnish Software Group
> Mobile: +47 98062619
> We Make Websites Fly
> www.varnish-software.com
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160127/59667727/attachment.html>

From varnish at tengu.ch  Thu Jan 28 09:00:45 2016
From: varnish at tengu.ch (=?UTF-8?Q?C=c3=a9dric_Jeanneret?=)
Date: Thu, 28 Jan 2016 10:00:45 +0100
Subject: ACLs limts/perfs ?
Message-ID: <56A9D8BD.3080208@tengu.ch>

Hello,

We're currently wanting to implement spamhaus "drop" and "edrop" lists
so that we can deny access to those bad networks.

In order to do so, we've created a small script that downloads those
lists, parse them and create an ACL, in for form:
acl spamhaus {
 "net"/netmask;
  ...
}

After the whole process is finished the ACL holds more than 800 entries.

Will it be an issue for a high-traffic application? Will varnish have
some performance issues with such a long list?

If so, what do you propose as alternative? Multiple ACL blocks?

Thank you in advance for your help.

Cheers,

C.



From phk at phk.freebsd.dk  Thu Jan 28 09:33:17 2016
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Thu, 28 Jan 2016 09:33:17 +0000
Subject: ACLs limts/perfs ?
In-Reply-To: <56A9D8BD.3080208@tengu.ch>
References: <56A9D8BD.3080208@tengu.ch>
Message-ID: <78956.1453973597@critter.freebsd.dk>

--------
In message <56A9D8BD.3080208 at tengu.ch>, =?UTF-8?Q?C=c3=a9dric_Jeanneret?= write
s:

>After the whole process is finished the ACL holds more than 800 entries.

Varnish compiles ACL's into optimized C-code (you can see this if
you use the -C option), and 800 entries should not be a problem
at all.

If your system uses the GCC compiler, compilation will take longer
time as the length of your ACLs grow, but 800 entries shouldn't be
an issue at all.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



From varnish at tengu.ch  Thu Jan 28 09:44:07 2016
From: varnish at tengu.ch (=?UTF-8?Q?C=c3=a9dric_Jeanneret?=)
Date: Thu, 28 Jan 2016 10:44:07 +0100
Subject: ACLs limts/perfs ?
In-Reply-To: <78956.1453973597@critter.freebsd.dk>
References: <56A9D8BD.3080208@tengu.ch> <78956.1453973597@critter.freebsd.dk>
Message-ID: <56A9E2E7.9090309@tengu.ch>

On 01/28/2016 10:33 AM, Poul-Henning Kamp wrote:
> --------
> In message <56A9D8BD.3080208 at tengu.ch>, =?UTF-8?Q?C=c3=a9dric_Jeanneret?= write
> s:
> 
>> After the whole process is finished the ACL holds more than 800 entries.
> 
> Varnish compiles ACL's into optimized C-code (you can see this if
> you use the -C option), and 800 entries should not be a problem
> at all.
> 
> If your system uses the GCC compiler, compilation will take longer
> time as the length of your ACLs grow, but 800 entries shouldn't be
> an issue at all.
> 

Hello,

Thanks for your answer ? that was also our thoughts. It's pretty
impressive to see how far we can go with varnish, as well as how many
connections it can handle without a single leap :).

Regarding the GCC: I think it uses it (default debian install) ? is
there a way to make the compilation a bit faster? Our vcl is big (really
big)? and it takes some minutes to compile each time. Yes. Minutes. [khof].

Cheers,

C.



From tobias.eichelbroenner at lamp-solutions.de  Thu Jan 28 10:38:02 2016
From: tobias.eichelbroenner at lamp-solutions.de (=?UTF-8?Q?Tobias_Eichelbr=c3=b6nner?=)
Date: Thu, 28 Jan 2016 11:38:02 +0100
Subject: ACLs limts/perfs ?
In-Reply-To: <56A9E2E7.9090309@tengu.ch>
References: <56A9D8BD.3080208@tengu.ch> <78956.1453973597@critter.freebsd.dk>
 <56A9E2E7.9090309@tengu.ch>
Message-ID: <56A9EF8A.3070502@lamp-solutions.de>

Hi,
> Thanks for your answer ? that was also our thoughts. It's pretty
> impressive to see how far we can go with varnish, as well as how many
> connections it can handle without a single leap :).

we had once a varnish setup running containing VCL-Code with several
million lines generated by a script. Varnish was doing fine delivering
some 1000/s.

> Regarding the GCC: I think it uses it (default debian install) ? is
> there a way to make the compilation a bit faster? Our vcl is big (really
> big)? and it takes some minutes to compile each time. Yes. Minutes. [khof].

We also hat do deal with this. It took us more than an hour to compile
the vcl. Our solutions was to compile it on a backend server, copy the
.so File to the frontends and load it into the running Varnishes there.

Sincerly,

Tobias

P.S.: By the way. Later we took the data contained within the VCL and
put it into a redis db. We wrote a simple vmod to query the information
out of the database and does all required logic based on the data with
some simple lines of code. Maybe this is a better solution for you, too.


-- 
LAMP solutions GmbH
Gostenhofer Hauptstrasse 35
90443 Nuernberg

Amtsgericht Nuernberg: HRB 22366
Geschaeftsfuehrer: Heiko Schubert

Es gelten unsere allgemeinen Geschaeftsbedingungen.
http://www.lamp-solutions.de/agb/

Telefon      : 0911 / 376 516 0
Fax          : 0911 / 376 516 11
E-Mail       : support at lamp-solutions.de
Web          : www.lamp-solutions.de
Facebook     : http://www.facebook.com/LAMPsolutions
Twitter      : http://twitter.com/#!/lampsolutions



From kb at kbojens.de  Thu Jan 28 11:39:31 2016
From: kb at kbojens.de (Kai Bojens)
Date: Thu, 28 Jan 2016 12:39:31 +0100
Subject: ACLs limts/perfs ?
In-Reply-To: <56A9EF8A.3070502@lamp-solutions.de>
References: <56A9D8BD.3080208@tengu.ch> <78956.1453973597@critter.freebsd.dk>
 <56A9E2E7.9090309@tengu.ch> <56A9EF8A.3070502@lamp-solutions.de>
Message-ID: <20160128113931.GA5528@voyager.kbojens.de>

On 28/01/2016, Tobias Eichelbr?nner wrote:
 
> we had once a varnish setup running containing VCL-Code with several
> million lines generated by a script. Varnish was doing fine delivering
> some 1000/s.

Just out of curiosity: why would you have several million lines? I somehow
can't see any use case in which this could happen. But maybe I haven't seen
very much of varnish at all ;)



From tobias.eichelbroenner at lamp-solutions.de  Thu Jan 28 13:20:57 2016
From: tobias.eichelbroenner at lamp-solutions.de (=?UTF-8?Q?Tobias_Eichelbr=c3=b6nner?=)
Date: Thu, 28 Jan 2016 14:20:57 +0100
Subject: ACLs limts/perfs ?
In-Reply-To: <20160128113931.GA5528@voyager.kbojens.de>
References: <56A9D8BD.3080208@tengu.ch> <78956.1453973597@critter.freebsd.dk>
 <56A9E2E7.9090309@tengu.ch> <56A9EF8A.3070502@lamp-solutions.de>
 <20160128113931.GA5528@voyager.kbojens.de>
Message-ID: <56AA15B9.9060803@lamp-solutions.de>

Hi Kai,

> Just out of curiosity: why would you have several million lines? I somehow
> can't see any use case in which this could happen.

to get the actual use case we had, think of a adservers delivering
different ads based on browser, country, connection, and other criteria.
Now you start just redirecting users to external partners if you cannot
serf a proper ad yourself.
You start with some simple rules hardcoded in your caching proxy so that
your backend is not bothered to calculate anything. As it works great
you start to add additional rules.
Then you write a script that ads new rules taken from a database....

Of cause we did not use varnish as a caching proxy anymore, but more as
an application server. But especially when it comes to performance
varnish does a great job running simple applications.
I think the varnish plus package offered from the Varnish guys has
features that are not limited to simple caching (like paywalls)

Sincerly,

Tobias

-- 
LAMP solutions GmbH
Gostenhofer Hauptstrasse 35
90443 Nuernberg

Amtsgericht Nuernberg: HRB 22366
Geschaeftsfuehrer: Heiko Schubert

Es gelten unsere allgemeinen Geschaeftsbedingungen.
http://www.lamp-solutions.de/agb/

Telefon      : 0911 / 376 516 0
Fax          : 0911 / 376 516 11
E-Mail       : support at lamp-solutions.de
Web          : www.lamp-solutions.de
Facebook     : http://www.facebook.com/LAMPsolutions
Twitter      : http://twitter.com/#!/lampsolutions



From lee at leetrout.com  Thu Jan 28 17:01:20 2016
From: lee at leetrout.com (Lee Trout)
Date: Thu, 28 Jan 2016 12:01:20 -0500
Subject: Trouble with caching POSTs (using 4.1 & libvmod-bodyaccess)
In-Reply-To: <CAMN4Rj+wr=YkwVchVU-BkMGQf+eiZ_-sW8pSBJUcsEGfM-O99g@mail.gmail.com>
References: <CAMN4RjKVEZWKsfgWYNgs6O1JNXb__AnfrkVZ1t1p73qWykV5Nw@mail.gmail.com>
 <20160127070919.GB2656@nerd.dk>
 <CAMN4Rj+wr=YkwVchVU-BkMGQf+eiZ_-sW8pSBJUcsEGfM-O99g@mail.gmail.com>
Message-ID: <CAMN4RjJBSUDrb7JkgOA0+FG_AZ7VBv8W=e_=GudcHDvy6M0uzg@mail.gmail.com>

Hey everyone. Big thanks to Arianna for the quick fix on the bodyaccess
vmod.

Unfortunately I'm still not seeing varnish sending the original body with
the request.

Varnish is sending:

POST /lee/models/HelloWorld/ HTTP/1.1
Authorization: Basic bGVlOmU0YjFjMTAyZTI1YmYyNjk2Njk1NzRkMjE2MDlhMGZm
Host: 127.0.0.1:12080
User-Agent: Paw/2.2.9 (Macintosh; OS X/10.11.3) GCDHTTPRequest
X-Forwarded-For: 10.0.2.2
x-meth: POST
Accept-Encoding: gzip
X-Varnish: 3

I'm assuming at this point this just isn't doable.

Has anyone successfully cached a POST request?

On Wed, Jan 27, 2016 at 8:36 AM, Lee Trout <lee at leetrout.com> wrote:

> Yes I did. I checked out 4.1 and did make clean and then reconfigured and
> ran make again and make install again. No issues.
>
>
> On Wednesday, January 27, 2016, Andreas Plesner <apj at mutt.dk> wrote:
>
>> On Wed, Jan 27, 2016 at 01:29:59AM -0500, Lee Trout wrote:
>> >
>> > So I moved on to 4.1 and rebuilt libvmod-bodyaccess but unfortunately
>> now
>> > I'm just seeing a crash so I'm sure I've done something silly.
>>
>> Did you use the 4.1 branch of the vmod?
>>
>> --
>> Andreas
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
> --
> Sorry I fat thumbed this on an iPhone
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/vinyl-misc/attachments/20160128/aa83dddc/attachment.html>

From yamakasi.014 at gmail.com  Sat Jan 30 22:26:18 2016
From: yamakasi.014 at gmail.com (Matt .)
Date: Sat, 30 Jan 2016 23:26:18 +0100
Subject: Rewrite name.domain.tld to domain.tld/foo/bar-name
Message-ID: <CAPNQp07aPDwRcFOVRXDP-A-h=jAw13Sz19J-FqKcBrj8htSsRA@mail.gmail.com>

HI,

I'm trying to rewrite a domainname what is typed in the addressbar but
keep it overthere that way.

At the moment my vcl ends up with:

    set req.http.Host = regsub(req.http.X-Redirect-To, {"^([^/]+)/.*"}, "\1");
                set req.url = regsub(req.http.X-Redirect-To,
{"[^/]+(/.*)"}, "\1");
                std.syslog(29, "Redirecting to host "+req.http.Host+";
URI="+req.url);
                return (pass);

But this really seems to redirect.

What are my options here ?

Thanks,

Matt



From cfernand at sju.edu  Sun Jan 31 02:19:57 2016
From: cfernand at sju.edu (=?utf-8?Q?Carlos_M=2E_Fern=C3=A1ndez?=)
Date: Sat, 30 Jan 2016 21:19:57 -0500 (EST)
Subject: Rewrite name.domain.tld to domain.tld/foo/bar-name
In-Reply-To: <CAPNQp07aPDwRcFOVRXDP-A-h=jAw13Sz19J-FqKcBrj8htSsRA@mail.gmail.com>
References: <CAPNQp07aPDwRcFOVRXDP-A-h=jAw13Sz19J-FqKcBrj8htSsRA@mail.gmail.com>
Message-ID: <AB71914F-30F7-4D68-954A-C29ACDFE091A@sju.edu>

Which version of Varnish?

It seems that you want to change the URL presented to the backend. In that case, you should look into the URI and host name in the backend request instead of the client request, likely in vcl_backend_fetch (version 4.0+).

Best regards,
--
Carlos M. Fern?ndez
Enterprise Systems Manager
Saint Joseph's University
W: 610-660-1501
M: 215-316-1193
E: cfernand at sju.edu

> On Jan 30, 2016, at 17:39, Matt . <yamakasi.014 at gmail.com> wrote:
> 
> HI,
> 
> I'm trying to rewrite a domainname what is typed in the addressbar but
> keep it overthere that way.
> 
> At the moment my vcl ends up with:
> 
>    set req.http.Host = regsub(req.http.X-Redirect-To, {"^([^/]+)/.*"}, "\1");
>                set req.url = regsub(req.http.X-Redirect-To,
> {"[^/]+(/.*)"}, "\1");
>                std.syslog(29, "Redirecting to host "+req.http.Host+";
> URI="+req.url);
>                return (pass);
> 
> But this really seems to redirect.
> 
> What are my options here ?
> 
> Thanks,
> 
> Matt
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

From yamakasi.014 at gmail.com  Sun Jan 31 03:00:21 2016
From: yamakasi.014 at gmail.com (Matt .)
Date: Sun, 31 Jan 2016 04:00:21 +0100
Subject: Rewrite name.domain.tld to domain.tld/foo/bar-name
In-Reply-To: <AB71914F-30F7-4D68-954A-C29ACDFE091A@sju.edu>
References: <CAPNQp07aPDwRcFOVRXDP-A-h=jAw13Sz19J-FqKcBrj8htSsRA@mail.gmail.com>
 <AB71914F-30F7-4D68-954A-C29ACDFE091A@sju.edu>
Message-ID: <CAPNQp07Jn0G8RxDnHmRafesQq=3O=FKzNqSwi3je9ejbDfE-iw@mail.gmail.com>

varnish 4.0

The issue is that when I set a /$1 behind my destination it redirects
to the destination and doesn't rewrite.

2016-01-31 3:19 GMT+01:00 Carlos M. Fern?ndez <cfernand at sju.edu>:
> Which version of Varnish?
>
> It seems that you want to change the URL presented to the backend. In that case, you should look into the URI and host name in the backend request instead of the client request, likely in vcl_backend_fetch (version 4.0+).
>
> Best regards,
> --
> Carlos M. Fern?ndez
> Enterprise Systems Manager
> Saint Joseph's University
> W: 610-660-1501
> M: 215-316-1193
> E: cfernand at sju.edu
>
>> On Jan 30, 2016, at 17:39, Matt . <yamakasi.014 at gmail.com> wrote:
>>
>> HI,
>>
>> I'm trying to rewrite a domainname what is typed in the addressbar but
>> keep it overthere that way.
>>
>> At the moment my vcl ends up with:
>>
>>    set req.http.Host = regsub(req.http.X-Redirect-To, {"^([^/]+)/.*"}, "\1");
>>                set req.url = regsub(req.http.X-Redirect-To,
>> {"[^/]+(/.*)"}, "\1");
>>                std.syslog(29, "Redirecting to host "+req.http.Host+";
>> URI="+req.url);
>>                return (pass);
>>
>> But this really seems to redirect.
>>
>> What are my options here ?
>>
>> Thanks,
>>
>> Matt
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From yamakasi.014 at gmail.com  Sun Jan 31 03:20:16 2016
From: yamakasi.014 at gmail.com (Matt .)
Date: Sun, 31 Jan 2016 04:20:16 +0100
Subject: Rewrite name.domain.tld to domain.tld/foo/bar-name
In-Reply-To: <CAPNQp07Jn0G8RxDnHmRafesQq=3O=FKzNqSwi3je9ejbDfE-iw@mail.gmail.com>
References: <CAPNQp07aPDwRcFOVRXDP-A-h=jAw13Sz19J-FqKcBrj8htSsRA@mail.gmail.com>
 <AB71914F-30F7-4D68-954A-C29ACDFE091A@sju.edu>
 <CAPNQp07Jn0G8RxDnHmRafesQq=3O=FKzNqSwi3je9ejbDfE-iw@mail.gmail.com>
Message-ID: <CAPNQp05MV=F75jR_dkqWYA1dLY+Qi8WPWXZ0QVA-daMaxeW-rQ@mail.gmail.com>

I'm actually kinda stuck here with the following:

I redirect as I said:

name.domain.tld -> domain.tld/foo/bar-name

this works OK, but as I can access domain.tld/foo/bar/name also
normally it works with local javascript:

/js/libs/etc...

but when I access that page using my rewrite it cannot find the root
ofcourse for that javascript folder and I cannot place it on a static
server, that is a big issue.

so: username.domain.tld/js is not found and my script falls back to /js....


can I do something on apache about this or do I need to handle it on
varnish as that actually does the rewrite

2016-01-31 4:00 GMT+01:00 Matt . <yamakasi.014 at gmail.com>:
> varnish 4.0
>
> The issue is that when I set a /$1 behind my destination it redirects
> to the destination and doesn't rewrite.
>
> 2016-01-31 3:19 GMT+01:00 Carlos M. Fern?ndez <cfernand at sju.edu>:
>> Which version of Varnish?
>>
>> It seems that you want to change the URL presented to the backend. In that case, you should look into the URI and host name in the backend request instead of the client request, likely in vcl_backend_fetch (version 4.0+).
>>
>> Best regards,
>> --
>> Carlos M. Fern?ndez
>> Enterprise Systems Manager
>> Saint Joseph's University
>> W: 610-660-1501
>> M: 215-316-1193
>> E: cfernand at sju.edu
>>
>>> On Jan 30, 2016, at 17:39, Matt . <yamakasi.014 at gmail.com> wrote:
>>>
>>> HI,
>>>
>>> I'm trying to rewrite a domainname what is typed in the addressbar but
>>> keep it overthere that way.
>>>
>>> At the moment my vcl ends up with:
>>>
>>>    set req.http.Host = regsub(req.http.X-Redirect-To, {"^([^/]+)/.*"}, "\1");
>>>                set req.url = regsub(req.http.X-Redirect-To,
>>> {"[^/]+(/.*)"}, "\1");
>>>                std.syslog(29, "Redirecting to host "+req.http.Host+";
>>> URI="+req.url);
>>>                return (pass);
>>>
>>> But this really seems to redirect.
>>>
>>> What are my options here ?
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From yamakasi.014 at gmail.com  Sun Jan 31 05:08:59 2016
From: yamakasi.014 at gmail.com (Matt .)
Date: Sun, 31 Jan 2016 06:08:59 +0100
Subject: Avoid trailingslash redirect
Message-ID: <CAPNQp04mtyGqTgPyfLUwnoO0s0_VgUoeAnaXd7QjJDygKd_Nzg@mail.gmail.com>

when I have a rewrite rule which does:

subdomain.domain.tld

with a destination host

domain.tld/subdomain and I place a $1 like

domain.tld/subdomain$1

my subdomain.domain.tld gets redirected to domain.tld/subdomain/

so there is no rewrite anymore but all my links behind
subdomain.domain.tld/foo work perfectly.

When I place a slash in front of the /$1 my /foo doesn't work anymore.

When I place nothing at all I can visit my subdomain.domain.tld and
get the page but no links work anymore as I'm missing the $1

I am using this vmod:
https://www.varnish-cache.org/vmod/database-driven-rewrites

Anyone knows why this happens ?

Cheers,

Matt



From yamakasi.014 at gmail.com  Sun Jan 31 12:51:57 2016
From: yamakasi.014 at gmail.com (Matt .)
Date: Sun, 31 Jan 2016 13:51:57 +0100
Subject: Rewrite name.domain.tld to domain.tld/foo/bar-name
In-Reply-To: <CAPNQp05MV=F75jR_dkqWYA1dLY+Qi8WPWXZ0QVA-daMaxeW-rQ@mail.gmail.com>
References: <CAPNQp07aPDwRcFOVRXDP-A-h=jAw13Sz19J-FqKcBrj8htSsRA@mail.gmail.com>
 <AB71914F-30F7-4D68-954A-C29ACDFE091A@sju.edu>
 <CAPNQp07Jn0G8RxDnHmRafesQq=3O=FKzNqSwi3je9ejbDfE-iw@mail.gmail.com>
 <CAPNQp05MV=F75jR_dkqWYA1dLY+Qi8WPWXZ0QVA-daMaxeW-rQ@mail.gmail.com>
Message-ID: <CAPNQp07LzZB5J8CyZ3iMwXsPukCB4LXCv18dWGuSCVATdbn7sA@mail.gmail.com>

OK, it seems that I have a trailingslash which is added and messes up

I see domain.tld/foo/subdomain//bla and I expect domain.tld/foo/subdomain/bla

this is actually what I do subdomain.domain.tld
domain.tld/foo/subdomain/$1  $url  (.*)

Is this a normal behaviour ?


2016-01-31 4:20 GMT+01:00 Matt . <yamakasi.014 at gmail.com>:
> I'm actually kinda stuck here with the following:
>
> I redirect as I said:
>
> name.domain.tld -> domain.tld/foo/bar-name
>
> this works OK, but as I can access domain.tld/foo/bar/name also
> normally it works with local javascript:
>
> /js/libs/etc...
>
> but when I access that page using my rewrite it cannot find the root
> ofcourse for that javascript folder and I cannot place it on a static
> server, that is a big issue.
>
> so: username.domain.tld/js is not found and my script falls back to /js....
>
>
> can I do something on apache about this or do I need to handle it on
> varnish as that actually does the rewrite
>
> 2016-01-31 4:00 GMT+01:00 Matt . <yamakasi.014 at gmail.com>:
>> varnish 4.0
>>
>> The issue is that when I set a /$1 behind my destination it redirects
>> to the destination and doesn't rewrite.
>>
>> 2016-01-31 3:19 GMT+01:00 Carlos M. Fern?ndez <cfernand at sju.edu>:
>>> Which version of Varnish?
>>>
>>> It seems that you want to change the URL presented to the backend. In that case, you should look into the URI and host name in the backend request instead of the client request, likely in vcl_backend_fetch (version 4.0+).
>>>
>>> Best regards,
>>> --
>>> Carlos M. Fern?ndez
>>> Enterprise Systems Manager
>>> Saint Joseph's University
>>> W: 610-660-1501
>>> M: 215-316-1193
>>> E: cfernand at sju.edu
>>>
>>>> On Jan 30, 2016, at 17:39, Matt . <yamakasi.014 at gmail.com> wrote:
>>>>
>>>> HI,
>>>>
>>>> I'm trying to rewrite a domainname what is typed in the addressbar but
>>>> keep it overthere that way.
>>>>
>>>> At the moment my vcl ends up with:
>>>>
>>>>    set req.http.Host = regsub(req.http.X-Redirect-To, {"^([^/]+)/.*"}, "\1");
>>>>                set req.url = regsub(req.http.X-Redirect-To,
>>>> {"[^/]+(/.*)"}, "\1");
>>>>                std.syslog(29, "Redirecting to host "+req.http.Host+";
>>>> URI="+req.url);
>>>>                return (pass);
>>>>
>>>> But this really seems to redirect.
>>>>
>>>> What are my options here ?
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc at varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From yamakasi.014 at gmail.com  Sun Jan 31 17:31:37 2016
From: yamakasi.014 at gmail.com (Matt .)
Date: Sun, 31 Jan 2016 18:31:37 +0100
Subject: Catch /js on all domains and rewrite the host
Message-ID: <CAPNQp07AgRTZ_6cmRTRRPvzdJos+bLUYys37sfOw=mQsnuec3A@mail.gmail.com>

Hi All,

I'm fuguring out how I can accomplish the following:

I have subdomain.domain.tld but as this is a rewrite the olders like
/js are not directly available on it.

As my subdomains .domain.tld are dynamic I wonder if I can catch /js
from reg.url for all incoming host(requests) and rewrite it to a
source folder  on another host so it can be served on:

subdomain.domain.tld/js

My script lookups /js/whatever.js which is needed for these libs.

Is there some catchall technique to accomplish this ?

Thanks!

Matt