From straightflush at gmail.com Fri Dec 2 19:50:04 2011 From: straightflush at gmail.com (AD) Date: Fri, 2 Dec 2011 14:50:04 -0500 Subject: per_call in vmod Message-ID: hello, How does per_call know what to cache on in the vmod? Is it based on the parameters to the function or based on the request in the VCL? I have run some tests previously where i make multiple http requests with different URLs and priv_call returns the same cached response. Looking at this page https://www.varnish-cache.org/trac/wiki/ArchitectureVmodStorage it seems if i call a function in a vmod with priv_call with different parameters i should be able to cache them independently. How is this supposed to work? -------------- next part -------------- An HTML attachment was scrubbed... URL: From john at 8t8.eu Mon Dec 5 10:03:20 2011 From: john at 8t8.eu (john at 8t8.eu) Date: Mon, 5 Dec 2011 10:03:20 +0000 Subject: varnishstat n_waitinglist question Message-ID: Hi, I am testing a new setup (squid -> varnish) and I am graphing varnish In order to compare setups. I have read Kristian's blog entry http://kristianlyng.wordpress.com/2009/12/08/varnishstat-for-dummies/ . While trying to setup cache health monitoring I have noticed that n_waitinglist as reported by Varnishstat keeps growing. Can anyone tell me what that means and if I should be about it? I'm running varnish-3.0.2-1.el5.x86_64 on Centos 6 if that makes any difference. Cheers, John -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at varnish-software.com Mon Dec 5 10:12:50 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Mon, 5 Dec 2011 11:12:50 +0100 Subject: per_call in vmod In-Reply-To: References: Message-ID: <20111205101250.GA3717@freud.kly.no> On Fri, Dec 02, 2011 at 02:50:04PM -0500, AD wrote: > hello, > > How does per_call know what to cache on in the vmod? Is it based on the > parameters to the function or based on the request in the VCL? I have run > some tests previously where i make multiple http requests with different > URLs and priv_call returns the same cached response. > > Looking at this page > https://www.varnish-cache.org/trac/wiki/ArchitectureVmodStorage it seems if > i call a function in a vmod with priv_call with different parameters i > should be able to cache them independently. How is this supposed to work? It is cached... per occurrence in VCL. Meaning: For the same call, you will get the same private data regardless of input. It's up to you whether the vmod needs to take the parameters into account. A good example of why this makes sense can be found in the header vmod[1]. It has functions that take multiple arguments, but the first is a regular expression. The regular expression is expected (required) to be static. In other words: You write the regular expression by hand, but the headers it are of course variable. The vmod can use per_call storage to store the compiled regular expression, since the regex never changes. Or: sub vcl_recv { somevmod.foo("blatti"); somevmod.foo("blatti"); somevmod.foo("blatti"); somevmod.foo(req.http.someheader); } will get exactly 4 priv_vcl structures, regardless of the value of req.http.someheader. https://github.com/varnish/libvmod-header - Kristian From mit at stagename.com Mon Dec 5 18:10:15 2011 From: mit at stagename.com (Mit Rowe) Date: Mon, 5 Dec 2011 13:10:15 -0500 Subject: Method in health check Message-ID: Is there a way to specify the method used during health checks, such as using "HEAD" instead of "GET"? (current healthcheck pasted below) probe healthcheck { .url = "/api/system/check/"; .interval = 2s; .timeout = 0.3s; .window = 1; .threshold = 1; .initial = 0; .expected_response = 200; } -- Will 'Mit' Rowe Stagename* *1-866-326-3098 mit at stagename.com www.stagename.com Twitter: @stagename *The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of this information by persons or entities other than the intended recipient is prohibited. If you received this transmission in error, please contact the sender and delete all material contained herein from your computer.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Mon Dec 5 18:14:14 2011 From: perbu at varnish-software.com (Per Buer) Date: Mon, 5 Dec 2011 19:14:14 +0100 Subject: Method in health check In-Reply-To: References: Message-ID: On Mon, Dec 5, 2011 at 7:10 PM, Mit Rowe wrote: > Is there a way to specify the method used during health checks, such as > using "HEAD" instead of "GET"? Yes. probe rawprobe { "HEAD / HTTP/1.1" "Host: www.foo.bar" "Connection: close"; } This is documented in the reference material. Available in "man vcl". Per. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.fernandezcrisial at gmail.com Mon Dec 5 18:24:33 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Mon, 5 Dec 2011 15:24:33 -0300 Subject: Method in health check In-Reply-To: References: Message-ID: Try something like.. this is my probe at ".backend" on my directors: .probe = { .timeout = 3s; .interval = 10s; .window = 10; .threshold = 8; .request = "HEAD / HTTP/1.1" "Host: yourdomain.com" "Connection: close" "Accept-Encoding: compress, gzip" ; } -- Roberto O. Fern?ndez Crisial @rofc On Mon, Dec 5, 2011 at 3:10 PM, Mit Rowe wrote: > Is there a way to specify the method used during health checks, such as > using "HEAD" instead of "GET"? > > (current healthcheck pasted below) > > probe healthcheck { > .url = "/api/system/check/"; > .interval = 2s; > .timeout = 0.3s; > .window = 1; > .threshold = 1; > .initial = 0; > .expected_response = 200; > } > > > > > -- > Will 'Mit' Rowe > Stagename* > *1-866-326-3098 > mit at stagename.com > www.stagename.com > Twitter: @stagename > > *The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of this > information by persons or entities other than the intended recipient is > prohibited. If you received this transmission in error, please contact the > sender and delete all material contained herein from your computer.* > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Dec 5 18:31:41 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 05 Dec 2011 18:31:41 +0000 Subject: Method in health check In-Reply-To: Your message of "Mon, 05 Dec 2011 13:10:15 EST." Message-ID: <57382.1323109901@critter.freebsd.dk> In message , Mit Rowe writes: >Is there a way to specify the method used during health checks, such as >using "HEAD" instead of "GET"? You can specify the full request string with ".request" and thus customize it any way you want. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From mdepot at comcast.net Mon Dec 5 20:12:11 2011 From: mdepot at comcast.net (Mike Depot) Date: Mon, 05 Dec 2011 15:12:11 -0500 Subject: Ubuntu Apt Repo Message-ID: <4EDD259B.4050005@comcast.net> In http://repo.varnish-cache.org/ubuntu/dists/ I would expect to see subdirs named after Ubuntu releases: maverick, natty, oneiric, etc. But currently it contains subdirs for Debian release names instead of Ubuntu names. That means this doc page : http://www.varnish-cache.org/installation/ubuntu would not work as intended. Was that intentional, or just does the ubuntu dir just have an accidental copy of the debian content right now? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Mon Dec 5 20:17:09 2011 From: perbu at varnish-software.com (Per Buer) Date: Mon, 5 Dec 2011 21:17:09 +0100 Subject: Ubuntu Apt Repo In-Reply-To: <4EDD259B.4050005@comcast.net> References: <4EDD259B.4050005@comcast.net> Message-ID: On Mon, Dec 5, 2011 at 9:12 PM, Mike Depot wrote: > In http://repo.varnish-cache.org/ubuntu/dists/ > I would expect to see subdirs named after Ubuntu releases: maverick, > natty, oneiric, etc. > Only Lucid is supported. (The packages should work fine on any other release. At least any newer than 10.04). -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From minoritystorm at gmail.com Tue Dec 6 04:11:02 2011 From: minoritystorm at gmail.com (Brain Stormer) Date: Tue, 6 Dec 2011 05:11:02 +0100 Subject: Tune Varnish for pipe only operations Message-ID: Hello all, I am currently having Varnish deployed as a front-end proxy for a number of back-end servers (media streaming). The back-end servers only communicate with HTTP in the initialization and then complete the session with binary streams. So basically, I do some checks at the `vcl_recv` then if everything is okay I `pipe` them to the back-ends. Currently, I have absolutely NO use from Varnish capabilities of caching. I am currently facing CPU overhead at the `varnishd` process, I have tuned lots of parts in Varnish, So... My question now is, How to tune Varnish to give full attention performance wise to piped requests, putting into mind that all sessions keeps up for hours long. Another question is, Do I still have to define a caching store ? I read on some online resources that caching store is still needed for storage regarding Varnish threads. Awaiting your enlightening ! Thanks, MS -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at simplercomputing.net Tue Dec 6 04:32:28 2011 From: mark at simplercomputing.net (Mark E) Date: Mon, 05 Dec 2011 21:32:28 -0700 Subject: Strip port number In-Reply-To: References: Message-ID: <4EDD9ADC.1040101@simplercomputing.net> Hi, I'm testing Varnish and have it on 6081. Apache is on port 80. So in the browser I use http://site:6081 (where "site" is my actual domain name) to land on the home page. But the home page is not loading, Apache acts like it doesn't recognize the domain at all and returns the default Apache page that would be returned when a domain isn't recognized. So, I think maybe what's happening is that Varnish is passing the port in the host header - e.g. mysite:6081 In varnishlog I can see these lines: ----------------- 12 RxRequest c GET 12 RxURL c / 12 RxProtocol c HTTP/1.1 12 RxHeader c Host: site:6081 ------------------ I have a line in vcl_recv like this that tries to strip the port number off the host header: set req.http.Host = regsub(req.http.host, ":[0-9]+", ""); But that doesn't seem to remedy the problem. So I changed it to this thinking maybe the regex isn't right: set req.http.Host = "site" And even that doesn't work. Apache still returns it's default page instead of the domain's home page. "varnishlog -V" shows "varnish-2.1.3 SVN" so I assume it's Varnish v2.1.3 Anyone know how I can get vcl to work right to load the domain's home page so I can get on with testing? Mark From varnish at mm.quex.org Tue Dec 6 04:54:55 2011 From: varnish at mm.quex.org (Michael Alger) Date: Tue, 6 Dec 2011 12:54:55 +0800 Subject: Strip port number In-Reply-To: <4EDD9ADC.1040101@simplercomputing.net> References: <4EDD9ADC.1040101@simplercomputing.net> Message-ID: <20111206045455.GA13486@grum.quex.org> On Mon, Dec 05, 2011 at 09:32:28PM -0700, Mark E wrote: > > I'm testing Varnish and have it on 6081. Apache is on port 80. > > So in the browser I use http://site:6081 (where "site" is my actual > domain name) to land on the home page. But the home page is not > loading, Apache acts like it doesn't recognize the domain at all and > returns the default Apache page that would be returned when a domain > isn't recognized. > > So, I think maybe what's happening is that Varnish is passing the > port in the host header - e.g. mysite:6081 > [...] > I have a line in vcl_recv like this that tries to strip the port > number off the host header: > [...] > set req.http.Host = "site" > > And even that doesn't work. Apache still returns it's default page > instead of the domain's home page. > > Anyone know how I can get vcl to work right to load the domain's > home page so I can get on with testing? Should be working, so I think something else is the issue. Here's some things to consider off the top of my head. If none of them help, maybe post your vcl_recv() function, anonymized if necessary. - Are you sure your second attempt was fetched from the origin, and not a cached result from earlier attempts? - If you have any conditional logic in vcl_recv double-check you're not returning before you overwrite the host header or something along those lines. - Double-check you're sending a host header it'll understand, e.g. including or excluding the "www." in front as required. - Is Varnish using the correct backend for that host? - It might be worth watching the log output and seeing what headers Varnish is sending in the request to the backend; and/or you could modify your backend so the default page for unrecognised hosts is a script that outputs the value of the host header it received, and maybe also the date+time the response was generated. From mattias at nucleus.be Tue Dec 6 07:28:05 2011 From: mattias at nucleus.be (Mattias Geniar) Date: Tue, 6 Dec 2011 08:28:05 +0100 Subject: Strip port number In-Reply-To: <20111206045455.GA13486@grum.quex.org> References: <4EDD9ADC.1040101@simplercomputing.net> <20111206045455.GA13486@grum.quex.org> Message-ID: <18834F5BEC10824891FB8B22AC821A5A01C6D69B@nucleus-srv01.Nucleus.local> > - Is Varnish using the correct backend for that host? That's probably your biggest bet: if you have Apache's Virtual Hosts bound on the public IP of your server, but you've defined your backend in Varnish to connect to 127.0.0.1, Apache won't be able to match the hostname with the IP address and doesn't know which Virtual Host to serve. Mattias From simon at darkmere.gen.nz Tue Dec 6 09:09:27 2011 From: simon at darkmere.gen.nz (Simon Lyall) Date: Tue, 6 Dec 2011 22:09:27 +1300 (NZDT) Subject: Tune Varnish for pipe only operations In-Reply-To: References: Message-ID: I think varnish might not be the right tool for you, perhaps haproxy might better suit your needs. On Tue, 6 Dec 2011, Brain Stormer wrote: > I am currently having Varnish deployed as a front-end proxy for a number of > back-end servers (media streaming). > > The back-end servers only communicate with HTTP in the initialization and > then complete the session with binary streams. So basically, I do some > checks at the `vcl_recv` then if everything is okay I `pipe` them to the > back-ends. > > Currently, I have absolutely NO use from Varnish capabilities of caching. > > I am currently facing CPU overhead at the `varnishd` process, I have tuned > lots of parts in Varnish, So... > > My question now is, How to tune Varnish to give full attention performance > wise to piped requests, putting into mind that all sessions keeps up for > hours long. > > Another question is, Do I still have to define a caching store ? I read on > some online resources that caching store is still needed for storage > regarding Varnish threads. > > Awaiting your?enlightening?! > > > > Thanks, > MS > > -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. From minoritystorm at gmail.com Tue Dec 6 11:52:58 2011 From: minoritystorm at gmail.com (Brain Stormer) Date: Tue, 6 Dec 2011 12:52:58 +0100 Subject: Tune Varnish for pipe only operations In-Reply-To: References: Message-ID: Well, the back-end servers are some how breaking the HTTP protocol in some way, also it is HTTP/0.9 only, it just not well support by HAPROXY unless running it in binary proxying which won't give me the capability of doing the initial security checks and restrictions over the negotiation packets, Varnish was the only proxy tolerating all the protocol issues and still working! So what you say, is Varnish cannot be tuned for doing such job efficiently ? Thanks, MS On Tue, Dec 6, 2011 at 10:09 AM, Simon Lyall wrote: > > I think varnish might not be the right tool for you, perhaps haproxy might > better suit your needs. > > > On Tue, 6 Dec 2011, Brain Stormer wrote: > >> I am currently having Varnish deployed as a front-end proxy for a number >> of >> back-end servers (media streaming). >> >> The back-end servers only communicate with HTTP in the initialization and >> then complete the session with binary streams. So basically, I do some >> checks at the `vcl_recv` then if everything is okay I `pipe` them to the >> back-ends. >> >> Currently, I have absolutely NO use from Varnish capabilities of caching. >> >> I am currently facing CPU overhead at the `varnishd` process, I have tuned >> lots of parts in Varnish, So... >> >> My question now is, How to tune Varnish to give full attention performance >> wise to piped requests, putting into mind that all sessions keeps up for >> hours long. >> >> Another question is, Do I still have to define a caching store ? I read on >> some online resources that caching store is still needed for storage >> regarding Varnish threads. >> >> Awaiting your enlightening ! >> >> >> >> Thanks, >> MS >> >> >> > -- > Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ > "To stay awake all night adds a day to your life" - Stilgar | eMT. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at ibotty.net Tue Dec 6 21:59:52 2011 From: me at ibotty.net (me at ibotty.net) Date: Tue, 06 Dec 2011 22:59:52 +0100 Subject: mysterious stalls Message-ID: <4EDE9058.3000802@ibotty.net> hi, i have strange not easily reproducible stalls. on a debian squeeze varnish 3.0.2-1~1squeeze1 runs in front of very slow php scripts on apache 2.2.16-6+squeeze3. the machine is a virtual machine with 4gb of ram and 2 cpus. the vcl w/o comments is below, the varnishlog with maybe too much output is at [1]. the vcl is at [2]. i hope, i did not forget necessary information. the problem is the following. sometimes (in this example at about 20:43) it stalls for about one to three minutes. there are some timeouts in that timeframe, but i fail to see the reason. without varnish in front, apache does not show such stalls (but because it has been so extremly slow, noone really knows). i found one bug (#951, [3]), that described similar problems, but here, we explicitly return (pass) these urls. so the workaround mentioned there (req.hash_ignore_busy=true) as expected does not work. if you have any pointer on how to proceed, i'd be glad to know. [1] http://pastebin.com/z78RQwra or http://pastebin.com/download.php?i=z78RQwra for download [2] http://pastebin.com/vGzfHCZC [3] https://www.varnish-cache.org/trac/ticket/951 thanks in advance, tobias florek From cdgraff at gmail.com Wed Dec 7 03:05:24 2011 From: cdgraff at gmail.com (Alejandro) Date: Wed, 7 Dec 2011 00:05:24 -0300 Subject: HA with Varnish? advice request Message-ID: Hi guys, I need your advice... I'm using varnish into multiple server, some for caching dynamic content and other only for static content. On front of the Dynamic Content Varnish y have Cisco ACE Load balancer, but on front of the Static Varnish, the traffic is much higher and can't use Cisco ACE, for this reason I'm using RoundRobin DNS into this 2 servers. This week I had, some network issue with one of this servers and when the network go down I loose 50% of the traffic... here my question, or advice request... How I can have HA with 2 varnish server? I'm thinking on Heartbeat or Heartbeat+ldirectord, please fell free to send your suggestions. Sorry for my English and regards from Argentina Alejandro -------------- next part -------------- An HTML attachment was scrubbed... URL: From allan_wind at lifeintegrity.com Wed Dec 7 05:34:26 2011 From: allan_wind at lifeintegrity.com (Allan Wind) Date: Wed, 7 Dec 2011 00:34:26 -0500 Subject: HA with Varnish? advice request In-Reply-To: References: Message-ID: <20111207053426.GA7422@vent.lifeintegrity.localnet> On 2011-12-07 00:05:24, Alejandro wrote: > Hi guys, > > I need your advice... I'm using varnish into multiple server, some for > caching dynamic content and other only for static content. > > On front of the Dynamic Content Varnish y have Cisco ACE Load balancer, but > on front of the Static Varnish, the traffic is much higher and can't use > Cisco ACE, for this reason I'm using RoundRobin DNS into this 2 servers. > > This week I had, some network issue with one of this servers and when the > network go down I loose 50% of the traffic... > > here my question, or advice request... How I can have HA with 2 varnish > server? I'm thinking on Heartbeat or Heartbeat+ldirectord, please fell free > to send your suggestions. As your clients do not fall back to the 2nd IP it means you need one (active/passive) or two vips (active/active) that can be shared between your two varnish servers. If you have cachable dynamic content then your two Varnish instances may return different content to the same client. This may or may not be an issue. Others will have to weight in if cache invalidation of Varnish entries is a doable strategy (if you cannot set your expiration headers correctly). On the positive side you have a warm cache should one of your Varnish instances fail. With a active/passive fail over the cold cache may be problematic. That said I am surprised that you think Varnish will be able to handle your traffic but a pair of load balancers cannot. If the Cisco ACE is not up to it have you looked at haproxy? You still have the same vip issues, but load balancers are stateless and give you persistence connections to minimize differences between Varnish instances. /Allan -- Allan Wind Life Integrity, LLC From john at 8t8.eu Wed Dec 7 06:54:19 2011 From: john at 8t8.eu (john at 8t8.eu) Date: Wed, 7 Dec 2011 06:54:19 +0000 Subject: HA with Varnish? advice request In-Reply-To: <20111207053426.GA7422@vent.lifeintegrity.localnet> References: <20111207053426.GA7422@vent.lifeintegrity.localnet> Message-ID: I'm succesfully using lvs in director mode to load balance and provide ha for multiple varnish caches. It works really well. On Dec 7, 2011 5:36 a.m., "Allan Wind" wrote: > On 2011-12-07 00:05:24, Alejandro wrote: > > Hi guys, > > > > I need your advice... I'm using varnish into multiple server, some for > > caching dynamic content and other only for static content. > > > > On front of the Dynamic Content Varnish y have Cisco ACE Load balancer, > but > > on front of the Static Varnish, the traffic is much higher and can't use > > Cisco ACE, for this reason I'm using RoundRobin DNS into this 2 servers. > > > > This week I had, some network issue with one of this servers and when the > > network go down I loose 50% of the traffic... > > > > here my question, or advice request... How I can have HA with 2 varnish > > server? I'm thinking on Heartbeat or Heartbeat+ldirectord, please fell > free > > to send your suggestions. > > As your clients do not fall back to the 2nd IP it means you need > one (active/passive) or two vips (active/active) that can be > shared between your two varnish servers. > > If you have cachable dynamic content then your two Varnish > instances may return different content to the same client. This > may or may not be an issue. Others will have to weight in if > cache invalidation of Varnish entries is a doable strategy > (if you cannot set your expiration headers correctly). On the > positive side you have a warm cache should one of your Varnish > instances fail. With a active/passive fail over the cold cache > may be problematic. > > That said I am surprised that you think Varnish will be able to > handle your traffic but a pair of load balancers cannot. If the > Cisco ACE is not up to it have you looked at haproxy? You still > have the same vip issues, but load balancers are stateless and > give you persistence connections to minimize differences between > Varnish instances. > > > /Allan > -- > Allan Wind > Life Integrity, LLC > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdgraff at gmail.com Wed Dec 7 14:13:06 2011 From: cdgraff at gmail.com (Alejandro) Date: Wed, 7 Dec 2011 11:13:06 -0300 Subject: HA with Varnish? advice request In-Reply-To: References: <20111207053426.GA7422@vent.lifeintegrity.localnet> Message-ID: Thanks guys for the answers... I try to create, Active/Active model... I'll check this soft like wackamole or director... 2011/12/7 john at 8t8.eu > I'm succesfully using lvs in director mode to load balance and provide ha > for multiple varnish caches. It works really well. > On Dec 7, 2011 5:36 a.m., "Allan Wind" > wrote: > >> On 2011-12-07 00:05:24, Alejandro wrote: >> > Hi guys, >> > >> > I need your advice... I'm using varnish into multiple server, some for >> > caching dynamic content and other only for static content. >> > >> > On front of the Dynamic Content Varnish y have Cisco ACE Load balancer, >> but >> > on front of the Static Varnish, the traffic is much higher and can't use >> > Cisco ACE, for this reason I'm using RoundRobin DNS into this 2 servers. >> > >> > This week I had, some network issue with one of this servers and when >> the >> > network go down I loose 50% of the traffic... >> > >> > here my question, or advice request... How I can have HA with 2 varnish >> > server? I'm thinking on Heartbeat or Heartbeat+ldirectord, please fell >> free >> > to send your suggestions. >> >> As your clients do not fall back to the 2nd IP it means you need >> one (active/passive) or two vips (active/active) that can be >> shared between your two varnish servers. >> >> If you have cachable dynamic content then your two Varnish >> instances may return different content to the same client. This >> may or may not be an issue. Others will have to weight in if >> cache invalidation of Varnish entries is a doable strategy >> (if you cannot set your expiration headers correctly). On the >> positive side you have a warm cache should one of your Varnish >> instances fail. With a active/passive fail over the cold cache >> may be problematic. >> >> That said I am surprised that you think Varnish will be able to >> handle your traffic but a pair of load balancers cannot. If the >> Cisco ACE is not up to it have you looked at haproxy? You still >> have the same vip issues, but load balancers are stateless and >> give you persistence connections to minimize differences between >> Varnish instances. >> >> >> /Allan >> -- >> Allan Wind >> Life Integrity, LLC >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan.martinez at masterion.com Wed Dec 7 16:52:35 2011 From: ivan.martinez at masterion.com (Ivan Martinez) Date: Wed, 07 Dec 2011 17:52:35 +0100 Subject: 503 Service Unavailable when using firewall In-Reply-To: References: Message-ID: <874a02e4fa0526a965493cc5741d2a09@mail.masterion.com> Hello all, In a CentOS 6 server, I'm running Varnish in port 80 serving pages from a Zotonic site in port 8000. It works fine when I leave all the ports open. However, if I close all ports from 0 to 631 and explicitly open 80 and 8000, the following happens: - I can see the website in server:8000. - In server:80, I get the following after some seconds: Error 503 Service Unavailable Service Unavailable Guru Meditation: XID: 544990083 Varnish cache server I have read about the sysctl issues with firewalls but it seems I have it right: [root at server ivanmr]# sysctl -A | grep tw net.ipv4.tcp_max_tw_buckets = 2048 net.ipv4.tcp_tw_recycle = 0 net.ipv4.tcp_tw_reuse = 0 varnish.log is empty. What can be the problem?. Thank you. Ivan From hugo.cisneiros at gmail.com Wed Dec 7 17:04:48 2011 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Wed, 7 Dec 2011 15:04:48 -0200 Subject: 503 Service Unavailable when using firewall In-Reply-To: <874a02e4fa0526a965493cc5741d2a09@mail.masterion.com> References: <874a02e4fa0526a965493cc5741d2a09@mail.masterion.com> Message-ID: On Wed, Dec 7, 2011 at 2:52 PM, Ivan Martinez wrote: > In a CentOS 6 server, I'm running Varnish in port 80 serving pages from a > Zotonic site in port 8000. It works fine when I leave all the ports open. > However, if I close all ports from 0 to 631 and explicitly open 80 and > 8000, the following happens: > > - I can see the website in server:8000. > - In server:80, I get the following after some seconds: > > Error 503 Service Unavailable > > Service Unavailable > Guru Meditation: > > XID: 544990083 > > Varnish cache server > Error 503 happens when Varnish could not contact any backends. You can check if a backend is up and healthy using the command: varnishadm -T localhost: debug.health The backend must be healthy. If it's sick, it means that varnish can't connect on port server:8000 on your site for some reason (nc/telnet/wget/curl can be useful to test here). Check if the firewall is blocking the output, and it's accepting established/related connections on the backend. -- []'s Hugo www.devin.com.br -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.fernandezcrisial at gmail.com Wed Dec 7 17:43:50 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Wed, 7 Dec 2011 14:43:50 -0300 Subject: 503 Service Unavailable when using firewall In-Reply-To: <874a02e4fa0526a965493cc5741d2a09@mail.masterion.com> References: <874a02e4fa0526a965493cc5741d2a09@mail.masterion.com> Message-ID: Ivan, Have you tried telnet from Varnish server to Zotonic server? $ telnet zotonic.address 8000 Check if there is any response. -- Roberto O. Fern?ndez Crisial @rofc On Wed, Dec 7, 2011 at 1:52 PM, Ivan Martinez wrote: > Hello all, > In a CentOS 6 server, I'm running Varnish in port 80 serving pages from a > Zotonic site in port 8000. It works fine when I leave all the ports open. > However, if I close all ports from 0 to 631 and explicitly open 80 and > 8000, the following happens: > > - I can see the website in server:8000. > - In server:80, I get the following after some seconds: > > Error 503 Service Unavailable > > Service Unavailable > Guru Meditation: > > XID: 544990083 > > Varnish cache server > > I have read about the sysctl issues with firewalls but it seems I have it > right: > > [root at server ivanmr]# sysctl -A | grep tw > net.ipv4.tcp_max_tw_buckets = 2048 > net.ipv4.tcp_tw_recycle = 0 > net.ipv4.tcp_tw_reuse = 0 > > varnish.log is empty. What can be the problem?. Thank you. > > Ivan > > ______________________________**_________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ivan.martinez at masterion.com Wed Dec 7 17:53:26 2011 From: ivan.martinez at masterion.com (Ivan Martinez) Date: Wed, 07 Dec 2011 18:53:26 +0100 Subject: 503 Service Unavailable when using firewall In-Reply-To: References: <874a02e4fa0526a965493cc5741d2a09@mail.masterion.com> Message-ID: <9508827cac559bb85b66142b864b2d59@mail.masterion.com> Thank you Roberto, There is only one server running both services, and if I can browse http://server:8000 from my local machine, I suppose the telnet works as well. Ivan On Wed, 7 Dec 2011 14:43:50 -0300, Roberto O. Fern?ndez Crisial wrote: > Ivan, > Have you tried telnet from Varnish server to Zotonic server? > $ telnet zotonic.address 8000 > Check if there is any response. > -- > Roberto O. Fern?ndez Crisial > @rofc > > On Wed, Dec 7, 2011 at 1:52 PM, Ivan Martinez wrote: > >> Hello all, >> In a CentOS 6 server, I'm running Varnish in port 80 serving pages from a Zotonic site in port 8000. It works fine when I leave all the ports open. However, if I close all ports from 0 to 631 and explicitly open 80 and 8000, the following happens: >> >> - I can see the website in server:8000. >> - In server:80, I get the following after some seconds: >> >> Error 503 Service Unavailable >> >> Service Unavailable >> Guru Meditation: >> >> XID: 544990083 >> >> Varnish cache server >> >> I have read about the sysctl issues with firewalls but it seems I have it right: >> >> [root at server ivanmr]# sysctl -A | grep tw >> net.ipv4.tcp_max_tw_buckets = 2048 >> net.ipv4.tcp_tw_recycle = 0 >> net.ipv4.tcp_tw_reuse = 0 >> >> varnish.log is empty. What can be the problem?. Thank you. >> >> Ivan >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org [1] >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc [2] Links: ------ [1] mailto:varnish-misc at varnish-cache.org [2] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc [3] mailto:ivan.martinez at masterion.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.fernandezcrisial at gmail.com Wed Dec 7 18:15:35 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Wed, 7 Dec 2011 15:15:35 -0300 Subject: 503 Service Unavailable when using firewall In-Reply-To: <9508827cac559bb85b66142b864b2d59@mail.masterion.com> References: <874a02e4fa0526a965493cc5741d2a09@mail.masterion.com> <9508827cac559bb85b66142b864b2d59@mail.masterion.com> Message-ID: Ivan, Try to disable SELinux (enforced by default), and re-check your iptables rules. Regards, Roberto. 2011/12/7 Ivan Martinez > ** > > Thank you Roberto, > > There is only one server running both services, and if I can browse > http://server:8000 from my local machine, I suppose the telnet works as > well. > > Ivan > > On Wed, 7 Dec 2011 14:43:50 -0300, Roberto O. Fern?ndez Crisial wrote: > > Ivan, > Have you tried telnet from Varnish server to Zotonic server? > $ telnet zotonic.address 8000 > Check if there is any response. > -- > Roberto O. Fern?ndez Crisial > @rofc > > On Wed, Dec 7, 2011 at 1:52 PM, Ivan Martinez > wrote: > >> Hello all, >> In a CentOS 6 server, I'm running Varnish in port 80 serving pages from a >> Zotonic site in port 8000. It works fine when I leave all the ports open. >> However, if I close all ports from 0 to 631 and explicitly open 80 and >> 8000, the following happens: >> >> - I can see the website in server:8000. >> - In server:80, I get the following after some seconds: >> >> Error 503 Service Unavailable >> >> Service Unavailable >> Guru Meditation: >> >> XID: 544990083 >> >> Varnish cache server >> >> I have read about the sysctl issues with firewalls but it seems I have it >> right: >> >> [root at server ivanmr]# sysctl -A | grep tw >> net.ipv4.tcp_max_tw_buckets = 2048 >> net.ipv4.tcp_tw_recycle = 0 >> net.ipv4.tcp_tw_reuse = 0 >> >> varnish.log is empty. What can be the problem?. Thank you. >> >> Ivan >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.birdsong at gmail.com Wed Dec 7 18:49:20 2011 From: david.birdsong at gmail.com (David Birdsong) Date: Wed, 7 Dec 2011 10:49:20 -0800 Subject: 503 Service Unavailable when using firewall In-Reply-To: <9508827cac559bb85b66142b864b2d59@mail.masterion.com> References: <874a02e4fa0526a965493cc5741d2a09@mail.masterion.com> <9508827cac559bb85b66142b864b2d59@mail.masterion.com> Message-ID: 2011/12/7 Ivan Martinez : > Thank you Roberto, > > There is only one server running both services, and if I can browse > http://server:8000 from my local machine, I suppose the telnet works as > well. Right, but you should still try to telnet *from* the varnish machine to the backend despite the fact that your local machine can reach the backend. > > Ivan > > On Wed, 7 Dec 2011 14:43:50 -0300, Roberto O. Fern?ndez Crisial wrote: > > Ivan, > > Have you tried telnet from Varnish server to Zotonic server? > $ telnet zotonic.address 8000 > Check if there is any response. > -- > Roberto O. Fern?ndez Crisial > @rofc > > On Wed, Dec 7, 2011 at 1:52 PM, Ivan Martinez > wrote: >> >> Hello all, >> In a CentOS 6 server, I'm running Varnish in port 80 serving pages from a >> Zotonic site in port 8000. It works fine when I leave all the ports open. >> However, if I close all ports from 0 to 631 and explicitly open 80 and 8000, >> the following happens: >> >> - I can see the website in server:8000. >> - In server:80, I get the following after some seconds: >> >> Error 503 Service Unavailable >> >> Service Unavailable >> Guru Meditation: >> >> XID: 544990083 >> >> Varnish cache server >> >> I have read about the sysctl issues with firewalls but it seems I have it >> right: >> >> [root at server ivanmr]# sysctl -A | grep tw >> net.ipv4.tcp_max_tw_buckets = 2048 >> net.ipv4.tcp_tw_recycle = 0 >> net.ipv4.tcp_tw_reuse = 0 >> >> varnish.log is empty. What can be the problem?. Thank you. >> >> Ivan >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From ivan.martinez at masterion.com Wed Dec 7 18:54:50 2011 From: ivan.martinez at masterion.com (Ivan Martinez) Date: Wed, 07 Dec 2011 19:54:50 +0100 Subject: varnish-misc Digest, Vol 69, Issue 7 In-Reply-To: References: Message-ID: Thank you Hugo, looks like I'm blocking internal traffic: [root at server ivanmr]# varnishadm -T 127.0.0.1:6082 debug.health Connection failed (127.0.0.1:6082) [root at server ivanmr]# nmap 127.0.0.1 -p 8000 Starting Nmap 5.21 ( http://nmap.org ) at 2011-12-07 18:52 UTC sendto in send_ip_packet: sendto(4, packet, 44, 0, 127.0.0.1, 16) => Operation not permitted Offending packet: TCP 127.0.0.1:43982 > 127.0.0.1:8000 S ttl=52 id=29152 iplen=44 seq=521959048 win=1024 sendto in send_ip_packet: sendto(4, packet, 44, 0, 127.0.0.1, 16) => Operation not permitted Offending packet: TCP 127.0.0.1:43983 > 127.0.0.1:8000 S ttl=48 id=55619 iplen=44 seq=522024585 win=1024 Nmap scan report for localhost (127.0.0.1) Host is up. PORT STATE SERVICE 8000/tcp filtered http-alt Nmap done: 1 IP address (1 host up) scanned in 2.07 seconds I will review my firewall configuration again... :-( Ivan > Message: 3 > Date: Wed, 7 Dec 2011 15:04:48 -0200 > From: "Hugo Cisneiros (Eitch)" > To: varnish-misc at varnish-cache.org > Subject: Re: 503 Service Unavailable when using firewall > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > > Error 503 happens when Varnish could not contact any backends. You > can > check if a backend is up and healthy using the command: > > varnishadm -T localhost: debug.health > > The backend must be healthy. If it's sick, it means that varnish > can't > connect on port server:8000 on your site for some reason > (nc/telnet/wget/curl can be useful to test here). Check if the > firewall is > blocking the output, and it's accepting established/related > connections on > the backend. > > -- > []'s > Hugo > www.devin.com.br From christian at moc.net Thu Dec 8 09:37:49 2011 From: christian at moc.net (Christian Jul Jensen) Date: Thu, 08 Dec 2011 10:37:49 +0100 Subject: Maitaining state in ESI includes Message-ID: <4EE0856D.4060107@moc.net> Hi I asked about this previously, but maybe I didn't explain my problem well enough, at least I am getting some different answers on whether the following is possible. Scenario: We want to embed a stand-alone application into pages generated by a CMS. The CMS pages are fully cacheable. The embedded application has a mix of cacheable and uncacheable states defined by GET-parameters. Which state should be shown is not known by the CMS, so proper GET-parameters can not be provided in the ESI-tag, but will be links back to the page from the embedded application itself. Questions: Is it possible to forward GET (and POST) parameters from the main request to the ESI request? If so, is it possible to filter the forwarded parameters by some regexp, to only provide the relevant parameters. thanks. -- Christian Jul Jensen MOC - http://moc.net @christianjul From ivan.martinez at masterion.com Thu Dec 8 11:12:34 2011 From: ivan.martinez at masterion.com (Ivan Martinez) Date: Thu, 08 Dec 2011 12:12:34 +0100 Subject: 503 Service Unavailable when using firewall In-Reply-To: References: Message-ID: <22d58bb8c056fc3d0670e91215013d9f@mail.masterion.com> Yes, I'm using Firewall Builder and had "lo" configured as "Unnumbered interface". Changed to "Static IP address" and now everything works fine. "Unnumbered" worked in another server, I don't know why... Thank you everybody and sorry for wasting your time with silly mistakes. Ivan > Thank you Hugo, looks like I'm blocking internal traffic: > > [root at server ivanmr]# varnishadm -T 127.0.0.1:6082 debug.health > Connection failed (127.0.0.1:6082) > [root at server ivanmr]# nmap 127.0.0.1 -p 8000 > > Starting Nmap 5.21 ( http://nmap.org ) at 2011-12-07 18:52 UTC > sendto in send_ip_packet: sendto(4, packet, 44, 0, 127.0.0.1, 16) => > Operation not permitted > Offending packet: TCP 127.0.0.1:43982 > 127.0.0.1:8000 S ttl=52 > id=29152 iplen=44 seq=521959048 win=1024 > sendto in send_ip_packet: sendto(4, packet, 44, 0, 127.0.0.1, 16) => > Operation not permitted > Offending packet: TCP 127.0.0.1:43983 > 127.0.0.1:8000 S ttl=48 > id=55619 iplen=44 seq=522024585 win=1024 > Nmap scan report for localhost (127.0.0.1) > Host is up. > PORT STATE SERVICE > 8000/tcp filtered http-alt > > Nmap done: 1 IP address (1 host up) scanned in 2.07 seconds > > I will review my firewall configuration again... :-( > Ivan From jacquiebell4 at yahoo.com Sat Dec 3 15:25:22 2011 From: jacquiebell4 at yahoo.com (Jacquie Bulla) Date: Sat, 3 Dec 2011 09:25:22 -0600 Subject: Varnish cache? Message-ID: <26F7B01B-8A8A-4F43-BFAA-410EC9B054D5@yahoo.com> Hello. I recently upgraded my iPhone to 5.0. I have been trying to get to my forum on ford-trucks.com like I always did before, but now I only get this: Error 503 File is Temporarily UnavailableFile is Temporarily UnavailableGuru Meditation: XID: 2037754862 Then there is a link to the varnish website, which I have no clue about. Any ideas? I even called Apple support, and they just told me I shouldn't have this problem. Any suggestions are appreciated. Thank you. Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Thu Dec 8 12:36:37 2011 From: perbu at varnish-software.com (Per Buer) Date: Thu, 8 Dec 2011 13:36:37 +0100 Subject: Varnish cache? In-Reply-To: <26F7B01B-8A8A-4F43-BFAA-410EC9B054D5@yahoo.com> References: <26F7B01B-8A8A-4F43-BFAA-410EC9B054D5@yahoo.com> Message-ID: Hi Jacquie. The reason you are seeing this is because there is an error on the ford-trucks.com website. It has nothing to do with your phone and there is nothing you can do about it. The people running the ford-trucks.com website need to fix the problem and if you give them time they probably will. So you can just relax and maybe browse the GM trucks website instead. ;-) Regards, Per. On Sat, Dec 3, 2011 at 4:25 PM, Jacquie Bulla wrote: > Hello. I recently upgraded my iPhone to 5.0. I have been trying to get to > my forum on ford-trucks.com like I always did before, but now I only get > this: Error 503 File is Temporarily UnavailableFile is Temporarily > UnavailableGuru Meditation: XID: 2037754862 > Then there is a link to the varnish website, which I have no clue about. > Any ideas? I even called Apple support, and they just told me I shouldn't > have this problem. Any suggestions are appreciated. Thank you. > Sent from my iPhone > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Thu Dec 8 13:41:17 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 08 Dec 2011 14:41:17 +0100 Subject: Tune Varnish for pipe only operations In-Reply-To: (Brain Stormer's message of "Tue, 6 Dec 2011 05:11:02 +0100") References: Message-ID: <87pqfzmb9e.fsf@qurzaw.varnish-software.com> ]] Brain Stormer > I am currently facing CPU overhead at the `varnishd` process, I have tuned > lots of parts in Varnish, So... What have you tuned and how? What's your varnishstat numbers? > Another question is, Do I still have to define a caching store ? I > read on some online resources that caching store is still needed for > storage regarding Varnish threads. Yes, you need to define some storage. It won't be used, if you're just piping, though, so can be tiny. -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From jason at pethub.com Fri Dec 9 07:08:50 2011 From: jason at pethub.com (Jason Farnsworth) Date: Fri, 9 Dec 2011 07:08:50 +0000 Subject: Rewriting/enforcing SSL behing an SSL termination point Message-ID: Hi everyone, new Varnish user and new to the list here, but I've quickly become a strong advocate of using Varnish in our organization based on my initial experiences with it. However, I'm having a difficult time finding information on what I think would be a fairly common problem. We want to enforce the use of www. (which we currently do with mod_rewrite) AND enforce the use of HTTPS - site wide. I'm aware that we can do both of these with mod_rewrite, but there are various reasons I'd like to keep this at the Varnish level? We are hosted on Amazon Web Services and all SSL termination is done by an Elastic Load Balancer. So all I'm looking to do is re-write URLs like this? http://domain.com -> https://www.domain.com http://www.domain.com -> https://www.domain.com https://domain.com -> https://www.domain.com Of course, we also want to pass on the x-forwarded-proto header (which is pretty well documented, no problem there.) The URL re-write directions on the website address rewriting the host and the path well, but not SSL. It would make me quite happy if we could use Varnish for all this. Regards, Jason Farnsworth From varnish at mm.quex.org Fri Dec 9 07:45:13 2011 From: varnish at mm.quex.org (Michael Alger) Date: Fri, 9 Dec 2011 15:45:13 +0800 Subject: Rewriting/enforcing SSL behing an SSL termination point In-Reply-To: References: Message-ID: <20111209074513.GA10322@grum.quex.org> On Fri, Dec 09, 2011 at 07:08:50AM +0000, Jason Farnsworth wrote: > I'm having a difficult time finding information on what I think > would be a fairly common problem. We want to enforce the use of > www. (which we currently do with mod_rewrite) AND enforce the use > of HTTPS - site wide. I'm aware that we can do both of these with > mod_rewrite, but there are various reasons I'd like to keep this at > the Varnish level? Since Varnish doesn't handle HTTPS there's a sort of level of abstracton here. You need some other indicator as to whether the connection was made via SSL, which it sounds like you have: > We are hosted on Amazon Web Services and all SSL termination is done > by an Elastic Load Balancer. So all I'm looking to do is re-write > URLs like this? > > http://domain.com -> https://www.domain.com > http://www.domain.com -> https://www.domain.com > https://domain.com -> https://www.domain.com > > Of course, we also want to pass on the x-forwarded-proto header > (which is pretty well documented, no problem there.) I take it the X-Forwarded-Proto header a) indicates whether the connection was made via SSL or not and b) can be trusted? If so then you're set. If not you'll need to find something in the request you can use for this purpose. > The URL re-write directions on the website address rewriting the > host and the path well, but not SSL. It would make me quite happy > if we could use Varnish for all this. Once you have a mechanism for determining how the original connection was made, it's pretty straightforward. In vcl_recv you just check for that header e.g. if (req.http.X-Forwarded-Proto ~ "SSL") or whatever, and if it wasn't SSL then you issue a redirect to the same URL with an https:// prefix in the usual manner. From joshua.strebel at gmail.com Fri Dec 9 08:24:26 2011 From: joshua.strebel at gmail.com (joshua strebel) Date: Fri, 9 Dec 2011 01:24:26 -0700 Subject: .vcl rule to pass no content Message-ID: <08BFD9F5-6800-4C62-B1B7-5ED69580101C@gmail.com> Looking for the best way to not cache content that is an empty page. From time to time a page will return blank, not even an html tag. I need to make sure we pass on these. Is there .vcl for checking body size? We are using Varnish 3+ -- Joshua Strebel Founder @ Page.ly twitter: @strebel From perbu at varnish-software.com Fri Dec 9 08:48:48 2011 From: perbu at varnish-software.com (Per Buer) Date: Fri, 9 Dec 2011 09:48:48 +0100 Subject: Rewriting/enforcing SSL behing an SSL termination point In-Reply-To: References: Message-ID: On Fri, Dec 9, 2011 at 8:08 AM, Jason Farnsworth wrote: > We are hosted on Amazon Web Services and all SSL termination is done by an > Elastic Load Balancer. So all I'm looking to do is re-write URLs like > this > > http://domain.com -> https://www.domain.com > http://www.domain.com -> https://www.domain.com > https://domain.com -> https://www.domain.com Varnish will not rewrite the actual content coming from the backend. We can however, _redirect_ the client whenever they ask for a http:// URL. We use the following code on varnish-cache.org to do this: in vcl_recv: if ( (req.http.host ~ "(?i)www.varnish-cache.org") && !(client.ip ~ localhost)) { set req.http.x-redir-url = "https://" + req.http.host + req.url; error 750 req.http.x-redir-url; } (..) sub vcl_error { # standard redirection in VCL: if (obj.status == 750) { set obj.http.Location = obj.response; set obj.status = 302; return(deliver); } } Since we have an SSL terminator in front of Varnish client.ip is localhost when there is SSL present. You might want to change the code to test X-Forwarded-Proto for whatever it is set to. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlane at ahbelo.com Fri Dec 9 17:25:58 2011 From: rlane at ahbelo.com (Lane, Richard) Date: Fri, 9 Dec 2011 17:25:58 +0000 Subject: .vcl rule to pass no content In-Reply-To: <<08BFD9F5-6800-4C62-B1B7-5ED69580101C@gmail.com>> Message-ID: The should work for you if you get a content type and length from backend. This works on varnish 2.15 I placed this is my fetch block. # Don't cache html objects smaller than 100 bytes if ((beresp.status == 200) && (beresp.http.Content-Length ~ "^([0-2]|\d)\d") && ( beresp.http.Content-type ~ "html") ) { log "TooSmall: Pass on ( " req.url " ) small objects: " beresp.http.Content-Length ; return(pass); } Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlane at ahbelo.com Fri Dec 9 19:06:55 2011 From: rlane at ahbelo.com (Lane, Richard) Date: Fri, 9 Dec 2011 19:06:55 +0000 Subject: .vcl rule to pass no content In-Reply-To: Message-ID: Joshua, Actually the code I sent looks for a length that is 299 bytes or smaller. So if you wanted to look for less you would need to change up the regex on Content-Length to something like If less than 10 bytes - ?^[0-9]? If less than 30 bytes - ?^[0-2]\d? However this may not work for you either because the Content-Length includes headers also. I will give your question a little more thought. Cheers, Richard On 12/9/11 11:25 AM, "Lane, Richard" wrote: The should work for you if you get a content type and length from backend. This works on varnish 2.15 I placed this is my fetch block. # Don't cache html objects smaller than 100 bytes if ((beresp.status == 200) && (beresp.http.Content-Length ~ "^([0-2]|\d)\d") && ( beresp.http.Content-type ~ "html") ) { log "TooSmall: Pass on ( " req.url " ) small objects: " beresp.http.Content-Length ; return(pass); } Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdgraff at gmail.com Sat Dec 10 04:03:55 2011 From: cdgraff at gmail.com (Alejandro) Date: Sat, 10 Dec 2011 01:03:55 -0300 Subject: Director with Sticky session Message-ID: Hi guys, I would like to know if currently the director support Sticky session? I read here but I don't found nothing related: https://www.varnish-cache.org/docs/3.0/tutorial/advanced_backend_servers.html?highlight=director thanks, Alejandro From ruben at varnish-software.com Sat Dec 10 10:38:11 2011 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Sat, 10 Dec 2011 11:38:11 +0100 Subject: Director with Sticky session In-Reply-To: References: Message-ID: Hi Alejandro, As of today there is no support for that, although I know the idea has been in the air for a very long time. See this thread: https://www.varnish-cache.org/lists/pipermail/varnish-misc/2010-May/004184.html It has also been a bug, shopping list item and it lives now here: https://www.varnish-cache.org/trac/wiki/Future_VCL#Stickydirector Kristian, Martin, Lasse and I discussed this yesterday afternoon, so it is a very relevant subject... But I cannot tell you anything about it's priority as I have no idea about that. What I know is that there are other relevant features/changes with higher priority. I am pretty sure that someone in the list can give you details on that (if there are any). Best, - Rub?n Romero Varnish Software On 12/10/11, Alejandro wrote: > Hi guys, > > I would like to know if currently the director support Sticky session? > I read here but I don't found nothing related: > https://www.varnish-cache.org/docs/3.0/tutorial/advanced_backend_servers.html?highlight=director > > thanks, > Alejandro > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Rub?n Romero, Global Sales Executive Phone: +47 21 98 92 62 / Mobile: +47 959 64 088 / Skype: ruben_varnish / GTalk: *Varnish makes websites fly!* Whitepapers | Video | Twitter | LinkedIn From perbu at varnish-software.com Sat Dec 10 13:51:16 2011 From: perbu at varnish-software.com (Per Buer) Date: Sat, 10 Dec 2011 14:51:16 +0100 Subject: Director with Sticky session In-Reply-To: References: Message-ID: Hi. On Sat, Dec 10, 2011 at 5:03 AM, Alejandro wrote: > Hi guys, > > I would like to know if currently the director support Sticky session? > Sort of. The client director can do load balancing on the basis some input, maybe a session cookie. See "man VCL", search for "client director". The problem with however is that the server setting the initial cookie might not get the request back if you just use the default hashing. If you can live with that, or better, you're able to set the cookie in Varnish itself and then restart the transaction you'll be fine. So, as you see see, all the pieces are here. There is just a bit of customer assembly required. :-) Or you can just use the client.ip as input to the client director. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdgraff at gmail.com Sun Dec 11 05:52:36 2011 From: cdgraff at gmail.com (Alejandro) Date: Sun, 11 Dec 2011 02:52:36 -0300 Subject: Unknown Error with INFLATE Message-ID: Hi Guys, Need your help for understand my issue... I have 1 varnish 3.0.2 and 2 backends apache/php 5.3.8, the health-check work perfect but I get this error 0 Debug - "Unknown INFLATE=-3 (incorrect data check)%0a" Here the logs lines: 0 Backend_health - www[1] Still healthy 4--X-RH 10 8 10 0.000495 0.000291 HTTP/1.1 200 OK 0 Backend_health - www[0] Still healthy 4--X-RH 10 8 10 0.001459 0.000843 HTTP/1.1 200 OK 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1323582308 1.0 0 Debug - "Unknown INFLATE=-3 (incorrect data check)%0a" 4 BackendOpen b www[0] 128.204.194.99 34700 128.204.194.97 81 4 TxRequest b GET 4 TxURL b / 4 TxProtocol b HTTP/1.1 4 TxHeader b Host: web2 4 TxHeader b X-Forwarded-For: 201.231.169.XXX 4 TxHeader b X-UserAgent: Default 4 TxHeader b User-Agent: Default 4 TxHeader b X-Grace: 6m 4 TxHeader b Cookie: CLIPSHARE=094162b7900bd00bfe9d7b07758401f2 4 TxHeader b X-Varnish: 452193219 4 TxHeader b Accept-Encoding: gzip 4 RxProtocol b HTTP/1.1 4 RxStatus b 200 4 RxResponse b OK 4 RxHeader b Date: Sun, 11 Dec 2011 05:45:09 GMT 4 RxHeader b Server: Apache/2.2.3 (CentOS) 4 RxHeader b X-Powered-By: PHP/5.3.8 4 RxHeader b Expires: Thu, 19 Nov 1981 08:52:00 GMT 4 RxHeader b Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 4 RxHeader b Pragma: no-cache 4 RxHeader b Content-Encoding: gzip 4 RxHeader b Connection: close 4 RxHeader b Transfer-Encoding: chunked 4 RxHeader b Content-Type: text/html; charset=UTF-8 4 Fetch_Body b 3(chunked) cls -1 mklen 1 4 BackendClose b www[0] 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1323582311 1.0 0 Debug - "Unknown INFLATE=-3 (incorrect data check)%0a" Please fell free to request more info I don't know what more I can send. Regards, Alejandro -------------- next part -------------- An HTML attachment was scrubbed... URL: From cdgraff at gmail.com Sun Dec 11 06:12:39 2011 From: cdgraff at gmail.com (Alejandro) Date: Sun, 11 Dec 2011 03:12:39 -0300 Subject: Unknown Error with INFLATE In-Reply-To: References: Message-ID: Guys, I found the issue... Into my application "CLIPSHARE" have one option for "Gzip Encoding", this option enabled, break the connection with varnish. Will be great now why, but for the moment I can continue with the varnish installation... If anyone now way this break the varnish, please tell me. Thanks, Alejandro 2011/12/11 Alejandro : > Hi Guys, > > Need your help for understand my issue... > > I have 1 varnish 3.0.2 and 2 backends apache/php 5.3.8, the health-check > work perfect but I get this error > > ? ? 0 Debug ? ? ? ?- "Unknown INFLATE=-3 (incorrect data check)%0a" > > Here the logs lines: > > ?0 Backend_health - www[1] Still healthy 4--X-RH 10 8 10 0.000495 0.000291 > HTTP/1.1 200 OK > ? ? 0 Backend_health - www[0] Still healthy 4--X-RH 10 8 10 0.001459 > 0.000843 HTTP/1.1 200 OK > ? ? 0 CLI ? ? ? ? ?- Rd ping > ? ? 0 CLI ? ? ? ? ?- Wr 200 19 PONG 1323582308 1.0 > ? ? 0 Debug ? ? ? ?- "Unknown INFLATE=-3 (incorrect data check)%0a" > ? ? 4 BackendOpen ?b www[0] 128.204.194.99 34700 128.204.194.97 81 > ? ? 4 TxRequest ? ?b GET > ? ? 4 TxURL ? ? ? ?b / > ? ? 4 TxProtocol ? b HTTP/1.1 > ? ? 4 TxHeader ? ? b Host: web2 > ? ? 4 TxHeader ? ? b X-Forwarded-For: 201.231.169.XXX > ? ? 4 TxHeader ? ? b X-UserAgent: Default > ? ? 4 TxHeader ? ? b User-Agent: Default > ? ? 4 TxHeader ? ? b X-Grace: 6m > ? ? 4 TxHeader ? ? b Cookie: CLIPSHARE=094162b7900bd00bfe9d7b07758401f2 > ? ? 4 TxHeader ? ? b X-Varnish: 452193219 > ? ? 4 TxHeader ? ? b Accept-Encoding: gzip > ? ? 4 RxProtocol ? b HTTP/1.1 > ? ? 4 RxStatus ? ? b 200 > ? ? 4 RxResponse ? b OK > ? ? 4 RxHeader ? ? b Date: Sun, 11 Dec 2011 05:45:09 GMT > ? ? 4 RxHeader ? ? b Server: Apache/2.2.3 (CentOS) > ? ? 4 RxHeader ? ? b X-Powered-By: PHP/5.3.8 > ? ? 4 RxHeader ? ? b Expires: Thu, 19 Nov 1981 08:52:00 GMT > ? ? 4 RxHeader ? ? b Cache-Control: no-store, no-cache, must-revalidate, > post-check=0, pre-check=0 > ? ? 4 RxHeader ? ? b Pragma: no-cache > ? ? 4 RxHeader ? ? b Content-Encoding: gzip > ? ? 4 RxHeader ? ? b Connection: close > ? ? 4 RxHeader ? ? b Transfer-Encoding: chunked > ? ? 4 RxHeader ? ? b Content-Type: text/html; charset=UTF-8 > ? ? 4 Fetch_Body ? b 3(chunked) cls -1 mklen 1 > ? ? 4 BackendClose b www[0] > ? ? 0 CLI ? ? ? ? ?- Rd ping > ? ? 0 CLI ? ? ? ? ?- Wr 200 19 PONG 1323582311 1.0 > ? ? 0 Debug ? ? ? ?- "Unknown INFLATE=-3 (incorrect data check)%0a" > > Please fell free to request more info I don't know what more I can send. > > Regards, > Alejandro From arjen at thenextweb.com Sun Dec 11 18:59:40 2011 From: arjen at thenextweb.com (Arjen Schat) Date: Sun, 11 Dec 2011 19:59:40 +0100 Subject: send reason along with backend request Message-ID: <2B3A03D6-75C2-4E5D-B4B5-3959911DACA8@thenextweb.com> HI All, Is there a way to send an extra header to the backend request if a miss occurs? There are 4 scenario's I can think off: purged, old, not set and lru. This would make debugging and stats a lot easier. Thanks, Arjen From straightflush at gmail.com Mon Dec 12 01:47:51 2011 From: straightflush at gmail.com (AD) Date: Sun, 11 Dec 2011 20:47:51 -0500 Subject: send reason along with backend request In-Reply-To: <2B3A03D6-75C2-4E5D-B4B5-3959911DACA8@thenextweb.com> References: <2B3A03D6-75C2-4E5D-B4B5-3959911DACA8@thenextweb.com> Message-ID: you should be able to add a req.http.header in your vcl_fetch. I dont believe vcl_fetch gets called unless you are in a cache_miss and you need to contact the backend. sub vcl_fetch { req.http.X-Custom-Header = "test"; } Not sure varnish exposes the miss "reason" though, aside from the fact that it is either not cacheable or it is and just isn't in the cache. On Sun, Dec 11, 2011 at 1:59 PM, Arjen Schat wrote: > HI All, > > Is there a way to send an extra header to the backend request if a miss > occurs? > There are 4 scenario's I can think off: purged, old, not set and lru. This > would make debugging and stats a lot easier. > > > Thanks, > Arjen > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrot at carrotis.com Tue Dec 13 01:11:50 2011 From: carrot at carrotis.com (Calvin Park) Date: Tue, 13 Dec 2011 10:11:50 +0900 Subject: response time in varnishncsa Message-ID: Hello ! I'm using varnish 3.0.x Are there any plans for introducing response time in the varnishncsa log format? I mean response time as in '%D' from the apache2 log format. I would really like to see the time-taken field in the varnishnca log From hugo.cisneiros at gmail.com Tue Dec 13 14:35:46 2011 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Tue, 13 Dec 2011 12:35:46 -0200 Subject: Hostnames in backend definition... Message-ID: Hi, I'm setting up varnish to access a load balancer from amazon ec2. I have two problems: - since the IP address for the load balancer changes very often, does varnish obey the ttl rule for resolving the hostname if I specify the hostname in vcl? - i can use multiple availability zones, meaning the hostname will resolve to 4 IPs, one for each zone. I got an error saying that varnish couldn't handle multiple IP addresses on backend definition, so I chose only one IP and it works :) is there a way to use multiple ips? Thanks! -- []'s Hugo www.devin.com.br From roberto.fernandezcrisial at gmail.com Tue Dec 13 14:43:54 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Tue, 13 Dec 2011 11:43:54 -0300 Subject: Hostnames in backend definition... In-Reply-To: References: Message-ID: Hugo, Have you tried Elastic IP? I think this resolves the first problem you have. Best, Roberto O. Fern?ndez Crisial @rofc On Tue, Dec 13, 2011 at 11:35 AM, Hugo Cisneiros (Eitch) < hugo.cisneiros at gmail.com> wrote: > Hi, > > I'm setting up varnish to access a load balancer from amazon ec2. I > have two problems: > > - since the IP address for the load balancer changes very often, does > varnish obey the ttl rule for resolving the hostname if I specify the > hostname in vcl? > > - i can use multiple availability zones, meaning the hostname will > resolve to 4 IPs, one for each zone. I got an error saying that > varnish couldn't handle multiple IP addresses on backend definition, > so I chose only one IP and it works :) is there a way to use multiple > ips? > > Thanks! > > -- > []'s > Hugo > www.devin.com.br > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugo.cisneiros at gmail.com Tue Dec 13 16:01:27 2011 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Tue, 13 Dec 2011 14:01:27 -0200 Subject: Hostnames in backend definition... In-Reply-To: References: Message-ID: 2011/12/13 Roberto O. Fern?ndez Crisial : > Have you tried Elastic IP? I think this resolves the first problem you have. I think Elastic IPs can be associated only with instances. I'm using Elastic Load Balancer (ELB) that uses Auto-Scaling: when it needs, it automatically spawns new servers and they get into this "dynamic" ELB. The IP for this Load Balancer changes very often and Amazon says that you must always use CNAME to their hostname (because of these changes). In other words, I think I can't use a fixed IP address on Varnish backend configuration. :( Thanks -- []'s Hugo Cisneiros (Eitch) http://www.devin.com.br From drais at icantclick.org Tue Dec 13 17:22:06 2011 From: drais at icantclick.org (david raistrick) Date: Tue, 13 Dec 2011 12:22:06 -0500 (EST) Subject: Hostnames in backend definition... In-Reply-To: References: Message-ID: On Tue, 13 Dec 2011, Hugo Cisneiros (Eitch) wrote: > I'm setting up varnish to access a load balancer from amazon ec2. I > have two problems: For those of us familiar with EC2 architecture, can I get some clarification? You're trying to use varnish in front of ELB, with the ELB hostnames as your backends? I would strongly recommend against this. > - since the IP address for the load balancer changes very often, does > varnish obey the ttl rule for resolving the hostname if I specify the > hostname in vcl? ...for exactly this reason. You have no control, knowledge, or insight into when the IP address behind the ELB hostname changes, and varnish effectively caches the hostname on startup. When the internal IP of ELB changes (and it will!) your traffic will stop. > - i can use multiple availability zones, meaning the hostname will > resolve to 4 IPs, one for each zone. I got an error saying that > varnish couldn't handle multiple IP addresses on backend definition, > so I chose only one IP and it works :) is there a way to use multiple > ips? You have to define each AZ as a seperate backend in this case (using hostnames or IP addresses), but you have the same problems as above. I've sort of been threatening to get one of my team to work up a patch to varnish to do some DNS sanity, but until it actually becomes a major blocker for us (we have other workarounds in place - like not using ELB and using internal dynamic config re-generators) I can't get the resource to do it... I personally feel that 99% of the software proxy/loadbalancing solutions are in sorry shape when it comes to dealing with an environment like AWS/EC2 where dynamic dhcp and dynamic dns hostnames are a fact of life. -- david raistrick http://www.netmeister.org/news/learn2quote.html drais at icantclick.org http://www.expita.com/nomime.html From drais at icantclick.org Tue Dec 13 17:28:08 2011 From: drais at icantclick.org (david raistrick) Date: Tue, 13 Dec 2011 12:28:08 -0500 (EST) Subject: Hostnames in backend definition... In-Reply-To: References: Message-ID: On Tue, 13 Dec 2011, Roberto O. Fern?ndez Crisial wrote: > Hugo, > Have you tried Elastic IP? I think this resolves the first problem you have. First - elastic IP doesn't resolve the the core problem (lack of DNS ttl). This only maps an static -external- address to a still dynamic internal address. For me - I configure ec2-184-73-211-97.compute-1.amazonaws.com (an elastic IP) as a backend. Both varnish and the backend are internal to EC2. Varnish stores this IP because of the AWS split horizon magic: c2-184-73-211-97.compute-1.amazonaws.com has address 10.210.214.95 Now, I replace the instance that is running on that EIP - the old internal IP gets picked up by another customer, and they start recieving my traffic. The new internal IP behind the EIP is now 10.194.22.15, but he never gets any traffic. I have to reload, then unload, the varnish config (or stop/start varnish) to get varnish to be aware of the change. For the OP's situation, he's using Amazon's Elastic Loadbalancer solution, which has neither a static external nor static internal IP address - amazon clearly states that you should -only- use the cname to send traffic to ELB because they perform migration and scaling operations behind the scenes that the customer is never informed of. -- david raistrick http://www.netmeister.org/news/learn2quote.html drais at icantclick.org http://www.expita.com/nomime.html From hugo.cisneiros at gmail.com Tue Dec 13 17:38:32 2011 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Tue, 13 Dec 2011 15:38:32 -0200 Subject: Hostnames in backend definition... In-Reply-To: References: Message-ID: On Tue, Dec 13, 2011 at 3:22 PM, david raistrick wrote: >> I'm setting up varnish to access a load balancer from amazon ec2. I >> have two problems: > For those of us familiar with EC2 architecture, can I get some > clarification? > > You're trying to use varnish in front of ELB, with the ELB hostnames as your > backends? > > I would strongly recommend against this. > >> - since the IP address for the load balancer changes very often, does >> varnish obey the ttl rule for resolving the hostname if I specify the >> hostname in vcl? > > ...for exactly this reason. > > You have no control, knowledge, or insight into when the IP address behind > the ELB hostname changes, and varnish effectively caches the hostname on > startup. ?When the internal IP of ELB changes (and it will!) your traffic > will stop. That's the scenario. The idea for this is to use the ELB as a "fallback" backend. When my main backend fails, varnish will switch to another backend. This fallback backend would be an ELB because of the autoscaling feature for cost efficiency. One instance will be up, waiting for traffic, but getting nothing most of the time. When the main backend fail, all of the traffic (it's a lot) will go to the ELB that will spawn and terminate new servers as necessary. I can use Elastic IPs with fixed instances, but this would kill the "elasticity". And as you said, since varnish caches DNS on startup/vcl.load only, this will not work for me... That's sad :( Thanks for the information! :) -- []'s Hugo www.devin.com.br From drais at icantclick.org Tue Dec 13 17:52:39 2011 From: drais at icantclick.org (david raistrick) Date: Tue, 13 Dec 2011 12:52:39 -0500 (EST) Subject: Hostnames in backend definition... In-Reply-To: References: Message-ID: Oh. I did actually have a suggestion, just forgot to give it. Spin up yet another piece of reverse proxy software that -can- do dns lookups, and make that your backend on localhost, and point it to ELB. I haven't tested this (that is, I dont know what software would handle DNS right - I have tested pound, haproxy, and varnish. We use nginx, too, but only to localhost - maybe I can get a chance to try a dns update test). I've heard good things about perlbal, despite it's name, particularly with regards to dynamic reconfiguration...so that's one possibility. http://www.danga.com/perlbal/ If you do find one that works for you, let us know? :) -- david raistrick http://www.netmeister.org/news/learn2quote.html drais at icantclick.org http://www.expita.com/nomime.html From hugo.cisneiros at gmail.com Tue Dec 13 18:30:02 2011 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Tue, 13 Dec 2011 16:30:02 -0200 Subject: Hostnames in backend definition... In-Reply-To: References: Message-ID: On Tue, Dec 13, 2011 at 3:52 PM, david raistrick wrote: > Oh. ?I did actually have a suggestion, just forgot to give it. > > Spin up yet another piece of reverse proxy software that -can- do dns > lookups, and make that your backend on localhost, and point it to ELB. > > I haven't tested this (that is, I dont know what software would handle DNS > right - I have tested pound, haproxy, and varnish. ?We use nginx, too, but > only to localhost - maybe I can get a chance to try a dns update test). > > I've heard good things about perlbal, despite it's name, particularly with > regards to dynamic reconfiguration...so that's one possibility. > http://www.danga.com/perlbal/ > > If you do find one that works for you, let us know? :) We have thought the same here, but the suggestions are great! To begin, I'm going to use fixed instances, then in the next week I'll try to evaluate these possibilities and incorporate ELB... I'll report here any results :) Thanks a lot! -- []'s Hugo www.devin.com.br From jhalfmoon at milksnot.com Tue Dec 13 23:16:04 2011 From: jhalfmoon at milksnot.com (Johnny Halfmoon) Date: Wed, 14 Dec 2011 00:16:04 +0100 Subject: Possible bug: Missing X-Forwarded-For and X-Varnish headers (Varnish 2.1.5 64bit Linux) Message-ID: <4EE7DCB4.3050107@milksnot.com> Hi, I have a problem with Varnish (2.1.5) not adding 'X-Forwarded-For' and an 'X-Varnish' to a large portion of the backend requests it makes. It looks like a bug in Varnish. The details: A Varnish 2.1.5 / 64bit Linux server is placed in front of 4 backends. The site it is serving has been running for a very long time already without problems. Recently I got an email from a web-developper telling me that he saw a large portion of the requests coming from the Varnish server without X-Forwarded-For headers. When I researched the subject I discovered he was right. This situation was probably already like this for a long time, too. Varnish is supposed to add an 'X-Forwarded-For' and an 'X-Varnish' header to each request it does to a backend, when not in 'pipe mode' that is. When using the varnishlog tool I can see that Varnish says that it is doing this for every request it issues. When I started inspecting tcpdumps of the traffic on the Varnish server I discovered the following: Varnish is handling more traffic than it shows in its logs. The traffic that is not showing up in the logs is the same traffic that the backends see as having no XFF and no X-Varnish header. After closer inspection of the tcpdumps I found out that each time that Varnish receives a response containing a 'Keep-Alive:' and / or 'Connection: Keep-Alive' header, it will stop sending the XFF and X-Varnish headers. It looks very much like Varnish enters 'pipe mode' at that moment, because the traffic without those headers is also not registered in its logs. What's also noticable is that the 'Host:' header, that was first with a lower case 'h' is now spelled with an uppercase 'H'. And also, suddenly a 'Connection: Keep-Alive' header is added to all requests coming from Varnish. These are more indicators that Varnish is now in a kind of 'pipe mode'. I checked to see if the VCL code for the site is configured to go into pipe mode : It is most definitely not. I've added an real life example of a tcpdump below, showing the first few requests going well, and after a certain response, containing the 'Keep-Alive:' and 'Connection: Keep-Alive' header, Varnish stops acting as it should, and starts ommiting the XFF and X-Varnish headers. This is one example of, but I have lots more similar examples for the same site. In the example, I've placed a properly visible marker to indicate where things go wrong. Can someone tell me what's going on here? This looks very much like a bug in Varnish, but I have been searching for the cause of this problem for so long already, that I might very well have overlooked something terribly obvious. Any help will be greatly appreciated. Cheers, Johnny #=== The following is text exctracted from a tcpdump on the Varnish server's network interface ============ GET Accept: application/javascript, */*;q=0.8 Referer: Accept-Language: nl-NL User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) If-Modified-Since: Tue, 13 Dec 2011 19:56:19 GMT Cookie: Accept-Encoding: gzip host: X-Varnish: 1148200324 X-Forwarded-For: HTTP/1.1 200 OK Date: Tue, 13 Dec 2011 20:05:00 GMT Last-Modified: Tue, 13 Dec 2011 20:05:00 GMT Cache-Control: private, must-revalidate Expires: Wed, 16 Aug 2000 10:00:00 GMT Content-Type: text/html;charset=utf-8 Vary: Accept-Encoding Content-Encoding: gzip P3P: policyref="/w3c/p3p.xml", CP="NON DSP COR CURa OUR NOR" Content-Length: 3917 GET Accept: */* Referer: Accept-Language: nl UA-CPU: x86 If-Modified-Since: Tue, 13 Dec 2011 20:04:19 GMT User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB7.2; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) Cookie: Accept-Encoding: gzip host: X-Varnish: 1148200496 X-Forwarded-For: HTTP/1.1 200 OK Date: Tue, 13 Dec 2011 20:05:00 GMT Last-Modified: Tue, 13 Dec 2011 20:05:00 GMT Cache-Control: private, must-revalidate Expires: Wed, 16 Aug 2000 10:00:00 GMT Content-Type: text/html;charset=utf-8 Vary: Accept-Encoding Content-Encoding: gzip P3P: policyref="/w3c/p3p.xml", CP="NON DSP COR CURa OUR NOR" Content-Length: 3917 POST Accept: */* Content-Type: application/x-www-form-urlencoded X-Requested-With: XMLHttpRequest Referer: Accept-Language: nl User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) Content-Length: 112 Connection: Keep-Alive Cache-Control: no-cache Cookie: Accept-Encoding: gzip host: X-Varnish: 1148201219 X-Forwarded-For: HTTP/1.1 200 OK Date: Tue, 13 Dec 2011 20:05:00 GMT Content-Type: text/html; charset=utf-8 Cache-Control: max-age=0 Expires: Tue, 13 Dec 2011 20:05:00 GMT P3P: policyref="/w3c/p3p.xml", CP="NON DSP COR CURa OUR NOR" Vary: Accept-Encoding Content-Encoding: gzip Content-Length: 732 Keep-Alive: timeout=3, max=9999 Connection: Keep-Alive <==========================================================================> <=== From here on, Varnish no longer appends either X-Varnish or X-Forwarded-For headers to backend requests. ===> <==========================================================================> GET Accept: image/png, image/svg+xml, image/*;q=0.8, */*;q=0.5 Referer: Accept-Language: nl-NL User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) Accept-Encoding: gzip, deflate Host: Connection: Keep-Alive Cookie: HTTP/1.1 200 OK Date: Tue, 13 Dec 2011 20:05:01 GMT Server: Apache Last-Modified: Tue, 13 Dec 2011 08:12:33 GMT ETag: "303232-197-4b3f4d02f5e40" Accept-Ranges: bytes Content-Length: 407 Cache-Control: max-age=86400 Expires: Wed, 14 Dec 2011 20:05:01 GMT P3P: policyref="/w3c/p3p.xml", CP="NON DSP COR CURa OUR NOR" Keep-Alive: timeout=3, max=9998 Connection: Keep-Alive Content-Type: image/gif GET Accept: text/css Referer: Accept-Language: nl-NL User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) Accept-Encoding: gzip, deflate Host: Connection: Keep-Alive Cookie: HTTP/1.1 200 OK Date: Tue, 13 Dec 2011 20:05:07 GMT Server: Apache Last-Modified: Tue, 13 Dec 2011 08:12:33 GMT ETag: "59b39-3f9-4b3f4d02f5e40" Accept-Ranges: bytes Cache-Control: max-age=86400 Expires: Wed, 14 Dec 2011 20:05:07 GMT Vary: Accept-Encoding Content-Encoding: gzip P3P: policyref="/w3c/p3p.xml", CP="NON DSP COR CURa OUR NOR" Content-Length: 506 Keep-Alive: timeout=3, max=9997 Connection: Keep-Alive Content-Type: text/css From perbu at varnish-software.com Wed Dec 14 08:36:31 2011 From: perbu at varnish-software.com (Per Buer) Date: Wed, 14 Dec 2011 09:36:31 +0100 Subject: Possible bug: Missing X-Forwarded-For and X-Varnish headers (Varnish 2.1.5 64bit Linux) In-Reply-To: <4EE7DCB4.3050107@milksnot.com> References: <4EE7DCB4.3050107@milksnot.com> Message-ID: Hi Johnny, On Wed, Dec 14, 2011 at 12:16 AM, Johnny Halfmoon wrote: > Hi, > > I have a problem with Varnish (2.1.5) not adding 'X-Forwarded-For' and an > 'X-Varnish' to a large portion of the backend requests it makes. Typically this happens when people are piping requests and forget to set a Connection: close. So then the TCP socket stays open and the browsers keeps on talking to the backend. Is this what is going on with you? -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhalfmoon at milksnot.com Wed Dec 14 09:39:51 2011 From: jhalfmoon at milksnot.com (Johnny Halfmoon) Date: Wed, 14 Dec 2011 10:39:51 +0100 Subject: Possible bug: Missing X-Forwarded-For and X-Varnish headers (Varnish 2.1.5 64bit Linux) In-Reply-To: References: <4EE7DCB4.3050107@milksnot.com> Message-ID: <4EE86EE7.80603@milksnot.com> On 12/14/2011 09:36 AM, Per Buer wrote: > Hi Johnny, > > On Wed, Dec 14, 2011 at 12:16 AM, Johnny Halfmoon > wrote: > > Hi, > > I have a problem with Varnish (2.1.5) not adding 'X-Forwarded-For' and an 'X-Varnish' to a large portion of the backend requests it makes. > > > Typically this happens when people are piping requests and forget to set a Connection: close. So then the TCP socket stays open and the browsers keeps on talking to the backend. Is this what is going on with you? > Unfortunately, no. Pipe connections are correctly configured to close by this config : sub vcl_pipe { set req.http.connection = "close"; } And besides that: When I monitor all requests for this site (lets call it site X), I see only GET and POST requests and nothing that should trigger 'pipe mode' for this site. There are other sites also running on the Varnish server, some of which trigger 'pipe mode' sometimes, but these sites use seperate backends, so this should be of no influence to the traffic to site X. Cheers, Johnny From apj at mutt.dk Wed Dec 14 09:49:38 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Wed, 14 Dec 2011 10:49:38 +0100 Subject: Possible bug: Missing X-Forwarded-For and X-Varnish headers (Varnish 2.1.5 64bit Linux) In-Reply-To: <4EE86EE7.80603@milksnot.com> References: <4EE7DCB4.3050107@milksnot.com> <4EE86EE7.80603@milksnot.com> Message-ID: <20111214094938.GO3214@nerd.dk> On Wed, Dec 14, 2011 at 10:39:51AM +0100, Johnny Halfmoon wrote: > > > >Typically this happens when people are piping requests and forget to set a > >Connection: close. So then the TCP socket stays open and the browsers keeps > >on talking to the backend. Is this what is going on with you? > > > Unfortunately, no. Pipe connections are correctly configured to close by this > config : varnishlog and corresponding pcap dumps would be very helpful. -- Andreas From mhettwer at team.mobile.de Wed Dec 14 10:09:37 2011 From: mhettwer at team.mobile.de (Hettwer, Marian) Date: Wed, 14 Dec 2011 10:09:37 +0000 Subject: Hostnames in backend definition... In-Reply-To: Message-ID: Hi there, On 13.12.11 18:22, "david raistrick" wrote: > >I've sort of been threatening to get one of my team to work up a patch to >varnish to do some DNS sanity, but until it actually becomes a major >blocker for us (we have other workarounds in place - like not using ELB >and using internal dynamic config re-generators) I can't get the resource >to do it... Does varnish cache DNS entries forever and does the dns lookup only once at startup time? If so, why would it do that? The initial DNS answer would include a TTL, indicating how long the record is valid. Proper behaviour would be to do another DNS questions, as soon as the TTL is not valid anymore. Not behaving correctly in DNS is bad. Perhaps someone could enlighten me why one would go that road? (more or less the same behaviour is known to me by java, and I hate it. Squid is doing the same too. And it's a pain). Cheers, Marian From phk at phk.freebsd.dk Wed Dec 14 10:56:17 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 14 Dec 2011 10:56:17 +0000 Subject: Hostnames in backend definition... In-Reply-To: Your message of "Wed, 14 Dec 2011 10:09:37 GMT." Message-ID: <95486.1323860177@critter.freebsd.dk> In message , "Hettwer, Marian" writes: >Does varnish cache DNS entries forever and does the dns lookup only once >at startup time? > >If so, why would it do that? Varnish does the DNS lookup at VCL compiler time. This is to avoid DNS-server trouble impacting varnish doing its job. We have talked about having the health-probe do DNS lookups in the background, but there are some interesting cornercases that needs to be dealt with, for instance: What happens when we suddenly get two IP#'s back, or when we get no ip numbers back. The advantage to doing the DNS lookup and error checking at VCL compile time, is that we have a CLI connection to report trouble on. >The initial DNS answer would include a TTL, indicating how long the record >is valid. Yes, and no standardized API returns that TTL to the application program, so all you can do is call getaddrinfo() periodically, at whatever frequency you deem appropriate :-( -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jhalfmoon at milksnot.com Wed Dec 14 11:16:49 2011 From: jhalfmoon at milksnot.com (Johnny Halfmoon) Date: Wed, 14 Dec 2011 12:16:49 +0100 Subject: Possible bug: Missing X-Forwarded-For and X-Varnish headers (Varnish 2.1.5 64bit Linux) In-Reply-To: <20111214094938.GO3214@nerd.dk> References: <4EE7DCB4.3050107@milksnot.com> <4EE86EE7.80603@milksnot.com> <20111214094938.GO3214@nerd.dk> Message-ID: <4EE885A1.7000700@milksnot.com> On 12/14/2011 10:49 AM, Andreas Plesner Jacobsen wrote: > On Wed, Dec 14, 2011 at 10:39:51AM +0100, Johnny Halfmoon wrote: >>> Typically this happens when people are piping requests and forget to set a >>> Connection: close. So then the TCP socket stays open and the browsers keeps >>> on talking to the backend. Is this what is going on with you? >>> >> Unfortunately, no. Pipe connections are correctly configured to close by this >> config : > varnishlog and corresponding pcap dumps would be very helpful. > In my initial post there's already a real-life example snapshot from a single TCP session between the Varnish server and a backend, illustrating the issue. As for a Varnishlog; the backend requests are not logged, so those do not show up in the log. I'll prepare an integrated backend/frontend varnishlog, to post here, where at least the frontend requests and responses will be visible. And I'll see if I can post the tcpdump corresponding to that varnishlog too. The dumps are huge, even on short time spans, so they tend to take a while to be processed. From mark.nellemann at gmail.com Wed Dec 14 11:55:52 2011 From: mark.nellemann at gmail.com (Mark Nellemann) Date: Wed, 14 Dec 2011 12:55:52 +0100 Subject: Varnishncsa (3.0.2) showing 0 bytes in logfile Message-ID: <8ECA4E8F-39A7-4375-86FD-C5542FBDC589@gmail.com> Hi, I am using Varnish 3.0.2-1~1lucid1 as cache for IIS Smoothstreaming files. I have noticed that some requests gets logged as 0 bytes in the varnishncsa.log file. These requests comes from another varnish server, but that should not matter right? Requests for other hosts to other urls are correct. xxx.xxx.xxx.xxx - - [14/Dec/2011:11:46:48 +0100] "GET http://someurl/path/ism/2011-12/trailer.ism/QualityLevels(1400000)/Fragments(video=775200000) HTTP/1.1" 200 0 "http://otherurl/resources/slv/Player.xap" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MS-RTC LM 8)" Any idea why this is happening ? Or perhaps there are better ways to count traffic ? /Mark From hugo.cisneiros at gmail.com Wed Dec 14 12:46:51 2011 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Wed, 14 Dec 2011 10:46:51 -0200 Subject: send reason along with backend request In-Reply-To: References: <2B3A03D6-75C2-4E5D-B4B5-3959911DACA8@thenextweb.com> Message-ID: On Sun, Dec 11, 2011 at 11:47 PM, AD wrote: > you should be able to add a req.http.header in your vcl_fetch. ?I dont > believe vcl_fetch gets called unless you are in a cache_miss and you need to > contact the backend. In vcl_fetch varnish already contacted the backend and fetched the answer. This is why beresp.* variable is already there. You can add headers before the backend request on vcl_miss or vcl_hit (in your case, vcl_miss). -- []'s Hugo www.devin.com.br From mib at electronic-minds.de Wed Dec 14 13:14:17 2011 From: mib at electronic-minds.de (Michael Borejdo) Date: Wed, 14 Dec 2011 13:14:17 +0000 Subject: Varnish chunks response with esi-fragments Message-ID: Hi list, being new to varnish and http-caching in general, i have a few questions, i'd love to ask. My Setup: varnish [3.0.2] (80) <-> nginx (8080) [1.1.10] <-> php-fpm (9000) (same behaviour using static html only) I have a page (/outer.html) with two esi-fragments. (esi:include tags). (/inner.html) I expect varnish to (very simplified): - receive a request for /outer.html - recognize and parse the esi-tags in /outer.html - do two additional requests to fetch the esi-srcs (inner.html) - respond with the assembled page. Varnish is sending a response, which has no content-length, hence the Transfer-Encoding is chunked. In my current setup, outer.html loads and the (correctly evaluated) esi-tags appear one after another in the chunked response. Q) Why is varnish not buffering the response and sending it "as a whole" to the client, but chunking the response? Q) Is this the expected behaviour? Q) How can i prevent the chunked response but tell varnish to buffer the whole page (including its esi-fragments) and send it fully to the client? ? thanks ? Mike ? PS. I can provide the logs for all layers, if this will help From arjen at thenextweb.com Wed Dec 14 14:08:58 2011 From: arjen at thenextweb.com (Arjen Schat) Date: Wed, 14 Dec 2011 15:08:58 +0100 Subject: send reason along with backend request In-Reply-To: References: <2B3A03D6-75C2-4E5D-B4B5-3959911DACA8@thenextweb.com> Message-ID: <51942A0D-BB21-477C-A264-365DD947D98D@arjenschat.com> Yep, that is too late in the process. Is the reason for the call to vcl_miss exposed? This would allow vcl_miss to set an extra bereq. Or is there a way to see which stats counters are updated by lookup? Thank you, Arjen On 14 dec. 2011, at 13:46, Hugo Cisneiros (Eitch) wrote: > On Sun, Dec 11, 2011 at 11:47 PM, AD wrote: >> you should be able to add a req.http.header in your vcl_fetch. I dont >> believe vcl_fetch gets called unless you are in a cache_miss and you need to >> contact the backend. > > In vcl_fetch varnish already contacted the backend and fetched the > answer. This is why beresp.* variable is already there. You can add > headers before the backend request on vcl_miss or vcl_hit (in your > case, vcl_miss). > > -- > []'s > Hugo > www.devin.com.br > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From hugo.cisneiros at gmail.com Wed Dec 14 14:23:01 2011 From: hugo.cisneiros at gmail.com (Hugo Cisneiros (Eitch)) Date: Wed, 14 Dec 2011 12:23:01 -0200 Subject: send reason along with backend request In-Reply-To: <51942A0D-BB21-477C-A264-365DD947D98D@arjenschat.com> References: <2B3A03D6-75C2-4E5D-B4B5-3959911DACA8@thenextweb.com> <51942A0D-BB21-477C-A264-365DD947D98D@arjenschat.com> Message-ID: On Wed, Dec 14, 2011 at 12:08 PM, Arjen Schat wrote: > Yep, that is too late in the process. > > Is the reason for the call to vcl_miss exposed? This would allow vcl_miss to set an extra bereq. Or is there a way to see which stats counters are updated by lookup? I didn't quite understand the question, but vcl_miss is always called when a lookup was made on an object and it wasn't in the cache (or the object's ttl is expired in the cache). I think that was the original question. So in vcl_miss, you can define your headers using the bereq. variable, for example: set bereq.http.X-Radical-Header = "value" And they will be send after vcl_miss returns (fetch). -- []'s Hugo www.devin.com.br From mhettwer at team.mobile.de Wed Dec 14 15:26:22 2011 From: mhettwer at team.mobile.de (Hettwer, Marian) Date: Wed, 14 Dec 2011 15:26:22 +0000 Subject: Hostnames in backend definition... In-Reply-To: <95486.1323860177@critter.freebsd.dk> Message-ID: Hi there, On 14.12.11 11:56, "Poul-Henning Kamp" wrote: >In message , "Hettwer, Marian" >writes: > >>Does varnish cache DNS entries forever and does the dns lookup only once >>at startup time? >> >>If so, why would it do that? > >Varnish does the DNS lookup at VCL compiler time. > >This is to avoid DNS-server trouble impacting varnish doing its job. > >We have talked about having the health-probe do DNS lookups in the >background, but there are some interesting cornercases that needs >to be dealt with, for instance: What happens when we suddenly get >two IP#'s back, or when we get no ip numbers back. Well, that would either mean, set the backend to fail, or use the first IP provided. But as you said, you had your talks about it, so I guess the decision is settled for now. > >The advantage to doing the DNS lookup and error checking at VCL compile >time, is that we have a CLI connection to report trouble on. I can understand this. On the other hand, even in your own datacenter setup, one might need to change dns records. Restarting a service for that is just... Well... painful. (I'm just thinking about our hundreds of tomcats, where java is also treating DNS not as one might expect. But this gets off-topic). > >>The initial DNS answer would include a TTL, indicating how long the >>record >>is valid. > >Yes, and no standardized API returns that TTL to the application >program, so all you can do is call getaddrinfo() periodically, at >whatever frequency you deem appropriate :-( Oh crap. So that's the reason why there's so much OSS around which just plain caches DNS forever after startup? I wonder how caching dns servers are doing it (thinking of pdns-resolver). On the other hand, I've seen a whole shitload of badly behaving dns cachers. So probably it again boils down to no standardized way to do it. Crap indeed. But thanks for your insights! :) Cheers, Marian From phk at phk.freebsd.dk Thu Dec 15 00:26:13 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 15 Dec 2011 00:26:13 +0000 Subject: Hostnames in backend definition... In-Reply-To: Your message of "Wed, 14 Dec 2011 15:26:22 GMT." Message-ID: <10746.1323908773@critter.freebsd.dk> In message , "Hettwer, Marian" writes: >>We have talked about having the health-probe do DNS lookups in the >>background, but there are some interesting cornercases that needs >>to be dealt with, for instance: What happens when we suddenly get >>two IP#'s back, or when we get no ip numbers back. > >But as you said, you had your talks about it, so I guess the decision is >settled for now. No, not really: Absent a consistent proposal on how it should work, we have not changed anything yet. Ideas are always welcome, but they need to be thought through and all the odd-ball cases considered if we are to adobt them. >I can understand this. On the other hand, even in your own datacenter >setup, one might need to change dns records. Restarting a service for that >is just... Well... painful. You don't need to restart any service: Just load your VCL again and switch to the newly loaded VCL and you won't see your service as much as blink. Varnish was built to do that kind of change on the fly, and we're proud of how well it works. >So that's the reason why there's so much OSS around which just plain >caches DNS forever after startup? yes. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jason at pethub.com Thu Dec 15 04:40:02 2011 From: jason at pethub.com (Jason Farnsworth) Date: Thu, 15 Dec 2011 04:40:02 +0000 Subject: Rewriting/enforcing SSL behing an SSL termination point In-Reply-To: Message-ID: This is great, I'll give this a shot and report back! From: Per Buer > Date: Fri, 9 Dec 2011 09:48:48 +0100 To: Jason Farnsworth > Cc: "varnish-misc at varnish-cache.org" > Subject: Re: Rewriting/enforcing SSL behing an SSL termination point On Fri, Dec 9, 2011 at 8:08 AM, Jason Farnsworth > wrote: We are hosted on Amazon Web Services and all SSL termination is done by an Elastic Load Balancer. So all I'm looking to do is re-write URLs like this http://domain.com -> https://www.domain.com http://www.domain.com -> https://www.domain.com https://domain.com -> https://www.domain.com Varnish will not rewrite the actual content coming from the backend. We can however, _redirect_ the client whenever they ask for a http:// URL. We use the following code on varnish-cache.org to do this: in vcl_recv: if ( (req.http.host ~ "(?i)www.varnish-cache.org") && !(client.ip ~ localhost)) { set req.http.x-redir-url = "https://" + req.http.host + req.url; error 750 req.http.x-redir-url; } (..) sub vcl_error { # standard redirection in VCL: if (obj.status == 750) { set obj.http.Location = obj.response; set obj.status = 302; return(deliver); } } Since we have an SSL terminator in front of Varnish client.ip is localhost when there is SSL present. You might want to change the code to test X-Forwarded-Proto for whatever it is set to. -- [http://www.varnish-software.com/sites/default/files/varnishsoft_white_190x47.png] Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Thu Dec 15 09:35:41 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 15 Dec 2011 10:35:41 +0100 Subject: Varnish chunks response with esi-fragments In-Reply-To: References: Message-ID: <20111215093541.GB11804@err.no> ]] Michael Borejdo > Q) Why is varnish not buffering the response and sending it "as a whole" to the client, but chunking the response? Because that works better with how Varnish works internally. > Q) Is this the expected behaviour? Yes. > Q) How can i prevent the chunked response but tell varnish to buffer > the whole page (including its esi-fragments) and send it fully to the > client? Why do you want to do this? You can do it, but it'll be a bit of a workaround by doing something like using yourself as a backend for the ESI-ed request. I guess something like: backend myself { .host = "localhost"; .port = "80"; } sub vcl_recv { if (! req.http.x-esi) { set req.backend = myself; set req.http.x-esi = "yes, please"; } } sub vcl_hash { hash_data(req.http.x-esi); } sub vcl_fetch { if (req.http.x-esi) { set beresp.do_esi = true; } } might work, possibly with some extra fluff to ensure you just use ESI for the relevant URLs. Cache efficientcy will also go down this way. Regards, -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Thu Dec 15 09:37:32 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 15 Dec 2011 10:37:32 +0100 Subject: Varnishncsa (3.0.2) showing 0 bytes in logfile In-Reply-To: <8ECA4E8F-39A7-4375-86FD-C5542FBDC589@gmail.com> References: <8ECA4E8F-39A7-4375-86FD-C5542FBDC589@gmail.com> Message-ID: <20111215093732.GC11804@err.no> ]] Mark Nellemann > Any idea why this is happening ? > Or perhaps there are better ways to count traffic ? Can you capture the corresponding varnishlog fragment? -- Tollef Fog Heen Technical lead, Varnish Software t: +47 21 98 92 64 From personal at arjenschat.com Fri Dec 9 17:36:58 2011 From: personal at arjenschat.com (Arjen Schat) Date: Fri, 9 Dec 2011 18:36:58 +0100 Subject: Extra header to let the backend know why the cached missed Message-ID: <944FDE46-9338-4AC9-9743-D9175366AEBD@arjenschat.com> Hi Gents, Is there a way to add an extra header to the backend request on a miss? Eg x-varnish-miss-reason: purged / expired / lru / no-cache In this way the backend knows why the request was made. Thank you, Arjen From thierry.magnien at sfr.com Thu Dec 15 16:29:29 2011 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Thu, 15 Dec 2011 17:29:29 +0100 Subject: Change backend when pass Message-ID: <4A029B1A60B8E340A50D654D2F130DAA2FF4BEC0DA@EXCV001.encara.local.ads> Hi, I have a setup where I don't manage objects' TTL because the origin server is handled by someone else. To setup correct caching options, I use an nginx server as reverse proxy. So, my setup is : Client -> varnish -> nginx -> origin server The nginx fixes the TTL options so that varnish just has to respect what the backend tells him. The problem with this setup is the following : for uncacheable objects, I have an extra step (nginx) that I would like to avoid. I would like to setup something like this: - in vcl_recv I select nginx as my default backend - if the object is non-cacheable, return hit_for_pass - in vcl_pass, set backend to point to the origin server. So each time I get a hit_for_pass, all subsequent requests for the object will go directly to the origin server and not through the nginx. Does this sound good to you or am I completely mistaken ? Thanks, Thierry From drais at icantclick.org Thu Dec 15 17:01:03 2011 From: drais at icantclick.org (david raistrick) Date: Thu, 15 Dec 2011 12:01:03 -0500 (EST) Subject: Hostnames in backend definition... In-Reply-To: <10746.1323908773@critter.freebsd.dk> References: <10746.1323908773@critter.freebsd.dk> Message-ID: On Thu, 15 Dec 2011, Poul-Henning Kamp wrote: > You don't need to restart any service: Just load your VCL again and > switch to the newly loaded VCL and you won't see your service as much > as blink. Don't forget to unload your old VCL, too (at least for 2.x), iirc. -- david raistrick http://www.netmeister.org/news/learn2quote.html drais at icantclick.org http://www.expita.com/nomime.html From phk at phk.freebsd.dk Thu Dec 15 17:04:36 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 15 Dec 2011 17:04:36 +0000 Subject: Hostnames in backend definition... In-Reply-To: Your message of "Thu, 15 Dec 2011 12:01:03 EST." Message-ID: <6347.1323968676@critter.freebsd.dk> In message , david rai strick writes: >On Thu, 15 Dec 2011, Poul-Henning Kamp wrote: > >> You don't need to restart any service: Just load your VCL again and >> switch to the newly loaded VCL and you won't see your service as much >> as blink. > >Don't forget to unload your old VCL, too (at least for 2.x), iirc. Why ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From drais at icantclick.org Thu Dec 15 17:06:40 2011 From: drais at icantclick.org (david raistrick) Date: Thu, 15 Dec 2011 12:06:40 -0500 (EST) Subject: Hostnames in backend definition... In-Reply-To: <6347.1323968676@critter.freebsd.dk> References: <6347.1323968676@critter.freebsd.dk> Message-ID: On Thu, 15 Dec 2011, Poul-Henning Kamp wrote: >>> switch to the newly loaded VCL and you won't see your service as much >>> as blink. >> >> Don't forget to unload your old VCL, too (at least for 2.x), iirc. > > Why ? So you stop sending probes to the old backend IP addresses. Here are my notes from the discussion around it: well, yes, you need to discard backends for probes to stop this is intentional vcl.discard old_config ... @keen discarding the boot vcl made my stale healthchecks stop fyi I didn't think of the boot one just discarding right away should work. I think there's a race condition if you discard it within a _really_ short time that you might hit Mithrandir, huh ? phk: at least in 2.1, I've seen asserts when people discard too quickly. Mithrandir, I hope that one is solved in 2.1.4 possibly 2.1.3 -- david raistrick http://www.netmeister.org/news/learn2quote.html drais at icantclick.org http://www.expita.com/nomime.html From mark.nellemann at gmail.com Thu Dec 15 17:27:32 2011 From: mark.nellemann at gmail.com (Mark N.) Date: Thu, 15 Dec 2011 18:27:32 +0100 Subject: Varnishncsa (3.0.2) showing 0 bytes in logfile In-Reply-To: <20111215093732.GC11804@err.no> References: <8ECA4E8F-39A7-4375-86FD-C5542FBDC589@gmail.com> <20111215093732.GC11804@err.no> Message-ID: Hi, On Thursday, December 15, 2011 at 10:37 AM, Tollef Fog Heen wrote: > ]] Mark Nellemann > > > Any idea why this is happening ? > > Or perhaps there are better ways to count traffic ? > > > > > Can you capture the corresponding varnishlog fragment? > This is one: 16 SessionOpen c xxx.xxx.xxx.xxx 36981 :80 16 ReqEnd c 2077822584 1323969936.139942884 1323969936.150170088 0.000026703 0.010180473 0.000046730 16 SessionClose c Stream error /Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Thu Dec 15 18:13:32 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 15 Dec 2011 18:13:32 +0000 Subject: Hostnames in backend definition... In-Reply-To: Your message of "Thu, 15 Dec 2011 12:06:40 EST." Message-ID: <6526.1323972812@critter.freebsd.dk> In message , david rai strick writes: >>>> switch to the newly loaded VCL and you won't see your service as much >>>> as blink. >>> >>> Don't forget to unload your old VCL, too (at least for 2.x), iirc. >> >> Why ? > >So you stop sending probes to the old backend IP addresses. Here are my >notes from the discussion around it: Ahh, that's a different and only slightly related issue, but your advice seems to be opposite of what would be warranted, it's the unloading that causes the crash in that case... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jason at pethub.com Thu Dec 15 20:51:01 2011 From: jason at pethub.com (Jason Farnsworth) Date: Thu, 15 Dec 2011 20:51:01 +0000 Subject: Rewriting/enforcing SSL behing an SSL termination point In-Reply-To: Message-ID: Alright, I had to re-write the rule a bit since I'm watching for X-Forwarded-Proto, and want to handle domain.com as well as www.domain.com. I've put this rule together, and it seems to be working OK? sub vcl_recv { if((req.http.host ~ "^(?i)pethub.com") || ((req.http.host ~ "^(?i)www.pethub.com") && (req.http.X-Forwarded-Proto !~ "(?i)https"))){ set req.http.x-redir-url = "https://www.pethub.com" + req.url; error 750 req.http.x-redir-url; } } sub vcl_error { if (obj.status == 750) { set obj.http.Location = obj.response; set obj.status = 302; return(deliver); } Thanks for the info! Jason From: Jason Farnsworth > Date: Wed, 14 Dec 2011 20:39:59 -0800 To: Per Buer > Cc: "varnish-misc at varnish-cache.org" > Subject: Re: Rewriting/enforcing SSL behing an SSL termination point This is great, I'll give this a shot and report back! From: Per Buer > Date: Fri, 9 Dec 2011 09:48:48 +0100 To: Jason Farnsworth > Cc: "varnish-misc at varnish-cache.org" > Subject: Re: Rewriting/enforcing SSL behing an SSL termination point On Fri, Dec 9, 2011 at 8:08 AM, Jason Farnsworth > wrote: We are hosted on Amazon Web Services and all SSL termination is done by an Elastic Load Balancer. So all I'm looking to do is re-write URLs like this http://domain.com -> https://www.domain.com http://www.domain.com -> https://www.domain.com https://domain.com -> https://www.domain.com Varnish will not rewrite the actual content coming from the backend. We can however, _redirect_ the client whenever they ask for a http:// URL. We use the following code on varnish-cache.org to do this: in vcl_recv: if ( (req.http.host ~ "(?i)www.varnish-cache.org") && !(client.ip ~ localhost)) { set req.http.x-redir-url = "https://" + req.http.host + req.url; error 750 req.http.x-redir-url; } (..) sub vcl_error { # standard redirection in VCL: if (obj.status == 750) { set obj.http.Location = obj.response; set obj.status = 302; return(deliver); } } Since we have an SSL terminator in front of Varnish client.ip is localhost when there is SSL present. You might want to change the code to test X-Forwarded-Proto for whatever it is set to. -- [http://www.varnish-software.com/sites/default/files/varnishsoft_white_190x47.png] Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From dbaker at flightaware.com Thu Dec 15 23:15:43 2011 From: dbaker at flightaware.com (Daniel Baker) Date: Thu, 15 Dec 2011 23:15:43 +0000 Subject: Assert error in vfp_esi_bytes_gg() Message-ID: <8514345A-F42C-4DF1-AC70-BA6C42C2F745@flightaware.com> Hello, all. We're running varnish 3.0.2 out of FreeBSD ports and are seeing the same crash on multiple different servers assert error in vfp_esi_bytes_gg on multiple servers: Dec 12 03:09:24 bilee varnishd[1967]: Child (2372) Panic message: Assert error in vfp_esi_bytes_gg(), cache_esi_fetch.c line 273: Condition(i >= VGZ_OK) not true. thread = (cache-worker) ident = FreeBSD,8.2-STABLE,amd64,-smalloc,-smalloc,-hclassic,kqueue Dec 13 18:17:35 rokit varnishd[6636]: Child (6637) Panic message: Assert error in vfp_esi_bytes_gg(), cache_esi_fetch.c line 273: Condition(i >= VGZ_OK) not true. thread = (cache-worker) ident = FreeBSD,8.2-STABLE,amd64,-smalloc,-smalloc,-hclassic,kqueue Despite the error, it does not look like we actually got the core dumps. It looks like there have been some possibly related commits in the last few weeks although the commit msg doesn't seem to reflect addressing a specific issue. Has anyone else seen this in 3.0.2 and/or HEAD? Thanks, Daniel From mhettwer at team.mobile.de Fri Dec 16 13:11:49 2011 From: mhettwer at team.mobile.de (Hettwer, Marian) Date: Fri, 16 Dec 2011 13:11:49 +0000 Subject: Hostnames in backend definition... In-Reply-To: <10746.1323908773@critter.freebsd.dk> Message-ID: On 15.12.11 01:26, "Poul-Henning Kamp" wrote: >In message , "Hettwer, Marian" >writes: > >>>We have talked about having the health-probe do DNS lookups in the >>>background, but there are some interesting cornercases that needs >>>to be dealt with, for instance: What happens when we suddenly get >>>two IP#'s back, or when we get no ip numbers back. >> >>But as you said, you had your talks about it, so I guess the decision is >>settled for now. > >No, not really: Absent a consistent proposal on how it should work, >we have not changed anything yet. Since I'm not a programmer, all I can say is, honor TTL in dns answers and asked again after the TTL expired. But you will correctly reply, that this is not an easy task. So I'd better shut up now ;) > >Ideas are always welcome, but they need to be thought through and all >the odd-ball cases considered if we are to adobt them. > >>I can understand this. On the other hand, even in your own datacenter >>setup, one might need to change dns records. Restarting a service for >>that >>is just... Well... painful. > >You don't need to restart any service: Just load your VCL again and >switch to the newly loaded VCL and you won't see your service as much >as blink. Oh right. Of course. :) > >Varnish was built to do that kind of change on the fly, and we're >proud of how well it works. And you guys can be proud of varnish! I'm using it wherever it's appropriate and honestly, varnish is probably the most interesting piece of software I stumbled upon in my last 6 years as a professional administrator. Just keep up the good job! Cheers, Marian From cosimo at streppone.it Mon Dec 19 14:27:02 2011 From: cosimo at streppone.it (Cosimo Streppone) Date: Mon, 19 Dec 2011 15:27:02 +0100 Subject: Rewriting/enforcing SSL behing an SSL termination point In-Reply-To: References: Message-ID: On Thu, 15 Dec 2011 21:51:01 +0100, Jason Farnsworth wrote: > I've put this rule together, and it seems to be working Thanks. Added to the wiki, may be useful to others (/me). "Redirect to HTTPS", https://www.varnish-cache.org/trac/wiki/VCLExampleRedirectInVCL -- Cosimo From jason at pethub.com Tue Dec 20 10:04:26 2011 From: jason at pethub.com (Jason Farnsworth) Date: Tue, 20 Dec 2011 10:04:26 +0000 Subject: Rewriting/enforcing SSL behing an SSL termination point In-Reply-To: Message-ID: Great! FYI - we've had this running on our production site for a few days now with no issues at all. It's working very well. On 12/19/11 6:27 AM, "Cosimo Streppone" wrote: >On Thu, 15 Dec 2011 21:51:01 +0100, Jason Farnsworth >wrote: > >> I've put this rule together, and it seems to be working > >Thanks. Added to the wiki, may be useful to others (/me). > >"Redirect to HTTPS", > >https://www.varnish-cache.org/trac/wiki/VCLExampleRedirectInVCL > >-- >Cosimo > From parsimail2001 at yahoo.se Wed Dec 21 23:33:33 2011 From: parsimail2001 at yahoo.se (Your Friend) Date: Wed, 21 Dec 2011 23:33:33 +0000 (GMT) Subject: Yet another SSLquestion Message-ID: <1324510413.36799.YahooMailNeo@web26301.mail.ukl.yahoo.com> Hi guys, I have been thinking about that how I can use ssl with nginx and varnish. (I use nginx as backend) and I know that Varnish does not support SSL yet. In my case I want to use SSL on some pages like www.domain.com/login/ ? domain.com/register/. Does anyone have any solution? Would this work to create a VCL rule to send all https requests to nginx? Kind regards, Ali -------------- next part -------------- An HTML attachment was scrubbed... URL: From pprocacci at datapipe.com Thu Dec 22 01:47:48 2011 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Wed, 21 Dec 2011 19:47:48 -0600 Subject: Yet another SSLquestion In-Reply-To: <1324510413.36799.YahooMailNeo@web26301.mail.ukl.yahoo.com> References: <1324510413.36799.YahooMailNeo@web26301.mail.ukl.yahoo.com> Message-ID: <20111222014748.GL3214@nat.myhome> You wouldn't send https requests anywhere because they are already using the ssl port (443). You'd only be interested in using port 80 requests (i.e. http). An example of how to do this is here: https://www.varnish-cache.org/trac/wiki/VCLExampleRedirectInVCL ~Paul On Wed, Dec 21, 2011 at 11:33:33PM +0000, Your Friend wrote: > Hi guys, > > > I have been thinking about that how I can use ssl with nginx and varnish. (I use nginx as backend) and I know that Varnish does not support SSL yet. > > In my case I want to use SSL on some pages like > www.domain.com/login/ ? > domain.com/register/. > > Does anyone have any solution? Would this work to create a VCL rule to send all https requests to nginx? > > Kind regards, Ali > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From parsimail2001 at yahoo.se Thu Dec 22 13:31:34 2011 From: parsimail2001 at yahoo.se (Your Friend) Date: Thu, 22 Dec 2011 13:31:34 +0000 (GMT) Subject: SV: Yet another SSLquestion In-Reply-To: <20111222014748.GL3214@nat.myhome> References: <1324510413.36799.YahooMailNeo@web26301.mail.ukl.yahoo.com> <20111222014748.GL3214@nat.myhome> Message-ID: <1324560694.43010.YahooMailNeo@web26306.mail.ukl.yahoo.com> Hi Paul, Thanks for your reply and link. It was very helpful. Kind regards, Ali ________________________________ You wouldn't send https requests anywhere because they are already using the ssl port (443).? You'd only be interested in using port 80 requests (i.e. http). An example of how to do this is here: https://www.varnish-cache.org/trac/wiki/VCLExampleRedirectInVCL ~Paul On Wed, Dec 21, 2011 at 11:33:33PM +0000, Your Friend wrote: > Hi guys, > > > I have been thinking about that how I can use ssl with nginx and varnish. (I use nginx as backend) and I know that Varnish does not support SSL yet. > > In my case I want to use SSL on some pages like > www.domain.com/login/ ? > domain.com/register/. > > Does anyone have any solution? Would this work to create a VCL rule to send all https requests to nginx? > > Kind regards, Ali > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger.szerb at gmail.com Thu Dec 22 15:44:07 2011 From: roger.szerb at gmail.com (Szerb Roger) Date: Thu, 22 Dec 2011 16:44:07 +0100 Subject: sporadic unknown result for backend probes (random director) Message-ID: Hi! We are investigating apache2 threads spiking issue, but bumped into a backend probe problem(?). Apache2 backends configured for keep-alive connections, varnish poll probes just check if static.txt file exists on the backend host, but varnishlog says sporadic 4--X--- . Can anyone give a hint? Thanks and Merry Christmas! :) VCL: probe healthcheck { .url = "/static.txt"; .timeout = 300ms; .interval = 2s; .window = 10; .threshold = 2; } backend hnode { .host = "10.0.80.2"; .port = "8081"; .probe = healthcheck; } backend h2node { .host = "10.0.80.2"; .port = "8082"; .probe = healthcheck; } backend hFBnode { .host = "10.0.80.2"; .port = "8083"; .probe = healthcheck; } ... varnishlog | grep health 0 Backend_health - hnode Still healthy 4--X-RH 10 2 10 0.000484 0.000537 HTTP/1.1 200 OK 0 Backend_health - h2node Still healthy 4--X-RH 10 2 10 0.000512 0.000602 HTTP/1.1 200 OK 0 Backend_health - hFBnode Still healthy 4--X-RH 10 2 10 0.000495 0.000579 HTTP/1.1 200 OK 0 Backend_health - hnode Still healthy 4--X-RH 10 2 10 0.000568 0.000544 HTTP/1.1 200 OK 0 Backend_health - h2node Still healthy 4--X-RH 10 2 10 0.000452 0.000564 HTTP/1.1 200 OK 0 Backend_health - hFBnode Still healthy 4--X-RH 10 2 10 0.000648 0.000597 HTTP/1.1 200 OK ??????? 0 Backend_health - hnode Still healthy 4--X--- 9 2 10 0.000000 0.000544 0 Backend_health - h2node Still healthy 4--X--- 9 2 10 0.000000 0.000564 0 Backend_health - hFBnode Still healthy 4--X--- 9 2 10 0.000000 0.000597 ??????? 0 Backend_health - hnode Still healthy 4--X-RH 9 2 10 0.000914 0.000637 HTTP/1.1 200 OK 0 Backend_health - h2node Still healthy 4--X-RH 9 2 10 0.000873 0.000641 HTTP/1.1 200 OK 0 Backend_health - hFBnode Still healthy 4--X-RH 9 2 10 0.000802 0.000648 HTTP/1.1 200 OK 0 Backend_health - hnode Still healthy 4--X-RH 9 2 10 0.000672 0.000646 HTTP/1.1 200 OK 0 Backend_health - h2node Still healthy 4--X-RH 9 2 10 0.000551 0.000619 HTTP/1.1 200 OK 0 Backend_health - hFBnode Still healthy 4--X-RH 9 2 10 0.000578 0.000631 HTTP/1.1 200 OK 0 Backend_health - hnode Still healthy 4--X-RH 9 2 10 0.000647 0.000646 HTTP/1.1 200 OK 0 Backend_health - h2node Still healthy 4--X-RH 9 2 10 0.000585 0.000610 HTTP/1.1 200 OK 0 Backend_health - hFBnode Still healthy 4--X-RH 9 2 10 0.000545 0.000609 HTTP/1.1 200 OK From lixinfish at gmail.com Fri Dec 23 11:26:13 2011 From: lixinfish at gmail.com (Lx Yu) Date: Fri, 23 Dec 2011 19:26:13 +0800 Subject: Can varnish be used on a e-commerce website where almost all pages are dynamic? Message-ID: <53B0E90D-2F29-4604-8F9E-0541A6E26C2E@gmail.com> Hello, I'm working on a e-commerce site, and the content of it is almost fully dynamic. Most user must login to purchase, and every login user in have a collection of personalized items displayed in every page. And every item's status is in real-time(whether it's in stock, and how many left). So for now, we dynamically generate every webpage, and use a CDN to delivery all static files(images, css, js). Then, can varnish be used in such situation where almost all things is dynamic? As I see Facebook is also using varnish, and I'm wondering how fb did it. -- Lx Yu, Developer, ELEME Inc. Email: lixinfish at gmail.com From samu at linuxasylum.net Fri Dec 23 13:19:17 2011 From: samu at linuxasylum.net (Samuele Giovanni Tonon) Date: Fri, 23 Dec 2011 14:19:17 +0100 Subject: debugging varnish with vlc_error Message-ID: <4EF47FD5.8020804@linuxasylum.net> hello, i need to do some deep analysys on how varnish is acting on our production environment. thanks to some previous email in the list, i was able to send to syslog some information. i have some questions: does this configuration is useful to trace guru meditation ? if not do you have any advice? I would like also to print the hostname of the backend which is replying with the problem; the only thing i was able to come close is req.backend but this just print the name of the director and not the backend name nor the backend ip; any idea ? for example at the moment i'm getting : Dec 23 14:18:20 isvar01 varnishd[1893]: V Err '/' 'Service Unavailable' '2011782056' [www.myhost.net] {director_islig} here's my configuration: i'm running varnish varnish-3.0.2-1.el5.x86_64 on centos. sub vcl_error { if (obj.status >= 500) { std.syslog(180, "V Err '" + req.url + "' '" + obj.response + "' '" + req.xid + "' [" + req.http.host + "] {" + req.backend + "}" ); } } req.backend prints the name of my director which is configured as follow: director director_islig round-robin { { .backend = islig01_local; } { .backend = islig02_local; } { .backend = islig03_local; } { .backend = web102; } { .backend = web103; } } backend web103 { .host = "192.168.1.13"; .probe = { .url = "/"; .interval = 60s; .timeout = 10 s; .window = 5; .threshold = 1; } } Many thanks Samuele From me at ibotty.net Mon Dec 26 18:16:39 2011 From: me at ibotty.net (tob) Date: Mon, 26 Dec 2011 19:16:39 +0100 Subject: Can varnish be used on a e-commerce website where almost all pages are dynamic? In-Reply-To: <53B0E90D-2F29-4604-8F9E-0541A6E26C2E@gmail.com> References: <53B0E90D-2F29-4604-8F9E-0541A6E26C2E@gmail.com> Message-ID: <4EF8BA07.8010405@ibotty.net> hi, as nobody more experienced than me bothered to answer up to now, i might just tell you some very basic things. most dynamic pages only have a small part of it, that really is dynamic. sometimes, that part can be loaded with ajax, so the first page to get is cached. if this does not work, edge side includes might provide an answer. that way you can assemble the dynamic pages on the server. check the wiki for details. good luck, tobi From perbu at varnish-software.com Tue Dec 27 09:40:13 2011 From: perbu at varnish-software.com (Per Buer) Date: Tue, 27 Dec 2011 10:40:13 +0100 Subject: Can varnish be used on a e-commerce website where almost all pages are dynamic? In-Reply-To: <53B0E90D-2F29-4604-8F9E-0541A6E26C2E@gmail.com> References: <53B0E90D-2F29-4604-8F9E-0541A6E26C2E@gmail.com> Message-ID: Hi, On Fri, Dec 23, 2011 at 12:26 PM, Lx Yu wrote: > Hello, I'm working on a e-commerce site, and the content of it is almost > fully dynamic. > > Most user must login to purchase, and every login user in have a > collection of personalized items displayed in every page. Yes. As someone already pointed out - mostly people use ESI for this. Or Javascript. And every item's status is in real-time(whether it's in stock, and how many > left). > That's a rather soft real time requirement. You can probably cache those for at least a second or two or ten. Nobody will know and your database will love it. So for now, we dynamically generate every webpage, and use a CDN to > delivery all static files(images, css, js). > > Then, can varnish be used in such situation where almost all things is > dynamic? Yes. -- Per Buer Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From lixinfish at gmail.com Tue Dec 27 10:23:18 2011 From: lixinfish at gmail.com (Lx Yu) Date: Tue, 27 Dec 2011 18:23:18 +0800 Subject: Can varnish be used on a e-commerce website where almost all pages are dynamic? In-Reply-To: References: <53B0E90D-2F29-4604-8F9E-0541A6E26C2E@gmail.com> Message-ID: <3DA09B57-D900-4446-8938-568019C1073D@gmail.com> Thank you very much! I'll go try for it. :) -- Lx Yu, Developer, ELEME Inc. Email: lixinfish at gmail.com On Dec 27, 2011, at 5:40 PM, Per Buer wrote: > Hi, > > On Fri, Dec 23, 2011 at 12:26 PM, Lx Yu wrote: > Hello, I'm working on a e-commerce site, and the content of it is almost fully dynamic. > > Most user must login to purchase, and every login user in have a collection of personalized items displayed in every page. > > Yes. As someone already pointed out - mostly people use ESI for this. Or Javascript. > > And every item's status is in real-time(whether it's in stock, and how many left). > > That's a rather soft real time requirement. You can probably cache those for at least a second or two or ten. Nobody will know and your database will love it. > > So for now, we dynamically generate every webpage, and use a CDN to delivery all static files(images, css, js). > > Then, can varnish be used in such situation where almost all things is dynamic? > > Yes. > > -- > > Per Buer > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > Varnish makes websites fly! > Whitepapers | Video | Twitter > > From bedis9 at gmail.com Tue Dec 27 17:56:16 2011 From: bedis9 at gmail.com (Baptiste) Date: Tue, 27 Dec 2011 18:56:16 +0100 Subject: Tune Varnish for pipe only operations In-Reply-To: References: Message-ID: Hi, I'd be keen to help you doing this with HAPRoxy. You could play with some ACLs to make haproxy swap from a http to tcp backend depending on your needs. That way, you could analyse your request then let the stream go. You can contact me offlist if you need some help. cheers On Tue, Dec 6, 2011 at 12:52 PM, Brain Stormer wrote: > Well, the back-end servers are some how breaking the HTTP protocol in some > way, also it is HTTP/0.9 only, it just not well support by HAPROXY unless > running it in binary proxying which won't give me the capability of doing > the initial security checks and restrictions over the negotiation packets, > Varnish was the only proxy tolerating all the protocol issues and still > working! > > So what you say, is Varnish cannot be tuned for doing such job efficiently ? > > > > Thanks, > MS > > > On Tue, Dec 6, 2011 at 10:09 AM, Simon Lyall wrote: >> >> >> I think varnish might not be the right tool for you, perhaps haproxy might >> better suit your needs. >> >> >> On Tue, 6 Dec 2011, Brain Stormer wrote: >>> >>> I am currently having Varnish deployed as a front-end proxy for a number >>> of >>> back-end servers (media streaming). >>> >>> The back-end servers only communicate with HTTP in the initialization and >>> then complete the session with binary streams. So basically, I do some >>> checks at the `vcl_recv` then if everything is okay I `pipe` them to the >>> back-ends. >>> >>> Currently, I have absolutely NO use from Varnish capabilities of caching. >>> >>> I am currently facing CPU overhead at the `varnishd` process, I have >>> tuned >>> lots of parts in Varnish, So... >>> >>> My question now is, How to tune Varnish to give full attention >>> performance >>> wise to piped requests, putting into mind that all sessions keeps up for >>> hours long. >>> >>> Another question is, Do I still have to define a caching store ? I read >>> on >>> some online resources that caching store is still needed for storage >>> regarding Varnish threads. >>> >>> Awaiting your?enlightening?! >>> >>> >>> >>> Thanks, >>> MS >>> >>> >> >> -- >> Simon Lyall ?| ?Very Busy ?| ?Web: http://www.darkmere.gen.nz/ >> "To stay awake all night adds a day to your life" - Stilgar | eMT. > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From Jeff.Stroomer at disney.com Tue Dec 27 18:22:46 2011 From: Jeff.Stroomer at disney.com (Stroomer, Jeff) Date: Tue, 27 Dec 2011 10:22:46 -0800 Subject: How does saint mode work when director is being used? Message-ID: <4F86EEFDF44D2D41951D491B61B7D43324A3686C53@SM-CALA-VXMB06B.swna.wdpr.disney.com> Varnish folks, How does saint mode work when a director is being used? The discussion here https://www.varnish-cache.org/trac/wiki/VCLExampleSaintMode mentions this case briefly, but doesn't have much detail. To make this more concrete, let's say I am using a round-robin director to distribute traffic across three back ends named A, B, and C, and suppose that in vcl_fetch I see a 500 response that came from A. If I set saint mode to 20 seconds on the response, does that mean that the object is blacklisted on A for 20 seconds, and therefore subsequent requests for that object go to B and C? And if saint mode subsequently also blacklists the object on B and C as well, does that mean that grace mode kicks in, and a cached copy is returned? Assuming I have this right, then it sounds like the backend health checks used by the director are a large-scale way to blacklist all requests for a back end, whereas saint mode is a small-scale object-by-object way to blacklist a back end. Thanks in advance for the help, Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From ljorg6 at gmail.com Wed Dec 28 14:12:45 2011 From: ljorg6 at gmail.com (=?ISO-8859-1?Q?Lars_J=F8rgensen?=) Date: Wed, 28 Dec 2011 15:12:45 +0100 Subject: Purge based on url postfix Message-ID: <4EFB23DD.9030108@gmail.com> Hi, We're having trouble getting our heavily customized backend to purge changed pages automatically from Varnish. As a workaround, I would like to enable CMS editors to call content with a postfixed string to purge it. Like this: www.website.com/index.html$PURGE That should purge index.html from the cache. I'm stumped on how to extract the original url from that request, though. I got this far: sub vcl_recv { if (req.url ~ "\$PURGE$") { $original_url = some magic code to strip "$PURGE" from req.url set req.x-purge = 1; set req.url = $original_url; } } sub vcl_hit { if (req.x-purge == 1) { purge; error 200 "Purged"; } } sub vcl_miss { if (req.x-purge == 1) { error 404 "Page not in cache"; } } Is this the right way to go about it, and can somebody supply the rather essential missing piece? -- Lars From kristian at varnish-software.com Thu Dec 29 10:01:26 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Thu, 29 Dec 2011 11:01:26 +0100 Subject: How does saint mode work when director is being used? In-Reply-To: <4F86EEFDF44D2D41951D491B61B7D43324A3686C53@SM-CALA-VXMB06B.swna.wdpr.disney.com> References: <4F86EEFDF44D2D41951D491B61B7D43324A3686C53@SM-CALA-VXMB06B.swna.wdpr.disney.com> Message-ID: <20111229100126.GA5758@freud.kly.no> Greetings, On Tue, Dec 27, 2011 at 10:22:46AM -0800, Stroomer, Jeff wrote: > To make this more concrete, let's say I am using a round-robin > director to distribute traffic across three back ends named A, B, and > C, and suppose that in vcl_fetch I see a 500 response that came from > A. If I set saint mode to 20 seconds on the response, does that mean > that the object is blacklisted on A for 20 seconds, and therefore > subsequent requests for that object go to B and C? And if saint > mode subsequently also blacklists the object on B and C as well, does > that mean that grace mode kicks in, and a cached copy is returned? Correct. > Assuming I have this right, then it sounds like the backend health > checks used by the director are a large-scale way to blacklist all > requests for a back end, whereas saint mode is a small-scale > object-by-object way to blacklist a back end. That's precisely it. I couldn't have said it better myself. - Kristian From lampe at hauke-lampe.de Thu Dec 29 11:35:57 2011 From: lampe at hauke-lampe.de (Hauke Lampe) Date: Thu, 29 Dec 2011 12:35:57 +0100 Subject: Purge based on url postfix In-Reply-To: <4EFB23DD.9030108@gmail.com> References: <4EFB23DD.9030108@gmail.com> Message-ID: <4EFC509D.9090206@hauke-lampe.de> On 28.12.2011 15:12, Lars J?rgensen wrote: > I'm stumped on how to extract the original url from that request, > though. regsub(str, regex, sub) Returns a copy of str with the first occurrence of the regular expression regex replaced with sub. Within sub, 0 (which can also be spelled &) is replaced with the entire matched string, and n is replaced with the contents of subgroup n in the matched string. https://www.varnish-cache.org/docs/3.0/reference/vcl.html#functions i.e. something like this should work: set req.url = regsub(req.url, "\$PURGE$", ""); > As a workaround, I would like to enable CMS editors to call content with a postfixed string to purge it. Like this: www.website.com/index.html$PURGE I use a special request header to force a cache miss and retrieve updated content from the backend: if (req.http.X-FreshForce == "yes") { # Force a cache miss set req.hash_always_miss = true; } CMS editors have a button in their browsers to enable the X-FreshForce header, e.g. by using http://www.garethhunt.com/modifyheaders/ or similar. Hauke -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: From ochs at marcochs.com Thu Dec 29 20:15:52 2011 From: ochs at marcochs.com (Marc Ochs) Date: Thu, 29 Dec 2011 15:15:52 -0500 Subject: RHEL5 build issue Message-ID: I'm having some issue building from souce where when I try to run make I get a bunch of errors like this: make[3]: Entering directory `/data/home/pkgbuild/varnish-3.0.2/lib/libvarnishcompat' if /bin/sh ../../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I../.. -I../../include -g -O2 -pthread -Wextra -Wno-missing-field-initializers -Wno-sign-compare -MT daemon.lo -MD -MP -MF ".deps/daemon.Tpo" -c -o daemon.lo daemon.c; \ then mv -f ".deps/daemon.Tpo" ".deps/daemon.Plo"; else rm -f ".deps/daemon.Tpo"; exit 1; fi ../../libtool: line 852: X--tag=CC: command not found ../../libtool: line 885: libtool: ignoring unknown tag : command not found ../../libtool: line 852: X--mode=compile: command not found ../../libtool: line 1018: *** Warning: inferring the mode of operation is deprecated.: command not found ../../libtool: line 1019: *** Future versions of Libtool will require --mode=MODE be specified.: command not found ../../libtool: line 1162: Xgcc: command not found ... Something with my echo or sed but I'm not sure... any help greatly appreciated. Sorry in advance if I'm missing something super obvious. Marc Full output: [pkgbuild at s605202nj2vl803 varnish-3.0.2]$ make cd . && /bin/sh /data/home/pkgbuild/varnish-3.0.2/missing --run autoheader rm -f stamp-h1 touch config.h.in cd . && /bin/sh ./config.status config.h config.status: creating config.h config.status: config.h is unchanged make all-recursive make[1]: Entering directory `/data/home/pkgbuild/varnish-3.0.2' Making all in include make[2]: Entering directory `/data/home/pkgbuild/varnish-3.0.2/include' make all-am make[3]: Entering directory `/data/home/pkgbuild/varnish-3.0.2/include' make[3]: Leaving directory `/data/home/pkgbuild/varnish-3.0.2/include' make[2]: Leaving directory `/data/home/pkgbuild/varnish-3.0.2/include' Making all in lib make[2]: Entering directory `/data/home/pkgbuild/varnish-3.0.2/lib' Making all in libvarnishcompat make[3]: Entering directory `/data/home/pkgbuild/varnish-3.0.2/lib/libvarnishcompat' if /bin/sh ../../libtool --tag=CC --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I. -I../.. -I../../include -g -O2 -pthread -Wextra -Wno-missing-field-initializers -Wno-sign-compare -MT daemon.lo -MD -MP -MF ".deps/daemon.Tpo" -c -o daemon.lo daemon.c; \ then mv -f ".deps/daemon.Tpo" ".deps/daemon.Plo"; else rm -f ".deps/daemon.Tpo"; exit 1; fi ../../libtool: line 852: X--tag=CC: command not found ../../libtool: line 885: libtool: ignoring unknown tag : command not found ../../libtool: line 852: X--mode=compile: command not found ../../libtool: line 1018: *** Warning: inferring the mode of operation is deprecated.: command not found ../../libtool: line 1019: *** Future versions of Libtool will require --mode=MODE be specified.: command not found ../../libtool: line 1162: Xgcc: command not found ../../libtool: line 1162: X-std=gnu99: command not found ../../libtool: line 1162: X-DHAVE_CONFIG_H: command not found ../../libtool: line 1162: X-I.: command not found ../../libtool: line 1162: X-I.: command not found ../../libtool: line 1162: X-I../..: No such file or directory ../../libtool: line 1162: X-I../../include: No such file or directory ../../libtool: line 1162: X-g: command not found ../../libtool: line 1162: X-O2: command not found ../../libtool: line 1162: X-pthread: command not found ../../libtool: line 1162: X-Wextra: command not found ../../libtool: line 1162: X-Wno-missing-field-initializers: command not found ../../libtool: line 1162: X-Wno-sign-compare: command not found ../../libtool: line 1162: X-MT: command not found ../../libtool: line 1162: Xdaemon.lo: command not found ../../libtool: line 1162: X-MD: command not found ../../libtool: line 1162: X-MP: command not found ../../libtool: line 1162: X-MF: command not found ../../libtool: line 1162: X.deps/daemon.Tpo: No such file or directory ../../libtool: line 1162: X-c: command not found ../../libtool: line 1214: Xdaemon.lo: command not found ../../libtool: line 1219: libtool: compile: cannot determine name of library object from `': command not found make[3]: *** [daemon.lo] Error 1 make[3]: Leaving directory `/data/home/pkgbuild/varnish-3.0.2/lib/libvarnishcompat' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/data/home/pkgbuild/varnish-3.0.2/lib' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/data/home/pkgbuild/varnish-3.0.2' make: *** [all] Error 2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From iradik at gmail.com Fri Dec 16 13:16:37 2011 From: iradik at gmail.com (Steve A) Date: Fri, 16 Dec 2011 13:16:37 -0000 Subject: generate a list of objects Message-ID: Hi, Is there any way to generate a list of object keys (host + url) that are present in a running varnish cache? thanks, steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From yamsergey at gmail.com Mon Dec 19 15:10:48 2011 From: yamsergey at gmail.com (=?KOI8-R?B?88XSx8XKIPHN3cnLz9c=?=) Date: Mon, 19 Dec 2011 18:10:48 +0300 Subject: Varnish 3.0.1 Backend conditional requests Message-ID: Hello. How to enable Backend conditional requests (If-None-Match and If-Modified-Since) for the Varnish 3.0.1. Now it doesn't send to me headers (If-None-Match and If-Modified-Since) when max-age=0 or expired, on first response i set ETag and Last-Modified headers. Subs and stale_obj from https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests throw exception in "*.vcl" file when compile. From yamsergey at gmail.com Mon Dec 19 15:26:23 2011 From: yamsergey at gmail.com (=?KOI8-R?B?88XSx8XKIPHN3cnLz9c=?=) Date: Mon, 19 Dec 2011 18:26:23 +0300 Subject: Varnish 3.0.1 Backend conditional requests In-Reply-To: References: Message-ID: Hello. How to enable Backend conditional requests (If-None-Match and If-Modified-Since) for the Varnish 3.0.1. Now it doesn't send to me headers (If-None-Match and If-Modified-Since) when max-age=0 or expired, on first response i set ETag and Last-Modified headers. Subs and stale_obj from https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests throw exception in "*.vcl" file when compile. From dbenavid at terra.cl Wed Dec 21 21:29:31 2011 From: dbenavid at terra.cl (Daniel Benavides) Date: Wed, 21 Dec 2011 18:29:31 -0300 Subject: Problem with Virtual Host Message-ID: <4EF24FBB.3040507@terra.cl> mrs. I'm cashing server, if backend page not is virtual host, the cashing working. but the backend page is a virtual host, the cashing not working I can not modify de webservers. example ---working backend default { .host = "www.sgi.com"; .port = "80"; } ---- Not working backend default { .host = "www.omegasystems.cl"; .port = "80"; } Please help me