From tietje at topconcepts.com Fri Aug 1 15:03:30 2008 From: tietje at topconcepts.com (Sven Tietje) Date: Fri, 1 Aug 2008 17:03:30 +0200 Subject: two questions Message-ID: <000a01c8f3e7$c3978060$9e00a8c0@stade.topconcepts.net> hi, two questions - perhas quit simple, but somehow i got into trouble with it: aim: I want to cache all image and css - files. It works fine, but: 1. for example: I open /blah.html. Vvrnish caches all necessary files. if I reload /blah.html, images and css will be served out of varnish cache. then I open /blub.html. everything works fine. Now, I click again on the link to /blah.html. all images and files are now requested from the backend. varnish says: "cache miss". why? 2. Can I tell varnish to ignore the max-age sent by the backend? The user should receive max-age=0, but I want varnish to get all images and css into cache and to serve it. I do not want to cache the browser any data and I do not want to purge data from cache automatically by varnish. If somethings is changed in the content management system varnish will be told to purge the relevant urls. my version: varnishd (varnish-1.2) thanks & nice weekend sven From damien.sarazin at smile.fr Mon Aug 4 09:43:11 2008 From: damien.sarazin at smile.fr (Damien Sarazin) Date: Mon, 04 Aug 2008 11:43:11 +0200 Subject: Strange problem with url redirecting when clicking on a link Message-ID: <4896CF2F.3030209@smile.fr> Hello, i replaced Squid by Varnish on web sites that are in production (like a lot of people here :)), and i haven't had any problems with those until the last one. The issue is that when the client clicked on a link, the url was redirect towards the port 81 and given that varnish is listening on the port 80 the pages were not normally displayed. I don't understand how this could happen since when i was browsing myself i never had the same problem. However i looked in the varnishlog and here is the result : varnishncsa -c -r /var/log/varnish/diagnostic-port | grep ":81" 213.56.83.241 - - [01/Aug/2008:14:27:46 +0200] "GET http://www.website.com:81/football/fil-info/brann-ou-ventspils-pour-l-om-178141/ HTTP/1.1" 200 92149 "http://www.website.com/football/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)" 213.56.83.241 - - [01/Aug/2008:14:29:12 +0200] "GET http://www.website.com:81/football HTTP/1.1" 301 239 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)" 213.56.83.241 - - [01/Aug/2008:14:29:12 +0200] "GET http://www.website.com:81/football/ HTTP/1.1" 200 134824 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)" Apache is listening on the same OS on the port 81, but transactions are not allowed on this port by the firewall. (The firewall is not on this computer). i am using the trunk version 2945. Varnish is listening on the 2 interfaces of the computer. The default file looks like this : VARNISH_LISTEN_ADDRESS= VARNISH_LISTEN_PORT=80 VARNISH_ADMIN_LISTEN_ADDRESS=localhost VARNISH_ADMIN_LISTEN_PORT=6082 VARNISH_MIN_THREADS=1 VARNISH_MAX_THREADS=1000 VARNISH_THREAD_TIMEOUT=120 VARNISH_STORAGE_FILE=/var/varnish/$INSTANCE/varnish_storage.bin VARNISH_STORAGE_SIZE=5G VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}" VARNISH_TTL=0 The VCL file : backend default { .host = "localhost"; .port = "81"; } sub vcl_recv { remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip; set req.backend = default; if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } lookup; } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { pipe; } if (req.url ~ "\.(css|gif|jpg|jpeg|png|ico|swf|js)$") { lookup; } if (req.http.Expect) { pipe; } if (req.http.Authenticate) { pass; } if (req.request != "GET" && req.request != "HEAD") { pass; } if (req.http.Cache-Control ~ "no-cache") { pass; } set req.grace = 30s; lookup; } sub vcl_hash { set req.hash += req.url; set req.hash += req.http.host; hash; } sub vcl_hit { if (!obj.cacheable) { pass; } if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } set obj.http.X-Cache = "HIT"; deliver; } sub vcl_fetch { if (!obj.valid) { error; } if (!obj.cacheable) { pass; } if (obj.http.Set-Cookie) { pass; } if (obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "(no-cache|no-store|private)") { pass; } set obj.grace if (req.http.Authorization && !obj.http.Cache-Control ~ "public") { pass; } insert; } sub vcl_deliver { set resp.http.Via = "Varnish"; deliver; } I have the same configuration, except for the "VARNISH_LISTEN_ADDRESS=" line, on other sites and this url redirecting on port 81 never happened before. And as i said before, while i was browsing myself on this website, i didn't encounter this issue. I don't know if it is random, if it comes from the browser of the client, from the loadbalancer, from the max_threads or from another thing. Have you any idea about this ? Before trying to add a rule that searches for ":81" and remove it in the url, i wanted to have your opinion and know if someone had already seen this problem. Best regards, -- Damien Sarazin From Phuwadon at sanookonline.co.th Mon Aug 4 09:43:53 2008 From: Phuwadon at sanookonline.co.th (Phuwadon Danrahan) Date: Mon, 4 Aug 2008 16:43:53 +0700 Subject: Question about purge_url & purge_hash In-Reply-To: References: Message-ID: Hi all, I just found the way to do purge_hash in vcl. Try it yourself, it supports regular expression in both req.url & req.http.host. VCL_RECV if (req.request == "PURGE") { if (client.ip ~ backoffice) { set req.http.x = req.url "#" req.http.host "#"; purge_hash(req.http.x); error 200 "HASHPURGED"; } } -----Original Message----- From: varnish-misc-bounces at projects.linpro.no [mailto:varnish-misc-bounces at projects.linpro.no] On Behalf Of Phuwadon Danrahan Sent: Monday, July 28, 2008 5:29 PM To: varnish-misc at projects.linpro.no Subject: Question about purge_url & purge_hash Hi all, I configured the Varnish 1.1.2-r2635 to allow purging from backoffice web servers. The first version of this concept is very easy, just send PURGE request for specific URL to Varnish servers. Configuraiton, VCL_RECV if (req.request == "PURGE") { if (client.ip ~ backoffice) { lookup; } } VCL_HIT if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged"; } This above configuration is working fine but the application owner would like to purge some URL that contains multiple paging in only one PURGE request. The example is http://myhost.com/path/to/index.php?id=123&page=1 http://myhost.com/path/to/index.php?id=123&page=2 http://myhost.com/path/to/index.php?id=123&page=3 Sometimes, the maximum page can be more than 1000. So, we think about purge_url(req.http) but it always return MISS when we input the URL with REGEX. What did we test the VCL configuration 1. use WFETCH to get content from varnish with the following request headers, GET /path/to/file.gif HTTP/1.0\r\n Host: myhost.com\r\n Accept-Encoding: gzip\r\n 2. use WFETCH to purge content from varnish with the following request headers, PURGE /path/to/file.gif HTTP/1.0\r\n Host: myhost.com\r\n Accept-Encoding: gzip\r\n We can purged the object in this step 3. We tried, PURGE \.gif$ HTTP/1.0\r\n Host: myhost.com\r\n Accept-Encoding: gzip\r\n But this one can not be purged which we require this kind of purging style (allow REGEX) Could anyone provide the valid use of purge_url or purge_hash in both vcl_recv and vcl_hit? and how can we input the REGEX url into purging request? Thank you. Phuwadon D. _______________________________________________ varnish-misc mailing list varnish-misc at projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc From phk at phk.freebsd.dk Mon Aug 4 09:55:05 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 04 Aug 2008 09:55:05 +0000 Subject: Strange problem with url redirecting when clicking on a link In-Reply-To: Your message of "Mon, 04 Aug 2008 11:43:11 +0200." <4896CF2F.3030209@smile.fr> Message-ID: <9802.1217843705@critter.freebsd.dk> In message <4896CF2F.3030209 at smile.fr>, Damien Sarazin writes: >The issue is that when the client clicked on a link, the url was >redirect towards the port 81 and given that varnish is listening on the >port 80 the pages were not normally displayed. That sounds utterly impossible. Is there any chance that the client has server:80 configured as HTTP proxy ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From damien.sarazin at smile.fr Mon Aug 4 10:06:03 2008 From: damien.sarazin at smile.fr (Damien Sarazin) Date: Mon, 04 Aug 2008 12:06:03 +0200 Subject: Strange problem with url redirecting when clicking on a link In-Reply-To: <9802.1217843705@critter.freebsd.dk> References: <9802.1217843705@critter.freebsd.dk> Message-ID: <4896D48B.3020703@smile.fr> Hi again, i called the client, and he told me that he hasn't configured anything on his firefox. So it seems like it doesn't come from an HTTP proxy setting. Poul-Henning Kamp wrote: > In message <4896CF2F.3030209 at smile.fr>, Damien Sarazin writes: > > >> The issue is that when the client clicked on a link, the url was >> redirect towards the port 81 and given that varnish is listening on the >> port 80 the pages were not normally displayed. >> > > That sounds utterly impossible. > > Is there any chance that the client has server:80 configured as > HTTP proxy ? > > -- Damien Sarazin Administration syst?me Smile ? Motoristes Internet www.smile.fr Tel : 04 99 77 20 17 Fax : 04 67 58 25 14 Mailto : damien.sarazin at smile.fr From lukas.loesche at bertelsmann.de Mon Aug 4 10:20:51 2008 From: lukas.loesche at bertelsmann.de (Loesche, Lukas, ZD-Scoyo) Date: Mon, 04 Aug 2008 12:20:51 +0200 Subject: Strange problem with url redirecting when clicking on a link In-Reply-To: <4896D48B.3020703@smile.fr> Message-ID: Hi, > i called the client, and he told me that he hasn't configured anything > on his firefox. So it seems like it doesn't come from an HTTP proxy setting. you could try to grep the website's source code for "SERVER_PORT" or equivalent, depending on the language the site is written in. I'm quite certain that this problem is unrelated to varnish. My guess is, that on one or more pages of your webapp something like dynamic linking is in use. In some situations, like when you have links or images from an external host and your using the same pages for http and https it makes sense to check which protocol or port the client used, to modify the links accordingly. So I'd guess that instead of using relative linking that one or two pages are building them using environment variables like the SERVER_PORT. As you said, the local apache is running on Port 81 so it doesn't look incidental. On the other hand, might also be something completely different :-) Cheers, -- Lukas From phk at phk.freebsd.dk Mon Aug 4 13:12:43 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 04 Aug 2008 13:12:43 +0000 Subject: Strange problem with url redirecting when clicking on a link In-Reply-To: Your message of "Mon, 04 Aug 2008 12:06:03 +0200." <4896D48B.3020703@smile.fr> Message-ID: <2225.1217855563@critter.freebsd.dk> In message <4896D48B.3020703 at smile.fr>, Damien Sarazin writes: >Hi again, > >i called the client, and he told me that he hasn't configured anything >on his firefox. So it seems like it doesn't come from an HTTP proxy setting. I can imagine no way a tcp connection for port 81 ends up on port 80 unless somebody does the proxything along the way. But once it does, there is nothing varnish can do about it, running on port 80. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From damien.sarazin at smile.fr Mon Aug 4 13:27:11 2008 From: damien.sarazin at smile.fr (Damien Sarazin) Date: Mon, 04 Aug 2008 15:27:11 +0200 Subject: Strange problem with url redirecting when clicking on a link In-Reply-To: <2225.1217855563@critter.freebsd.dk> References: <2225.1217855563@critter.freebsd.dk> Message-ID: <489703AF.1040107@smile.fr> Yeah, i agree with you, i don't understand how a request addressed to port 81 can be logged and processed by varnish which listens on port 80. Apache should receive this ! Anyway, thanks for your feedback. I'll try to contact the guys who are in charge of the loadbalancing and of the routing. Poul-Henning Kamp wrote: > In message <4896D48B.3020703 at smile.fr>, Damien Sarazin writes: > >> Hi again, >> >> i called the client, and he told me that he hasn't configured anything >> on his firefox. So it seems like it doesn't come from an HTTP proxy setting. >> > > I can imagine no way a tcp connection for port 81 ends up on port > 80 unless somebody does the proxything along the way. > > But once it does, there is nothing varnish can do about it, running > on port 80. > > -- Damien Sarazin Administration syst?me Smile ? Motoristes Internet www.smile.fr Tel : 04 99 77 20 17 Fax : 04 67 58 25 14 Mailto : damien.sarazin at smile.fr From phk at phk.freebsd.dk Mon Aug 4 13:29:38 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 04 Aug 2008 13:29:38 +0000 Subject: Strange problem with url redirecting when clicking on a link In-Reply-To: Your message of "Mon, 04 Aug 2008 15:27:11 +0200." <489703AF.1040107@smile.fr> Message-ID: <2311.1217856578@critter.freebsd.dk> In message <489703AF.1040107 at smile.fr>, Damien Sarazin writes: >Yeah, i agree with you, i don't understand how a request addressed to = >port 81 can be logged and processed by varnish which listens on port 80. = >Apache should receive this ! >Anyway, thanks for your feedback. I'll try to contact the guys who are = >in charge of the loadbalancing and of the routing. Consider if you could catch it in flagante delicto with snort or tcpdump ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From damien.sarazin at smile.fr Mon Aug 4 14:17:26 2008 From: damien.sarazin at smile.fr (Damien Sarazin) Date: Mon, 04 Aug 2008 16:17:26 +0200 Subject: Strange problem with url redirecting when clicking on a link In-Reply-To: <2311.1217856578@critter.freebsd.dk> References: <2311.1217856578@critter.freebsd.dk> Message-ID: <48970F76.5040707@smile.fr> Here is another lead that might be near the truth ! I am not allowed to do tests on the website until tomorrow morning since there is too much traffic atm. But i'll try to set UseCanonicalName Off in apache conf and to catch paquets with tcpdump. I'll tell you if it resolved the problem. > Hi Damien > > I've seen this happen where you have a bad link that say points to > "http://200.200.200.200/index.html" - Apache will rewrite it to > "http://my.fqdn.com:81/index.html" - where my.fqdn.com:81 come from the > Apache config. > > I guess there is an option in apache to turn that behaviour off - or > alternatively look through your hosted pages for links that look "odd"? > > Cheers > > Alex > > -- Alex Harrington - Network Development Manager Longhill High School > t: 01273 391672 e: alex at longhill.org.uk Poul-Henning Kamp wrote: > In message <489703AF.1040107 at smile.fr>, Damien Sarazin writes: > > >> Yeah, i agree with you, i don't understand how a request addressed to = >> port 81 can be logged and processed by varnish which listens on port 80. = >> Apache should receive this ! >> Anyway, thanks for your feedback. I'll try to contact the guys who are = >> in charge of the loadbalancing and of the routing. >> > > Consider if you could catch it in flagante delicto with snort or > tcpdump ? > > From varnish-misc at projects.linpro.no Wed Aug 6 07:32:04 2008 From: varnish-misc at projects.linpro.no (varnish-misc at projects.linpro.no) Date: Wed, 6 Aug 2008 09:32:04 +0200 (CEST) Subject: Angelina Jolie Free Video. Message-ID: <20080806143202.4170.qmail@p2p-90-36-r.line1.ru> An HTML attachment was scrubbed... URL: From anders at fupp.net Wed Aug 6 14:18:24 2008 From: anders at fupp.net (Anders Nordby) Date: Wed, 6 Aug 2008 16:18:24 +0200 Subject: redirect in varnish In-Reply-To: <9571.1209419236@critter.freebsd.dk> References: <9571.1209419236@critter.freebsd.dk> Message-ID: <20080806141824.GA84142@fupp.net> Hi, On Mon, Apr 28, 2008 at 09:47:16PM +0000, Poul-Henning Kamp wrote: > if (req.http.host == "www.bla.com") { > set req.http.host = "www.bla1.com"; > } Trying a VCL like this (this is my entire VCL, used as an example), I do not get a redirect: backend default { .host = "192.168.0.1"; .port = "80"; } sub vcl_recv { set req.http.host = "www.foobar.no"; set req.http.url = "/"; lookup; } Instead of doing a redirect, it fetches pages off the default backend. Why? How do I redirect to another URL on a different host, not served by Varnish? PS: Another thing, the first backend seems to become the default one, even though there is no backend called "default". Shouldn't it respond with 503 Service Unavailable instead? I like to control what is default and not. Cheers, -- Anders. From phk at phk.freebsd.dk Wed Aug 6 14:28:01 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 06 Aug 2008 14:28:01 +0000 Subject: redirect in varnish In-Reply-To: Your message of "Wed, 06 Aug 2008 16:18:24 +0200." <20080806141824.GA84142@fupp.net> Message-ID: <25956.1218032881@critter.freebsd.dk> In message <20080806141824.GA84142 at fupp.net>, Anders Nordby writes: >Hi, > >On Mon, Apr 28, 2008 at 09:47:16PM +0000, Poul-Henning Kamp wrote: >> if (req.http.host == "www.bla.com") { >> set req.http.host = "www.bla1.com"; >> } > >Trying a VCL like this (this is my entire VCL, used as an example), I do >not get a redirect: > >backend default { > .host = "192.168.0.1"; > .port = "80"; >} > >sub vcl_recv { > set req.http.host = "www.foobar.no"; > set req.http.url = "/"; > lookup; >} > >Instead of doing a redirect, it fetches pages off the default backend. >Why? How do I redirect to another URL on a different host, not served by >Varnish? You need to define that host as a backend. >PS: Another thing, the first backend seems to become the default one, >even though there is no backend called "default". Shouldn't it respond >with 503 Service Unavailable instead? I like to control what is default >and not. No, the first backend in the VCL is your default, no matter what it is called. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Aug 6 14:29:03 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 06 Aug 2008 14:29:03 +0000 Subject: redirect in varnish In-Reply-To: Your message of "Wed, 06 Aug 2008 14:28:01 GMT." <25956.1218032881@critter.freebsd.dk> Message-ID: <25985.1218032943@critter.freebsd.dk> In message <25956.1218032881 at critter.freebsd.dk>, "Poul-Henning Kamp" writes: >>PS: Another thing, the first backend seems to become the default one, >>even though there is no backend called "default". Shouldn't it respond >>with 503 Service Unavailable instead? I like to control what is default >>and not. > >No, the first backend in the VCL is your default, no matter what it >is called. Actually, using the one named "default" as default is not a bad idea. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From anders at fupp.net Wed Aug 6 14:57:33 2008 From: anders at fupp.net (Anders Nordby) Date: Wed, 6 Aug 2008 16:57:33 +0200 Subject: redirect in varnish In-Reply-To: <25956.1218032881@critter.freebsd.dk> References: <20080806141824.GA84142@fupp.net> <25956.1218032881@critter.freebsd.dk> Message-ID: <20080806145732.GA85993@fupp.net> Hi, On Wed, Aug 06, 2008 at 02:28:01PM +0000, Poul-Henning Kamp wrote: >>Instead of doing a redirect, it fetches pages off the default backend. >>Why? How do I redirect to another URL on a different host, not served by >>Varnish? > You need to define that host as a backend. Did that (and changed req.http.url to req.url - my mistake). This makes Varnish fetch the content on that url/host, but instead of redirecting (301/302 http code) it just fetches and returns the content (200 http code). I want to redirect, not just fetch the content. Also it would be nice to specify 301 or 302 http code for the redirect. Bye, -- Anders. From phk at phk.freebsd.dk Wed Aug 6 16:05:24 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 06 Aug 2008 16:05:24 +0000 Subject: redirect in varnish In-Reply-To: Your message of "Wed, 06 Aug 2008 16:57:33 +0200." <20080806145732.GA85993@fupp.net> Message-ID: <26315.1218038724@critter.freebsd.dk> In message <20080806145732.GA85993 at fupp.net>, Anders Nordby writes: >Hi, > >On Wed, Aug 06, 2008 at 02:28:01PM +0000, Poul-Henning Kamp wrote: >>>Instead of doing a redirect, it fetches pages off the default backend. >>>Why? How do I redirect to another URL on a different host, not served by >>>Varnish? >> You need to define that host as a backend. > >Did that (and changed req.http.url to req.url - my mistake). This makes >Varnish fetch the content on that url/host, but instead of redirecting >(301/302 http code) it just fetches and returns the content (200 http >code). I want to redirect, not just fetch the content. Also it would be >nice to specify 301 or 302 http code for the redirect. error 301; Set the headers in vcl_error{} -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From armdan20 at gmail.com Thu Aug 7 10:22:51 2008 From: armdan20 at gmail.com (andan andan) Date: Thu, 7 Aug 2008 12:22:51 +0200 Subject: Issue related to ticket 278 Message-ID: Hi. We are experimenting exactly the same issue related to ticket 278. In the case that the URL is involved with the problem, the failed URL is /blogs_opinion/ We have two Varnish, and both presents the problem, when we restart them, varnish servers back to normal. Thanks in advance. Best Regards. From armdan20 at gmail.com Thu Aug 7 10:49:06 2008 From: armdan20 at gmail.com (andan andan) Date: Thu, 7 Aug 2008 12:49:06 +0200 Subject: Issue related to ticket 278 In-Reply-To: References: Message-ID: Sorrry, I forgot give data: Varnish trunk rev 2980 RHEL 5.2 x86_64 Thanks. 2008/8/7 andan andan : > Hi. > > We are experimenting exactly the same issue related to ticket 278. > > In the case that the URL is involved with the problem, the failed URL > is /blogs_opinion/ > > We have two Varnish, and both presents the problem, when we restart > them, varnish servers back to normal. > > Thanks in advance. > Best Regards. > From damien.sarazin at smile.fr Thu Aug 7 13:32:45 2008 From: damien.sarazin at smile.fr (Damien Sarazin) Date: Thu, 07 Aug 2008 15:32:45 +0200 Subject: Strange problem with url redirecting when clicking on a link In-Reply-To: <48970F76.5040707@smile.fr> References: <2311.1217856578@critter.freebsd.dk> <48970F76.5040707@smile.fr> Message-ID: <489AF97D.3090604@smile.fr> Hi, i finally succeeded in finding the origin of the problem and in reproducting it on a test website. It comes from rewriterules in the apache configuration. First i noticed in firebug that the url with ":81" had a code "301" but there was not the redirected url with code "200". The rewriterules rules were written like this : RewriteRule ^/test.html / [R=301,L] For this rewriterule to work with Varnish you have to modify it like that : RewriteRules ^/test.html http://www.website.com/ [R=301,L] Else if you don't want to modify your Apache configuration, you can try to add this in your vcl file : In sub vcl_fetch { if (obj.http.location ~ ":81"){ set obj.http.location = regsub(obj.http.location,"\:81",""); } } Best Regards, Damien S. From ianevans at digitalhit.com Fri Aug 8 17:27:38 2008 From: ianevans at digitalhit.com (Ian M. Evans) Date: Fri, 08 Aug 2008 13:27:38 -0400 Subject: Ignoring cookies except for one area Message-ID: <489C820A.90803@digitalhit.com> Thinking of using Varnish to handle the spikes we getting during some big news events. Besides our advertisers, we currently only use cookies to stop ballot stuffing for our online polls. The only problem is that our cookies are set for "/" because even though this looks like a directory path: /poll/vote/173 /poll is actually an extensionless PHP script and /vote/173 its pathinfo. Looking at some VCL examples, am I correct in assuming that I could just set Varnish to strip the cookie if the url doesn't not contain /poll? Would that be: if (req.url ! [absolutely no idea since I suck at regex]) { remove req.http.cookie; lookup; } Thanks! From ask at develooper.com Fri Aug 8 17:59:01 2008 From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Fri, 8 Aug 2008 10:59:01 -0700 Subject: Strange problem with url redirecting when clicking on a link In-Reply-To: <489AF97D.3090604@smile.fr> References: <2311.1217856578@critter.freebsd.dk> <48970F76.5040707@smile.fr> <489AF97D.3090604@smile.fr> Message-ID: On Aug 7, 2008, at 6:32 AM, Damien Sarazin wrote: > The rewriterules rules were written like this : > > RewriteRule ^/test.html / [R=301,L] > > For this rewriterule to work with Varnish you have to modify it like > that : > > RewriteRules ^/test.html http://www.website.com/ [R=301,L] It sounds like you have Apache configured with Port 81 When what you wanted to have was Listen 81 Port 80 - ask -- http://develooper.com/ - http://askask.com/ From ask at develooper.com Fri Aug 8 18:00:21 2008 From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Fri, 8 Aug 2008 11:00:21 -0700 Subject: Ignoring cookies except for one area In-Reply-To: <489C820A.90803@digitalhit.com> References: <489C820A.90803@digitalhit.com> Message-ID: <29D31253-FDD2-4B4F-B584-39292E66C6D7@develooper.com> On Aug 8, 2008, at 10:27 AM, Ian M. Evans wrote: > The only problem is that our cookies are set for "/" because even > though > this looks like a directory path: > > /poll/vote/173 > > /poll is actually an extensionless PHP script and /vote/173 its > pathinfo. The cookies don't care how the script is picked - you could still set them for /poll/ - ask -- http://develooper.com/ - http://askask.com/ From ianevans at digitalhit.com Fri Aug 8 18:50:27 2008 From: ianevans at digitalhit.com (Ian M. Evans) Date: Fri, 08 Aug 2008 14:50:27 -0400 Subject: Ignoring cookies except for one area In-Reply-To: <29D31253-FDD2-4B4F-B584-39292E66C6D7@develooper.com> References: <489C820A.90803@digitalhit.com> <29D31253-FDD2-4B4F-B584-39292E66C6D7@develooper.com> Message-ID: <489C9573.5030000@digitalhit.com> Ask Bj?rn Hansen wrote: > The cookies don't care how the script is picked - you could still set > them for /poll/ Great thanks! From ianevans at digitalhit.com Fri Aug 8 19:20:19 2008 From: ianevans at digitalhit.com (Ian M. Evans) Date: Fri, 08 Aug 2008 15:20:19 -0400 Subject: Ads, Analytics and cookies... Message-ID: <489C9C73.5000208@digitalhit.com> Okay, well, I thought I was doing the cache friendly thing by moving my poll only cookie to the /poll/ path (Re: Ignoring cookies except for one area) That worked, but after checking out my cookies in Firefox, I see that Google Analytics and the Openx Ad server toss a bunch of cookies in '/' for my domain. So of course a cookie is attached to each page, image, javascript and CSS file. Since I assume that some of the main sites in need of Varnish would be the high-traffic, digg-friendly, ad-supported, Analtyics-using sites what's the way around this? From what I've been reading I can use the VCL rules to strip the cookies off the static stuff like .css and .jpg, and .js...but what about the dynamically-created pages? And how do I know where cookie stripping might hurt ad delivery? I didn't see anything about dealing with ad servers in the FAQ, so I thought I'd ask. From jnerin+varnish at gmail.com Fri Aug 8 21:49:00 2008 From: jnerin+varnish at gmail.com (=?UTF-8?Q?Jorge_Ner=C3=ADn?=) Date: Fri, 8 Aug 2008 23:49:00 +0200 Subject: Ads, Analytics and cookies... In-Reply-To: <30df48470808081443o63035d04ka0debc24c3cc8083@mail.gmail.com> References: <489C9C73.5000208@digitalhit.com> <30df48470808081443o63035d04ka0debc24c3cc8083@mail.gmail.com> Message-ID: <30df48470808081449x3496268dnaa8972c47d55b5dc@mail.gmail.com> As far as I know if the backend does not change the object requested dynamically (page source code, image, etc) using conditions that depends on the value of cookies it is safe to be cached in varnish (or a public proxy). You will (should) not be using third party cookies to generate different contents, and the google analytics server and the Openx Ad server will deffinetly be generating different content using the cookies. But the parsing code is a javascript included in your page but (this is important) served by their servers, so it can never be cached by varnish because the client browser ask for it directly to the third party servers. Greetings, hope it makes sense. On Fri, Aug 8, 2008 at 21:20, Ian M. Evans wrote: > Okay, well, I thought I was doing the cache friendly thing by moving my > poll only cookie to the /poll/ path (Re: Ignoring cookies except for one > area) > > That worked, but after checking out my cookies in Firefox, I see that > Google Analytics and the Openx Ad server toss a bunch of cookies in '/' > for my domain. So of course a cookie is attached to each page, image, > javascript and CSS file. > > Since I assume that some of the main sites in need of Varnish would be > the high-traffic, digg-friendly, ad-supported, Analtyics-using sites > what's the way around this? > > From what I've been reading I can use the VCL rules to strip the > cookies off the static stuff like .css and .jpg, and .js...but what > about the dynamically-created pages? > > And how do I know where cookie stripping might hurt ad delivery? > > I didn't see anything about dealing with ad servers in the FAQ, so I > thought I'd ask. > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > -- Jorge Ner?n -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianevans at digitalhit.com Fri Aug 8 22:03:35 2008 From: ianevans at digitalhit.com (Ian M. Evans) Date: Fri, 08 Aug 2008 18:03:35 -0400 Subject: Ads, Analytics and cookies... In-Reply-To: <30df48470808081443o63035d04ka0debc24c3cc8083@mail.gmail.com> References: <489C9C73.5000208@digitalhit.com> <30df48470808081443o63035d04ka0debc24c3cc8083@mail.gmail.com> Message-ID: <489CC2B7.4080501@digitalhit.com> Jorge Ner?n wrote: >But the parsing code is a javascript included in your page but (this is >important) served by their servers, so it can never be cached by >varnish because the client browser ask for it directly to the third >party servers. Two problems... 1) The OpenX ad server is a server you run ON YOUR OWN SITE, so even though the ads may be coming from another network (setting its own cookies) the OpenX server cookies are generated by your own domain and not by a third party server. It's used by sites who want to sell ads directly or geotarget certain ad networks to say North American or EU visitors. As I'm reading more about cookies today, I'm assuming having a CNAME of, say, an "ads" subdomain would keep the cookies off the "www." stuff. I'll have to check with the OpenX people. 2) The Google Analytics cookies code also sets cookies on your own domain. Reading more about cookies today than I have in the decade of my site. I'm a writer/photographer who has to toss on the sysadmin hat every few days. :-) From kova70 at gmail.com Sat Aug 9 01:59:12 2008 From: kova70 at gmail.com (Raymond Hall) Date: Fri, 8 Aug 2008 20:59:12 -0500 Subject: varnishncsa and virtual hosts In-Reply-To: <20080725131803.GA41345@fupp.net> References: <53dee190807250244x464de7dr99d492e77d27f29d@mail.gmail.com> <20080725131803.GA41345@fupp.net> Message-ID: <53dee190808081859w6e028464i5388c353148b8f1e@mail.gmail.com> A pretty simple hack to varnishncsa did the trick nicely: diff -ru varnish-2.0.tp2/bin/varnishncsa/varnishncsa.c varnish-2.0.tp2.hack/bin/varnishncsa/varnishncsa.c --- varnish-2.0.tp2/bin/varnishncsa/varnishncsa.c 2008-07-05 16:41:36.000000000 -0500 +++ varnish-2.0.tp2.hack/bin/varnishncsa/varnishncsa.c 2008-07-22 07:07:03.000000000 -0500 @@ -424,6 +424,15 @@ strftime(tbuf, sizeof tbuf, "[%d/%b/%Y:%T %z]", &lp->df_t); fprintf(fo, "%s ", tbuf); + if (lp->df_Host) { + if (strncmp(lp->df_Host, "http://", 7) == 0) { + fprintf(fo, lp->df_Host + 7); + } else { + fprintf(fo, lp->df_Host); + } + fprintf(fo, " "); + } + /* * Fake "%r". This would be a lot easier if Varnish * normalized the request URL. This makes varnishncsa to log the host header separately, just after the date sting. That way you can tell AWstats to use that field for %virtualname It's ugly but less uglier than the other workarounds I've seen :) Regards, Raymond Hall On Fri, Jul 25, 2008 at 8:18 AM, Anders Nordby wrote: > Hi, > > I'm using an ugly cron-based work-aroud. ;-) Seems to work ok. But yes, > I'd appreciate very much if we could write separate logs for different > virtual hosts. Preferably, even without running one instance of > varnishncsa for every log. > > On Fri, Jul 25, 2008 at 04:44:28AM -0500, Raymond Hall wrote: >> I've run through pretty much all posts regarding separating >> varnishncsa's output per hosts, as requested in RxHeader. >> So far, no one has posted a reliable solution ... if only varnishlog >> accepted both -o and -w ... or varnishncsa RxHeader worked... :) >> No use redirecting varnishlog -o -c RxHeader '^Host: www.example.com' >> to a fifo and ther asking varnishncsa to read from it. >> >> Has anyone found a reliable solution to this? (ie. other than >> resorting to ugly cron-based workarounds....) >> I'm using varnish-2.0-tp2 >> >> TIA, >> Ray >> >> -- >> I'd rather have an inch of a dog than miles of pedigree. >> >> Dana Burnet >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc > > > -- > Anders. > -- I'd rather have an inch of a dog than miles of pedigree. Dana Burnet From kova70 at gmail.com Sat Aug 9 02:11:10 2008 From: kova70 at gmail.com (Raymond Hall) Date: Fri, 8 Aug 2008 21:11:10 -0500 Subject: Ads, Analytics and cookies... In-Reply-To: <489CC2B7.4080501@digitalhit.com> References: <489C9C73.5000208@digitalhit.com> <30df48470808081443o63035d04ka0debc24c3cc8083@mail.gmail.com> <489CC2B7.4080501@digitalhit.com> Message-ID: <53dee190808081911i3ca53ebaxff91fba66c9a2a2@mail.gmail.com> On Fri, Aug 8, 2008 at 5:03 PM, Ian M. Evans wrote: > Two problems... > > 1) The OpenX ad server is a server you run ON YOUR OWN SITE, so even Don't route the adserver through varnish, use a different IP & domain for that, or if you _need_ to go through varnish, at least use a different domain so you can skip the lookup and pass the request based on host header. > 2) The Google Analytics cookies code also sets cookies on your own domain. Yes, and if you leave the default rule that pass'ses all requests having a cookie this will in effect bypass the cache for all your pages. I'm running drupal behind varnish and it sets a lot of / cookies too, so I removed the default cookies => pass rule and instead I'm controlling which places are cached and which are passed using the url. regards, Raymond Hall From kova70 at gmail.com Sat Aug 9 02:24:43 2008 From: kova70 at gmail.com (Raymond Hall) Date: Fri, 8 Aug 2008 21:24:43 -0500 Subject: weird multisite behavior Message-ID: <53dee190808081924g39c09a76m5156d5bf52d085a6@mail.gmail.com> Hi there, My varnish setup is pretty simple, basically a director with 5 random backends with same weight and some url-based exceptions sent to a different backend. In the 5 director-backends I'm hosting several sites. I was using varnish just for one of the sites, and after Saturday when I added 3 more sites, some weird things started to happen, but only very occasionally: 1- the proxy would answer all pages really fast, but for / it took a loong time, and then it delivered the / from a different site! 2- we have some xhr includes in the / page, and those blocks would sometimes load the / from a different site. The / returned in all cases was from the same site. We have a lot of mod_rewrite "magic" going on in the apaches, and I just read a post of a kind of similar problem that was solved redirecting to the whole domain instead of just to / ... could this be the problem? I still have to debug more, but I had to move the 3 added domains back to an ugly webmux LB ... I'll post more details as soon as I get time to experiment. Meanwhile I was wondering if someone has had these kind of problems with multisites. TIA, Raymond Hall -- I'd rather have an inch of a dog than miles of pedigree. Dana Burnet From ianevans at digitalhit.com Sat Aug 9 03:14:23 2008 From: ianevans at digitalhit.com (Ian M. Evans) Date: Fri, 08 Aug 2008 23:14:23 -0400 Subject: Ads, Analytics and cookies... In-Reply-To: <53dee190808081911i3ca53ebaxff91fba66c9a2a2@mail.gmail.com> References: <489C9C73.5000208@digitalhit.com> <30df48470808081443o63035d04ka0debc24c3cc8083@mail.gmail.com> <489CC2B7.4080501@digitalhit.com> <53dee190808081911i3ca53ebaxff91fba66c9a2a2@mail.gmail.com> Message-ID: <489D0B8F.1060301@digitalhit.com> Raymond Hall wrote: > Don't route the adserver through varnish, use a different IP & domain Does it have to be a fully different domain, or can it be a subdomain like ads.example.com? >> 2) The Google Analytics cookies code also sets cookies on your own domain. > Yes, and if you leave the default rule that pass'ses all requests > having a cookie this will in effect bypass the cache for all your Yes, in my reading today I thought I saw stuff like that...if I'm not mistaken I thought I saw some discussion about handling Google Analytics cookies differently in a later release. From jnerin+varnish at gmail.com Sat Aug 9 08:28:23 2008 From: jnerin+varnish at gmail.com (=?UTF-8?Q?Jorge_Ner=C3=ADn?=) Date: Sat, 9 Aug 2008 10:28:23 +0200 Subject: Ads, Analytics and cookies... In-Reply-To: <30df48470808090126r74b68490s4e6f90a8e8e2f259@mail.gmail.com> References: <489C9C73.5000208@digitalhit.com> <30df48470808081443o63035d04ka0debc24c3cc8083@mail.gmail.com> <489CC2B7.4080501@digitalhit.com> <30df48470808090126r74b68490s4e6f90a8e8e2f259@mail.gmail.com> Message-ID: <30df48470808090128n39fe86d3s4113089550b233bf@mail.gmail.com> On Sat, Aug 9, 2008 at 00:03, Ian M. Evans wrote: > Jorge Ner?n wrote: > > >But the parsing code is a javascript included in your page but (this > is >important) served by their servers, so it can never be cached by > >varnish because the client browser ask for it directly to the third > >party servers. > > Two problems... > > 1) The OpenX ad server is a server you run ON YOUR OWN SITE, so even > though the ads may be coming from another network (setting its own > cookies) the OpenX server cookies are generated by your own domain and > not by a third party server. It's used by sites who want to sell ads > directly or geotarget certain ad networks to say North American or EU > visitors. As I'm reading more about cookies today, I'm assuming having a > CNAME of, say, an "ads" subdomain would keep the cookies off the "www." > stuff. I'll have to check with the OpenX people. > > 2) The Google Analytics cookies code also sets cookies on your own domain. > > Reading more about cookies today than I have in the decade of my site. > I'm a writer/photographer who has to toss on the sysadmin hat every few > days. :-) Yes I know about the google analytics, I use them. The way they work is setting AND parsing the cookies using the javascript you insert in your page with the src attribute pointing to the google analytics server. I cache pages, images, css, javascript with these cookies set with no problems in the analytics logs. To know if you can cache with these cookies you only need to compare the source code of two request of the same url to your backend server, one with cookies and one without, if it doesn't change then it's almost safe to cache it. I didn't know about the OpenX server, but I'm sure that you can cache pages with add serving code that is only based in javascript loaded from an external server (be it a third party or a subdomain of your own). This two cookies are being put by javascript code that runs in your domain because you put it inside your pages with a varnishncsa truncated the URL to 256 bytes, which is a problem in this case. I found in the archives a patch, which increases the limit to 65535 bytes ( http://projects.linpro.no/pipermail/varnish-misc/2007-September/001057.html), but the post is from Sept 2007 and I could not succesfull patch it to the latest svn-trunk. Is it possible to patch this in the trunk? I think, the limitation to 256 bytes might be a problem for other users too.. Thanks, Hermann-Marcus Behrens -- citybeat.de: Bremens gr??tes Stadtmagazin im Internet. Statistik M?rz 2008: Absolut eindeutige Besucher: 101.711 Pageviews: 1.728.618 Visits: 254.239 (alle Zahlen Google Analytics) citybeat.de Zum Huchtinger Bahnhof 13 28259 Bremen Tel: 0421 - 16 80 80 - 0 Fax: 0421 - 16 80 80 - 80 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ottolski at web.de Tue Aug 12 07:53:25 2008 From: ottolski at web.de (Sascha Ottolski) Date: Tue, 12 Aug 2008 09:53:25 +0200 Subject: want to allow or deny an URL based on a timestamp Message-ID: <200808120953.25459.ottolski@web.de> Hi all, I'm certain that it's possible, but am not sure how to do it: I want to let my application create "encrypted" URLs, that are valid only for a specific amount of time. I guess with the help of embedded C and manually constructed hash keys this should be doable. Now I'm wondering if may be someone already has done something like this, or as other ideas to achieve this? My idea is basically inspired by a lighttpd module: http://trac.lighttpd.net/trac/wiki/Docs:ModSecDownload The workflow would be something like - "decrpyt" incoming URL - rewrite URL, extract timestamp - if not in range, send 404 (or what seems appropriate) - if timestamp is ok, set hash key - deliver object from cache or pull from backend Thanks for any pointer, Sascha From phk at phk.freebsd.dk Tue Aug 12 09:15:15 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 12 Aug 2008 09:15:15 +0000 Subject: Truncated URL in varnishncsa In-Reply-To: Your message of "Mon, 11 Aug 2008 15:27:09 +0200." Message-ID: <17200.1218532515@critter.freebsd.dk> In message , "Herman n-Marcus Behrens" writes: >Is it possible to patch this in the trunk? I think, the limitation to 256 >bytes might be a problem for other users too.. In -trunk we have changed the format of the shmlog to support records up to 64k in length, but there are still some leftover code that (needlessly) truncated to 256 bytes. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From michael at dynamine.net Tue Aug 12 16:23:07 2008 From: michael at dynamine.net (Michael S. Fischer) Date: Tue, 12 Aug 2008 09:23:07 -0700 Subject: want to allow or deny an URL based on a timestamp In-Reply-To: <200808120953.25459.ottolski@web.de> References: <200808120953.25459.ottolski@web.de> Message-ID: <86db848d0808120923l7f8fcc3ayde45725fc86ef8db@mail.gmail.com> Nearly every modern webserver has optimized file transfers using sendfile(2). You're not going to get any better performance by shifting the burden of this task to your caching proxies. --Michael On Tue, Aug 12, 2008 at 12:53 AM, Sascha Ottolski wrote: > Hi all, > > I'm certain that it's possible, but am not sure how to do it: I want to > let my application create "encrypted" URLs, that are valid only for a > specific amount of time. I guess with the help of embedded C and > manually constructed hash keys this should be doable. Now I'm wondering > if may be someone already has done something like this, or as other > ideas to achieve this? > > My idea is basically inspired by a lighttpd module: > http://trac.lighttpd.net/trac/wiki/Docs:ModSecDownload > > The workflow would be something like > > - "decrpyt" incoming URL > - rewrite URL, extract timestamp > - if not in range, send 404 (or what seems appropriate) > - if timestamp is ok, set hash key > - deliver object from cache or pull from backend > > > Thanks for any pointer, > > Sascha > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From darryl.dixon at winterhouseconsulting.com Tue Aug 12 22:02:07 2008 From: darryl.dixon at winterhouseconsulting.com (Darryl Dixon - Winterhouse Consulting) Date: Wed, 13 Aug 2008 10:02:07 +1200 (NZST) Subject: want to allow or deny an URL based on a timestamp In-Reply-To: <86db848d0808120923l7f8fcc3ayde45725fc86ef8db@mail.gmail.com> References: <200808120953.25459.ottolski@web.de> <86db848d0808120923l7f8fcc3ayde45725fc86ef8db@mail.gmail.com> Message-ID: <34826.118.93.3.38.1218578527.squirrel@services.directender.co.nz> I'm not sure how the original post specifically relates to binary file downloads (not familiar with the lighttpd module referenced). However, I was going to comment that the requested behaviour is really better implemented at the backend. My quick sketch of how this might work is something like: 1) Backend generates time-sensitive URL and flags it with some application-specific header "X-MyApp-Cachetimeout: 500" for eg, a 500 second lifetime. 2) At the same time, it applies all the usual Cache-Control: private etc for any intermediate caches that may live in the request chain. 3) Varnish watches the returned content and when it sees X-MyApp-Cachetimeout it knows to cache the object and set the obj.ttl to the referenced value. ...Time passes... 4) Object expires and Varnish goes to fetch it from the backend, which of course returns 404 or whatever as the URL has expired. In this way, the internals of the application are hidden from Varnish as much as possible and localised inside the application itself. Hope this helps. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com > Nearly every modern webserver has optimized file transfers using > sendfile(2). You're not going to get any better performance by shifting > the > burden of this task to your caching proxies. > > --Michael > > On Tue, Aug 12, 2008 at 12:53 AM, Sascha Ottolski wrote: > >> Hi all, >> >> I'm certain that it's possible, but am not sure how to do it: I want to >> let my application create "encrypted" URLs, that are valid only for a >> specific amount of time. I guess with the help of embedded C and >> manually constructed hash keys this should be doable. Now I'm wondering >> if may be someone already has done something like this, or as other >> ideas to achieve this? >> >> My idea is basically inspired by a lighttpd module: >> http://trac.lighttpd.net/trac/wiki/Docs:ModSecDownload >> >> The workflow would be something like >> >> - "decrpyt" incoming URL >> - rewrite URL, extract timestamp >> - if not in range, send 404 (or what seems appropriate) >> - if timestamp is ok, set hash key >> - deliver object from cache or pull from backend >> >> >> Thanks for any pointer, >> >> Sascha >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-misc >> >> > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > From ottolski at web.de Wed Aug 13 06:29:54 2008 From: ottolski at web.de (Sascha Ottolski) Date: Wed, 13 Aug 2008 08:29:54 +0200 Subject: want to allow or deny an URL based on a timestamp In-Reply-To: <34826.118.93.3.38.1218578527.squirrel@services.directender.co.nz> References: <200808120953.25459.ottolski@web.de> <86db848d0808120923l7f8fcc3ayde45725fc86ef8db@mail.gmail.com> <34826.118.93.3.38.1218578527.squirrel@services.directender.co.nz> Message-ID: <200808130829.54792.ottolski@web.de> Am Mittwoch 13 August 2008 00:02:07 schrieb Darryl Dixon - Winterhouse Consulting: > 4) Object expires and Varnish goes to fetch it from the backend, > which of course returns 404 or whatever as the URL has expired. Darryl, thanks for your reply. This would of course be a straight forward approach, but in my case, it's vital that the objects live almost forever in the cache. Cheers, Sascha From ay at vg.no Fri Aug 15 09:02:50 2008 From: ay at vg.no (Audun Ytterdal) Date: Fri, 15 Aug 2008 11:02:50 +0200 Subject: Memory Usage when using malloc Message-ID: <48A5463A.9080700@vg.no> I need some help to understand this I have a 64bit server with 32GB of RAM and 60GB SWAP And varnish is running with these parameters /usr/sbin/varnishd -a :80 -f /etc/varnish/nettby.vcl -T 127.0.0.1:82 -t 120 -u varnish -g varnish -p thread_pool_add_delay 100 -p thread_pool_timeout 600 -p client_http11 on -p lru_interval 3600 -s malloc,30G -P /var/run/varnish.pid malloc'ed 30G of memory.. So in theory I should not even need swap. But [root at sol ~]# free -m total used free shared buffers cached Mem: 32242 31735 507 0 0 92 -/+ buffers/cache: 31642 600 Swap: 59871 51322 8549 The machine is using 51GB of swap and is swapping in pages frequently [root at sol ~]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 52549748 446992 780 96284 29 11 48 157 1 1 4 2 92 1 0 0 0 52549708 530360 780 96348 280 0 280 0 9654 6935 4 8 87 1 0 0 0 52549708 528160 780 96420 0 0 0 0 6785 7560 2 2 96 0 0 0 0 52549664 526668 780 96316 316 0 316 0 6570 7350 2 2 95 1 0 0 0 52549648 524932 780 96396 104 0 104 0 6300 7456 2 1 96 0 0 0 0 52549644 523372 780 96400 28 0 28 0 6013 7099 2 2 96 0 0 0 0 52549640 521884 788 96424 28 0 28 28 5649 7343 2 2 96 0 0 [root at sol ~]# varnishstat -1 client_conn 2089633996 362.23 Client connections accepted client_req 8345199749 1446.62 Client requests received cache_hit 8289156139 1436.90 Cache hits cache_hitpass 1610 0.00 Cache hits for pass cache_miss 53623537 9.30 Cache misses backend_conn 53621117 9.30 Backend connections success backend_fail 4031 0.00 Backend connections failures backend_reuse 0 0.00 Backend connections reuses backend_recycle 0 0.00 Backend connections recycles backend_unused 0 0.00 Backend connections unused n_srcaddr 7689 . N struct srcaddr n_srcaddr_act 4164 . N active struct srcaddr n_sess_mem 59601 . N struct sess_mem n_sess 64719 . N struct sess n_object 2025596 . N struct object n_objecthead 2025576 . N struct objecthead n_smf 0 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 106 . N struct vbe_conn n_bereq 96 . N struct bereq n_wrk 78 . N worker threads n_wrk_create 15968 0.00 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 191919689 33.27 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 210128619 36.43 N overflowed work requests n_wrk_drop 31202342 5.41 N dropped work requests n_backend 1 . N backends n_expired 31600718 . N expired objects n_lru_nuked 20008124 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 779599235 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 196 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 6674989721 1157.09 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 2073648637 359.46 Total Sessions s_req 8345234372 1446.62 Total Requests s_pipe 0 0.00 Total pipe s_pass 0 0.00 Total pass s_fetch 53620083 9.29 Total fetch s_hdrbytes 2924438007698 506943.34 Total header bytes s_bodybytes 13839780950697 2399088.22 Total body bytes sess_closed 54123498 9.38 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 42322579 7.34 Session Read Ahead sess_herd 8270331676 1433.64 Session herd shm_records 347828089203 60295.05 SHM records shm_writes 14631506809 2536.33 SHM writes shm_flushes 908314 0.16 SHM flushes due to overflow shm_cont 14245861 2.47 SHM MTX contention sm_nreq 0 0.00 allocator requests sm_nobj 0 . outstanding allocations sm_balloc 0 . bytes allocated sm_bfree 0 . bytes free sma_nreq 127269201 22.06 SMA allocator requests sma_nobj 4019319 . SMA outstanding allocations sma_nbytes 32212251961 . SMA outstanding bytes sma_balloc 891934739829 . SMA bytes allocated sma_bfree 859722487868 . SMA bytes free backend_req 53621117 9.30 Backend requests made ***************************************************************** Denne fotnoten bekrefter at denne e-postmeldingen ble skannet av MailSweeper og funnet fri for virus. ***************************************************************** This footnote confirms that this email message has been swept by MailSweeper for the presence of computer viruses. ***************************************************************** From jdrukman at gmail.com Fri Aug 15 22:32:34 2008 From: jdrukman at gmail.com (Jon Drukman) Date: Fri, 15 Aug 2008 15:32:34 -0700 Subject: Waiting list? Message-ID: We've got a varnish server that we're trying to put into production, but it seems to get itself into a very bad state after only an hour or so of being up. All requests end up like this: 92 SessionOpen c 64.81.61.44 46900 92 ReqStart c 64.81.61.44 46900 1629712861 92 RxRequest c GET 92 RxURL c / 92 RxProtocol c HTTP/1.1 92 RxHeader c User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3 92 RxHeader c Host: go24.cn 92 RxHeader c Accept: */* 92 VCL_call c recv 92 VCL_return c lookup 92 VCL_call c hash 92 VCL_return c hash 92 Debug c "on waiting list on obj 1629712739" 0 StatAddr 64.81.61.44 0 nan 16 16 0 0 3 5483 637585 That's it. After the "waiting list" message the request just hangs indefinitely. The only noteworthy piece of information I can think of is that the connection between the varnish server and the back end is slow and prone to packet loss. (Varnish is in China, backend is in the USA.) From ask at develooper.com Sun Aug 17 07:09:24 2008 From: ask at develooper.com (=?ISO-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Sun, 17 Aug 2008 00:09:24 -0700 Subject: Ads, Analytics and cookies... In-Reply-To: <489C9C73.5000208@digitalhit.com> References: <489C9C73.5000208@digitalhit.com> Message-ID: <08260D8D-FADF-40DB-8699-06900D960927@develooper.com> On Aug 8, 2008, at 12:20, Ian M. Evans wrote: [....] > So of course a cookie is attached to each page, image, > javascript and CSS file. > > Since I assume that some of the main sites in need of Varnish would be > the high-traffic, digg-friendly, ad-supported, Analtyics-using sites > what's the way around this? It doesn't really answer your question - but it usually makes sense to serve the static content (css/js/images) from a completely different domain. For example at http://www.yellowbot.com/ we use st.ypbot.net to have a domain that's "cookie free" for static content. Yahoo uses "l.yimg.com". You don't want the browsers to "upload" the cookies a gazillion times to a server that doesn't even look at them anyway. This also gives you more flexibility to have your site served by one set of servers and the static content served by another set of servers. - ask -- http://develooper.com/ - http://askask.com/ From phk at phk.freebsd.dk Sun Aug 17 07:21:43 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 17 Aug 2008 07:21:43 +0000 Subject: Ads, Analytics and cookies... In-Reply-To: Your message of "Sun, 17 Aug 2008 00:09:24 MST." <08260D8D-FADF-40DB-8699-06900D960927@develooper.com> Message-ID: <3493.1218957703@critter.freebsd.dk> In message <08260D8D-FADF-40DB-8699-06900D960927 at develooper.com>, =?ISO-8859-1? Q?Ask_Bj=F8rn_Hansen?= writes: >It doesn't really answer your question - but it usually makes sense to >serve the static content (css/js/images) from a completely different >domain. For example at http://www.yellowbot.com/ we use st.ypbot.net >to have a domain that's "cookie free" for static content. Yahoo >uses "l.yimg.com". > >You don't want the browsers to "upload" the cookies a gazillion times >to a server that doesn't even look at them anyway. This also gives >you more flexibility to have your site served by one set of servers >and the static content served by another set of servers. And just to be totally clear: you need different domains because that's what cookies select on but you can serve the domains from the same servers if you want. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From perbu at linpro.no Mon Aug 18 10:10:40 2008 From: perbu at linpro.no (Per Buer) Date: Mon, 18 Aug 2008 12:10:40 +0200 Subject: Waiting list? In-Reply-To: References: Message-ID: <48A94AA0.5070500@linpro.no> Jon Drukman skrev: > 92 Debug c "on waiting list on obj 1629712739" > 0 StatAddr 64.81.61.44 0 nan 16 16 0 0 3 5483 637585 > > That's it. After the "waiting list" message the request just hangs > indefinitely. You might want to have a look at session 1629712739 - earlier in the logs. I'm not sure how many people are running varnish with the origin server available on a flaky WAN connection. There might be issues. Per. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature URL: From phk at phk.freebsd.dk Tue Aug 19 20:09:28 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 19 Aug 2008 20:09:28 +0000 Subject: Varnish Errata 2008-08-19 #292 r3110 Message-ID: <71539.1219176568@critter.freebsd.dk> In response to a bug with particularly nasty effects, I have created a new wiki page: http://varnish.projects.linpro.no/wiki/TroubleLog I have not defined a specific official policy for this page, but the intent is that the page will be used as a clearing-house for "ohh-shit!" kind of stuff, so that people do not have to trawl the mailing list archives to learn about it. In addition, I will also post the entries here on varnish-misc, so that you do not have to check that wikipage every hour. Our first entry: 2008-08-19 Wrong headers can be sent to backend * Ticket: #292 * Fixed in: r3110 * Releases affected: All 1.x, 2.0-tp1, 2.0-tp2 If the client sends a Connection: header and the VCL code deletes one or more HTTP headers, the wrong set of headers may be sent to the backend. The typical result is that the next header after the one listed in Connection: is deleted, but depending on the exact headers, more complex behaviour can arise, for instance if the Host: header is deleted this way. Sorry about that... Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From mj at jarn.com Thu Aug 21 11:31:22 2008 From: mj at jarn.com (=?ISO-8859-1?Q?Martijn_Pieters_=B7_Jarn?=) Date: Thu, 21 Aug 2008 13:31:22 +0200 Subject: Run-away log, how to analyse Message-ID: <80FC249D-EB55-4BAD-BF67-0F432914F5BE@jarn.com> We run varnishd from trunk, revision 3020. This morning the 13GB harddisk of one of our varnish servers was filled to the brim by a rapidly expanding varnishlog output (varnishlog -a -w). The logfile grew 5 GB in about 15 minutes. I managed to move the logfile off the system and start varnish again (running out of disk space caused it to die, it seems). I'd like to analyse this log to see what caused it to fill so quickly. Any hints on where I should begin to look, and what tools to use? -- _______________________________________________________ Martijn Pieters ? Senior Developer, Jarn ? www.jarn.com Plone Solutions, Development, Hosting and Support _______________________________________________________ Plone Solutions is now known as Jarn www.jarn.com/name-change From duja at torlen.net Tue Aug 26 08:44:03 2008 From: duja at torlen.net (duja at torlen.net) Date: Tue, 26 Aug 2008 10:44:03 +0200 Subject: Header array implented? Message-ID: Hi, Just got of the vacation and read through alot that has happened around varnish. Looks good, nice work :) One question that I have, is it possible in 2.0 to use Headers like arrays e.g Cookie[user_id] , maybe on its way in to the trunk? Or do I still have to use regexp for that kind of stuff? Regards D From phk at phk.freebsd.dk Tue Aug 26 08:56:02 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 26 Aug 2008 08:56:02 +0000 Subject: Header array implented? In-Reply-To: Your message of "Tue, 26 Aug 2008 10:44:03 +0200." Message-ID: <51026.1219740962@critter.freebsd.dk> In message , duja at torlen.net writes: >Just got of the vacation and read through alot that has happened around varnish. Looks good, nice work :) > >One question that I have, is it possible in 2.0 to use Headers like arrays e.g Cookie[user_id] , maybe on its way in to the trunk? > >Or do I still have to use regexp for that kind of stuff? Sorry, I've never gotten around to that, and I won't promise I'll manage for 2.0 -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From isaac.z.grant at gmail.com Thu Aug 28 00:18:50 2008 From: isaac.z.grant at gmail.com (Isaac Grant) Date: Wed, 27 Aug 2008 20:18:50 -0400 Subject: varnish inside openvz: memory usage issue Message-ID: Hi, I'm running varnish inside an openvz's container. Recently, I started to see problems with varnish using all available memory. It began to happen when we started delivering bigger files: videos (10 to 50MB). Inside a virtual machine having a total 4GB of virtual memory, varnish will eat all memory in less than one hour, at 20-30 req/s. When all memory is used by varnishd, it stops to deliver content. No more process can fork, the only solution is to kill varnishd from outside the VM.. I use version 1.1.2 and tried 2.0-tp2 with similar result. I tested with a malloc and file storage, same result again. I (think I) understand the varnish's design, but should varnish eat ALL the memory like that ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at linpro.no Thu Aug 28 08:28:42 2008 From: perbu at linpro.no (Per Buer) Date: Thu, 28 Aug 2008 10:28:42 +0200 Subject: varnish inside openvz: memory usage issue In-Reply-To: References: Message-ID: <48B661BA.1070801@linpro.no> Isaac Grant skrev: > (..) > I (think I) understand the varnish's design, but should varnish eat ALL > the memory like that ? Varnish uses the VM to differentiate between what's cached in memory and what is put on disk. If the VM doesn't have any other use for the memory it will give it to varnish. The problem is that you are running VM inside a box (the openvz container) which artificially limits the memory usage. If this was a sane VM it would probably page out a lot of Varnish cache and not run out. I guess this is a bug or a weakness of openvz, first and foremost. You could work around the problem by limiting the cache size. I don't think you will encounter the same problem with Xen or any other virtualization solution where you have a separate VM for your VPS. Per. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature URL: From duja at torlen.net Thu Aug 28 15:59:43 2008 From: duja at torlen.net (E.Torlen) Date: Thu, 28 Aug 2008 17:59:43 +0200 Subject: Header array implented? In-Reply-To: <51026.1219740962@critter.freebsd.dk> References: <51026.1219740962@critter.freebsd.dk> Message-ID: <48B6CB6F.5000901@torlen.net> Maybe for 2.1 :) ? Poul-Henning Kamp skrev: > In message , duja at torlen.net writes: > > >> Just got of the vacation and read through alot that has happened around varnish. Looks good, nice work :) >> >> One question that I have, is it possible in 2.0 to use Headers like arrays e.g Cookie[user_id] , maybe on its way in to the trunk? >> >> Or do I still have to use regexp for that kind of stuff? >> > > Sorry, I've never gotten around to that, and I won't promise I'll > manage for 2.0 > > From perbu at linpro.no Thu Aug 28 17:22:28 2008 From: perbu at linpro.no (Per Buer) Date: Thu, 28 Aug 2008 19:22:28 +0200 Subject: varnish inside openvz: memory usage issue In-Reply-To: <45F46E1F2ED4774BA3371546DFBE068205AEB92810@EXCMAIL.Akersgt.local> References: , <48B661BA.1070801@linpro.no> <45F46E1F2ED4774BA3371546DFBE068205AEB92810@EXCMAIL.Akersgt.local> Message-ID: <48B6DED4.3000701@linpro.no> Audun Ytterdal skrev: > > Unfortunatly it seems like it does use up all available memory, even when you try to limit with malloc. > > http://varnish.projects.linpro.no/ticket/295 Are you sure this is the same issue? Does it go unresponsive like the OP reported and need powercycling? Per. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature URL: