From goafter1981 at gmail.com Wed Jun 1 00:41:51 2011 From: goafter1981 at gmail.com (xwu) Date: Wed, 01 Jun 2011 08:41:51 +0800 Subject: Long accpet time issue in varnish 2.1.5 Message-ID: <1306888911.2951.14.camel@xwu-ThinkPad-X201> Dear folks, In our varnish settings, we wait for long time to get response from varnish frequently. From Reqend log, we have the following finding: The '824436860' transaction takes 193.6 seconds from accepting request to dealing with request. I am not sure what's wrong. I'll supplement my vcl and start paremeter in the bottom. 0 WorkThread - 0x7f848dbfcbc0 start 30 ReqStart c 119.167.245.10 2539 824436860 30 RxRequest c GET 30 RxURL c /javascripts/jquery/jquery.backgroundPosition.js?1306827037 30 RxProtocol c HTTP/1.1 30 RxHeader c Host: dev.yottaa.com.try.yottaa.net 30 RxHeader c User-Agent: YottaaMonitor 30 RxHeader c Accept: */* 30 RxHeader c Accept-Language: en-us,en;q=0.5 30 RxHeader c Accept-Encoding: gzip,deflate 30 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 30 RxHeader c Keep-Alive: 115 30 RxHeader c Connection: keep-alive 30 RxHeader c Referer: http://dev.yottaa.com.try.yottaa.net/ 30 RxHeader c Cookie: y=b92319a06dcf012ebe4712313909bca1; _dpu_session=BAh7CDoQX2NzcmZfdG9rZW4iMU9vK3VEK2wxUUlMWDR2M2NDMUc5ZTQrM2NDN1p4VVk5N0dKNlVHYm5FUzg9OhN1c2VyLnJldHVybl90byIGLzoPc2Vzc2lvbl9pZCIlNTA3ZDE5ODBjYzZlMzk0YTI2ZWE0YmQyNWEyZTg2MmM%3D--97597402d695f4c7a7b7dd 30 VCL_call c recv 30 VCL_return c pass 30 VCL_call c hash 30 VCL_return c hash 30 VCL_call c pass 30 VCL_return c pass 30 Backend c 23 yo a 23 TxRequest b GET 23 TxURL b /javascripts/jquery/jquery.backgroundPosition.js?1306827037 23 TxProtocol b HTTP/1.1 23 TxHeader b Host: dev.yottaa.com.try.yottaa.net 23 TxHeader b User-Agent: YottaaMonitor 23 TxHeader b Accept: */* 23 TxHeader b Accept-Language: en-us,en;q=0.5 23 TxHeader b Accept-Encoding: gzip,deflate 23 TxHeader b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 23 TxHeader b Referer: http://dev.yottaa.com.try.yottaa.net/ 23 TxHeader b Cookie: y=b92319a06dcf012ebe4712313909bca1; _dpu_session=BAh7CDoQX2NzcmZfdG9rZW4iMU9vK3VEK2wxUUlMWDR2M2NDMUc5ZTQrM2NDN1p4VVk5N0dKNlVHYm5FUzg9OhN1c2VyLnJldHVybl90byIGLzoPc2Vzc2lvbl9pZCIlNTA3ZDE5ODBjYzZlMzk0YTI2ZWE0YmQyNWEyZTg2MmM%3D--97597402d695f4c7a7b7dd 23 TxHeader b X-Forwarded-For: 119.167.245.10 23 TxHeader b X-Varnish: 824436860 23 RxProtocol b HTTP/1.1 23 RxStatus b 200 23 RxResponse b OK 23 RxHeader b Server: nginx/0.8.54 23 RxHeader b Date: Tue, 31 May 2011 07:33:41 GMT 23 RxHeader b Content-Type: application/x-javascript 23 RxHeader b Last-Modified: Tue, 31 May 2011 07:30:37 GMT 23 RxHeader b Vary: Accept-Encoding 23 RxHeader b Expires: Thu, 31 Dec 2037 23:55:55 GMT 23 RxHeader b Cache-Control: max-age=315360000 23 RxHeader b Content-Encoding: deflate 23 RxHeader b Content-Length: 807 23 RxHeader b X-Yottaa-Cache: hit 23 RxHeader b X-Yottaa-Timestamp: 12345678911 23 RxHeader b X-Yottaa-Optimizations: compression 23 RxHeader b X-Yottaa-Timings: 0,620,620 23 RxHeader b X-Yottaa-SessionId: b2a163ef-b80d-46bd-bb7e-e27e3b0d3d08 23 RxHeader b X-Yottaa-V2R-Thread: 49 23 RxHeader b X-Yottaa-R2O-Thread: hit-in-cache? 30 TTL c 824436860 RFC 315360000 1306858991 0 0 315360000 0 30 VCL_call c fetch 30 VCL_Log c enter fetch 30 VCL_Log c hit 30 TTL c 824436 30 ObjProtocol c HTTP/1.1 30 ObjStatus c 200 30 ObjResponse c OK 30 ObjHeader c Server: nginx/0.8.54 30 ObjHeader c Date: Tue, 31 May 2011 07:33:41 GMT 30 ObjHeader c Content-Type: application/x-javascript 30 ObjHeader c Last-Modified: Tue, 31 May 2011 07:30:37 GMT 30 ObjHeader c Vary: Accept-Encoding 30 ObjHeader c Expires: Thu, 31 Dec 2037 23:55:55 GMT 30 ObjHeader c Cache-Control: max-age=315360000 30 ObjHeader c Content-Encoding: deflate 30 ObjHeader c Content-Length: 807 30 ObjHeader c X-Yottaa-Cache: hit 30 ObjHeader c X-Yottaa-Timestamp: 12345678911 30 ObjHeader c X-Yottaa-Optimizations: compression 30 ObjHeader c X-Yottaa-Timings: 0,620,620 30 ObjHeader c X-Yottaa-SessionId: b2a163ef-b80d-46bd-bb7e-e27e3b0d3d08 30 ObjHeader c X-Yottaa-V2R-Thread: 49 30 ObjHeader c X-Yottaa-R2O-Thread: hit-in-cache? 23 Fetch_Body b 4 0 1 23 Length b 807 23 BackendReuse b a 30 VCL_call c deliver 30 VCL_return c deliver 30 TxProtocol c HTTP/1.1 30 TxStatus c 200 30 TxResponse c OK 30 TxHeader c Server: nginx/0.8.54 30 TxHeader c Content-Type: application/x-javascript 30 TxHeader c Last-Modified: Tue, 31 May 2011 07:30:37 GMT 30 TxHeader c Vary: Accept-Encoding 30 TxHeader c Expires: Thu, 31 Dec 2037 23:55:55 GMT 30 TxHeader c Cache-Control: max-age=315360000 30 TxHeader c Content-Encoding: deflate 30 TxHeader c X-Yottaa-Cache: hit 30 TxHeader c X-Yottaa-Timestamp: 12345678911 30 TxHeader c X-Yottaa-Optimizations: compression 30 TxHeader c X-Yottaa-Timings: 0,620,620 30 TxHeader c X-Yottaa-SessionId: b2a163ef-b80d-46bd-bb7e-e27e3b0d3d08 30 TxHeader c X-Yottaa-V2R-Thread: 49 30 TxHeader c X-Yottaa-R2O-Thread: hit-in-cache? 30 TxHeader c Content-Length: 807 30 TxHeader c Date: Tue, 31 May 2011 16:23:11 GMT 30 TxHeader c X-Varnish: 824436860 30 TxHeader c Age: 0 30 TxHeader c Via: 1.1 varnish 30 TxHeader c Connection: keep-alive 30 TxHeader c X-Cache: MISS 30 Length c 807 30 ReqEnd c 824436860 1306858991.862764597 1306858991.865270615 193.628781319 0.002455711 0.000050306 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/yottaa.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -s malloc,5 Below is the VCL configuration: backend a { .host = "192.168.1.1"; .port = "80"; .probe = { .request = "TRACE / HTTP/1.1" "Max-Forwards: 0" "Connection: close"; .timeout = 0.3 s; .window = 5; .threshold = 3; } .connect_timeout = 2s; .first_byte_timeout = 330s; .between_bytes_timeout = 330s; } backend b { .host = "192.168.1.2"; .port = "80"; .probe = { .request = "TRACE / HTTP/1.1" "Max-Forwards: 0" "Connection: close"; .timeout = 0.3 s; .window = 5; .threshold = 3; } .connect_timeout = 2s; .first_byte_timeout = 330s; .between_bytes_timeout = 330s; } director yo client { { .backend = a; .weight = 1; } { .backend = b; .weight = 1; } } sub vcl_recv { if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { unset req.http.cookie; } if (req.backend.healthy) { set req.grace = 30s; } else { set req.grace = 1h; } set req.backend = yo; /* Load balance by user agent */ set client.identity = req.http.host; } sub vcl_fetch { set beresp.grace = 1h; log "enter fetch"; log beresp.http.X-y-Cache; if (beresp.http.X-y-Cache && !beresp.http.X-y-Cache ~ "hit") { log "Cache miss from tpu"; log "set ttl 0s " ; set beresp.ttl= 0s; return (pass); } if (!beresp.cacheable) { set beresp.ttl= 0s; return (pass); } if (beresp.http.Set-Cookie) { set beresp.ttl= 0s; return (pass); } if (beresp.http.Content-Type && beresp.http.Content-Type ~ "html" ) { return (pass); } return (deliver); } sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } return (deliver); } From tfheen at varnish-software.com Wed Jun 1 06:52:15 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 01 Jun 2011 08:52:15 +0200 Subject: percent sign in vcl strings In-Reply-To: <4DE524FA.5070100@sentia.nl> (Camiel Dobbelaar's message of "Tue, 31 May 2011 19:27:22 +0200") References: <4DE524FA.5070100@sentia.nl> Message-ID: <8739juoxgg.fsf@qurzaw.varnish-software.com> ]] Camiel Dobbelaar | What about allowing "%%" for a literal "%" like printf() ? It's already changed in 3.0. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From patrick.cao_huu_thien at upmc.fr Wed Jun 1 11:02:22 2011 From: patrick.cao_huu_thien at upmc.fr (Patrick CAO HUU THIEN) Date: Wed, 1 Jun 2011 13:02:22 +0200 Subject: time out on big object ?? In-Reply-To: <4DE3C245.9030406@uplex.de> References: <4DE3C245.9030406@uplex.de> Message-ID: 2011/5/30 Geoff Simmons > > The default value of the send_timeout parameter is 600 seconds: > > "Send timeout for client connections. If no data has been sent to the > client in this many seconds, the session is closed. See setsockopt(2) > under SO_SNDTIMEO for more information." > > Probably what's happening is that Varnish needs more than 10 minutes to > read the 630 MB monster from your backend, during which the client > connection sits idle, and the timeout elapses. > > You could set a higher timeout with -p send_timeout=, but > you're probably much better off having VCL return pass on that > particular URL. Unless you really need Varnish to take up 630 MB of > space just to cache your Ubuntu ISO. > Yes, that's exactly what I want. So I just add this line to the vcl_recv function: sub vcl_recv { if (req.http.host ~ "^xxxx") { set req.backend = xxxx; *if (req.url ~ "^/iso/") {* * return(pass);* * }* The time out is always activated ?! The only difference is what, now, wget can use the "206 Partial content" option, but firefox can't :( I am sure what I fail on something, because with the pass action, no time must be send to the client anymore. -- patrick -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Wed Jun 1 12:11:19 2011 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 01 Jun 2011 14:11:19 +0200 Subject: time out on big object ?? In-Reply-To: References: <4DE3C245.9030406@uplex.de> Message-ID: <4DE62C67.1050504@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 06/ 1/11 01:02 PM, Patrick CAO HUU THIEN wrote: >> >> You could set a higher timeout with -p send_timeout=, but >> you're probably much better off having VCL return pass on that >> particular URL. Unless you really need Varnish to take up 630 MB of >> space just to cache your Ubuntu ISO. >> > > Yes, that's exactly what I want. > > So I just add this line to the vcl_recv function: > > sub vcl_recv { > if (req.http.host ~ "^xxxx") { > set req.backend = xxxx; > *if (req.url ~ "^/iso/") {* > * return(pass);* > * }* Um, I don't follow. You *do* want Varnish to cache the file? If so, then you don't want to return pass on the URL. In that case, you need a larger value for send_timeout (assuming my guess is right that send_timeout aborted your connection). > The time out is always activated ?! > The only difference is what, now, wget can use the "206 Partial content" > option, but firefox can't :( > > I am sure what I fail on something, because with the pass action, no time > must be send to the client anymore. There's always a send_timeout, 10 minutes by default, but you can increase it. If you have return(pass) on the URL, then the timeout won't interfere with the transmission, but then Varnish won't be caching the object. If you want Varnish to cache and transmit the object, you need to remove return(pass), but set a higher value for send_timeout. Try starting varnishd with -p send_timeout=1200 -- in your first mail, it looked like 53% had been transmitted in 10 minutes. so the timeout would have to be about twice as long. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (SunOS) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJN5ixnAAoJEOUwvh9pJNURL6wP/jgu2Zaos7vD9UGPq4377U2E 8lF+68u79UKl9C6bL1+Av1g6LE4K0AwB00+r+yaNs5O4cuAZx4YdqzphBXEnH3zH Np7NZQY+bw8WGXu7hIg/N9b4bhnqzqIOcrPhbRHE1/T1zqrArD/5mbfFMkGeoYxd eTXP/TsN2Ng557WMLc2EQaxGxjKpzRTtovx8LjA+KksGB3JTMuPDNCuD/3lg5Evr IrH/V0AqqN3Vn7LJt0BN/FU6NL71BEmjBj8srZ5fAxsxQQn79vU0mgWySaO8orvK XnyJJ0B/119VhaXNVzWEY16AkZctZB3TDV8wmHsKF420LIYPpP6q1iWK9viL18zl JzaE3A2qrTUichV4b+Mz+Dkrk9lIPmPwlFzJA++S397havl1nhWNRgCHShpLeGb6 JYC6VxrvW7uuyW1SrwetwQ7CoqcXY1NInj0h9fgAPqL87gb4Eiq+fZ3U4MDmALns 2PxBoHzVz5L+3GbidBXPgM63PFnW10erFI94Hl4nJjW5ZCiKGGHqlT3JzXl0hP4U W985xyWG5w0oHjwOD2Y2GjQWSLN/NfMTrM5fozsH68knSMmdZx3fHFmfQTzIqB59 H3/H1EwBXwCPGmOeDJm6JOfzYSwM61k3j7RdXh1hH8i+6X5ZWqLF9QBZCLE0ps1D 4ME5q1xH+9kBIB+MKat6 =IyMc -----END PGP SIGNATURE----- From jeanmarc.pouchoulon at gmail.com Wed Jun 1 17:39:32 2011 From: jeanmarc.pouchoulon at gmail.com (jean-marc pouchoulon) Date: Wed, 1 Jun 2011 19:39:32 +0200 Subject: Write error, retval = 62780, len = 305169, errno = Success moodle backend Message-ID: Hello varnish list , I got a problem with Content-Disposition: attachment; filename="Activite.... with a resulting error *"Write error, retval = 62780, len = 305169, errno = Success"* I saw some post about write error and I have modified tcp mem value ins sysctl.conf without any success The backend in the following log is a moodle server but I got the same problem with another type of backend and I have to modify vcl with if (req.http.Content-Disposition ~ "inline; filename=.*.pdf$") { return(pass); } I'm using varnish-2.1.5 SVN 0843d7a on linux centos 5 direct acces to apache backend works fine. the size of file is in Ko not Mo un 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Server: Apache/2.2.3 (CentOS) Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c X-Powered-By: PHP/5.1.6 Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Expires: Thu, 01 Jan 1970 00:00:00 GMT Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Cache-Control: private, must-revalidate, pre-check=0, post-check=0, max-age=0 Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Pragma: no-cache Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Last-Modified: Thu, 05 May 2011 13:44:07 GMT Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Content-Disposition: attachment; filename="Activite_12.docx" Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Accept-Ranges: none Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Content-Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Content-Length: 304590 Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Date: Wed, 01 Jun 2011 11:11:49 GMT Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c X-Varnish: 1757900317 Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Age: 0 Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Via: 1.1 varnish Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c Connection: keep-alive Jun 1 14:23:27 v222ent-varnish2 logger: 20 TxHeader c X-Cache: MISSV2 Jun 1 14:23:27 v222ent-varnish2 logger: 20 Debug c *"Write error, retval = 62780, len = 305169, errno = Success"* Jun 1 14:23:27 v222ent-varnish2 logger: -1 Length - 304590 Jun 1 14:23:27 v222ent-varnish2 logger: 20 ReqEnd c 1757900317 1306926709.079653263 1306926709.155108452 3.159499168 0.07 any ideas ? thanks jean-marc -------------- next part -------------- An HTML attachment was scrubbed... URL: From ghstridr at gmail.com Wed Jun 1 21:39:32 2011 From: ghstridr at gmail.com (Mike Gracy) Date: Wed, 1 Jun 2011 14:39:32 -0700 Subject: docs or examples of vcl.inline seem to be scarce. Message-ID: I'm trying to figure out how to add a backend to an existing config: vcl.inline boot "backend wiki { .host = "wiki.idle-gaems.com"; .port = "80"; }" 105 64 Unknown request. Type 'help' for more info. Too many parameters Is this only for the use of including C code? From ghstridr at gmail.com Thu Jun 2 00:06:53 2011 From: ghstridr at gmail.com (Mike Gracy) Date: Wed, 1 Jun 2011 17:06:53 -0700 Subject: docs or examples of vcl.inline seem to be scarce. In-Reply-To: References: Message-ID: For reference of what I'm trying to do: (Varnish 2.1.5, running in debug mode) In backend specification starting at: (input Line 1 Pos 1) backend default { .host = wiki.idle-games.com; .port = 80; } #######----------------------------------------------------- Running VCC-compiler failed, exit 1 VCL compilation failed vcl.inline new backend default "{ .host = 'wiki.idle-games.com'; .port = '80'; }" 105 64 Unknown request. Type 'help' for more info. Too many parameters On Wed, Jun 1, 2011 at 2:39 PM, Mike Gracy wrote: > I'm trying to figure out how to add a backend to an existing config: > vcl.inline boot "backend wiki { .host = "wiki.idle-gaems.com"; .port = "80"; }" > 105 64 > Unknown request. > Type 'help' for more info. > Too many parameters > > Is this only for the use of including C code? > From checker at d6.com Thu Jun 2 05:56:37 2011 From: checker at d6.com (Chris Hecker) Date: Wed, 01 Jun 2011 22:56:37 -0700 Subject: esi question Message-ID: <4DE72615.9080209@d6.com> I'm trying to figure out if I can use ESI to solve a problem, and I'm wondering if there's a way to take parameters from the main url and pass them to the ESI included url, so like: http://blah.com/foo?bar=true and inside, I want "?bar=true" to be passed to the ESI. Do I need to do this manually with regexes and whatnot in the vcl, or is there a way to automate it? I've seen something about last-mod-time or something being passed, but it wasn't clear from the docs... Or, am I missing something and this is a dumb idea? Thanks, Chris From ruben at varnish-software.com Thu Jun 2 13:37:35 2011 From: ruben at varnish-software.com (=?ISO-8859-1?Q?Rub=E9n_Romero?=) Date: Thu, 2 Jun 2011 15:37:35 +0200 Subject: Varnish Cache 3.0 Release Party site - http://v3party.varnish-cache.org In-Reply-To: References: Message-ID: Hei again, The site is here: http://v3party.varnish-cache.org/ Paul Krischer just announced the launch of this awesome site (based in the drupal7releaseparty.org site) so we can rejoice celebrating the launch of Varnish Cache 3.0 in two weeks from now. Check his tweet and blog post telling how this happened: http://sqyd.net/node/5 http://twitter.com/#!/SqyD/status/76254093131259904 Use the hash #v3rp on images, videos, blog tags and tweets so we can add it automatically to the stream. We have twitter integration now and are working in Flickr integration as well. Will add YouTube and probably Facebook integration later. So please go ahead and add you own party or register if you are coming to the events in Buenos Aires, Hamburg, London, Milano Oslo, Santa Clara (the 15th @Velocity) or Zurich. I hear rumours of parties in Paris and Sao Paulo as well. If you want to have your own event listed just get in touch and we'll make it happen ;) Thanks to KPN for hosting, Varnish Software for awesomeness, Savant for artwork and specially Paul for driving this forward! Full credits in the About and Credit pages of the site. Have a great day everyone! Regards, - Rub?n Romero Varnish Software -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul at krischer.nl Thu Jun 2 14:04:45 2011 From: paul at krischer.nl (Paul Krischer) Date: Thu, 2 Jun 2011 16:04:45 +0200 Subject: Varnish Cache 3.0 Release Party site - http://v3party.varnish-cache.org In-Reply-To: References: Message-ID: Hi all, Building this site over a couple of evenings was a lot of fun. I hope you enjoy it. One minor adjustment: "If you want to have your own event listed just get in touch and we'll make it happen" Even better than that: You can make it happen yourself! Just create an account on the site and add your own event yourself: http://v3party.varnish-cache.org/node/add/release-party After creating it people you can add information afterwards, people van signup for your event and you don't need to bother Rub?n or me. Off course we're there to keep an eye out for party crashers ;-) Cheers, Paul Krischer 2011/6/2 Rub?n Romero : > Hei again, > > The site is here: http://v3party.varnish-cache.org/ > > Paul Krischer just announced the launch of this awesome site (based in the > drupal7releaseparty.org site) so we can rejoice celebrating the launch of > Varnish Cache 3.0 in two weeks from now. Check his tweet and blog post > telling how this happened: > http://sqyd.net/node/5 > http://twitter.com/#!/SqyD/status/76254093131259904 > > Use the hash #v3rp on images, videos, blog tags and tweets so we can add it > automatically to the stream. We have twitter integration now and are working > in Flickr integration as well. Will add YouTube and probably Facebook > integration later. > > So please go ahead and add you own party or register if you are coming to > the events in Buenos Aires, Hamburg, London, Milano Oslo, Santa Clara (the > 15th @Velocity) or Zurich. I hear rumours of parties in Paris and Sao Paulo > as well. If you want to have your own event listed just get in touch and > we'll make it happen ;) > > Thanks to KPN for hosting, Varnish Software for awesomeness, Savant for > artwork and specially Paul for driving this forward! Full credits in the > About and Credit pages of the site. > > Have a great day everyone! > > Regards, > - > Rub?n Romero > Varnish Software > From isp at daviesinc.com Fri Jun 3 20:28:52 2011 From: isp at daviesinc.com (Chris Davies) Date: Fri, 3 Jun 2011 16:28:52 -0400 Subject: Minor typo in documentation Message-ID: <9B7B071C-023D-4456-B374-9FD626F7141F@daviesinc.com> Typos on http://www.varnish-cache.org/docs/trunk/tutorial/purging.html#bans This probably comes from the sphinx markup sub vcl_recv { if (req.request == "BAN") { # Same ACL check as above: if (!client.ip ~ purge) { error 405 "Not allowed."; } # ban("req.http.host == " req.http.host # "&& req.url == " req.url); ban("req.http.host == " + req.http.host + "&& req.url == " + req.url); # Throw a synthetic page so the # request won't go to the backend. # error 200 "Ban added" error 200 "Ban added"; } } missing + for concatenation in ban, missing ; at end of line for error 200 From mattias at nucleus.be Mon Jun 6 09:40:40 2011 From: mattias at nucleus.be (Mattias Geniar) Date: Mon, 6 Jun 2011 11:40:40 +0200 Subject: Varnish <3 Party in Belgium, Antwerp Message-ID: <18834F5BEC10824891FB8B22AC821A5A017349F4@nucleus-srv01.Nucleus.local> Hi there! We're gladly hosting a Varnish 3.0 release party, to celebrate (hell, any reason to drink beers is a good one, isn't it?). Feel free to sign up here; http://v3party.varnish-cache.org/belgium/release-party-nucleusbe Current address listed is our office, but if the weather allows it we'll be going to a nearby terrace for some beers in the sun. Please confirm via the site or just reply here to me personally, so we can have an idea how many people may be interested. Cheers for the Varnish crew! Regards, Mattias -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Tue Jun 7 06:11:58 2011 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 07 Jun 2011 08:11:58 +0200 Subject: Conditional backend requests on repository branch experimental-ims Message-ID: <4DEDC12E.9010702@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello all, The code for the proposed conditional backend request feature (usually If-Modified-Since requests to backends) is now up to date with the current trunk and available on an experimental branch of the source repository called experimental-ims. On an updated git working tree you can say: $ git branch -a ... and you should see something like 'remotes/origin/experimental-ims' in the list. Then to work with the branch: $ git checkout -b experimental-ims remotes/origin/experimental-ims To switch back to the master branch: $ git checkout master The feature is documented here: http://www.varnish-cache.org/trac/wiki/BackendConditionalRequests Please note that this code is experimental, and will not be included in the upcoming 3.0.0 release. The way that this works has not been changed from the previous versions distributed as patches, but will likely be changed in future versions. Testing the mainline 3.0 version is the highest priority for the Varnish team right now, so feedback on that is most important. If you also get a chance to test the IMS feature, it will be a great help toward getting an "official" version settled, so let us know how it goes. Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJN7cEuAAoJEOUwvh9pJNURH9cQAKhwMbFRc7XBBwro2oKE57c4 jRH36dZMV9/PgAsgg4LlCWbt3+ceG9607gigHocY4VY6viF4ppsK3daxGYL4qxMj st4hwJwyZculNDr3Fbg/du13/o4Dexl5Rhy49g1AtxgDCkioT4xo5Y+9OPfI2nAq KbiThND+yThCcaOUDssZZsp9ed3JEPD71SNE5Xh0JupXTgXc8RTU2n32XYZE6HwF re4Lfpk1RA9xKFg+S/Dz2zAKvJTFIU9liV9abrdQTnHyhsloDyf04QLg7bVnZ28c 5PXWQlo2PSxujcLJvr9uKr5CaS+CNWRADd/8DIH6q2tf6pMX3VfwryjbRh2KpT4I /GHYEJsyCk0nUNNEwzTPXhZE6zT9yvRPU/qhx09fOsg+X7FEDyZJhTLvgiqIKaFd OESrgyQE0wBu/5O+fg7dY+U8rPzQI5aKZWrBYLHIwBA4puCUpq1Fm2+/FZY6vGd5 GVDbCMNF+Q4HADfYAQZG8X753DQ5O0ZzVvlKP2hmh+r13UTcx45wmFiXitZJFdy4 BRuNlsJAt13qIgP/I38hCJEay6OgDIyhRtyTkX/aZckHSX/rI/4rjqN7bsHOMIOE 33LHwP9VV0OaDPhov9iKKF6/aMqCJ6w5mRUp+qDHdePgQv2uER/HC3qBEtpErB00 4LsJb64hxNQds4/nMjCs =n6yS -----END PGP SIGNATURE----- From joerg.wiegleb at online.de Fri Jun 3 11:11:31 2011 From: joerg.wiegleb at online.de (=?iso-8859-1?q?J=F6rg_Wiegleb?=) Date: Fri, 3 Jun 2011 13:11:31 +0200 Subject: varnish POST issue Message-ID: <201106031311.31640.joerg.wiegleb@online.de> Hey @all, whenever i send an POST-request varnish comes up with a 503 server-error. this bug(?) is already discussed on the web, but i can?t see how to solve this issue! may be it belongs to the POST-size of the request? Varnishlog tells that i am sending a 31 TxRequest - POST 31 TxHeader - Content-Type: application/x-www-form-urlencoded but i am getting 9 FetchError c backend write error: 11 31 BackendClose - [backend name] 9 VCL_call c error 9 VCL_return c deliver 9 Length c 488 9 VCL_call c deliver 9 VCL_return c deliver 9 TxProtocol c HTTP/1.1 9 TxStatus c 503 need help to fix it! From k at kevinkevin.com Mon Jun 6 16:10:35 2011 From: k at kevinkevin.com (kevin) Date: Mon, 6 Jun 2011 12:10:35 -0400 Subject: Remote backend servers Message-ID: <106b01cc2464$45818f30$d084ad90$@com> Hello, I realize this is a generalized question, but I was hoping to get some feedback as far as if anyone has implemented backend servers in the varnish director configuration that are remotely in another data center. The varnish frontend would have a 100meg connectivity over the internet to the backend server. Would this be considered in line with best practices? What would be the potential problems? Obviously the timeout variables would have to be adjusted. Any input or ideas would be greatly appreciated! Thanks, Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtshilston at gmail.com Tue Jun 7 09:39:55 2011 From: rtshilston at gmail.com (Robert Shilston) Date: Tue, 7 Jun 2011 10:39:55 +0100 Subject: Remote backend servers In-Reply-To: <106b01cc2464$45818f30$d084ad90$@com> References: <106b01cc2464$45818f30$d084ad90$@com> Message-ID: > I realize this is a generalized question, but I was hoping to get some feedback as far as if anyone has implemented backend servers in the varnish director configuration that are remotely in another data center. > > The varnish frontend would have a 100meg connectivity over the internet to the backend server. Would this be considered in line with best practices? > > What would be the potential problems? Obviously the timeout variables would have to be adjusted. > We had to do this for a week - we ran a backend on ADSL about 20ms from a Varnish node in a data center. We had no problems with it. Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From audun at ytterdal.net Tue Jun 7 09:58:05 2011 From: audun at ytterdal.net (Audun Ytterdal) Date: Tue, 7 Jun 2011 11:58:05 +0200 Subject: Remote backend servers In-Reply-To: <106b01cc2464$45818f30$d084ad90$@com> References: <106b01cc2464$45818f30$d084ad90$@com> Message-ID: <9CF886C5-A346-4E53-95B7-A4A4641B9D20@ytterdal.net> We do this for as a layer in front of remote API calls to 3rd party providers (mostly to be able to terminate long running serverside queries. Php is notoriously bad at timeouts. Works like a charm. Den 6. juni 2011 kl. 18:10 skrev "kevin" : > Hello, > > > > > > I realize this is a generalized question, but I was hoping to get some feedback as far as if anyone has implemented backend servers in the varnish director configuration that are remotely in another data center. > > > > The varnish frontend would have a 100meg connectivity over the internet to the backend server. Would this be considered in line with best practices? > > > > What would be the potential problems? Obviously the timeout variables would have to be adjusted. > > > > Any input or ideas would be greatly appreciated! > > > > Thanks, > > > > Kevin > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish-announce-bounces at varnish-cache.org Tue Jun 7 12:04:58 2011 From: varnish-announce-bounces at varnish-cache.org (varnish-announce-bounces at varnish-cache.org) Date: Tue, 07 Jun 2011 14:04:58 +0200 Subject: Forward of moderated message Message-ID: An embedded message was scrubbed... From: Jean-Francois Laurens Subject: Re: Varnish 3.0.0 beta 2 released Date: Tue, 7 Jun 2011 13:43:16 +0200 Size: 4762 URL: From tfheen at varnish-software.com Tue Jun 7 12:19:31 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 07 Jun 2011 14:19:31 +0200 Subject: Minor typo in documentation In-Reply-To: <9B7B071C-023D-4456-B374-9FD626F7141F@daviesinc.com> (Chris Davies's message of "Fri, 3 Jun 2011 16:28:52 -0400") References: <9B7B071C-023D-4456-B374-9FD626F7141F@daviesinc.com> Message-ID: <87tyc1kf58.fsf@qurzaw.varnish-software.com> ]] Chris Davies | Typos on http://www.varnish-cache.org/docs/trunk/tutorial/purging.html#bans | | This probably comes from the sphinx markup Thanks, fixed. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From patrick.cao_huu_thien at upmc.fr Tue Jun 7 13:01:14 2011 From: patrick.cao_huu_thien at upmc.fr (Patrick CAO HUU THIEN) Date: Tue, 7 Jun 2011 15:01:14 +0200 Subject: time out on big object ?? In-Reply-To: <4DE62C67.1050504@uplex.de> References: <4DE3C245.9030406@uplex.de> <4DE62C67.1050504@uplex.de> Message-ID: Geoff Simmons a dit le 06/01/2011 02:11 PM: >> Um, I don't follow. You **do** want Varnish to cache the file? If so, then> you don't want to return pass on the URL. In that case, you need a> larger value for send_timeout (assuming my guess is right that> send_timeout aborted your connection). Damned. No. I don't want to cache big objects. sorry for my bad english ... :( > > There's always a send_timeout, 10 minutes by default, but you can> increase it. If you have return(pass) on the URL, then the timeout won't> interfere with the transmission, but then Varnish won't be caching the> object. Are you sure about the disconnection between the pass action and the send_timeout parameter ? Ok, I understand the time-out of 10min. But with the return(pass) I've just added, varnish continue to make a time-out after 10min. So I have to put the send_timeout option to a higher value even if I make a return(pass). 1200s may not be enough because personal or foreign connection may be slower then the one I test at home. I put 6000s to be large :) thanks for your help Goeff Patrick -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Tue Jun 7 14:36:39 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 07 Jun 2011 16:36:39 +0200 Subject: Forward of moderated message In-Reply-To: (varnish-announce-bounces@varnish-cache.org's message of "Tue, 07 Jun 2011 14:04:58 +0200") References: Message-ID: <87lixdk8so.fsf@qurzaw.varnish-software.com> ]] | > * the ?stats? command has been removed from the CLI interface as it did | > not work well with the new dynamic counters. | | What would be the recommended way to catch statistics, for cacti as an example ? Make it talk to varnishstat either through inetd or some other service. Eventually, I want to expose the numbers through a vmod so you can just expose a JSON object or similar, but we're not there yet. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From mhettwer at team.mobile.de Tue Jun 7 15:50:13 2011 From: mhettwer at team.mobile.de (Hettwer, Marian) Date: Tue, 7 Jun 2011 16:50:13 +0100 Subject: Forward of moderated message In-Reply-To: <87lixdk8so.fsf@qurzaw.varnish-software.com> Message-ID: On 07.06.11 16:36, "Tollef Fog Heen" wrote: >]] > >| > * the ?stats? command has been removed from the CLI interface as it >did >| > not work well with the new dynamic counters. >| >| What would be the recommended way to catch statistics, for cacti as an >example ? > >Make it talk to varnishstat either through inetd or some other service. >Eventually, I want to expose the numbers through a vmod so you can just >expose a JSON object or similar, but we're not there yet. I like the munin graphs (and munin itself). It also uses "varnishstat -1" as its source of data. Common answer here is: run varnishstat :) ./Marian From l at lrowe.co.uk Tue Jun 7 23:04:59 2011 From: l at lrowe.co.uk (Laurence Rowe) Date: Wed, 8 Jun 2011 00:04:59 +0100 Subject: time out on big object ?? In-Reply-To: References: <4DE3C245.9030406@uplex.de> <4DE62C67.1050504@uplex.de> Message-ID: On 7 June 2011 14:01, Patrick CAO HUU THIEN wrote: > Geoff Simmons a dit le 06/01/2011 02:11 PM: > >> >> Um, I don't follow. You *do* want Varnish to cache the file? If so, then >> you don't want to return pass on the URL. In that case, you need a >> larger value for send_timeout (assuming my guess is right that >> send_timeout aborted your connection). > > Damned. No. I don't want to cache big objects. > sorry for my bad english ... :( > >> >> There's always a send_timeout, 10 minutes by default, but you can >> increase it. If you have return(pass) on the URL, then the timeout won't >> interfere with the transmission, but then Varnish won't be caching the >> object. > > Are you sure about the disconnection between the pass action and the > send_timeout parameter ? > > Ok, I understand the time-out of 10min. > But with the return(pass) I've just added, varnish continue to make > a time-out after 10min. > > So I have to put the send_timeout option to a higher value even if I > make a return(pass). 1200s may not be enough because personal or > foreign connection may be slower then the one I test at home. I put > 6000s to be large :) > > thanks for your help Goeff If you don't want to cache the large object at all then you can return (pipe); instead. I do this for large media files proxied by Varnish to prevent reading the entire file into memory before delivering it to the client. It sounds like you may still need to increase the time out value though, which is a parameter to varnishd. Laurence From nkinkade at creativecommons.org Tue Jun 7 23:23:33 2011 From: nkinkade at creativecommons.org (Nathan Kinkade) Date: Tue, 7 Jun 2011 19:23:33 -0400 Subject: Remote backend servers In-Reply-To: <9CF886C5-A346-4E53-95B7-A4A4641B9D20@ytterdal.net> References: <106b01cc2464$45818f30$d084ad90$@com> <9CF886C5-A346-4E53-95B7-A4A4641B9D20@ytterdal.net> Message-ID: On Tue, Jun 7, 2011 at 05:58, Audun Ytterdal wrote: > We do this for as a layer in front of remote API calls to 3rd party > providers (mostly to be able to terminate long running serverside queries. > Php is notoriously bad at timeouts. Works like a charm. We don't do this on a permanent basis, but on a number of occasions we have, when moving a service from one machine to another remote one. We set up the service on the new machine, sync everything, then tell Varnish to forward everything to new remote backend, then we flip the DNS switch. In that scenario we are essentially using Varnish to cover us while DNS converges. We've never had to adjust any timeout values, though we are on decent networks. Nathan From news at byteworkshosting.net Wed Jun 8 08:01:00 2011 From: news at byteworkshosting.net (=?iso-8859-15?q?J=F6rg_Wiegleb?=) Date: Wed, 8 Jun 2011 10:01:00 +0200 Subject: varnish POST issue In-Reply-To: References: Message-ID: <201106081001.00340.news@byteworkshosting.net> solved in ticket #849 by nathan. nathan wrote: sub vcl_recv { /* pipeline post requests trac #4124 */ if (req.request == "POST") { return (pipe); } } sub vcl_pipe { /* Force the connection to be closed afterwards so subsequent reqs don't use pipe */ set bereq.http.connection = "close"; } From samuel.roze at gmail.com Wed Jun 8 13:01:51 2011 From: samuel.roze at gmail.com (Samuel ROZE) Date: Wed, 8 Jun 2011 15:01:51 +0200 Subject: Reversing SSH HTTP tunnel Message-ID: Hello, I'm using Varnish as a front reverse-proxy, and it works very well. I would like to add a new functionnality: an SSH HTTP tunnel, using "httptunnel". That's what I made: # hts --forward-port localhost:22 2280 And, in the varnish configuration: ---------------------------------------------------------- [...] backend hts { .host = "127.0.0.1"; .port = "2280"; .first_byte_timeout = 600s; } [...] sub vcl_recv { if (req.http.host ~ "tunnel.example.com") { set req.backend = hts; } [...] } else { set req.backend = nginx; } } ---------------------------------------------------------- So I just want Varnish to redirect HTTP request to "tunnel.example.com" to the HTS backend, but it doesn't work when I run the httptunnel client and I make the SSH connection: $ htc --forward-port 8001 tunnel.example.com:80 $ ssh root at localhost -p 8001 ssh_exchange_identification: Connection closed by remote host ------------------------------------------------------ 17 RxRequest c POST 17 RxURL c /index.html?crap=1307533809 17 RxProtocol c HTTP/1.0 17 RxHeader c Host: tunnel.example.com 17 RxHeader c Content-Length: 102400 17 RxHeader c X-Forwarded-For: 10.6.2.154 17 RxHeader c Cache-Control: max-age=259200 17 RxHeader c Connection: keep-alive 17 VCL_call c recv 17 VCL_return c pass 17 VCL_call c hash 17 VCL_return c hash 17 VCL_call c pass 17 VCL_return c pass 19 BackendOpen b hts 127.0.0.1 35066 127.0.0.1 2280 17 Backend c 19 hts hts 19 TxRequest b POST 19 TxURL b /index.html?crap=1307533809 19 TxProtocol b HTTP/1.0 19 TxHeader b Host: tunnel.example.com 19 TxHeader b Content-Length: 102400 19 TxHeader b X-Forwarded-For: 10.6.2.154, X.X.X.X 19 TxHeader b X-Varnish: 255974636 17 FetchError c backend write error: 11 (Resource temporarily unavailable) 19 BackendClose b hts 17 VCL_call c error 17 VCL_return c deliver 17 VCL_call c deliver 17 VCL_return c deliver 17 TxProtocol c HTTP/1.1 17 TxStatus c 503 17 TxResponse c Service Unavailable 17 TxHeader c Server: Varnish 17 TxHeader c Retry-After: 0 17 TxHeader c Content-Type: text/html; charset=utf-8 17 TxHeader c Content-Length: 418 17 TxHeader c Date: Wed, 08 Jun 2011 11:56:04 GMT 17 TxHeader c X-Varnish: 255974636 17 TxHeader c Age: 5 17 TxHeader c Via: 1.1 varnish 17 TxHeader c Connection: close 17 Length c 418 17 ReqEnd c 255974636 1307534159.114364386 1307534164.113938570 0.000126123 4.999477625 0.000096560 17 SessionClose c error ------------------------------------------------------ But, when I use the HTS port number, it works: $ htc --forward-port 8001 tunnel.example.com:2280 $ ssh root at localhost -p 8001 Do you have some idea? Regards, Samuel ROZE. From christian.albrecht at easynet.com Wed Jun 8 15:39:33 2011 From: christian.albrecht at easynet.com (Christian Albrecht) Date: Wed, 8 Jun 2011 17:39:33 +0200 Subject: Reversing SSH HTTP tunnel In-Reply-To: References: Message-ID: <4DEF97B5.3040609@easynet.com> On 06/08/2011 03:01 PM, Samuel ROZE wrote: > Hello, > > I'm using Varnish as a front reverse-proxy, and it works very well. I > would like to add a new functionnality: an SSH HTTP tunnel, using > "httptunnel". That's what I made: > > # hts --forward-port localhost:22 2280 > > And, in the varnish configuration: > > ---------------------------------------------------------- > [...] > backend hts { > .host = "127.0.0.1"; > .port = "2280"; > .first_byte_timeout = 600s; > } > > [...] > sub vcl_recv { > if (req.http.host ~ "tunnel.example.com") { > set req.backend = hts; > } > [...] > } else { > set req.backend = nginx; > } > } > ---------------------------------------------------------- > > So I just want Varnish to redirect HTTP request to > "tunnel.example.com" to the HTS backend, but it doesn't work when I > run the httptunnel client and I make the SSH connection: > > $ htc --forward-port 8001 tunnel.example.com:80 > $ ssh root at localhost -p 8001 > ssh_exchange_identification: Connection closed by remote host It looks like a tcpwrapper (/etc/hosts.allow, /etc/hosts.deny) issue. Do you have enabled localhost for using sshd? Try to add a line like in /etc/hosts.allow sshd: 127.0.0.1: allow Brgds, Christian From samuel.roze at gmail.com Wed Jun 8 15:53:24 2011 From: samuel.roze at gmail.com (Samuel ROZE) Date: Wed, 8 Jun 2011 17:53:24 +0200 Subject: Reversing SSH HTTP tunnel In-Reply-To: <4DEF97B5.3040609@easynet.com> References: <4DEF97B5.3040609@easynet.com> Message-ID: 2011/6/8 Christian Albrecht : >> $ htc --forward-port 8001 tunnel.example.com:80 >> $ ssh root at localhost -p 8001 >> ssh_exchange_identification: Connection closed by remote host > > It looks like a tcpwrapper (/etc/hosts.allow, /etc/hosts.deny) issue. Do you > have enabled localhost for using sshd? > > Try to add a line like in /etc/hosts.allow > > sshd: 127.0.0.1: allow It can't be that because when I connect my HTTPTunnel Client directly on my tunnel server, it works! But, it doesn't works when the client requests pass by Varnish... Any other idea? Thanks! Samuel ROZE. From jim at freesolutions.net Wed Jun 8 15:54:47 2011 From: jim at freesolutions.net (Jim Bailey) Date: Wed, 08 Jun 2011 16:54:47 +0100 Subject: Add a trailing slash to a directory url Message-ID: Hello, I have been looking through the docs and some of the archives but can not find anything helpful. I need to configure Varnish to add a trailing "/" to directory urls for example www.example.org/somedir with Apache adds a trailing slash to make it www.example.com/somedir/ and the index.php is picked up by default. Is this possible in Varnish and could some one post examples or point to some relevant examples or docs? Regards Jim From kacperw at gmail.com Wed Jun 8 18:26:29 2011 From: kacperw at gmail.com (Kacper Wysocki) Date: Wed, 8 Jun 2011 20:26:29 +0200 Subject: Language of Varnish 3.0 Message-ID: Hello, For those of you who don't know me... I write programs in VCL, like security.vcl (as opposed to simple request flow control).. like some of you I know you're out there.. I'm playing with V3.. wondering if there are debs available.. Too bad it breaks nearly all nontrivial VCL scripts out there.. "+" for strings, and return (restart); # used to be just restart; and std.log("foo"); # used to be just log import std; # only allowed once in entire program.. or is it? sometimes, you need to import again.. these changes taken together make it very difficult (read: not doable) to write portable vcl.. even in the name of language normalization that is painful. I understand the parser is now much easier to handle, so perchance it is time now to apply a little syntactic sugar to the mix? These issues should be obvious to anyone who's looked at V3, and I am merely bringing this up here before release to avoid having the same conversation 1k times with different people. -- http://u.rdir.it/ From phk at phk.freebsd.dk Wed Jun 8 18:38:46 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 08 Jun 2011 18:38:46 +0000 Subject: Language of Varnish 3.0 In-Reply-To: Your message of "Wed, 08 Jun 2011 20:26:29 +0200." Message-ID: <92706.1307558326@critter.freebsd.dk> In message , Kacper Wysocki writes: >these changes taken together make it very difficult (read: not doable) >to write portable vcl.. Unfortunately one cannot make an omelette without breaking the eggs. In general we aim to make such changes to VCL only for major releases (ie: 2.0 -> 3.0) so you shouldn't have trouble until 4.0 again. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From pprocacci at datapipe.com Wed Jun 8 18:39:36 2011 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Wed, 8 Jun 2011 13:39:36 -0500 Subject: Add a trailing slash to a directory url In-Reply-To: References: Message-ID: <20110608183936.GB96249@nat.myhome> Hello, The following isn't tested, but it assumes all your resources end with a 2 or 3 character extension to avoid adding a slash to those resources. ###################################################### if(req.url !~ "\..{2,3}$" && req.url !~ "/$"){ set req.url = regsub(req.url, "$", "/"); } ###################################################### Maybe someone else has something better. ~Paul On Wed, Jun 08, 2011 at 04:54:47PM +0100, Jim Bailey wrote: > Hello, > > I have been looking through the docs and some of the archives but can not > find anything helpful. I need to configure Varnish to add a trailing "/" > to directory urls for example www.example.org/somedir with Apache adds a > trailing slash to make it www.example.com/somedir/ and the index.php is > picked up by default. > > Is this possible in Varnish and could some one post examples or point to > some relevant examples or docs? > > > Regards Jim > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From guly at luv.guly.org Wed Jun 8 18:43:32 2011 From: guly at luv.guly.org (Sandro guly Zaccarini) Date: Wed, 8 Jun 2011 20:43:32 +0200 Subject: Add a trailing slash to a directory url In-Reply-To: <20110608183936.GB96249@nat.myhome> References: <20110608183936.GB96249@nat.myhome> Message-ID: <20110608184332.GE2927@shivaya.guly.org> On Wed, Jun 08, 2011 at 01:39:36PM -0500, Paul A. Procacci wrote: > > Maybe someone else has something better. > I'd go with thorough backend configuration instead sz -- /"\ taste your favourite IT consultant \ / gpg public key http://www.guly.org/guly.asc X / \ From pprocacci at datapipe.com Wed Jun 8 18:53:32 2011 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Wed, 8 Jun 2011 13:53:32 -0500 Subject: Add a trailing slash to a directory url In-Reply-To: <20110608184332.GE2927@shivaya.guly.org> References: <20110608183936.GB96249@nat.myhome> <20110608184332.GE2927@shivaya.guly.org> Message-ID: <20110608185332.GC96249@nat.myhome> You are right. This type of thing is better suited for the backend. ~Paul On Wed, Jun 08, 2011 at 08:43:32PM +0200, Sandro guly Zaccarini wrote: > On Wed, Jun 08, 2011 at 01:39:36PM -0500, Paul A. Procacci wrote: > > > > Maybe someone else has something better. > > > > I'd go with thorough backend configuration instead > > sz > -- > /"\ taste your favourite IT consultant > \ / gpg public key http://www.guly.org/guly.asc > X > / \ > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From kacperw at gmail.com Wed Jun 8 19:16:27 2011 From: kacperw at gmail.com (Kacper Wysocki) Date: Wed, 8 Jun 2011 21:16:27 +0200 Subject: Language of Varnish 3.0 In-Reply-To: <92706.1307558326@critter.freebsd.dk> References: <92706.1307558326@critter.freebsd.dk> Message-ID: On Wed, Jun 8, 2011 at 8:38 PM, Poul-Henning Kamp wrote: > In message , Kacper Wysocki > ?writes: > >>these changes taken together make it very difficult (read: not doable) >>to write portable vcl.. > > Unfortunately one cannot make an omelette without breaking the eggs. Are you referring to all those vcl scripts out there as the eggs? If you are the one making the omlette I am the one feeding the unhappy hens... True, you can not change syntax without changing syntax... but you could add syntactic sugar to make things more managable. This was not possible when going from posix regex to pcre without supporting both, which would be fugly, but is reasonably managable with these new changes, unless we call all the little eggs "legacy" and the stance on legacy is like the stance against terrorists - no compromise. > In general we aim to make such changes to VCL only for major > releases (ie: 2.0 -> 3.0) so you shouldn't have trouble until > 4.0 again. Or 3.1, as the case may be? 0K out, still just clearing the air. From phk at phk.freebsd.dk Wed Jun 8 19:29:28 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 08 Jun 2011 19:29:28 +0000 Subject: Language of Varnish 3.0 In-Reply-To: Your message of "Wed, 08 Jun 2011 21:16:27 +0200." Message-ID: <92943.1307561368@critter.freebsd.dk> In message , Kacper Wysocki writes: >Or 3.1, as the case may be? No, that is not a major version, that is only a minor version. Listen, I'm not doing this kind of stuff to hurt anybody. Backwards compatibility and "syntactic sugare" is always a tradeoff of where you want to spend your effort. With the limited amount of developers and testers we have available in Varnish, I have prioritized getting important stuff done, which in this particular context included VMODS which caused a pretty extensive rewrite of the VCL compiler. In all likelyhood, some of your VCL scripts should be converted to a VMOD (Sorry: you cannot write VMOD's in VCL yet) and once you realize what options that gives you, I think you will see the VCL improvements from 2.X to 3.X as quite sensible. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From l at lrowe.co.uk Wed Jun 8 21:51:45 2011 From: l at lrowe.co.uk (Laurence Rowe) Date: Wed, 8 Jun 2011 22:51:45 +0100 Subject: Reversing SSH HTTP tunnel In-Reply-To: References: <4DEF97B5.3040609@easynet.com> Message-ID: On 8 June 2011 16:53, Samuel ROZE wrote: > 2011/6/8 Christian Albrecht : >>> $ htc --forward-port 8001 tunnel.example.com:80 >>> $ ssh root at localhost -p 8001 >>> ssh_exchange_identification: Connection closed by remote host >> >> It looks like a tcpwrapper (/etc/hosts.allow, /etc/hosts.deny) issue. Do you >> have enabled localhost for using sshd? >> >> Try to add a line like in /etc/hosts.allow >> >> sshd: 127.0.0.1: allow > > It can't be that because when I connect my HTTPTunnel Client directly > on my tunnel server, it works! But, it doesn't works when the client > requests pass by Varnish... Any other idea? You will want to at least return (pipe); to forward ssh connections. I really have no idea how Varnish will cope with that sort of traffic or whether you'll have to up the timeouts to a ridiculous degree (and perhaps cause stability problems elsewhere). Laurence From ghstridr at gmail.com Wed Jun 8 23:51:49 2011 From: ghstridr at gmail.com (Mike Gracy) Date: Wed, 8 Jun 2011 16:51:49 -0700 Subject: How does one start varnishd to listen on multiple ports? Message-ID: I have -a 0.0.0.0:8080 -a 0.0.0.0:9010 in the /etc/default/varnish but it will only listen on 9010. DAEMON_OPTS="-a 0.0.0.0:8080 \ -a 0.0.0.0:9010 \ -T localhost:2000 \ -f /etc/varnish/default.vcl \ -n $INSTANCE" I have the same problem when running it from the command line. It only starts listening on the last port listed. If I start two separate instances, changing the port and -n values, it works. From pprocacci at datapipe.com Wed Jun 8 23:57:20 2011 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Wed, 8 Jun 2011 18:57:20 -0500 Subject: How does one start varnishd to listen on multiple ports? In-Reply-To: References: Message-ID: <20110608235720.GF96249@nat.myhome> >From the documentation: ########################################### -a address[:port][,address[:port][...] Listen for client requests on the specified address and port. The address can be a host name (alocalhosta), an IPv4 dot- ted-quad (a127.0.0.1a), or an IPv6 address enclosed in square brackets (a[::1]a). If address is not specified, varnishd will listen on all available IPv4 and IPv6 interfaces. If port is not specified, the default HTTP port as listed in /etc/services is used. Multiple listening addresses and ports can be specia fied as a whitespace- or comma-separated list. ########################################### What you want is to separate ports by a comma. Each addition 'a' flag replaces the first. ~Paul On Wed, Jun 08, 2011 at 04:51:49PM -0700, Mike Gracy wrote: > I have -a 0.0.0.0:8080 -a 0.0.0.0:9010 in the /etc/default/varnish but > it will only listen on 9010. > > DAEMON_OPTS="-a 0.0.0.0:8080 \ > -a 0.0.0.0:9010 \ > -T localhost:2000 \ > -f /etc/varnish/default.vcl \ > -n $INSTANCE" > > I have the same problem when running it from the command line. It > only starts listening on the last port listed. > If I start two separate instances, changing the port and -n values, it works. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From pprocacci at datapipe.com Wed Jun 8 23:58:47 2011 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Wed, 8 Jun 2011 18:58:47 -0500 Subject: How does one start varnishd to listen on multiple ports? In-Reply-To: <20110608235720.GF96249@nat.myhome> References: <20110608235720.GF96249@nat.myhome> Message-ID: <20110608235847.GG96249@nat.myhome> I just realized I stated to separate ports by a comma, but what I meant is address. i.e. 127.0.0.1:1234,127.0.0.1:5678 ~Paul On Wed, Jun 08, 2011 at 06:57:20PM -0500, Paul A. Procacci wrote: > From the documentation: > > ########################################### > -a address[:port][,address[:port][...] > Listen for client requests on the specified address and port. > The address can be a host name (alocalhosta), an IPv4 dot- > ted-quad (a127.0.0.1a), or an IPv6 address enclosed in square > brackets (a[::1]a). If address is not specified, varnishd will > listen on all available IPv4 and IPv6 interfaces. If port is > not specified, the default HTTP port as listed in /etc/services > is used. Multiple listening addresses and ports can be specia > fied as a whitespace- or comma-separated list. > ########################################### > > What you want is to separate ports by a comma. Each addition 'a' flag replaces the first. > > ~Paul > > On Wed, Jun 08, 2011 at 04:51:49PM -0700, Mike Gracy wrote: > > I have -a 0.0.0.0:8080 -a 0.0.0.0:9010 in the /etc/default/varnish but > > it will only listen on 9010. > > > > DAEMON_OPTS="-a 0.0.0.0:8080 \ > > -a 0.0.0.0:9010 \ > > -T localhost:2000 \ > > -f /etc/varnish/default.vcl \ > > -n $INSTANCE" > > > > I have the same problem when running it from the command line. It > > only starts listening on the last port listed. > > If I start two separate instances, changing the port and -n values, it works. > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From ghstridr at gmail.com Thu Jun 9 00:04:58 2011 From: ghstridr at gmail.com (Mike Gracy) Date: Wed, 8 Jun 2011 17:04:58 -0700 Subject: How does one start varnishd to listen on multiple ports? In-Reply-To: <20110608235847.GG96249@nat.myhome> References: <20110608235720.GF96249@nat.myhome> <20110608235847.GG96249@nat.myhome> Message-ID: Thanks, just figured that out after sending that. I keep looking at the website docs, which don't help. On a related note you have to also have unique -n instances when doing this (the varnishd told me so). So this is what I now have in /etc/default/varnish: DAEMON_OPTS="-a 0.0.0.0:8080,0.0.0.0:9010 \ -T localhost:2000 \ -f /etc/varnish/default.vcl \ -n $INSTANCE1 \ -n $INSTANCE2" On Wed, Jun 8, 2011 at 4:58 PM, Paul A. Procacci wrote: > I just realized I stated to separate ports by a comma, but what I meant is address. > > i.e. ?127.0.0.1:1234,127.0.0.1:5678 > > ~Paul > > On Wed, Jun 08, 2011 at 06:57:20PM -0500, Paul A. Procacci wrote: >> From the documentation: >> >> ########################################### >> ? ? ? ?-a address[:port][,address[:port][...] >> ? ? ? ? ? ? ? Listen for client requests on the specified ?address ?and ?port. >> ? ? ? ? ? ? ? The ?address ?can ?be ?a ?host ?name (alocalhosta), an IPv4 dot- >> ? ? ? ? ? ? ? ted-quad (a127.0.0.1a), or an IPv6 address ?enclosed ?in ?square >> ? ? ? ? ? ? ? brackets ?(a[::1]a). ?If address is not specified, varnishd will >> ? ? ? ? ? ? ? listen on all available IPv4 and IPv6 interfaces. ? If ?port ?is >> ? ? ? ? ? ? ? not ?specified, the default HTTP port as listed in /etc/services >> ? ? ? ? ? ? ? is used. ?Multiple listening addresses and ports can ?be ?specia >> ? ? ? ? ? ? ? fied as a whitespace- or comma-separated list. >> ########################################### >> >> What you want is to separate ports by a comma. ?Each addition 'a' flag replaces the first. >> >> ~Paul >> >> On Wed, Jun 08, 2011 at 04:51:49PM -0700, Mike Gracy wrote: >> > I have -a 0.0.0.0:8080 -a 0.0.0.0:9010 in the /etc/default/varnish but >> > it will only listen on 9010. >> > >> > ?DAEMON_OPTS="-a 0.0.0.0:8080 \ >> > ? ? ? ? ? ? ? -a 0.0.0.0:9010 \ >> > ? ? ? ? ? ? ? -T localhost:2000 \ >> > ? ? ? ? ? ? ? -f /etc/varnish/default.vcl \ >> > ? ? ? ? ? ? ? -n $INSTANCE" >> > >> > I have the same problem when running it from the command line. ?It >> > only starts listening on the last port listed. >> > If I start two separate instances, changing the port and -n values, it works. >> > >> > _______________________________________________ >> > varnish-misc mailing list >> > varnish-misc at varnish-cache.org >> > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > ________________________________ > > This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. > From paulo at aliancaproject.com Thu Jun 9 00:13:08 2011 From: paulo at aliancaproject.com (Paulo Paracatu) Date: Wed, 8 Jun 2011 21:13:08 -0300 Subject: Conditional backend requests on repository branch experimental-ims In-Reply-To: <4DEDC12E.9010702@uplex.de> References: <4DEDC12E.9010702@uplex.de> Message-ID: Hello, Yesterday I finally had the time to download this and make the RPM to install in my caches servers. I did not have the time to tune the VCL very much yet, or test it, but from what I did until now, I'm getting this result: Monitoring eth0... (press CTRL-C to stop) rx: 17985.20 kB/s 51195 p/s *tx: 82989.47 kB/s 69083 p/s* 0+00:12:14 Hitrate ratio: 10 72 72 Hitrate avg: 0.7245 0.7202 0.7202 758943 1059.66 1033.98 client_conn - Client connections accepted 2006428 2647.15 2733.55 client_req - Client requests received 1200744 1622.48 1635.89 cache_hit - Cache hits 29541 40.99 40.25 cache_hitpass - Cache hits for pass 491576 620.80 669.72 cache_miss - Cache misses 64840 110.96 88.34 backend_conn - Backend conn. success 50 2.00 0.07 backend_fail - Backend conn. failures 748067 927.70 1019.16 backend_reuse - Backend conn. reuses 343 0.00 0.47 backend_toolate - Backend conn. was closed 748505 926.70 1019.76 backend_recycle - Backend conn. recycles 101 0.00 0.14 backend_retry - Backend conn. retry 60 0.00 0.08 fetch_head - Fetch head 463996 462.85 632.15 fetch_length - Fetch with Length 91229 147.95 124.29 fetch_chunked - Fetch chunked 4732 9.00 6.45 fetch_close - Fetch wanted close 12 0.00 0.02 fetch_failed - Fetch failed * 252531 416.87 344.05 fetch_304 - Fetch no body (304)* The default ttl is set to 60s, and people are able to refresh the cache using the no-cache (ctrl f5). Ah, before the patch I was hitting much more RX than now, so I guess it is really working and helping. If you have any tips, I'd be glad to test. Thanks you! ;) 2011/6/7 Geoff Simmons > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Hello all, > > The code for the proposed conditional backend request feature (usually > If-Modified-Since requests to backends) is now up to date with the > current trunk and available on an experimental branch of the source > repository called experimental-ims. > > On an updated git working tree you can say: > > $ git branch -a > > ... and you should see something like 'remotes/origin/experimental-ims' > in the list. Then to work with the branch: > > $ git checkout -b experimental-ims remotes/origin/experimental-ims > > To switch back to the master branch: > > $ git checkout master > > The feature is documented here: > > http://www.varnish-cache.org/trac/wiki/BackendConditionalRequests > > Please note that this code is experimental, and will not be included in > the upcoming 3.0.0 release. The way that this works has not been changed > from the previous versions distributed as patches, but will likely be > changed in future versions. > > Testing the mainline 3.0 version is the highest priority for the Varnish > team right now, so feedback on that is most important. If you also get a > chance to test the IMS feature, it will be a great help toward getting > an "official" version settled, so let us know how it goes. > > > Best, > Geoff > - -- > UPLEX Systemoptimierung > Schwanenwik 24 > 22087 Hamburg > http://uplex.de/ > Mob: +49-176-63690917 > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.14 (Darwin) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQIcBAEBCAAGBQJN7cEuAAoJEOUwvh9pJNURH9cQAKhwMbFRc7XBBwro2oKE57c4 > jRH36dZMV9/PgAsgg4LlCWbt3+ceG9607gigHocY4VY6viF4ppsK3daxGYL4qxMj > st4hwJwyZculNDr3Fbg/du13/o4Dexl5Rhy49g1AtxgDCkioT4xo5Y+9OPfI2nAq > KbiThND+yThCcaOUDssZZsp9ed3JEPD71SNE5Xh0JupXTgXc8RTU2n32XYZE6HwF > re4Lfpk1RA9xKFg+S/Dz2zAKvJTFIU9liV9abrdQTnHyhsloDyf04QLg7bVnZ28c > 5PXWQlo2PSxujcLJvr9uKr5CaS+CNWRADd/8DIH6q2tf6pMX3VfwryjbRh2KpT4I > /GHYEJsyCk0nUNNEwzTPXhZE6zT9yvRPU/qhx09fOsg+X7FEDyZJhTLvgiqIKaFd > OESrgyQE0wBu/5O+fg7dY+U8rPzQI5aKZWrBYLHIwBA4puCUpq1Fm2+/FZY6vGd5 > GVDbCMNF+Q4HADfYAQZG8X753DQ5O0ZzVvlKP2hmh+r13UTcx45wmFiXitZJFdy4 > BRuNlsJAt13qIgP/I38hCJEay6OgDIyhRtyTkX/aZckHSX/rI/4rjqN7bsHOMIOE > 33LHwP9VV0OaDPhov9iKKF6/aMqCJ6w5mRUp+qDHdePgQv2uER/HC3qBEtpErB00 > 4LsJb64hxNQds4/nMjCs > =n6yS > -----END PGP SIGNATURE----- > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Thu Jun 9 06:32:27 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 09 Jun 2011 08:32:27 +0200 Subject: Language of Varnish 3.0 In-Reply-To: (Kacper Wysocki's message of "Wed, 8 Jun 2011 20:26:29 +0200") References: Message-ID: <878vtbwm4k.fsf@qurzaw.varnish-software.com> ]] Kacper Wysocki Hi, | I'm playing with V3.. wondering if there are debs available.. Yes, they're linked to from the normal download pages on varnish-cache.org | Too bad it breaks nearly all nontrivial VCL scripts out there.. | | "+" for strings, and Yes, people were continually confused about how to concatenate strings so we changed it. | return (restart); # used to be just restart; This was changed in 2.1 which supports both syntaxes. | and | std.log("foo"); # used to be just log Yes, there was no reason for it to stay in the main namespace. | import std; # only allowed once in entire program.. or is it? | sometimes, you need to import again.. Why would you need to reimport? | these changes taken together make it very difficult (read: not doable) | to write portable vcl.. | even in the name of language normalization that is painful. | I understand the parser is now much easier to handle, so perchance it | is time now to apply a little syntactic sugar to the mix? We're not going to make any significant changes to the VCL compiler less than a week before release. Cheers, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From samuel.roze at gmail.com Thu Jun 9 09:23:24 2011 From: samuel.roze at gmail.com (Samuel ROZE) Date: Thu, 9 Jun 2011 11:23:24 +0200 Subject: Reversing SSH HTTP tunnel In-Reply-To: References: <4DEF97B5.3040609@easynet.com> Message-ID: It works using the return(pipe) ! Thanks for you reply! -- Samuel 2011/6/8 Laurence Rowe : > On 8 June 2011 16:53, Samuel ROZE wrote: >> 2011/6/8 Christian Albrecht : >>>> $ htc --forward-port 8001 tunnel.example.com:80 >>>> $ ssh root at localhost -p 8001 >>>> ssh_exchange_identification: Connection closed by remote host >>> >>> It looks like a tcpwrapper (/etc/hosts.allow, /etc/hosts.deny) issue. Do you >>> have enabled localhost for using sshd? >>> >>> Try to add a line like in /etc/hosts.allow >>> >>> sshd: 127.0.0.1: allow >> >> It can't be that because when I connect my HTTPTunnel Client directly >> on my tunnel server, it works! But, it doesn't works when the client >> requests pass by Varnish... Any other idea? > > You will want to at least return (pipe); to forward ssh connections. I > really have no idea how Varnish will cope with that sort of traffic or > whether you'll have to up the timeouts to a ridiculous degree (and > perhaps cause stability problems elsewhere). > > Laurence > From dennis.hendriksen at kalooga.com Fri Jun 10 06:32:11 2011 From: dennis.hendriksen at kalooga.com (Dennis Hendriksen) Date: Fri, 10 Jun 2011 08:32:11 +0200 Subject: thread pool issues Message-ID: <1307687531.1937.12.camel@kalooga-dennis> Hi there, We're running Varnish 2.0.6 on a dual quad core server which is doing about 500 req/s with a 97% hit ratio serving mostly images with. When we increase the load to about 800 req/s than we encounter two problems that seem to be related with the thread pool increase. When we double the varnish load then the "N worker threads limited" increases rapidly (100k+) while the "N worker threads created" does not increase (8 pools, min pool size 25, max pool size 1000). Varnish is unresponsive and client connections hang. At other times we see the number of worker threads increasing but again connections 'hang' while Varnish doesn't show any dropped connections (only overflows). The backends are fast and can easily handle such kind of load. I've posted our Varnish config below. Is there anyone out there who knows how to solve our problem? Any help would be greatly appreciated! Greetings, Dennis varnishadm -T localhost:6082 param.show accept_fd_holdoff 50 [ms] acceptor default (epoll, poll) auto_restart on [bool] backend_http11 on [bool] between_bytes_timeout 60.000000 [s] cache_vbe_conns off [bool] cc_command "exec cc -fpic -shared -Wl,-x -o %o %s" cli_buffer 8192 [bytes] cli_timeout 5 [seconds] client_http11 off [bool] clock_skew 10 [s] connect_timeout 0.400000 [s] default_grace 10 default_ttl 120 [seconds] diag_bitmap 0x0 [bitmap] err_ttl 0 [seconds] esi_syntax 0 [bitmap] fetch_chunksize 128 [kilobytes] first_byte_timeout 60.000000 [s] group varnish (103) listen_address :80 listen_depth 1024 [connections] log_hashstring off [bool] log_local_address off [bool] lru_interval 2 [seconds] max_esi_includes 5 [includes] max_restarts 4 [restarts] obj_workspace 8192 [bytes] overflow_max 100 [%] ping_interval 3 [seconds] pipe_timeout 60 [seconds] prefer_ipv6 off [bool] purge_dups on [bool] purge_hash on [bool] rush_exponent 3 [requests per request] send_timeout 600 [seconds] sess_timeout 5 [seconds] sess_workspace 16384 [bytes] session_linger 50 [ms] session_max 100000 [sessions] shm_reclen 255 [bytes] shm_workspace 8192 [bytes] srcaddr_hash 1049 [buckets] srcaddr_ttl 0 [seconds] thread_pool_add_delay 20 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_fail_delay 200 [milliseconds] thread_pool_max 1000 [threads] thread_pool_min 25 [threads] thread_pool_purge_delay 1000 [milliseconds] thread_pool_stack unlimited [bytes] thread_pool_timeout 300 [seconds] thread_pools 8 [pools] user varnish (101) vcl_trace off [bool] From kristian at varnish-software.com Fri Jun 10 14:29:56 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Fri, 10 Jun 2011 16:29:56 +0200 Subject: thread pool issues In-Reply-To: <1307687531.1937.12.camel@kalooga-dennis> References: <1307687531.1937.12.camel@kalooga-dennis> Message-ID: <20110610142956.GG3015@freud.kly.no> Greetings, On Fri, Jun 10, 2011 at 08:32:11AM +0200, Dennis Hendriksen wrote: > We're running Varnish 2.0.6 on a dual quad core server which is doing > about 500 req/s with a 97% hit ratio serving mostly images with. When we > increase the load to about 800 req/s than we encounter two problems that > seem to be related with the thread pool increase. You really should see if you can't move to at least Varnish 2.1.5. > When we double the varnish load then the "N worker threads limited" > increases rapidly (100k+) while the "N worker threads created" does not > increase (8 pools, min pool size 25, max pool size 1000). Varnish is > unresponsive and client connections hang. That'll give you 200 threads at startup. I would typically recommend something closer to minimum 500, pools 2 and max 5000. You also want to reduce the thread_pool_add_delay from the (2.0.6) default 20ms to 2ms for instance. That will limit the rate that threads are started at, and 20ms is often way too slow. How many connections (not requests) are you doing during these tests? > At other times we see the number of worker threads increasing but again > connections 'hang' while Varnish doesn't show any dropped connections > (only overflows). Do you use keep-alive and long-lasting connections? You may want to see if reducing session_linger helps. Are you testing with real traffic or synthetic tests? If possible, varnishstat -1 output would be useful. - Kristian From roberto.fernandezcrisial at gmail.com Fri Jun 10 15:10:27 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Fri, 10 Jun 2011 12:10:27 -0300 Subject: Varnish 3.0 Release Party :: Buenos Aires, Argentina Message-ID: Here is some update information for #V3RP in Buenos Aires, Argentina! http://bit.ly/ll7o2t Enjoy us! Some stuff at my blog (spanish),http://bit.ly/lNJ54D Regards, Roberto @rofc -------------- next part -------------- An HTML attachment was scrubbed... URL: From nfn at gmx.com Fri Jun 10 16:03:10 2011 From: nfn at gmx.com (Nuno Neves) Date: Fri, 10 Jun 2011 16:03:10 +0000 Subject: Purgin when obj.ttl = 0 Message-ID: <20110610160310.278240@gmx.com> Hello, I run varnish 2.1 and my front page is cached for 300s. I would like to purge this page when the object expires. I placed this in vcl_hit: sub vcl_hit { if (obj.ttl < 1s) { set obj.ttl = 0s; error 200 "Purged."; } } Is this correct? Can obj.ttl be negative or when it's expired will be = 0? Thanks Nuno -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.fernandezcrisial at gmail.com Fri Jun 10 16:11:19 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Fri, 10 Jun 2011 13:11:19 -0300 Subject: Purgin when obj.ttl = 0 In-Reply-To: <20110610160310.278240@gmx.com> References: <20110610160310.278240@gmx.com> Message-ID: Nuno, Once the object is purged, its TTL it set to 300s on the next request. I think you should use something like this: sub vcl_hit { if (obj.ttl < 1s) { purge_url(req.url); } } Regards, Roberto O. Fern?ndez Crisial @rofc On Fri, Jun 10, 2011 at 1:03 PM, Nuno Neves wrote: > Hello, > > I run varnish 2.1 and my front page is cached for 300s. > I would like to purge this page when the object expires. > > I placed this in vcl_hit: > > sub vcl_hit { > if (obj.ttl < 1s) { > set obj.ttl = 0s; > error 200 "Purged."; > } > } > > Is this correct? > Can obj.ttl be negative or when it's expired will be = 0? > > Thanks > > Nuno > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan.b.lynch at gmail.com Fri Jun 10 17:17:41 2011 From: ryan.b.lynch at gmail.com (Ryan Lynch) Date: Fri, 10 Jun 2011 10:17:41 -0700 Subject: Config reloading and persistent client connections. Message-ID: When switching to a new config, how does Varnish handle a new request that re-uses an existing (persistent, keep-alive) client connection? In other words, given a persistent connection, does Varnish handle further requests using the old config or the new config? Based on "man varnishd", I believe that Varnish will always use the new config, regardless of keep-alives. Here's the quote that makes me think so, from the section detailing the "vcl.use configname" option: "Start using the configuration specified by configname for all new requests. Existing requests will con? tinue using whichever configuration was in use when they arrived." If I'm wrong about this, I'd appreciate somebody letting me know. Also, is it possible to change this behavior? Can Varnish handle config reloads similar to how Apache's "graceful" restarts work? (I.e., continue using the old config for new requests on existing connections, so that any given client doesn't experience the config change until the connection breaks.) -Ryan From guly at luv.guly.org Sat Jun 11 14:47:14 2011 From: guly at luv.guly.org (Sandro guly Zaccarini) Date: Sat, 11 Jun 2011 16:47:14 +0200 Subject: mascotte name Message-ID: <20110611144714.GG1188@shivaya.guly.org> I think we should know the name of the bunny. Does he/she have one? sz -- /"\ taste your favourite IT consultant \ / gpg public key http://www.guly.org/guly.asc X / \ From cato.aune at aunenett.st.no Sat Jun 11 22:24:08 2011 From: cato.aune at aunenett.st.no (Cato Aune) Date: Sun, 12 Jun 2011 00:24:08 +0200 Subject: Cryillic letters in rewrite? Message-ID: <1307831048.2187.16.camel@cato-laptop> Hi, is it possible to rewrite a request for http://company.ru/ to http://company.ru/company-?????? ? When I try the following in default.vcl set req.url = "/company-??????"; I get an error message when trying to access http://company.ru The requested URL /company-?77777720?77777640?77777720?77777676?77777721?77777601?77777721?77777601?77777720?77777670?77777721?77777617 was not found on this server. varnishlog shows 14 TxURL b /company-?77777720?77777640?77777720?77777676?77777721?77777601?77777721?77777601?77777720?77777670?77777721?77777617 varnishd -C shows this VRT_l_req_url(sp, VRT_regsub(sp, 0, VRT_r_req_url(sp), VGC_re_1, "/company-\37777777720\37777777640\37777777720\37777777676\37777777721 \37777777601\37777777721\37777777601\37777777720\37777777670\37777777721 \37777777617"), vrt_magic_string_end); It seems that /company-%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D1%8F works in the browser, but if I try to use that encondig in the .vcl file, I get Message from VCC-compiler: Control character in %xx escape (input Line 24 Pos 41) set req.url = "/company-%D0%A0%D0%BE%D1%81%D1%81%D0%B8% D1%8F"; ----------------------------------------###----------------------------------- Running VCC-compiler failed, exit 1 VCL compilation failed Any suggestions on how to make this work? Regards, Cato From ruben at varnish-software.com Sat Jun 11 23:00:25 2011 From: ruben at varnish-software.com (=?ISO-8859-1?Q?Rub=E9n_Romero?=) Date: Sun, 12 Jun 2011 01:00:25 +0200 Subject: mascotte name In-Reply-To: <20110611144714.GG1188@shivaya.guly.org> References: <20110611144714.GG1188@shivaya.guly.org> Message-ID: On Sat, Jun 11, 2011 at 4:47 PM, Sandro guly Zaccarini wrote: > I think we should know the name of the bunny. Does he/she have one? Yeah, we call it Claudia. What do you think? :) > > sz > -- > ?/"\ ? taste your favourite IT consultant > ?\ / ? gpg public key http://www.guly.org/guly.asc > ? X > ?/ \ > > -- Best regards, -- Rub?n Romero | Varnish Software e-mail ruben at varnish-software.com?/?skype: ruben_varnish P: +47 21 98 92?62 /?M: +47 95 96 40 88 Online Sales Chat: http://www.varnish-software.com/contact-us ====================== ?Varnish makes websites fly! ====================== www.varnish-software.com twitter.com/varnishsoftware linkedin.com/companies/varnish-software Want to learn more about Varnish, its features, get tips and news? http://www.varnish-software.com/whitepapers http://www.varnish-software.com/about/newsletter From guly at luv.guly.org Mon Jun 13 15:10:32 2011 From: guly at luv.guly.org (Sandro guly Zaccarini) Date: Mon, 13 Jun 2011 17:10:32 +0200 Subject: mascotte name In-Reply-To: References: <20110611144714.GG1188@shivaya.guly.org> Message-ID: <20110613151032.GN1188@shivaya.guly.org> On Sun, Jun 12, 2011 at 01:00:25AM +0200, Rub?n Romero wrote: > > Yeah, we call it Claudia. What do you think? > great, i like it :) thanks sz -- /"\ taste your favourite IT consultant \ / gpg public key http://www.guly.org/guly.asc X / \ From dennis.hendriksen at kalooga.com Tue Jun 14 15:20:36 2011 From: dennis.hendriksen at kalooga.com (Dennis Hendriksen) Date: Tue, 14 Jun 2011 17:20:36 +0200 Subject: thread pool issues In-Reply-To: <20110610142956.GG3015@freud.kly.no> References: <1307687531.1937.12.camel@kalooga-dennis> <20110610142956.GG3015@freud.kly.no> Message-ID: <1308064836.9921.83.camel@kalooga-dennis> Hi Kristian, Thank you for your suggestions. We've upgraded our Varnish config to 2.1.5 which decreases the thread_pool_add_delay from 20ms to 2ms. I've included a varnishstat listing below. The numbers reflect live testing, (our experiences with synthetic tests is that it is very hard to imitate real life behavior) > I would typically recommend something closer to minimum 500, pools 2 and > max 5000. Currently we use 8 pools because the server has 2x4 CPU cores. Is there an advantage to use less pools than the number of CPU cores? When we increase the number of threads the problem with "N worker threads limited" is solved! :-) > How many connections (not requests) are you doing during these tests? ls -1 /proc//fd | wc -l gives us ~1300 (single load) and ~2600 (double load) file descriptors (=connections?). > Do you use keep-alive and long-lasting connections? You may want to see > if reducing session_linger helps. Requests mostly arrive from web browsers. netstat -tna | wc -l ~12000 tcp connections (single load) Unfortunately Varnish, after facing double load, now gets very 'unresponsive' after a while. Client requests are not answered by varnish resulting in long waiting times (10+ seconds) or timeouts. We do not have bandwidth issues. Is it possible that in our use case we've reached the limit of what Varnish can handle? Greetings and thanks for the help so far! Dennis varnishstat -1 client_conn 696307 177.40 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 965174 245.90 Client requests received cache_hit 925943 235.91 Cache hits cache_hitpass 5 0.00 Cache hits for pass cache_miss 39125 9.97 Cache misses backend_conn 4568 1.16 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 3 0.00 Backend conn. failures backend_reuse 34683 8.84 Backend conn. reuses backend_toolate 79 0.02 Backend conn. was closed backend_recycle 34768 8.86 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 0 0.00 Fetch head fetch_length 24818 6.32 Fetch with Length fetch_chunked 14426 3.68 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 1 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed n_sess_mem 2235 . N struct sess_mem n_sess 1787 . N struct sess n_object 34379 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 34516 . N struct objectcore n_objecthead 22424 . N struct objecthead n_smf 0 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 6 . N struct vbe_conn n_wrk 280 . N worker threads n_wrk_create 280 0.07 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 9693 2.47 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 0 0.00 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 4 . N backends n_expired 385 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 370058 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 815230 207.70 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 696245 177.39 Total Sessions s_req 965174 245.90 Total Requests s_pipe 4 0.00 Total pipe s_pass 120 0.03 Total pass s_fetch 39245 10.00 Total fetch s_hdrbytes 285675067 72783.46 Total header bytes s_bodybytes 10667879292 2717931.03 Total body bytes sess_closed 30597 7.80 Session Closed sess_pipeline 1238 0.32 Session Pipeline sess_readahead 537 0.14 Session Read Ahead sess_linger 955973 243.56 Session Linger sess_herd 891554 227.15 Session herd shm_records 39223429 9993.23 SHM records shm_writes 4022999 1024.97 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 1578 0.40 SHM MTX contention shm_cycles 15 0.00 SHM cycles through buffer sm_nreq 0 0.00 allocator requests sm_nobj 0 . outstanding allocations sm_balloc 0 . bytes allocated sm_bfree 0 . bytes free sma_nreq 71633 18.25 SMA allocator requests sma_nobj 66455 . SMA outstanding allocations sma_nbytes 608883602 . SMA outstanding bytes sma_balloc 2206748168 . SMA bytes allocated sma_bfree 1597864566 . SMA bytes free sms_nreq 0 0.00 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 0 . SMS bytes allocated sms_bfree 0 . SMS bytes freed backend_req 39247 10.00 Backend requests made n_vcl 2 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 1 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 1 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 0 0.00 N objects tested n_purge_re_test 0 0.00 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 0 0.00 HCB Lookups without lock hcb_lock 0 0.00 HCB Lookups with lock hcb_insert 0 0.00 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 3925 1.00 Client uptime backend_retry 2 0.00 Backend conn. retry dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 0 0.00 Fetch no body (304) On Fri, 2011-06-10 at 16:29 +0200, Kristian Lyngstol wrote: > Greetings, > > On Fri, Jun 10, 2011 at 08:32:11AM +0200, Dennis Hendriksen wrote: > > We're running Varnish 2.0.6 on a dual quad core server which is doing > > about 500 req/s with a 97% hit ratio serving mostly images with. When we > > increase the load to about 800 req/s than we encounter two problems that > > seem to be related with the thread pool increase. > > You really should see if you can't move to at least Varnish 2.1.5. > > > When we double the varnish load then the "N worker threads limited" > > increases rapidly (100k+) while the "N worker threads created" does not > > increase (8 pools, min pool size 25, max pool size 1000). Varnish is > > unresponsive and client connections hang. > > That'll give you 200 threads at startup. > > I would typically recommend something closer to minimum 500, pools 2 and > max 5000. > > You also want to reduce the thread_pool_add_delay from the (2.0.6) > default 20ms to 2ms for instance. That will limit the rate that threads > are started at, and 20ms is often way too slow. > > How many connections (not requests) are you doing during these tests? > > > At other times we see the number of worker threads increasing but again > > connections 'hang' while Varnish doesn't show any dropped connections > > (only overflows). > > Do you use keep-alive and long-lasting connections? You may want to see > if reducing session_linger helps. > > Are you testing with real traffic or synthetic tests? > > If possible, varnishstat -1 output would be useful. > > - Kristian > From shib4u at gmail.com Thu Jun 16 11:43:40 2011 From: shib4u at gmail.com (Shibashish) Date: Thu, 16 Jun 2011 17:13:40 +0530 Subject: Alert on certain error like 503 Message-ID: Hi, Can varnish be made to send out an alert (mail) in case of an error like 503? I do have a monitoring solution but i can't add all the urls for monitoring. Sometimes varnish gives a 503 and i need to be alerted during that time. Another option can be to keep looking for 503 errors. log them as they occur and keep watching that file. ShiB. while ( ! ( succeed = try() ) ); -------------- next part -------------- An HTML attachment was scrubbed... URL: From robrecc at gmail.com Thu Jun 16 12:24:22 2011 From: robrecc at gmail.com (Robert Recchia) Date: Thu, 16 Jun 2011 08:24:22 -0400 Subject: varnish syntax Message-ID: I run word press and munin on the same server - i don't want varnish caching anything /munin so in older varnish releases all i did was add this to the vcl file sub vcl_recv { if (req.url ~ "munin") { pass; } it seems varnish-2.1.5-1 does not like this syntax and wont start. Does anyone know the equivilant to what it would be in this version -- Robert Recchia From enno at tty.nl Thu Jun 16 12:35:19 2011 From: enno at tty.nl (Enno van Amerongen) Date: Thu, 16 Jun 2011 14:35:19 +0200 Subject: varnish syntax In-Reply-To: References: Message-ID: <4DF9F887.3020309@tty.nl> try sub vcl_recv { if (req.url ~ "^/munin") { return (pass); } } http://www.varnish-cache.org/docs/2.1/reference/vcl.html Robert Recchia wrote: > I run word press and munin on the same server - i don't want varnish > caching anything /munin so in older varnish releases all i did was add > this to the vcl file > > sub vcl_recv { > > if (req.url ~ "munin") { > > pass; > > } > > > it seems varnish-2.1.5-1 does not like this syntax and wont start. > Does anyone know the equivilant to what it would be in this version > > > > From geoff at uplex.de Thu Jun 16 12:45:25 2011 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 16 Jun 2011 14:45:25 +0200 Subject: Alert on certain error like 503 In-Reply-To: References: Message-ID: <4DF9FAE5.1080200@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 06/16/11 01:43 PM, Shibashish wrote: > > Can varnish be made to send out an alert (mail) in case of an error like > 503? I do have a monitoring solution but i can't add all the urls for > monitoring. Sometimes varnish gives a 503 and i need to be alerted during > that time. > > Another option can be to keep looking for 503 errors. log them as they occur > and keep watching that file. If you just want to know that 503s are happening at all, you can monitor varnishlog for "NNN TxStatus c 503", although that won't tell you which URL got the 503. For that, it gets a bit tricky, you'd have to first watch for "NNN RxURL c /some/path", and then watch for "NNN c TxStatus" with the same NNN, emitting an alert for 503 and forgetting the URL otherwise. At any rate, monitoring the varnishlog is the way to do this, we wouldn't the Varnish demon itself to start sending email. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (SunOS) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJN+frlAAoJEOUwvh9pJNURn9EP/RxjFK2y8CsrRTcU6MX3E8lQ KbZrF5zCoOC40uLShdVmRSQpugMYfJG30+iSwcqea+mLaavlQQCO1i3WkAc5SEfh hFkz/W/vkqFG1ssAmvgxSA+SE0VH0wns7bbkiifGcYktzFumouchcjxpWLXegFyZ vGj4V3H7v4kv0dUIdnxYT5TcZKPRiZg3HEyF8TjyAz2X1ry0snfgLHQrQIctReh1 y5rlmCWxIZOZaX36VKT2I0IVy55+mwaYpYXE9POYR9iMUbZtMcB1OGAAOQfANa86 CPsSKE/+AmvlX0if6qM76Fl0kLxAHJu8UgoylBXtmf8n82QXqzp3Uad5g5+5zfp1 3FceOk575rRbH/FhxXW/ZebsisUULaTX2X0orsKy+2E70Mm9rYFuWCyrMgb1BZqM sqyApUj7MECBSGow3kunp0dHa7MOaKHgtq/oIRm8ji6owG+yfro/ndw/Ek+xUtJY vA+VK+9rQuf/LpEgv2YcA7nfsSS50rZAHZ+as3PSO6w6Alfa8LPsVOIDByM0RV7a naQkQi4/qbOM/CBTuR86TqvspXUeRyUgsjO5F9Rqn3i8ADzr18+wS04EVUY1LW29 lgPGe95OJUrAbxwuoKBxL/rMPQSj2UZ8Lqap+2qnaLWCjWvj5kfYGS5O+UL+adjE BqX2nKhmv1zevfpV40vk =vyPY -----END PGP SIGNATURE----- From roberto.fernandezcrisial at gmail.com Thu Jun 16 13:03:56 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Thu, 16 Jun 2011 10:03:56 -0300 Subject: varnish syntax In-Reply-To: References: Message-ID: Varnish updated VCL syntax on 2.1.x, don't forget to use "return(pass)" instead just "pass". The same for "return(deliver)", "return(error)"... Best regards, Roberto O. Fern?ndez Crisial @rofc On Thu, Jun 16, 2011 at 9:24 AM, Robert Recchia wrote: > I run word press and munin on the same server - i don't want varnish > caching anything /munin so in older varnish releases all i did was add > this to the vcl file > > sub vcl_recv { > > if (req.url ~ "munin") { > > pass; > > } > > > it seems varnish-2.1.5-1 does not like this syntax and wont start. > Does anyone know the equivilant to what it would be in this version > > > > -- > Robert Recchia > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn at ruberg.no Thu Jun 16 13:07:09 2011 From: bjorn at ruberg.no (=?utf-8?Q?Bj=C3=B8rn?= Ruberg) Date: Thu, 16 Jun 2011 15:07:09 +0200 Subject: Alert on certain error like 503 In-Reply-To: <4DF9FAE5.1080200@uplex.de> References: <4DF9FAE5.1080200@uplex.de> Message-ID: Geoff Simmons writes: > On 06/16/11 01:43 PM, Shibashish wrote: >> >> Can varnish be made to send out an alert (mail) in case of an error like >> 503? I do have a monitoring solution but i can't add all the urls for >> monitoring. Sometimes varnish gives a 503 and i need to be alerted during >> that time. >> >> Another option can be to keep looking for 503 errors. log them as they occur >> and keep watching that file. > > If you just want to know that 503s are happening at all, you can monitor > varnishlog for "NNN TxStatus c 503", although that won't tell you which > URL got the 503. > > For that, it gets a bit tricky, you'd have to first watch for "NNN RxURL > c /some/path", and then watch for "NNN c TxStatus" with the same NNN, > emitting an alert for 503 and forgetting the URL otherwise. # varnishlog -c -i RxURL,TxStatus -o TxStatus 503 > At any rate, monitoring the varnishlog is the way to do this, we > wouldn't the Varnish demon itself to start sending email. Indeed. However, on a not-too-busy server, you could run some cron job which runs varnishlog like shown above. You might want to investigate running the above as a daemon writing to a file, then read from that file in your cron job. The frequency of the cron job must be adjusted to your server and to how important it is to catch all occurrences, but you should probably prepare for overlaps and/or gaps :-) -- Bj?rn From jonathan.hursey at adrevolution.com Thu Jun 16 14:18:04 2011 From: jonathan.hursey at adrevolution.com (Jonathan Hursey) Date: Thu, 16 Jun 2011 09:18:04 -0500 Subject: Alert on certain error like 503 In-Reply-To: References: Message-ID: I have varnishncsa writing to a log file and a shell script that greps out all 5XX errors then reports via Nagios if they exceed a certain number. On Thu, Jun 16, 2011 at 6:43 AM, Shibashish wrote: > Hi, > > Can varnish be made to send out an alert (mail) in case of an error like > 503? I do have a monitoring solution but i can't add all the urls for > monitoring. Sometimes varnish gives a 503 and i need to be alerted during > that time. > > Another option can be to keep looking for 503 errors. log them as they > occur and keep watching that file. > > ShiB. > while ( ! ( succeed = try() ) ); > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- *Jonathan M. Hursey* *Linux Systems Administrator* * * -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlevine at iwin.com Thu Jun 16 17:10:42 2011 From: jlevine at iwin.com (Joshua Levine) Date: Thu, 16 Jun 2011 10:10:42 -0700 Subject: trouble with varnishncsa Message-ID: <8F36D4C4-A189-4E21-AC31-73D289AB954B@iwin.com> I am using ubuntu lucid's default varnish-2.1. A `man varnishncsa` still shows the -b option in the build, however evoking it results in: -b is not valid for varnishncsa Clues? Thank you, Joshua From audun at ytterdal.net Fri Jun 17 00:26:37 2011 From: audun at ytterdal.net (Audun Ytterdal) Date: Thu, 16 Jun 2011 17:26:37 -0700 Subject: Alert on certain error like 503 In-Reply-To: References: Message-ID: I actually just have a Custom errorpage with javascript that post all relevant info from the client to a phpscript. That phpscript could easily have logic for Sending mail Den 16. juni 2011 kl. 07:18 skrev Jonathan Hursey : > I have varnishncsa writing to a log file and a shell script that greps out all 5XX errors then reports via Nagios if they exceed a certain number. > > On Thu, Jun 16, 2011 at 6:43 AM, Shibashish wrote: > Hi, > > Can varnish be made to send out an alert (mail) in case of an error like 503? I do have a monitoring solution but i can't add all the urls for monitoring. Sometimes varnish gives a 503 and i need to be alerted during that time. > > Another option can be to keep looking for 503 errors. log them as they occur and keep watching that file. > > ShiB. > while ( ! ( succeed = try() ) ); > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > -- > Jonathan M. Hursey > Linux Systems Administrator > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoiz.shine at gmail.com Fri Jun 17 04:19:08 2011 From: amoiz.shine at gmail.com (Sharl.Jimh.Tsin) Date: Fri, 17 Jun 2011 12:19:08 +0800 Subject: Varnish 3.0.0 released In-Reply-To: <878vt18l4l.fsf@qurzaw.varnish-software.com> References: <878vt18l4l.fsf@qurzaw.varnish-software.com> Message-ID: <4DFAD5BC.5010108@gmail.com> On 2011?06?17? 06:30, Tollef Fog Heen wrote: > It is my pleasure to announce Varnish 3.0.0 ? our best release so > far. There are lots of changes with the most notable ones being: > > * Module support through VMODs. > > * Compression and uncompression support, including stiching together > compressed ESI fragments. > > * Preliminary streaming support, both on miss and on pass. > > * Much improved documentation. > > * Better default values for parameters. > > * Varnishncsa now has custom log format support. > > * Varnishlog, varnishncsa and varnishhist can now filter out records > that match multiple expressions. > > This is just a short summary of all the changes, please see the > changelog for a fuller list. Please note that VCL has changed in > backwards-incompatible fashion, but please see the changelog and > documentation for more details. > > Enjoy! Is Varnish-Cache.org site down? -- Best regards, Sharl.Jimh.Tsin (From China *Obviously Taiwan INCLUDED*) Using Gmail? Please read this important notice: http://www.fsf.org/campaigns/jstrap/gmail?10073. From simon at darkmere.gen.nz Fri Jun 17 06:36:16 2011 From: simon at darkmere.gen.nz (Simon Lyall) Date: Fri, 17 Jun 2011 18:36:16 +1200 (NZST) Subject: varnishncsa filtering? Message-ID: Using 2.1.5 ( thought I'd wait a little while before trying 3.0.0 ) I'm trying to write a little regex to look for a certain part of the URL but nothing seems to work. $ varnishncsa -d -I / $ varnishncsa -d | grep -c / 29701 $ varnishncsa -d | grep -c js 1295 $ varnishncsa -d -I js $ varnishncsa -d -I 6666 202.180.67.113 - - [17/Jun/2011:18:31:24 +1200] "(null) (null) (null)" (null) - "-" "-" 118.93.33.248 - - [17/Jun/2011:18:31:49 +1200] "(null) (null) (null)" (null) - "-" "-" 222.154.227.11 - - [17/Jun/2011:18:32:17 +1200] "(null) (null) (null)" (null) - "-" "-" 202.124.109.104 - - [17/Jun/2011:18:32:44 +1200] "(null) (null) (null)" (null) - "-" "-" 203.109.207.17 - - [17/Jun/2011:18:33:08 +1200] "(null) (null) (null)" (null) - "-" "-" What am I missing? -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. From kristian at varnish-software.com Fri Jun 17 07:41:34 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Fri, 17 Jun 2011 09:41:34 +0200 Subject: Varnish 3.0.0 released In-Reply-To: <4DFAD5BC.5010108@gmail.com> References: <878vt18l4l.fsf@qurzaw.varnish-software.com> <4DFAD5BC.5010108@gmail.com> Message-ID: <20110617074134.GA2881@freud.kly.no> On Fri, Jun 17, 2011 at 12:19:08PM +0800, Sharl.Jimh.Tsin wrote: > Is Varnish-Cache.org site down? It was experiencing some issues earlier, but it's fixed now. - Kristian From adi at netstyle.ch Fri Jun 17 09:10:25 2011 From: adi at netstyle.ch (Adrian Lienhard) Date: Fri, 17 Jun 2011 11:10:25 +0200 Subject: Alert on certain error like 503 In-Reply-To: References: Message-ID: <272A8E05-1D60-4272-9372-CA073650AA00@netstyle.ch> What we do is to write to syslog: C{ #include }C ... sub vcl_error { C{ syslog(LOG_ERR, "Varnish vcl_error"); }C ... return(deliver); } Cheers, Adrian On Jun 17, 2011, at 02:26 , Audun Ytterdal wrote: > I actually just have a Custom errorpage with javascript that post all relevant info from the client to a phpscript. That phpscript could easily have logic for > Sending mail > > Den 16. juni 2011 kl. 07:18 skrev Jonathan Hursey : > >> I have varnishncsa writing to a log file and a shell script that greps out all 5XX errors then reports via Nagios if they exceed a certain number. >> >> On Thu, Jun 16, 2011 at 6:43 AM, Shibashish wrote: >> Hi, >> >> Can varnish be made to send out an alert (mail) in case of an error like 503? I do have a monitoring solution but i can't add all the urls for monitoring. Sometimes varnish gives a 503 and i need to be alerted during that time. >> >> Another option can be to keep looking for 503 errors. log them as they occur and keep watching that file. >> >> ShiB. >> while ( ! ( succeed = try() ) ); >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> >> >> -- >> Jonathan M. Hursey >> Linux Systems Administrator >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From hensel.tobias at googlemail.com Fri Jun 17 09:29:03 2011 From: hensel.tobias at googlemail.com (Tobias Hensel) Date: Fri, 17 Jun 2011 11:29:03 +0200 Subject: Unknown variable 'beresp.url' Message-ID: <4DFB1E5F.4060206@googlemail.com> Hello, I try doing my first steps with varnish. I use debian and installed varnish from your repos. varnishd -V says varnishd (varnish-2.1.5 SVN ) I try to rebuild some of your examples. On http://www.varnish-cache.org/docs/2.1/tutorial/vcl.html Example 2 - manipulating beresp says: sub vcl_fetch { if (beresp.url ~ "\.(png|gif|jpg)$") { unset beresp.http.set-cookie; set beresp.ttl = 3600; } } If I use this in my config (I did not set anything else there) I get the following error: root at debian:~# /etc/init.d/varnish start Starting HTTP accelerator: varnishd failed! storage_file: filename: /var/lib/varnish/varnish_storage.bin size 500 MB. Message from VCC-compiler: Unknown variable 'beresp.url' At: (input Line 14 Pos 10) if (beresp.url ~ "\.(png|gif|jpg)$") { ---------##########------------------------ Running VCC-compiler failed, exit 1 VCL compilation failed Did I miss something? Regards Tobias From twiztar at gmail.com Fri Jun 17 09:32:29 2011 From: twiztar at gmail.com (Erik Weber) Date: Fri, 17 Jun 2011 11:32:29 +0200 Subject: Unknown variable 'beresp.url' In-Reply-To: <4DFB1E5F.4060206@googlemail.com> References: <4DFB1E5F.4060206@googlemail.com> Message-ID: On Fri, Jun 17, 2011 at 11:29 AM, Tobias Hensel wrote: > Hello, > > I try doing my first steps with varnish. > I use debian and installed varnish from your repos. > varnishd -V says varnishd (varnish-2.1.5 SVN ) > > I try to rebuild some of your examples. > On http://www.varnish-cache.org/docs/2.1/tutorial/vcl.html > > Example 2 - manipulating beresp says: > > sub vcl_fetch { > ? if (beresp.url ~ "\.(png|gif|jpg)$") { You wanna use req.url, e.g. sub vcl_fetch { if (req.url ~ ....) -- Erik From hensel.tobias at googlemail.com Fri Jun 17 09:39:17 2011 From: hensel.tobias at googlemail.com (Tobias Hensel) Date: Fri, 17 Jun 2011 11:39:17 +0200 Subject: Unknown variable 'beresp.url' In-Reply-To: References: <4DFB1E5F.4060206@googlemail.com> Message-ID: <4DFB20C5.3050603@googlemail.com> > You wanna use req.url, e.g. > sub vcl_fetch { > if (req.url ~ ....) > Thank you for your fast answer! I thought I wanna use the statements shown in the example. In the Internet, I see a lot of vcl examples where beresp.url is used with 2.1 even in the official docs it is shown. So I'm a little bit irritated that this does not work. From mattias at nucleus.be Fri Jun 17 09:50:09 2011 From: mattias at nucleus.be (Mattias Geniar) Date: Fri, 17 Jun 2011 11:50:09 +0200 Subject: Alert on certain error like 503 In-Reply-To: <272A8E05-1D60-4272-9372-CA073650AA00@netstyle.ch> References: <272A8E05-1D60-4272-9372-CA073650AA00@netstyle.ch> Message-ID: <18834F5BEC10824891FB8B22AC821A5A01734CDB@nucleus-srv01.Nucleus.local> > What we do is to write to syslog: > > C{ #include }C > ... > sub vcl_error { > C{ syslog(LOG_ERR, "Varnish vcl_error"); }C > ... > return(deliver); > } > Have you had a chance to test how that relates to performance when you start to get a lot of errors being thrown? I can imagine pushing every error to syslog can become a bottleneck when it reaches a certain amount? Regards, Mattias From lajo at kb.dk Fri Jun 17 12:12:01 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Fri, 17 Jun 2011 12:12:01 +0000 Subject: What happened to "log"? Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC26699@EXCHANGE-02.kb.dk> Hi, varnishlog generates an awful lot of output, and I used to be happy to insert easily searchable points in it with 'log "That thing happened now";' The log keyword is no longer recognized in 3.0. Is there a way to achieve similar functionality? Also, currently I'm trying to strip some cookies. Is there some way to see what the value of req.http.Cookie ends up being after mangling it? -- Lars -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry.magnien at sfr.com Fri Jun 17 12:17:05 2011 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Fri, 17 Jun 2011 14:17:05 +0200 Subject: What happened to "log"? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC26699@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC26699@EXCHANGE-02.kb.dk> Message-ID: <4A029B1A60B8E340A50D654D2F130DAA2FE81FB04A@EXCV001.encara.local.ads> Hi, The ? log ? function has been included in the std VMOD. Just add an ? import std ; ? statement at the beginning of your VCL and use ? std.log(? something ?). Regards, Thierry De : varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] De la part de Lars J?rgensen Envoy? : vendredi 17 juin 2011 14:12 ? : 'varnish-misc at varnish-cache.org' Objet : What happened to "log"? Hi, varnishlog generates an awful lot of output, and I used to be happy to insert easily searchable points in it with 'log "That thing happened now";' The log keyword is no longer recognized in 3.0. Is there a way to achieve similar functionality? Also, currently I'm trying to strip some cookies. Is there some way to see what the value of req.http.Cookie ends up being after mangling it? -- Lars -------------- next part -------------- An HTML attachment was scrubbed... URL: From lajo at kb.dk Fri Jun 17 12:28:59 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Fri, 17 Jun 2011 12:28:59 +0000 Subject: Best practice for SSL passthrough? Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk> Hi, We're building a varnish cache to put on a separate machine in front of a site that uses both http and https. Varnish is of course only supposed to cache http requests, but what is the best practice to pass https through to the backend? I'm thinking about putting a separate varnish instance up on port 443 that does a pipe on all requests. Lars -------------- next part -------------- An HTML attachment was scrubbed... URL: From k at kevinkevin.com Fri Jun 17 12:34:01 2011 From: k at kevinkevin.com (Kevinkevin) Date: Fri, 17 Jun 2011 08:34:01 -0400 Subject: Best practice for SSL passthrough? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk> Message-ID: <908BE3AD-B890-42CB-BDDC-54155926D737@kevinkevin.com> I have nginx listening on port 443 , proxying the requests. The certificate can be easily setup with nginx. It works flawlessly. On Jun 17, 2011, at 8:28 AM, Lars J?rgensen wrote: > Hi, > > > > We're building a varnish cache to put on a separate machine in front of a site that uses both http and https. Varnish is of course only supposed to cache http requests, but what is the best practice to pass https through to the backend? I'm thinking about putting a separate varnish instance up on port 443 that does a pipe on all requests. > > > > > > Lars > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From stewsnooze at gmail.com Fri Jun 17 12:35:53 2011 From: stewsnooze at gmail.com (Stewart Robinson) Date: Fri, 17 Jun 2011 13:35:53 +0100 Subject: Best practice for SSL passthrough? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk> Message-ID: <30CF80FB-A985-4DF4-B389-92E855AB1821@gmail.com> Hi Lars, A few people on the list have previously spoken about using stunnel or pound in front of Varnish to handle the SSL traffic. http://www.gossamer-threads.com/lists/varnish/misc/18857 http://www.gossamer-threads.com/lists/varnish/misc/14215?do=post_view_threaded Stew On 17 Jun 2011, at 13:28, Lars J?rgensen wrote: > Hi, > > We're building a varnish cache to put on a separate machine in front of a site that uses both http and https. Varnish is of course only supposed to cache http requests, but what is the best practice to pass https through to the backend? I'm thinking about putting a separate varnish instance up on port 443 that does a pipe on all requests. > > > Lars > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattias at nucleus.be Fri Jun 17 12:36:19 2011 From: mattias at nucleus.be (Mattias Geniar) Date: Fri, 17 Jun 2011 14:36:19 +0200 Subject: Best practice for SSL passthrough? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk> Message-ID: <18834F5BEC10824891FB8B22AC821A5A01734CE4@nucleus-srv01.Nucleus.local> > We're building a varnish cache to put on a separate machine in front of a site that uses both http and https. > Varnish is of course only supposed to cache http requests, but what is the best practice to > pass https through to the backend? I'm thinking about putting a separate varnish instance > up on port 443 that does a pipe on all requests. We're actually a pretty big fan of running Nginx on port 443 for our SSL (so let Nginx encrypt all traffic), and let it talk to the backend port 80 (your varnish) as a proxy configuration. If I'm not mistaken, that's how varnish-cache.org works as well. With Nginx you can pass some extra headers to your backend as well, so you can differentiate between HTTP and HTTPs traffic. Regards, Mattias From lajo at kb.dk Fri Jun 17 12:38:40 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Fri, 17 Jun 2011 12:38:40 +0000 Subject: Language of Varnish 3.0 In-Reply-To: <878vtbwm4k.fsf@qurzaw.varnish-software.com> References: <878vtbwm4k.fsf@qurzaw.varnish-software.com> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC26739@EXCHANGE-02.kb.dk> || and || std.log("foo"); # used to be just log | Yes, there was no reason for it to stay in the main namespace. Ah, so there's my log keyword. Where was this change documented? I'm having a hard time finding up-to-date documentation on varnish - the website still says "log" is good (https://www.varnish-cache.org/docs/trunk/reference/vcl.html#syntax). -- Lars From lajo at kb.dk Fri Jun 17 12:55:14 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Fri, 17 Jun 2011 12:55:14 +0000 Subject: varnishncsa filtering? In-Reply-To: References: Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC2678F@EXCHANGE-02.kb.dk> Hi Simon, Try $ varnishncsa -d | grep \/ Lars -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Simon Lyall Sent: Friday, June 17, 2011 8:36 AM To: varnish-misc at varnish-cache.org Subject: varnishncsa filtering? Using 2.1.5 ( thought I'd wait a little while before trying 3.0.0 ) I'm trying to write a little regex to look for a certain part of the URL but nothing seems to work. $ varnishncsa -d -I / $ varnishncsa -d | grep -c / 29701 $ varnishncsa -d | grep -c js 1295 $ varnishncsa -d -I js $ varnishncsa -d -I 6666 202.180.67.113 - - [17/Jun/2011:18:31:24 +1200] "(null) (null) (null)" (null) - "-" "-" 118.93.33.248 - - [17/Jun/2011:18:31:49 +1200] "(null) (null) (null)" (null) - "-" "-" 222.154.227.11 - - [17/Jun/2011:18:32:17 +1200] "(null) (null) (null)" (null) - "-" "-" 202.124.109.104 - - [17/Jun/2011:18:32:44 +1200] "(null) (null) (null)" (null) - "-" "-" 203.109.207.17 - - [17/Jun/2011:18:33:08 +1200] "(null) (null) (null)" (null) - "-" "-" What am I missing? -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From geoff at uplex.de Fri Jun 17 13:16:12 2011 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 17 Jun 2011 15:16:12 +0200 Subject: Alert on certain error like 503 In-Reply-To: <18834F5BEC10824891FB8B22AC821A5A01734CDB@nucleus-srv01.Nucleus.local> References: <272A8E05-1D60-4272-9372-CA073650AA00@netstyle.ch> <18834F5BEC10824891FB8B22AC821A5A01734CDB@nucleus-srv01.Nucleus.local> Message-ID: <4DFB539C.6090903@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 06/17/11 11:50 AM, Mattias Geniar wrote: >> What we do is to write to syslog: >> >> C{ #include }C >> ... >> sub vcl_error { >> C{ syslog(LOG_ERR, "Varnish vcl_error"); }C >> ... >> return(deliver); >> } >> > > Have you had a chance to test how that relates to performance when you > start to get a lot of errors being thrown? I can imagine pushing every > error to syslog can become a bottleneck when it reaches a certain > amount? If all you want is a syslog entry any time vcl_error() is called, just scan varnishlog for occurrences of VCL_call matching error. There's no reason to tie up the varnishd demon with calls to syslog() just for that. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Schwanenwik 24 22087 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (SunOS) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJN+1ObAAoJEOUwvh9pJNURt5YP/AxtiC/qFZBFDPkFDrqIEs0N g9DXWtenXVv6ZcnuIpD4SBZ3zwfvFVtwGasUGyqAXOSFFCLtDksKyHgrK0X1OYGC ZSEWJakAYFxptC+MHKP96AEseZg4sLJyOC7BVQYST+pT8bfvK5pmXHFtRi+QprJk jd8F84aXQRbmxIoV7E9Px9cTgCz2w8SXNMksVWiYoOOltTBNsm4bMc3GcY2jcI5r fT6kwOQQE5SpLo/8Lhy6dcWNN9JrZS8LHaXUUdoYctPI0kzoUIoOnYBz2+rVBCOY UEavdGQ1s2T4w/gVaH3ERRApGe6Wv37NXUT93oS3e/p4YyE7Utb0u0n9MPc52l59 Fy+fsZIYMws9zMHb5+0uC7v/3Elf1WrTEKgAK1GNjgR9RePF5M0b69J73N05Hjbl K7fdRqjzW3Ezrf6TcRtzwmDK8Du32wdjHjv7Prx/d37xvOx64MmIOlMj829qy9MQ E2tJP/wSXFTFLzMpz62PbB8ScmXHiORTIIFQvZjoQ8rTnboLiZiTDBxrivE0ZzsG nI/+U0ERvuy+ufeuPAGH8H0DnGc+h8AKtxqqs8Vet02/xI+pF5fUyMncjBki2Ge9 MmIcIXpR98ud4CMARaUg9dWh3UtcFNntnzvPgpWtVvYhoFyzfy9j6kHGnYmhn9G6 fTy6fRcTfMs4Q03LC4cU =8UrV -----END PGP SIGNATURE----- From lajo at kb.dk Fri Jun 17 13:23:28 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Fri, 17 Jun 2011 13:23:28 +0000 Subject: Best practice for SSL passthrough? In-Reply-To: <18834F5BEC10824891FB8B22AC821A5A01734CE4@nucleus-srv01.Nucleus.local> References: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk> <18834F5BEC10824891FB8B22AC821A5A01734CE4@nucleus-srv01.Nucleus.local> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC267BB@EXCHANGE-02.kb.dk> > We're actually a pretty big fan of running Nginx on port 443 for our SSL > (so let Nginx encrypt all traffic), and let it talk to the backend port > 80 (your varnish) as a proxy configuration. Let me get this straight: HTTPS -> nginx -> HTTP -> Varnish -> HTTP -> Backend? Is that how it works? That seems pretty neat because then I can cache https requests too. I don't know too much about ssl, but it seems to me the backend might be confused at getting regular http requests when it expects https? Or this might not be an issue? > With Nginx you can pass some extra headers to your backend as well, so > you can differentiate between HTTP and HTTPs traffic. In what case would I want to do that? -- Lars From kristian at varnish-software.com Fri Jun 17 13:27:13 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Fri, 17 Jun 2011 15:27:13 +0200 Subject: Unknown variable 'beresp.url' In-Reply-To: <4DFB20C5.3050603@googlemail.com> References: <4DFB1E5F.4060206@googlemail.com> <4DFB20C5.3050603@googlemail.com> Message-ID: <20110617132713.GF2881@freud.kly.no> Hi Tobias, On Fri, Jun 17, 2011 at 11:39:17AM +0200, Tobias Hensel wrote: > >You wanna use req.url, e.g. > >sub vcl_fetch { > > if (req.url ~ ....) > > Thank you for your fast answer! > > I thought I wanna use the statements shown in the example. > In the Internet, I see a lot of vcl examples where beresp.url is > used with 2.1 even in the official docs it is shown. So I'm a little > bit irritated that this does not work. Unfortunately we had a typo that snuck into the docs, which was fixed in March. And unfortunately again, the docs online for 2.1 are only updated when we pull changes into the 2.1 branch preparing for an other 2.1-release. Since we haven't had a release of 2.1 since the documentation-typo was fixed, the issue persists on the 2.1 web, which is a bit unfortunate. Sorry you got hit by this, but I'm glad you got it working :) - Kristian From stewsnooze at gmail.com Fri Jun 17 13:37:29 2011 From: stewsnooze at gmail.com (Stewart Robinson) Date: Fri, 17 Jun 2011 14:37:29 +0100 Subject: Maintaining historical docs and yay! Message-ID: <8C5B6E19-147F-4CA9-99C5-DD18AB79C256@gmail.com> Hi, I'm very happy Varnish 3 came out and it looks fantastic. On a historical note I think it is important that we have links to the historical documentation from http://www.varnish-cache.org/docs A good proportion of users will be using the stock version of Varnish that is packaged with their distribution. Can we find a way to link to historical copies of the documentation on http://www.varnish-cache.org/docs other than latest stable and trunk. I'm not asking that we go back and regenerate docs for 2.0.x but we could maintain the 2.1.x docs and 3.x docs as separate links on the site. Perhaps this will happen automatically. In which case I'm sorry but yay Varnish 3.0 Stewart Robinson -------------- next part -------------- An HTML attachment was scrubbed... URL: From twiztar at gmail.com Fri Jun 17 13:44:18 2011 From: twiztar at gmail.com (Erik Weber) Date: Fri, 17 Jun 2011 15:44:18 +0200 Subject: Best practice for SSL passthrough? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC267BB@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk> <18834F5BEC10824891FB8B22AC821A5A01734CE4@nucleus-srv01.Nucleus.local> <6D2C830A0941EA40B6B483FE6EC98ADBC267BB@EXCHANGE-02.kb.dk> Message-ID: On Fri, Jun 17, 2011 at 3:23 PM, Lars J?rgensen wrote: >> With Nginx you can pass some extra headers to your backend as well, so >> you can differentiate between HTTP and HTTPs traffic. > > In what case would I want to do that? In cases where you hard link uri's. E.g. lets say you have all your images hosted on a sub called images.example.com. Based on the existence of a specificed header, X-Use-HTTPS in this example, you can make sure that your backend provides an output with https-uris. -- Erik From magnus at hagander.net Fri Jun 17 15:03:49 2011 From: magnus at hagander.net (Magnus Hagander) Date: Fri, 17 Jun 2011 17:03:49 +0200 Subject: Unknown variable 'beresp.url' In-Reply-To: <20110617132713.GF2881@freud.kly.no> References: <4DFB1E5F.4060206@googlemail.com> <4DFB20C5.3050603@googlemail.com> <20110617132713.GF2881@freud.kly.no> Message-ID: On Fri, Jun 17, 2011 at 15:27, Kristian Lyngstol wrote: > Hi Tobias, > > On Fri, Jun 17, 2011 at 11:39:17AM +0200, Tobias Hensel wrote: >> >You wanna use req.url, e.g. >> >sub vcl_fetch { >> > ?if (req.url ~ ....) >> >> Thank you for your fast answer! >> >> I thought I wanna use the statements shown in the example. >> In the Internet, I see a lot of vcl examples where beresp.url is >> used with 2.1 even in the official docs it is shown. So I'm a little >> bit irritated that this does not work. > > Unfortunately we had a typo that snuck into the docs, which was fixed in > March. And unfortunately again, the docs online for 2.1 are only updated > when we pull changes into the 2.1 branch preparing for an other > 2.1-release. Since we haven't had a release of 2.1 since the > documentation-typo was fixed, the issue persists on the 2.1 web, which > is a bit unfortunate. Would it be much work to change it so the 2.1 docs on the web pull from the latest branch-tip for 2.1? ;) -- ?Magnus Hagander ?Me: http://www.hagander.net/ ?Work: http://www.redpill-linpro.com/ From mattias at nucleus.be Fri Jun 17 15:11:25 2011 From: mattias at nucleus.be (Mattias Geniar) Date: Fri, 17 Jun 2011 17:11:25 +0200 Subject: Best practice for SSL passthrough? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC267BB@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk><18834F5BEC10824891FB8B22AC821A5A01734CE4@nucleus-srv01.Nucleus.local> <6D2C830A0941EA40B6B483FE6EC98ADBC267BB@EXCHANGE-02.kb.dk> Message-ID: <18834F5BEC10824891FB8B22AC821A5A01734CF1@nucleus-srv01.Nucleus.local> > Let me get this straight: HTTPS -> nginx -> HTTP -> Varnish -> HTTP -> > Backend? > > Is that how it works? That seems pretty neat because then I can cache https > requests too. Yup, that's it exactly. And Nginx is pretty performant as well, so it completes Varnish in that matter. > > I don't know too much about ssl, but it seems to me the backend might be > confused at getting regular http requests when it expects https? Or this > might not be an issue? It's not really an issue, except when your application excepts to know when it's hitting HTTP or HTTPs requests. See below. :-) > > > With Nginx you can pass some extra headers to your backend as well, so > > you can differentiate between HTTP and HTTPs traffic. > > In what case would I want to do that? Depends on your application, but you may want to limit certain functions to only the HTTPs site (say: login, credit card information, ...). Also keep in mind that if you have redirects (say: in a .htaccess file for Apache), that redirect to HTTPs when it receives an HTTP call, you'll hit an infinite loop since all your requests will hit the backend as HTTP. Regards, Mattias From hensel.tobias at googlemail.com Fri Jun 17 16:52:35 2011 From: hensel.tobias at googlemail.com (Tobias Hensel) Date: Fri, 17 Jun 2011 18:52:35 +0200 Subject: Unknown variable 'beresp.url' In-Reply-To: <20110617132713.GF2881@freud.kly.no> References: <4DFB1E5F.4060206@googlemail.com> <4DFB20C5.3050603@googlemail.com> <20110617132713.GF2881@freud.kly.no> Message-ID: Hi Kristian, thank you for your answer. When looking into the branch docs its clear. 2011/6/17 Kristian Lyngstol : > Hi Tobias, > > On Fri, Jun 17, 2011 at 11:39:17AM +0200, Tobias Hensel wrote: >> >You wanna use req.url, e.g. >> >sub vcl_fetch { >> > ?if (req.url ~ ....) >> >> Thank you for your fast answer! >> >> I thought I wanna use the statements shown in the example. >> In the Internet, I see a lot of vcl examples where beresp.url is >> used with 2.1 even in the official docs it is shown. So I'm a little >> bit irritated that this does not work. > > Unfortunately we had a typo that snuck into the docs, which was fixed in > March. And unfortunately again, the docs online for 2.1 are only updated > when we pull changes into the 2.1 branch preparing for an other > 2.1-release. Since we haven't had a release of 2.1 since the > documentation-typo was fixed, the issue persists on the 2.1 web, which > is a bit unfortunate. > > Sorry you got hit by this, but I'm glad you got it working :) > > - Kristian > From ruben at varnish-software.com Fri Jun 17 20:49:56 2011 From: ruben at varnish-software.com (=?ISO-8859-1?Q?Rub=E9n_Romero?=) Date: Fri, 17 Jun 2011 22:49:56 +0200 Subject: Do you have Varnish 3 party pictures and want to share? :-) Message-ID: Hi everyone! 18 parties all in all! Wow. Hope you have had your fun this far! As most people has already celebrated (some parties are hold tomorrow) I would like to ask you to not forget to tag your pictures with #v3rp. We only have Flickr integration on the site, but I have noticed that almost no one uses Flickr anymore :-) I was thinking about adding all of the pictures (from twitpic and what not) to one album in Flickr where we can see how the fun went everywhere. So feel free to send me links to your pictures or even the pictures themselves and I will upload them to Flickr and let you all know once that is done, or at least have all the links in one page so people can easily have an overview. So, please do. I will add my own, I promise ;-) Have a great weekend everyone and enjoy the release! All the best, -- Rub?n Romero | Varnish Software http://pad.lv/~huayra -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at varnish-software.com Fri Jun 17 23:31:43 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Sat, 18 Jun 2011 01:31:43 +0200 Subject: Do you have Varnish 3 party pictures and want to share? :-) In-Reply-To: References: Message-ID: <20110617233143.GD22667@luke.kly.no> On Fri, Jun 17, 2011 at 10:49:56PM +0200, Rub?n Romero wrote: > As most people has already celebrated (some parties are hold tomorrow) I > would like to ask you to not forget to tag your pictures with #v3rp. We only > have Flickr integration on the site, but I have noticed that almost no one > uses Flickr anymore :-) Out of the 271 pictures I had by the end of the day, I ended up sharing 10. https://picasaweb.google.com/kristianlyng/Varnish30PartyOslo And while some are ok, I'm not too happy with the outcome, as I _am_ a novice photographer... But there you are. > I was thinking about adding all of the pictures (from twitpic and what not) > to one album in Flickr where we can see how the fun went everywhere. So feel > free to send me links to your pictures or even the pictures themselves and I > will upload them to Flickr and let you all know once that is done, or at > least have all the links in one page so people can easily have an overview. All my pictures are CC-BY-SA so you /could/ copy them all, but a page that links to albums of relevance is much preferred. I like my visitors using "my" album/gallery. Stats, feedback and the ability to retract images if the people in them ask me to are some of the reasons. It just feels more polite to link than copy. If what you had in mind was using one or two pictures from different galleries to make an entirely new collection, that's a different matter of course. But if it's just an unfiltered copy, then just link to it instead please :) - Kristian From kristian at varnish-software.com Fri Jun 17 23:45:27 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Sat, 18 Jun 2011 01:45:27 +0200 Subject: Updating the 2.1 documentation on Web (was: Unknown variable 'beresp.url') In-Reply-To: References: <4DFB1E5F.4060206@googlemail.com> <4DFB20C5.3050603@googlemail.com> <20110617132713.GF2881@freud.kly.no> Message-ID: <20110617234527.GE22667@luke.kly.no> On Fri, Jun 17, 2011 at 05:03:49PM +0200, Magnus Hagander wrote: > On Fri, Jun 17, 2011 at 15:27, Kristian Lyngstol > wrote: > > Unfortunately we had a typo that snuck into the docs, which was > > fixed in March. And unfortunately again, the docs online for 2.1 are > > only updated when we pull changes into the 2.1 branch preparing for > > an other > > 2.1-release. > > Would it be much work to change it so the 2.1 docs on the web pull > from the latest branch-tip for 2.1? ;) Probably not. Tollef? The /doc/2.1 stuff in sphinx is rebuilt manually, on commit to 2.1 or on tag or something similar? Oh, and we're going to need /doc/3.0 now :) Maybe I should file one of them tickets I keep hearing about. - Kristian From simon at darkmere.gen.nz Sat Jun 18 08:00:21 2011 From: simon at darkmere.gen.nz (Simon Lyall) Date: Sat, 18 Jun 2011 20:00:21 +1200 (NZST) Subject: varnishncsa filtering? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC2678F@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC2678F@EXCHANGE-02.kb.dk> Message-ID: On Fri, 17 Jun 2011, Lars J?rgensen wrote: > $ varnishncsa -d | grep \/ That is probably what I'll end up doing, but does "varnishncsa -I" work at all? -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. From perbu at varnish-software.com Sat Jun 18 13:29:09 2011 From: perbu at varnish-software.com (Per Buer) Date: Sat, 18 Jun 2011 15:29:09 +0200 Subject: Language of Varnish 3.0 In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC26739@EXCHANGE-02.kb.dk> References: <878vtbwm4k.fsf@qurzaw.varnish-software.com> <6D2C830A0941EA40B6B483FE6EC98ADBC26739@EXCHANGE-02.kb.dk> Message-ID: On Fri, Jun 17, 2011 at 2:38 PM, Lars J?rgensen wrote: > || and > || ?std.log("foo"); # used to be just log > > | Yes, there was no reason for it to stay in the main namespace. > > Ah, so there's my log keyword. Where was this change documented? I'm having a hard time finding up-to-date documentation on varnish - the website still says "log" is good (https://www.varnish-cache.org/docs/trunk/reference/vcl.html#syntax). The standard module has it's own man page. std.log should be there. I've ripped the reference to log out of man vcl now. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Whitepapers?| Video?| Twitter From kacperw at gmail.com Sat Jun 18 17:22:49 2011 From: kacperw at gmail.com (Kacper Wysocki) Date: Sat, 18 Jun 2011 19:22:49 +0200 Subject: Language of Varnish 3.0 In-Reply-To: <92943.1307561368@critter.freebsd.dk> References: <92943.1307561368@critter.freebsd.dk> Message-ID: On Wed, Jun 8, 2011 at 9:29 PM, Poul-Henning Kamp wrote: > In message , Kacper Wysocki > ?writes: > >>Or 3.1, as the case may be? > > No, that is not a major version, that is only a minor version. will hold you to it :d > Listen, I'm not doing this kind of stuff to hurt anybody. Duly noted. > Backwards compatibility and "syntactic sugare" is always a > tradeoff of where you want to spend your effort. > > With the limited amount of developers and testers we have available > in Varnish, I have prioritized getting important stuff done, which > in this particular context included VMODS which caused a pretty > extensive rewrite of the VCL compiler. I paid particular attention to this part of the code as my interests often lie within language design. This thread is not a rally against change or progress, the answers you have given are appreciated > In all likelyhood, some of your VCL scripts should be converted > to a VMOD (Sorry: you cannot write VMOD's in VCL yet) and once > you realize what options that gives you, I think you will see > the VCL improvements from 2.X to 3.X as quite sensible. Most of us, me included, are very excited about the options VMODs give us and look forward to VCL VMODs too. 0K From geoff at uplex.de Sun Jun 19 18:30:37 2011 From: geoff at uplex.de (Geoff Simmons) Date: Sun, 19 Jun 2011 20:30:37 +0200 Subject: Conditional backend requests on repository branch experimental-ims In-Reply-To: References: <4DEDC12E.9010702@uplex.de> Message-ID: <4DFE404D.1040705@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 6/9/11 2:13 AM, Paulo Paracatu wrote: > > Yesterday I finally had the time to download this and make the RPM to > install in my caches servers. Thanks for trying it out, and sorry for the late response. The experimental-ims branch in the repository is now up to date with the current master branch, including the 3.0 release. > * 252531 416.87 344.05 fetch_304 - Fetch no body (304)* > > Ah, before the patch I was hitting much more RX than now, so I guess it > is really working and helping. > > If you have any tips, I'd be glad to test. Since a few people have asked how to make sure that the conditional requests are working, I added a section about verification to the Wiki page: http://www.varnish-cache.org/trac/wiki/BackendConditionalRequests#Verification Non-zero values of fetch_304 definitely indicate that Varnish has sent conditional requests. Watching varnishlog is the best way to know exactly what's going on. As for testing: any exposure to a live web site will be very helpful. All of my testing has been with varnishtest test cases and synthetic performance tests. I haven't been able to break it for a while using load generators for heavy load and high concurrency, but nothing beats the real thing. I haven't tested it with ESI, which has been improved for 3.0. Varnish with the IMS feature should work smoothly with conditional requests for ESI includes, but I'm not familiar with ESI and haven't confirmed that. (I have confirmed that conditional requests work correctly with objects that are compressed in the cache.) The feature introduces some additional copying -- Varnish always creates a new object from a backend fetch, and the IMS function copies the body and headers from the stale object to the new object when a 304 is returned. It would be good to test it with large objects to see if there are signs of performance loss. Thanks again for the feedback! Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJN/kBMAAoJEOUwvh9pJNURvoEP/2GJlEN5f3AGjHV/+uOD85T5 4gK+f2xVSYX+Eaav1Pi/qcgrrIZ1o51fxAAxK4r5X9G2roMvjK0S6COAuXy8fjg2 CMbPRxwzJ38P38QHP8aDz+He0zLE3EjaprMPafHy2xReZdr3YLCMUtW+EVJHVWIh 1/OVfglqQ34XU7sbngoLFxUB+pVfn0NshbOnK6Wvqoy4uo5IoqiHyNtn3Ts6/ZNu jgEc5pUYUQ1O5tsQFHtQlx7kyZcRz51bFPKG5gVxnOqH8FMSRR7LIuv2UEo8AOxw qhsNWDnswFGciQwoeILzeoT8Y3z8oUtMvTtUfEHEmuZ5jX4xzgbqwlbjRitSLCDa +IKifG6+PTFNXYpYyca6DV7PBT9GehQPdmAzk0Ql36A/X6GBZHdbiOF+hLGF+Kf+ 7EVHcFf52/xztfU1bcxFkoYOSuxMVHB0WGxBFAMEjLI/U+Qg8iInEI1sHUC+aM8Q kWxcFB/MLBtd4ZtwtVP9aMc4Ev+uXUq164Gfv0biXJci0me3E5EUDf5LTaD8WqNW q2n0gbCl0Dih2qAj2wdQDi40CE0PpUJEVGGoKVrYiMs/evL+rGCXiy/NqKUGdEKq S3PRewWHfRNqpeo/UeRNh/LTCKsKRRSFrrbzGleE/GeG38dIYoclCEwWl8oUD20o jAs9+aJVt1axvzF/O5Jm =oGO8 -----END PGP SIGNATURE----- From lajo at kb.dk Mon Jun 20 09:20:02 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Mon, 20 Jun 2011 09:20:02 +0000 Subject: Best practice for SSL passthrough? In-Reply-To: <18834F5BEC10824891FB8B22AC821A5A01734CF1@nucleus-srv01.Nucleus.local> References: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk><18834F5BEC10824891FB8B22AC821A5A01734CE4@nucleus-srv01.Nucleus.local> <6D2C830A0941EA40B6B483FE6EC98ADBC267BB@EXCHANGE-02.kb.dk> <18834F5BEC10824891FB8B22AC821A5A01734CF1@nucleus-srv01.Nucleus.local> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC26B31@EXCHANGE-02.kb.dk> >> Is that how it works? That seems pretty neat because then I can cache https >> requests too. >Yup, that's it exactly. And Nginx is pretty performant as well, so it completes Varnish in that matter. Great, I'll probably look into it. So far I've found out that https is only really used in a redirect during login, and after that it's straight back to http. I have therefore no need to cache encrypted pages. If I can avoid learning new tech (nginx), I would prefer it at this point. So I did this: backend default { .host = "1.2.3.62"; .port = "80"; } backend default_ssl { .host = "1.2.3.62"; .port = "443"; } sub vcl_recv { if (server.port == 443) { set req.backend = default_ssl; return(pipe); } else { set req.backend = default; } . . . } But it doesn't work, all I get is this: 15 SessionOpen c 10.6.0.18 52617 :443 15 Debug c herding 15 SessionClose c timeout 15 StatSess c 10.6.0.18 52617 0 1 0 0 0 0 0 0 15 SessionOpen c 10.6.0.18 52622 :443 15 Debug c herding 15 SessionClose c timeout 15 StatSess c 10.6.0.18 52622 0 1 0 0 0 0 0 0 Am I doing something obviously wrong? Is "server.port" not available in 3.0? I tried putting a std.log("HTTPS"); in the backend select statement above, but it doesn't get logged. Looks like that if-statement is wrong? -- Lars From andrea.campi at zephirworks.com Mon Jun 20 15:25:28 2011 From: andrea.campi at zephirworks.com (Andrea Campi) Date: Mon, 20 Jun 2011 17:25:28 +0200 Subject: [Announce] TextMate plugin for Varnish Message-ID: Hi all, in the days leading up the party I forgot to post it here; admittedly, it might also be of limited interest. But, for all you Mac users using TextMate, this may be useful: https://github.com/zephirworks/Varnish.tmbundle Andrea From cooltechemail at gmail.com Mon Jun 20 19:31:01 2011 From: cooltechemail at gmail.com (Vince) Date: Tue, 21 Jun 2011 03:31:01 +0800 Subject: varnish is using all file descriptors Message-ID: Hi, We had a problem with our varnish server recently that sometimes the server will use all file descriptors available to it, which is set to 65536, and start refusing connections. I am wondering how to find out what's the real problem behind it. The only thing I can think of is that I configured max_restarts=2 and we have 3 backends. When the the first backend returns a 404, varnish will try the second, and when the second backend returns 404 again, varnish will try the third one. Will this be a problem or cause any dead loops? I would appreciate if anyone would shed some light on it problem. Basically when we encounter this problem everything seems fine and it is hard for us to diagnose the problem Thank you very much Vince -------------- next part -------------- An HTML attachment was scrubbed... URL: From cooltechemail at gmail.com Mon Jun 20 19:47:10 2011 From: cooltechemail at gmail.com (Vince) Date: Tue, 21 Jun 2011 03:47:10 +0800 Subject: customized 502/503 error page for varnish Message-ID: Hi All, Is there anyway to have a customized error page? For example, if there is something wrong with the backend, varnish will throw some thing like this to the end user: *Error 503 Service Unavailable Service Unavailable Guru Meditation: XID: 228357344 Varnish cache server* Is there anyway to replace this with a user friendly error page? Thank you, Vince -------------- next part -------------- An HTML attachment was scrubbed... URL: From twiztar at gmail.com Mon Jun 20 19:53:12 2011 From: twiztar at gmail.com (Erik Weber) Date: Mon, 20 Jun 2011 21:53:12 +0200 Subject: customized 502/503 error page for varnish In-Reply-To: References: Message-ID: On Mon, Jun 20, 2011 at 9:47 PM, Vince wrote: > Hi All, > Is there anyway to have a customized error page? Check vcl_error in this example: http://www.nedproductions.biz/wiki/a-perfected-varnish-reverse-caching-proxy-vcl-script -- Erik From apj at mutt.dk Mon Jun 20 19:56:56 2011 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Mon, 20 Jun 2011 21:56:56 +0200 Subject: customized 502/503 error page for varnish In-Reply-To: References: Message-ID: <20110620195655.GN960@nerd.dk> On Tue, Jun 21, 2011 at 03:47:10AM +0800, Vince wrote: > > Is there anyway to have a customized error page? You can use the synthetic keyword in vcl. If you look in vcl_error in the default vcl (listed in man vcl), you'll see an example. -- Andreas From darvin.denmian at gmail.com Mon Jun 20 20:01:13 2011 From: darvin.denmian at gmail.com (Darvin Denmian) Date: Mon, 20 Jun 2011 17:01:13 -0300 Subject: varnish is using all file descriptors In-Reply-To: References: Message-ID: What OS are you using? On Mon, Jun 20, 2011 at 4:31 PM, Vince wrote: > Hi, > > We had a problem with our varnish server recently that sometimes the server > will use all file descriptors available to it, which is set to 65536, and > start refusing connections. I am wondering how to find out what's the real > problem behind it. > > The only thing I can think of is that I configured max_restarts=2 and we > have 3 backends. When the the first backend returns a 404, varnish will try > the second, and when the second backend returns 404 again, varnish will try > the third one. Will this be a problem or cause any dead loops? > > I would appreciate if anyone would shed some light on it problem. Basically > when we encounter this problem everything seems fine and it is hard for us > to diagnose the problem > > Thank you very much > > Vince > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Mon Jun 20 20:02:44 2011 From: perbu at varnish-software.com (Per Buer) Date: Mon, 20 Jun 2011 22:02:44 +0200 Subject: varnish is using all file descriptors In-Reply-To: References: Message-ID: On Mon, Jun 20, 2011 at 9:31 PM, Vince wrote > Hi, > We had a problem with our varnish server recently that sometimes the server > will use all file descriptors available to it, which is set to 65536, and > start refusing connections. I am wondering how to find out what's the real > problem behind it. Why the insanely low number? Out of the box my laptop has over half a million. With 2minutes timeout for reuse on a tcp socket you would run of sockets in a jiffie - oh, wait, that's what happening. :-) I would set up waaaay up. FD's are really cheap. You could start to tune your TCP stack and/or reduce the number of Varnish threads but unless you are on a embedded system I wouldn't do it. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Whitepapers?| Video?| Twitter From cooltechemail at gmail.com Mon Jun 20 20:08:35 2011 From: cooltechemail at gmail.com (Vince) Date: Tue, 21 Jun 2011 04:08:35 +0800 Subject: varnish is using all file descriptors In-Reply-To: References: Message-ID: We are using CentOS 5.6 64-bit I'll try set it to a higher number. Do we need to turn any kernel parameters if we set it to a really high number, say 1M, on CentOS? However, a higher number doesn't mean we have found the problem. Our varnish server is serving about 300-400 requests per second. When it runs ok we checked /proc/PID/fd and it uses less than 100 file descriptors. But when it goes wrong all 65536 file descriptors have been used. Is this normal? Thank you! On Tue, Jun 21, 2011 at 4:02 AM, Per Buer wrote: > On Mon, Jun 20, 2011 at 9:31 PM, Vince wrote > > Hi, > > We had a problem with our varnish server recently that sometimes the > server > > will use all file descriptors available to it, which is set to 65536, and > > start refusing connections. I am wondering how to find out what's the > real > > problem behind it. > > Why the insanely low number? Out of the box my laptop has over half a > million. With 2minutes timeout for reuse on a tcp socket you would run > of sockets in a jiffie - oh, wait, that's what happening. :-) > > I would set up waaaay up. FD's are really cheap. You could start to > tune your TCP stack and/or reduce the number of Varnish threads but > unless you are on a embedded system I wouldn't do it. > > -- > Per Buer, CEO > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > Varnish makes websites fly! > Whitepapers | Video | Twitter > -------------- next part -------------- An HTML attachment was scrubbed... URL: From darvin.denmian at gmail.com Mon Jun 20 20:13:22 2011 From: darvin.denmian at gmail.com (Darvin Denmian) Date: Mon, 20 Jun 2011 17:13:22 -0300 Subject: varnish is using all file descriptors In-Reply-To: References: Message-ID: Maybe the following link can help you : http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/ On Mon, Jun 20, 2011 at 5:08 PM, Vince wrote: > > We are using CentOS 5.6 64-bit > I'll try set it to a higher number. Do we need to turn any kernel parameters if we set it to a really high number, say 1M, on CentOS? > However, a higher number doesn't mean we have found the problem. Our varnish server is serving about 300-400 requests per second. When it runs ok we checked /proc/PID/fd and it uses less than 100 file descriptors. But when it goes wrong all 65536 file descriptors have been used. Is this normal? > Thank you! > > On Tue, Jun 21, 2011 at 4:02 AM, Per Buer wrote: >> >> On Mon, Jun 20, 2011 at 9:31 PM, Vince wrote >> > Hi, >> > We had a problem with our varnish server recently that sometimes the server >> > will use all file descriptors available to it, which is set to 65536, and >> > start refusing connections. I am wondering how to find out what's the real >> > problem behind it. >> >> Why the insanely low number? Out of the box my laptop has over half a >> million. With 2minutes timeout for reuse on a tcp socket you would run >> of sockets in a jiffie - oh, wait, that's what happening. :-) >> >> I would set up waaaay up. FD's are really cheap. You could start to >> tune your TCP stack and/or reduce the number of Varnish threads but >> unless you are on a embedded system I wouldn't do it. >> >> -- >> Per Buer, CEO >> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer >> Varnish makes websites fly! >> Whitepapers?| Video?| Twitter > From TFigueiro at au.westfield.com Tue Jun 21 04:23:11 2011 From: TFigueiro at au.westfield.com (Thiago Figueiro) Date: Tue, 21 Jun 2011 04:23:11 +0000 Subject: RHEL 6 SElinux policy for 2.1.5 Message-ID: <22F96A574165AA4482FF290947347774025FAD@AUPDC00-MBX01P.au.ad.westfield.com> I used the 2.1.5 EPEL 6 packages from http://download.fedora.redhat.com/pub/epel/6/x86_64/repoview/varnish.html in our systems but it looks they are lacking the selinux policy. It's not hard to create a policy but it's a repetitive and boring task. See below for the policy we're using; no guarantees it will work with your particular configuration but hopefully that will save you some time and point you in the right direction. I'm not sure why varnishd needs to write to /etc/varnish but oh well... :-) # cat varnish.te module varnish 1.0; require { type home_root_t; type varnishd_t; type boot_t; type varnishlog_t; type file_t; type varnishd_etc_t; class dir { write remove_name search getattr add_name }; class file { execute setattr read create write getattr unlink open }; } #============= varnishd_t ============== allow varnishd_t boot_t:dir getattr; allow varnishd_t file_t:dir search; allow varnishd_t file_t:file { read write getattr }; allow varnishd_t home_root_t:dir getattr; allow varnishd_t varnishd_etc_t:dir { write remove_name add_name }; allow varnishd_t varnishd_etc_t:file { write create unlink execute setattr }; #============= varnishlog_t ============== allow varnishlog_t varnishd_etc_t:dir search; allow varnishlog_t varnishd_etc_t:file open; allow varnishlog_t varnishd_etc_t:file read; ______________________________________________________ CONFIDENTIALITY NOTICE This electronic mail message, including any and/or all attachments, is for the sole use of the intended recipient(s), and may contain confidential and/or privileged information, pertaining to business conducted under the direction and supervision of the sending organization. All electronic mail messages, which may have been established as expressed views and/or opinions (stated either within the electronic mail message or any of its attachments), are left to the sole responsibility of that of the sender, and are not necessarily attributed to the sending organization. Unauthorized interception, review, use, disclosure or distribution of any such information contained within this electronic mail message and/or its attachment(s), is (are) strictly prohibited. If you are not the intended recipient, please contact the sender by replying to this electronic mail message, along with the destruction all copies of the original electronic mail message (along with any attachments). ______________________________________________________ From perbu at varnish-software.com Tue Jun 21 08:17:29 2011 From: perbu at varnish-software.com (Per Buer) Date: Tue, 21 Jun 2011 10:17:29 +0200 Subject: varnish is using all file descriptors In-Reply-To: References: Message-ID: Hi Vince. There are to limits on file descriptors on a Linux system. There is the global one, which was the one I thought you have been adjusting ( /proc/sys/kernel/fs/something) and the per process tree one, rlimits (kernel part), also known as ulimits (user space part). A Varnish server will need quite a few FDs of the first kind, not so many of the second. So, you're right, you have a problem. Check to see whether the same problem is present at the backend. You might have a slow backend that is piling up connections. If so, you could work around the problem by setting some very agressive timeouts. You'll serve some errors, but at least the site won't go down. Per. On Mon, Jun 20, 2011 at 10:08 PM, Vince wrote: > We are using CentOS 5.6 64-bit > I'll try set it to a higher number. Do we need to turn any kernel parameters > if we set it to a really high number, say 1M, on CentOS? > However, a higher number doesn't mean we have found the problem. Our varnish > server is serving about 300-400 requests per second. When it runs ok we > checked /proc/PID/fd and it uses less than 100 file descriptors. But when it > goes wrong all 65536 file descriptors have been used. Is this normal? > Thank you! > > On Tue, Jun 21, 2011 at 4:02 AM, Per Buer > wrote: >> >> On Mon, Jun 20, 2011 at 9:31 PM, Vince wrote >> > Hi, >> > We had a problem with our varnish server recently that sometimes the >> > server >> > will use all file descriptors available to it, which is set to 65536, >> > and >> > start refusing connections. I am wondering how to find out what's the >> > real >> > problem behind it. >> >> Why the insanely low number? Out of the box my laptop has over half a >> million. With 2minutes timeout for reuse on a tcp socket you would run >> of sockets in a jiffie - oh, wait, that's what happening. :-) >> >> I would set up waaaay up. FD's are really cheap. You could start to >> tune your TCP stack and/or reduce the number of Varnish threads but >> unless you are on a embedded system I wouldn't do it. >> >> -- >> Per Buer, CEO >> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer >> Varnish makes websites fly! >> Whitepapers?| Video?| Twitter > > -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Whitepapers?| Video?| Twitter From lajo at kb.dk Tue Jun 21 10:53:29 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Tue, 21 Jun 2011 10:53:29 +0000 Subject: Best practice for SSL passthrough? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC26B31@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC266BC@EXCHANGE-02.kb.dk><18834F5BEC10824891FB8B22AC821A5A01734CE4@nucleus-srv01.Nucleus.local> <6D2C830A0941EA40B6B483FE6EC98ADBC267BB@EXCHANGE-02.kb.dk> <18834F5BEC10824891FB8B22AC821A5A01734CF1@nucleus-srv01.Nucleus.local> <6D2C830A0941EA40B6B483FE6EC98ADBC26B31@EXCHANGE-02.kb.dk> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC27168@EXCHANGE-02.kb.dk> Hi, Not a solution to below problem, but I found a very easy workaround. Posting it here in case it has relevance to other users. Problem: HTTP and HTTPS goes to same cached FQDN. Varnish only does HTTP, how to pass HTTPS through to the backend? Solution: xinetd can do this out of the box. Create a file in /etc/xinetd.d containing stuff like below and restart xinetd. service some_unique_name { type = UNLISTED flags = REUSE socket_type = stream wait = no user = root redirect = back.end.ip.address 443 port = 443 } A big caveat to this: You will not be able to cache HTTPS traffic. In my case that's not a problem but YMMV. Lars -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Lars J?rgensen Sent: Monday, June 20, 2011 11:20 AM To: varnish-misc at varnish-cache.org Subject: RE: Best practice for SSL passthrough? >> Is that how it works? That seems pretty neat because then I can cache https >> requests too. >Yup, that's it exactly. And Nginx is pretty performant as well, so it completes Varnish in that matter. Great, I'll probably look into it. So far I've found out that https is only really used in a redirect during login, and after that it's straight back to http. I have therefore no need to cache encrypted pages. If I can avoid learning new tech (nginx), I would prefer it at this point. So I did this: backend default { .host = "1.2.3.62"; .port = "80"; } backend default_ssl { .host = "1.2.3.62"; .port = "443"; } sub vcl_recv { if (server.port == 443) { set req.backend = default_ssl; return(pipe); } else { set req.backend = default; } . . . } But it doesn't work, all I get is this: 15 SessionOpen c 10.6.0.18 52617 :443 15 Debug c herding 15 SessionClose c timeout 15 StatSess c 10.6.0.18 52617 0 1 0 0 0 0 0 0 15 SessionOpen c 10.6.0.18 52622 :443 15 Debug c herding 15 SessionClose c timeout 15 StatSess c 10.6.0.18 52622 0 1 0 0 0 0 0 0 Am I doing something obviously wrong? Is "server.port" not available in 3.0? I tried putting a std.log("HTTPS"); in the backend select statement above, but it doesn't get logged. Looks like that if-statement is wrong? -- Lars _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From harri.paivaniemi at tieto.com Tue Jun 21 11:47:16 2011 From: harri.paivaniemi at tieto.com (Harri =?ISO-8859-1?Q?P=E4iv=E4niemi?=) Date: Tue, 21 Jun 2011 14:47:16 +0300 Subject: X-forwarded-for with a load balancer? Message-ID: <1308656836.19498.5.camel@hjp.tieto> Hi, We have a varnish behind a load balancer that add's x-forwarded-for- gheader to give a client ip. How to get varnish to just forward that allready set header that it recieves from lb to backend instead that varnish is setting it again to lb-ip? Thanks. -hjp -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue Jun 21 13:42:49 2011 From: perbu at varnish-software.com (Per Buer) Date: Tue, 21 Jun 2011 15:42:49 +0200 Subject: X-forwarded-for with a load balancer? In-Reply-To: <1308656836.19498.5.camel@hjp.tieto> References: <1308656836.19498.5.camel@hjp.tieto> Message-ID: Hi. 2011/6/21 Harri P?iv?niemi : > > We have a varnish behind a load balancer that add's x-forwarded-for- gheader > to give a client ip. > > How to get varnish to just forward that allready set header that it recieves > from lb to backend instead that varnish is setting it again to lb-ip? The built in code looks like this: # if (req.restarts == 0) { # if (req.http.x-forwarded-for) { # set req.http.X-Forwarded-For = # req.http.X-Forwarded-For + ", " + client.ip; # } else { # set req.http.X-Forwarded-For = client.ip; # } # } So your original IP should still be there, together with the LB-IP. You could of course override this code as usual and not mess with the header, or rename the original. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer Varnish makes websites fly! Whitepapers?| Video?| Twitter From bedis9 at gmail.com Wed Jun 22 05:32:23 2011 From: bedis9 at gmail.com (Baptiste) Date: Wed, 22 Jun 2011 07:32:23 +0200 Subject: X-forwarded-for with a load balancer? In-Reply-To: <1308656836.19498.5.camel@hjp.tieto> References: <1308656836.19498.5.camel@hjp.tieto> Message-ID: Hi, What kind of LB do you use? Usually, you can turn your load-balancer in transparent mode. It will then present client IP to Varnish instead of using xff header. In this mode, the varnish default gateway must be the LB. Regards 2011/6/21 Harri P?iv?niemi : > Hi, > > We have a varnish behind a load balancer that add's x-forwarded-for- gheader > to give a client ip. > > How to get varnish to just forward that allready set header that it recieves > from lb to backend instead that varnish is setting it again to lb-ip? > > Thanks. > > -hjp > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > From jonathan.hursey at adrevolution.com Thu Jun 23 00:34:10 2011 From: jonathan.hursey at adrevolution.com (Jonathan Hursey) Date: Wed, 22 Jun 2011 19:34:10 -0500 Subject: varnishlog Message-ID: Hello all. Congrats on V3, I'll be implementing it here shortly! As for my current install, I'm trying to figure out how to capture the "worker threads limited" count using varnishlog or some other facility so I can report on the output. Would anyone happen to know how I could go about doing this? I would really appreciate your help. Thanks so much. -- *Jonathan M. Hursey* *Linux Systems Administrator* * * -------------- next part -------------- An HTML attachment was scrubbed... URL: From cosimo at streppone.it Thu Jun 23 07:31:13 2011 From: cosimo at streppone.it (Cosimo Streppone) Date: Thu, 23 Jun 2011 09:31:13 +0200 Subject: varnishlog In-Reply-To: References: Message-ID: On Thu, 23 Jun 2011 02:34:10 +0200, Jonathan Hursey wrote: > I'm trying to figure out how to capture the "worker threads limited" > count Try: varnishstat -1 | grep 'worker threads limited' | awk '{ print $2 }' -- Cosimo From jonathan.hursey at adrevolution.com Thu Jun 23 16:35:00 2011 From: jonathan.hursey at adrevolution.com (Jonathan Hursey) Date: Thu, 23 Jun 2011 11:35:00 -0500 Subject: varnishlog In-Reply-To: References: Message-ID: Exactly what I was looking for ! I am in your debt! Do you know if there is a way to load sites into the cache without having to click through all their internal links? Thanks again ! Varnish Rocks! On Thu, Jun 23, 2011 at 2:31 AM, Cosimo Streppone wrote: > On Thu, 23 Jun 2011 02:34:10 +0200, Jonathan Hursey < > jonathan.hursey at adrevolution.**com > > wrote: > > I'm trying to figure out how to capture the "worker threads limited" count >> > > Try: > > varnishstat -1 | grep 'worker threads limited' | awk '{ print $2 }' > > -- > Cosimo > > ______________________________**_________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc > -- *Jonathan M. Hursey* *Linux Systems Administrator* * * -------------- next part -------------- An HTML attachment was scrubbed... URL: From flatcircle at hotmail.com Fri Jun 24 20:00:39 2011 From: flatcircle at hotmail.com (Flatcircle) Date: Fri, 24 Jun 2011 22:00:39 +0200 Subject: Varnish redirection Message-ID: Hello, I saw that it is possible to redirect domains with a 'dirty' hack. (http://www.varnish-cache.org/trac/wiki/VCLExampleRedirectInVCL) This works well. However, is it also possible to redirect subdirectories on a domain? For example: http://www.example.com/somesubdirectory should redirect to http://www.adifferentsite.com Is this possible with Varnish? (and is so, tips are welcome :-) Regards. From twiztar at gmail.com Fri Jun 24 21:04:55 2011 From: twiztar at gmail.com (Erik Weber) Date: Fri, 24 Jun 2011 23:04:55 +0200 Subject: Varnish redirection In-Reply-To: References: Message-ID: On Fri, Jun 24, 2011 at 10:00 PM, Flatcircle wrote: > Hello, > > I saw that it is possible to redirect domains with a 'dirty' hack. > (http://www.varnish-cache.org/trac/wiki/VCLExampleRedirectInVCL) > This works well. > > However, is it also possible to redirect subdirectories on a domain? > > For example: > > http://www.example.com/somesubdirectory > > should redirect to > > http://www.adifferentsite.com sub vcl_recv { if (req.url ~ "^/somesubdirectory/?$") { error 750 "http://www.adifferentsite.com"; } } sub vcl_error { if (obj.status == 750) { set obj.http.Location = obj.response; set obj.status = 302; return (deliver); } } -- Erik From jonathan.hursey at adrevolution.com Sat Jun 25 07:40:09 2011 From: jonathan.hursey at adrevolution.com (Jonathan Hursey) Date: Sat, 25 Jun 2011 02:40:09 -0500 Subject: Looking for varnish 3 documentation Message-ID: Hello ALL, where can I find some Varnish V3 docs? Thanks! -- *Jonathan M. Hursey* *Linux Systems Administrator* * * -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Sat Jun 25 08:29:00 2011 From: perbu at varnish-software.com (Per Buer) Date: Sat, 25 Jun 2011 10:29:00 +0200 Subject: Looking for varnish 3 documentation In-Reply-To: References: Message-ID: Hi, On Sat, Jun 25, 2011 at 9:40 AM, Jonathan Hursey < jonathan.hursey at adrevolution.com> wrote: > Hello ALL, > where can I find some Varnish V3 docs? Thanks! > http://www.varnish-cache.org/docs/trunk/ -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.hursey at adrevolution.com Sat Jun 25 08:36:37 2011 From: jonathan.hursey at adrevolution.com (Jonathan Hursey) Date: Sat, 25 Jun 2011 03:36:37 -0500 Subject: Looking for varnish 3 documentation In-Reply-To: References: Message-ID: thanks friend! On Sat, Jun 25, 2011 at 3:29 AM, Per Buer wrote: > Hi, > > On Sat, Jun 25, 2011 at 9:40 AM, Jonathan Hursey < > jonathan.hursey at adrevolution.com> wrote: > >> Hello ALL, >> where can I find some Varnish V3 docs? Thanks! >> > > http://www.varnish-cache.org/docs/trunk/ > > > -- > Per Buer, CEO > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > *Varnish makes websites fly!* > Whitepapers | Video | > Twitter > > > -- *Jonathan M. Hursey* *Linux Systems Administrator* * * -------------- next part -------------- An HTML attachment was scrubbed... URL: From mls at pooteeweet.org Mon Jun 27 14:41:18 2011 From: mls at pooteeweet.org (Lukas Kahwe Smith) Date: Mon, 27 Jun 2011 16:41:18 +0200 Subject: Is LCI on the radar? In-Reply-To: References: <4E0D7A8A-2219-4B7E-BBD5-5BF5DBC54047@pooteeweet.org> Message-ID: <4B5D88B4-F620-43C0-AB8A-DE19565B22DF@pooteeweet.org> On 31.05.2011, at 12:58, Laurence Rowe wrote: > On 31 May 2011 10:34, Per Buer wrote: > Hi > > On Tue, May 31, 2011 at 12:23 AM, Lukas Kahwe Smith wrote: > Hi, > > I assume some of you have stumbled over LCI by now: > http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt > > This is actually quite interesting. For an application we are building we are looking to create an invalidation service to which the various independent frontend server applications can register and which gets notified by the backend. Of course the frontends then have to figure out which pages all need to be invalidated. The original article will be easy. Some of the category overviews will also be easy to delete. What will already get harder is invalidating all articles that reference the given article and worse yet would be if we start caching search results. > > So I am wondering if you guys are looking at LCI for a future varnish impovement and if someone has build something like this on top of varnish today already that could maybe help us here. > > I'm pretty sure this can be implemented in VCL. No need to place it on the radar. I have an upcoming blog-post describing something similar. It might get a bit hairy with all the regular expression so it might be cleaner in a module. > > I experimented with something that sounds similar. Each page set a header recording the the content item ids that were used in rendering the page. They could then be purged with a regex including any dependents id. http://dev.plone.org/collective/browser/experimental.depends/trunk/varnish.vcl > > It works when you update or delete a content item, but it can't help the case where you add a new content item and want that to appear in listing. So we are looking to implement this. However I have one question: How well does this perform if you start to have 100k, 1M or more objects in your varnish cache? Does varnish create some sort of index of all headers? I assume even if it did, it cant really leverage it with a regexp. Just dont want to kill my varnish servers CPU/harddrive when I start to purge stuff. If someone could give me some indication of what to expect it would be very good. We will in the end of course have to do our own benchmarks, but it would be good to be able to control expectations :) I guess in the long run if one would want to properly implement LCI it would be necessary to maybe use an sqlite DB and some inline C magic to parse the relevant headers in there and then use that for lookups when doing a PURGE. Does anyone have an idea how much effort it would be to properly implement LCI in Varnish and how we could maybe organize funding among all interested parties? regards, Lukas Kahwe Smith mls at pooteeweet.org From l at lrowe.co.uk Mon Jun 27 15:45:33 2011 From: l at lrowe.co.uk (Laurence Rowe) Date: Mon, 27 Jun 2011 16:45:33 +0100 Subject: Is LCI on the radar? In-Reply-To: <4B5D88B4-F620-43C0-AB8A-DE19565B22DF@pooteeweet.org> References: <4E0D7A8A-2219-4B7E-BBD5-5BF5DBC54047@pooteeweet.org> <4B5D88B4-F620-43C0-AB8A-DE19565B22DF@pooteeweet.org> Message-ID: On 27 June 2011 15:41, Lukas Kahwe Smith wrote: > > On 31.05.2011, at 12:58, Laurence Rowe wrote: > >> On 31 May 2011 10:34, Per Buer wrote: >> Hi >> >> On Tue, May 31, 2011 at 12:23 AM, Lukas Kahwe Smith wrote: >> Hi, >> >> I assume some of you have stumbled over LCI by now: >> http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt >> >> This is actually quite interesting. For an application we are building we are looking to create an invalidation service to which the various independent frontend server applications can register and which gets notified by the backend. Of course the frontends then have to figure out which pages all need to be invalidated. The original article will be easy. Some of the category overviews will also be easy to delete. What will already get harder is invalidating all articles that reference the given article and worse yet would be if we start caching search results. >> >> So I am wondering if you guys are looking at LCI for a future varnish impovement and if someone has build something like this on top of varnish today already that could maybe help us here. >> >> I'm pretty sure this can be implemented in VCL. No need to place it on the radar. I have an upcoming blog-post describing something similar. It might get a bit hairy with all the regular expression so it might be cleaner in a module. >> >> I experimented with something that sounds similar. Each page set a header recording the the content item ids that were used in rendering the page. They could then be purged with a regex including any dependents id. http://dev.plone.org/collective/browser/experimental.depends/trunk/varnish.vcl >> >> It works when you update or delete a content item, but it can't help the case where you add a new content item and want that to appear in listing. > > So we are looking to implement this. However I have one question: > How well does this perform if you start to have 100k, 1M or more objects in your varnish cache? > Does varnish create some sort of index of all headers? I assume even if it did, it cant really leverage it with a regexp. > > Just dont want to kill my varnish servers CPU/harddrive when I start to purge stuff. If someone could give me some indication of what to expect it would be very good. We will in the end of course have to do our own benchmarks, but it would be good to be able to control expectations :) > > I guess in the long run if one would want to properly implement LCI it would be necessary to maybe use an sqlite DB and some inline C magic to parse the relevant headers in there and then use that for lookups when doing a PURGE. > > Does anyone have an idea how much effort it would be to properly implement LCI in Varnish and how we could maybe organize funding among all interested parties? This type of purge (which in Varnish 3 is renames 'ban') adds the expression to the ban list. When any object is found in the cache the expressions in the ban list are checked to decide whether to call vcl_hit or vcl_miss. To prevent the ban list getting too long another thread periodically works its way through all objects in the cache removing those that have been banned and updating the pointer to the place in the ban list it has checked so on subsequent requests fewer ban expressions need to be checked. I've not done any benchmarking on this, but for me even thousands of regular expression checks will be much faster than re-requesting the page from the CMS. So with my Varnish config, a PURGE request is cheap (it only results in adding an entry to the ban list, not checking against the entire contents of the cache) and the additional cost of checking each GET request is not noticeable. So I don't really see the need for LCI support given the existing support for bans. Laurence From mls at pooteeweet.org Mon Jun 27 17:22:05 2011 From: mls at pooteeweet.org (Lukas Kahwe Smith) Date: Mon, 27 Jun 2011 19:22:05 +0200 Subject: Is LCI on the radar? In-Reply-To: References: <4E0D7A8A-2219-4B7E-BBD5-5BF5DBC54047@pooteeweet.org> <4B5D88B4-F620-43C0-AB8A-DE19565B22DF@pooteeweet.org> Message-ID: <4AE4176D-2A32-4F6E-A41E-390025BBAF18@pooteeweet.org> On 27.06.2011, at 17:45, Laurence Rowe wrote: > This type of purge (which in Varnish 3 is renames 'ban') adds the > expression to the ban list. When any object is found in the cache the > expressions in the ban list are checked to decide whether to call > vcl_hit or vcl_miss. To prevent the ban list getting too long another > thread periodically works its way through all objects in the cache > removing those that have been banned and updating the pointer to the > place in the ban list it has checked so on subsequent requests fewer > ban expressions need to be checked. Ok, that sounds pretty much like genius to me :) > I've not done any benchmarking on this, but for me even thousands of > regular expression checks will be much faster than re-requesting the > page from the CMS. So with my Varnish config, a PURGE request is cheap > (it only results in adding an entry to the ban list, not checking > against the entire contents of the cache) and the additional cost of > checking each GET request is not noticeable. So I don't really see the > need for LCI support given the existing support for bans. well LCI support would just mean Varnish plays along with a published standard for this kind of feature, which is always a plus. regards, Lukas Kahwe Smith mls at pooteeweet.org From jonathan.hursey at adrevolution.com Mon Jun 27 21:11:20 2011 From: jonathan.hursey at adrevolution.com (Jonathan Hursey) Date: Mon, 27 Jun 2011 16:11:20 -0500 Subject: Varnish with Wordpress Message-ID: after months of tinkering finally getting awesome hit avg Hitrate ratio: 10 49 49 Hitrate avg: 0.9869 0.9712 0.9712 -- *Jonathan M. Hursey* *Linux Systems Administrator* * * -------------- next part -------------- An HTML attachment was scrubbed... URL: From k at kevinkevin.com Mon Jun 27 22:15:28 2011 From: k at kevinkevin.com (kevin) Date: Mon, 27 Jun 2011 18:15:28 -0400 Subject: Varnish + Joomla Message-ID: <013801cc3517$b957a2f0$2c06e8d0$@com> Hello, I was curious if anyone out there has used varnish and joomla together (successfully). I would love to see a working configuration of varnish 2.1 (or even 3.0) with Joomla. I am particularly interested in how to deal with the PHPSESSID session cookie that is created (by joomla) for every single visitor, logged in or not. I have played around with vcl_hash among other things , but my hit rates aren't that good and I would love to see another working config, or if anyone who knows joomla & varnish could at least give me some pointers. Thanks in advance, Kevin From nkinkade at creativecommons.org Mon Jun 27 22:19:08 2011 From: nkinkade at creativecommons.org (Nathan Kinkade) Date: Mon, 27 Jun 2011 18:19:08 -0400 Subject: Varnish with Wordpress In-Reply-To: References: Message-ID: On Mon, Jun 27, 2011 at 17:11, Jonathan Hursey wrote: > after months of tinkering finally getting awesome hit avg > Hitrate ratio: ? ? ? 10 ? ? ? 49 ? ? ? 49 > Hitrate avg: ? ? 0.9869 ? 0.9712 ? 0.9712 Care to share the relevant VCL rules? From isp at daviesinc.com Tue Jun 28 00:49:00 2011 From: isp at daviesinc.com (Chris Davies) Date: Mon, 27 Jun 2011 20:49:00 -0400 Subject: [SPAM] Varnish + Joomla In-Reply-To: <013801cc3517$b957a2f0$2c06e8d0$@com> References: <013801cc3517$b957a2f0$2c06e8d0$@com> Message-ID: <067CC1F0-37B7-4C57-81E6-E73C53F6F8F2@daviesinc.com> Joomla had some issues with varnish until we started running mod_rpaf with apache. They rely on the IP address REMOTE_ADDR for quite a bit of their internal stuff. Then, some of the themes (rockettheme for one) rely on it. We worked with it for quite a bit but I was never too happy with it. You can write VCL to determine if they are not logged in and remove that cookie to get it to cache. I'll have to look through my notes. On Jun 27, 2011, at 6:15 PM, kevin wrote: > I was curious if anyone out there has used varnish and joomla together > (successfully). I would love to see a working configuration of varnish 2.1 > (or even 3.0) with Joomla. I am particularly interested in how to deal with > the PHPSESSID session cookie that is created (by joomla) for every single > visitor, logged in or not. From isp at daviesinc.com Tue Jun 28 00:57:42 2011 From: isp at daviesinc.com (Chris Davies) Date: Mon, 27 Jun 2011 20:57:42 -0400 Subject: Varnish with Wordpress In-Reply-To: References: Message-ID: http://code.google.com/p/wordpress-varnish-esi/wiki/VCL here's a start for wordpress - I am not 100% happy with it, but, it is running in production on a few servers with minor issues. You can remove the ESI and reset the 12 hour timeout to something more reasonable 19+08:59:27 Hitrate ratio: 10 100 128 Hitrate avg: 1.0000 0.9945 0.9943 about 150rps on this machine On Jun 27, 2011, at 6:19 PM, Nathan Kinkade wrote: > On Mon, Jun 27, 2011 at 17:11, Jonathan Hursey > wrote: >> after months of tinkering finally getting awesome hit avg >> Hitrate ratio: 10 49 49 >> Hitrate avg: 0.9869 0.9712 0.9712 From jonathan.hursey at adrevolution.com Tue Jun 28 00:58:42 2011 From: jonathan.hursey at adrevolution.com (Jonathan Hursey) Date: Mon, 27 Jun 2011 19:58:42 -0500 Subject: Varnish with Wordpress In-Reply-To: References: Message-ID: holy dang. never mind! On Mon, Jun 27, 2011 at 7:57 PM, Chris Davies wrote: > http://code.google.com/p/wordpress-varnish-esi/wiki/VCL > > here's a start for wordpress - I am not 100% happy with it, but, it is > running in production on a few servers with minor issues. > > You can remove the ESI and reset the 12 hour timeout to something more > reasonable > > 19+08:59:27 > Hitrate ratio: 10 100 128 > Hitrate avg: 1.0000 0.9945 0.9943 > > about 150rps on this machine > > On Jun 27, 2011, at 6:19 PM, Nathan Kinkade wrote: > > > On Mon, Jun 27, 2011 at 17:11, Jonathan Hursey > > wrote: > >> after months of tinkering finally getting awesome hit avg > >> Hitrate ratio: 10 49 49 > >> Hitrate avg: 0.9869 0.9712 0.9712 > > -- *Jonathan M. Hursey* *Linux Systems Administrator* * * -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.hursey at adrevolution.com Tue Jun 28 01:00:46 2011 From: jonathan.hursey at adrevolution.com (Jonathan Hursey) Date: Mon, 27 Jun 2011 20:00:46 -0500 Subject: Varnish with Wordpress In-Reply-To: References: Message-ID: should varnish be expected to act as a complete failure server if the wp-content directory is removed on the wordpress install. That just happened to us and were trying to figure out how to get varnish to continue to serve up the cached content for 30+ wp sites whose content was removed. On Mon, Jun 27, 2011 at 7:58 PM, Jonathan Hursey < jonathan.hursey at adrevolution.com> wrote: > holy dang. never mind! > > > On Mon, Jun 27, 2011 at 7:57 PM, Chris Davies wrote: > >> http://code.google.com/p/wordpress-varnish-esi/wiki/VCL >> >> here's a start for wordpress - I am not 100% happy with it, but, it is >> running in production on a few servers with minor issues. >> >> You can remove the ESI and reset the 12 hour timeout to something more >> reasonable >> >> 19+08:59:27 >> Hitrate ratio: 10 100 128 >> Hitrate avg: 1.0000 0.9945 0.9943 >> >> about 150rps on this machine >> >> On Jun 27, 2011, at 6:19 PM, Nathan Kinkade wrote: >> >> > On Mon, Jun 27, 2011 at 17:11, Jonathan Hursey >> > wrote: >> >> after months of tinkering finally getting awesome hit avg >> >> Hitrate ratio: 10 49 49 >> >> Hitrate avg: 0.9869 0.9712 0.9712 >> >> > > > -- > *Jonathan M. Hursey* > *Linux Systems Administrator* > > * > * > > -- *Jonathan M. Hursey* *Linux Systems Administrator* * * -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew at matthewwilkes.co.uk Tue Jun 28 01:06:21 2011 From: matthew at matthewwilkes.co.uk (Matthew Wilkes) Date: Tue, 28 Jun 2011 02:06:21 +0100 Subject: Varnish with Wordpress In-Reply-To: References: Message-ID: <94305693-66FE-4DB4-B966-C7B47F81C5BF@matthewwilkes.co.uk> On 2011-06-28, at 0200, Jonathan Hursey wrote: > should varnish be expected to act as a complete failure server if the wp-content directory is removed on the wordpress install. That just happened to us and were trying to figure out how to get varnish to continue to serve up the cached content for 30+ wp sites whose content was removed. This is what file permissions and backups are for. Matt From k at kevinkevin.com Tue Jun 28 01:42:46 2011 From: k at kevinkevin.com (kevin) Date: Mon, 27 Jun 2011 21:42:46 -0400 Subject: Varnish + Joomla In-Reply-To: <067CC1F0-37B7-4C57-81E6-E73C53F6F8F2@daviesinc.com> References: <013801cc3517$b957a2f0$2c06e8d0$@com> <067CC1F0-37B7-4C57-81E6-E73C53F6F8F2@daviesinc.com> Message-ID: <014d01cc3534$af825eb0$0e871c10$@com> I have been attempting to do the following (with no luck so far), since (from what I understand) the logged in session cookie is a long random hash name : [varnish 3.0] vcl_recv : set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js|base_domain|fbsetting|PHPSESSID)=[^;]*", ""); if ( !(regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js|base_domain|fbsetting|simpleCart)=[^;]*", ""))){ set req.backend = iamloggedin; return(pass); } vcl_hash : if ( !(regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js|base_domain|fbsetting|simpleCart|PHPSESSID)=[^;]*" , ""))){ hash_data(req.http.Cookie); } I know I'm missing something so any way you can assist would be greatly appreciated. I don't consider myself a varnish expert by any means so I'm positive there is a way to do this safely. My tests over the course of the past month have produce mixed results, such as caching the login message of another user (oops). Thanks! ~kevin > Joomla had some issues with varnish until we started running mod_rpaf with apache. They rely on the IP address REMOTE_ADDR for quite a bit of their internal stuff. Then, some of the themes (rockettheme > for one) rely on it. We worked with it for quite a bit but I was never too happy with it. > > You can write VCL to determine if they are not logged in and remove that cookie to get it to cache. I'll have to look through my notes. On Jun 27, 2011, at 6:15 PM, kevin wrote: >> I was curious if anyone out there has used varnish and joomla together >> (successfully). I would love to see a working configuration of varnish 2.1 >> (or even 3.0) with Joomla. I am particularly interested in how to deal with >> the PHPSESSID session cookie that is created (by joomla) for every single >> visitor, logged in or not. From k at kevinkevin.com Tue Jun 28 01:46:27 2011 From: k at kevinkevin.com (kevin) Date: Mon, 27 Jun 2011 21:46:27 -0400 Subject: Varnish with Wordpress In-Reply-To: References: Message-ID: <015c01cc3535$3317d340$994779c0$@com> > should varnish be expected to act as a complete failure server if the wp-content directory is removed on the wordpress install. That just happened to us and were trying to figure out how to get varnish > to continue to serve up the cached content for 30+ wp sites whose content was removed. Separate cache times for media might avoid that issue : if (req.url ~ ".(gif|jpg|jpeg|png|css|js|java|fla|swf)$") { set beresp.http.cache-control = "public, max-age=2592000"; C{ #include static char timebuf[30]; char *format = {"%a, %d %b %Y %H:%M:%S GMT"}; struct tm timestruct; time_t now; time(&now); now+=2592000; gmtime_r(&now, ×truct); strftime(timebuf, 30, format, ×truct); VRT_SetHdr(sp, HDR_BERESP, "\010Expires:", timebuf, vrt_magic_string_end); }C set beresp.ttl = 12h; return(deliver); } Just set "12h" to whatever you want. Also : don't delete your data ;) ~kevin From simon at darkmere.gen.nz Tue Jun 28 03:59:40 2011 From: simon at darkmere.gen.nz (Simon Lyall) Date: Tue, 28 Jun 2011 15:59:40 +1200 (NZST) Subject: Varnish with Wordpress In-Reply-To: <015c01cc3535$3317d340$994779c0$@com> References: <015c01cc3535$3317d340$994779c0$@com> Message-ID: On Mon, 27 Jun 2011, kevin wrote: > Separate cache times for media might avoid that issue : > > if (req.url ~ ".(gif|jpg|jpeg|png|css|js|java|fla|swf)$") { > set beresp.http.cache-control = "public, max-age=2592000"; > C{ > #include > static char timebuf[30]; > char *format = {"%a, %d %b %Y %H:%M:%S GMT"}; > struct tm timestruct; > time_t now; > time(&now); > now+=2592000; > gmtime_r(&now, ×truct); > strftime(timebuf, 30, format, ×truct); > VRT_SetHdr(sp, HDR_BERESP, "\010Expires:", timebuf, > vrt_magic_string_end); > }C > set beresp.ttl = 12h; > return(deliver); > } Not sure you need all that trouble to generate the expires header. I copied the example of this page: https://www.varnish-cache.org/trac/wiki/VCLExampleLongerCaching so I only end up with a Last-Modified, Cache-Control and Date. Server: Apache Last-Modified: Wed, 06 Jun 2007 21:12:47 GMT Cache-Control: max-age=32832000 Content-Type: text/html; charset=UTF-8 Content-Length: 63 Accept-Ranges: bytes Date: Tue, 28 Jun 2011 03:56:20 GMT Connection: keep-alive Cache-Info: Object-Age=510885, hits=247477, Cache-Host=moth, Backend-Host=apn207, healthy=yes -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. From tfheen at varnish-software.com Tue Jun 28 08:23:43 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 28 Jun 2011 10:23:43 +0200 Subject: Cryillic letters in rewrite? In-Reply-To: <1307831048.2187.16.camel@cato-laptop> (Cato Aune's message of "Sun, 12 Jun 2011 00:24:08 +0200") References: <1307831048.2187.16.camel@cato-laptop> Message-ID: <87wrg6uzxs.fsf@qurzaw.varnish-software.com> ]] Cato Aune | It seems that | /company-%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D1%8F | works in the browser, but if I try to use that encondig in the .vcl | file, I get | Message from VCC-compiler: | Control character in %xx escape | (input Line 24 Pos 41) | set req.url = "/company-%D0%A0%D0%BE%D1%81%D1%81%D0%B8% | D1%8F"; Try using set req.url = {"/company-%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D1%8F"}; Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Tue Jun 28 08:28:53 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 28 Jun 2011 10:28:53 +0200 Subject: varnishncsa filtering? In-Reply-To: (Simon Lyall's message of "Fri, 17 Jun 2011 18:36:16 +1200 (NZST)") References: Message-ID: <87sjquuzp6.fsf@qurzaw.varnish-software.com> ]] Simon Lyall | Using 2.1.5 ( thought I'd wait a little while before trying 3.0.0 ) | | I'm trying to write a little regex to look for a certain part of the | URL but nothing seems to work. With 2.1.5, varnishncsa -o RxURL \\.js$ ought to work with 3.0, the syntax is varnishncsa -m 'RxURL:\.js$' -I doesn't really work right with the tools that look at complete transactions. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From tfheen at varnish-software.com Tue Jun 28 08:39:42 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 28 Jun 2011 10:39:42 +0200 Subject: Updating the 2.1 documentation on Web In-Reply-To: <20110617234527.GE22667@luke.kly.no> (Kristian Lyngstol's message of "Sat, 18 Jun 2011 01:45:27 +0200") References: <4DFB1E5F.4060206@googlemail.com> <4DFB20C5.3050603@googlemail.com> <20110617132713.GF2881@freud.kly.no> <20110617234527.GE22667@luke.kly.no> Message-ID: <87oc1iuz75.fsf@qurzaw.varnish-software.com> ]] Kristian Lyngstol | On Fri, Jun 17, 2011 at 05:03:49PM +0200, Magnus Hagander wrote: | > On Fri, Jun 17, 2011 at 15:27, Kristian Lyngstol | > wrote: | > > Unfortunately we had a typo that snuck into the docs, which was | > > fixed in March. And unfortunately again, the docs online for 2.1 are | > > only updated when we pull changes into the 2.1 branch preparing for | > > an other > > 2.1-release. | > | > Would it be much work to change it so the 2.1 docs on the web pull | > from the latest branch-tip for 2.1? ;) | | Probably not. | | Tollef? I fixed this independently yesterday, so fixed now. | The /doc/2.1 stuff in sphinx is rebuilt manually, on commit to 2.1 or on | tag or something similar? It's now rebuilt on each commit. | Oh, and we're going to need /doc/3.0 now :) Yup, did that yesterday. -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From phk at phk.freebsd.dk Tue Jun 28 08:42:55 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 28 Jun 2011 08:42:55 +0000 Subject: Is LCI on the radar? In-Reply-To: Your message of "Mon, 27 Jun 2011 16:41:18 +0200." <4B5D88B4-F620-43C0-AB8A-DE19565B22DF@pooteeweet.org> Message-ID: <49097.1309250575@critter.freebsd.dk> In message <4B5D88B4-F620-43C0-AB8A-DE19565B22DF at pooteeweet.org>, Lukas Kahwe S mith writes: >> I assume some of you have stumbled over LCI by now: >> http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt It's not a proposal I particular is happy about because it is very expensive to implement one way or the other. But I made a quick attempt and you can implement this using VCL and bans in Varnish already, so I don't expect to do more about it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From lajo at kb.dk Tue Jun 28 08:48:04 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Tue, 28 Jun 2011 08:48:04 +0000 Subject: lurker? Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> Hi, I constantly get tons of these in varnishlog: 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" What are they? I'm running 3.0. -- Lars -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.cao_huu_thien at upmc.fr Tue Jun 7 12:56:31 2011 From: patrick.cao_huu_thien at upmc.fr (Patrick CAO HUU THIEN) Date: Tue, 07 Jun 2011 12:56:31 -0000 Subject: time out on big object ?? In-Reply-To: <4DE62C67.1050504@uplex.de> References: <4DE3C245.9030406@uplex.de> <4DE62C67.1050504@uplex.de> Message-ID: <4DEE1FFA.5080801@upmc.fr> Geoff Simmons a dit le 06/01/2011 02:11 PM: > > Um, I don't follow. You *do* want Varnish to cache the file? If so, then > you don't want to return pass on the URL. In that case, you need a > larger value for send_timeout (assuming my guess is right that > send_timeout aborted your connection). Damned. No. I don't want to cache big objects. sorry for my bad english ... :( > > There's always a send_timeout, 10 minutes by default, but you can > increase it. If you have return(pass) on the URL, then the timeout won't > interfere with the transmission, but then Varnish won't be caching the > object. Are you sure about the disconnection between the pass action and the send_timeout parameter ? Ok, I understand the time-out of 10min. But with the return(pass) I've just added, varnish continue to make a time-out after 10min. So I have to put the send_timeout option to a higher value even if I make a return(pass). 1200s may not be enough because personal or foreign connection may be slower then the one I test at home. I put 6000s to be large :) thanks for your help Goeff Patrick From anand at rediff-inc.com Thu Jun 16 12:09:13 2011 From: anand at rediff-inc.com (Anand) Date: Thu, 16 Jun 2011 17:39:13 +0530 Subject: Alert on certain error like 503 In-Reply-To: References: Message-ID: Instead act in a correct way to handle those errors. Two remedies from my side: 1. 503 normally occurs when your backend does not respond. Increase you timeouts. Connect timeout by default is 0.004 sec. 2. Redirect it to another URL using vcl_error. Intelligence with elegant user experience. Regards, Anand _____ From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Shibashish Sent: 16 June 2011 17:14 To: varnish-misc at varnish-cache.org Subject: Alert on certain error like 503 Hi, Can varnish be made to send out an alert (mail) in case of an error like 503? I do have a monitoring solution but i can't add all the urls for monitoring. Sometimes varnish gives a 503 and i need to be alerted during that time. Another option can be to keep looking for 503 errors. log them as they occur and keep watching that file. ShiB. while ( ! ( succeed = try() ) ); -------------- next part -------------- An HTML attachment was scrubbed... URL: From mls at pooteeweet.org Tue Jun 28 08:49:56 2011 From: mls at pooteeweet.org (Lukas Kahwe Smith) Date: Tue, 28 Jun 2011 10:49:56 +0200 Subject: Is LCI on the radar? In-Reply-To: <49097.1309250575@critter.freebsd.dk> References: <49097.1309250575@critter.freebsd.dk> Message-ID: <087AF654-7980-4AE2-A8DE-34D6937FB081@pooteeweet.org> On 28.06.2011, at 10:42, Poul-Henning Kamp wrote: > In message <4B5D88B4-F620-43C0-AB8A-DE19565B22DF at pooteeweet.org>, Lukas Kahwe S > mith writes: > >>> I assume some of you have stumbled over LCI by now: >>> http://www.ietf.org/id/draft-nottingham-linked-cache-inv-00.txt > > It's not a proposal I particular is happy about because it is very > expensive to implement one way or the other. > > But I made a quick attempt and you can implement this using VCL and bans > in Varnish already, so I don't expect to do more about it. I would appreciate it a lot if your tests and Laurence's solution [1] could somehow we documented on the wiki. We will happily beta test it :) regards, Lukas Kahwe Smith mls at pooteeweet.org [1] http://dev.plone.org/collective/browser/experimental.depends/trunk/varnish.vcl From kristian at varnish-software.com Tue Jun 28 09:11:47 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Tue, 28 Jun 2011 11:11:47 +0200 Subject: lurker? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> Message-ID: <20110628091147.GB21377@freud.kly.no> On Tue, Jun 28, 2011 at 08:48:04AM +0000, Lars J?rgensen wrote: > I constantly get tons of these in varnishlog: > > 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" (....) > 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" > > What are they? I'm running 3.0. Are you sure it's 3.0.0 and not some pre-release? I assume this is coming in truly constantly, as a stream, not as bursts? - Kristian From phk at phk.freebsd.dk Tue Jun 28 09:13:02 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 28 Jun 2011 09:13:02 +0000 Subject: lurker? In-Reply-To: Your message of "Tue, 28 Jun 2011 08:48:04 GMT." <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> Message-ID: <54965.1309252382@critter.freebsd.dk> In message <6D2C830A0941EA40B6B483FE6EC98ADBC28503 at EXCHANGE-02.kb.dk>, =?iso-88 59-1?Q?Lars_J=F8rgensen?= writes: > >I constantly get tons of these in varnishlog: > > 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" That is the ban-lurker saying "hello" looking for stuff to do. Don't worry about it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From armdan20 at gmail.com Tue Jun 28 09:29:07 2011 From: armdan20 at gmail.com (andan andan) Date: Tue, 28 Jun 2011 11:29:07 +0200 Subject: varnishncsa dies on HUP signal. Message-ID: Hello. I'm testing Varnish 3.0 on CentOS 5 (2.6.18-238.5.1.el5 x86_64) using rpms from http://repo.varnish-cache.org/. varnishncsa dies on logrotate, the logrotate script do a kill -HUP to force refresh log files, this works fine on varnish 2.1.5 and previous versions. Debugging with strace: root 24777 0.0 0.0 95272 356 ? Ss 10:59 0:00 /usr/bin/varnishncsa -c -w /var/log/varnish/varnishncsa.log -D -P /var/run/varnishncsa.pid $> strace -p 24777 Process 24777 attached - interrupt to quit restart_syscall(<... resuming interrupted call ...>) = 0 nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, NULL) = 0 .... nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, 0) = ? ERESTART_RESTARTBLOCK (To be restarted) --- SIGHUP (Hangup) @ 0 (0) --- rt_sigreturn(0x1) = -1 EINTR (Interrupted system call) write(2, "Assert error in vsl_nextlog(), v"..., 95) = 95 write(2, " errno = 4 (Interrupted system "..., 38) = 38 rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0 tgkill(24777, 24777, SIGABRT) = 0 --- SIGABRT (Aborted) @ 0 (0) --- Process 24777 detached Thanks in advance. Kind Regards. From lajo at kb.dk Tue Jun 28 10:54:14 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Tue, 28 Jun 2011 10:54:14 +0000 Subject: varnishncsa dies on HUP signal. In-Reply-To: References: Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC28564@EXCHANGE-02.kb.dk> Same on RHEL 6, varnish compiled from source. I've just filed bug #947 on it. Lars -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of andan andan Sent: Tuesday, June 28, 2011 11:29 AM To: varnish-misc at varnish-cache.org Subject: varnishncsa dies on HUP signal. Hello. I'm testing Varnish 3.0 on CentOS 5 (2.6.18-238.5.1.el5 x86_64) using rpms from http://repo.varnish-cache.org/. varnishncsa dies on logrotate, the logrotate script do a kill -HUP to force refresh log files, this works fine on varnish 2.1.5 and previous versions. Debugging with strace: root 24777 0.0 0.0 95272 356 ? Ss 10:59 0:00 /usr/bin/varnishncsa -c -w /var/log/varnish/varnishncsa.log -D -P /var/run/varnishncsa.pid $> strace -p 24777 Process 24777 attached - interrupt to quit restart_syscall(<... resuming interrupted call ...>) = 0 nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, NULL) = 0 .... nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, NULL) = 0 nanosleep({0, 50000000}, 0) = ? ERESTART_RESTARTBLOCK (To be restarted) --- SIGHUP (Hangup) @ 0 (0) --- rt_sigreturn(0x1) = -1 EINTR (Interrupted system call) write(2, "Assert error in vsl_nextlog(), v"..., 95) = 95 write(2, " errno = 4 (Interrupted system "..., 38) = 38 rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0 tgkill(24777, 24777, SIGABRT) = 0 --- SIGABRT (Aborted) @ 0 (0) --- Process 24777 detached Thanks in advance. Kind Regards. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From kristian at varnish-software.com Tue Jun 28 10:56:17 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Tue, 28 Jun 2011 12:56:17 +0200 Subject: lurker? In-Reply-To: <54965.1309252382@critter.freebsd.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> Message-ID: <20110628105617.GD21377@freud.kly.no> On Tue, Jun 28, 2011 at 09:13:02AM +0000, Poul-Henning Kamp wrote: > In message <6D2C830A0941EA40B6B483FE6EC98ADBC28503 at EXCHANGE-02.kb.dk>, =?iso-88 > 59-1?Q?Lars_J=F8rgensen?= writes: > > > > >I constantly get tons of these in varnishlog: > > > > 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" > > That is the ban-lurker saying "hello" looking for stuff to do. > > Don't worry about it. I would still like to know if it's happening constantly or not, though... It could be harmless (or a optimization opportunity for your bans) or it could be something else. Most likely it's the former. Do you have any bans? Like PHK says, it's the ban-lurker saying "hi", but it should only do that if you have bans on your ban list that do not use "obj" but "req". - Kristian From cato.aune at aunenett.st.no Tue Jun 28 11:45:08 2011 From: cato.aune at aunenett.st.no (Cato Aune) Date: Tue, 28 Jun 2011 13:45:08 +0200 Subject: Cryillic letters in rewrite? In-Reply-To: <87wrg6uzxs.fsf@qurzaw.varnish-software.com> References: <1307831048.2187.16.camel@cato-laptop> <87wrg6uzxs.fsf@qurzaw.varnish-software.com> Message-ID: <4E09BEC4.70409@aunenett.st.no> Thanks Tollef, that solved the problem. Regards, Cato Den 28. juni 2011 10:23, skrev Tollef Fog Heen: > ]] Cato Aune > > | It seems that > | /company-%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D1%8F > | works in the browser, but if I try to use that encondig in the .vcl > | file, I get > | Message from VCC-compiler: > | Control character in %xx escape > | (input Line 24 Pos 41) > | set req.url = "/company-%D0%A0%D0%BE%D1%81%D1%81%D0%B8% > | D1%8F"; > > Try using > > set req.url = {"/company-%D0%A0%D0%BE%D1%81%D1%81%D0%B8%D1%8F"}; > > Regards, From tfheen at varnish-software.com Tue Jun 28 12:30:18 2011 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 28 Jun 2011 14:30:18 +0200 Subject: Maintaining historical docs and yay! In-Reply-To: <8C5B6E19-147F-4CA9-99C5-DD18AB79C256@gmail.com> (Stewart Robinson's message of "Fri, 17 Jun 2011 14:37:29 +0100") References: <8C5B6E19-147F-4CA9-99C5-DD18AB79C256@gmail.com> Message-ID: <87liwmt9yd.fsf@qurzaw.varnish-software.com> ]] Stewart Robinson Hi, | I'm very happy Varnish 3 came out and it looks fantastic. On a | historical note I think it is important that we have links to the | historical documentation from http://www.varnish-cache.org/docs | | A good proportion of users will be using the stock version of Varnish | that is packaged with their distribution. Can we find a way to link to | historical copies of the documentation on | http://www.varnish-cache.org/docs other than latest stable and | trunk. I'm not asking that we go back and regenerate docs for 2.0.x | but we could maintain the 2.1.x docs and 3.x docs as separate links on | the site. Just for the record, this is now in place, the ?latest stable? link pointed to 2.1, not 3.0, but it now points to 3.0 and there's a 2.1 link too. I have no intention of removing that any time soon. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 92 64 From roberto.fernandezcrisial at gmail.com Tue Jun 28 16:29:54 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Tue, 28 Jun 2011 13:29:54 -0300 Subject: Types Message-ID: Hi guys, I need to know which should be the best way to use "if (req.http.host == server.ip)" at vcl_recv subrutine with Varnish 3.0? I've been working with this "if" on Varnish 2.1.5 and it works fine. Now, with Varnish 3.0, I got "Comparison of different types: STRING '==' IP" error msg Thank you, Roberto @rofc -------------- next part -------------- An HTML attachment was scrubbed... URL: From fla_torres at yahoo.com.br Tue Jun 28 17:21:09 2011 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Tue, 28 Jun 2011 14:21:09 -0300 Subject: Types In-Reply-To: References: Message-ID: <4E0A0D85.7070603@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/28/2011 01:29 PM, Roberto O. Fern?ndez Crisial wrote: > Hi guys, > > I need to know which should be the best way to use "if > (req.http.host == server.ip)" at vcl_recv subrutine with Varnish 3.0? > I've been working with this "if" on Varnish 2.1.5 and it works fine. > Now, with Varnish 3.0, I got "Comparison of different types: STRING > '==' IP" error msg > Hello, Try: if (req.http.host == "192.168.0.1") { ... } Hope this helps. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4KDYMACgkQNRQApncg297NagCg4QXu4GP7cM9VdtJjxeDVt5zK /VYAoIIsETLFnuozIiaSFJhH1EIfZyed =jGdK -----END PGP SIGNATURE----- From roberto.fernandezcrisial at gmail.com Tue Jun 28 17:38:49 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Tue, 28 Jun 2011 14:38:49 -0300 Subject: Types In-Reply-To: <4E0A0D85.7070603@yahoo.com.br> References: <4E0A0D85.7070603@yahoo.com.br> Message-ID: Flavio, I need to use "if (req.http.host == server.ip)" because I had a lot of servers using the same VCL. Those servers are behind a HW balancer, which tests servers' status asking their IP addres, like: HEAD / HTTP/1.1 Host: www.xxx.yyy.zzz So I return "error 200" and the balancer understand servers are alive. My VCL lines are: if (req.http.host == server.ip) { error 200 "Balancer"; } Regards, Roberto @rofc 2011/6/28 Flavio Torres > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/28/2011 01:29 PM, Roberto O. Fern?ndez Crisial wrote: > > Hi guys, > > > > I need to know which should be the best way to use "if > > (req.http.host == server.ip)" at vcl_recv subrutine with Varnish 3.0? > > I've been working with this "if" on Varnish 2.1.5 and it works fine. > > Now, with Varnish 3.0, I got "Comparison of different types: STRING > > '==' IP" error msg > > > > Hello, > > Try: > > if (req.http.host == "192.168.0.1") { > ... > } > > Hope this helps. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk4KDYMACgkQNRQApncg297NagCg4QXu4GP7cM9VdtJjxeDVt5zK > /VYAoIIsETLFnuozIiaSFJhH1EIfZyed > =jGdK > -----END PGP SIGNATURE----- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fla_torres at yahoo.com.br Tue Jun 28 18:37:41 2011 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Tue, 28 Jun 2011 15:37:41 -0300 Subject: Types In-Reply-To: References: <4E0A0D85.7070603@yahoo.com.br> Message-ID: <4E0A1F75.20003@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/28/2011 02:38 PM, Roberto O. Fern?ndez Crisial wrote: > Flavio, > > I need to use "if (req.http.host == server.ip)" because I had a > lot of servers using the same VCL. Those servers are behind a HW > balancer, which tests servers' status asking their IP addres, > like: > > HEAD / HTTP/1.1 Host: www.xxx.yyy.zzz > > So I return "error 200" and the balancer understand servers are > alive. My VCL lines are: > > if (req.http.host == server.ip) { error 200 "Balancer"; } > Roberto, Sorry for my misunderstanding. Try server.ip in quotation marks: if (req.http.host == "server.ip"){ error 200 "Balancer"; } Btw, here I'm using the following vcl for the same purpose: vcl_recv { ... if (req.url == "/healthcheck_varnish.html") { error 200 "LIVE"; } ... } $ curl -H "Host: 127.0.0.1" http://localhost/healthcheck_varnish.html 200 LIVE

Error 200 LIVE

LIVE

Guru Meditation:

XID: 1871742327

Varnish cache server

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4KH3EACgkQNRQApncg2958hACfRulThp9qPHXP7az0ecmkSXT0 nioAn1bqfP4qCOT6yTMB9W4U49Du/iUS =dAa2 -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.fernandezcrisial at gmail.com Tue Jun 28 19:38:29 2011 From: roberto.fernandezcrisial at gmail.com (=?ISO-8859-1?Q?Roberto_O=2E_Fern=E1ndez_Crisial?=) Date: Tue, 28 Jun 2011 16:38:29 -0300 Subject: Types In-Reply-To: <4E0A1F75.20003@yahoo.com.br> References: <4E0A0D85.7070603@yahoo.com.br> <4E0A1F75.20003@yahoo.com.br> Message-ID: Flavio, I will try. Other option is to test "client.ip", as far I know HW balancers' IP address I will show "error 200" to them only. Thank you, Roberto @rofc 2011/6/28 Flavio Torres > ** > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/28/2011 02:38 PM, Roberto O. Fern?ndez Crisial wrote: > > Flavio, > > > > I need to use "if (req.http.host == server.ip)" because I had a > > lot of servers using the same VCL. Those servers are behind a HW > > balancer, which tests servers' status asking their IP addres, > > like: > > > > HEAD / HTTP/1.1 Host: www.xxx.yyy.zzz > > > > So I return "error 200" and the balancer understand servers are > > alive. My VCL lines are: > > > > if (req.http.host == server.ip) { error 200 "Balancer"; } > > > > > Roberto, > > Sorry for my misunderstanding. Try server.ip in quotation marks: > > > > if (req.http.host == "server.ip"){ > error 200 "Balancer"; > } > > > > Btw, here I'm using the following vcl for the same purpose: > > vcl_recv { > ... > if (req.url == "/healthcheck_varnish.html") { > error 200 "LIVE"; > } > ... > } > > > $ curl -H "Host: 127.0.0.1" http://localhost/healthcheck_varnish.html > > > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd" > > > > > 200 LIVE > > >

Error 200 LIVE

>

LIVE

>

Guru Meditation:

>

XID: 1871742327

>
>

Varnish cache server

> > > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk4KH3EACgkQNRQApncg2958hACfRulThp9qPHXP7az0ecmkSXT0 > nioAn1bqfP4qCOT6yTMB9W4U49Du/iUS > =dAa2 > -----END PGP SIGNATURE----- > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lajo at kb.dk Wed Jun 29 07:14:48 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Wed, 29 Jun 2011 07:14:48 +0000 Subject: lurker? In-Reply-To: <54965.1309252382@critter.freebsd.dk> References: Your message of "Tue, 28 Jun 2011 08:48:04 GMT." <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC286A8@EXCHANGE-02.kb.dk> >> >>I constantly get tons of these in varnishlog: >> >> 0 Debug - "lurker: 0x7f9018ffd970 3.1536e+08 0" > That is the ban-lurker saying "hello" looking for stuff to do. > Don't worry about it. I am worrying a little bit because it seems to snowball. This morning we had clients off of the Varnish (testing a deployment), and there was nothing going through varnish. It is logging about a 100 lines of those lurker things *a second*. Is that normal? -- Lars From lajo at kb.dk Wed Jun 29 07:17:18 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Wed, 29 Jun 2011 07:17:18 +0000 Subject: lurker? In-Reply-To: <20110628105617.GD21377@freud.kly.no> References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> <20110628105617.GD21377@freud.kly.no> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> > > Don't worry about it. > I would still like to know if it's happening constantly or not, > though... It could be harmless (or a optimization opportunity for your > bans) or it could be something else. Most likely it's the former. > Do you have any bans? Not that I know of. How do they look? I mean, it may be a term I'm unfamiliar with. There is no "ban" keyword in my VCL, but I am not aware if a "ban" is something else. I actually thought it was a purge, when looking at varnishadm... -- Lars From kristian at varnish-software.com Wed Jun 29 07:47:31 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Wed, 29 Jun 2011 09:47:31 +0200 Subject: lurker? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> <20110628105617.GD21377@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> Message-ID: <20110629074731.GA8986@freud.kly.no> On Wed, Jun 29, 2011 at 07:17:18AM +0000, Lars J?rgensen wrote: > > > Don't worry about it. > > > I would still like to know if it's happening constantly or not, > > though... It could be harmless (or a optimization opportunity for your > > bans) or it could be something else. Most likely it's the former. > > > Do you have any bans? > > Not that I know of. How do they look? I mean, it may be a term I'm > unfamiliar with. There is no "ban" keyword in my VCL, but I am not > aware if a "ban" is something else. I actually thought it was a purge, > when looking at varnishadm... What was known as purges in Varnish 2.x is known as bans in 3.0, so a bit of confusion... Use 'ban.list' on varnishadm and it should list any bans. You can also change the ban_lurker_sleep param to a higher value, though it shouldn't be a real problem. If you do find that you use bans, make sure they don't use 'req', but 'obj' instead where possible. I've explained bans and the ban lurker further in a blog post [1]. As for 100 per second.... The default sleep for the ban lurker is 0.01s, which means 100 per second, but again, smart bans shouldn't trigger log activity like what you're seeing and neither should an empty ban list. [1] https://kristianlyng.wordpress.com/2010/07/28/smart-bans-with-varnish/ - Kristian From lajo at kb.dk Wed Jun 29 10:50:05 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Wed, 29 Jun 2011 10:50:05 +0000 Subject: lurker? In-Reply-To: <20110629074731.GA8986@freud.kly.no> References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> <20110628105617.GD21377@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> <20110629074731.GA8986@freud.kly.no> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC28719@EXCHANGE-02.kb.dk> > What was known as purges in Varnish 2.x is known as bans in 3.0, so a > bit of confusion... > Use 'ban.list' on varnishadm and it should list any bans. You can also > change the ban_lurker_sleep param to a higher value, though it shouldn't > be a real problem. I actually had three bans. And now I'm really confused: I needed to purge a few pages from the cache, so I did ban.url /da/index.html and a couple other in varnishadm. I thought this would purge the url from the cache. It did, but why do they appear when I issue a ban.list? I thought it was a one-time purge command, not a list that will forever ban those urls from entering the cache. How do I purge a url once from the cache using varnishadm? I've restarted varnish to remove the bans (is there any other way?). I'll check whether the lurker starts ... lurking? again -- Lars From perbu at varnish-software.com Wed Jun 29 10:53:16 2011 From: perbu at varnish-software.com (Per Buer) Date: Wed, 29 Jun 2011 12:53:16 +0200 Subject: lurker? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC28719@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> <20110628105617.GD21377@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> <20110629074731.GA8986@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC28719@EXCHANGE-02.kb.dk> Message-ID: On Wed, Jun 29, 2011 at 12:50 PM, Lars J?rgensen wrote: > > What was known as purges in Varnish 2.x is known as bans in 3.0, so a > > bit of confusion... > > > Use 'ban.list' on varnishadm and it should list any bans. You can also > > change the ban_lurker_sleep param to a higher value, though it shouldn't > > be a real problem. > > I actually had three bans. And now I'm really confused: I needed to purge a > few pages from the cache, so I did > > ban.url /da/index.html > > and a couple other in varnishadm. I thought this would purge the url from > the cache. It did, but why do they appear when I issue a ban.list? I thought > it was a one-time purge command, not a list that will forever ban those urls > from entering the cache. How do I purge a url once from the cache using > varnishadm? > It would be a great help to you if you read the explanation given here: https://www.varnish-cache.org/docs/trunk/tutorial/purging.html Please let me know if it is unclear in any way and I'll try to rectify that. Cheers, Per. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnishlist at realvideosite.com Wed Jun 29 16:16:26 2011 From: varnishlist at realvideosite.com (Varnish List) Date: Wed, 29 Jun 2011 12:16:26 -0400 Subject: varnishlog and varnishncsa to syslog Message-ID: Does anyone know if its possible to send the output of varnishlog and varnishncsa to syslog? I am using syslog-ng and would like to use the filters there to separate errors, and everything else into separate log files with regex via syslog-ng.conf. Any help is appreciated! Thanks, Drew -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnishlist at realvideosite.com Wed Jun 29 16:21:02 2011 From: varnishlist at realvideosite.com (Varnish List) Date: Wed, 29 Jun 2011 12:21:02 -0400 Subject: varnishstat explanation? Message-ID: Does anyone know if there is a complete explanation for the statistics / coutners displayed by varnishstat? I found https://www.varnish-cache.org/trac/wiki/StatsExplained and http://kristianlyng.wordpress.com/2009/12/08/varnishstat-for-dummies/ but neither is complete. I'm trying to get a grasp on the SMA counters right now: SMA allocator requests SMA outstanding allocations SMA outstanding bytes SMA bytes allocated SMA bytes free Thanks! Drew -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacolmode at wp.pl Wed Jun 29 16:37:44 2011 From: jacolmode at wp.pl (Jacek Placek) Date: Wed, 29 Jun 2011 18:37:44 +0200 Subject: streaming service Message-ID: <4e0b54d82cd799.04026933@wp.pl> Hi, I'm planning on using Varnish in my project as a reverse proxy cache. Mainly to cache dynamically created streaming content at the backend side. The process is CPU intensive and takes a fair amount of time to complete. I would like to ask you all which of the following requirements can be accomplished by using the build-in functions, writing a dedicated module (version 3.0) or changing the default functionality by patching the source: 1) rate-limiting - a memcached set of counters, ex. ip address/time, uri/time, ... 2) streaming - content generated via a backend should be streamed ASAP to the client and saved to cache at the same time - ideally while the content is generated it would be very good to have the ability to stream to more then one client (non blocking object) 3) request manipulation - incoming request from a client is a HTTP GET (domain/object_id) and should be transformed into HTTP POST to a backend app extended with additional set of parameters (headers) taken from db (memcache, mongodb, tokyo tyrant, mysql) 4) cache expire - based on popularity and size of an object - free space should be reclaimed by purging least popular objects (amount of traffic they generate) Thank you in advance, Jacek. From perbu at varnish-software.com Wed Jun 29 17:59:30 2011 From: perbu at varnish-software.com (Per Buer) Date: Wed, 29 Jun 2011 19:59:30 +0200 Subject: varnishlog and varnishncsa to syslog In-Reply-To: References: Message-ID: On Wed, Jun 29, 2011 at 6:16 PM, Varnish List wrote: > Does anyone know if its possible to send the output of varnishlog and > varnishncsa to syslog? I am using syslog-ng and would like to use the > filters there to separate errors, and everything else into separate log > files with regex via syslog-ng.conf. > This is a syslog-ng question, not a varnish question. And I believe the question is "Can I pipe stdout from varnishlog or varnishncsa into syslog with syslog-ng?" and the answer, according to my googling is "no, but this is on the todo list" - see the syslog-ng FAQ for details. Cheers, Per. > > -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnishlist at realvideosite.com Wed Jun 29 21:16:05 2011 From: varnishlist at realvideosite.com (Varnish List) Date: Wed, 29 Jun 2011 17:16:05 -0400 Subject: varnishlog and varnishncsa to syslog In-Reply-To: References: Message-ID: Thanks for your help. My question was if i can get varnishlog and varnishncsa to syslog. I mentioned i was using syslog-ng as additional info in case it helped. It looks like i can daemonize varnishlog and varnishncsa and send stdout to a program like logger ( http://linux.about.com/library/cmd/blcmdl1_logger.htm) which will send it to syslog and syslog-ng will pick that up. It may not be the best way to do it but it seems to work. Thanks, Drew On Wed, Jun 29, 2011 at 1:59 PM, Per Buer wrote: > On Wed, Jun 29, 2011 at 6:16 PM, Varnish List < > varnishlist at realvideosite.com> wrote: > >> Does anyone know if its possible to send the output of varnishlog and >> varnishncsa to syslog? I am using syslog-ng and would like to use the >> filters there to separate errors, and everything else into separate log >> files with regex via syslog-ng.conf. >> > > This is a syslog-ng question, not a varnish question. And I believe the > question is "Can I pipe stdout from varnishlog or varnishncsa into syslog > with syslog-ng?" and the answer, according to my googling is "no, but this > is on the todo list" - see the syslog-ng FAQ for details. > > Cheers, > > Per. > >> >> -- > Per Buer, CEO > Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer > *Varnish makes websites fly!* > Whitepapers | Video | > Twitter > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guly at luv.guly.org Wed Jun 29 23:08:26 2011 From: guly at luv.guly.org (Sandro guly Zaccarini) Date: Thu, 30 Jun 2011 01:08:26 +0200 Subject: varnishlog and varnishncsa to syslog In-Reply-To: References: Message-ID: <20110629230826.GD9238@shivaya.guly.org> rsyslog can monitor dir/files, so you could tell him to monitor /var/log/varnish/ncsa.log and send it through syslog to @somehost. and FWIW syslog-ng has been depreciated in favour of rsyslog. maybe wrong list indeed :) sz -- /"\ taste your favourite IT consultant \ / gpg public key http://www.guly.org/guly.asc X / \ From l at lrowe.co.uk Wed Jun 29 23:18:37 2011 From: l at lrowe.co.uk (Laurence Rowe) Date: Thu, 30 Jun 2011 00:18:37 +0100 Subject: lurker? In-Reply-To: References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> <20110628105617.GD21377@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> <20110629074731.GA8986@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC28719@EXCHANGE-02.kb.dk> Message-ID: On 29 June 2011 11:53, Per Buer wrote: > It would be a great help to you if you read the?explanation?given here:?https://www.varnish-cache.org/docs/trunk/tutorial/purging.html > Please let me know if it is unclear in any way and I'll try to rectify that. It's unclear to me what the ``purge;`` command does? Does it remove varies perhaps? Ideally there would be some discussion on varies on that page. It appears in an example on https://www.varnish-cache.org/docs/trunk/reference/vcl.html but the specific command is not described there either. Laurence From jonathan.hursey at adrevolution.com Thu Jun 30 00:49:34 2011 From: jonathan.hursey at adrevolution.com (Jonathan Hursey) Date: Wed, 29 Jun 2011 19:49:34 -0500 Subject: backend question Message-ID: Hey folks, Anyone running a varnish server on the same server they host the backend content? Is this not recommended? -- *Jonathan M. Hursey* *Linux Systems Administrator* * * -------------- next part -------------- An HTML attachment was scrubbed... URL: From pprocacci at datapipe.com Thu Jun 30 01:15:09 2011 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Wed, 29 Jun 2011 20:15:09 -0500 Subject: backend question In-Reply-To: References: Message-ID: <20110630011509.GD21730@nat.myhome> Yes we do that for various clients of ours. The ones with a budget mainly. I see no downside to it personally. I think the recomendation to _NOT_ do that would depend on traffic patterns. On Wed, Jun 29, 2011 at 07:49:34PM -0500, Jonathan Hursey wrote: > Hey folks, > Anyone running a varnish server on the same server they host the backend > content? Is this not recommended? > > -- > *Jonathan M. Hursey* > *Linux Systems Administrator* > > * > * > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Paul Procacci Manager, UNIX Support Services Datapipe Managed Global IT Services 1.201.792.4847 (international) 1.888.749.5821 (toll free) ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From alexus at gmail.com Thu Jun 30 04:13:15 2011 From: alexus at gmail.com (alexus) Date: Thu, 30 Jun 2011 00:13:15 -0400 Subject: errno = Connection reset by peer Message-ID: [root at pvprx01 ~]# varnishlog | grep Debug | grep -v herding 73 Debug - "Write error, retval = -1, len = 11066, errno = Connection reset by peer" 91 Debug - "Write error, retval = -1, len = 8997, errno = Connection reset by peer" 85 Debug - "Write error, retval = -1, len = 58827, errno = Connection reset by peer" 0 Debug - "VCL_error(404, Unknown virtual host)" 90 Debug c "Write error, retval = -1, len = 12958, errno = Connection reset by peer" 40 Debug - "Write error, retval = -1, len = 11067, errno = Connection reset by peer" 18 Debug c "Write error, retval = -1, len = 9750, errno = Connection reset by peer" 26 Debug c "Write error, retval = -1, len = 8717, errno = Connection reset by peer" 72 Debug - "Write error, retval = -1, len = 11066, errno = Connection reset by peer" 120 Debug c "Write error, retval = -1, len = 11061, errno = Connection reset by peer" 40 Debug c "Write error, retval = -1, len = 8478, errno = Connection reset by peer" 17 Debug c "Write error, retval = -1, len = 44476, errno = Connection reset by peer" 90 Debug c "Write error, retval = -1, len = 25625, errno = Connection reset by peer" 0 Debug - "VCL_error(404, Unknown virtual host)" 105 Debug c "Write error, retval = -1, len = 8478, errno = Connection reset by peer" 70 Debug - "Write error, retval = -1, len = 12077, errno = Connection reset by peer" 95 Debug c "Write error, retval = -1, len = 11060, errno = Connection reset by peer" 112 Debug c "Write error, retval = -1, len = 50238, errno = Connection reset by peer" 42 Debug c "Write error, retval = -1, len = 35693, errno = Connection reset by peer" 116 Debug c "Write error, retval = -1, len = 32880, errno = Connection reset by peer" 0 Debug - "VCL_error(404, Unknown virtual host)" 85 Debug c "Write error, retval = -1, len = 9659, errno = Connection reset by peer" 56 Debug c "Write error, retval = -1, len = 51829, errno = Connection reset by peer" 77 Debug c "Write error, retval = -1, len = 10107, errno = Connection reset by peer" 42 Debug c "Write error, retval = -1, len = 22616, errno = Connection reset by peer" 70 Debug c "Write error, retval = -1, len = 10212, errno = Connection reset by peer" 0 Debug - "VCL_error(404, Unknown virtual host)" other then vcl_error(404, unknown virtual host) what that other message really mean? -- http://alexus.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From checker at d6.com Thu Jun 30 07:33:18 2011 From: checker at d6.com (Chris Hecker) Date: Thu, 30 Jun 2011 00:33:18 -0700 Subject: backend question In-Reply-To: <20110630011509.GD21730@nat.myhome> References: <20110630011509.GD21730@nat.myhome> Message-ID: <4E0C26BE.6000909@d6.com> I do this on my two main sites and it works great. I am definitely on a budget, so I'd agree with that part! Varnish allows me to take a crappy cheap dedicated machine (P4 2.8ghz, 1gb ram) and get reddited without a problem. I run it in front of a MediaWiki and a Wordpress site, and it takes both of them from ~15 reqs/sec to > 300 rps, with the former being 100% cpu maxed, and the latter being about 10% cpu usage. It'll max out the 100mbps link without breaking a sweat. I don't even bother with all of the wordpress and mediawiki caching stuff that everybody talks about all the time (I still use APC for various other reasons, though, and I use W3TC for managing AWS CF, but I have all its local caching features turned off), because I just solve all that with varnish, way better than the plugins possibly can. Note: I did some load tests and it turns out apache httpd isn't as slow as most articles on the internet claim (it'll do 300 rps on a simple php file as well, and it was basically identical to nginx+php-fpm on this hardware), it's just that wordpress is a giant pig. Anyway, it works great for small sites. I wonder how large a site you could serve on one of the AWS EC2 micro instances (which are in the free tier) with varnish, actually. They suck for httpd because if the CPU spikes too high AWS will throttle you to nothing, but varnish could conceivably fix that problem. Oh well, not enough time to experiment with this. Chris On 2011/06/29 18:15, Paul A. Procacci wrote: > Yes we do that for various clients of ours. The ones with a budget mainly. I see no downside to it personally. I think the recomendation to _NOT_ do that would depend on traffic patterns. > > On Wed, Jun 29, 2011 at 07:49:34PM -0500, Jonathan Hursey wrote: >> Hey folks, >> Anyone running a varnish server on the same server they host the backend >> content? Is this not recommended? >> >> -- >> *Jonathan M. Hursey* >> *Linux Systems Administrator* >> >> * >> * > >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > -- > Paul Procacci > Manager, UNIX Support Services > Datapipe Managed Global IT Services > 1.201.792.4847 (international) > 1.888.749.5821 (toll free) > > ________________________________ > > This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > From matthew at matthewwilkes.co.uk Thu Jun 30 08:28:29 2011 From: matthew at matthewwilkes.co.uk (Matthew Wilkes) Date: Thu, 30 Jun 2011 09:28:29 +0100 Subject: backend question In-Reply-To: <4E0C26BE.6000909@d6.com> References: <20110630011509.GD21730@nat.myhome> <4E0C26BE.6000909@d6.com> Message-ID: <0412B406-C021-479A-9391-199BFDB1FC40@matthewwilkes.co.uk> On 2011-06-30, at 0833, Chris Hecker wrote: > > I wonder how large a site you could serve on one of the AWS EC2 micro instances (which are in the free tier) with varnish, actually. They suck for httpd because if the CPU spikes too high AWS will throttle you to nothing, but varnish could conceivably fix that problem. Oh well, not enough time to experiment with this. I don't know about one, but on 6 I've served http://www.homelandsecuritydialogue.org/ when it was getting > 100 rps. About 1/5 of requests caused some sort of cache perge and many were esi; We moved between 1 and 8 as the traffic spiked, currently at one instance with cold caches (Varnish, database and HD) as it's not in use even as an archive anymore. Matt From lajo at kb.dk Thu Jun 30 10:23:45 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Thu, 30 Jun 2011 10:23:45 +0000 Subject: lurker? In-Reply-To: References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> <20110628105617.GD21377@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> <20110629074731.GA8986@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC28719@EXCHANGE-02.kb.dk> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC289D3@EXCHANGE-02.kb.dk> > It would be a great help to you if you read the?explanation?given here:?https://www.varnish- > cache.org/docs/trunk/tutorial/purging.html > Please let me know if it is unclear in any way and I'll try to rectify that. That page confirms what I've learned until now, thank you for the link. Just one question: It says "Usually a purge is invoked through HTTP with the method PURGE". Usually? Is there another way? I would like to do it via varnishadm, if possible. -- Lars From lajo at kb.dk Thu Jun 30 10:36:44 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Thu, 30 Jun 2011 10:36:44 +0000 Subject: lurker? In-Reply-To: <20110629074731.GA8986@freud.kly.no> References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> <20110628105617.GD21377@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> <20110629074731.GA8986@freud.kly.no> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC28A47@EXCHANGE-02.kb.dk> > What was known as purges in Varnish 2.x is known as bans in 3.0, so a > bit of confusion... Is that really the case? According to the documentation, both concepts still exists. As I understand them, a ban is a permanent filter, a purge is a deletion of cache object(s). -- Lars From perbu at varnish-software.com Thu Jun 30 10:48:49 2011 From: perbu at varnish-software.com (Per Buer) Date: Thu, 30 Jun 2011 12:48:49 +0200 Subject: lurker? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC28A47@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> <20110628105617.GD21377@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> <20110629074731.GA8986@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC28A47@EXCHANGE-02.kb.dk> Message-ID: On Thu, Jun 30, 2011 at 12:36 PM, Lars J?rgensen wrote: > > What was known as purges in Varnish 2.x is known as bans in 3.0, so a > > bit of confusion... > > Is that really the case? According to the documentation, both concepts > still exists. As I understand them, a ban is a permanent filter, a purge is > a deletion of cache object(s). > No. Bans are not permanent. They are *lazy*. They are applied at a later time, when the relevant object is requested. If the object is *newer* than the ban the band doesn't apply. -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at varnish-software.com Thu Jun 30 10:51:53 2011 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Thu, 30 Jun 2011 12:51:53 +0200 Subject: lurker? In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC28A47@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> <20110628105617.GD21377@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> <20110629074731.GA8986@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC28A47@EXCHANGE-02.kb.dk> Message-ID: <20110630105153.GD2672@freud.kly.no> On Thu, Jun 30, 2011 at 10:36:44AM +0000, Lars J?rgensen wrote: > > What was known as purges in Varnish 2.x is known as bans in 3.0, so a > > bit of confusion... > > Is that really the case? According to the documentation, both concepts > still exists. As I understand them, a ban is a permanent filter, a > purge is a deletion of cache object(s). In Varnish 2: - purge() from vcl AND cli took a regular expression that added a filter on a list. No objects in cache at the time of entry (that is: when purge() was run) that matches that regular expression would be served again - a new object would be fetched from the backend instead if requeted. - No memory was freed using purge(), excet using the 'ban lurker' (which was, confusingly, never called the "purge lurker"). - "nuke" was an internal Varnish term that could free a single object in cache. It was never exposed in CLI or VCL. In Varnish 3: - ban() from vcl AND cli takes regular expressions and is the same as purge() from Varnish 2.0. - No memory is freed using ban() in Varnish 3.0. - purge; is exposed to VCL: This is a way to expose the previously internal 'nuke' mechanism, and allows you to remove a single object and all its variants. It only works on a single object+variants, and does not take a regular expression. It acts immediately and has no lingering effect save that some memory is freed and an object is removed from cache. So yes, both ban() and purge; is available in Varnish 3. But what was known as purge() in Varnish 2 has been re-named, and purge; in Varnish 3 is something entirely different, more similar to setting obj.ttl and obj.grace to 0s for all variants of an object in vcl_hit. This name-mixing is unfortunate, but believed to be less confusing in the future when the ban()-vs-purge; terminology has had a chance to settle itself in our documentation, wiki and daily talk. The "PURGE" method (all caps) is usually used to reference a VCL implementation of the fictional http PURGE method, which is intended to do what purge; does in Varnish 3. In Varnish 2, it was usually implemented by setting ttl and grace to 0s in vcl_hit, which would not take care of all variants of an object. I hope this clears things up a bit. - Kristian From lajo at kb.dk Thu Jun 30 10:59:44 2011 From: lajo at kb.dk (=?iso-8859-1?Q?Lars_J=F8rgensen?=) Date: Thu, 30 Jun 2011 10:59:44 +0000 Subject: Bans vs. Purges (was RE: lurker?) In-Reply-To: References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> <20110628105617.GD21377@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> <20110629074731.GA8986@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC28A47@EXCHANGE-02.kb.dk> Message-ID: <6D2C830A0941EA40B6B483FE6EC98ADBC28A6B@EXCHANGE-02.kb.dk> >> Is that really the case? According to the documentation, both concepts still exists. As I understand them, a ban >> is a permanent filter, a purge is a deletion of cache object(s). > No. Bans are not permanent. They are *lazy*. They are applied at a later time, when the relevant object is > requested. If the object is *newer* than the ban the band doesn't apply. Huh. Please walk me through an example like this: 1. User requests index.html, it's a miss, so varnish fetches it, stores it in the cache and delivers it 2. I ban index.html 3. User requests index.html. The current object in the cache is banned, so a new version is fetched, stored and deliverd 4. User requests index.html. There are two of those in the cache, but one is older than the ban, so it will not be served. What is the purpose of banning an object when you could just delete it? The way I understand it, when a ban is in effect, multiple copies of the same object will exist in the cache, and varnish has to check a ban list to find out which ones to serve. Seems a convoluted way of doing things, but I suspect I'm missing a major point somewhere. I'm sorry for wasting time if everybody else understands this, but it is puzzling to me. -- Lars From ck-lists at cksoft.de Thu Jun 30 11:17:05 2011 From: ck-lists at cksoft.de (Christian Kratzer) Date: Thu, 30 Jun 2011 13:17:05 +0200 (CEST) Subject: Bans vs. Purges (was RE: lurker?) In-Reply-To: <6D2C830A0941EA40B6B483FE6EC98ADBC28A6B@EXCHANGE-02.kb.dk> References: <6D2C830A0941EA40B6B483FE6EC98ADBC28503@EXCHANGE-02.kb.dk> <54965.1309252382@critter.freebsd.dk> <20110628105617.GD21377@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC286B7@EXCHANGE-02.kb.dk> <20110629074731.GA8986@freud.kly.no> <6D2C830A0941EA40B6B483FE6EC98ADBC28A47@EXCHANGE-02.kb.dk> <6D2C830A0941EA40B6B483FE6EC98ADBC28A6B@EXCHANGE-02.kb.dk> Message-ID: Hi, On Thu, 30 Jun 2011, Lars J?rgensen wrote: >>> Is that really the case? According to the documentation, both concepts still exists. As I understand them, a ban >>> is a permanent filter, a purge is a deletion of cache object(s). > >> No. Bans are not permanent. They are *lazy*. They are applied at a later time, when the relevant object is >> requested. If the object is *newer* than the ban the band doesn't apply. > > Huh. > > Please walk me through an example like this: > > 1. User requests index.html, it's a miss, so varnish fetches it, stores it in the cache and delivers it > 2. I ban index.html > 3. User requests index.html. The current object in the cache is banned, so a new version is fetched, stored and deliverd > 4. User requests index.html. There are two of those in the cache, but one is older than the ban, so it will not be served. > > What is the purpose of banning an object when you could just delete it? The way I understand it, when a ban is in effect, multiple copies of the same object will exist in the cache, and varnish has to check a ban list to find out which ones to serve. Seems a convoluted way of doing things, but I suspect I'm missing a major point somewhere. the purpose of the ban.list is to avoid walking through potentially hundreds of thousands of objects and finding the correct once to delete. Adding the expression to a short list to check on every hit is by far the smaller cost. This has the same effect as deleting the respective objects from the cache. It's just cheaper for varnish. > I'm sorry for wasting time if everybody else understands this, but it is puzzling to me. Greetings Christian -- Christian Kratzer CK Software GmbH Email: ck at cksoft.de Wildberger Weg 24/2 Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht Stuttgart Web: http://www.cksoft.de/ Geschaeftsfuehrer: Christian Kratzer From flatcircle at hotmail.com Thu Jun 30 16:41:30 2011 From: flatcircle at hotmail.com (Flatcircle) Date: Thu, 30 Jun 2011 18:41:30 +0200 Subject: Drupal 7.x & Varnish 3.x Message-ID: Has anyone succeeded in getting Varnish 3.x and Drupal 7.x up & running? Since Varnish 3 the info at https://www.varnish-cache.org/trac/wiki/VarnishAndDrupal is outdated. Below is the content of my default.vcl: I can start Varnish (3.0) but it seems that Varnish isn't caching. 1) In Firebug: I only see one Varnish timestamp instead of two (so varnish isn't serving from cache) (f.e. X-Varnish 453348501) 2) When I check with 'varnishlog -b' and refresh my Drupal pages, I see that Varnish still makes requests to the web backend instead of serving from cache. Any ideas what's missing in my default.vcl to get Varnish 3.0 up & running with Drupal 7? -------------------------- backend default { .host = "127.0.0.1"; .port = "8080"; .connect_timeout = 600s; .first_byte_timeout = 600s; .between_bytes_timeout = 600s; } sub vcl_recv { if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } // Remove has_js and Google Analytics cookies. set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", ""); if (req.http.Authorization || req.http.Cookie) { /* Not cacheable by default */ return (pass); } // Remove a ";" prefix, if present. set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); // Remove empty cookies. if (req.http.Cookie ~ "^\s*$") { unset req.http.Cookie; } // Skip the Varnish cache for install, update, and cron if (req.url ~ "install\.php|update\.php|cron\.php") { return (pass); } // Normalize the Accept-Encoding header // as per: http://varnish-cache.org/wiki/FAQ/Compression if (req.http.Accept-Encoding) { if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { # No point in compressing these remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } else { # Unknown or deflate algorithm remove req.http.Accept-Encoding; } } // Let's have a little grace set req.grace = 30s; return (lookup); } // Strip any cookies before an image/js/css is inserted into cache. sub vcl_fetch { if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { unset beresp.http.set-cookie; } } sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; set obj.http.Retry-After = "5"; synthetic {" "} + obj.status + " " + obj.response + {"

Error "} + obj.status + " " + obj.response + {"

"} + obj.response + {"

Guru Meditation:

XID: "} + req.xid + {"

Varnish cache server

"}; return (deliver); } -------------------------- Regards. From phk at phk.freebsd.dk Thu Jun 30 12:04:07 2011 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 30 Jun 2011 12:04:07 +0000 Subject: Bans vs. Purges (was RE: lurker?) In-Reply-To: Your message of "Thu, 30 Jun 2011 10:59:44 GMT." <6D2C830A0941EA40B6B483FE6EC98ADBC28A6B@EXCHANGE-02.kb.dk> Message-ID: <2774.1309435447@critter.freebsd.dk> In message <6D2C830A0941EA40B6B483FE6EC98ADBC28A6B at EXCHANGE-02.kb.dk>, =?iso-88 59-1?Q?Lars_J=F8rgensen?= writes: >3. User requests index.html. The current object in the cache is banned, so a new version is fetched, stored and deliverd ... and the banned object is expired immediately. >What is the purpose of banning an object when you could just delete it? Banning allows you to use a regular expression on any header you want. "just deleting .*[.].jpg" would take walking over the entire cache to find all matching objects. Varnish does this with "lazy evaluation", we only check when we are about to deliver an object. This minimizes the work, for instance objects that expire after then ban was entered without being hit in the meantime, are not checked. Purging works by doing a lookup, so you can only do it with a hash-value (typically URL+Host:), and it expires the objects it find. Purging is much cheaper, because we can go directly after the objects we want. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ccastro at altavoz.net Thu Jun 30 16:55:49 2011 From: ccastro at altavoz.net (Claudio Castro) Date: Thu, 30 Jun 2011 12:55:49 -0400 Subject: backend question In-Reply-To: References: Message-ID: <4E0CAA95.4050807@altavoz.net> Apache and Varnish fights for RAM, swap usage goes up and you get some instability. Nothing to worry if you got a small/medium site, i use it only on emergency times :D Greets, Jonathan Hursey wrote: > Hey folks, > Anyone running a varnish server on the same server they host the > backend content? Is this not recommended? > > -- > *Jonathan M. Hursey* > /Linux Systems Administrator/ > > / > / > > ------------------------------------------------------------------------ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Claudio Castro N. Jefe de Plataforma AltaVoz S.A. http://www.altavoz.net Vi?a del Mar: 2 Poniente 355 of 53 +56 32 276 8060 Santiago: Guardia Vieja 255, oficina 612 +56 2 585 4264 From perbu at varnish-software.com Thu Jun 30 17:08:40 2011 From: perbu at varnish-software.com (Per Buer) Date: Thu, 30 Jun 2011 19:08:40 +0200 Subject: backend question In-Reply-To: <4E0CAA95.4050807@altavoz.net> References: <4E0CAA95.4050807@altavoz.net> Message-ID: Hi, It should be easy to configure your way around this. Limit the size of Varnish Cache and limit the number of Apache processes. This might slow down the traffic that is not served from cache, but that far better then the whole thing exploding. Per. On Thu, Jun 30, 2011 at 6:55 PM, Claudio Castro wrote: > Apache and Varnish fights for RAM, swap usage goes up and you get some > instability. Nothing to worry if you got a small/medium site, i use it only > on emergency times :D > > Greets, > > Jonathan Hursey wrote: > >> Hey folks, Anyone running a varnish server on the same server they >> host the backend content? Is this not recommended? >> -- >> *Jonathan M. Hursey* >> /Linux Systems Administrator/ >> >> / >> / >> >> ------------------------------**------------------------------** >> ------------ >> >> >> ______________________________**_________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc >> > > > -- > Claudio Castro N. > Jefe de Plataforma > AltaVoz S.A. > http://www.altavoz.net > Vi?a del Mar: > 2 Poniente 355 of 53 > +56 32 276 8060 > Santiago: > Guardia Vieja 255, oficina 612 > +56 2 585 4264 > > > > ______________________________**_________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc > -- Per Buer, CEO Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer *Varnish makes websites fly!* Whitepapers | Video | Twitter -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtshilston at gmail.com Thu Jun 30 17:51:34 2011 From: rtshilston at gmail.com (Rob S) Date: Thu, 30 Jun 2011 18:51:34 +0100 Subject: NaN in varnishlog ReqEnd Message-ID: Hi, I'm running varnish 2.1.5 (SVN 0843d7a), and I'm seeing quite odd numbers in my ReqEnd. 601 ReqEnd c 483016025 1309454855.855156898 1309454865.751157045 -9.895982027 nan nan The documentation at https://www.varnish-cache.org/trac/wiki/Varnishlog doesn't really explain any reason for having a negative fourth number, or for nans in the fifth and sixth positions. I think this is connected with ESIs - the page being loaded is one which includes several ESI fragments. I'm looking at the ReqEnd line as I'm trying to identify whether this page is being slow to generate, or if the http clients are being slow to retrieve the data. Can anyone offer more information on this? Rob