From bluethundr at gmail.com Mon Jun 2 16:08:20 2014 From: bluethundr at gmail.com (Tim Dunphy) Date: Mon, 2 Jun 2014 12:08:20 -0400 Subject: varnish is sticking too much to web nodes Message-ID: We have two out of six web nodes showing extremely high CPU usage compared to the others in the farm. We're using the round-robin load balancing algorithm. Using a tool like 'htop' shows us that all the hungry processes belong to apache. Yet diffing the configuration files between the nodes with low cpu usage and the ones with high cpu usage shows no difference at all. The theory is that varnish is throwing traffic at these two hosts too often. So I was just wondering if there could be any way to diagnose this from a Varnish perspective. What clues could tools like varnishtop and varnishstat or varnishlog yield in investigating this type of problem? Here's my varnish vcl (cleaned of any too-identifying info) in case anyone can offer an opinion as to why this is going on: backend web1 { .host = "10.10.1.98"; .port = "80"; .connect_timeout = 90s; .first_byte_timeout = 90s; .between_bytes_timeout = 90s; .max_connections = 70; .probe = { .url = "/healthcheck.php"; .timeout = 5s; .interval = 30s; .window = 10; .threshold = 1; } } backend web2 { .host = "10.10.1.99"; .port = "80"; .connect_timeout = 90s; .first_byte_timeout = 90s; .between_bytes_timeout = 90s; .max_connections = 70; .probe = { .url = "/healthcheck.php"; .timeout = 5s; .interval = 30s; .window = 10; .threshold = 1; } } backend web3 { .host = "10.10.1.100"; .port = "80"; .connect_timeout = 90s; .first_byte_timeout = 90s; .between_bytes_timeout = 90s; .max_connections = 70; .probe = { .url = "/healthcheck.php"; .timeout = 5s; .interval = 30s; .window = 10; .threshold = 1; } } backend web4 { .host = "10.10.1.101"; .port = "80"; .connect_timeout = 90s; .first_byte_timeout = 90s; .between_bytes_timeout = 90s; .max_connections = 70; .probe = { .url = "/healthcheck.php"; .timeout = 5s; .interval = 30s; .window = 10; .threshold = 1; } } backend web5 { .host = "10.10.1.235"; .port = "80"; .connect_timeout = 90s; .first_byte_timeout = 90s; .between_bytes_timeout = 90s; .max_connections = 70; .probe = { .url = "/healthcheck.php"; .timeout = 5s; .interval = 30s; .window = 10; .threshold = 1; } } backend web6 { .host = "10.10.1.236"; .port = "80"; .connect_timeout = 90s; .first_byte_timeout = 90s; .between_bytes_timeout = 90s; .max_connections = 70; .probe = { .url = "/healthcheck.php"; .timeout = 5s; .interval = 30s; .window = 10; .threshold = 1; } } backend varnish1 { .host = "10.10.1.96"; .port = "80"; .connect_timeout = 90s; .first_byte_timeout = 90s; .between_bytes_timeout = 90s; .max_connections = 1000; } backend varnish2 { .host = "10.10.1.97"; .port = "80"; .connect_timeout = 90s; .first_byte_timeout = 90s; .between_bytes_timeout = 90s; .max_connections = 1000; } acl purge { "localhost"; "127.0.0.1"; "10.10.1.96"; "10.10.1.97"; } director www round-robin { { .backend = web1; } { .backend = web2; } { .backend = web3; } { .backend = web4; } { .backend = web5; } { .backend = web6; } } director cache round-robin { { .backend = varnish1; } { .backend = varnish2; } } sub vcl_recv { set req.backend = www; if (req.http.User-Agent ~ "typhoeus") { error 403 "Temporarily disabled"; } set req.grace = 6h; if (! req.backend.healthy) { set req.grace = 24h; } set req.http.X-Forwarded-For = req.http.X-Forwarded-For ", " client.ip; /*if (req.http.host ~ "^origin\.test\.(.+\.|)mywebsite.com$") { return(pass); }*/ if (req.http.host ~ "^origin\.(.+\.|)mywebsite\.com$") { return(pass); } if (req.http.host ~ "test.cms.mywebsite.com|test.cms-2.mywebsite.com| test.facebook.mywebsite.com") { return (pass); } if (req.http.host ~ "test.cache-only.mywebsite.com") { set req.backend = cache; } /* if (req.http.host ~ "test\.*\.mywebsite.com") { if (req.http.host !~ "test\.(.+\.|)m\.mywebsite\.com$") { set req.http.host = "test.mywebsite.com"; } } else { return (pass); }*/ if (req.http.host ~ ".*\.mywebsite.com") { if (req.http.host !~ "(.+\.|)m\.mywebsite\.com$") { set req.http.host = "www.mywebsite.com"; } } else { return (pass); } /* if (req.url ~ "^/node/75402/tweettracker_brand/config/xml") { error 403 "Temporarily disabled"; } */ if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } return (lookup); } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { return (pipe); } if (req.request != "GET" && req.request != "HEAD") { return (pass); } if (req.url ~ "eyeblaster/addineyeV2.html\?.*") { set req.url = "/eyeblaster/addineyeV2.html"; } if (req.url ~ "sites/all/modules/custom/bravo_ad/ads.html\?.*") { set req.url = "/sites/all/modules/custom/bravo_ad/ads.html"; } if (req.url ~ "sites/ m.mywebsite.com/modules/custom/bravo_ad/ads.html\?.*") { set req.url = "/sites/ m.mywebsite.com/modules/custom/bravo_ad/ads.html"; } if (req.url ~ "ahah_helper\.php|bravo_points\.php|install\.php|update\.php|cron\.php|/json(:?\?.*)?$") { return (pass); } if (req.http.Authorization) { return (pass); } if (req.url ~ "login" || req.url ~ "logout") { return (pass); } if (req.url ~ "^/admin/" || req.url ~ "^/node/add/") { return (pass); } if (req.http.Cache-Control ~ "no-cache") { //return (pass); } if (req.http.Cookie ~ "(VARNISH|DRUPAL_UID|LOGGED_IN|SESS|_twitter_sess)") { set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", ""); set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); } else { unset req.http.Cookie; } if (req.restarts == 0) { if (client.ip == "10.10.1.96" || client.ip == "10.10.1.97") { set req.backend = www; } elsif (server.ip == "10.10.1.96") { set req.backend = varnish2; } else { set req.backend = varnish1; } } elsif (req.restarts >= 2) { return (pass); } if (req.restarts >= 2) { return (pass); } if (req.url ~ "\.(ico|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|ICO|JPG|JPEG|PNG|GIF|GZ|TGZ|BZ2|TBZ|MP3|OOG|SWF)") { unset req.http.Accept-Encoding; } if (req.url ~ "^/(sites/all/modules/mywebsite_admanager/includes/ads.php|doubleclick/DARTIframe.html)(\?.*|)$") { set req.url = regsub(req.url, "\?.*$", ""); } /* keep user-agents if (! req.http.User-Agent ~ "MSIE") { set req.http.User-Agent = "Mozilla"; } */ if (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } return (lookup); } sub vcl_pipe { set bereq.http.connection = "close"; return (pipe); } sub vcl_pass { return (pass); } sub vcl_hash { set req.hash += req.url; set req.hash += req.http.host; if (req.http.Cookie ~ "VARNISH|DRUPAL_UID|LOGGED_IN") { set req.hash += req.http.Cookie; } return (hash); } sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } /* if (!obj.cacheable) { return (pass); } return (deliver); */ } sub vcl_fetch { if (beresp.status == 500) { set req.http.X-Varnish-Error = "1"; restart; } set beresp.grace = 6h; # Set a short circuit cache lifetime for resp codes above 302 if (beresp.status > 302) { set beresp.ttl = 60s; set beresp.http.Cache-Control = "max-age = 60"; } if (beresp.http.Edge-control ~ "no-store") { set beresp.http.storage = "1"; set beresp.cacheable = false; return (pass); } if (beresp.status >= 300 || !beresp.cacheable) { set beresp.http.Varnish-X-Cacheable = "Not Cacheable"; set beresp.http.storage = "1"; return (pass); } if (beresp.http.Set-Cookie) { return (pass); } if (beresp.cacheable) { unset beresp.http.expires; set beresp.ttl = 600s; set beresp.http.Cache-Control = "max-age = 600"; if (req.url ~ "\.(ico|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|ICO|JPG|JPEG|PNG|GIF|GZ|TGZ|BZ2|TBZ|MP3|OOG|SWF)") { set beresp.ttl = 43829m; set beresp.http.Cache-Control = "max-age = 1000000"; } /* if (req.url ~ "json(:?\??.*)?$") { set beresp.ttl = 30s; set beresp.http.Cache-Control = "max-age = 30"; } */ } return (deliver); } sub vcl_deliver { if (obj.hits > 0) { set resp.http.Varnish-X-Cache = "HIT"; set resp.http.Varnish-X-Cache-Hits = obj.hits; } else { set resp.http.Varnish-X-Cache = "MISS"; } return (deliver); } sub vcl_error { if (req.restarts == 0) { return (restart); } if (req.http.X-Varnish-Error != "1") { set req.http.X-Varnish-Error = "1"; return (restart); } set obj.http.Content-Type = "text/html; charset=utf-8"; synthetic {" > mywebsite.com "} obj.status " " obj.response {"
"}; return (deliver); } Thanks Tim -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B -------------- next part -------------- An HTML attachment was scrubbed... URL: From guery.b at gmail.com Tue Jun 3 08:08:54 2014 From: guery.b at gmail.com (=?ISO-8859-1?Q?Boris_Gu=E9ry?=) Date: Tue, 03 Jun 2014 10:08:54 +0200 Subject: varnish is sticking too much to web nodes In-Reply-To: References: Message-ID: <538D8296.7080705@gmail.com> Maybe you can start to check the request counters of Apache using its status module. You should be able to see how much requests your webserver is handling at a given time. Check http://httpd.apache.org/docs/2.2/mod/mod_status.html It should also be easy to create a script which check every n seconds the apache status of each servers and aggregate data for further analysis. Le 02/06/14 18:08, Tim Dunphy a ?crit : > > We have two out of six web nodes showing extremely high CPU usage > compared to the others in the farm. > > We're using the round-robin load balancing algorithm. Using a tool > like 'htop' shows us that all the hungry processes belong to apache. > Yet diffing the configuration files between the nodes with low cpu > usage and the ones with high cpu usage shows no difference at all. > > The theory is that varnish is throwing traffic at these two hosts too > often. > > So I was just wondering if there could be any way to diagnose this > from a Varnish perspective. What clues could tools like varnishtop and > varnishstat or varnishlog yield in investigating this type of problem? > > Here's my varnish vcl (cleaned of any too-identifying info) in case > anyone can offer an opinion as to why this is going on: > > backend web1 { > .host = "10.10.1.98"; > .port = "80"; > .connect_timeout = 90s; > .first_byte_timeout = 90s; > .between_bytes_timeout = 90s; > .max_connections = 70; > .probe = { > .url = "/healthcheck.php"; > .timeout = 5s; > .interval = 30s; > .window = 10; > .threshold = 1; > } > } > > backend web2 { > .host = "10.10.1.99"; > .port = "80"; > .connect_timeout = 90s; > .first_byte_timeout = 90s; > .between_bytes_timeout = 90s; > .max_connections = 70; > .probe = { > .url = "/healthcheck.php"; > .timeout = 5s; > .interval = 30s; > .window = 10; > .threshold = 1; > } > } > > backend web3 { > .host = "10.10.1.100"; > .port = "80"; > .connect_timeout = 90s; > .first_byte_timeout = 90s; > .between_bytes_timeout = 90s; > .max_connections = 70; > .probe = { > .url = "/healthcheck.php"; > .timeout = 5s; > .interval = 30s; > .window = 10; > .threshold = 1; > } > } > > backend web4 { > .host = "10.10.1.101"; > .port = "80"; > .connect_timeout = 90s; > .first_byte_timeout = 90s; > .between_bytes_timeout = 90s; > .max_connections = 70; > .probe = { > .url = "/healthcheck.php"; > .timeout = 5s; > .interval = 30s; > .window = 10; > .threshold = 1; > } > } > > backend web5 { > .host = "10.10.1.235"; > .port = "80"; > .connect_timeout = 90s; > .first_byte_timeout = 90s; > .between_bytes_timeout = 90s; > .max_connections = 70; > .probe = { > .url = "/healthcheck.php"; > .timeout = 5s; > .interval = 30s; > .window = 10; > .threshold = 1; > } > } > > backend web6 { > .host = "10.10.1.236"; > .port = "80"; > .connect_timeout = 90s; > .first_byte_timeout = 90s; > .between_bytes_timeout = 90s; > .max_connections = 70; > .probe = { > .url = "/healthcheck.php"; > .timeout = 5s; > .interval = 30s; > .window = 10; > .threshold = 1; > } > } > > backend varnish1 { > .host = "10.10.1.96"; > .port = "80"; > .connect_timeout = 90s; > .first_byte_timeout = 90s; > .between_bytes_timeout = 90s; > .max_connections = 1000; > } > > backend varnish2 { > .host = "10.10.1.97"; > .port = "80"; > .connect_timeout = 90s; > .first_byte_timeout = 90s; > .between_bytes_timeout = 90s; > .max_connections = 1000; > } > > acl purge { > "localhost"; > "127.0.0.1"; > "10.10.1.96"; > "10.10.1.97"; > } > > director www round-robin { > { .backend = web1; } > { .backend = web2; } > { .backend = web3; } > { .backend = web4; } > { .backend = web5; } > { .backend = web6; } > > > } > > director cache round-robin { > { .backend = varnish1; } > { .backend = varnish2; } > } > > sub vcl_recv { > set req.backend = www; > if (req.http.User-Agent ~ "typhoeus") { > error 403 "Temporarily disabled"; > } > set req.grace = 6h; > if (! req.backend.healthy) { > set req.grace = 24h; > } > set req.http.X-Forwarded-For = req.http.X-Forwarded-For ", " > client.ip; > /*if (req.http.host ~ "^origin\.test\.(.+\.|)mywebsite.com > $") { > return(pass); > }*/ > if (req.http.host ~ "^origin\.(.+\.|)mywebsite\.com$") { > return(pass); > } > if (req.http.host ~ "test.cms.mywebsite.com > |test.cms-2.mywebsite.com > |test.facebook.mywebsite.com > ") { > return (pass); > } > if (req.http.host ~ "test.cache-only.mywebsite.com > ") { > set req.backend = cache; > } > /* if (req.http.host ~ "test\.*\.mywebsite.com > ") { > if (req.http.host !~ "test\.(.+\.|)m\.mywebsite\.com$") { > set req.http.host = "test.mywebsite.com > "; > } > } else { > return (pass); > }*/ > > if (req.http.host ~ ".*\.mywebsite.com ") { > if (req.http.host !~ "(.+\.|)m\.mywebsite\.com$") { > set req.http.host = "www.mywebsite.com > "; > } > } else { > return (pass); > } > > /* > if (req.url ~ "^/node/75402/tweettracker_brand/config/xml") { > error 403 "Temporarily disabled"; > } > */ > if (req.request == "PURGE") { > if (!client.ip ~ purge) { > error 405 "Not allowed."; > } > return (lookup); > } > if (req.request != "GET" && > req.request != "HEAD" && > req.request != "PUT" && > req.request != "POST" && > req.request != "TRACE" && > req.request != "OPTIONS" && > req.request != "DELETE") { > return (pipe); > } > if (req.request != "GET" && req.request != "HEAD") { > return (pass); > } > if (req.url ~ "eyeblaster/addineyeV2.html\?.*") { > set req.url = "/eyeblaster/addineyeV2.html"; > } > if (req.url ~ "sites/all/modules/custom/bravo_ad/ads.html\?.*") { > set req.url = "/sites/all/modules/custom/bravo_ad/ads.html"; > } > if (req.url ~ > "sites/m.mywebsite.com/modules/custom/bravo_ad/ads.html\?.* > ") { > set req.url = > "/sites/m.mywebsite.com/modules/custom/bravo_ad/ads.html > "; > } > if (req.url ~ > "ahah_helper\.php|bravo_points\.php|install\.php|update\.php|cron\.php|/json(:?\?.*)?$") > { > return (pass); > } > if (req.http.Authorization) { > return (pass); > } > if (req.url ~ "login" || req.url ~ "logout") { > return (pass); > } > if (req.url ~ "^/admin/" || req.url ~ "^/node/add/") { > return (pass); > } > if (req.http.Cache-Control ~ "no-cache") { > //return (pass); > } > if (req.http.Cookie ~ > "(VARNISH|DRUPAL_UID|LOGGED_IN|SESS|_twitter_sess)") { > set req.http.Cookie = regsuball(req.http.Cookie, > "(^|;\s*)(__[a-z]+|has_js)=[^;]*", ""); > set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); > } else { > unset req.http.Cookie; > } > if (req.restarts == 0) { > if (client.ip == "10.10.1.96" || client.ip == "10.10.1.97") { > set req.backend = www; > } elsif (server.ip == "10.10.1.96") { > set req.backend = varnish2; > } else { > set req.backend = varnish1; > } > } elsif (req.restarts >= 2) { > return (pass); > } > if (req.restarts >= 2) { > return (pass); > } > if (req.url ~ > "\.(ico|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|ICO|JPG|JPEG|PNG|GIF|GZ|TGZ|BZ2|TBZ|MP3|OOG|SWF)") > { > unset req.http.Accept-Encoding; > } > if (req.url ~ > "^/(sites/all/modules/mywebsite_admanager/includes/ads.php|doubleclick/DARTIframe.html)(\?.*|)$") > { > set req.url = regsub(req.url, "\?.*$", ""); > } > /* keep user-agents > if (! req.http.User-Agent ~ "MSIE") { > set req.http.User-Agent = "Mozilla"; > } > */ > if (req.http.Accept-Encoding ~ "gzip") { > set req.http.Accept-Encoding = "gzip"; > } elsif (req.http.Accept-Encoding ~ "deflate") { > set req.http.Accept-Encoding = "deflate"; > } else { > unset req.http.Accept-Encoding; > } > return (lookup); > } > > sub vcl_pipe { > set bereq.http.connection = "close"; > return (pipe); > } > > sub vcl_pass { > return (pass); > } > > sub vcl_hash { > set req.hash += req.url; > set req.hash += req.http.host; > if (req.http.Cookie ~ "VARNISH|DRUPAL_UID|LOGGED_IN") { > set req.hash += req.http.Cookie; > } > return (hash); > } > > sub vcl_hit { > if (req.request == "PURGE") { > set obj.ttl = 0s; > error 200 "Purged."; > } > /* > if (!obj.cacheable) { > return (pass); > } > return (deliver); > */ > } > > sub vcl_fetch { > if (beresp.status == 500) { > set req.http.X-Varnish-Error = "1"; > restart; > } > set beresp.grace = 6h; > # Set a short circuit cache lifetime for resp codes above 302 > if (beresp.status > 302) { > set beresp.ttl = 60s; > set beresp.http.Cache-Control = "max-age = 60"; > } > if (beresp.http.Edge-control ~ "no-store") { > set beresp.http.storage = "1"; > set beresp.cacheable = false; > return (pass); > } > if (beresp.status >= 300 || !beresp.cacheable) { > set beresp.http.Varnish-X-Cacheable = "Not Cacheable"; > set beresp.http.storage = "1"; > return (pass); > } > if (beresp.http.Set-Cookie) { > return (pass); > } > if (beresp.cacheable) { > unset beresp.http.expires; > set beresp.ttl = 600s; > set beresp.http.Cache-Control = "max-age = 600"; > if (req.url ~ > "\.(ico|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|ICO|JPG|JPEG|PNG|GIF|GZ|TGZ|BZ2|TBZ|MP3|OOG|SWF)") > { > set beresp.ttl = 43829m; > set beresp.http.Cache-Control = "max-age = 1000000"; > } > /* > if (req.url ~ "json(:?\??.*)?$") { > set beresp.ttl = 30s; > set beresp.http.Cache-Control = "max-age = 30"; > } > */ > } > return (deliver); > } > > sub vcl_deliver { > if (obj.hits > 0) { > set resp.http.Varnish-X-Cache = "HIT"; > set resp.http.Varnish-X-Cache-Hits = obj.hits; > } else { > set resp.http.Varnish-X-Cache = "MISS"; > } > return (deliver); > } > > sub vcl_error { > if (req.restarts == 0) { > return (restart); > } > if (req.http.X-Varnish-Error != "1") { > set req.http.X-Varnish-Error = "1"; > return (restart); > } > set obj.http.Content-Type = "text/html; charset=utf-8"; > synthetic {" > > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd" > > > > > mywebsite.com <http://mywebsite.com> "} obj.status " " > obj.response {" > > > >
> src="http://www.mywebsite.com/media/site-unavailable/site-unavailable.jpg" > width="1000" height="700" > > >
> > > "}; > return (deliver); > } > > > Thanks > Tim > -- > GPG me!! > > gpg --keyserver pool.sks-keyservers.net > --recv-keys F186197B > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Boris Gu?ry twitter: @borisguery skype: borisguery mobile: +33 6 86 83 03 12 pgp: 0x034C6265 irc: bgy @ freenode -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry.magnien at sfr.com Tue Jun 3 08:39:20 2014 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Tue, 3 Jun 2014 08:39:20 +0000 Subject: varnish is sticking too much to web nodes In-Reply-To: <538D8296.7080705@gmail.com> References: <538D8296.7080705@gmail.com> Message-ID: <5D103CE839D50E4CBC62C9FD7B83287CC9BC4C5A@EXCN013.encara.local.ads> Hi, Maybe a first step would be to upgrade to a more recent version. From what I see in your vcl, you're still using a 2.x varnish, which is much deprecated. Regards, Thierry De?: varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Boris Gu?ry Envoy??: mardi 3 juin 2014 10:09 ??: varnish-misc at varnish-cache.org Objet?: Re: varnish is sticking too much to web nodes Maybe you can start to check the request counters of Apache using its status module. You should be able to see how much requests your webserver is handling at a given time. Check http://httpd.apache.org/docs/2.2/mod/mod_status.html It should also be easy to create a script which check every n seconds the apache status of each servers and aggregate data for further analysis. Le 02/06/14 18:08, Tim Dunphy a ?crit?: We have two out of six web nodes showing extremely high CPU usage compared to the others in the farm.? We're using the round-robin load balancing algorithm. Using a tool like 'htop' shows us that all the hungry processes belong to apache. Yet diffing the configuration files between the nodes with low cpu usage and the ones with high cpu usage shows no difference at all. The theory is that varnish is throwing traffic at these two hosts too often. So I was just wondering if there could be any way to diagnose this from a Varnish perspective. What clues could tools like varnishtop and varnishstat or varnishlog yield in investigating this type of problem? Here's my varnish vcl (cleaned of any too-identifying info) in case anyone can offer an opinion as to why this is going on: backend web1 { ? ? .host = "10.10.1.98"; ? ? .port = "80"; ? ? .connect_timeout = 90s; ? ? .first_byte_timeout = 90s; ? ? .between_bytes_timeout = 90s; ? ? .max_connections = 70; ? ? .probe = { ? ? ? ? .url = "/healthcheck.php"; ? ? ? ? .timeout = 5s; ? ? ? ? .interval = 30s; ? ? ? ? .window = 10; ? ? ? ? .threshold = 1; ? ? } } backend web2 { ? ? .host = "10.10.1.99"; ? ? .port = "80"; ? ? .connect_timeout = 90s; ? ? .first_byte_timeout = 90s; ? ? .between_bytes_timeout = 90s; ? ? .max_connections = 70; ? ? .probe = { ? ? ? ? .url = "/healthcheck.php"; ? ? ? ? .timeout = 5s; ? ? ? ? .interval = 30s; ? ? ? ? .window = 10; ? ? ? ? .threshold = 1; ? ? } } backend web3 { ? ? .host = "10.10.1.100"; ? ? .port = "80"; ? ? .connect_timeout = 90s; ? ? .first_byte_timeout = 90s; ? ? .between_bytes_timeout = 90s; ? ? .max_connections = 70; ? ? .probe = { ? ? ? ? .url = "/healthcheck.php"; ? ? ? ? .timeout = 5s; ? ? ? ? .interval = 30s; ? ? ? ? .window = 10; ? ? ? ? .threshold = 1; ? ? } } backend web4 { ? ? .host = "10.10.1.101"; ? ? .port = "80"; ? ? .connect_timeout = 90s; ? ? .first_byte_timeout = 90s; ? ? .between_bytes_timeout = 90s; ? ? .max_connections = 70; ? ? .probe = { ? ? ? ? .url = "/healthcheck.php"; ? ? ? ? .timeout = 5s; ? ? ? ? .interval = 30s; ? ? ? ? .window = 10; ? ? ? ? .threshold = 1; ? ? } } backend web5 { ? ? .host = "10.10.1.235"; ? ? .port = "80"; ? ? .connect_timeout = 90s; ? ? .first_byte_timeout = 90s; ? ? .between_bytes_timeout = 90s; ? ? .max_connections = 70; ? ? .probe = { ? ? ? ? .url = "/healthcheck.php"; ? ? ? ? .timeout = 5s; ? ? ? ? .interval = 30s; ? ? ? ? .window = 10; ? ? ? ? .threshold = 1; ? ? } } backend web6 { ? ? .host = "10.10.1.236"; ? ? .port = "80"; ? ? .connect_timeout = 90s; ? ? .first_byte_timeout = 90s; ? ? .between_bytes_timeout = 90s; ? ? .max_connections = 70; ? ? .probe = { ? ? ? ? .url = "/healthcheck.php"; ? ? ? ? .timeout = 5s; ? ? ? ? .interval = 30s; ? ? ? ? .window = 10; ? ? ? ? .threshold = 1; ? ? } } backend varnish1 { ? ? .host = "10.10.1.96"; ? ? .port = "80"; ? ? .connect_timeout = 90s; ? ? .first_byte_timeout = 90s; ? ? .between_bytes_timeout = 90s; ? ? .max_connections = 1000; } ? backend varnish2 { ? ? .host = "10.10.1.97"; ? ? .port = "80"; ? ? .connect_timeout = 90s; ? ? .first_byte_timeout = 90s; ? ? .between_bytes_timeout = 90s; ? ? .max_connections = 1000; } acl purge { ? ? "localhost"; ? ? "127.0.0.1"; ? ? "10.10.1.96"; ? ? "10.10.1.97"; } ? director www round-robin { ? ? { .backend = web1; } ? ? { .backend = web2; } ? ? { .backend = web3; } ? ? { .backend = web4; } ? ? { .backend = web5; } ? ? { .backend = web6; } } director cache round-robin { ? ? { .backend = varnish1; } ? ? { .backend = varnish2; } } ? sub vcl_recv { ? ? set req.backend = www; ? ? if (req.http.User-Agent ~ "typhoeus") { ? ? ? ? error 403 "Temporarily disabled"; ? ? } ? ? set req.grace = 6h; ? ? if (! req.backend.healthy) { ? ? ? ? set req.grace = 24h; ? ? } ? ? set req.http.X-Forwarded-For = req.http.X-Forwarded-For ", " client.ip; ? ? /*if (req.http.host ~ "^origin\.test\.(.+\.|)mywebsite.com$") { ? ? ? return(pass); ? ? }*/? ? ? if (req.http.host ~ "^origin\.(.+\.|)mywebsite\.com$") { ? ? ? return(pass); ? ? } ? ? if (req.http.host ~ "test.cms.mywebsite.com|test.cms-2.mywebsite.com|test.facebook.mywebsite.com") { ? ? ? ? return (pass); ? ? } ? ? if (req.http.host ~ "test.cache-only.mywebsite.com") { ? ? ? ? set req.backend = cache; ? ? } ? ? /* if (req.http.host ~ "test\.*\.mywebsite.com") { ? ? ? if (req.http.host !~ "test\.(.+\.|)m\.mywebsite\.com$") { ? ? ? ? set req.http.host = "test.mywebsite.com"; ? ? ? } ? ? } else { ? ? ? ? return (pass); ? ? }*/ ? ? ?if (req.http.host ~ ".*\.mywebsite.com") { ? ? ? if (req.http.host !~ "(.+\.|)m\.mywebsite\.com$") { ? ? ? ? set req.http.host = "www.mywebsite.com"; ? ? ? } ? ? } else { ? ? ? ? return (pass); ? ? } ? ? /* ? ? if (req.url ~ "^/node/75402/tweettracker_brand/config/xml") { ? ? ? ? error 403 "Temporarily disabled"; ? ? } */ ? ? if (req.request == "PURGE") { ? ? ? ? if (!client.ip ~ purge) { ? ? ? ? ? ? error 405 "Not allowed."; ? ? ? ? } ? ? ? ? return (lookup); ? ? } ? ? if (req.request != "GET" && ? ? ? ? req.request != "HEAD" && ? ? ? ? req.request != "PUT" && ? ? ? ? req.request != "POST" && ? ? ? ? req.request != "TRACE" && ? ? ? ? req.request != "OPTIONS" && ? ? ? ? req.request != "DELETE") { ? ? ? ? ? ? return (pipe); ? ? } ? ? if (req.request != "GET" && req.request != "HEAD") { ? ? ? ? return (pass); ? ? } ? ? if (req.url ~ "eyeblaster/addineyeV2.html\?.*") { ? ? ? ? set req.url = "/eyeblaster/addineyeV2.html"; ? ? } ? ? if (req.url ~ "sites/all/modules/custom/bravo_ad/ads.html\?.*") { ? ? ? ? set req.url = "/sites/all/modules/custom/bravo_ad/ads.html"; ? ? } ? ? if (req.url ~ "sites/m.mywebsite.com/modules/custom/bravo_ad/ads.html\?.*") { ? ? ? ? set req.url = "/sites/m.mywebsite.com/modules/custom/bravo_ad/ads.html"; ? ? } ? ? if (req.url ~ "ahah_helper\.php|bravo_points\.php|install\.php|update\.php|cron\.php|/json(:?\?.*)?$") { ? ? ? ? return (pass); ? ? } ? ? if (req.http.Authorization) { ? ? ? ? return (pass); ? ? } ? ? if (req.url ~ "login" || req.url ~ "logout") { ? ? ? ? return (pass); ? ? } ? ? if (req.url ~ "^/admin/" || req.url ~ "^/node/add/") { ? ? ? ? return (pass); ? ? } ? ? if (req.http.Cache-Control ~ "no-cache") { ? ? ? ? //return (pass); ? ? } ? ? if (req.http.Cookie ~ "(VARNISH|DRUPAL_UID|LOGGED_IN|SESS|_twitter_sess)") { ? ? ? ? set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", ""); ? ? ? ? set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); ? ? } else { ? ? ? ? unset req.http.Cookie; ? ? } ? ? if (req.restarts == 0) { ? ? ? ? if (client.ip == "10.10.1.96" || client.ip == "10.10.1.97") { ? ? ? ? ? ? set req.backend = www; ? ? ? ? } elsif (server.ip == "10.10.1.96") { ? ? ? ? ? ? set req.backend = varnish2; ? ? ? ? } else { ? ? ? ? ? ? set req.backend = varnish1; ? ? ? ? } ? ? } elsif (req.restarts >= 2) { ? ? ? ? return (pass); ? ? } ? ? if (req.restarts >= 2) { ? ? ? ? return (pass); ? ? } ? ? if (req.url ~ "\.(ico|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|ICO|JPG|JPEG|PNG|GIF|GZ|TGZ|BZ2|TBZ|MP3|OOG|SWF)") { ? ? ? ? unset req.http.Accept-Encoding; ? ? } ? ? if (req.url ~ "^/(sites/all/modules/mywebsite_admanager/includes/ads.php|doubleclick/DARTIframe.html)(\?.*|)$") { ? ? ? ? set req.url = regsub(req.url, "\?.*$", ""); ? ? } /* keep user-agents ? ? if (! req.http.User-Agent ~ "MSIE") { ? ? ? ? set req.http.User-Agent = "Mozilla"; ? ? } */ ? ? if (req.http.Accept-Encoding ~ "gzip") { ? ? ? ? set req.http.Accept-Encoding = "gzip"; ? ? } elsif (req.http.Accept-Encoding ~ "deflate") { ? ? ? ? set req.http.Accept-Encoding = "deflate"; ? ? } else { ? ? ? ? unset req.http.Accept-Encoding; ? ? } ? ? return (lookup); } ? sub vcl_pipe { ? ? set bereq.http.connection = "close"; ? ? return (pipe); } ? sub vcl_pass { ? ? return (pass); } ? sub vcl_hash { ? ? set req.hash += req.url; ? ? set req.hash += req.http.host; ? ? if (req.http.Cookie ~ "VARNISH|DRUPAL_UID|LOGGED_IN") { ? ? ? ? set req.hash += req.http.Cookie; ? ? } ? ? return (hash); } sub vcl_hit { ? ? if (req.request == "PURGE") { ? ? ? ? set obj.ttl = 0s; ? ? ? ? error 200 "Purged."; ? ? } ? ? /* ? ? if (!obj.cacheable) { ? ? ? ? return (pass); ? ? } ? ? return (deliver); ? ? */ } ? sub vcl_fetch { ? ? if (beresp.status == 500) { ? ? ? ? set req.http.X-Varnish-Error = "1"; ? ? ? ? restart; ? ? } ? ? set beresp.grace = 6h; ? ? # Set a short circuit cache lifetime for resp codes above 302 ? ? if (beresp.status > 302) { set beresp.ttl = 60s; set beresp.http.Cache-Control = "max-age = 60"; ? ? } ? ? if (beresp.http.Edge-control ~ "no-store") { ? ? ? ? set beresp.http.storage = "1"; ? ? ? ? set beresp.cacheable = false; ? ? ? ? return (pass); ? ? } ? ? if (beresp.status >= 300 || !beresp.cacheable) { ? ? ? ? set beresp.http.Varnish-X-Cacheable = "Not Cacheable"; ? ? ? ? set beresp.http.storage = "1"; ? ? ? ? return (pass); ? ? } ? ? if (beresp.http.Set-Cookie) { ? ? ? ? return (pass); ? ? } ? ? if (beresp.cacheable) { ? ? ? ? unset beresp.http.expires; ? ? ? ? set beresp.ttl = 600s; ? ? ? ? set beresp.http.Cache-Control = "max-age = 600"; ? ? ? ? if (req.url ~ "\.(ico|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|ICO|JPG|JPEG|PNG|GIF|GZ|TGZ|BZ2|TBZ|MP3|OOG|SWF)") { ? ? ? ? ? ? set beresp.ttl = 43829m; ? ? ? ? ? ? set beresp.http.Cache-Control = "max-age = 1000000"; ? ? ? ? } ? ? ? ? /* ? ? ? ? if (req.url ~ "json(:?\??.*)?$") { ? ? ? ? ? ? set beresp.ttl = 30s; ? ? ? ? ? ? set beresp.http.Cache-Control = "max-age = 30"; ? ? ? ? } ? ? ? ? */ ? ? } ? ? return (deliver); } ? sub vcl_deliver { ? ? if (obj.hits > 0) { ? ? ? ? set resp.http.Varnish-X-Cache = "HIT"; ? ? ? ? set resp.http.Varnish-X-Cache-Hits = obj.hits; ? ? } else { ? ? ? ? set resp.http.Varnish-X-Cache = "MISS"; ? ? } ? ? return (deliver); } ? sub vcl_error { ? ? if (req.restarts == 0) { ? ? ? ? return (restart); ? ? } ? ? if (req.http.X-Varnish-Error != "1") { ? ? ? ? set req.http.X-Varnish-Error = "1"; ? ? ? ? return (restart); ? ? } ? ? set obj.http.Content-Type = "text/html; charset=utf-8"; ? ? synthetic {" ? ? ? ? > ? ? ? ? ? ? mywebsite.com "} obj.status " " obj.response {" ? ? ? ? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
? ? ? ? "}; ? ? return (deliver); } Thanks Tim -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Boris Gu?ry twitter: @borisguery skype: borisguery mobile: +33 6 86 83 03 12 pgp: 0x034C6265 irc: bgy @ freenode From lkarsten at varnish-software.com Tue Jun 3 12:16:35 2014 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Tue, 3 Jun 2014 14:16:35 +0200 Subject: Varnish 4 - VCL Compilation failed - Valid VCL In-Reply-To: References: Message-ID: <20140603121634.GD13788@immer.varnish-software.com> On Tue, May 20, 2014 at 09:57:49PM -0700, Hugues Alary wrote: > After looking into this more, it seems to be just a small permission > problem. Hi. This sounds like something that should be looked into. Can you please file a bug on it in the bugtracker: https://www.varnish-cache.org/trac/newticket Thanks. -- Lasse Karstensen From lkarsten at varnish-software.com Tue Jun 3 12:45:20 2014 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Tue, 3 Jun 2014 14:45:20 +0200 Subject: host is lagging In-Reply-To: References: Message-ID: <20140603124519.GE13788@immer.varnish-software.com> On Fri, May 30, 2014 at 03:53:04PM -0400, Tim Dunphy wrote: > A couple of the web servers out of a farm of 6 machines we're load > balancing with varnish is showing high cpu use. We're balancing them using > the round robin LB scheme. I was just wondering if there are any varnish > based tools (varnishstat, varnishlog, varnishtop or something else) that > might be able to tell me which hosts are underperforming. Given that you are on 4.0, you can use a vsl-query to varnishlog to pick out backend requests that are taking a long time to complete. # varnishlog -q 'Timestamp:Beresp[2] > 0.3' (or if you're looking for client-side response times) # varnishlog -q 'Timestamp:Resp[2] > 0.3' Unit is in seconds, so "slow" means 300ms in this example. The Backend log lines will tell you which backend was used. If you are on 3.0, you'll have to parse the response time from ReqEnd yourself. Format is documented in the wiki. > Also I was wondering if maybe there might be a better scheme to try (like > least-conn?) that might be better for alleviating high CPU usage or high > load? You can adjust the weights if they're not equally powerful, that is about it I think. There have been talk about a least-connections backend director for 4.0, but I don't think anyone has completed/published one yet. The best solution is of course to lower the load on your backend servers. Fix the application, disable backend-side gzip, make sure connections are reused, limit concurrent backend connections or increase TTL if possible. Unless you have a very odd webapp, you are IO-bound and not CPU-bound. Buying more memory chips is usually a cheap and simple short term fix for this. :-) -- Lasse Karstensen From lkarsten at varnish-software.com Tue Jun 3 12:54:00 2014 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Tue, 3 Jun 2014 14:54:00 +0200 Subject: Varnish log where obj are stored In-Reply-To: <20140528140903.2b2b6a31@lobo.lobo.dom> References: <20140528140903.2b2b6a31@lobo.lobo.dom> Message-ID: <20140603125400.GF13788@immer.varnish-software.com> On Wed, May 28, 2014 at 02:09:03PM +0200, marco at nucleus.it wrote: > Hi, > using more than one storage like -s othersram=malloc,500M -s > testram=malloc,50M how can i check from the varnishlog or other tools > where the object is storage in cache ? > I know that i can force the storage in vcl_fetch with set > beresp.storage = "name_og_the_storage"; but how can i verify this ? In 4.0, the Storage log entry in varnishlog for backend requests give you this: [..] - TTL VCL 120 60 86400 1401799601 - VCL_return deliver - Storage malloc s0 >From vsl(7): Storage - Where object is stored Type and name of the storage backend the object is stored in. The format is: %s %s | | | +- Name of storage backend +---- Type ("malloc", "file", "persistent" etc.) -- Lasse Karstensen From varnish at tengu.ch Thu Jun 5 05:41:25 2014 From: varnish at tengu.ch (=?UTF-8?B?Q8OpZHJpYyBKZWFubmVyZXQ=?=) Date: Thu, 05 Jun 2014 07:41:25 +0200 Subject: Catch "req.restarts" in logs Message-ID: <53900305.9050200@tengu.ch> Hello, I'm wondering if there's a way to catch the req.restarts in the logs. We currently have a bloc telling Varnish the following: sub vcl_error { [?] if (obj.status >= 500 && obj.status <= 504 && req.restarts < 1) { return(restart); } [?] } Is there a way to send the req.restarts in logs? I guess this would be done in the vcl_recv routine though, maybe through std.log()? How to catch it in varnishncsa? We're currently running varnish 3.0.5 Thanks! Cheers, C. From dridi.boukelmoune at zenika.com Thu Jun 5 06:02:53 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Thu, 5 Jun 2014 08:02:53 +0200 Subject: Catch "req.restarts" in logs In-Reply-To: <53900305.9050200@tengu.ch> References: <53900305.9050200@tengu.ch> Message-ID: Hi C?dric, If you want this information in varnishncsa, use std.log and catch it with an extended variable in your format pattern (it's described in the man). Cheers, Dridi On Thu, Jun 5, 2014 at 7:41 AM, C?dric Jeanneret wrote: > Hello, > > I'm wondering if there's a way to catch the req.restarts in the logs. We > currently have a bloc telling Varnish the following: > > sub vcl_error { > [?] > if (obj.status >= 500 && obj.status <= 504 && req.restarts < 1) { > return(restart); > } > [?] > } > > Is there a way to send the req.restarts in logs? I guess this would be > done in the vcl_recv routine though, maybe through std.log()? How to > catch it in varnishncsa? > > We're currently running varnish 3.0.5 > > Thanks! > > Cheers, > > C. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From diraol at diraol.eng.br Mon Jun 9 23:24:30 2014 From: diraol at diraol.eng.br (Diego Rabatone) Date: Mon, 9 Jun 2014 20:24:30 -0300 Subject: Cross-Domain Caching Message-ID: Hi friends, I am new at the caching World, and I have looked for some solution to cross-domain caching, but I have not found any solution. I have a blog that embed some dataviz webservices, and I want to cache this content that I insert using Iframes. The blog is: http://blog.estadaodados.com there is any way of doing this? At this moment I have a Varnish + nginx + php-fpm setup on a dev server (not the link that I posted), and it seems to be working ok, but there is no caching of the Iframes. Thanks. -------------------------------- Diego Rabatone Oliveira diraol(arroba)diraol(ponto)eng(ponto)br Identica: (@diraol) http://identi.ca/diraol Twitter: @diraol -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at ifixit.com Tue Jun 10 00:08:03 2014 From: james at ifixit.com (James Pearson) Date: Mon, 09 Jun 2014 17:08:03 -0700 Subject: Cross-Domain Caching In-Reply-To: References: Message-ID: <1402358526-sup-7879@geror.local> Excerpts from Diego Rabatone's message of 2014-06-09 16:24:30 -0700: > I have a blog that embed some dataviz webservices, and I want to cache this > content that I insert using Iframes. > The blog is: http://blog.estadaodados.com there is any way of doing this? In general, caching is done by the content provider. So, if you administer the webservices you're embedding, you should cache them appropriately there. If you don't own the content you're embedding, you should first consider whether you're allowed to embed it. :) After that, you have two basic options: convince the site administrator to implement appropriate caching, or do it yourself in a mirror. The way a caching mirror might work is that, rather than using an iframe, your backend code fetches the service via curl, caches it locally (on disk, in memcache, etc.), then inserts the fetched data directly into the html you serve your visitors. This gives you control over the content, but be aware that it may be against the content owner's terms of service. - P -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From hernan at cmsmedios.com Tue Jun 10 01:07:28 2014 From: hernan at cmsmedios.com (=?UTF-8?Q?Hern=C3=A1n_Marsili?=) Date: Mon, 9 Jun 2014 22:07:28 -0300 Subject: varnish mp4 streaming Message-ID: Hi, We are working on a VIDEO SITE for a sports sites. Currently working with Varnish 3.0.5. The average video size is 100mb. We have a 4gb malloc storage. Right now, we are handling mp4 as just a regular file. No problems so far. A couple of questions: 1) STREAM. We tried this on the vcl but we don't see much improvement. Should we use it anyway? if (req.url ~ "\.mp4") { set beresp.do_*stream* = true; set beresp.http.X-Cacheable-TTL = "stream"; } 1) VARNISH STREAM BRANCH. We find a all branch of varnish for streaming. https://github.com/mbgrydeland/varnish-cache-streaming. Does anyway knows if this is now part of Varnish 3.0.5 or Varnish 4? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Tue Jun 10 04:24:13 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Tue, 10 Jun 2014 06:24:13 +0200 Subject: varnish mp4 streaming In-Reply-To: References: Message-ID: Hi, On Tue, Jun 10, 2014 at 3:07 AM, Hern?n Marsili wrote: > Hi, > > We are working on a VIDEO SITE for a sports sites. Currently working with > Varnish 3.0.5. The average video size is 100mb. We have a 4gb malloc > storage. > > Right now, we are handling mp4 as just a regular file. No problems so far. A > couple of questions: > > 1) STREAM. We tried this on the vcl but we don't see much improvement. > Should we use it anyway? > > if (req.url ~ "\.mp4") { > > set beresp.do_stream = true; > > set beresp.http.X-Cacheable-TTL = "stream"; > > } IIRC do_stream is useful for the client that triggers a refresh in v3. > 1) VARNISH STREAM BRANCH. We find a all branch of varnish for streaming. > https://github.com/mbgrydeland/varnish-cache-streaming. Does anyway knows if > this is now part of Varnish 3.0.5 or Varnish 4? Varnish 4 has a full streaming support! Dridi From perbu at varnish-software.com Tue Jun 10 06:55:42 2014 From: perbu at varnish-software.com (Per Buer) Date: Tue, 10 Jun 2014 08:55:42 +0200 Subject: Cross-Domain Caching In-Reply-To: References: Message-ID: Hi Diego, On Tue, Jun 10, 2014 at 1:24 AM, Diego Rabatone wrote: > Hi friends, > > I am new at the caching World, and I have looked for some solution to > cross-domain caching, but I have not found any solution. > > I have a blog that embed some dataviz webservices, and I want to cache > this content that I insert using Iframes. > The blog is: http://blog.estadaodados.com there is any way of doing this? > Absolutely. It is a pretty good way to make sure that the content you embed performs well in addition to lessening the load on the server you are embedding content from. > At this moment I have a Varnish + nginx + php-fpm setup on a dev server > (not the link that I posted), and it seems to be working ok, but there is > no caching of the Iframes. > No. This is expected. In order to cache those iframes you would need to set up the server you embedding content from as a backend on your varnish and preferably give it a new hostname. Then, in the vcl_recv you replace req.http.host to the original hostname and set the backend. Then regular caching rules will be applied. In Varnish 4.0 you could set the cache time to one hour and you could add a keep of maybe a couple of hours. That will enable IMS so you wouldn't have to refetch the content, only revalidate it. As James said, should make sure you have the authors permission, but if you do it through Varnish at least there won't be much of a impact on their server when you do this. Cheers, Per. -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at smartjog.com Tue Jun 10 09:28:06 2014 From: guillaume.quintard at smartjog.com (Guillaume Quintard) Date: Tue, 10 Jun 2014 11:28:06 +0200 Subject: varnish mp4 streaming In-Reply-To: References: Message-ID: <5396CFA6.2080106@smartjog.com> On 06/10/2014 03:07 AM, Hern?n Marsili wrote: > Hi, > > We are working on a VIDEO SITE for a sports sites. Currently working > with Varnish 3.0.5. The average video size is 100mb. We have a 4gb > malloc storage. > > Right now, we are handling mp4 as just a regular file. No problems so > far. A couple of questions: > > 1) STREAM. We tried this on the vcl but we don't see much improvement. > Should we use it anyway? > > if (req.url ~ "\.mp4") { > > set beresp.do_*stream* = true; > > set beresp.http.X-Cacheable-TTL = "stream"; > > } > > > 1) VARNISH STREAM BRANCH. We find a all branch of varnish for > streaming. https://github.com/mbgrydeland/varnish-cache-streaming. > Does anyway knows if this is now part of Varnish 3.0.5 or Varnish 4? > > You may have two problems: - the fact that your file is being downloaded, meaning that only the client who triggered the backend request will profit from the streaming. This is solved in v4, and data will be send as soon as available, for all threads. - the video metadata is at the end of the file, meaning that your clients must retrieve all the file before starting to read. You're probably in this case if the video takes a long time to read, even if cached. To mitigate the second problem, you may use qt-faststart (https://github.com/danielgtaylor/qtfaststart) to move the meta data at the beginning of the file, or switch to a chunked protocol. (hopefully, if per-request vmods calls come to fruition, or the vfp mature a bit more, we'll have a vmod to qt-faststartify files on the fly). Hope that helps. -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From hernan at cmsmedios.com Tue Jun 10 12:51:50 2014 From: hernan at cmsmedios.com (=?UTF-8?Q?Hern=C3=A1n_Marsili?=) Date: Tue, 10 Jun 2014 09:51:50 -0300 Subject: varnish mp4 streaming In-Reply-To: <5396CFA6.2080106@smartjog.com> References: <5396CFA6.2080106@smartjog.com> Message-ID: Hi, We are encoding our videos with ffmpeg using the FASTSTART movflag, so i think qtfaststart is no needed. We will start looking at v4 right away. Thank you all! Saludos, Hern?n. [image: logo tfs] http://www.cms-medios.com | http://blog.tfsla.com | facebook.com/cmsmedioscel +54 [911] 4945 2272 | skype hmarsili | Linkedin ar.linkedin.com/in/hmarsiliSuscribite a nuestras novedades por e-mail o RSS feed o Twitter @tfsla >> Argentina +54 11 4711-8999 | USA +1 305 722-5130 | M?xico +52 55 5350-1090 | Espa?a +34 93 179-0330 | El Salvador +503 21 13-9730 | Venezuela +58 212 335-1180 | Colombia +57 1 508-7840 On Tue, Jun 10, 2014 at 6:28 AM, Guillaume Quintard < guillaume.quintard at smartjog.com> wrote: > On 06/10/2014 03:07 AM, Hern?n Marsili wrote: > > Hi, > We are working on a VIDEO SITE for a sports sites. Currently > working with Varnish 3.0.5. The average video size is 100mb. We have a 4gb > malloc storage. > > Right now, we are handling mp4 as just a regular file. No problems so > far. A couple of questions: > > 1) STREAM. We tried this on the vcl but we don't see much improvement. > Should we use it anyway? > > if (req.url ~ "\.mp4") { > > set beresp.do_*stream* = true; > > set beresp.http.X-Cacheable-TTL = "stream"; > > } > > > 1) VARNISH STREAM BRANCH. We find a all branch of varnish for streaming. > https://github.com/mbgrydeland/varnish-cache-streaming. Does anyway knows > if this is now part of Varnish 3.0.5 or Varnish 4? > > You may have two problems: > - the fact that your file is being downloaded, meaning that only the > client who triggered the backend request will profit from the streaming. > This is solved in v4, and data will be send as soon as available, for all > threads. > - the video metadata is at the end of the file, meaning that your clients > must retrieve all the file before starting to read. You're probably in this > case if the video takes a long time to read, even if cached. > > To mitigate the second problem, you may use qt-faststart ( > https://github.com/danielgtaylor/qtfaststart) to move the meta data at > the beginning of the file, or switch to a chunked protocol. > > (hopefully, if per-request vmods calls come to fruition, or the vfp mature > a bit more, we'll have a vmod to qt-faststartify files on the fly). > > Hope that helps. > > -- > Guillaume Quintard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From diraol at diraol.eng.br Wed Jun 11 18:10:35 2014 From: diraol at diraol.eng.br (Diego Rabatone) Date: Wed, 11 Jun 2014 15:10:35 -0300 Subject: Cross-Domain Caching In-Reply-To: References: Message-ID: Hi James and Per, thank you very much for answering my question. First of all, I have the permission to embed the content. All embeds are from websites that are offer the service of building data visualizations and infographics, me and my team use them to build our products and they all have embeding options. I like the option to only revalidate the contet, instead of refetching it, but I didn't understand completly how could I do it Per. I will try and example, tell me if I'm making any conecptual mistake. My cached domain is: *estadaodados.com * , this is were I have full control and where my Varnish is. On *estadaodados.com/iframe * I want to embed, using an Iframe, a page from *example.com *, exactly this page: *example.com/123* So, instead of using ** I should set *example.com * as a *backend* on my varnish with: backend infograph { .host="example.com" } And set this vcl_recv rule: sub vlc_recv { if (req.http.host ~ "infographics.estadaodados.com") { set req.backend_hint = example.com } And then use the iframe with: ** That's it? Thanks for your attention, Best, -------------------------------- Diego Rabatone Oliveira diraol(arroba)diraol(ponto)eng(ponto)br Identica: (@diraol) http://identi.ca/diraol Twitter: @diraol 2014-06-10 3:55 GMT-03:00 Per Buer : > Hi Diego, > > > On Tue, Jun 10, 2014 at 1:24 AM, Diego Rabatone > wrote: > >> Hi friends, >> >> I am new at the caching World, and I have looked for some solution to >> cross-domain caching, but I have not found any solution. >> >> I have a blog that embed some dataviz webservices, and I want to cache >> this content that I insert using Iframes. >> The blog is: http://blog.estadaodados.com there is any way of doing this? >> > > Absolutely. It is a pretty good way to make sure that the content you > embed performs well in addition to lessening the load on the server you are > embedding content from. > > >> At this moment I have a Varnish + nginx + php-fpm setup on a dev server >> (not the link that I posted), and it seems to be working ok, but there is >> no caching of the Iframes. >> > > No. This is expected. In order to cache those iframes you would need to > set up the server you embedding content from as a backend on your varnish > and preferably give it a new hostname. Then, in the vcl_recv you replace > req.http.host to the original hostname and set the backend. Then regular > caching rules will be applied. In Varnish 4.0 you could set the cache time > to one hour and you could add a keep of maybe a couple of hours. That will > enable IMS so you wouldn't have to refetch the content, only revalidate it. > > > As James said, should make sure you have the authors permission, but if > you do it through Varnish at least there won't be much of a impact on their > server when you do this. > > Cheers, > > Per. > > -- > *Per Buer* > CTO | Varnish Software > Phone: +47 958 39 117 | Skype: per.buer > We Make Websites Fly! > > Winner of the Red Herring Top 100 Global Award 2013 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Wed Jun 11 19:26:08 2014 From: perbu at varnish-software.com (Per Buer) Date: Wed, 11 Jun 2014 21:26:08 +0200 Subject: Cross-Domain Caching In-Reply-To: References: Message-ID: Hi Diego. What you need to understand is how the server which you are embedding the content from works. It will not serve up the right content unless you give it the right Host: header ( https://www.varnish-software.com/blog/getting-virtual-hosts-right-varnish-cache explain how virtual hosts actually works). The thing you are missing is something like this: sub vlc_recv { if (req.http.host ~ "infographics.estadaodados.com") { set req.http.host = "example.com"; # could also override req.url here - but better to keep req.url the same (for simplicity). set req.backend_hint = infograph; } On Wed, Jun 11, 2014 at 8:10 PM, Diego Rabatone wrote: > Hi James and Per, > > thank you very much for answering my question. > > First of all, I have the permission to embed the content. All embeds are > from websites that are offer the service of building data visualizations > and infographics, me and my team use them to build our products and they > all have embeding options. > > I like the option to only revalidate the contet, instead of refetching it, > but I didn't understand completly how could I do it Per. > > I will try and example, tell me if I'm making any conecptual mistake. > > My cached domain is: *estadaodados.com * , this > is were I have full control and where my Varnish is. > > On *estadaodados.com/iframe * I want to > embed, using an Iframe, a page from *example.com *, > exactly this page: *example.com/123* > > So, instead of using ** > I should set *example.com * as a *backend* on my > varnish with: > > backend infograph { > .host="example.com" > } > > And set this vcl_recv rule: > > sub vlc_recv { > if (req.http.host ~ "infographics.estadaodados.com") { > set req.backend_hint = example.com > } > > And then use the iframe with: > > > * * > > That's it? > > Thanks for your attention, > > Best, > > > -------------------------------- > Diego Rabatone Oliveira > diraol(arroba)diraol(ponto)eng(ponto)br > Identica: (@diraol) http://identi.ca/diraol > Twitter: @diraol > > > 2014-06-10 3:55 GMT-03:00 Per Buer : > > Hi Diego, >> >> >> On Tue, Jun 10, 2014 at 1:24 AM, Diego Rabatone >> wrote: >> >>> Hi friends, >>> >>> I am new at the caching World, and I have looked for some solution to >>> cross-domain caching, but I have not found any solution. >>> >>> I have a blog that embed some dataviz webservices, and I want to cache >>> this content that I insert using Iframes. >>> The blog is: http://blog.estadaodados.com there is any way of doing >>> this? >>> >> >> Absolutely. It is a pretty good way to make sure that the content you >> embed performs well in addition to lessening the load on the server you are >> embedding content from. >> >> >>> At this moment I have a Varnish + nginx + php-fpm setup on a dev server >>> (not the link that I posted), and it seems to be working ok, but there is >>> no caching of the Iframes. >>> >> >> No. This is expected. In order to cache those iframes you would need to >> set up the server you embedding content from as a backend on your varnish >> and preferably give it a new hostname. Then, in the vcl_recv you replace >> req.http.host to the original hostname and set the backend. Then regular >> caching rules will be applied. In Varnish 4.0 you could set the cache time >> to one hour and you could add a keep of maybe a couple of hours. That will >> enable IMS so you wouldn't have to refetch the content, only revalidate it. >> >> >> As James said, should make sure you have the authors permission, but if >> you do it through Varnish at least there won't be much of a impact on their >> server when you do this. >> >> Cheers, >> >> Per. >> >> -- >> *Per Buer* >> CTO | Varnish Software >> Phone: +47 958 39 117 | Skype: per.buer >> We Make Websites Fly! >> >> Winner of the Red Herring Top 100 Global Award 2013 >> >> >> > -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From diraol at diraol.eng.br Wed Jun 11 20:07:47 2014 From: diraol at diraol.eng.br (Diego Rabatone) Date: Wed, 11 Jun 2014 17:07:47 -0300 Subject: Cross-Domain Caching In-Reply-To: References: Message-ID: Thanks Per, I have read the link you suggested and it made all more clear to me now. It looks that my varnish is now "working", but I don't know if it is caching or now, how to test if it is a cached version or if it is just a redirect? Thanks again. -------------------------------- Diego Rabatone Oliveira diraol(arroba)diraol(ponto)eng(ponto)br Identica: (@diraol) http://identi.ca/diraol Twitter: @diraol 2014-06-11 16:26 GMT-03:00 Per Buer : > Hi Diego. > > What you need to understand is how the server which you are embedding the > content from works. It will not serve up the right content unless you give > it the right Host: header ( > https://www.varnish-software.com/blog/getting-virtual-hosts-right-varnish-cache > explain how virtual hosts actually works). > > The thing you are missing is something like this: > > sub vlc_recv { > if (req.http.host ~ "infographics.estadaodados.com") { > set req.http.host = "example.com"; > # could also override req.url here - but better to keep req.url the > same (for simplicity). > set req.backend_hint = infograph; > } > > > > On Wed, Jun 11, 2014 at 8:10 PM, Diego Rabatone > wrote: > >> Hi James and Per, >> >> thank you very much for answering my question. >> >> First of all, I have the permission to embed the content. All embeds are >> from websites that are offer the service of building data visualizations >> and infographics, me and my team use them to build our products and they >> all have embeding options. >> >> I like the option to only revalidate the contet, instead of refetching >> it, but I didn't understand completly how could I do it Per. >> >> I will try and example, tell me if I'm making any conecptual mistake. >> >> My cached domain is: *estadaodados.com * , this >> is were I have full control and where my Varnish is. >> >> On *estadaodados.com/iframe * I want to >> embed, using an Iframe, a page from *example.com *, >> exactly this page: *example.com/123* >> >> So, instead of using ** >> I should set *example.com * as a *backend* on my >> varnish with: >> >> backend infograph { >> .host="example.com" >> } >> >> And set this vcl_recv rule: >> >> sub vlc_recv { >> if (req.http.host ~ "infographics.estadaodados.com") { >> set req.backend_hint = example.com >> } >> >> And then use the iframe with: >> >> >> * * >> >> That's it? >> >> Thanks for your attention, >> >> Best, >> >> >> -------------------------------- >> Diego Rabatone Oliveira >> diraol(arroba)diraol(ponto)eng(ponto)br >> Identica: (@diraol) http://identi.ca/diraol >> Twitter: @diraol >> >> >> 2014-06-10 3:55 GMT-03:00 Per Buer : >> >> Hi Diego, >>> >>> >>> On Tue, Jun 10, 2014 at 1:24 AM, Diego Rabatone >>> wrote: >>> >>>> Hi friends, >>>> >>>> I am new at the caching World, and I have looked for some solution to >>>> cross-domain caching, but I have not found any solution. >>>> >>>> I have a blog that embed some dataviz webservices, and I want to cache >>>> this content that I insert using Iframes. >>>> The blog is: http://blog.estadaodados.com there is any way of doing >>>> this? >>>> >>> >>> Absolutely. It is a pretty good way to make sure that the content you >>> embed performs well in addition to lessening the load on the server you are >>> embedding content from. >>> >>> >>>> At this moment I have a Varnish + nginx + php-fpm setup on a dev server >>>> (not the link that I posted), and it seems to be working ok, but there is >>>> no caching of the Iframes. >>>> >>> >>> No. This is expected. In order to cache those iframes you would need to >>> set up the server you embedding content from as a backend on your varnish >>> and preferably give it a new hostname. Then, in the vcl_recv you replace >>> req.http.host to the original hostname and set the backend. Then regular >>> caching rules will be applied. In Varnish 4.0 you could set the cache time >>> to one hour and you could add a keep of maybe a couple of hours. That will >>> enable IMS so you wouldn't have to refetch the content, only revalidate it. >>> >>> >>> As James said, should make sure you have the authors permission, but if >>> you do it through Varnish at least there won't be much of a impact on their >>> server when you do this. >>> >>> Cheers, >>> >>> Per. >>> >>> -- >>> *Per Buer* >>> CTO | Varnish Software >>> Phone: +47 958 39 117 | Skype: per.buer >>> We Make Websites Fly! >>> >>> Winner of the Red Herring Top 100 Global Award 2013 >>> >>> >>> >> > > > -- > *Per Buer* > CTO | Varnish Software > Phone: +47 958 39 117 | Skype: per.buer > We Make Websites Fly! > > Winner of the Red Herring Top 100 Global Award 2013 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Wed Jun 11 20:29:38 2014 From: perbu at varnish-software.com (Per Buer) Date: Wed, 11 Jun 2014 22:29:38 +0200 Subject: Cross-Domain Caching In-Reply-To: References: Message-ID: Look at the headers. If there are two numbers next on the X-Varnish header it's a cache hit. One number - cache miss. On Wed, Jun 11, 2014 at 10:07 PM, Diego Rabatone wrote: > Thanks Per, > > I have read the link you suggested and it made all more clear to me now. > > It looks that my varnish is now "working", but I don't know if it is > caching or now, how to test if it is a cached version or if it is just a > redirect? > > Thanks again. > > > -------------------------------- > Diego Rabatone Oliveira > diraol(arroba)diraol(ponto)eng(ponto)br > Identica: (@diraol) http://identi.ca/diraol > Twitter: @diraol > > > 2014-06-11 16:26 GMT-03:00 Per Buer : > > Hi Diego. >> >> What you need to understand is how the server which you are embedding the >> content from works. It will not serve up the right content unless you give >> it the right Host: header ( >> https://www.varnish-software.com/blog/getting-virtual-hosts-right-varnish-cache >> explain how virtual hosts actually works). >> >> The thing you are missing is something like this: >> >> sub vlc_recv { >> if (req.http.host ~ "infographics.estadaodados.com") { >> set req.http.host = "example.com"; >> # could also override req.url here - but better to keep req.url the >> same (for simplicity). >> set req.backend_hint = infograph; >> } >> >> >> >> On Wed, Jun 11, 2014 at 8:10 PM, Diego Rabatone >> wrote: >> >>> Hi James and Per, >>> >>> thank you very much for answering my question. >>> >>> First of all, I have the permission to embed the content. All embeds are >>> from websites that are offer the service of building data visualizations >>> and infographics, me and my team use them to build our products and they >>> all have embeding options. >>> >>> I like the option to only revalidate the contet, instead of refetching >>> it, but I didn't understand completly how could I do it Per. >>> >>> I will try and example, tell me if I'm making any conecptual mistake. >>> >>> My cached domain is: *estadaodados.com * , >>> this is were I have full control and where my Varnish is. >>> >>> On *estadaodados.com/iframe * I want to >>> embed, using an Iframe, a page from *example.com *, >>> exactly this page: *example.com/123* >>> >>> So, instead of using ** >>> I should set *example.com * as a *backend* on my >>> varnish with: >>> >>> backend infograph { >>> .host="example.com" >>> } >>> >>> And set this vcl_recv rule: >>> >>> sub vlc_recv { >>> if (req.http.host ~ "infographics.estadaodados.com") { >>> set req.backend_hint = example.com >>> } >>> >>> And then use the iframe with: >>> >>> >>> * * >>> >>> That's it? >>> >>> Thanks for your attention, >>> >>> Best, >>> >>> >>> -------------------------------- >>> Diego Rabatone Oliveira >>> diraol(arroba)diraol(ponto)eng(ponto)br >>> Identica: (@diraol) http://identi.ca/diraol >>> Twitter: @diraol >>> >>> >>> 2014-06-10 3:55 GMT-03:00 Per Buer : >>> >>> Hi Diego, >>>> >>>> >>>> On Tue, Jun 10, 2014 at 1:24 AM, Diego Rabatone >>>> wrote: >>>> >>>>> Hi friends, >>>>> >>>>> I am new at the caching World, and I have looked for some solution to >>>>> cross-domain caching, but I have not found any solution. >>>>> >>>>> I have a blog that embed some dataviz webservices, and I want to cache >>>>> this content that I insert using Iframes. >>>>> The blog is: http://blog.estadaodados.com there is any way of doing >>>>> this? >>>>> >>>> >>>> Absolutely. It is a pretty good way to make sure that the content you >>>> embed performs well in addition to lessening the load on the server you are >>>> embedding content from. >>>> >>>> >>>>> At this moment I have a Varnish + nginx + php-fpm setup on a dev >>>>> server (not the link that I posted), and it seems to be working ok, but >>>>> there is no caching of the Iframes. >>>>> >>>> >>>> No. This is expected. In order to cache those iframes you would need to >>>> set up the server you embedding content from as a backend on your varnish >>>> and preferably give it a new hostname. Then, in the vcl_recv you replace >>>> req.http.host to the original hostname and set the backend. Then regular >>>> caching rules will be applied. In Varnish 4.0 you could set the cache time >>>> to one hour and you could add a keep of maybe a couple of hours. That will >>>> enable IMS so you wouldn't have to refetch the content, only revalidate it. >>>> >>>> >>>> As James said, should make sure you have the authors permission, but if >>>> you do it through Varnish at least there won't be much of a impact on their >>>> server when you do this. >>>> >>>> Cheers, >>>> >>>> Per. >>>> >>>> -- >>>> *Per Buer* >>>> CTO | Varnish Software >>>> Phone: +47 958 39 117 | Skype: per.buer >>>> We Make Websites Fly! >>>> >>>> Winner of the Red Herring Top 100 Global Award 2013 >>>> >>>> >>>> >>> >> >> >> -- >> *Per Buer* >> CTO | Varnish Software >> Phone: +47 958 39 117 | Skype: per.buer >> We Make Websites Fly! >> >> Winner of the Red Herring Top 100 Global Award 2013 >> >> >> > -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From infos at opendoc.net Wed Jun 18 14:43:44 2014 From: infos at opendoc.net (Alexandre) Date: Wed, 18 Jun 2014 16:43:44 +0200 Subject: Question about varnishstat Message-ID: <53A1A5A0.9020106@opendoc.net> Hello, I use varnishstat to watch hosts state, and I wonder what is the number 18446744073709551615 ? Would you have any idea? example: --- varnishstat -1 . . . VBE.web01(X.X.X.X,,1234).vcls 1 . VCL references VBE.web01_MC(X.X.X.X,,1234).happy 18446744073709551615 . Happy health probes VBE.web02(X.X.X.X,,1234).vcls 1 . VCL references VBE.web02(X.X.X.X,,1234).happy 18446744073709551615 . Happy health probes VBE.web03(X.X.X.X,,1234).vcls 1 . VCL references VBE.web03(X.X.X.X,,1234).happy 18446744073709551615 . Happy health probes VBE.web04(X.X.X.X,,1234).vcls 1 . VCL references . . . --- Thank you for your feedback. Alex. From andrew.langhorn at digital.cabinet-office.gov.uk Wed Jun 18 15:16:16 2014 From: andrew.langhorn at digital.cabinet-office.gov.uk (Andrew Langhorn) Date: Wed, 18 Jun 2014 16:16:16 +0100 Subject: Question about varnishstat In-Reply-To: <53A1A5A0.9020106@opendoc.net> References: <53A1A5A0.9020106@opendoc.net> Message-ID: It?s 2^64, which is also the answer to the cheeseboard problem. In this case, I suspect it?s an unsigned bigint. The default value for that is 18446744073709551615. In Varnish terms, I?m not sure what that field should display on a happy system, though. On 18 June 2014 at 15:49:56, Alexandre (infos at opendoc.net) wrote: Hello, I use varnishstat to watch hosts state, and I wonder what is the number 18446744073709551615 ? Would you have any idea? example: --- varnishstat -1 . . . VBE.web01(X.X.X.X,,1234).vcls 1 . VCL references VBE.web01_MC(X.X.X.X,,1234).happy 18446744073709551615 . Happy health probes VBE.web02(X.X.X.X,,1234).vcls 1 . VCL references VBE.web02(X.X.X.X,,1234).happy 18446744073709551615 . Happy health probes VBE.web03(X.X.X.X,,1234).vcls 1 . VCL references VBE.web03(X.X.X.X,,1234).happy 18446744073709551615 . Happy health probes VBE.web04(X.X.X.X,,1234).vcls 1 . VCL references . . . --- Thank you for your feedback. Alex. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hernan at cmsmedios.com Wed Jun 18 17:14:43 2014 From: hernan at cmsmedios.com (=?UTF-8?Q?Hern=C3=A1n_Marsili?=) Date: Wed, 18 Jun 2014 14:14:43 -0300 Subject: memory alloc and probe Message-ID: Hi, We have two problems while using Varnish: 1) we actually don't know how to determine if the amount of memory we assigned to the MALLOC is enough or not. varnishstat -1 show this: SMA.s0.c_req 58035 34.08 Allocator requests SMA.s0.c_fail 0 0.00 Allocator failures SMA.s0.c_bytes 4830340101 2836371.17 Bytes allocated SMA.s0.c_freed 2864339421 1681937.42 Bytes freed SMA.s0.g_alloc 44952 . Allocations outstanding SMA.s0.g_bytes 1966000680 . Bytes outstanding SMA.s0.g_space 181482968 . Bytes available SMA.Transient.c_req 72960 42.84 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 4366160011 2563805.06 Bytes allocated SMA.Transient.c_freed 4365783718 2563584.10 Bytes freed SMA.Transient.g_alloc 350 . Allocations outstanding SMA.Transient.g_bytes 376293 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available 2) PROBE: we configured the probes as follows. What we noticed is, once the backend is mark as sick, even when the service is restored, Varnish continues to answering with a GURU MEDITATION. Is like VARNISH never checks back again the backend to see if it is healthy again. Is that a misconfiguration we have? .probe = { .url = "http://www.tycsports.com/balancer/alive1.html"; .interval = 30s; #chequear cada X seg .timeout = 10s; #timeout de la prueba .window = 10; #el probing funciona como sliding window .threshold = 8; # cuantos del window tienen que estar ok para declararl$ Any help will be much appreciated! tnx :) Regards, Hern?n. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry.magnien at sfr.com Thu Jun 19 06:07:56 2014 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Thu, 19 Jun 2014 06:07:56 +0000 Subject: memory alloc and probe In-Reply-To: References: Message-ID: <5D103CE839D50E4CBC62C9FD7B83287CC9BE6E4E@EXCN015.encara.local.ads> Hi, 1) Tou have to check if you have objects that are removed from cache because of lack of space. They are marked as "n_lru_nuked" in varnishstat. 2) With your probe configuration, varnish will have to get at least 8 good results out of 10 from probes in order to consider the backend healthy. With an interval of 30s, this can take quite long ;-) Regards, Thierry ----------------------------- De?: varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Hern?n Marsili Envoy??: mercredi 18 juin 2014 19:15 ??: varnish-misc Objet?: memory alloc and probe Hi, We have two problems while using Varnish: 1) we actually don't know how to determine if the amount of memory we assigned to the MALLOC is enough or not. varnishstat -1 show this: ? SMA.s0.c_req ? ? ? ? ? ? ? 58035? ? ? ? 34.08 Allocator requests SMA.s0.c_fail? ? ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 Allocator failures SMA.s0.c_bytes? ? ? ? 4830340101 ? 2836371.17 Bytes allocated SMA.s0.c_freed? ? ? ? 2864339421 ? 1681937.42 Bytes freed SMA.s0.g_alloc ? ? ? ? ? ? 44952? ? ? ? ? . ? Allocations outstanding SMA.s0.g_bytes? ? ? ? 1966000680? ? ? ? ? . ? Bytes outstanding SMA.s0.g_space ? ? ? ? 181482968? ? ? ? ? . ? Bytes available SMA.Transient.c_req? ? ? ? 72960? ? ? ? 42.84 Allocator requests SMA.Transient.c_fail ? ? ? ? ? 0 ? ? ? ? 0.00 Allocator failures SMA.Transient.c_bytes ? 4366160011 ? 2563805.06 Bytes allocated SMA.Transient.c_freed ? 4365783718 ? 2563584.10 Bytes freed SMA.Transient.g_alloc? ? ? ? ? 350? ? ? ? ? . ? Allocations outstanding SMA.Transient.g_bytes ? ? ? 376293? ? ? ? ? . ? Bytes outstanding SMA.Transient.g_space? ? ? ? ? ? 0? ? ? ? ? . ? Bytes available 2) PROBE: we configured the probes as follows. What we noticed is, once the backend is mark as sick, even when the service is restored, Varnish continues to answering with a GURU MEDITATION. Is like VARNISH never checks back again the backend to see if it is healthy again. Is that a misconfiguration we have? ?.probe = { ? ? ? .url = "http://www.tycsports.com/balancer/alive1.html"; ? ? ? .interval = 30s; ? #chequear cada X seg ? ? ? .timeout = 10s;? ? #timeout de la prueba ? ? ? .window = 10;? ? ? #el probing funciona como sliding window ? ? ? .threshold = 8; ? # cuantos del window tienen que estar ok para declararl$ ? Any help will be much appreciated! tnx :) Regards, Hern?n. From infos at opendoc.net Thu Jun 19 06:54:33 2014 From: infos at opendoc.net (Alexandre) Date: Thu, 19 Jun 2014 08:54:33 +0200 Subject: Question about varnishstat In-Reply-To: References: <53A1A5A0.9020106@opendoc.net> Message-ID: <53A28929.3080505@opendoc.net> I noticed that if I stop a web server managed by varnish sevrer, this number decreased Alex. On 18/06/14 17:16, Andrew Langhorn wrote: > It?s 2^64, which is also the answer to the cheeseboard problem. In this > case, I suspect it?s an unsigned bigint. The default value for that is > 18446744073709551615. In Varnish terms, I?m not sure what that field > should display on a happy system, though. > > On 18 June 2014 at 15:49:56, Alexandre (infos at opendoc.net > ) wrote: > >> Hello, >> >> I use varnishstat to watch hosts state, and I wonder what is the number >> 18446744073709551615 ? Would you have any idea? >> >> example: >> --- >> varnishstat -1 >> . >> . >> . >> VBE.web01(X.X.X.X,,1234).vcls 1 . VCL references >> VBE.web01_MC(X.X.X.X,,1234).happy 18446744073709551615 . >> Happy health probes >> VBE.web02(X.X.X.X,,1234).vcls 1 . VCL references >> VBE.web02(X.X.X.X,,1234).happy 18446744073709551615 . Happy >> health probes >> VBE.web03(X.X.X.X,,1234).vcls 1 . VCL references >> VBE.web03(X.X.X.X,,1234).happy 18446744073709551615 . Happy >> health probes >> VBE.web04(X.X.X.X,,1234).vcls 1 . VCL references >> . >> . >> . >> --- >> >> >> Thank you for your feedback. >> >> Alex. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From glenn at squixa.com Thu Jun 19 08:21:13 2014 From: glenn at squixa.com (Glenn Slaven) Date: Thu, 19 Jun 2014 18:21:13 +1000 Subject: memory alloc and probe In-Reply-To: <5D103CE839D50E4CBC62C9FD7B83287CC9BE6E4E@EXCN015.encara.local.ads> References: <5D103CE839D50E4CBC62C9FD7B83287CC9BE6E4E@EXCN015.encara.local.ads> Message-ID: <19EC89AC-A853-4ED0-A619-43BF93AA5B64@squixa.com> Are the probe thresholds and counts configurable? > On 19 Jun 2014, at 4:07 pm, "MAGNIEN, Thierry" wrote: > > Hi, > > 1) Tou have to check if you have objects that are removed from cache because of lack of space. They are marked as "n_lru_nuked" in varnishstat. > > 2) With your probe configuration, varnish will have to get at least 8 good results out of 10 from probes in order to consider the backend healthy. With an interval of 30s, this can take quite long ;-) > > Regards, > Thierry > > > > ----------------------------- > De : varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Hern?n Marsili > Envoy? : mercredi 18 juin 2014 19:15 > ? : varnish-misc > Objet : memory alloc and probe > > Hi, > > We have two problems while using Varnish: > > 1) we actually don't know how to determine if the amount of memory we assigned to the MALLOC is enough or not. varnishstat -1 show this: > > SMA.s0.c_req 58035 34.08 Allocator requests > SMA.s0.c_fail 0 0.00 Allocator failures > SMA.s0.c_bytes 4830340101 2836371.17 Bytes allocated > SMA.s0.c_freed 2864339421 1681937.42 Bytes freed > SMA.s0.g_alloc 44952 . Allocations outstanding > SMA.s0.g_bytes 1966000680 . Bytes outstanding > SMA.s0.g_space 181482968 . Bytes available > SMA.Transient.c_req 72960 42.84 Allocator requests > SMA.Transient.c_fail 0 0.00 Allocator failures > SMA.Transient.c_bytes 4366160011 2563805.06 Bytes allocated > SMA.Transient.c_freed 4365783718 2563584.10 Bytes freed > SMA.Transient.g_alloc 350 . Allocations outstanding > SMA.Transient.g_bytes 376293 . Bytes outstanding > SMA.Transient.g_space 0 . Bytes available > 2) PROBE: we configured the probes as follows. What we noticed is, once the backend is mark as sick, even when the service is restored, Varnish continues to answering with a GURU MEDITATION. Is like VARNISH never checks back again the backend to see if it is healthy again. Is that a misconfiguration we have? > > .probe = { > .url = "http://www.tycsports.com/balancer/alive1.html"; > .interval = 30s; #chequear cada X seg > .timeout = 10s; #timeout de la prueba > .window = 10; #el probing funciona como sliding window > .threshold = 8; # cuantos del window tienen que estar ok para declararl$ > > Any help will be much appreciated! tnx :) > > Regards, > Hern?n. > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From glenn at squixa.com Thu Jun 19 08:21:13 2014 From: glenn at squixa.com (Glenn Slaven) Date: Thu, 19 Jun 2014 18:21:13 +1000 Subject: memory alloc and probe In-Reply-To: <5D103CE839D50E4CBC62C9FD7B83287CC9BE6E4E@EXCN015.encara.local.ads> References: <5D103CE839D50E4CBC62C9FD7B83287CC9BE6E4E@EXCN015.encara.local.ads> Message-ID: <19EC89AC-A853-4ED0-A619-43BF93AA5B64@squixa.com> Are the probe thresholds and counts configurable? > On 19 Jun 2014, at 4:07 pm, "MAGNIEN, Thierry" wrote: > > Hi, > > 1) Tou have to check if you have objects that are removed from cache because of lack of space. They are marked as "n_lru_nuked" in varnishstat. > > 2) With your probe configuration, varnish will have to get at least 8 good results out of 10 from probes in order to consider the backend healthy. With an interval of 30s, this can take quite long ;-) > > Regards, > Thierry > > > > ----------------------------- > De : varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Hern?n Marsili > Envoy? : mercredi 18 juin 2014 19:15 > ? : varnish-misc > Objet : memory alloc and probe > > Hi, > > We have two problems while using Varnish: > > 1) we actually don't know how to determine if the amount of memory we assigned to the MALLOC is enough or not. varnishstat -1 show this: > > SMA.s0.c_req 58035 34.08 Allocator requests > SMA.s0.c_fail 0 0.00 Allocator failures > SMA.s0.c_bytes 4830340101 2836371.17 Bytes allocated > SMA.s0.c_freed 2864339421 1681937.42 Bytes freed > SMA.s0.g_alloc 44952 . Allocations outstanding > SMA.s0.g_bytes 1966000680 . Bytes outstanding > SMA.s0.g_space 181482968 . Bytes available > SMA.Transient.c_req 72960 42.84 Allocator requests > SMA.Transient.c_fail 0 0.00 Allocator failures > SMA.Transient.c_bytes 4366160011 2563805.06 Bytes allocated > SMA.Transient.c_freed 4365783718 2563584.10 Bytes freed > SMA.Transient.g_alloc 350 . Allocations outstanding > SMA.Transient.g_bytes 376293 . Bytes outstanding > SMA.Transient.g_space 0 . Bytes available > 2) PROBE: we configured the probes as follows. What we noticed is, once the backend is mark as sick, even when the service is restored, Varnish continues to answering with a GURU MEDITATION. Is like VARNISH never checks back again the backend to see if it is healthy again. Is that a misconfiguration we have? > > .probe = { > .url = "http://www.tycsports.com/balancer/alive1.html"; > .interval = 30s; #chequear cada X seg > .timeout = 10s; #timeout de la prueba > .window = 10; #el probing funciona como sliding window > .threshold = 8; # cuantos del window tienen que estar ok para declararl$ > > Any help will be much appreciated! tnx :) > > Regards, > Hern?n. > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From thierry.magnien at sfr.com Thu Jun 19 08:25:18 2014 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Thu, 19 Jun 2014 08:25:18 +0000 Subject: memory alloc and probe In-Reply-To: <19EC89AC-A853-4ED0-A619-43BF93AA5B64@squixa.com> References: <5D103CE839D50E4CBC62C9FD7B83287CC9BE6E4E@EXCN015.encara.local.ads> <19EC89AC-A853-4ED0-A619-43BF93AA5B64@squixa.com> Message-ID: <5D103CE839D50E4CBC62C9FD7B83287CC9BE71E9@EXCN015.encara.local.ads> Yes, see here: https://www.varnish-cache.org/docs/4.0/reference/vcl.html#probes Same config options are available for 3.x releases. Thierry -----Message d'origine----- De?: Glenn Slaven [mailto:glenn at squixa.com] Envoy??: jeudi 19 juin 2014 10:21 ??: MAGNIEN, Thierry Cc?: Hern?n Marsili; varnish-misc Objet?: Re: memory alloc and probe Are the probe thresholds and counts configurable? > On 19 Jun 2014, at 4:07 pm, "MAGNIEN, Thierry" wrote: > > Hi, > > 1) Tou have to check if you have objects that are removed from cache because of lack of space. They are marked as "n_lru_nuked" in varnishstat. > > 2) With your probe configuration, varnish will have to get at least 8 good results out of 10 from probes in order to consider the backend healthy. With an interval of 30s, this can take quite long ;-) > > Regards, > Thierry > > > > ----------------------------- > De : varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Hern?n Marsili > Envoy? : mercredi 18 juin 2014 19:15 > ? : varnish-misc > Objet : memory alloc and probe > > Hi, > > We have two problems while using Varnish: > > 1) we actually don't know how to determine if the amount of memory we assigned to the MALLOC is enough or not. varnishstat -1 show this: > > SMA.s0.c_req 58035 34.08 Allocator requests > SMA.s0.c_fail 0 0.00 Allocator failures > SMA.s0.c_bytes 4830340101 2836371.17 Bytes allocated > SMA.s0.c_freed 2864339421 1681937.42 Bytes freed > SMA.s0.g_alloc 44952 . Allocations outstanding > SMA.s0.g_bytes 1966000680 . Bytes outstanding > SMA.s0.g_space 181482968 . Bytes available > SMA.Transient.c_req 72960 42.84 Allocator requests > SMA.Transient.c_fail 0 0.00 Allocator failures > SMA.Transient.c_bytes 4366160011 2563805.06 Bytes allocated > SMA.Transient.c_freed 4365783718 2563584.10 Bytes freed > SMA.Transient.g_alloc 350 . Allocations outstanding > SMA.Transient.g_bytes 376293 . Bytes outstanding > SMA.Transient.g_space 0 . Bytes available > 2) PROBE: we configured the probes as follows. What we noticed is, once the backend is mark as sick, even when the service is restored, Varnish continues to answering with a GURU MEDITATION. Is like VARNISH never checks back again the backend to see if it is healthy again. Is that a misconfiguration we have? > > .probe = { > .url = "http://www.tycsports.com/balancer/alive1.html"; > .interval = 30s; #chequear cada X seg > .timeout = 10s; #timeout de la prueba > .window = 10; #el probing funciona como sliding window > .threshold = 8; # cuantos del window tienen que estar ok para declararl$ > > Any help will be much appreciated! tnx :) > > Regards, > Hern?n. > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From dhose0 at gmail.com Thu Jun 19 11:03:07 2014 From: dhose0 at gmail.com (Daniel) Date: Thu, 19 Jun 2014 13:03:07 +0200 Subject: Question regarding Varnish memory usage Message-ID: Hi, I have some questions regarding Varnish 4 and some problems we have with it's memory usage. We are using Varnish 4 on a virtual server (CentOS 6.5) with 14 GB memory and 9 GB malloc. The TTL in use is 5m and the grace time is 60m. Varnish sits in front of an API that is responding with JSON only. On the API servers we are using Nginx, and according to the body_bytes_sent variable in Nginx we can see that the average response size is 4.4 Kbyte and in 1 hour the API is sending around 300 Mbytes of data. Nginx is also compressing the response before sending it to Varnish. The number of requests to the API servers in 1 hour is between 60 000 and 70 000. The problem we see is that when we reach around 73 000 objects in Varnish all the memory (9G) is used and the lru_nuked counter starts to increase. When Varnish is restarted most of the memory is used after 1 to 1.5 hour. As mentioned before the TTL is 5 minutes and the grace time is 60 minutes. I am really confused why we are seeing this behaviour, especially when the API servers is only sending around 300 Mbytes of data in 1 hour time. I hope somebody can shed some light into this, thanks! ## Backend request ## * << BeReq >> 808484921 - Begin bereq 808484920 fetch - Timestamp Start: 1403174559.768064 0.000000 0.000000 - BereqMethod GET - BereqURL request_url - BereqProtocol HTTP/1.1 - BereqHeader Content-Type: application/hal+json - BereqHeader Accept: application/hal+json - BereqHeader Cneonction: close - BereqHeader Host: backend-servers:8080 - BereqHeader X-Forwarded-For: x.x.x.x - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 808484921 - VCL_call BACKEND_FETCH - VCL_return fetch - Backend 46 backend backend3(x.x.x.x,,8080) - Timestamp Bereq: 1403174559.768100 0.000036 0.000036 - Timestamp Beresp: 1403174560.085275 0.317211 0.317176 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespResponse OK - BerespHeader Server: nginx/1.6.0 - BerespHeader Date: Thu, 19 Jun 2014 10:42:46 GMT - BerespHeader Content-Type: application/hal+json; charset=utf-8 - BerespHeader Transfer-Encoding: chunked - BerespHeader Connection: keep-alive - BerespHeader Vary: Accept-Encoding - BerespHeader Status: 200 OK - BerespHeader X-Frame-Options: SAMEORIGIN - BerespHeader X-XSS-Protection: 1; mode=block - BerespHeader X-Content-Type-Options: nosniff - BerespHeader X-UA-Compatible: chrome=1 - BerespHeader Cache-Control: max-age=0, private, must-revalidate - BerespHeader X-Runtime: 0.182126 - BerespHeader Content-Encoding: gzip - TTL RFC 0 -1 -1 1403174560 1403174560 1403174566 0 0 - VCL_call BACKEND_RESPONSE - TTL VCL 300 10 0 1403174560 - TTL VCL 300 3600 0 1403174560 - VCL_return deliver - Storage malloc s0 - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjResponse OK - ObjHeader Server: nginx/1.6.0 - ObjHeader Date: Thu, 19 Jun 2014 10:42:46 GMT - ObjHeader Content-Type: application/hal+json; charset=utf-8 - ObjHeader Vary: Accept-Encoding - ObjHeader Status: 200 OK - ObjHeader X-Frame-Options: SAMEORIGIN - ObjHeader X-XSS-Protection: 1; mode=block - ObjHeader X-Content-Type-Options: nosniff - ObjHeader X-UA-Compatible: chrome=1 - ObjHeader Cache-Control: max-age=0, private, must-revalidate - ObjHeader X-Runtime: 0.182126 - ObjHeader Content-Encoding: gzip - Fetch_Body 2 chunked stream - Gzip u F - 5876 21006 80 80 46942 - Timestamp BerespBody: 1403174560.085712 0.317648 0.000436 - BackendReuse 46 backend3(x.x.x.x,,8080) - Length 5876 - BereqAcct 471 0 471 657 5888 6545 - End ## default_vcl ## # This is a Varnish 4.x VCL file vcl 4.0; import directors; import std; backend backend1 { .host = "x.x.x.x"; .port = "8080"; .probe = { .url = "/ping"; .timeout = 4s; .interval = 5s; .window = 5; .threshold = 4; } } backend backend2 { .host = "x.x.x.x"; .port = "8080"; .probe = { .url = "/ping"; .timeout = 4s; .interval = 5s; .window = 5; .threshold = 4; } } backend backend3 { .host = "x.x.x.x"; .port = "8080"; .probe = { .url = "/ping"; .timeout = 4s; .interval = 5s; .window = 5; .threshold = 4; } } # Below is an example redirector based on round-robin requests sub vcl_init { new sservice = directors.round_robin(); sservice.add_backend(backend1); sservice.add_backend(backend2); sservice.add_backend(backend3); } acl local { "x.x.x.x"/24; } # Handle the HTTP request received by the client sub vcl_recv { set req.backend_hint = sservice.backend(); if (req.url ~ "^/ping$") { if (client.ip ~ local) { return (synth(900, "pong")); } } if (! req.http.Authorization ~ "apitokens") { return (synth(401, "Restricted")); } return (hash); } sub vcl_backend_response { set beresp.ttl = 5m; set beresp.grace = 60m; return (deliver); } sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } return (deliver); } sub vcl_synth { if (resp.status == 900) { set resp.status = 200; set resp.http.Content-Type = "text/plain; charset=utf-8"; synthetic("pong"); return (deliver); } set resp.http.Content-Type = "application/hal+json"; return (deliver); } sub vcl_backend_error { set beresp.http.Content-Type = "application/hal+json"; return (deliver); } ## varnishstat -1 ## MAIN.uptime 236760 1.00 Child process uptime MAIN.sess_conn 217727 0.92 Sessions accepted MAIN.sess_drop 0 0.00 Sessions dropped MAIN.sess_fail 0 0.00 Session accept failures MAIN.sess_pipe_overflow 0 0.00 Session pipe overflow MAIN.client_req_400 0 0.00 Client requests received, subject to 400 errors MAIN.client_req_411 0 0.00 Client requests received, subject to 411 errors MAIN.client_req_413 0 0.00 Client requests received, subject to 413 errors MAIN.client_req_417 0 0.00 Client requests received, subject to 417 errors MAIN.client_req 5316027 22.45 Good Client requests received MAIN.cache_hit 2061461 8.71 Cache hits MAIN.cache_hitpass 0 0.00 Cache hits for pass MAIN.cache_miss 3204801 13.54 Cache misses MAIN.backend_conn 107148 0.45 Backend conn. success MAIN.backend_unhealthy 0 0.00 Backend conn. not attempted MAIN.backend_busy 0 0.00 Backend conn. too many MAIN.backend_fail 0 0.00 Backend conn. failures MAIN.backend_reuse 4253086 17.96 Backend conn. reuses MAIN.backend_toolate 82192 0.35 Backend conn. was closed MAIN.backend_recycle 4335308 18.31 Backend conn. recycles MAIN.backend_retry 0 0.00 Backend conn. retry MAIN.fetch_head 0 0.00 Fetch no body (HEAD) MAIN.fetch_length 305 0.00 Fetch with Length MAIN.fetch_chunked 4359871 18.41 Fetch chunked MAIN.fetch_eof 0 0.00 Fetch EOF MAIN.fetch_bad 0 0.00 Fetch bad T-E MAIN.fetch_close 0 0.00 Fetch wanted close MAIN.fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed MAIN.fetch_zero 0 0.00 Fetch zero len body MAIN.fetch_1xx 0 0.00 Fetch no body (1xx) MAIN.fetch_204 0 0.00 Fetch no body (204) MAIN.fetch_304 0 0.00 Fetch no body (304) MAIN.fetch_failed 4994 0.02 Fetch body failed MAIN.pools 2 . Number of thread pools MAIN.threads 125 . Total number of threads MAIN.threads_limited 0 0.00 Threads hit max MAIN.threads_created 24761 0.10 Threads created MAIN.threads_destroyed 24636 0.10 Threads destoryed MAIN.threads_failed 0 0.00 Thread creation failed MAIN.thread_queue_len 0 . Length of session queue MAIN.busy_sleep 44391 0.19 Number of requests sent to sleep on busy objhdr MAIN.busy_wakeup 44390 0.19 Number of requests woken after sleep on busy objhdr MAIN.sess_queued 22381 0.09 Sessions queued for thread MAIN.sess_dropped 0 0.00 Sessions dropped for thread MAIN.n_object 72729 . N struct object MAIN.n_vampireobject 0 . N unresurrected objects MAIN.n_objectcore 79071 . N struct objectcore MAIN.n_objecthead 69256 . N struct objecthead MAIN.n_waitinglist 21819 . N struct waitinglist MAIN.n_backend 3 . N backends MAIN.n_expired 2977375 . N expired objects MAIN.n_lru_nuked 1336131 . N LRU nuked objects MAIN.n_lru_moved 1928525 . N LRU moved objects MAIN.losthdr 0 0.00 HTTP header overflows MAIN.s_sess 217727 0.92 Total Sessions MAIN.s_req 5316027 22.45 Total Requests MAIN.s_pipe 0 0.00 Total pipe MAIN.s_pass 0 0.00 Total pass MAIN.s_fetch 3204801 13.54 Total fetch MAIN.s_synth 53251 0.22 Total synth MAIN.s_req_hdrbytes 2351626120 9932.53 Request header bytes MAIN.s_req_bodybytes 0 0.00 Request body bytes MAIN.s_resp_hdrbytes 3914065512 16531.79 Response header bytes MAIN.s_resp_bodybytes 78724096530 332505.90 Reponse body bytes MAIN.s_pipe_hdrbytes 0 0.00 Pipe request header bytes MAIN.s_pipe_in 0 0.00 Piped bytes from client MAIN.s_pipe_out 0 0.00 Piped bytes to client MAIN.sess_closed 47598 0.20 Session Closed MAIN.sess_pipeline 0 0.00 Session Pipeline MAIN.sess_readahead 0 0.00 Session Read Ahead MAIN.sess_herd 2769201 11.70 Session herd MAIN.shm_records 666122457 2813.49 SHM records MAIN.shm_writes 27933605 117.98 SHM writes MAIN.shm_flushes 0 0.00 SHM flushes due to overflow MAIN.shm_cont 1780 0.01 SHM MTX contention MAIN.shm_cycles 315 0.00 SHM cycles through buffer MAIN.sms_nreq 0 0.00 SMS allocator requests MAIN.sms_nobj 0 . SMS outstanding allocations MAIN.sms_nbytes 0 . SMS outstanding bytes MAIN.sms_balloc 0 . SMS bytes allocated MAIN.sms_bfree 0 . SMS bytes freed MAIN.backend_req 4360234 18.42 Backend requests made MAIN.n_vcl 4 0.00 N vcl total MAIN.n_vcl_avail 4 0.00 N vcl available MAIN.n_vcl_discard 0 0.00 N vcl discarded MAIN.bans 1 . Count of bans MAIN.bans_completed 1 . Number of bans marked 'completed' MAIN.bans_obj 0 . Number of bans using obj.* MAIN.bans_req 0 . Number of bans using req.* MAIN.bans_added 1 0.00 Bans added MAIN.bans_deleted 0 0.00 Bans deleted MAIN.bans_tested 0 0.00 Bans tested against objects (lookup) MAIN.bans_obj_killed 0 0.00 Objects killed by bans (lookup) MAIN.bans_lurker_tested 0 0.00 Bans tested against objects (lurker) MAIN.bans_tests_tested 0 0.00 Ban tests tested against objects (lookup) MAIN.bans_lurker_tests_tested 0 0.00 Ban tests tested against objects (lurker) MAIN.bans_lurker_obj_killed 0 0.00 Objects killed by bans (lurker) MAIN.bans_dups 0 0.00 Bans superseded by other bans MAIN.bans_lurker_contention 0 0.00 Lurker gave way for lookup MAIN.bans_persisted_bytes 13 . Bytes used by the persisted ban lists MAIN.bans_persisted_fragmentation 0 . Extra bytes in persisted ban lists due to fragmentation MAIN.exp_mailed 5722364 24.17 Number of objects mailed to expiry thread MAIN.exp_received 5722364 24.17 Number of objects received by expiry thread MAIN.hcb_nolock 5266297 22.24 HCB Lookups without lock MAIN.hcb_lock 3204827 13.54 HCB Lookups with lock MAIN.hcb_insert 3204827 13.54 HCB Inserts MAIN.esi_errors 0 0.00 ESI parse errors (unlock) MAIN.esi_warnings 0 0.00 ESI parse warnings (unlock) MAIN.vmods 2 . Loaded VMODs MAIN.n_gzip 4083400 17.25 Gzip operations MAIN.n_gunzip 5065871 21.40 Gunzip operations MAIN.vsm_free 972096 . Free VSM space MAIN.vsm_used 83962512 . Used VSM space MAIN.vsm_cooling 0 . Cooling VSM space MAIN.vsm_overflow 0 . Overflow VSM space MAIN.vsm_overflowed 0 0.00 Overflowed VSM space MGT.uptime 307850 1.30 Management process uptime MGT.child_start 2 0.00 Child process started MGT.child_exit 0 0.00 Child process normal exit MGT.child_stop 0 0.00 Child process unexpected exit MGT.child_died 1 0.00 Child process died (signal) MGT.child_dump 0 0.00 Child process core dumped MGT.child_panic 1 0.00 Child process panic MEMPOOL.vbc.live 88 . In use MEMPOOL.vbc.pool 15 . In Pool MEMPOOL.vbc.sz_wanted 88 . Size requested MEMPOOL.vbc.sz_needed 120 . Size allocated MEMPOOL.vbc.allocs 107148 0.45 Allocations MEMPOOL.vbc.frees 107060 0.45 Frees MEMPOOL.vbc.recycle 105673 0.45 Recycled from pool MEMPOOL.vbc.timeout 75842 0.32 Timed out from pool MEMPOOL.vbc.toosmall 0 0.00 Too small to recycle MEMPOOL.vbc.surplus 2032 0.01 Too many for pool MEMPOOL.vbc.randry 1475 0.01 Pool ran dry MEMPOOL.busyobj.live 58 . In use MEMPOOL.busyobj.pool 11 . In Pool MEMPOOL.busyobj.sz_wanted 65536 . Size requested MEMPOOL.busyobj.sz_needed 65568 . Size allocated MEMPOOL.busyobj.allocs 4391285 18.55 Allocations MEMPOOL.busyobj.frees 4391227 18.55 Frees MEMPOOL.busyobj.recycle 4380692 18.50 Recycled from pool MEMPOOL.busyobj.timeout 231100 0.98 Timed out from pool MEMPOOL.busyobj.toosmall 0 0.00 Too small to recycle MEMPOOL.busyobj.surplus 916 0.00 Too many for pool MEMPOOL.busyobj.randry 10593 0.04 Pool ran dry MEMPOOL.req0.live 14 . In use MEMPOOL.req0.pool 10 . In Pool MEMPOOL.req0.sz_wanted 65536 . Size requested MEMPOOL.req0.sz_needed 65568 . Size allocated MEMPOOL.req0.allocs 1408010 5.95 Allocations MEMPOOL.req0.frees 1407996 5.95 Frees MEMPOOL.req0.recycle 1407941 5.95 Recycled from pool MEMPOOL.req0.timeout 167174 0.71 Timed out from pool MEMPOOL.req0.toosmall 0 0.00 Too small to recycle MEMPOOL.req0.surplus 0 0.00 Too many for pool MEMPOOL.req0.randry 69 0.00 Pool ran dry MEMPOOL.sess0.live 19 . In use MEMPOOL.sess0.pool 10 . In Pool MEMPOOL.sess0.sz_wanted 384 . Size requested MEMPOOL.sess0.sz_needed 416 . Size allocated MEMPOOL.sess0.allocs 108788 0.46 Allocations MEMPOOL.sess0.frees 108769 0.46 Frees MEMPOOL.sess0.recycle 108735 0.46 Recycled from pool MEMPOOL.sess0.timeout 76175 0.32 Timed out from pool MEMPOOL.sess0.toosmall 0 0.00 Too small to recycle MEMPOOL.sess0.surplus 0 0.00 Too many for pool MEMPOOL.sess0.randry 53 0.00 Pool ran dry MEMPOOL.req1.live 14 . In use MEMPOOL.req1.pool 11 . In Pool MEMPOOL.req1.sz_wanted 65536 . Size requested MEMPOOL.req1.sz_needed 65568 . Size allocated MEMPOOL.req1.allocs 1409105 5.95 Allocations MEMPOOL.req1.frees 1409091 5.95 Frees MEMPOOL.req1.recycle 1408955 5.95 Recycled from pool MEMPOOL.req1.timeout 168231 0.71 Timed out from pool MEMPOOL.req1.toosmall 0 0.00 Too small to recycle MEMPOOL.req1.surplus 0 0.00 Too many for pool MEMPOOL.req1.randry 150 0.00 Pool ran dry MEMPOOL.sess1.live 19 . In use MEMPOOL.sess1.pool 13 . In Pool MEMPOOL.sess1.sz_wanted 384 . Size requested MEMPOOL.sess1.sz_needed 416 . Size allocated MEMPOOL.sess1.allocs 108939 0.46 Allocations MEMPOOL.sess1.frees 108920 0.46 Frees MEMPOOL.sess1.recycle 108788 0.46 Recycled from pool MEMPOOL.sess1.timeout 76251 0.32 Timed out from pool MEMPOOL.sess1.toosmall 0 0.00 Too small to recycle MEMPOOL.sess1.surplus 0 0.00 Too many for pool MEMPOOL.sess1.randry 151 0.00 Pool ran dry SMA.s0.c_req 10059622 42.49 Allocator requests SMA.s0.c_fail 1339270 5.66 Allocator failures SMA.s0.c_bytes 576850225314 2436434.47 Bytes allocated SMA.s0.c_freed 567228141680 2395793.81 Bytes freed SMA.s0.g_alloc 145458 . Allocations outstanding SMA.s0.g_bytes 9622083634 . Bytes outstanding SMA.s0.g_space 41592782 . Bytes available SMA.Transient.c_req 26057 0.11 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 11256624 47.54 Bytes allocated SMA.Transient.c_freed 11256624 47.54 Bytes freed SMA.Transient.g_alloc 0 . Allocations outstanding SMA.Transient.g_bytes 0 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.backend1(x.x.x.x,,8080).vcls 4 . VCL references VBE.backend1(x.x.x.x,,8080).happy 11493186249049505791 . Happy health probes VBE.backend1(x.x.x.x,,8080).bereq_hdrbytes 715578294 3022.38 Request header bytes VBE.backend1(x.x.x.x,,8080).bereq_bodybytes 0 0.00 Request body bytes VBE.backend1(x.x.x.x,,8080).beresp_hdrbytes 941776370 3977.77 Response header bytes VBE.backend1(x.x.x.x,,8080).beresp_bodybytes 17197789137 72638.07 Response body bytes VBE.backend1(x.x.x.x,,8080).pipe_hdrbytes 0 0.00 Pipe request header bytes VBE.backend1(x.x.x.x,,8080).pipe_out 0 0.00 Piped bytes to backend VBE.backend1(x.x.x.x,,8080).pipe_in 0 0.00 Piped bytes from backend VBE.backend2(x.x.x.x,,8080).vcls 4 . VCL references VBE.backend2(x.x.x.x,,8080).happy 14987979559889002495 . Happy health probes VBE.backend2(x.x.x.x,,8080).bereq_hdrbytes 715272478 3021.09 Request header bytes VBE.backend2(x.x.x.x,,8080).bereq_bodybytes 0 0.00 Request body bytes VBE.backend2(x.x.x.x,,8080).beresp_hdrbytes 941339831 3975.92 Response header bytes VBE.backend2(x.x.x.x,,8080).beresp_bodybytes 17174050588 72537.80 Response body bytes VBE.backend2(x.x.x.x,,8080).pipe_hdrbytes 0 0.00 Pipe request header bytes VBE.backend2(x.x.x.x,,8080).pipe_out 0 0.00 Piped bytes to backend VBE.backend2(x.x.x.x,,8080).pipe_in 0 0.00 Piped bytes from backend VBE.backend3(x.x.x.x,,8080).vcls 4 . VCL references VBE.backend3(x.x.x.x,,8080).happy 4611686018427363327 . Happy health probes VBE.backend3(x.x.x.x,,8080).bereq_hdrbytes 707848949 2989.73 Request header bytes VBE.backend3(x.x.x.x,,8080).bereq_bodybytes 0 0.00 Request body bytes VBE.backend3(x.x.x.x,,8080).beresp_hdrbytes 931448534 3934.15 Response header bytes VBE.backend3(x.x.x.x,,8080).beresp_bodybytes 16984021399 71735.18 Response body bytes VBE.backend3(x.x.x.x,,8080).pipe_hdrbytes 0 0.00 Pipe request header bytes VBE.backend3(x.x.x.x,,8080).pipe_out 0 0.00 Piped bytes to backend VBE.backend3(x.x.x.x,,8080).pipe_in 0 0.00 Piped bytes from backend LCK.sms.creat 0 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 0 0.00 Lock Operations LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.sma.creat 2 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 18686630 78.93 Lock Operations LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hcb.creat 1 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 6356203 26.85 Lock Operations LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 8976880 37.92 Lock Operations LCK.sessmem.creat 0 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 0 0.00 Lock Operations LCK.sess.creat 217727 0.92 Created locks LCK.sess.destroy 217689 0.92 Destroyed locks LCK.sess.locks 0 0.00 Lock Operations LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 19771639 83.51 Lock Operations LCK.herder.creat 0 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 0 0.00 Lock Operations LCK.wq.creat 3 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 14797256 62.50 Lock Operations LCK.objhdr.creat 3216824 13.59 Created locks LCK.objhdr.destroy 3147552 13.29 Destroyed locks LCK.objhdr.locks 60262894 254.53 Lock Operations LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 26069460 110.11 Lock Operations LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 18404226 77.73 Lock Operations LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 78936 0.33 Lock Operations LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 9477160 40.03 Lock Operations LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 124145 0.52 Lock Operations LCK.backend.creat 3 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 8991937 37.98 Lock Operations LCK.vcapace.creat 1 0.00 Created locks LCK.vcapace.destroy 0 0.00 Destroyed locks LCK.vcapace.locks 0 0.00 Lock Operations LCK.nbusyobj.creat 0 0.00 Created locks LCK.nbusyobj.destroy 0 0.00 Destroyed locks LCK.nbusyobj.locks 0 0.00 Lock Operations LCK.busyobj.creat 4391285 18.55 Created locks LCK.busyobj.destroy 4391227 18.55 Destroyed locks LCK.busyobj.locks 33222445 140.32 Lock Operations LCK.mempool.creat 6 0.00 Created locks LCK.mempool.destroy 0 0.00 Destroyed locks LCK.mempool.locks 17553267 74.14 Lock Operations LCK.vxid.creat 1 0.00 Created locks LCK.vxid.destroy 0 0.00 Destroyed locks LCK.vxid.locks 22801 0.10 Lock Operations LCK.pipestat.creat 1 0.00 Created locks LCK.pipestat.destroy 0 0.00 Destroyed locks LCK.pipestat.locks 0 0.00 Lock Operations -------------- next part -------------- An HTML attachment was scrubbed... URL: From enridaga at apache.org Thu Jun 19 16:33:15 2014 From: enridaga at apache.org (enridaga) Date: Thu, 19 Jun 2014 17:33:15 +0100 Subject: How to cache responses to POST requests Message-ID: Hi all, I have found something about it on SO [1] but unfortunately I can't translate the requests to GET. My use case is about caching time consuming calls to a remote REST service going through a corporate proxy. With the following vcl script I was able to use the corporate proxy and make POST requests work fine. However, the responses are not cached. Does anybody know the magic words? Thank you! Enrico # This is my script backend default { .host = "my-remote-service.org "; .port = "80"; } backend ouproxy { .host = "my-corporate-proxy.org "; .port = "80"; } sub vcl_recv { set req.url = "http://my-remote-service.org " + req.url; set req.backend = ouproxy; if(req.request=="POST"){ return (lookup); }else{ return (lookup); } } sub vcl_pass { if(req.request=="POST"){ set bereq.request = "POST"; } } [1] http://stackoverflow.com/questions/9023590/varnish-caching-of-post-requests Enrico Daga ------------------------------------------------------------------------------ enridaga From james at ifixit.com Thu Jun 19 17:16:39 2014 From: james at ifixit.com (James Pearson) Date: Thu, 19 Jun 2014 10:16:39 -0700 Subject: How to cache responses to POST requests In-Reply-To: References: Message-ID: <1403198156-sup-6799@geror.local> Excerpts from enridaga's message of 2014-06-19 09:33:15 -0700: > With the following vcl script I was able to use the corporate proxy > and make POST requests work fine. > However, the responses are not cached. What does your vcl_hash() look like? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From enridaga at apache.org Thu Jun 19 17:22:24 2014 From: enridaga at apache.org (enridaga) Date: Thu, 19 Jun 2014 18:22:24 +0100 Subject: How to cache responses to POST requests In-Reply-To: <1403198156-sup-6799@geror.local> References: <1403198156-sup-6799@geror.local> Message-ID: Hi James, thank you for your answer. I did not redefined vcl_hash, so I guess it behave as the commented default: # sub vcl_hash { # hash_data(req.url); # if (req.http.host) { # hash_data(req.http.host); # } else { # hash_data(server.ip); # } # return (hash); # } Enrico ------------------------------------------------------------------------------ enridaga On 19 June 2014 18:16, James Pearson wrote: > Excerpts from enridaga's message of 2014-06-19 09:33:15 -0700: >> With the following vcl script I was able to use the corporate proxy >> and make POST requests work fine. >> However, the responses are not cached. > > What does your vcl_hash() look like? > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From dhose0 at gmail.com Mon Jun 23 14:02:22 2014 From: dhose0 at gmail.com (Daniel) Date: Mon, 23 Jun 2014 16:02:22 +0200 Subject: varnishsizes (Varnish 4) Message-ID: <53A8336E.4060300@gmail.com> Hi, Installed Varnish4 using varnish-4.0.0-1.el6.x86_64.rpm from https://repo.varnish-cache.org/redhat/varnish-4.0/el6/x86_64/ but there is no varnishsizes binary installed. Is it just missing in the RPM package or is varnishsizes removed in Varnish 4? Thanks! /Daniel From daghf at varnish-software.com Mon Jun 23 14:25:36 2014 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Mon, 23 Jun 2014 16:25:36 +0200 Subject: varnishsizes (Varnish 4) In-Reply-To: <53A8336E.4060300@gmail.com> References: <53A8336E.4060300@gmail.com> Message-ID: Hi varnishsizes is now varnishhist -P size Regards, Dag On Mon, Jun 23, 2014 at 4:02 PM, Daniel wrote: > Hi, > > Installed Varnish4 using varnish-4.0.0-1.el6.x86_64.rpm from > https://repo.varnish-cache.org/redhat/varnish-4.0/el6/x86_64/ but there is > no varnishsizes binary installed. Is it just missing in the RPM package or > is varnishsizes removed in Varnish 4? > > > Thanks! > > > /Daniel > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Dag Haavi Finstad Software Developer | Varnish Software AS Mobile: +47 476 64 134 We Make Websites Fly! From itxx00 at gmail.com Wed Jun 25 05:56:02 2014 From: itxx00 at gmail.com (Gao Yongwei) Date: Wed, 25 Jun 2014 13:56:02 +0800 Subject: Does varnish4.0 libvmod_directors support dns director ? Message-ID: Hello, as in varnish3.0 there is a dns director, but it seems that it has been removed from 4.0 ? how can I get it work in 4.0? thanks. -- *Regards,* *GaoYongwei* -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Wed Jun 25 07:46:00 2014 From: perbu at varnish-software.com (Per Buer) Date: Wed, 25 Jun 2014 09:46:00 +0200 Subject: Does varnish4.0 libvmod_directors support dns director ? In-Reply-To: References: Message-ID: Hii Gao. On Wed, Jun 25, 2014 at 7:56 AM, Gao Yongwei wrote: > Hello, > as in varnish3.0 there is a dns director, but it seems that it has been > removed from 4.0 ? > how can I get it work in 4.0? thanks. > You would need to port it to 4.0. It was removed due to lack of use and fundamental incompatibility with IPv6. Per. -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dhose0 at gmail.com Wed Jun 25 14:59:01 2014 From: dhose0 at gmail.com (Daniel) Date: Wed, 25 Jun 2014 16:59:01 +0200 Subject: Question regarding Varnish memory usage In-Reply-To: References: Message-ID: <53AAE3B5.5000909@gmail.com> Hi again, Yesterday the Varnish (version 4.0.0) child process restarted with the error below in syslog. Varnish had been running for 7 days with 9 GB malloc on a host with 14 GB memory when the error occurred. The number of objects (MAIN.n_object) was approximately 70 000. Any ideas what happened here? I thought that 5 GB of "extra" memory would be more than enough. I have afterwards upgraded to version 4.0.1, changed to 8 GB malloc and lowered the grace time from 60 minutes to 45 minutes. Jun 24 14:48:58 varnish varnishd[48344]: Child (58488) Panic message: Assert error in mpl_alloc(), cache/cache_mempool.c line 80:#012 Condition((mi) != 0) not true.#012errno = 12 (Cannot allocate memory)#012thread = (MPL_req0)#012ident = Linux,2.6.32-431.17.1.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012 Thanks! Regards, Daniel From patrick at airseed.com Thu Jun 26 04:50:23 2014 From: patrick at airseed.com (Patrick Kaeding) Date: Wed, 25 Jun 2014 21:50:23 -0700 Subject: ubuntu source packages for varnish 4.0 Message-ID: Hi all, apologies if this is not the right mailing list for this question. I am having trouble getting the varnish 4.0 source, in ubuntu 14.04. I have added the repo.varnish-cache.org lines to my sources.list, as such: # cat /etc/apt/sources.list.d/varnish-cache.list deb https://repo.varnish-cache.org/ubuntu/ trusty varnish-4.0 deb-src https://repo.varnish-cache.org/ubuntu/ trusty varnish-4.0 When I `apt-get update`, I see the following lines (along with all the other usual lines), so I am pretty sure the sources.list changes above have taken effect: Get:3 https://repo.varnish-cache.org trusty InRelease Get:5 https://repo.varnish-cache.org trusty/varnish-4.0 Sources Get:6 https://repo.varnish-cache.org trusty/varnish-4.0 amd64 Packages However, when I try to get the varnish sources, I get this error: # apt-get source varnish Reading package lists... Done Building dependency tree Reading state information... Done E: Can not find version '4.0.1-1' of package 'varnish' E: Unable to find a source package for varnish Poking around in https://repo.varnish-cache.org/ubuntu/dists/trusty/varnish-4.0/source/, I see an empty Sources.gz file, and I don't see any source packages in https://repo.varnish-cache.org/ubuntu/pool/varnish-4.0/v/varnish/ (judging by file names). I don't know a lot about debian package repository layouts, though, so maybe I am just not looking in the right places. In googling for an answer to this issue, I came across https://www.varnish-cache.org/trac/ticket/1326, which sounds similar, but it looks like that issue was just user error. Is my issue also user error? Or is something wrong with the package publishing mechanism? Thanks! From perbu at varnish-software.com Thu Jun 26 10:22:07 2014 From: perbu at varnish-software.com (Per Buer) Date: Thu, 26 Jun 2014 12:22:07 +0200 Subject: ubuntu source packages for varnish 4.0 In-Reply-To: References: Message-ID: On Thu, Jun 26, 2014 at 6:50 AM, Patrick Kaeding wrote: > Hi all, apologies if this is not the right mailing list for this question. > > I am having trouble getting the varnish 4.0 source, in ubuntu 14.04. > I have added the repo.varnish-cache.org lines to my sources.list, as > such: > Hi. Please see https://www.varnish-cache.org/installation/ubuntu Set up precise sources. We don't build specifically for trusty, but the precise packages work great. -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From itxx00 at gmail.com Thu Jun 26 10:28:50 2014 From: itxx00 at gmail.com (Gao Yongwei) Date: Thu, 26 Jun 2014 18:28:50 +0800 Subject: Does varnish4.0 libvmod_directors support dns director ? In-Reply-To: References: Message-ID: 2014-06-25 15:46 GMT+08:00 Per Buer : > Hii Gao. > > > On Wed, Jun 25, 2014 at 7:56 AM, Gao Yongwei wrote: > >> Hello, >> as in varnish3.0 there is a dns director, but it seems that it has been >> removed from 4.0 ? >> how can I get it work in 4.0? thanks. >> > > > You would need to port it to 4.0. It was removed due to lack of use and > fundamental incompatibility with IPv6. > > > Thanks ?I'll have a try, :-) -- *Regards,* *GaoYongwei* -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at ifixit.com Thu Jun 26 18:30:14 2014 From: james at ifixit.com (James Pearson) Date: Thu, 26 Jun 2014 11:30:14 -0700 Subject: RxRequest METHOD_OTHER Message-ID: <1403807158-sup-9763@geror.local> Our application uses the PATCH HTTP method. One of our customers recently complained that it wasn't working. Looking at the output of varnishlog, those requests have RxRequest set to METHOD_OTHER, rather than PATCH. We're working around it, but I was wondering if anyone had seen this sort of thing before; searching around on the net I couldn't find any mention of METHOD_OTHER other than in the Apache httpd source code. They're also talking to their firewall vendor, which is where I suspect the problem originates, but I figured it wouldn't hurt to pursue multiple channels. - James -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From phk at phk.freebsd.dk Thu Jun 26 18:34:49 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 26 Jun 2014 18:34:49 +0000 Subject: RxRequest METHOD_OTHER In-Reply-To: <1403807158-sup-9763@geror.local> References: <1403807158-sup-9763@geror.local> Message-ID: <99518.1403807689@critter.freebsd.dk> In message <1403807158-sup-9763 at geror.local>, James Pearson writes: >Looking at the output of varnishlog, those >requests have RxRequest set to METHOD_OTHER, rather than PATCH. That is what Varnish received from the client, so you should look for the trouble in that direction. >They're also talking to their firewall vendor, which is where I suspect the >problem originates, but I figured it wouldn't hurt to pursue multiple channels. That sounds like a very probable cause of trouble. Maybe ask one of them to try from his home internet connection or using his phones "access-point" facility ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From tardis74 at yahoo.com Fri Jun 27 16:56:40 2014 From: tardis74 at yahoo.com (Steve Easley) Date: Fri, 27 Jun 2014 12:56:40 -0400 Subject: varnishlog issue with ESI and ReqURL? Message-ID: <9BDECC0C-F4E6-47E1-AF19-27E1152446AB@yahoo.com> When using varnishlog (4.0.1) on CentOS I notice its not grouping identical ReqURL tags together. As far as I can tell this is related to the ReqURL tag of ESI requests being (incorrectly?) associated with the top level request. For instance here is a request to a page containing one ESI tag (varnish was restarted to start with a clean cache) $ varnishlog -g request -i "Begin,End,ReqURL,BereqURL,Link" * << Request >> 2 - Begin req 1 rxreq - ReqURL /?foo=3 - Link bereq 3 fetch - Link req 4 esi - ReqURL /esi/csstheme/ - ReqURL /esi/csstheme/ - End ** << BeReq >> 3 -- Begin bereq 2 fetch -- BereqURL /?foo=3 -- End ** << Request >> 4 -- Begin req 2 esi -- Link bereq 5 fetch -- End *** << BeReq >> 5 --- Begin bereq 4 fetch --- BereqURL /esi/csstheme/ --- End Notice the ESI request (Request 4) has no ReqURL tag. Instead 2 tags seems (erroneously?) associated with top level request (Request 2). Here is what varnishtop looks like after hitting the same URL a few times: $ varnishtop -i ReqURL 10.01 ReqURL /esi/csstheme/ 9.39 ReqURL /?foo=3 8.75 ReqURL /esi/csstheme/ Not sure why its not grouping /esi/csstheme/. Here is my (very simple) VCL: sub vcl_recv { set req.backend_hint = cluster.backend(); set req.http.Surrogate-Capability = "varnish=ESI/1.0"; } sub vcl_deliver { } sub vcl_backend_response { if (beresp.http.Surrogate-Control ~ "ESI/1.0") { unset beresp.http.Surrogate-Control; set beresp.do_esi = true; } } From hugues at betabrand.com Fri Jun 27 20:15:48 2014 From: hugues at betabrand.com (Hugues Alary) Date: Fri, 27 Jun 2014 13:15:48 -0700 Subject: Varnish 4 - VCL Compilation failed - Valid VCL In-Reply-To: <20140603121634.GD13788@immer.varnish-software.com> References: <20140603121634.GD13788@immer.varnish-software.com> Message-ID: Just saw that email... I'll check the bug tracker and if this hasn't already been reported, I'll file it. -Hugues On Tue, Jun 3, 2014 at 5:16 AM, Lasse Karstensen < lkarsten at varnish-software.com> wrote: > On Tue, May 20, 2014 at 09:57:49PM -0700, Hugues Alary wrote: > > After looking into this more, it seems to be just a small permission > > problem. > > Hi. > > This sounds like something that should be looked into. > Can you please file a bug on it in the bugtracker: > > https://www.varnish-cache.org/trac/newticket > > Thanks. > > -- > Lasse Karstensen > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bluethundr at gmail.com Sun Jun 29 00:51:06 2014 From: bluethundr at gmail.com (Tim Dunphy) Date: Sat, 28 Jun 2014 20:51:06 -0400 Subject: pages are missing in cache Message-ID: Hey guys, I've setup varnish cache on my site. And I'm using this command to determine which pages are not being cached: varnishtop -b -i TxURL It seems that the following list (and then some) are all cache misses if what I'm reading is to be believed: list length 150 6.97 TxURL /cometchat/cometchat_receive.php 3.57 TxURL / 0.55 TxURL /server-status?auto 0.44 TxURL /registration.php 0.44 TxURL /lightbox/css/lightbox.css 0.44 TxURL /css/main_style.css 0.44 TxURL /js/jquery-1.7.2.min.js 0.44 TxURL /lightbox/js/lightbox.js 0.41 TxURL /cometchat/cometchatcss.php 0.41 TxURL /cometchat/cometchatjs.php 0.41 TxURL /images/img_fireit.jpg 0.41 TxURL /images/logo.png 0.41 TxURL /images/mainmenu_selectbg.png 0.41 TxURL /images/mainmenubg.png 0.41 TxURL /images/topmenu_divider.jpg 0.41 TxURL /images/headerblockbg.jpg 0.41 TxURL /images/icon_join.png 0.41 TxURL /images/icon_comments.png 0.41 TxURL /images/bg_viewcount.png 0.41 TxURL /cometchat/swf/sound.swf 0.41 TxURL /images/icon_twitter2.png 0.41 TxURL /images/icon_write.png 0.59 TxURL /joke_details.php?o=o&joke_id=id255111111111111111111111111 I'm just hoping if I present my VCL I can get some pointers on how I can improve my cache hit ratio. I'm sorry if I'm too much of a newb at this to be able to zero in on the most relevant parts of the VCL. # man page for details on VCL syntax and semantics. # # Default backend definition. Set this to point to your content # server. # probe index { .url = "/index.php"; .timeout = 5s; .interval = 2s; .window = 5; .threshold = 3; } backend web1 { .host = "10.10.10.94"; .port = "80"; .probe = index; .connect_timeout = 30s; .first_byte_timeout = 30s; .between_bytes_timeout = 30s; .max_connections = 70; } director www client { { .backend = web1 ; .weight = 2; } } acl purge { "localhost"; "127.0.0.1"; "10.10.10.115"; } sub vcl_recv { if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { set req.http.X-Forward-For = client.ip; } } set client.identity = req.http.cookie; if (req.request != "GET"&&req.request != "HEAD") { return (pass); } if (req.url ~ "\.(png|gif|jpeg|jpg|ico|swf|css|js|txt|xml)" || req.url ~ "/static/" || req.url ~ "/userimages/" || req.url ~ ?/admin/? || req.url ~ "/rockmongo/" || req.url ~ "/images/") { unset req.http.cookie; } if (req.url ~ "\.(conf|log|old|properties|tar|war)$") { error 403 "Forbidden"; } if (req.url ~ "^/healthcheck.php") { return(pass); } if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } return (lookup); } if (req.backend.healthy) { set req.grace = 30s; } else { set req.grace = 24h; } # Do not allow outside access to cron.php or install.php. if (req.url ~ "^/(cron|install)\.php$") { # Have Varnish throw the error directly. error 404 "Page not found."; } if (req.restarts == 0) { #Try the directory first. set req.backend = www; } else if (req.restarts == 1) { # Director has failed and we will try the backend web1. set req.backend = web1; } /*else if (req.restarts == 2) { # Director has failed and we will try the backend web1. set req.backend = web2; } */ else { # All back end servers have failed. Go to error page. error 503 "Service unavailable"; } return(lookup); } sub vcl_error { if ( obj.status == 503 ) { set obj.http.Content-Type = "text/html; charset=utf-8"; set obj.http.Retry-After = "5"; synthetic {" Page Unavailable

Page Unavailable

The page you requested is temporarily unavailable.

We're redirecting you to the homepage in 5 seconds.

(Error "} + obj.status + " " + obj.response + {")
"}; } } sub vcl_hash { hash_data(req.url); if (req.http.host) { set req.http.host = regsub(req.http.host,"^beta\.jokefire\.com$", "beta.jokefire.com"); hash_data(req.http.host); } else { hash_data(server.ip); } hash_data(req.url); if (req.http.host) { set req.http.host = regsub(req.http.host,"^ref\.jokefire\.com$", "ref.jokefire.com"); hash_data(req.http.host); } else { hash_data(server.ip); } if (req.url ~ "\.(html|htm)(\?[a-z0-9=]+)?$") { set req.http.X-HASH = regsub(req.http.X-HASH, "^/.*__", "/__"); } hash_data(req.http.X-HASH); return (hash); } sub vcl_pass { return (pass); } sub vcl_pipe { set bereq.http.connection = "close"; return (pipe); } sub vcl_fetch { if ( req.url ~ "\.(png|jpg|gif)$") { set beresp.ttl = 1w; unset beresp.http.Set-Cookie; } if ( req.url ~ "\.(css|js)$") { set beresp.ttl = 1h; unset beresp.http.Set-Cookie; } if ( req.url ~ "\.(html)$") { set beresp.ttl = 15m; unset beresp.http.Set-Cookie; } if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") { /* Mark as "Hit-For-Pass" for the next 2 minutes */ set beresp.ttl = 120 s; return (hit_for_pass); } if ( req.url ~ "/static/") { set beresp.ttl = 30m; unset beresp.http.set-cookie; } if ( beresp.status>= 500 && req.request != "POST") { return(restart); } } sub vcl_deliver { if (obj.hits> 0) { set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } unset resp.http.Server; unset resp.http.Via; unset resp.http.Age; unset resp.http.X-Varnish; return (deliver); } Thanks for any advice you may have! Tim -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Sun Jun 29 09:10:46 2014 From: perbu at varnish-software.com (Per Buer) Date: Sun, 29 Jun 2014 11:10:46 +0200 Subject: pages are missing in cache In-Reply-To: References: Message-ID: Hi Tim, On Sun, Jun 29, 2014 at 2:51 AM, Tim Dunphy wrote: > Hey guys, > > > I've setup varnish cache on my site. And I'm using this command to > determine which pages are not being cached: > > varnishtop -b -i TxURL > > It seems that the following list (and then some) are all cache misses if > what I'm reading is to be believed: > > list length 150 > > 6.97 TxURL /cometchat/cometchat_receive.php > 3.57 TxURL / > (..) The details of the VCL is not really important, because we can't really see what is going on your website. The traffic pattern is the important thing. I see you are setting some TTLs based on the URL. The better solution would be to clean up the backend and have it set proper cache-control headers. Your goal should be to set the TTL for as long av possible and have the application do a PURGE call out to Varnish whenever it changes. Increasing the TTL is more or less the only way of increasing the hit rate so you either have to suffer with slower updates and get cache invalidation going. Read the cache invalidation chapter in the docs. There are probably PHP examples that would fit your framework out there. If your backend application supports it and you are on Varnish 4.0 you could try playing around IMS requests. That way Varnish can revalidate objects and extent the TTL. If you haven't you might spend some reading the user guide. Hopefully that will explain the basics and give you and understanding of how you can improve your own Varnish installation. -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From aashisn at hotmail.com Sun Jun 29 11:04:38 2014 From: aashisn at hotmail.com (Ashish Nepal) Date: Sun, 29 Jun 2014 12:04:38 +0100 Subject: backend availability In-Reply-To: References: Message-ID: Hi all, Is there any work around so that i could say -> minimum set of backendserver has to be x if not available add from reserve pool. e.g. -> i have 5 web servers -> web1,2,3,4,5 i have reserve pool of 5 web servers -> web6,7,8,9,10 director activepool round-robin { { .backend = web1; } { .backend = web2; } { .backend = web3; } { .backend = web4; } { .backend = web5; } } director reservepool round-robin { { .backend = web6; } { .backend = web7; } { .backend = web8; } { .backend = web9; } { .backend = web10; } } and if web1 is down or web1/2 is down depends upon setting of what is minimum set of boxes required. i would like director to pull one of the box from reservepool. or, should this be worth having such feature on backendpolling? Regards Ashish From imer at imer.cc Mon Jun 30 08:19:54 2014 From: imer at imer.cc (Robin) Date: Mon, 30 Jun 2014 10:19:54 +0200 Subject: RX > TX, not storing files in cache as long as it should Message-ID: <53B11DAA.6070906@imer.cc> Hello, I recently set up a varnish cache to serve as an edge server to take some traffic load off our main server. What it's supposed to do is cache ~3GB (for now, will expand later) of static files which never change (new files will get a new url). This seems to work mostly ok, looking at the interface statistics after 2 days of runtime it turns out it's actually recieving more data than its sending. One of our nodes shows (for example): RX bytes:119388151300 (111.1 GiB) TX bytes:60809550709 (56.6 GiB) The static files range from a few MB to (the biggest) 1.2GB, usually are around max. 100MB though. They are originally served using nginx with the "expires max" directive set, this is reflected by the varnish cache too: http://puu.sh/9Qq3A/6c934ad71c.png As you can see though, the Age is 0 - which means (from what I understand) the file didn't exist yet and is just being downloaded by varnish - which is not true. The file has been downloaded by that node multiple times, each time with a varying content length: http://puu.sh/9Qqds/506e73e589.png My .vcl looks like this: > backend default { > .host = "backend ip"; > .port = "80"; > } > sub vcl_fetch { > if (beresp.ttl <= 0s || > beresp.http.Set-Cookie || > beresp.http.Vary == "*") { > /* > * Mark as "Hit-For-Pass" for the next 2 minutes > */ > set beresp.ttl = 120 s; > return (hit_for_pass); > } > set beresp.do_stream = true; > return (deliver); > } And the options passed to varnish are: > DAEMON_OPTS="-a :80 \ > -T localhost:6082 \ > -f /etc/varnish/default.vcl \ > -S /etc/varnish/secret \ > -p send_timeout=86400 \ > -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,7G" Any idea why this is happening? > n_lru_nuked 0 . N LRU nuked objects Means the cache isnt filled up, which it shouldnt be anyways - so why is it downloading it over and over again and why is it aborting prematurely and not downloading the full file to cache? Trying to download the file always results in working files, so does it cancel the download to varnish if the user aborts? If so this is really disadvantageous for us, cause it may cause a large file to be downloaded multiple times before finally ending up in the cache Hope someone can help me out - Robin From carlos.abalde at gmail.com Mon Jun 30 14:29:54 2014 From: carlos.abalde at gmail.com (Carlos Abalde) Date: Mon, 30 Jun 2014 16:29:54 +0200 Subject: ESI substitutions, rewrite VMOD & friends Message-ID: <2B682E2F-0B89-44CE-8DA5-C238711AEAB7@gmail.com> Hi all, I want to do something with Varnish Cache that probably is not possible. Anyway, I'm wondering if maybe I'm missing some possible workaround, or maybe there is something planned for future releases that could be useful. At the moment this is more a theoretical than a real thing, but I'm curious about it. I think that the real use case could look a little bit weird, so let me describe something similar, still weird, but sightly simpler and easier to explain :) For example, suppose you have some kind of key-value storage integrated with Varnish Cache (using the Memcached VMOD or anything similar). During vcl_deliver, you check the incoming URL, and if it matches some pattern, you hash a canonical representation of the URL and use that value to fetch some data from the key-value storage (it could be a small JSON string containing meta-information about the URL: authors, date... whatever). Let's assume the JSON strings are inserted in the storage by some other backend component not relevant to the discussion. So, at some point during vcl_deliver you have the HTML response ready to be delivered to the client side, and you also have some JSON data fetched from Memached you need to deliver to the client side, together with the HTML. In other words, the JSON contents are designed to be consumed in the client side using some JavaScript, and you need to attach them to the HTML. I'm wondering what are the options to do something like that in Varnish: - Someone could think about using some ad hoc HTTP header to transport the JSON value to the client side. However you cannot access HTTP headers using JavaScript. That's only possible during AJAX requests. - Someone could think about using a second AJAX request to fetch the JSON contents (i.e. execute the Memached GET request later), but doing extra requests is not an option. - Someone could think about using a temporary cookie to transport the JSON value to the client side (basically, an HTTP header that is readable using JavaScript). Some JavaScript could be used to extract the JSON data from the cookie and immediately remove the cookie. Sadly this is not going to work properly in the presence of concurrency in the client side. - Someone could suggest using the Rewrite VMOD and embed in the HTML some JavaScript + JSON, but I'm not sure about how stable / reasonable is that implementation, or even if it is possible to migrate that logic to Varnish 4. - Similar to the previous idea, someone could suggest using an ESI substitution, but I'm afraid that's not implemented in Varnish (I'm assuming here that is could be possible to add the ESI placeholder in all pages generated by the backend). Any other idea? :) Cheers, ? Carlos Abalde. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: