From matt.hanley at eloquent-technologies.com Thu Mar 3 17:25:43 2016 From: matt.hanley at eloquent-technologies.com (Matt Hanley) Date: Thu, 3 Mar 2016 17:25:43 +0000 Subject: ACLs (or ACL-like behaviour) for req.url regex expressions Message-ID: <9985DC0C-C2D6-4B14-8C77-E88A9E3CCAAF@eloquent-technologies.com> Hi all, Hopefully a quick question from a new Varnish user ? I haven?t been able to find anything in the docs for this. I?m separating static content and application servers and would like to do this based on several regex expressions on req.url in vcl_recv. Is it possible to use an ACL for this? acl static_assets { ? } if (req.url ~ static_assets) { set req.backend_hint = static; return(lookup); } set req.backend_hint = default; return(pass); If this isn?t possible, is there an alternative that?s more elegant than lots of `if` statements? Many thanks in advance Matt Hanley -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkarsten at varnish-software.com Fri Mar 4 13:30:40 2016 From: lkarsten at varnish-software.com (Lasse Karstensen) Date: Fri, 4 Mar 2016 14:30:40 +0100 Subject: Varnish modules package released (Was: Re: Update on vmods and packaging) In-Reply-To: <20151202144041.GB19950@immer.varnish-software.com> References: <20151202144041.GB19950@immer.varnish-software.com> Message-ID: <20160304133039.GA32277@immer.varnish-software.com> [crosspost to both -dev and -misc] On Wed, Dec 02, 2015 at 03:40:42PM +0100, Lasse Karstensen wrote: > Varnish Software (VS) is writing and maintaining a fair amount of vmods for > Varnish Cache. This includes major ones like vmod-header, which can be > said to be borderline core Varnish functionality. > We will be doing some changes on how these are maintained and distributed. [snip] > Further down the line it is likely that we will be combining the simple > vmods (with no third party/library dependencies) into a single > collection source package. Updates on that will come later. Hi all. We have now made an initial release of this collection. Tarball distribution is here: http://files.varnish-software.com/vmod/varnish-modules-0.9.0.tar.gz The source trees and distribution of the following vmods have been merged into this single project: cookie, vsthrottle, header, saintmode, softpurge, tcp, var, xkey. More vmods will be added at a later stage. This means that starting from today the authoritative location for development of these vmods is on: https://github.com/varnish/varnish-modules Releases of this tree will be a tarball with a proper changelog, suitable for packaging into rpm/deb/* packages by motivated third parties. It is my hope that this will significantly simplify the installation of these vmods, both for users installing from source, and for users installing pre-packaged versions from third-party yum/apt/* sources. 3.0 and 4.0 versions of the old vmods will be archived and made available in a similar manner. When that has been done, the old github projects for the vmods will be retired. -- Lasse Karstensen Varnish Software AS (VS hat) From bluethundr at gmail.com Sun Mar 6 04:40:40 2016 From: bluethundr at gmail.com (Tim Dunphy) Date: Sat, 5 Mar 2016 23:40:40 -0500 Subject: load balancing multiple sites with varnish Message-ID: Hey guys, I have a varnish config that's been working pretty nicely for the last year. But recently I had to build 3 Kibana (that's the logstash UI if you're not aware) nodes on aws. And I wanted to load balance them and perhaps do some caching on them. However when I tried to add the new kibana hosts to the config, instead of showing the kibana interface, it shows the default web page on my first apache server from the previous load balancing pool. That belongs to a host called 'web1' in the config. Loading the 3 kibana nodes individually shows that it's loading and running properly. Here's what I've done to try to get this to work: I added the 3 kibana nodes as logs1, 2 and 3 to the config: backend logs1 { .host = "52.xx.xxx.180"; .port = "80"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; .max_connections = 800; .probe = healthcheck; } backend logs2 { .host = "52.xx.xx.78"; .port = "80"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; .max_connections = 800; .probe = healthcheck; } backend logs3 { .host = "52.xx.xxx.209"; .port = "80"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; .max_connections = 800; .probe = healthcheck; } And setup a new load balancing section: sub vcl_init { new wiki = directors.round_robin(); wiki.add_backend(web1); ## <-- this is what shows up at logs.exampledomain.com wiki.add_backend(web2); wiki.add_backend(web3); } *sub vcl_init {* * new logs = directors.round_robin();* * logs.add_backend(logs1);* * logs.add_backend(logs2);* * logs.add_backend(logs3);* *}* In my vcl_recv I have the following: sub vcl_recv { # Authenticate the user using Apache basic auth if ( req.http.host ~ "^wiki\.exampledomain\.com$") { set req.backend_hint = wiki.backend(); if (!basicauth.match("/etc/httpd/auth", req.http.Authorization)) { return(synth(401, "Authentication required")); } } else if ( req.http.host ~ "^beta\.exampledomain\.com$" ) { set req.http.backend_hint = wiki.backend(); } if ( req.http.host ~ "^logs\.exampledomain\.com$") { set req.backend_hint = logs.backend(); } # Authenticate the user using Apache basic auth if ( req.http.host ~ "^wiki\.exampledomain\.com$") { set req.backend_hint = wiki.backend(); if (!basicauth.match("/etc/httpd/auth", req.http.Authorization)) { return(synth(401, "Authentication required")); } } else if ( req.http.host ~ "^beta\.exampledomain\.com$" ) { set req.http.backend_hint = wiki.backend(); } if ( req.http.host ~ "^logs\.exampledomain\.com$") { set req.backend_hint = logs.backend(); } ..... more stuff return(hash); } And instead of seeing this at logs.exampledomain.com: I see a web page that looks like this: How can I get varnish to serve the 'logs' load balancing pool correctly? Here's my whole vcl_recv in case that helps : vcl 4.0; import std; import directors; import basicauth; probe healthcheck { .url = "/healthcheck.php"; .timeout =1m; .interval = 1s; .window = 10; .threshold = 8; } backend default { .host = "107.xxx.xx.174"; .port = "80"; .connect_timeout = 1m; .first_byte_timeout = 1m; .between_bytes_timeout = 2s; .max_connections = 800; .probe = healthcheck; } backend web1 { .host = "107.xxx.xx.174"; .port = "80"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; .max_connections = 800; .probe = healthcheck; } backend web2 { .host = "107.xxx.xxx.53"; .port = "80"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; .max_connections = 800; .probe = healthcheck; } backend web3 { .host = "107.xxx.xx.38"; .port = "80"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; .max_connections = 800; .probe = healthcheck; } backend logs1 { .host = "52.xx.xxx.180"; .port = "80"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; .max_connections = 800; .probe = healthcheck; } backend logs2 { .host = "52.xx.xx.78"; .port = "80"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; .max_connections = 800; .probe = healthcheck; } backend logs3 { .host = "52.xx.xxx.209"; .port = "80"; .connect_timeout = 1s; .first_byte_timeout = 5s; .between_bytes_timeout = 2s; .max_connections = 800; .probe = healthcheck; } # access control list for "purge": open to only localhost and other local nodes acl purge { "127.0.0.1"; } sub vcl_init { new wiki = directors.round_robin(); wiki.add_backend(web1); wiki.add_backend(web2); wiki.add_backend(web3); } sub vcl_init { new logs = directors.round_robin(); logs.add_backend(logs1); logs.add_backend(logs2); logs.add_backend(logs3); } # vcl_recv is called whenever a request is received sub vcl_recv { # Authenticate the user using Apache basic auth if ( req.http.host ~ "^wiki\.exampledomain\.com$") { set req.backend_hint = wiki.backend(); if (!basicauth.match("/etc/httpd/auth", req.http.Authorization)) { return(synth(401, "Authentication required")); } } else if ( req.http.host ~ "^beta\.exampledomain\.com$" ) { set req.http.backend_hint = wiki.backend(); } if ( req.http.host ~ "^logs\.exampledomain\.com$") { set req.backend_hint = logs.backend(); } # Remove has_js and CloudFlare/Google Analytics __* cookies. set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", ""); # Remove a ";" prefix, if present. set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); set req.http.X-Forwarded-For = client.ip; set req.backend_hint= default; # This uses the ACL action called "purge". Basically if a request to # PURGE the cache comes from anywhere other than localhost, ignore it. if (req.method == "PURGE") {if (!client.ip ~ purge) {return(synth(405,"Not allowed."));} return(hash);} # Allows editing the wiki if (req.url ~ "&action=submit($|/)" ) { return(pass); } # Pass any requests that Varnish does not understand straight to the backend. if (req.method != "GET" && req.method != "HEAD" && req.method != "PUT" && req.method != "POST" && req.method != "TRACE" && req.method != "OPTIONS" && req.method != "DELETE") {return(pipe);} /* Non-RFC2616 or CONNECT which is weird. */ # Pass anything other than GET and HEAD directly. if (req.method != "GET" && req.method != "HEAD") {return(pass);} /* We only deal with GET and HEAD by default */ # Make images, etc cachable by unsetting cookie in request if (req.url ~ "^/images" ) { unset req.http.cookie; } # Pass requests from logged-in users directly. #if (req.http.Authorization || req.http.Cookie) if (req.http.Cookie) {return(pass);} /* Not cacheable by default */ # Pass any requests with the "If-None-Match" header directly. if (req.http.If-None-Match) {return(pass);} # Force lookup if the request is a no-cache request from the client. if (req.http.Cache-Control ~ "no-cache") {ban(req.url);} # normalize Accept-Encoding to reduce vary if (req.http.Accept-Encoding) { if (req.http.User-Agent ~ "MSIE 6") { unset req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } } # Set the X-Forwarded-For header so the backend can see the original # IP address. If one is already set by an upstream proxy, we'll just re-use that. if (client.ip ~ purge && req.http.X-Forwarded-For) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For; } else { set req.http.X-Forwarded-For = regsub(client.ip, ":.*", ""); } return(hash); } Thanks, Tim ? -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2016-03-05 23.35.25.png Type: image/png Size: 150608 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2016-03-05 23.37.29.png Type: image/png Size: 19480 bytes Desc: not available URL: From perbu at varnish-software.com Sun Mar 6 08:23:50 2016 From: perbu at varnish-software.com (Per Buer) Date: Sun, 6 Mar 2016 09:23:50 +0100 Subject: load balancing multiple sites with varnish In-Reply-To: References: Message-ID: Hi, On Sun, Mar 6, 2016 at 5:40 AM, Tim Dunphy wrote: > (..) > However when I tried to add the new kibana hosts to the config, instead of > showing the kibana interface, it shows the default web page on my first > apache server from the previous load balancing pool. That belongs to a host > called 'web1' in the config. > Common mistake. You've misconfigured the virtual hosts on the backends. The virtual host must match what Varnish sends to your backends. So if you are accessing your Varnish as "logs.foo.com" then the virtual host definition in Apache must be for that virtual host. I did a writeup on the topic a couple of years back. You might find it helpful: http://info.varnish-software.com/blog/getting-virtual-hosts-right-varnish-cache -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Mar 7 10:54:26 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 7 Mar 2016 11:54:26 +0100 Subject: ACLs (or ACL-like behaviour) for req.url regex expressions In-Reply-To: <9985DC0C-C2D6-4B14-8C77-E88A9E3CCAAF@eloquent-technologies.com> References: <9985DC0C-C2D6-4B14-8C77-E88A9E3CCAAF@eloquent-technologies.com> Message-ID: Hi Matt, On Thu, Mar 3, 2016 at 6:25 PM, Matt Hanley wrote: > Is it possible to use an ACL for this? > > acl static_assets { > ? > } Short answer, no. > If this isn?t possible, is there an alternative that?s more elegant than > lots of `if` statements? There's none AFAICT but you could try to get in touch with developers from UPLEX, they have an regex VMOD [1] that could be a nice fit for what you are trying to do. One could imagine something like: new static_assets = re.multi_regex( ... ); if (static_assets.match(req.url)) { set req.backend_hint = static; return(lookup); } Cheers, Dridi [1] https://code.uplex.de/uplex-varnish/libvmod-re From ruben at varnish-software.com Wed Mar 9 21:29:24 2016 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Wed, 9 Mar 2016 22:29:24 +0100 Subject: Varnish User Survey 2016 Message-ID: Hello, As PHK wrote recently to this list (yes, both), the Varnish project is undergoing a few adjustments and so it is time for us to get more knowledge on how you, our users, are currently using the software and the project's resources. We have a survey for that and it is now up: * https://www.varnish-cache.org/survey2016 Thank you in advance for helping us make Varnish better for you! Best regards, -- *Rub?n Romero*Community Manager Varnish Software Group Cell: +47 95964088 / Office: +47 21989260 Skype, Twitter & IRC: ruben_varnish We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From garry at 4ustralia.com Sun Mar 13 15:03:38 2016 From: garry at 4ustralia.com (Garry Hill) Date: Mon, 14 Mar 2016 00:33:38 +0930 Subject: VSF Rule for general spammy hackers Message-ID: <034c01d17d39$8761b370$96251a50$@com> Hi Varnish has made an incredible difference to my project that was struggling under heavy load I want to try to understand how to make some Custom rules to put into Varnish Security Firewall I still have loosers scanning my web sites with like http://website.com/index.php?option=com_user &task=register that are having a random go at breaking into Joomla or Worpress that I have neither of so they just waste my bandwidth I can't find much info on VSF out there and the forum is closed to posting can you help me? As I understand it I would make a custom rule like /etc/varnish/security/rules/myrules.vcf If I wanted to detect that Signature, Id need to make something like # File myrules.vcf sub vcl_recv { if (the URL requested contains "?option=com_user&task=register") { Tell him to bugger off } } Can you please show me how to do something like that? Thankyou Garry Hill Alice Springs, Outback Central Australia http://australiancommunitynetwork.com.au -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish-cache at otoh.org Sun Mar 13 17:46:00 2016 From: varnish-cache at otoh.org (Paul Armstrong) Date: Sun, 13 Mar 2016 17:46:00 +0000 Subject: VSF Rule for general spammy hackers In-Reply-To: <034c01d17d39$8761b370$96251a50$@com> References: <034c01d17d39$8761b370$96251a50$@com> Message-ID: <20160313174600.GY47547@suricate.otoh.org> At 2016-03-14T00:33+0930, Garry Hill wrote: > I still have loosers scanning my web sites with like > http://website.com/index.php?option=com_user > &task=register > > that are having a random go at breaking into Joomla or Worpress that I have > neither of so they just waste my bandwidth > > I can't find much info on VSF out there and the forum is closed to posting > can you help me? > sub vcl_recv { > > if (the URL requested contains "?option=com_user&task=register") { > Tell him to bugger off } > } > > Can you please show me how to do something like that? G'Day Garry. I've not used VSF, but it looks like it's just VCL. This will almost certainly work in the VSF ruleset, but if it chokes, just put it in your main ruleset. sub vcl_recv { if (req.url ~ "\?option=com_user&task=register") { return (synth(403)); } } If the synthetic 403 is too large and you just want to return a minimal page for such attacks: sub vcl_synth { if (req.url ~ "\?option=com_user&task=register") { synthetic(""); unset resp.http.X-Varnish; return(deliver); } } Paul From shvabauer at arsenal-d.uz Fri Mar 18 13:25:32 2016 From: shvabauer at arsenal-d.uz (=?UTF-8?B?0KjQstCw0LHQsNGD0Y3RgCDQn9Cw0LLQtdC7?=) Date: Fri, 18 Mar 2016 18:25:32 +0500 Subject: does it possible with varnish and how to do it Message-ID: <56EC01CC.7010601@arsenal-d.uz> Good day team, I'm using varnish in our projects already long time, so thank you very much for this great technology! Let me explain 1 thing I can't realize already 2 weeks. so, here is task. Please take a look, we are using some sort of security in links, so every client has own generated links. An example: client1: /2ca6d77ff25927870b135c389c4692ef/book/ a7a5b1ce35c454dbc4abf6425739e9cd890cd68c640d3747947bc502d9da9e2a5aa93c3069f29630/thing1.txt client2: /2ca6d77ff25927870b135c389c4692ef/book/ea0c89a5895321aa7af058492d7ba33e0395bc04efc490e4ad1d4c29f0788446b9d0fee67a606s4t/thing1.txt client3: /2ca6d77ff25927870b135c389c4692ef/book/ea0c89a5895320d96d798f0ec4dc13b69f6c342d2f5cf86d09374d141c9119515003b1abb2351d4f/thing1.txt So, based on clean URL we can't build cache, that's why we have to modify urls to cache them like just: /2ca6d77ff25927870b135c389c4692ef/thing1.txt But get proper response possible if URL like: /magazine/ea0c89a5895320d96d798f0ec4dc13b69f6c342d2f5cf86d09374d141c9119515003b1abb2351d4f/thing1.txt So, how is it possible to reach with varnish, to lookup objects by custom key, fetch by custom url and put to cache by custom key? I already created URL rewriting and it works properly, varnish fetching content and sends to client, but not caching it at all. Also, I need a way to pause some clients requesting same object till first will not reach it, so needs to make a queue. Here is my current config I'm using (please remove all things not important): sub vcl_recv { set req.backend_hint = vdir.backend(); # send all traffic to the vdir director set req.url = std.querysort(req.url); unset req.http.cookie; if (req.method == "GET" && (req.url ~ "book")) { std.log("we are passing it"); return (pass); } std.log("we are looking for the cache "); return (hash); } sub vcl_pipe { #std.log("piped url:::"+bereq.url); #if (bereq.url ~ "/origin/") { # set bereq.url = regsub(req.url, "(.*)/(.*)/(.*)/(.*)$", "\1/\4"); # std.log("piped::"+bereq.url); # } set bereq.http.Connection = "Close"; return (pipe); } # The data on which the hashing will take place sub vcl_hash { ###### here we should modify url to try to return from cache if (req.url ~ "/book/$") { set req.http.cacheurl = regsub(req.url, "(.*)/(.*)/(.*)/(.*)$", "\1/\4"); std.log("hash rewrite was:" + req.url); std.log("become ::: "+req.http.cacheurl); } else { set req.http.cacheurl = req.url; } hash_data(req.http.cacheurl); if (req.http.host) { hash_data(req.http.host); } else { hash_data(server.ip); } } sub vcl_backend_fetch { if (bereq.url ~ "/book/") { set bereq.url = regsub(bereq.url, "(.*)/(.*)/(.*)/(.*)$", "/magazine/\3/\4"); std.log("backend fetch::"+bereq.url); } return (fetch); } sub vcl_hit { # Called when a cache lookup is successful. if (obj.ttl >= 0s) { # A pure unadultered hit, deliver it return (deliver); } # if (!std.healthy(req.backend_hint) && (obj.ttl + obj.grace > 0s)) { # return (deliver); # } else { # return (fetch); # } # We have no fresh fish. Lets look at the stale ones. if (std.healthy(req.backend_hint)) { # Backend is healthy. Limit age to 10s. if (obj.ttl + 10s > 0s) { #set req.http.grace = "normal(limited)"; return (deliver); } else { # No candidate for grace. Fetch a fresh object. return(fetch); } } else { # backend is sick - use full grace if (obj.ttl + obj.grace > 0s) { #set req.http.grace = "full"; return (deliver); } else { # no graced object. return (fetch); } } # fetch & deliver once we get the result return (fetch); # Dead code, keep as a safeguard } sub vcl_miss { std.log("vcl_miss "+req.url); return (fetch); } # Handle the HTTP request coming from our backend sub vcl_backend_response { set beresp.grace = 20m; # Called after the response headers has been successfully retrieved from the backend. unset beresp.http.set-cookie; # Set 2min cache if unset for static files set beresp.ttl = 1200s; std.log("response url:::"+bereq.url); ###### here we should rewrite url to send it to wowza then if (bereq.url ~ "/live/") { set bereq.url = regsub(bereq.url, "(.*)/(.*)/(.*)/(.*)$", "\1/\4"); } # Allow stale content, in case the backend goes down. # make Varnish keep all objects for 6 hours beyond their TTL return (deliver); } sub vcl_deliver { if (req.url ~ "/book/") { set req.url = regsub(req.url, "(.*)/(.*)/(.*)/(.*)$", "\1/\4"); std.log("delivering :: "+req.url); } if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } # Please note that obj.hits behaviour changed in 4.0, now it counts per objecthead, not per object # and obj.hits may not be reset in some cases where bans are in use. See bug 1492 for details. # So take hits with a grain of salt set resp.http.X-Cache-Hits = obj.hits; # Remove some headers: PHP version unset resp.http.X-Powered-By; # Remove some headers: Apache version & OS unset resp.http.Server; unset resp.http.X-Drupal-Cache; unset resp.http.X-Varnish; unset resp.http.Via; unset resp.http.Link; unset resp.http.X-Generator; return (deliver); } sub vcl_purge { # Only handle actual PURGE HTTP methods, everything else is discarded if (req.method != "PURGE") { # restart request set req.http.X-Purge = "Yes"; return(restart); } } sub vcl_synth { if (resp.status == 720) { # We use this special error status 720 to force redirects with 301 (permanent) redirects # To use this, call the following from anywhere in vcl_recv: return (synth(720, "http://host/new.html")); set resp.http.Location = resp.reason; set resp.status = 301; return (deliver); } elseif (resp.status == 721) { # And we use error status 721 to force redirects with a 302 (temporary) redirect # To use this, call the following from anywhere in vcl_recv: return (synth(720, "http://host/new.html")); set resp.http.Location = resp.reason; set resp.status = 302; return (deliver); } return (deliver); } sub vcl_fini { # Called when VCL is discarded only after all requests have exited the VCL. # Typically used to clean up VMODs. return (ok); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From shvabauer at arsenal-d.uz Fri Mar 18 14:15:42 2016 From: shvabauer at arsenal-d.uz (=?UTF-8?B?0KjQstCw0LHQsNGD0Y3RgCDQn9Cw0LLQtdC7?=) Date: Fri, 18 Mar 2016 19:15:42 +0500 Subject: formatted:: does it possible with varnish and how to do it Message-ID: <56EC0D8E.7070102@arsenal-d.uz> Good day team, I'm using varnish in our projects already long time, so thank you very much for this great technology! Let me explain 1 thing I can't realize already 2 weeks. so, here is task. Please take a look, we are using some sort of security in links, so every client has own generated links. An example: client1: /2ca6d77ff25927870b135c389c4692ef/book/ a7a5b1ce35c454dbc4abf6425739e9cd890cd68c640d3747947bc502d9da9e2a5aa93c3069f29630/thing1.txt client2: /2ca6d77ff25927870b135c389c4692ef/book/ea0c89a5895321aa7af058492d7ba33e0395bc04efc490e4ad1d4c29f0788446b9d0fee67a606s4t/thing1.txt client3: /2ca6d77ff25927870b135c389c4692ef/book/ea0c89a5895320d96d798f0ec4dc13b69f6c342d2f5cf86d09374d141c9119515003b1abb2351d4f/thing1.txt So, based on clean URL we can't build cache, that's why we have to modify urls to cache them like just: /2ca6d77ff25927870b135c389c4692ef/thing1.txt But get proper response possible if URL like: /magazine/ea0c89a5895320d96d798f0ec4dc13b69f6c342d2f5cf86d09374d141c9119515003b1abb2351d4f/thing1.txt So, how is it possible to reach with varnish, to lookup objects by custom key, fetch by custom url and put to cache by custom key? I already created URL rewriting and it works properly, varnish fetching content and sends to client, but not caching it at all. Also, I need a way to pause some clients requesting same object till first will not reach it, so needs to make a queue. Here is my current config I'm using (please remove all things not important): sub vcl_recv { set req.backend_hint = vdir.backend(); # send all traffic to the vdir director set req.url = std.querysort(req.url); unset req.http.cookie; if (req.method == "GET" && (req.url ~ "book")) { std.log("we are passing it"); return (pass); } std.log("we are looking for the cache "); return (hash); } sub vcl_pipe { #std.log("piped url:::"+bereq.url); #if (bereq.url ~ "/origin/") { # set bereq.url = regsub(req.url, "(.*)/(.*)/(.*)/(.*)$", "\1/\4"); # std.log("piped::"+bereq.url); # } set bereq.http.Connection = "Close"; return (pipe); } # The data on which the hashing will take place sub vcl_hash { ###### here we should modify url to try to return from cache if (req.url ~ "/book/$") { set req.http.cacheurl = regsub(req.url, "(.*)/(.*)/(.*)/(.*)$", "\1/\4"); std.log("hash rewrite was:" + req.url); std.log("become ::: "+req.http.cacheurl); } else { set req.http.cacheurl = req.url; } hash_data(req.http.cacheurl); if (req.http.host) { hash_data(req.http.host); } else { hash_data(server.ip); } } sub vcl_backend_fetch { if (bereq.url ~ "/book/") { set bereq.url = regsub(bereq.url, "(.*)/(.*)/(.*)/(.*)$", "/magazine/\3/\4"); std.log("backend fetch::"+bereq.url); } return (fetch); } sub vcl_hit { # Called when a cache lookup is successful. if (obj.ttl >= 0s) { # A pure unadultered hit, deliver it return (deliver); } # if (!std.healthy(req.backend_hint) && (obj.ttl + obj.grace > 0s)) { # return (deliver); # } else { # return (fetch); # } # We have no fresh fish. Lets look at the stale ones. if (std.healthy(req.backend_hint)) { # Backend is healthy. Limit age to 10s. if (obj.ttl + 10s > 0s) { #set req.http.grace = "normal(limited)"; return (deliver); } else { # No candidate for grace. Fetch a fresh object. return(fetch); } } else { # backend is sick - use full grace if (obj.ttl + obj.grace > 0s) { #set req.http.grace = "full"; return (deliver); } else { # no graced object. return (fetch); } } # fetch & deliver once we get the result return (fetch); # Dead code, keep as a safeguard } sub vcl_miss { std.log("vcl_miss "+req.url); return (fetch); } # Handle the HTTP request coming from our backend sub vcl_backend_response { set beresp.grace = 20m; # Called after the response headers has been successfully retrieved from the backend. unset beresp.http.set-cookie; # Set 2min cache if unset for static files set beresp.ttl = 1200s; std.log("response url:::"+bereq.url); ###### here we should rewrite url to send it to wowza then if (bereq.url ~ "/live/") { set bereq.url = regsub(bereq.url, "(.*)/(.*)/(.*)/(.*)$", "\1/\4"); } # Allow stale content, in case the backend goes down. # make Varnish keep all objects for 6 hours beyond their TTL return (deliver); } sub vcl_deliver { if (req.url ~ "/book/") { set req.url = regsub(req.url, "(.*)/(.*)/(.*)/(.*)$", "\1/\4"); std.log("delivering :: "+req.url); } if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } # Please note that obj.hits behaviour changed in 4.0, now it counts per objecthead, not per object # and obj.hits may not be reset in some cases where bans are in use. See bug 1492 for details. # So take hits with a grain of salt set resp.http.X-Cache-Hits = obj.hits; # Remove some headers: PHP version unset resp.http.X-Powered-By; # Remove some headers: Apache version & OS unset resp.http.Server; unset resp.http.X-Drupal-Cache; unset resp.http.X-Varnish; unset resp.http.Via; unset resp.http.Link; unset resp.http.X-Generator; return (deliver); } sub vcl_purge { # Only handle actual PURGE HTTP methods, everything else is discarded if (req.method != "PURGE") { # restart request set req.http.X-Purge = "Yes"; return(restart); } } sub vcl_synth { if (resp.status == 720) { # We use this special error status 720 to force redirects with 301 (permanent) redirects # To use this, call the following from anywhere in vcl_recv: return (synth(720, "http://host/new.html")); set resp.http.Location = resp.reason; set resp.status = 301; return (deliver); } elseif (resp.status == 721) { # And we use error status 721 to force redirects with a 302 (temporary) redirect # To use this, call the following from anywhere in vcl_recv: return (synth(720, "http://host/new.html")); set resp.http.Location = resp.reason; set resp.status = 302; return (deliver); } return (deliver); } sub vcl_fini { # Called when VCL is discarded only after all requests have exited the VCL. # Typically used to clean up VMODs. return (ok); } From yianniska at gmail.com Mon Mar 21 19:43:24 2016 From: yianniska at gmail.com (Yiannis Karayiannidis) Date: Mon, 21 Mar 2016 21:43:24 +0200 Subject: Child died on request Message-ID: Hello, I would like to ask your for help for a really strange case that we have.. We have migrated from varnish 3.07 to varnish 4.03 few months ago. The last few days. we have a strange problem. Child died on request. The backends was unreachable at that time (we are using a hash director). We did get a panic message like : varnishd[30906]: Child (6755) Panic message:#012Assert error in VDI_GetFd(), cache/cache_dir.c line 111:#012 Condition((d) != NULL) not true. #012thread = (cache-worker)#012version = varnish-4.0.3 revision b8c4a34#012ident = Linux,3.10.0-327.10.1.el7.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012 Backtrace:#012 0x7eff9489f6b3: /usr/sbin/varnishd(+0x3e6b3) [0x7eff9489f6b3]#012 0x7eff948813c4: /usr/sbin/varnishd(VDI_GetFd+0xe4) [0x7eff948813c4]#012 0x7eff948a021b: /usr/sbin/varnishd(PipeRequest+0xcb) [0x7eff948a021b]#012 0x7eff948a432c: /usr/sbin/varnishd(CNT_Request+0xaec) [0x7eff948a432c] #012 0x7eff94898f2b: /usr/sbin/varnishd(HTTP1_Session+0x7eb) [0x7eff94898f2b]#012 0x7eff948a8747: /usr/sbin/varnishd(+0x47747) [0x7eff948a8747]#012 0x7eff948a9878: /usr/sbin/varnishd(SES_pool_accept_task+0x2b8) [0x7eff948a9878]#012 0x7eff948a291d: /usr/sbin/varnishd(Pool_Work_Thread+0x39d) [0x7eff948a291d]#012 0x7eff948b7bc2: /usr/sbin/varnishd(+0x56bc2) [0x7eff948b7bc2]#012 0x7eff93029dc5: /lib64/libpthread.so.0(+0x7dc5) [0x7eff93029dc5]#012req = 0x7eff64012020 {#012 sp = 0x7eff6281ae20, vxid = 1075806210, step = R_STP_PIPE,#012 req_body = R_BODY_PRESENT,#012 restarts = 0, esi_level = 0, #012 sp = 0x7eff6281ae20 {#012 fd = 122, vxid = 2064385,#012 client = 172.16.50.169 46012,#012 step = S_STP_WORKING,#012 },#012 worker = 0x7eff36b90c50 {#012 ws = 0x7eff36b90e70 {#012 id = "wrk",#012 {s,f,r,e} = {0x7eff36b90450,0x7eff36b90450,(nil),+2048},#012 }, #012 VCL::method = 0x0,#012 VCL::return = pipe,#012 },#012 ws = 0x7eff640121b8 {#012 id = "req",#012 {s,f,r,e} = {0x7eff64014010,+1480,(nil),+57360},#012 }, #012 http[req] = {#012 ws = 0x7eff640121b8[req]#012 "POST",#012 "/api/login",#012 "HTTP/1.1",#012 "Host: www.xxx.com",#012 "X-Real-IP: 188.73.254.105",#012 "X-Forwarded-Proto: https", #012 "Connection: close",#012 "Content-Length: 39",#012 "Accept-Encoding: gzip",#012 "CF-IPCountry: GR",#012 "CF-RAY: 28735bfd70642342-FRA",#012 "CF-Visitor: {"scheme":"https"}",#012 "accept: */*",#012 "accept-language: en-GR;q=1.0, el-GR;q=0.9",#012 "content-type: application/json",#012 "cookie: __cfduid=dbe3f7943ad953c0c32fed9c23d5ec53e1456395183; _ga=GA1.2.1866397751.1458056300; exauth=E1775E71AA7578E50ECCBB9172FEAF829C4BA0055456FBDFA4957FC670D753B0A5C63ADC9A28455744F5E6EF378B2F94D2BB62614DA5745938AC9E50B794FF294004BA0B2CCB7374BB04DC6CDDE90D49A84631723D9791AC71BCCBB29CFB854664E0B283C9B3873A9D754874BEF4A2B8043F01A713A8FD7754025F585B98EB4E4275D1E04012206BE1115ED491C3DBF3D2D34210DFC24235A21E125901CAC6E3EF6CD6B61B49F1EFF1ACE8C083B4C6EDF226682AF6AD4F2691752FA39C253AF1A06CF49FE01F99316540025A604FDB31A0F7F2F8449249CA92BA292D64AF6149123A2DAA2F034EB458E78A62CBF9D36F4FD0B9FDAF55430562595789B24AE667BE0C8EACA4711AD79BD2B2EE81670692C140D19A3B1F18D11CBC4A2CB9801ACF4A90F81DA9E0E0E6246522B4F72CD3B52CA283DB30B34D4574AC24AAC3452FE9",#012 "user-agent: xxx/com.bbb.ccc (37; OS Version 9.0.2 (Build 13A452))",#012 "x-requested-with: XMLHttpRequest",#012 "CF-Connecting-IP: 188.73.254.105",#012 "X-Forwarded-For: 188.73.254.105, 188.73.254.105, 173.16.50.169",#012 "X-defHash: /login + www.xxx.com",#012 },#012 vcl = {#012 srcname = {#012 "input",#012 "Builtin",#012 "origins-xxx.vcl",#012 "origins-xxx.vcl",#012 "origins-others.vcl",#012 "origins-staging.vcl",#012 "banlist.vcl",#012 },#012 },#012}, lin-varnish03 varnishd[30906]: Child cleanup complete lin-varnish03 varnishd[30906]: child (13007) Started Thanks in advance Yiannis -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue Mar 22 09:07:02 2016 From: perbu at varnish-software.com (Per Buer) Date: Tue, 22 Mar 2016 05:07:02 -0400 Subject: Child died on request In-Reply-To: References: Message-ID: On Mon, Mar 21, 2016 at 3:43 PM, Yiannis Karayiannidis wrote: > Hello, > I would like to ask your for help for a really strange case that we have.. > We have migrated from varnish 3.07 to varnish 4.03 few months ago. The > last few days. > we have a strange problem. > Child died on request. The backends was unreachable at that time (we are > using a hash director). > We did get a panic message like : > Look at this: https://www.varnish-cache.org/trac/ticket/1815 Looks like you have no backend and you're trying to pipe. -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue Mar 22 09:14:14 2016 From: perbu at varnish-software.com (Per Buer) Date: Tue, 22 Mar 2016 05:14:14 -0400 Subject: formatted:: does it possible with varnish and how to do it In-Reply-To: <56EC0D8E.7070102@arsenal-d.uz> References: <56EC0D8E.7070102@arsenal-d.uz> Message-ID: On Fri, Mar 18, 2016 at 10:15 AM, ???????? ????? wrote: > Good day team, > I'm using varnish in our projects already long time, so thank you very > much for this great technology! > Let me explain 1 thing I can't realize already 2 weeks. > > so, here is task. Please take a look, we are using some sort of security > in links, so every client has own generated links. > (..) > > # The data on which the hashing will take place > sub vcl_hash { > ###### here we should modify url to try to return from cache > if (req.url ~ "/book/$") { > set req.http.cacheurl = regsub(req.url, "(.*)/(.*)/(.*)/(.*)$", > "\1/\4"); > std.log("hash rewrite was:" + req.url); > std.log("become ::: "+req.http.cacheurl); > } else { > set req.http.cacheurl = req.url; > } > hash_data(req.http.cacheurl); > > if (req.http.host) { > hash_data(req.http.host); > } else { > hash_data(server.ip); > } > } > Here you hand over control to the builtin VCL. The builtin VCL will add req.url to the hash, which will more or less disable the caching. If you return(lookup) in vcl_hash you will stop processing. -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From shvabauer at arsenal-d.uz Tue Mar 22 09:18:12 2016 From: shvabauer at arsenal-d.uz (=?UTF-8?B?0KjQstCw0LHQsNGD0Y3RgCDQn9Cw0LLQtdC7?=) Date: Tue, 22 Mar 2016 14:18:12 +0500 Subject: formatted:: does it possible with varnish and how to do it In-Reply-To: References: <56EC0D8E.7070102@arsenal-d.uz> Message-ID: <56F10DD4.1040201@arsenal-d.uz> Good day Per, yes, I added return(lookup) in vcl_hash and an issue has gone! Thanks a lot for support and for great community! 22.03.2016 14:14, Per Buer ?????: > > > On Fri, Mar 18, 2016 at 10:15 AM, ???????? ????? > > wrote: > > Good day team, > I'm using varnish in our projects already long time, so thank you > very much for this great technology! > Let me explain 1 thing I can't realize already 2 weeks. > > so, here is task. Please take a look, we are using some sort of > security in links, so every client has own generated links. > > > > (..) > > > # The data on which the hashing will take place > sub vcl_hash { > ###### here we should modify url to try to return from cache > if (req.url ~ "/book/$") { > set req.http.cacheurl = regsub(req.url, > "(.*)/(.*)/(.*)/(.*)$", "\1/\4"); > std.log("hash rewrite was:" + req.url); > std.log("become ::: "+req.http.cacheurl); > } else { > set req.http.cacheurl = req.url; > } > hash_data(req.http.cacheurl); > > if (req.http.host) { > hash_data(req.http.host); > } else { > hash_data(server.ip); > } > } > > > Here you hand over control to the builtin VCL. The builtin VCL will > add req.url to the hash, which will more or less disable the caching. > > If you return(lookup) in vcl_hash you will stop processing. > > -- > *Per Buer* > CTO | Varnish Software AS > Cell: +47 95839117 > We Make Websites Fly! > www.varnish-software.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Mar 22 09:37:22 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 22 Mar 2016 10:37:22 +0100 Subject: Child died on request In-Reply-To: References: Message-ID: On Tue, Mar 22, 2016 at 10:07 AM, Per Buer wrote: > > Look at this: https://www.varnish-cache.org/trac/ticket/1815 > > Looks like you have no backend and you're trying to pipe. Looks like work for me. I'll try to get it fixed this week. Dridi From yianniska at gmail.com Tue Mar 22 09:48:19 2016 From: yianniska at gmail.com (Yiannis Karayiannidis) Date: Tue, 22 Mar 2016 11:48:19 +0200 Subject: Child died on request In-Reply-To: References: Message-ID: Hi all, first of all thanks for the reply. I' ve got one more question Is that case ( no backend and trying to pipe ) handled correctly in Varnish 4.1.2 If yes I could move on ?? the latest stable version. Thanks in advance Yiannis 2016-03-22 11:07 GMT+02:00 Per Buer : > > > On Mon, Mar 21, 2016 at 3:43 PM, Yiannis Karayiannidis < > yianniska at gmail.com> wrote: > >> Hello, >> I would like to ask your for help for a really strange case that we have.. >> We have migrated from varnish 3.07 to varnish 4.03 few months ago. The >> last few days. >> we have a strange problem. >> Child died on request. The backends was unreachable at that time (we are >> using a hash director). >> We did get a panic message like : >> > > Look at this: https://www.varnish-cache.org/trac/ticket/1815 > > Looks like you have no backend and you're trying to pipe. > > -- > *Per Buer* > CTO | Varnish Software AS > Cell: +47 95839117 > We Make Websites Fly! > www.varnish-software.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Mar 22 12:53:25 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 22 Mar 2016 13:53:25 +0100 Subject: Child died on request In-Reply-To: References: Message-ID: On Tue, Mar 22, 2016 at 10:48 AM, Yiannis Karayiannidis wrote: > > Hi all, > first of all thanks for the reply. > > I' ve got one more question > Is that case ( no backend and trying to pipe ) handled correctly in Varnish 4.1.2 > If yes I could move on ?? the latest stable version. Hi Yiannis, I believe this is a 4.0-only bug, you should be fine with 4.1.2 :) Cheers, Dridi From matrix at matrix2000.name Wed Mar 30 04:53:52 2016 From: matrix at matrix2000.name (matrix) Date: Wed, 30 Mar 2016 13:53:52 +0900 Subject: mp4 play issue for varnish4,1 Message-ID: Hi I am facing problem about mp4 play. I already set the stream with true. if (bereq.url ~ "^[^?]*\.(mp3|mp4)(\?.*)?$") { set beresp.do_stream = true; } File download is OK, but mp4 does not play immediately using chrome or Firefox. Does anyone have any solutions? (ex: Does Pipe during streaming. Best regards.