From varnish at mm.quex.org Wed Sep 1 02:54:16 2010 From: varnish at mm.quex.org (Michael Alger) Date: Wed, 1 Sep 2010 10:54:16 +0800 Subject: help understanding backend_health varnishlog line In-Reply-To: References: Message-ID: <20100901025416.GA300@grum.quex.org> On Tue, Aug 31, 2010 at 03:42:37PM -0700, David Birdsong wrote: > I'm trying to understand why varnish's health checks marks my > backend down at random intervals. Could somebody explain the > columns in these lines? > > varnishlog -i backend_health | grep -i sick > 0 Backend_health - mogfe Went sick 4--X--- 2 3 3 0.000000 0.000387 It's documented on this page of the wiki: http://varnish-cache.org/wiki/BackendPolling But it does look to me like varnish think it's sending the request okay, and not receiving any response. > My backend is nginx on the same machine. Varnish connects over > 127.0.0.1. The check stanza in nginx simply returns a 1x1 gif that > nginx keeps around in memory so I'm really confused what could be > causing the bad response. Maybe double-check that your probes are closing the connection each request (with the Connection: close header)? As a temporary workaround you could perhaps bump the window size so a single failed request won't knock the backend offline. From alessandro.ronchi at soasi.com Wed Sep 1 11:43:54 2010 From: alessandro.ronchi at soasi.com (Alessandro Ronchi) Date: Wed, 1 Sep 2010 13:43:54 +0200 Subject: filter known hack attempts Message-ID: is it possible to filter with vernish some known hack attempts, link /cltreq.asp?UL=1&ACT=4&BUILD=6254&STRMVER=4&CAPREQ=0 /_vti_bin/owssvr.dll/ and so on? it should be nice to have a list of those bot tries, to avoid unuseful requests and protect web servers... -- Alessandro Ronchi http://www.soasi.com Hobby & Giochi http://hobbygiochi.com http://www.facebook.com/hobbygiochi -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed Sep 1 11:49:16 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 01 Sep 2010 11:49:16 +0000 Subject: filter known hack attempts In-Reply-To: Your message of "Wed, 01 Sep 2010 13:43:54 +0200." Message-ID: <62020.1283341756@critter.freebsd.dk> In message , Ales sandro Ronchi writes: >is it possible to filter with vernish some known hack attempts, link >/cltreq.asp?UL=1&ACT=4&BUILD=6254&STRMVER=4&CAPREQ=0 >/_vti_bin/owssvr.dll/ Yes, Kristian did something "mod_security" like I think. It's pretty easy to blast individual bad things out of the way; sub vcl_recv { if (req.url ~ "_vti_bin/owssvr.dll") { error 503; } } -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kristian at varnish-software.com Wed Sep 1 11:56:30 2010 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Wed, 1 Sep 2010 13:56:30 +0200 Subject: filter known hack attempts In-Reply-To: <62020.1283341756@critter.freebsd.dk> References: <62020.1283341756@critter.freebsd.dk> Message-ID: <20100901115629.GH5349@sunrider> On Wed, Sep 01, 2010 at 11:49:16AM +0000, Poul-Henning Kamp wrote: > In message , Ales > sandro Ronchi writes: > > >is it possible to filter with vernish some known hack attempts, link > >/cltreq.asp?UL=1&ACT=4&BUILD=6254&STRMVER=4&CAPREQ=0 > >/_vti_bin/owssvr.dll/ > > Yes, Kristian did something "mod_security" like I think. Yeah, I helped Edward and Kacper out with "security.vcl", it's kept in svn under varnish-tools/security.vcl/. It was/is a PoC of how a more generic "security modules for VCL" could be done. Keep in mind that it wont catch data sent through POST data though. - Kristian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From tfheen at varnish-software.com Thu Sep 2 07:16:00 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 02 Sep 2010 09:16:00 +0200 Subject: Varnish-cache.org moved Message-ID: <87r5hcliof.fsf@qurzaw.linpro.no> Hi all, I just upgraded the machine hosting varnish-cache.org to a newer version of Ubuntu as well as moving it to a new data centre. I believe everything still works correctly, but if you see anything unusual or broken please drop me a mail and I'll take a look. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From varnish at mm.quex.org Fri Sep 3 10:44:56 2010 From: varnish at mm.quex.org (Michael Alger) Date: Fri, 3 Sep 2010 18:44:56 +0800 Subject: obj.status in synthetic responses Message-ID: <20100903104456.GA24973@grum.quex.org> I've noticed that in a setup that does something along the lines of: sub vcl_recv { ... if (something) { error 798; } } sub vcl_error { if (obj.status == 798) { set obj.status = 404; synthetic {"Oh no, a "} obj.status {" occurred!"}; } } The actual HTTP response code will be correct (404), but the text returned in the response will say the status was 798. This is pretty minor since the important bit (the HTTP response) uses the updated value of obj.status, but I thought it was curious the synthetic response text used the old value. From pablort+varnish at gmail.com Fri Sep 3 20:48:10 2010 From: pablort+varnish at gmail.com (pablort) Date: Fri, 3 Sep 2010 17:48:10 -0300 Subject: Varnish-cache.org moved In-Reply-To: <87r5hcliof.fsf@qurzaw.linpro.no> References: <87r5hcliof.fsf@qurzaw.linpro.no> Message-ID: What about that talk of having a face-lift on the homepage ? :D On Thu, Sep 2, 2010 at 4:16 AM, Tollef Fog Heen wrote: > > Hi all, > > I just upgraded the machine hosting varnish-cache.org to a newer version > of Ubuntu as well as moving it to a new data centre. I believe > everything still works correctly, but if you see anything unusual or > broken please drop me a mail and I'll take a look. > > Regards, > -- > Tollef Fog Heen > Varnish Software > t: +47 21 54 41 73 > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.robinson at stanford.edu Fri Sep 3 20:49:37 2010 From: jim.robinson at stanford.edu (James A. Robinson) Date: Fri, 03 Sep 2010 13:49:37 -0700 Subject: v2.1.3 w/ http_range_support strips Content-Range? Message-ID: <20100903204943.35DA522D1@lions.stanford.edu> Hi folks, We've got a problem dealing with Range requests via Varnish. We're using varnish-2.1.3 SVN 5049:5055, and have activated http_range_support. It looks as though the base logic is that Varnish will (a) strip Range from the request back to the backend, and (b) will surpress Content-Range from the response. HTTPH("Content-Range", H_Content_Range, 2, 3, HTTPH_R_PASS | HTTPH_A_PASS | HTTPH_R_FETCH | HTTPH_A_INS, 0, 0) /* RFC2616 14.16 */ HTTPH("Range", H_Range, 1, 0, HTTPH_R_FETCH | HTTPH_A_INS, 0, 0) /* RFC2616 14.35 */ When http_range_support is on, Varnish will deal with the incoming Range itself if it has the item in cache. Otherwise it appears to try and simply let the backend deal with it. Unfortunately it looks to me as though Varnish accidently strips out the Content-Range header in that situation. First, I want to show that our back end client handles the range requests. On the first request we ask for bytes 0-10: $ curl -i -H'Range: bytes=0-10' -H'Host: host.mydomain.org' -s http://apache/podcast/test.mp3 HTTP/1.1 206 Partial Content Date: Fri, 03 Sep 2010 18:34:48 GMT Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a Content-Range: bytes 0-10/15041999 Accept-Ranges: bytes Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT ETag: "1a8d50-e585cf-4c2cb0da" Content-Length: 11 Set-Cookie: JSESSIONID=79642FE5D7FEABBCFEC11F37B819A1F5.tomcat.mydomain.org; Path=/ Content-Type: audio/mpeg;charset=UTF-8 On the second we request bytes 10-20: $ curl -i -H'Range: bytes=10-20' -H'Host: host.mydomain.org' -s http://apache/podcast/test.mp3 HTTP/1.1 206 Partial Content Date: Fri, 03 Sep 2010 18:34:56 GMT Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a Content-Range: bytes 10-20/15041999 Accept-Ranges: bytes Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT ETag: "1a8d50-e585cf-4c2cb0da" Content-Length: 11 Set-Cookie: JSESSIONID=F915B3C040B37844E8DAA451377C0F8A.backend.mydomain.org; Path=/ Content-Type: audio/mpeg;charset=UTF-8 You can see that the apache server returns a Content-Range in both cases. Next, we try fetching via the front end varnish. Our first request succeeds, an an examination of varnishlog shows that varnish is stripping the incoming Range header when it makes the request to the backend, and is generating its own Content-Range header for the outgoing response: $ curl -i -H'Range: bytes=0-10' -s http://host.mydomain.org/podcast/test.mp3 HTTP/1.1 206 Partial Content Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a Date: Fri, 03 Sep 2010 18:18:10 GMT Content-Type: audio/mpeg;charset=UTF-8 Connection: keep-alive Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT ETag: "1a8d50-e585cf-4c2cb0da" Set-Cookie: JSESSIONID=EFC4B1B3B2B2774BE2B8FD7D26D179C6.tomcat.mydomain.org; Path=/ Accept-Ranges: bytes X-Varnish: 1884335308 Age: 1 Via: 1.1 varnish X-Varnish-Hostname: varnish1.mydomain.org X-Varnish-Cache: miss Content-Range: bytes 0-10/15041999 Content-Length: 11 So far so good! But at this point Varnish has added the resource to its hit-for-pass list (it has determined the item is not cachable). That ought to mean varnish will simply route the request back to the apache server and return the response: $ curl -i -H'Range: bytes=10-20' -s http://host.mydomain.org/podcast/test.mp3 HTTP/1.1 206 Partial Content Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a Date: Fri, 03 Sep 2010 18:18:11 GMT Content-Type: audio/mpeg;charset=UTF-8 Connection: keep-alive Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT ETag: "1a8d50-e585cf-4c2cb0da" Set-Cookie: JSESSIONID=E3144DE996CFA3B0BA4F2DE41A197510.tomcat.mydomain.org; Path=/ Content-Length: 11 Accept-Ranges: bytes X-Varnish: 1884335658 Age: 0 Via: 1.1 varnish X-Varnish-Hostname: varnish1.mydomain.org X-Varnish-Cache: miss Notice the Content-Range header has been stripped out for some reason when varnish transmits the response. This is not, as far as I can determine, to spec, since http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.7 indicates that a 206 response must include Either a Content-Range header field (section 14.16) indicating the range included with this response, or a multipart/byteranges Content-Type including Content-Range fields for each part. If a Content-Length header field is present in the response, its value MUST match the actual number of OCTETs transmitted in the message-body. Looking at varnishlog, the TxHeader made to the backend did contain the Range header, and Apache returned the requested portion and included a Content-Range response, but that header was never sent on to the client. If I clear the resource from Varnish, so that the resource is no longer in the hit-for-pass list, then re-request the 2nd range: $ curl http://host.mydomain.org/varnish/purge/podcast/test.mp3 ... success page returned indicating the item was added to the ban list ... $ curl -i -H'Range: bytes=10-20' -s http://host.mydomain.org/podcast/test.mp3 HTTP/1.1 206 Partial Content Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a Date: Fri, 03 Sep 2010 18:32:54 GMT Content-Type: audio/mpeg;charset=UTF-8 Connection: keep-alive Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT ETag: "1a8d50-e585cf-4c2cb0da" Set-Cookie: JSESSIONID=D496D1EEC4BB7E050D1776ED8C7209F7.tomcat.mydomain.org; Path=/ Accept-Ranges: bytes X-Varnish: 1884521183 Age: 1 Via: 1.1 varnish X-Varnish-Hostname: varnish1.mydomain.org X-Varnish-Cache: miss Content-Range: bytes 10-20/15041999 Content-Length: 11 The new Content-Range comes through. A 3rd request will, once again, return the result stripped of its Content-Range until I purge it. Here's what I see in varnishlog. On the first request the incoming Header specifies the range request: ... 12 RxHeader c Range: bytes=10-20 12 VCL_call c recv 12 VCL_acl c MATCH highwire_acl 127.0.0.1 12 VCL_return c lookup 12 VCL_call c hash 12 VCL_return c hash 12 VCL_call c miss 12 VCL_return c fetch At this point the TxHeader list indicates *no* Range request passed on, so apache returns the entire request: 15 RxProtocol b HTTP/1.1 15 RxStatus b 200 15 RxResponse b OK 15 RxHeader b Date: Fri, 03 Sep 2010 20:08:02 GMT 15 RxHeader b Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a 15 RxHeader b Accept-Ranges: bytes 15 RxHeader b Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT 15 RxHeader b ETag: "1a8d50-e585cf-4c2cb0da" 15 RxHeader b Content-Length: 15041999 15 RxHeader b Set-Cookie: JSESSIONID=35F2019D47A900C13424120CE340EBED.tomcat.mydomain.org; Path=/ 15 RxHeader b Content-Type: audio/mpeg;charset=UTF-8 Varnish then appears to deal with the Range request itself: 12 TTL c 1791943837 RFC 900 1283544482 0 0 0 0 12 VCL_call c fetch 12 VCL_return c pass ... 12 TxHeader c Content-Range: bytes 10-20/15041999 12 TxHeader c Content-Length: 11 ... 15 Length b 15041999 ... 12 VCL_call c deliver 12 VCL_acl c MATCH highwire_acl 127.0.0.1 12 VCL_return c deliver 12 TxProtocol c HTTP/1.1 12 TxStatus c 206 12 TxResponse c Partial Content ... 12 TxHeader c Content-Type: audio/mpeg;charset=UTF-8 12 TxHeader c Accept-Ranges: bytes 12 TxHeader c Date: Fri, 03 Sep 2010 20:08:03 GMT 12 TxHeader c X-Varnish: 1791943837 12 TxHeader c Age: 1 12 TxHeader c Via: 1.1 varnish 12 TxHeader c Connection: close 12 TxHeader c X-Varnish-Hostname: varnish-dev.mydomain.org 12 TxHeader c X-Varnish-Cache: miss 12 Length c 11 12 ReqEnd c 1791943837 1283544482.199923038 1283544483.532929897 0.000079155 1.332911968 0.000094891 12 SessionClose c Connection: close 12 StatSess c 127.0.0.1 37588 1 1 1 0 0 1 1239 11 Now on the 2nd request we see something different occuring. In this situation Varnish is transmitting the Range request to the backend, the backend is responding with status 206 and is providing a Content-Range header in the response. Varnish is allowing most of that response to come back, but is stripping out Content-Range: 12 RxHeader c Range: bytes=10-20 12 VCL_call c recv 12 VCL_acl c MATCH highwire_acl 127.0.0.1 12 VCL_return c lookup 12 VCL_call c hash 12 VCL_return c hash 12 HitPass c 1791943837 12 VCL_call c pass 12 VCL_return c pass 15 TxHeader b Range: bytes=10-20 15 RxProtocol b HTTP/1.1 15 RxStatus b 206 15 RxResponse b Partial Content ... 15 RxHeader b Content-Range: bytes 10-20/15041999 15 RxHeader b Accept-Ranges: bytes 15 RxHeader b Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT 15 RxHeader b ETag: "1a8d50-e585cf-4c2cb0da" 15 RxHeader b Content-Length: 11 ... 12 TTL c 1791943838 RFC 900 1283544541 0 0 0 0 12 VCL_call c fetch 12 VCL_return c pass 12 ObjProtocol c HTTP/1.1 12 ObjStatus c 206 12 ObjResponse c Partial Content ... 12 ObjHeader c Content-Type: audio/mpeg;charset=UTF-8 ... 15 Length b 11 12 VCL_call c deliver 12 VCL_acl c MATCH highwire_acl 127.0.0.1 12 VCL_return c deliver 12 TxProtocol c HTTP/1.1 12 TxStatus c 206 12 TxResponse c Partial Content ... 12 TxHeader c Content-Type: audio/mpeg;charset=UTF-8 ... 12 TxHeader c Content-Length: 11 12 TxHeader c Accept-Ranges: bytes 12 TxHeader c Date: Fri, 03 Sep 2010 20:09:01 GMT 12 TxHeader c X-Varnish: 1791943838 12 TxHeader c Age: 0 12 TxHeader c Via: 1.1 varnish 12 TxHeader c Connection: close 12 TxHeader c X-Varnish-Hostname: varnish-dev.mydomain.org 12 TxHeader c X-Varnish-Cache: miss 12 Length c 11 12 ReqEnd c 1791943838 1283544541.880913019 1283544541.910290003 0.000096083 0.029315948 0.000061035 12 SessionClose c Connection: close 12 StatSess c 127.0.0.1 51087 0 1 1 0 1 1 1202 11 Jim Robinson - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - James A. Robinson jim.robinson at stanford.edu Stanford University HighWire Press http://highwire.stanford.edu/ +1 650 7237294 (Work) +1 650 7259335 (Fax) From perbu at varnish-software.com Sun Sep 5 19:36:30 2010 From: perbu at varnish-software.com (Per Buer) Date: Sun, 5 Sep 2010 21:36:30 +0200 Subject: Varnish-cache.org moved In-Reply-To: References: <87r5hcliof.fsf@qurzaw.linpro.no> Message-ID: Coming right up - stay tuned. On Fri, Sep 3, 2010 at 10:48 PM, pablort > wrote: > What about that talk of having a face-lift on the homepage ? :D > > > > On Thu, Sep 2, 2010 at 4:16 AM, Tollef Fog Heen < > tfheen at varnish-software.com> wrote: > >> >> Hi all, >> >> I just upgraded the machine hosting varnish-cache.org to a newer version >> of Ubuntu as well as moving it to a new data centre. I believe >> everything still works correctly, but if you see anything unusual or >> broken please drop me a mail and I'll take a look. >> >> Regards, >> -- >> Tollef Fog Heen >> Varnish Software >> t: +47 21 54 41 73 >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From ionathan at gmail.com Mon Sep 6 14:40:01 2010 From: ionathan at gmail.com (Jonathan Leibiusky) Date: Mon, 6 Sep 2010 11:40:01 -0300 Subject: Caching doubt Message-ID: Hi! I'm playing with If-None-Match and Cache-Control: max-age headers and I have a doubt. Maybe it is more related to HTTP than Varnish, but it worth asking. Imagine that I have 3 user-agents, a varnish and my webserver. When the clients ask the first time for /example, varnish doesn't have anything cached so he ask the webserver for it. The webserver response has Cache-Control: max-age=180 and ETag: 123 (for example). So Varnish will cache the response for 180 seconds as well as the user-agents. After 180 seconds, the 3 user-agents will request /example for a conditional If-None-Match header because they got the ETag. Since it is a conditional request varnish will ask the webserver and will get a fresh response with all the caching headers. When the other 2 user-agent make the same conditional request they will generate another request to the webserver unless varnish has some mechanism to answer himself. Is that right? Does varnish has this mechanism? Thanks! Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bedis9 at gmail.com Tue Sep 7 04:34:22 2010 From: bedis9 at gmail.com (Bedis 9) Date: Tue, 7 Sep 2010 06:34:22 +0200 Subject: Caching doubt In-Reply-To: References: Message-ID: Hi, What kind of Vary header your webserver sends? If no, there is no reason to cache 3 versions of this object unless your conditionnal request does not match any header of the cached object. cheers On Mon, Sep 6, 2010 at 4:40 PM, Jonathan Leibiusky wrote: > Hi! > I'm playing with If-None-Match and Cache-Control: max-age?? headers and I > have a doubt. Maybe it is more related to HTTP than Varnish, but it worth > asking. > > Imagine that I have 3 user-agents, a varnish and my webserver. > When the clients ask the first time for /example, varnish doesn't have > anything cached so he ask the webserver for it. The webserver response has > Cache-Control: max-age=180 and ETag: 123 (for example). > So Varnish will cache the response for 180 seconds as well as the > user-agents. > After 180 seconds, the 3 user-agents will request /example for a conditional > If-None-Match header because they got the ETag. Since it is a conditional > request varnish will ask the webserver and will get a fresh response with > all the caching headers. When the other 2 user-agent make the same > conditional request they will generate another request to the webserver > unless varnish has some mechanism to answer himself. > Is that right? Does varnish has this mechanism? > > Thanks! > > Jonathan > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From ionathan at gmail.com Tue Sep 7 11:09:06 2010 From: ionathan at gmail.com (Jonathan Leibiusky) Date: Tue, 7 Sep 2010 08:09:06 -0300 Subject: Caching doubt In-Reply-To: References: Message-ID: I think I didn't explain myself good enough and after playing with varnish I realized that by default varnish won't forward to the backend If-None-Match and If-Modified-Since headers. Right now my backend is prepared to answer 304 when the resource hasn't been modified, so I would like varnish to cache an object for X seconds and after that use the ETag and Last-Modified sent by the backend to verify if the expired resource is still the same and not ask directly for a fresh one. Right now it always asks for a new one. On Tue, Sep 7, 2010 at 1:34 AM, Bedis 9 wrote: > Hi, > > What kind of Vary header your webserver sends? > If no, there is no reason to cache 3 versions of this object unless > your conditionnal request does not match any header of the cached > object. > > cheers > > On Mon, Sep 6, 2010 at 4:40 PM, Jonathan Leibiusky > wrote: > > Hi! > > I'm playing with If-None-Match and Cache-Control: max-age headers and I > > have a doubt. Maybe it is more related to HTTP than Varnish, but it worth > > asking. > > > > Imagine that I have 3 user-agents, a varnish and my webserver. > > When the clients ask the first time for /example, varnish doesn't have > > anything cached so he ask the webserver for it. The webserver response > has > > Cache-Control: max-age=180 and ETag: 123 (for example). > > So Varnish will cache the response for 180 seconds as well as the > > user-agents. > > After 180 seconds, the 3 user-agents will request /example for a > conditional > > If-None-Match header because they got the ETag. Since it is a conditional > > request varnish will ask the webserver and will get a fresh response with > > all the caching headers. When the other 2 user-agent make the same > > conditional request they will generate another request to the webserver > > unless varnish has some mechanism to answer himself. > > Is that right? Does varnish has this mechanism? > > > > Thanks! > > > > Jonathan > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtshilston at gmail.com Tue Sep 7 11:27:35 2010 From: rtshilston at gmail.com (Rob S) Date: Tue, 07 Sep 2010 12:27:35 +0100 Subject: SSL and Varnish Message-ID: <4C8621A7.50804@gmail.com> If you've used SSL with varnish in production, can you share any wisdom on your configuration and approaches? We've been happily using Varnish in production for just over a year, and now want to use it for an SSL site so that we can use ESI, load balancing and other such facilities offered by Varnish. I've searched over the Varnish website, and can't see any examples of a recommended SSL configuration. http://plone.org/documentation/kb/plone-behind-varnish-using-pound-for-ssl provides some documentation, but I suspect there are other options. I'll happily consolidate the answers into a coherent document for the website so that others can benefit in future. Thanks, Rob From kristian at varnish-software.com Tue Sep 7 11:43:38 2010 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Tue, 7 Sep 2010 13:43:38 +0200 Subject: Caching doubt In-Reply-To: References: Message-ID: <20100907114337.GD3475@sunrider> On Tue, Sep 07, 2010 at 08:09:06AM -0300, Jonathan Leibiusky wrote: > I think I didn't explain myself good enough and after playing with varnish I > realized that by default varnish won't forward to the backend If-None-Match > and If-Modified-Since headers. > Right now my backend is prepared to answer 304 when the resource hasn't been > modified, so I would like varnish to cache an object for X seconds and after > that use the ETag and Last-Modified sent by the backend to verify if the > expired resource is still the same and not ask directly for a fresh one. > Right now it always asks for a new one. Jonathan: We should take this on the support-list :) Anyway: As I said on support@: Varnish supports 304 to the client, but it does not send conditional GET requests to a web server. So Varnish will always get the entire object from the web server, but it will send "304 Not Modified" to the client if the conditional GET request indicates that the client already has the content. - Kristian -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 490 bytes Desc: Digital signature URL: From cosimo at streppone.it Tue Sep 7 13:34:48 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Tue, 07 Sep 2010 15:34:48 +0200 Subject: SSL and Varnish In-Reply-To: <4C8621A7.50804@gmail.com> References: <4C8621A7.50804@gmail.com> Message-ID: On Tue, 07 Sep 2010 13:27:35 +0200, Rob S wrote: > If you've used SSL with varnish in production, can you share any > wisdom on your configuration and approaches? > > We've been happily using Varnish in production for just over a year, and > now want to use it for an SSL site so that we can use ESI, load > balancing and other such facilities offered by Varnish. I've searched > over the Varnish website, and can't see any examples of a recommended > SSL configuration. We setup all varnish boxes where we need SSL with nginx on :443 and varnishd on :80. The local nginx server proxy-passes everything to localhost:80, with the following config bit: server { ssl on; ssl_certificate /etc/ssl/certs/your.crt; ssl_certificate_key /etc/ssl/private/your.key; listen 443 default ssl; access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; # Proxy any requests to the local varnish instance location / { proxy_set_header "Host:" $host; proxy_set_header "X-Forwarded-For" $proxy_add_x_forwarded_for; proxy_pass http://localhost:80; } } > I'll happily consolidate the answers into a coherent document for the > website so that others can benefit in future. Would be nice! -- Cosimo From jhayter at manta.com Tue Sep 7 14:14:53 2010 From: jhayter at manta.com (Jim Hayter) Date: Tue, 7 Sep 2010 10:14:53 -0400 Subject: SSL and Varnish In-Reply-To: <4C8621A7.50804@gmail.com> References: <4C8621A7.50804@gmail.com> Message-ID: We run nginx to accept both SSL and non-SSL requests. This allows me to have some logic to block certain requests at the nginx level and also for nginx to return a customized error page if varnishd is not running. Nginx on passes all remaining traffic to varnishd which sits in front of multiple web servers. -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Rob S Sent: Tuesday, September 07, 2010 7:28 AM To: varnish-misc at varnish-cache.org Subject: SSL and Varnish If you've used SSL with varnish in production, can you share any wisdom on your configuration and approaches? We've been happily using Varnish in production for just over a year, and now want to use it for an SSL site so that we can use ESI, load balancing and other such facilities offered by Varnish. I've searched over the Varnish website, and can't see any examples of a recommended SSL configuration. http://plone.org/documentation/kb/plone-behind-varnish-using-pound-for-s sl provides some documentation, but I suspect there are other options. I'll happily consolidate the answers into a coherent document for the website so that others can benefit in future. Thanks, Rob _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From perbu at varnish-software.com Thu Sep 9 10:27:36 2010 From: perbu at varnish-software.com (Per Buer) Date: Thu, 9 Sep 2010 12:27:36 +0200 Subject: New website: varnish-cache.org Message-ID: Hi. I'm very happy to announce the relaunch of our community site www.varnish-cache.org. The objective of the design was to make the important information more visible, to make the whole site visually more pleasing and aggregating some external, relevant information. As you can see there is a new color scheme and a new, modern logo as well. We're keeping trac - we'll be re-theming it during today or tomorrow. All existing links are redirected. If anyone would like to participate with developing the site further please let me know. I'll get busy cleaning up the wiki now, moving more and more content either into the main main web site or into the documentation. Help is more then welcome. Feedback is welcome. Cheers, Per. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From David at firechaser.com Thu Sep 9 10:55:44 2010 From: David at firechaser.com (David Murphy) Date: Thu, 9 Sep 2010 11:55:44 +0100 Subject: New website: varnish-cache.org In-Reply-To: References: Message-ID: Thanks for work on this. Looking forward to using new site. As you'r probably aware, there is a timeout issue at the moment for that URL (for me at least, from Spain). Last 5 hops in traceroute. 10 te0-3-0-0.ccr22.ham01.atlas.cogentco.com (130.117.1.102) 95.085 ms 86.536 ms te0-3-0-4.ccr22.ham01.atlas.cogentco.com (130.117.49.226) 86.714 ms 11 te3-8.ccr02.sto01.atlas.cogentco.com (130.117.51.194) 103.673 ms te1-1.ccr02.sto01.atlas.cogentco.com (130.117.3.206) 101.325 ms 106.260 ms 12 te3-8.ccr02.sto01.atlas.cogentco.com (130.117.51.194) 111.779 ms te1-1.ccr01.osl01.atlas.cogentco.com (130.117.3.42) 118.876 ms 110.179 ms 13 * te1-1.ccr01.osl01.atlas.cogentco.com (130.117.3.42) 121.294 ms * 14 * * * 15 * * * Best, David On 9 Sep 2010, at 12:27, Per Buer wrote: Hi. I'm very happy to announce the relaunch of our community site www.varnish-cache.org. The objective of the design was to make the important information more visible, to make the whole site visually more pleasing and aggregating some external, relevant information. As you can see there is a new color scheme and a new, modern logo as well. We're keeping trac - we'll be re-theming it during today or tomorrow. All existing links are redirected. If anyone would like to participate with developing the site further please let me know. I'll get busy cleaning up the wiki now, moving more and more content either into the main main web site or into the documentation. Help is more then welcome. Feedback is welcome. Cheers, Per. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.hongens at netmatch.nl Thu Sep 9 11:07:40 2010 From: a.hongens at netmatch.nl (=?ISO-8859-1?Q?Angelo_H=F6ngens?=) Date: Thu, 09 Sep 2010 13:07:40 +0200 Subject: New website: varnish-cache.org In-Reply-To: References: Message-ID: <4C88BFFC.9020206@netmatch.nl> On 9-9-2010 12:27, Per Buer wrote: > Hi. > > I'm very happy to announce the relaunch of our community site > www.varnish-cache.org . > > The objective of the design was to make the important information more > visible, to make the whole site visually more pleasing and aggregating > some external, relevant information. As you can see there is a new > color scheme and a new, modern logo as well. We're keeping trac - we'll > be re-theming it during today or tomorrow. All existing links are > redirected. Congrats! Although I'm not sure every new element is an improvement, overall the site looks much better and visually appealing! (although I also liked the old style.) -- With kind regards, Angelo H?ngens systems administrator MCSE on Windows 2003 MCSE on Windows 2000 MS Small Business Specialist ------------------------------------------ NetMatch tourism internet software solutions Ringbaan Oost 2b 5013 CA Tilburg +31 (0)13 5811088 +31 (0)13 5821239 A.Hongens at netmatch.nl www.netmatch.nl ------------------------------------------ From schmidt at ze.tum.de Thu Sep 9 11:23:34 2010 From: schmidt at ze.tum.de (Gerhard Schmidt) Date: Thu, 09 Sep 2010 13:23:34 +0200 Subject: vcl include inline config Message-ID: <4C88C3B6.7040406@ze.tum.de> Hi, is there a way to include config generated with vcl.inline via cli interface in another config (also generated via vcl.inline) this whould help to workaround some issues with vcl.inline and large configs (Bug #772 e.g.) Regards Estartu -- ------------------------------------------------- Gerhard Schmidt | E-Mail: schmidt at ze.tum.de TU-M?nchen | Jabber: estartu at ze.tum.de WWW & Online Services | Tel: 089/289-25270 | Fax: 089/289-25257 | PGP-Publickey auf Anfrage -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 544 bytes Desc: OpenPGP digital signature URL: From pablort+varnish at gmail.com Thu Sep 9 20:52:54 2010 From: pablort+varnish at gmail.com (pablort) Date: Thu, 9 Sep 2010 17:52:54 -0300 Subject: New website: varnish-cache.org In-Reply-To: <4C88BFFC.9020206@netmatch.nl> References: <4C88BFFC.9020206@netmatch.nl> Message-ID: It would be nice to have a new wiki too. :) On Thu, Sep 9, 2010 at 8:07 AM, Angelo H?ngens wrote: > On 9-9-2010 12:27, Per Buer wrote: > > Hi. > > > > I'm very happy to announce the relaunch of our community site > > www.varnish-cache.org . > > > > The objective of the design was to make the important information more > > visible, to make the whole site visually more pleasing and aggregating > > some external, relevant information. As you can see there is a new > > color scheme and a new, modern logo as well. We're keeping trac - we'll > > be re-theming it during today or tomorrow. All existing links are > > redirected. > > Congrats! Although I'm not sure every new element is an improvement, > overall the site looks much better and visually appealing! (although I > also liked the old style.) > > > -- > > > With kind regards, > > > Angelo H?ngens > systems administrator > > MCSE on Windows 2003 > MCSE on Windows 2000 > MS Small Business Specialist > ------------------------------------------ > NetMatch > tourism internet software solutions > > Ringbaan Oost 2b > 5013 CA Tilburg > +31 (0)13 5811088 > +31 (0)13 5821239 > > A.Hongens at netmatch.nl > www.netmatch.nl > ------------------------------------------ > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joe at joetify.com Fri Sep 10 01:46:58 2010 From: joe at joetify.com (Joe Williams) Date: Thu, 9 Sep 2010 18:46:58 -0700 Subject: further syslog examples Message-ID: <9D1653F3-87FF-4198-B907-E752EB379B01@joetify.com> Anyone have examples beyond what's on the wiki of logging varnish to syslog? Thanks. -Joe Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ Twitter: http://twitter.com/williamsjoe From checker at d6.com Fri Sep 10 17:12:59 2010 From: checker at d6.com (Chris Hecker) Date: Fri, 10 Sep 2010 10:12:59 -0700 Subject: regsub in error response Message-ID: <4C8A671B.5010702@d6.com> I'd like to do a 301 from a.com to b.com, but preserving the whole url, but I can't figure out how to do it. This doesn't work to send it to vcl_error: error 751 regsub(req.url,"a.com","b.com"); The regsub isn't allowed there. Can I stuff it in a header that I can get in vcl_error? This is CentOS: Name : varnish Arch : i386 Version : 2.0.6 Release : 2.el5 Thanks, Chris From varnish at mm.quex.org Sat Sep 11 09:52:05 2010 From: varnish at mm.quex.org (Michael Alger) Date: Sat, 11 Sep 2010 17:52:05 +0800 Subject: regsub in error response In-Reply-To: <4C8A671B.5010702@d6.com> References: <4C8A671B.5010702@d6.com> Message-ID: <20100911095205.GA28555@grum.quex.org> On Fri, Sep 10, 2010 at 10:12:59AM -0700, Chris Hecker wrote: > > I'd like to do a 301 from a.com to b.com, but preserving the whole > url, but I can't figure out how to do it. This doesn't work to > send it to vcl_error: > > error 751 regsub(req.url,"a.com","b.com"); > > The regsub isn't allowed there. Can I stuff it in a header that I > can get in vcl_error? You probably can. What I do for redirects is to rewrite the req.url and then use that in the error statement, i.e. set req.url = regsub (req.url, "rewritefrom", "rewriteto"); error 751 req.url; For your particular case where you don't actually want to rewrite the URL but instead only change the host part of it, I would do something like: if (req.url ~ "(?i)^/vanitypath") { set req.url = "http://b.com" req.url; error 751 req.url; } Note that the host isn't present in req.url (use req.http.Host for that) - so in the first example, "rewriteto" should start with a full protocol specification (http://..."). Most browsers work fine if they're told to redirect to /foo, but I don't really like relying on it. From amoiz.shine at gmail.com Sat Sep 11 13:03:15 2010 From: amoiz.shine at gmail.com (Sharl.Jimh.Tsin) Date: Sat, 11 Sep 2010 21:03:15 +0800 Subject: question about the multiple backends Message-ID: hi,everyone i have a varnish(2.1.3) server and some httpd server likes lighttpd?nginx and so on.now i let varnish send the request to different server by domain url. this is my configure: backend to_cherokee { .host = "127.0.0.1"; .port = "81"; } backend to_lighttpd { .host = "127.0.0.1"; .port = "82"; } backend to_nginx { .host = "127.0.0.1"; .port = "83"; } sub vcl_recv { if (req.http.host ~ "^(\w*\.)?aaaaaaaaa.cn$") { set req.backend = to_cherokee; } elseif (req.http.host ~ "^(\w*\.)?bbbbbbbbbbbb.cn$") { set req.backend = to_nginx; } else { set req.backend = to_lighttpd; } } BUT it don't works. any wrong with them? WANT your HELP!! Best regards, Sharl.Jimh.Tsin (From China) From fla_torres at yahoo.com.br Sat Sep 11 13:20:26 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Sat, 11 Sep 2010 10:20:26 -0300 Subject: question about the multiple backends In-Reply-To: References: Message-ID: <4C8B821A.9030404@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/11/2010 10:03 AM, Sharl.Jimh.Tsin wrote: > BUT it don't works. any wrong with them? WANT your HELP!! what do you get when you type aaaa or bbbbb ? -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyLghcACgkQNRQApncg297amwCg8RIET7msMeSLYKeT9WqtsCnZ YcoAmwbBnqm6EWkIFb5yOft/3CbZji38 =aIQm -----END PGP SIGNATURE----- From amoiz.shine at gmail.com Sun Sep 12 02:23:32 2010 From: amoiz.shine at gmail.com (Sharl.Jimh.Tsin) Date: Sun, 12 Sep 2010 10:23:32 +0800 Subject: question about the multiple backends In-Reply-To: <4C8B821A.9030404@yahoo.com.br> References: <4C8B821A.9030404@yahoo.com.br> Message-ID: <4C8C39A4.8010808@gmail.com> On 2010?09?11? 21:20, Flavio Torres wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 09/11/2010 10:03 AM, Sharl.Jimh.Tsin wrote: > >> BUT it don't works. any wrong with them? WANT your HELP!! >> > what do you get when you type aaaa or bbbbb ? > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkyLghcACgkQNRQApncg297amwCg8RIET7msMeSLYKeT9WqtsCnZ > YcoAmwbBnqm6EWkIFb5yOft/3CbZji38 > =aIQm > -----END PGP SIGNATURE----- > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > aaaa goto the first backend,bbbbb returns a 503 error. From amoiz.shine at gmail.com Sun Sep 12 07:40:25 2010 From: amoiz.shine at gmail.com (Sharl.Jimh.Tsin) Date: Sun, 12 Sep 2010 15:40:25 +0800 Subject: question about the multiple backends In-Reply-To: References: <4C8B821A.9030404@yahoo.com.br> <4C8C39A4.8010808@gmail.com> Message-ID: <4C8C83E9.9000500@gmail.com> On 2010?09?12? 14:08, Bayron Guevara wrote: > A 503 error means 'Service Unavailable' and often is caused by > mistakes in the backend definition or the proxy can't connect with the > Web Server or it has a misconfiguration so when a configurable timeout > is elapsed you get this error likely together with a Guru Medidation > message. Have you access to the Web Server directly typing? the domain > and port, example http://wwww.bbbbbbbbbbbb.cn > :83 ? What output do you get? Yes,i can access the backend server directly via domain:port. the backend works well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoiz.shine at gmail.com Mon Sep 13 02:22:50 2010 From: amoiz.shine at gmail.com (Sharl.Jimh.Tsin) Date: Mon, 13 Sep 2010 10:22:50 +0800 Subject: question about the multiple backends In-Reply-To: References: <4C8B821A.9030404@yahoo.com.br> <4C8C39A4.8010808@gmail.com> <4C8C83E9.9000500@gmail.com> Message-ID: here it is: connect_timeout 0.400000 [s] first_byte_timeout 60.000000 [s] between_bytes_timeout 60.000000 [s] Best regards, Sharl.Jimh.Tsin (From China) 2010/9/13 Bayron Guevara : > Sorry, the right param name is connect_timeout instead of connection_timeout > From scaunter at topscms.com Tue Sep 14 17:46:34 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Tue, 14 Sep 2010 13:46:34 -0400 Subject: regsub in error response In-Reply-To: <20100911095205.GA28555@grum.quex.org> References: <4C8A671B.5010702@d6.com> <20100911095205.GA28555@grum.quex.org> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C42C101@TMG-EVS02.torstar.net> You need to tell the browser the new location in vcl_error. I have something similar for mobile redirects. First in vcl_recv catch the event: if( ... some condition ... ) { set req.url = regsub(req.url, "(.*)", "/mobile\1"); error 750 "Moved Temporarily"; } Then handle it in vcl_error: if (obj.status == 750) { set obj.http.Location = req.url; set obj.status = 302; return(deliver); } You can specify obj.status = 301 of course. Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Michael Alger Sent: September-11-10 5:52 AM To: varnish-misc at varnish-cache.org Subject: Re: regsub in error response On Fri, Sep 10, 2010 at 10:12:59AM -0700, Chris Hecker wrote: > > I'd like to do a 301 from a.com to b.com, but preserving the whole > url, but I can't figure out how to do it. This doesn't work to > send it to vcl_error: > > error 751 regsub(req.url,"a.com","b.com"); > > The regsub isn't allowed there. Can I stuff it in a header that I > can get in vcl_error? You probably can. What I do for redirects is to rewrite the req.url and then use that in the error statement, i.e. set req.url = regsub (req.url, "rewritefrom", "rewriteto"); error 751 req.url; For your particular case where you don't actually want to rewrite the URL but instead only change the host part of it, I would do something like: if (req.url ~ "(?i)^/vanitypath") { set req.url = "http://b.com" req.url; error 751 req.url; } Note that the host isn't present in req.url (use req.http.Host for that) - so in the first example, "rewriteto" should start with a full protocol specification (http://..."). Most browsers work fine if they're told to redirect to /foo, but I don't really like relying on it. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From checker at d6.com Tue Sep 14 17:56:09 2010 From: checker at d6.com (Chris Hecker) Date: Tue, 14 Sep 2010 10:56:09 -0700 Subject: regsub in error response In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C42C101@TMG-EVS02.torstar.net> References: <4C8A671B.5010702@d6.com> <20100911095205.GA28555@grum.quex.org> <7F0AA702B8A85A4A967C4C8EBAD6902C42C101@TMG-EVS02.torstar.net> Message-ID: <4C8FB739.9010905@d6.com> But the req.url doesn't seem to have the domain name in it, which is what I want to redirect... Chris On 2010/09/14 10:46, Caunter, Stefan wrote: > You need to tell the browser the new location in vcl_error. I have > something similar for mobile redirects. > > First in vcl_recv catch the event: > > if( ... some condition ... ) { > set req.url = regsub(req.url, "(.*)", > "/mobile\1"); > error 750 "Moved Temporarily"; > } > > Then handle it in vcl_error: > > if (obj.status == 750) { > set obj.http.Location = req.url; > set obj.status = 302; > return(deliver); > } > > You can specify obj.status = 301 of course. > > Stefan Caunter :: Senior Systems Administrator :: TOPS > e: scaunter at topscms.com :: m: (416) 561-4871 > www.thestar.com www.topscms.com > > > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Michael > Alger > Sent: September-11-10 5:52 AM > To: varnish-misc at varnish-cache.org > Subject: Re: regsub in error response > > On Fri, Sep 10, 2010 at 10:12:59AM -0700, Chris Hecker wrote: >> >> I'd like to do a 301 from a.com to b.com, but preserving the whole >> url, but I can't figure out how to do it. This doesn't work to >> send it to vcl_error: >> >> error 751 regsub(req.url,"a.com","b.com"); >> >> The regsub isn't allowed there. Can I stuff it in a header that I >> can get in vcl_error? > > You probably can. What I do for redirects is to rewrite the req.url > and then use that in the error statement, i.e. > > set req.url = regsub (req.url, "rewritefrom", "rewriteto"); > error 751 req.url; > > For your particular case where you don't actually want to rewrite > the URL but instead only change the host part of it, I would do > something like: > > if (req.url ~ "(?i)^/vanitypath") > { > set req.url = "http://b.com" req.url; > error 751 req.url; > } > > Note that the host isn't present in req.url (use req.http.Host for > that) - so in the first example, "rewriteto" should start with a > full protocol specification (http://..."). > > Most browsers work fine if they're told to redirect to /foo, but I > don't really like relying on it. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From checker at d6.com Tue Sep 14 18:05:45 2010 From: checker at d6.com (Chris Hecker) Date: Tue, 14 Sep 2010 11:05:45 -0700 Subject: regsub in error response In-Reply-To: <4C8FB739.9010905@d6.com> References: <4C8A671B.5010702@d6.com> <20100911095205.GA28555@grum.quex.org> <7F0AA702B8A85A4A967C4C8EBAD6902C42C101@TMG-EVS02.torstar.net> <4C8FB739.9010905@d6.com> Message-ID: <4C8FB979.2000506@d6.com> Okay, figured this out. in vcl_recv I check req.http.Host, and if it's wrong, I set it correctly and throw 751, then in vcl_error: if(obj.status == 751) { # redirect set obj.http.Location = "http://" req.http.Host req.url; set obj.status = 301; This seems to work. Can anybody see any problems? Is it bad to set req.http.Host? Will that string concatenation above always be well formed? Thanks, Chris On 2010/09/14 10:56, Chris Hecker wrote: > > But the req.url doesn't seem to have the domain name in it, which is > what I want to redirect... > > Chris > > > > On 2010/09/14 10:46, Caunter, Stefan wrote: >> You need to tell the browser the new location in vcl_error. I have >> something similar for mobile redirects. >> >> First in vcl_recv catch the event: >> >> if( ... some condition ... ) { >> set req.url = regsub(req.url, "(.*)", >> "/mobile\1"); >> error 750 "Moved Temporarily"; >> } >> >> Then handle it in vcl_error: >> >> if (obj.status == 750) { >> set obj.http.Location = req.url; >> set obj.status = 302; >> return(deliver); >> } >> >> You can specify obj.status = 301 of course. >> >> Stefan Caunter :: Senior Systems Administrator :: TOPS >> e: scaunter at topscms.com :: m: (416) 561-4871 >> www.thestar.com www.topscms.com >> >> >> -----Original Message----- >> From: varnish-misc-bounces at varnish-cache.org >> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Michael >> Alger >> Sent: September-11-10 5:52 AM >> To: varnish-misc at varnish-cache.org >> Subject: Re: regsub in error response >> >> On Fri, Sep 10, 2010 at 10:12:59AM -0700, Chris Hecker wrote: >>> >>> I'd like to do a 301 from a.com to b.com, but preserving the whole >>> url, but I can't figure out how to do it. This doesn't work to >>> send it to vcl_error: >>> >>> error 751 regsub(req.url,"a.com","b.com"); >>> >>> The regsub isn't allowed there. Can I stuff it in a header that I >>> can get in vcl_error? >> >> You probably can. What I do for redirects is to rewrite the req.url >> and then use that in the error statement, i.e. >> >> set req.url = regsub (req.url, "rewritefrom", "rewriteto"); >> error 751 req.url; >> >> For your particular case where you don't actually want to rewrite >> the URL but instead only change the host part of it, I would do >> something like: >> >> if (req.url ~ "(?i)^/vanitypath") >> { >> set req.url = "http://b.com" req.url; >> error 751 req.url; >> } >> >> Note that the host isn't present in req.url (use req.http.Host for >> that) - so in the first example, "rewriteto" should start with a >> full protocol specification (http://..."). >> >> Most browsers work fine if they're told to redirect to /foo, but I >> don't really like relying on it. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> From scaunter at topscms.com Tue Sep 14 18:10:41 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Tue, 14 Sep 2010 14:10:41 -0400 Subject: regsub in error response In-Reply-To: <4C8FB979.2000506@d6.com> References: <4C8A671B.5010702@d6.com> <20100911095205.GA28555@grum.quex.org> <7F0AA702B8A85A4A967C4C8EBAD6902C42C101@TMG-EVS02.torstar.net> <4C8FB739.9010905@d6.com> <4C8FB979.2000506@d6.com> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C42C10C@TMG-EVS02.torstar.net> My condition is based on a check of req.url. #mobile rules if(! req.url ~ "(^/iphone|^/mobile|^/rss)" && req.http.user-agent ~ "([B b]lack[Bb]erry|[iI][eE][mM]obile)") { set req.url = regsub(req.url, "(.*)", "/mobile\1"); error 750 "Moved Temporarily"; } The req.http.Host header will always be present in HTTP/1.1 requests. Also, you're running 2.06 and I'm running 2.1.3 (which shouldn't matter although I haven't looked at the changelog). Stefan -----Original Message----- From: Chris Hecker [mailto:checker at d6.com] Sent: September-14-10 2:06 PM To: Caunter, Stefan Cc: varnish-misc at varnish-cache.org Subject: Re: regsub in error response Okay, figured this out. in vcl_recv I check req.http.Host, and if it's wrong, I set it correctly and throw 751, then in vcl_error: if(obj.status == 751) { # redirect set obj.http.Location = "http://" req.http.Host req.url; set obj.status = 301; This seems to work. Can anybody see any problems? Is it bad to set req.http.Host? Will that string concatenation above always be well formed? Thanks, Chris On 2010/09/14 10:56, Chris Hecker wrote: > > But the req.url doesn't seem to have the domain name in it, which is > what I want to redirect... > > Chris > > > > On 2010/09/14 10:46, Caunter, Stefan wrote: >> You need to tell the browser the new location in vcl_error. I have >> something similar for mobile redirects. >> >> First in vcl_recv catch the event: >> >> if( ... some condition ... ) { >> set req.url = regsub(req.url, "(.*)", >> "/mobile\1"); >> error 750 "Moved Temporarily"; >> } >> >> Then handle it in vcl_error: >> >> if (obj.status == 750) { >> set obj.http.Location = req.url; >> set obj.status = 302; >> return(deliver); >> } >> >> You can specify obj.status = 301 of course. >> >> Stefan Caunter :: Senior Systems Administrator :: TOPS >> e: scaunter at topscms.com :: m: (416) 561-4871 >> www.thestar.com www.topscms.com >> >> >> -----Original Message----- >> From: varnish-misc-bounces at varnish-cache.org >> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Michael >> Alger >> Sent: September-11-10 5:52 AM >> To: varnish-misc at varnish-cache.org >> Subject: Re: regsub in error response >> >> On Fri, Sep 10, 2010 at 10:12:59AM -0700, Chris Hecker wrote: >>> >>> I'd like to do a 301 from a.com to b.com, but preserving the whole >>> url, but I can't figure out how to do it. This doesn't work to >>> send it to vcl_error: >>> >>> error 751 regsub(req.url,"a.com","b.com"); >>> >>> The regsub isn't allowed there. Can I stuff it in a header that I >>> can get in vcl_error? >> >> You probably can. What I do for redirects is to rewrite the req.url >> and then use that in the error statement, i.e. >> >> set req.url = regsub (req.url, "rewritefrom", "rewriteto"); >> error 751 req.url; >> >> For your particular case where you don't actually want to rewrite >> the URL but instead only change the host part of it, I would do >> something like: >> >> if (req.url ~ "(?i)^/vanitypath") >> { >> set req.url = "http://b.com" req.url; >> error 751 req.url; >> } >> >> Note that the host isn't present in req.url (use req.http.Host for >> that) - so in the first example, "rewriteto" should start with a >> full protocol specification (http://..."). >> >> Most browsers work fine if they're told to redirect to /foo, but I >> don't really like relying on it. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> From justinp at distribion.com Tue Sep 14 22:53:04 2010 From: justinp at distribion.com (Justin Pasher) Date: Tue, 14 Sep 2010 17:53:04 -0500 Subject: Multiple log files for one instance Message-ID: <4C8FFCD0.30102@distribion.com> I'm thinking what I want to do is not possible, but I figured I'd give it a shot. I want to setup a generic Varnish instance that will be used to handle multiple web sites. All of the web sites are directed to the same backend and the correct site is served up by Apache using Virtual Hosts that match the corresponding Host: header. The idea is that a new site can have caching "turned on" simply by changing its IP address from the live server to the Varnish server. However, by doing so, all requests come into the same instance. This means that I cannot create separate log files for each site using varnishncsa. I've played around with some ideas in regards to matching server headers in the log, but I'm thinking this is not even possible because I would have to run a varnishncsa instance for each site but have them all talking to the same Varnish instance. ISTM that by doing so, varnish would "serve" the log to only one of the daemons and not all of them, making the logging ineffective. I could potentially still have my log analyzer work by read entirely from one file, but that would require tweaking the output format of varnishncsa which is not possible (without hacking the code). Can anyone think of a way to do something like this? Thanks. -- Justin Pasher From joe at joetify.com Tue Sep 14 22:59:38 2010 From: joe at joetify.com (Joe Williams) Date: Tue, 14 Sep 2010 15:59:38 -0700 Subject: further syslog examples In-Reply-To: <9D1653F3-87FF-4198-B907-E752EB379B01@joetify.com> References: <9D1653F3-87FF-4198-B907-E752EB379B01@joetify.com> Message-ID: <7277B0D0-F51B-49DF-854F-3DCD1784C839@joetify.com> I would really like to get varnishlog and/or varnishncsa writing to syslog. Is this possible? Thanks. -Joe On Sep 9, 2010, at 6:46 PM, Joe Williams wrote: > > Anyone have examples beyond what's on the wiki of logging varnish to syslog? > > Thanks. > > -Joe > > > > Name: Joseph A. Williams > Email: joe at joetify.com > Blog: http://www.joeandmotorboat.com/ > Twitter: http://twitter.com/williamsjoe > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ Twitter: http://twitter.com/williamsjoe From vlists at veus.hr Tue Sep 14 23:21:02 2010 From: vlists at veus.hr (Vladimir Vuksan) Date: Tue, 14 Sep 2010 19:21:02 -0400 Subject: further syslog examples In-Reply-To: <7277B0D0-F51B-49DF-854F-3DCD1784C839@joetify.com> References: <9D1653F3-87FF-4198-B907-E752EB379B01@joetify.com> <7277B0D0-F51B-49DF-854F-3DCD1784C839@joetify.com> Message-ID: <1284506462.13006.20.camel@zadria> Perhaps a "temporary" way to do is write a wrapper (in a language of your choice) that outputs anything that varnishncsa produces into syslog. Should be pretty trivial even though it's not pretty :-(. U Uto, 14. 09. 2010., u 15:59 -0700, Joe Williams je napisao/la: > I would really like to get varnishlog and/or varnishncsa writing to syslog. Is this possible? > > Thanks. > -Joe > > > On Sep 9, 2010, at 6:46 PM, Joe Williams wrote: > > > > > Anyone have examples beyond what's on the wiki of logging varnish to syslog? > > > > Thanks. > > > > -Joe > > > > > > > > Name: Joseph A. Williams > > Email: joe at joetify.com > > Blog: http://www.joeandmotorboat.com/ > > Twitter: http://twitter.com/williamsjoe > > > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > Name: Joseph A. Williams > Email: joe at joetify.com > Blog: http://www.joeandmotorboat.com/ > Twitter: http://twitter.com/williamsjoe > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From crcook at gmail.com Tue Sep 14 23:40:10 2010 From: crcook at gmail.com (Chris Cook) Date: Tue, 14 Sep 2010 19:40:10 -0400 Subject: Fixing backend URLs Message-ID: <2006354608710103182@unknownmsgid> I have an application that takes a URI that's submitted and spits back a different URI with a 302 redirect. The backend server translates this and sends back a new URI that contains the backend server port. I want to use Varnish to remove the backend server port before sending the request to the client. Varnish receives - http://domain.com:8080/blah/stuff Varnish makes it - http://domain.com/blah/stuff What's the best way to do this? I assume it involves vcl_fetch and some variants of the beresp, but I haven't been able to figure it out yet. Thanks! Chris From jlevine at iwin.com Tue Sep 14 23:43:47 2010 From: jlevine at iwin.com (Joshua Levine) Date: Tue, 14 Sep 2010 16:43:47 -0700 Subject: further syslog examples In-Reply-To: <7277B0D0-F51B-49DF-854F-3DCD1784C839@joetify.com> Message-ID: I wrap it, sending the data to a central syslog-ng for awstats and jawstats processing. ### ### I have edited /etc/init.d/varnish to pipe: ### /usr/bin/varnishncsa -f -b -P /var/run/varnishncsa-b.pid | /usr/bin/perl -w /usr/local/bin/syslog.pl local2 varnish-backend & /usr/bin/varnishncsa -f -c -P /var/run/varnishncsa-c.pid | /usr/bin/perl -w /usr/local/bin/syslog.pl local2 varnish-client & ### ### Syslog.pl is very simple: ### #!/usr/bin/perl -w use Sys::Syslog; $facility = shift; $app = shift; openlog($app, 'ndelay', $facility); while() { syslog('info', $_); # printf($_ . "\n"); } closelog(); ### ### varnish client syslog-ng.conf has ### source s_varnish_backend { file( "/var/log/varnish/varnishncsa.b.log" follow_freq(1) program_override("varnish-backend: ")); }; source s_varnish_client { file( "/var/log/varnish/varnishncsa.c.log" follow_freq(1) program_override("varnish-client: ")); }; ### ### central syslog server syslog-ng.conf has per domain processing based ### on the client header we set above, and the message body which has the ### host/domain data I parse ### filter f_hostA.domain.com-varnish-client { match(varnish-client value("PROGRAM")) and match(hostA.domain.com value("MESSAGE")) and not match (othercluster.domain.com value("HOST")); }; destination df_hostA.domain.com-varnish-client { file("/var/log/varnish/domain.com/hostA.domain.com-varnish-client.log"); }; log { source(s_remote); filter(f_hostA.domain.com-varnish-client); destination(df_hostA.domain.com-varnish-client); }; Cheers, Joshua On 9/14/10 3:59 PM, "Joe Williams" wrote: > > I would really like to get varnishlog and/or varnishncsa writing to syslog. Is > this possible? > > Thanks. > -Joe > > > On Sep 9, 2010, at 6:46 PM, Joe Williams wrote: > >> >> Anyone have examples beyond what's on the wiki of logging varnish to syslog? >> >> Thanks. >> >> -Joe >> >> >> >> Name: Joseph A. Williams >> Email: joe at joetify.com >> Blog: http://www.joeandmotorboat.com/ >> Twitter: http://twitter.com/williamsjoe >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > Name: Joseph A. Williams > Email: joe at joetify.com > Blog: http://www.joeandmotorboat.com/ > Twitter: http://twitter.com/williamsjoe > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From joe at joetify.com Tue Sep 14 23:53:19 2010 From: joe at joetify.com (Joe Williams) Date: Tue, 14 Sep 2010 16:53:19 -0700 Subject: further syslog examples In-Reply-To: References: Message-ID: Thanks Joshua and Vladimir, Currently I run varnishncsa and varnishlog using runit so I can have some log history but need to do some longer term centralized processing. I was hoping I wouldn't have to wrap it but your perl script looks simple enough, I'll give it a go. Much obliged. -Joe On Sep 14, 2010, at 4:43 PM, Joshua Levine wrote: > I wrap it, sending the data to a central syslog-ng for awstats and jawstats > processing. > > ### > ### I have edited /etc/init.d/varnish to pipe: > ### > /usr/bin/varnishncsa -f -b -P /var/run/varnishncsa-b.pid | /usr/bin/perl -w > /usr/local/bin/syslog.pl local2 varnish-backend & > > /usr/bin/varnishncsa -f -c -P /var/run/varnishncsa-c.pid | /usr/bin/perl -w > /usr/local/bin/syslog.pl local2 varnish-client & > > ### > ### Syslog.pl is very simple: > ### > #!/usr/bin/perl -w > use Sys::Syslog; > > $facility = shift; > $app = shift; > > openlog($app, 'ndelay', $facility); > > while() { > syslog('info', $_); > # printf($_ . "\n"); > } > > closelog(); > > ### > ### varnish client syslog-ng.conf has > ### > source s_varnish_backend { > file( "/var/log/varnish/varnishncsa.b.log" follow_freq(1) > program_override("varnish-backend: ")); > }; > source s_varnish_client { > file( "/var/log/varnish/varnishncsa.c.log" follow_freq(1) > program_override("varnish-client: ")); > }; > > > ### > ### central syslog server syslog-ng.conf has per domain processing based > ### on the client header we set above, and the message body which has the > ### host/domain data I parse > ### > filter f_hostA.domain.com-varnish-client { > match(varnish-client value("PROGRAM")) > and match(hostA.domain.com value("MESSAGE")) > and not match (othercluster.domain.com value("HOST")); > }; > > destination df_hostA.domain.com-varnish-client { > file("/var/log/varnish/domain.com/hostA.domain.com-varnish-client.log"); }; > log { > source(s_remote); > filter(f_hostA.domain.com-varnish-client); > destination(df_hostA.domain.com-varnish-client); > }; > > Cheers, > Joshua > > > On 9/14/10 3:59 PM, "Joe Williams" wrote: > >> >> I would really like to get varnishlog and/or varnishncsa writing to syslog. Is >> this possible? >> >> Thanks. >> -Joe >> >> >> On Sep 9, 2010, at 6:46 PM, Joe Williams wrote: >> >>> >>> Anyone have examples beyond what's on the wiki of logging varnish to syslog? >>> >>> Thanks. >>> >>> -Joe >>> >>> >>> >>> Name: Joseph A. Williams >>> Email: joe at joetify.com >>> Blog: http://www.joeandmotorboat.com/ >>> Twitter: http://twitter.com/williamsjoe >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> Name: Joseph A. Williams >> Email: joe at joetify.com >> Blog: http://www.joeandmotorboat.com/ >> Twitter: http://twitter.com/williamsjoe >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ Twitter: http://twitter.com/williamsjoe From jlevine at iwin.com Tue Sep 14 23:53:16 2010 From: jlevine at iwin.com (Joshua Levine) Date: Tue, 14 Sep 2010 16:53:16 -0700 Subject: further syslog examples In-Reply-To: Message-ID: Its all hacky, but please note, I have also added: killall /usr/bin/varnishncsa To the stop) function of the init script. Joshua On 9/14/10 4:43 PM, "Joshua Levine" wrote: > I wrap it, sending the data to a central syslog-ng for awstats and jawstats > processing. > > ### > ### I have edited /etc/init.d/varnish to pipe: > ### > /usr/bin/varnishncsa -f -b -P /var/run/varnishncsa-b.pid | /usr/bin/perl -w > /usr/local/bin/syslog.pl local2 varnish-backend & > > /usr/bin/varnishncsa -f -c -P /var/run/varnishncsa-c.pid | /usr/bin/perl -w > /usr/local/bin/syslog.pl local2 varnish-client & > > ### > ### Syslog.pl is very simple: > ### > #!/usr/bin/perl -w > use Sys::Syslog; > > $facility = shift; > $app = shift; > > openlog($app, 'ndelay', $facility); > > while() { > syslog('info', $_); > # printf($_ . "\n"); > } > > closelog(); > > ### > ### varnish client syslog-ng.conf has > ### > source s_varnish_backend { > file( "/var/log/varnish/varnishncsa.b.log" follow_freq(1) > program_override("varnish-backend: ")); > }; > source s_varnish_client { > file( "/var/log/varnish/varnishncsa.c.log" follow_freq(1) > program_override("varnish-client: ")); > }; > > > ### > ### central syslog server syslog-ng.conf has per domain processing based > ### on the client header we set above, and the message body which has the > ### host/domain data I parse > ### > filter f_hostA.domain.com-varnish-client { > match(varnish-client value("PROGRAM")) > and match(hostA.domain.com value("MESSAGE")) > and not match (othercluster.domain.com value("HOST")); > }; > > destination df_hostA.domain.com-varnish-client { > file("/var/log/varnish/domain.com/hostA.domain.com-varnish-client.log"); }; > log { > source(s_remote); > filter(f_hostA.domain.com-varnish-client); > destination(df_hostA.domain.com-varnish-client); > }; > > Cheers, > Joshua > > > On 9/14/10 3:59 PM, "Joe Williams" wrote: > >> >> I would really like to get varnishlog and/or varnishncsa writing to syslog. >> Is >> this possible? >> >> Thanks. >> -Joe >> >> >> On Sep 9, 2010, at 6:46 PM, Joe Williams wrote: >> >>> >>> Anyone have examples beyond what's on the wiki of logging varnish to syslog? >>> >>> Thanks. >>> >>> -Joe >>> >>> >>> >>> Name: Joseph A. Williams >>> Email: joe at joetify.com >>> Blog: http://www.joeandmotorboat.com/ >>> Twitter: http://twitter.com/williamsjoe >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> Name: Joseph A. Williams >> Email: joe at joetify.com >> Blog: http://www.joeandmotorboat.com/ >> Twitter: http://twitter.com/williamsjoe >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From varnish at mm.quex.org Wed Sep 15 02:56:15 2010 From: varnish at mm.quex.org (Michael Alger) Date: Wed, 15 Sep 2010 10:56:15 +0800 Subject: further syslog examples In-Reply-To: References: Message-ID: <20100915025615.GA497@grum.quex.org> On Tue, Sep 14, 2010 at 04:53:19PM -0700, Joe Williams wrote: > > Currently I run varnishncsa and varnishlog using runit so I can > have some log history but need to do some longer term centralized > processing. I was hoping I wouldn't have to wrap it but your perl > script looks simple enough, I'll give it a go. Don't forget there exists the "logger" command which will send anything sent to its stdin to syslog. No need to make a Perl script to wrap syslog... From joe at joetify.com Wed Sep 15 02:55:59 2010 From: joe at joetify.com (Joe Williams) Date: Tue, 14 Sep 2010 19:55:59 -0700 Subject: further syslog examples In-Reply-To: References: Message-ID: <27C4BEB6-4D7B-4E1E-AF87-DC0EA056D9FC@joetify.com> Joshua, Got this rolling on my systems just a few minutes ago, works perfectly. Thanks! -Joe On Sep 14, 2010, at 4:53 PM, Joshua Levine wrote: > > Its all hacky, but please note, I have also added: > > killall /usr/bin/varnishncsa > > To the stop) function of the init script. > > Joshua > > On 9/14/10 4:43 PM, "Joshua Levine" wrote: > >> I wrap it, sending the data to a central syslog-ng for awstats and jawstats >> processing. >> >> ### >> ### I have edited /etc/init.d/varnish to pipe: >> ### >> /usr/bin/varnishncsa -f -b -P /var/run/varnishncsa-b.pid | /usr/bin/perl -w >> /usr/local/bin/syslog.pl local2 varnish-backend & >> >> /usr/bin/varnishncsa -f -c -P /var/run/varnishncsa-c.pid | /usr/bin/perl -w >> /usr/local/bin/syslog.pl local2 varnish-client & >> >> ### >> ### Syslog.pl is very simple: >> ### >> #!/usr/bin/perl -w >> use Sys::Syslog; >> >> $facility = shift; >> $app = shift; >> >> openlog($app, 'ndelay', $facility); >> >> while() { >> syslog('info', $_); >> # printf($_ . "\n"); >> } >> >> closelog(); >> >> ### >> ### varnish client syslog-ng.conf has >> ### >> source s_varnish_backend { >> file( "/var/log/varnish/varnishncsa.b.log" follow_freq(1) >> program_override("varnish-backend: ")); >> }; >> source s_varnish_client { >> file( "/var/log/varnish/varnishncsa.c.log" follow_freq(1) >> program_override("varnish-client: ")); >> }; >> >> >> ### >> ### central syslog server syslog-ng.conf has per domain processing based >> ### on the client header we set above, and the message body which has the >> ### host/domain data I parse >> ### >> filter f_hostA.domain.com-varnish-client { >> match(varnish-client value("PROGRAM")) >> and match(hostA.domain.com value("MESSAGE")) >> and not match (othercluster.domain.com value("HOST")); >> }; >> >> destination df_hostA.domain.com-varnish-client { >> file("/var/log/varnish/domain.com/hostA.domain.com-varnish-client.log"); }; >> log { >> source(s_remote); >> filter(f_hostA.domain.com-varnish-client); >> destination(df_hostA.domain.com-varnish-client); >> }; >> >> Cheers, >> Joshua >> >> >> On 9/14/10 3:59 PM, "Joe Williams" wrote: >> >>> >>> I would really like to get varnishlog and/or varnishncsa writing to syslog. >>> Is >>> this possible? >>> >>> Thanks. >>> -Joe >>> >>> >>> On Sep 9, 2010, at 6:46 PM, Joe Williams wrote: >>> >>>> >>>> Anyone have examples beyond what's on the wiki of logging varnish to syslog? >>>> >>>> Thanks. >>>> >>>> -Joe >>>> >>>> >>>> >>>> Name: Joseph A. Williams >>>> Email: joe at joetify.com >>>> Blog: http://www.joeandmotorboat.com/ >>>> Twitter: http://twitter.com/williamsjoe >>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> >>> Name: Joseph A. Williams >>> Email: joe at joetify.com >>> Blog: http://www.joeandmotorboat.com/ >>> Twitter: http://twitter.com/williamsjoe >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > Name: Joseph A. Williams Email: joe at joetify.com Blog: http://www.joeandmotorboat.com/ Twitter: http://twitter.com/williamsjoe From varnish at mm.quex.org Wed Sep 15 03:11:09 2010 From: varnish at mm.quex.org (Michael Alger) Date: Wed, 15 Sep 2010 11:11:09 +0800 Subject: Multiple log files for one instance In-Reply-To: <4C8FFCD0.30102@distribion.com> References: <4C8FFCD0.30102@distribion.com> Message-ID: <20100915031109.GB497@grum.quex.org> On Tue, Sep 14, 2010 at 05:53:04PM -0500, Justin Pasher wrote: > > I want to setup a generic Varnish instance that will be used to > handle multiple web sites. All of the web sites are directed to > the same backend and the correct site is served up by Apache > using Virtual Hosts that match the corresponding Host: header. > The idea is that a new site can have caching "turned on" simply > by changing its IP address from the live server to the Varnish > server. However, by doing so, all requests come into the same > instance. This means that I cannot create separate log files for > each site using varnishncsa. > > Can anyone think of a way to do something like this? Thanks. I do this by writing a single log, and then splitting it every hour into the relevant individual logs. This might not be the most optimum way of doing it, but it was the easiest way to convert from our squid setup (where squid itself logged to multiple logfiles) without interfering with all my other scripts. The basic idea is to use varnishlog -c -a -w /var/log/client.log -D -P /var/run/varnishlog.pid and then periodically invoke another script which moves this "client.log" to a temporary directory and sends SIGHUP to the varnishlog process so it recreates a new logfile. Then, a Perl script uses varnishncsa -r /var/log/temp/client.log to read through the logfile, matching the hostname in each line and writing that to an appropriate file. After that's done, they're compressed and moved into another directory where they can be picked up by our log analysis and storage server. If you'd like to take this approach I can provide you with the scripts I'm using; written in Bash for the basic file handling and Perl for the log splitter. The limitation which will potentially hit you is my splitter script has a configuration section which defines which hostnames map to which logfile, with everything not explicitly matched going to a generic logfile. You could potentially run a varnishncsa to read the live log data, and split it on-the-fly. My system was built around an hourly collection of logfiles, so splitting it periodically hasn't made any difference to the functionality we get. From varnish at mm.quex.org Wed Sep 15 03:22:31 2010 From: varnish at mm.quex.org (Michael Alger) Date: Wed, 15 Sep 2010 11:22:31 +0800 Subject: Fixing backend URLs In-Reply-To: <2006354608710103182@unknownmsgid> References: <2006354608710103182@unknownmsgid> Message-ID: <20100915032231.GC497@grum.quex.org> On Tue, Sep 14, 2010 at 07:40:10PM -0400, Chris Cook wrote: > I have an application that takes a URI that's submitted and spits back > a different URI with a 302 redirect. The backend server translates > this and sends back a new URI that contains the backend server port. > I want to use Varnish to remove the backend server port before sending > the request to the client. > > Varnish receives - http://domain.com:8080/blah/stuff > Varnish makes it - http://domain.com/blah/stuff > > What's the best way to do this? I assume it involves vcl_fetch and > some variants of the beresp, but I haven't been able to figure it out > yet. I think you want something like this in vcl_fetch(): if (beresp.status == 301 || beresp.status == 302) { set beresp.http.Location = regsub(beresp.http.Location, "^(\w+://[^/]+):\d+", "\1"); } Seems to work okay in our UAT environment. The regex is hopefully matching any protocol specification (\w+://) and any hostname([^/]+). A colon after the hostname followed by any number of digits will then get dropped, but the request path should remain intact. Technically you can use a Location: header in other responses, so if that matters to you, you might want to change the if's to just a check if beresp.http.Location is present. From joe at joetify.com Wed Sep 15 03:24:46 2010 From: joe at joetify.com (Joe Williams) Date: Tue, 14 Sep 2010 20:24:46 -0700 Subject: further syslog examples In-Reply-To: <20100915025615.GA497@grum.quex.org> References: <20100915025615.GA497@grum.quex.org> Message-ID: <2228CE73-F23D-44C4-B257-A515C02C60D5@joetify.com> On Sep 14, 2010, at 7:56 PM, Michael Alger wrote: > On Tue, Sep 14, 2010 at 04:53:19PM -0700, Joe Williams wrote: >> >> Currently I run varnishncsa and varnishlog using runit so I can >> have some log history but need to do some longer term centralized >> processing. I was hoping I wouldn't have to wrap it but your perl >> script looks simple enough, I'll give it a go. > > Don't forget there exists the "logger" command which will send > anything sent to its stdin to syslog. No need to make a Perl script > to wrap syslog... Thanks Michael, that makes it even cleaner. -Joe From crcook at gmail.com Wed Sep 15 04:04:32 2010 From: crcook at gmail.com (Chris Cook) Date: Wed, 15 Sep 2010 00:04:32 -0400 Subject: Fixing backend URLs In-Reply-To: <20100915032231.GC497@grum.quex.org> References: <2006354608710103182@unknownmsgid> <20100915032231.GC497@grum.quex.org> Message-ID: <0FAEA775-D78B-4CD6-9C0C-972ADE57287F@gmail.com> Awesome - the regex works perfectly. I think I'll keep it with the redirect since those were the only area I was observing a suboptimal URI. Thanks! -Chris On Sep 14, 2010, at 11:22 PM, Michael Alger wrote: > On Tue, Sep 14, 2010 at 07:40:10PM -0400, Chris Cook wrote: >> I have an application that takes a URI that's submitted and spits back >> a different URI with a 302 redirect. The backend server translates >> this and sends back a new URI that contains the backend server port. >> I want to use Varnish to remove the backend server port before sending >> the request to the client. >> >> Varnish receives - http://domain.com:8080/blah/stuff >> Varnish makes it - http://domain.com/blah/stuff >> >> What's the best way to do this? I assume it involves vcl_fetch and >> some variants of the beresp, but I haven't been able to figure it out >> yet. > > I think you want something like this in vcl_fetch(): > > if (beresp.status == 301 || beresp.status == 302) > { > set beresp.http.Location = regsub(beresp.http.Location, "^(\w+://[^/]+):\d+", "\1"); > } > > Seems to work okay in our UAT environment. The regex is hopefully > matching any protocol specification (\w+://) and any hostname([^/]+). > A colon after the hostname followed by any number of digits will > then get dropped, but the request path should remain intact. > > Technically you can use a Location: header in other responses, so if > that matters to you, you might want to change the if's to just a > check if beresp.http.Location is present. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From mohanrao.01 at gmail.com Wed Sep 15 13:25:44 2010 From: mohanrao.01 at gmail.com (Mohan Rao) Date: Wed, 15 Sep 2010 09:25:44 -0400 Subject: Probing to see if Varnish is Alive. Message-ID: Hi, Is there a way to probe a varnish server to see if it is alive ? We have a couple of varnish servers that are load balanced. We would like the load balancer to stop sending traffic to a particular varnish server if we are updating that server. Is there a recommended way to do this ? Thanks & Regards, Mohan -------------- next part -------------- An HTML attachment was scrubbed... URL: From samcrawford at gmail.com Wed Sep 15 13:29:56 2010 From: samcrawford at gmail.com (samcrawford at gmail.com) Date: Wed, 15 Sep 2010 13:29:56 +0000 Subject: Probing to see if Varnish is Alive. In-Reply-To: References: Message-ID: <1303611368-1284557397-cardhu_decombobulator_blackberry.rim.net-358359348-@bda2055.bisx.produk.on.blackberry> In the past we've created a rule like this: if (req.url == "/varnish_alive.html") { error 200 "I am alive"; } We then have the load balancer use /varnish_alive.html and check for a http 200 response code. Thanks, Sam Sent from my BlackBerry? wireless device -----Original Message----- From: Mohan Rao Sender: varnish-misc-bounces at varnish-cache.org Date: Wed, 15 Sep 2010 09:25:44 To: Subject: Probing to see if Varnish is Alive. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From scaunter at topscms.com Wed Sep 15 14:55:04 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Wed, 15 Sep 2010 10:55:04 -0400 Subject: Probing to see if Varnish is Alive. In-Reply-To: References: Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C42C1F4@TMG-EVS02.torstar.net> If your load balancer can do a port 80 check (or whatever port you are using), you can block port 80 with pf to stop traffic flow, do your updating, and return it to service by unblocking the port. Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Mohan Rao Sent: September-15-10 9:26 AM To: varnish-misc at varnish-cache.org Subject: Probing to see if Varnish is Alive. Hi, Is there a way to probe a varnish server to see if it is alive ? We have a couple of varnish servers that are load balanced. We would like the load balancer to stop sending traffic to a particular varnish server if we are updating that server. Is there a recommended way to do this ? Thanks & Regards, Mohan -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.chiswell at mangahigh.com Wed Sep 15 14:59:11 2010 From: richard.chiswell at mangahigh.com (Richard Chiswell) Date: Wed, 15 Sep 2010 15:59:11 +0100 Subject: Probing to see if Varnish is Alive. In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C42C1F4@TMG-EVS02.torstar.net> References: <7F0AA702B8A85A4A967C4C8EBAD6902C42C1F4@TMG-EVS02.torstar.net> Message-ID: <4C90DF3F.4010703@mangahigh.com> I've got our load balancer requesting a MySQL driven PHP page via Apache and via Varnish. Every second, our load balancer probes a page on our site. This page is set to "pass" through Varnish to Apache which then asks PHP to check MySQL. If all is ok, the page returns the text "All checks passed" - if that text isn't there within 0.5s (for example, Varnish is unavailable, Apache is unavailable, MySQL is unavailable etc), then that server is removed from our load balanced pool. If we do maintenance, we just kill Varnish (our site is still accessible to ourselves via port 8080 which the Apache port), bring Varnish back up and then wait a couple of seconds for the load balancer to notice the "All checks passed" text is being returned. Rich On 15/09/2010 15:55, Caunter, Stefan wrote: > > If your load balancer can do a port 80 check (or whatever port you are > using), you can block port 80 with pf to stop traffic flow, do your > updating, and return it to service by unblocking the port. > > Stefan Caunter :: Senior Systems Administrator :: TOPS > > e: scaunter at topscms.com :: m: (416) 561-4871 > > www.thestar.com www.topscms.com > > *From:* varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] *On Behalf Of *Mohan Rao > *Sent:* September-15-10 9:26 AM > *To:* varnish-misc at varnish-cache.org > *Subject:* Probing to see if Varnish is Alive. > > Hi, > > Is there a way to probe a varnish server to see if it is alive ? We > have a couple of varnish servers that are load balanced. We would like > the load balancer to stop sending traffic to a particular varnish > server if we are updating that server. Is there a recommended way to > do this ? > > Thanks & Regards, > > Mohan > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal.taborsky at nrholding.com Wed Sep 15 15:39:39 2010 From: michal.taborsky at nrholding.com (Michal Taborsky - Netretail Holding) Date: Wed, 15 Sep 2010 17:39:39 +0200 Subject: Probing to see if Varnish is Alive. In-Reply-To: References: Message-ID: <4C90E8BB.7080107@nrholding.com> Hi Mohan, here is how we do it. We have also a couple of varnishes sitting in front of several backend servers. Each backend server responds to a simple http request on URL http://1.2.3.4/ping (1.2.3.4 being the backend server IP). We use this to check the backend server health in our VCL (in the probe statement). We also use this URL however to check the health of the varnish servers too. We just configured our load balancer to access the varnish URL (http://5.6.7.8/ping, 5.6.7.8 being the varnish IP address) and check it's http status code. Now, the load balancer will stop sending traffic to the varnish server either when it is down itself or when all it's backends are down, which is essentially the same thing, because it means the varnish cannot serve traffic. We are using restart in the VCL to make sure that single backend failure will not trigger this. If one backend goes down, the ping request is served by different live backend. Does it make sense? Michal Dne 15.9.2010 15:25, Mohan Rao napsal(a): > Hi, > Is there a way to probe a varnish server to see if it is alive ? We > have a couple of varnish servers that are load balanced. We would like > the load balancer to stop sending traffic to a particular varnish > server if we are updating that server. Is there a recommended way to > do this ? > Thanks & Regards, > Mohan > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -- Michal T?borsk? chief systems architect Netretail Holding, B.V. http://www.nrholding.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinp at distribion.com Wed Sep 15 17:33:58 2010 From: justinp at distribion.com (Justin Pasher) Date: Wed, 15 Sep 2010 12:33:58 -0500 Subject: Multiple log files for one instance In-Reply-To: <20100915031109.GB497@grum.quex.org> References: <4C8FFCD0.30102@distribion.com> <20100915031109.GB497@grum.quex.org> Message-ID: <4C910386.8010605@distribion.com> ----- Original Message ----- > From: Michael Alger > Date: Wed, 15 Sep 2010 11:11:09 +0800 > Subject: Re: Multiple log files for one instance > To: varnish-misc at varnish-cache.org > > > I do this by writing a single log, and then splitting it every hour > into the relevant individual logs. This might not be the most > optimum way of doing it, but it was the easiest way to convert from > our squid setup (where squid itself logged to multiple logfiles) > without interfering with all my other scripts. > ... > If you'd like to take this approach I can provide you with the > scripts I'm using; written in Bash for the basic file handling and > Perl for the log splitter. The limitation which will potentially hit > you is my splitter script has a configuration section which defines > which hostnames map to which logfile, with everything not explicitly > matched going to a generic logfile. > That sounds like something I can make work. If you don't mind, could you send your scripts over? That will save me a little time from throwing something together. Thanks. -- Justin Pasher From lbjay at reallywow.com Wed Sep 15 18:14:58 2010 From: lbjay at reallywow.com (Jay Luker) Date: Wed, 15 Sep 2010 14:14:58 -0400 Subject: Dealing with "http format error" Message-ID: I have a backend that I do not control that is causing an error in varnish due (I think) to a lack of http headers in the response. The URL in question is http://bit.ly/c8AXGI (shortened). If you look you'll see that the http response consists of "HTTP/1.0 200 OK" and that's all. No content-type, content-length, etc. This results in the following varnishlog output: Results 11 FetchError c http format error 12 BackendClose b ned 11 VCL_call c error ... My question is, since I don't have any control of this backend, is there some VCL magic I can use to fudge whatever missing headers are causing varnish to spit the bit on the response? I have already unsuccessfully tried the only obvious thing I could think of, which was to add to vcl_fetch: set obj.http.Content-Type = "text/plain; charset=ascii"; set obj.http.Content-Length = "100000"; set obj.http.Content-Encoding = "none"; I'm using v2.06. Thanks, --jay From engstaff at virtacore.com Fri Sep 17 20:36:09 2010 From: engstaff at virtacore.com (eng) Date: Fri, 17 Sep 2010 16:36:09 -0400 (EDT) Subject: Nagios Plugin Message-ID: <1460142507.63995.1284755769398.JavaMail.root@mail.dtgmail.com> Hello, Currently I am looking to install the varnish nagios plugin into XI and during the compiling of the plugin I have been running into some issues as seen below, grabbed source svn branch from 2.1x varnish-tools/nagios . Tried trunk as well w/ same results. Current version of varnish running 2.1 all from rpm install on CentOS 5. If anyone has any advice that would be greatly appreciated . varnish.x86_64 2.1.2-1.el5 installed varnish-debuginfo.x86_64 2.1.2-1.el5 installed varnish-libs.x86_64 2.1.2-1.el5 installed varnish-libs-devel.x86_64 2.1.2-1.el5 installed make all-am make[1]: Entering directory `/root/nagios-test' if gcc -DHAVE_CONFIG_H -I. -I. -I. -include config.h -I/usr/include/varnish -g -O2 -MT check_varnish-check_varnish.o -MD -MP -MF ".deps/check_varnish-check_varnish.Tpo" -c -o check_varnish-check_varnish.o `test -f 'check_varnish.c' || echo './'`check_varnish.c; \ then mv -f ".deps/check_varnish-check_varnish.Tpo" ".deps/check_varnish-check_varnish.Po"; else rm -f ".deps/check_varnish-check_varnish.Tpo"; exit 1; fi check_varnish.c: In function ?check_stats?: check_varnish.c:183: error: ?const struct vsc_main? has no member named ?sm_balloc? check_varnish.c:183: error: ?const struct vsc_main? has no member named ?sm_bfree? check_varnish.c:185: error: ?const struct vsc_main? has no member named ?sm_balloc? check_varnish.c: In function ?main?: check_varnish.c:252: warning: assignment makes pointer from integer without a cast check_varnish.c:285: warning: assignment makes pointer from integer without a cast make[1]: *** [check_varnish-check_varnish.o] Error 1 make[1]: Leaving directory `/root/nagios-test' make: *** [all] Error 2 Thanks Jason From simon at darkmere.gen.nz Sat Sep 18 03:58:22 2010 From: simon at darkmere.gen.nz (Simon Lyall) Date: Sat, 18 Sep 2010 15:58:22 +1200 (NZST) Subject: NYT: Varnish interview Message-ID: Jacob Harris from the New York times is interviewed here. About 80% of the interview is about how they use varnish: http://webpulp.tv/post/1049061452/new-york-times-jacob-harris Some more information here including VCL scripts. http://open.blogs.nytimes.com/2010/09/15/using-varnish-so-news-doesnt-break-your-server/ Only just picked up from the interview that ESI doesn't work on gzipped pages. -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. From tfheen at varnish-software.com Mon Sep 20 06:29:31 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 20 Sep 2010 08:29:31 +0200 Subject: Nagios Plugin In-Reply-To: <1460142507.63995.1284755769398.JavaMail.root@mail.dtgmail.com> (eng's message of "Fri, 17 Sep 2010 16:36:09 -0400 (EDT)") References: <1460142507.63995.1284755769398.JavaMail.root@mail.dtgmail.com> Message-ID: <878w2xorlg.fsf@qurzaw.linpro.no> ]] eng | Currently I am looking to install the varnish nagios plugin into XI and during the compiling of the plugin I have been running into some issues as seen below, grabbed source svn branch from 2.1x varnish-tools/nagios . Tried trunk as well w/ same results. Current version of varnish running 2.1 all from rpm install on CentOS 5. If anyone has any advice that would be greatly appreciated . It's known broken, I'm working on fixing it. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From j.vanarragon at lukkien.com Tue Sep 21 07:30:23 2010 From: j.vanarragon at lukkien.com (Jaap van Arragon) Date: Tue, 21 Sep 2010 09:30:23 +0200 Subject: Cachefile full Message-ID: Hello, Last Friday we?ve experienced a serious problem with Varnish. We cache a lot of images for our costumers and last Friday we saw that the varnish server wouldn?t serve any images. Is it possible that our 10 GB cachefile is full? Doesn?t Varnish, when it reach it?s limit (10GB), just throw out the oldest images present in the cache file? Thank you. Grt Jaap van Arragon -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue Sep 21 07:39:11 2010 From: perbu at varnish-software.com (Per Buer) Date: Tue, 21 Sep 2010 09:39:11 +0200 Subject: Cachefile full In-Reply-To: References: Message-ID: On Tue, Sep 21, 2010 at 9:30 AM, Jaap van Arragon wrote: > Hello, > > Last Friday we?ve experienced a serious problem with Varnish. We cache a > lot of images for our costumers and last Friday we saw that the varnish > server wouldn?t serve any images. Is it possible that our 10 GB cachefile is > full? Doesn?t Varnish, when it reach it?s limit (10GB), just throw out the > oldest images present in the cache file? > Yes. It does. There are some counters showing the LRU (last-recently-used, the algorithm that is used to decide what to kick out) at work - "varnishstat -1 |grep lru" shoud show you something. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From scaunter at topscms.com Tue Sep 21 14:16:47 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Tue, 21 Sep 2010 10:16:47 -0400 Subject: Varnish interview In-Reply-To: References: Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C42C816@TMG-EVS02.torstar.net> It is an interesting blog post, but it incorrectly implies that varnish cache timing is controlled by the Cache-control header; client browser behaviour is controlled by this header, not varnish. The time to live for objects is set in fetch, with additional timing settings accessible using grace. Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Simon Lyall Sent: September-17-10 11:58 PM To: varnish-misc at varnish-cache.org Subject: NYT: Varnish interview Jacob Harris from the New York times is interviewed here. About 80% of the interview is about how they use varnish: http://webpulp.tv/post/1049061452/new-york-times-jacob-harris Some more information here including VCL scripts. http://open.blogs.nytimes.com/2010/09/15/using-varnish-so-news-doesnt-br eak-your-server/ Only just picked up from the interview that ESI doesn't work on gzipped pages. -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From kristian at varnish-software.com Tue Sep 21 15:13:30 2010 From: kristian at varnish-software.com (=?UTF-8?Q?Kristian_Lyngst=C3=B8l?=) Date: Tue, 21 Sep 2010 17:13:30 +0200 Subject: Varnish interview In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C42C816@TMG-EVS02.torstar.net> References: <7F0AA702B8A85A4A967C4C8EBAD6902C42C816@TMG-EVS02.torstar.net> Message-ID: On Tue, Sep 21, 2010 at 4:16 PM, Caunter, Stefan wrote: > It is an interesting blog post, but it incorrectly implies that varnish > cache timing is controlled by the Cache-control header; client browser > behaviour is controlled by this header, not varnish. Varnish reads the Cache-Control header and uses s-maxage, max-age or Expires: as the TTL. What you set in vcl_fetch _overrides_ that. If you do not modify ttl in vcl_fetch, Varnish will use the Cache-Control header. - Kristian From scaunter at topscms.com Tue Sep 21 15:19:17 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Tue, 21 Sep 2010 11:19:17 -0400 Subject: Varnish interview In-Reply-To: References: <7F0AA702B8A85A4A967C4C8EBAD6902C42C816@TMG-EVS02.torstar.net> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C42C867@TMG-EVS02.torstar.net> -----Original Message----- From: Kristian Lyngst?l [mailto:kristian at varnish-software.com] Sent: September-21-10 11:14 AM To: Caunter, Stefan Cc: Simon Lyall; varnish-misc at varnish-cache.org Subject: Re: Varnish interview On Tue, Sep 21, 2010 at 4:16 PM, Caunter, Stefan wrote: > It is an interesting blog post, but it incorrectly implies that varnish > cache timing is controlled by the Cache-control header; client browser > behaviour is controlled by this header, not varnish. > Varnish reads the Cache-Control header and uses s-maxage, max-age or > Expires: as the TTL. What you set in vcl_fetch _overrides_ that. If > you do not modify ttl in vcl_fetch, Varnish will use the Cache-Control > header. Thanks for the clarification. Our webapp does not set it, so I had not seen this behaviour. Am I correct that we would see a two minute ttl in the absence of those headers? - Stef From kristian at varnish-software.com Tue Sep 21 15:35:17 2010 From: kristian at varnish-software.com (=?UTF-8?Q?Kristian_Lyngst=C3=B8l?=) Date: Tue, 21 Sep 2010 17:35:17 +0200 Subject: Varnish interview In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C42C867@TMG-EVS02.torstar.net> References: <7F0AA702B8A85A4A967C4C8EBAD6902C42C816@TMG-EVS02.torstar.net> <7F0AA702B8A85A4A967C4C8EBAD6902C42C867@TMG-EVS02.torstar.net> Message-ID: On Tue, Sep 21, 2010 at 5:19 PM, Caunter, Stefan wrote: >> Varnish reads the Cache-Control header and uses s-maxage, max-age or >> Expires: as the TTL. What you set in vcl_fetch _overrides_ that. If >> you do not modify ttl in vcl_fetch, Varnish will use the Cache-Control >> header. > > Thanks for the clarification. Our webapp does not set it, so I had not seen this behaviour. > > Am I correct that we would see a two minute ttl in the absence of those headers? Yes. In the absence of s-maxage, max-age and Expires, the default_ttl is used, which is 120 seconds by default. In varnishlog, you will see a line saying TTL: if it says TTL RFC, it's based on headers or default TTL. If it says VCL, it's set in VCL. - Kristian From marc.fournier at camptocamp.com Tue Sep 21 17:19:55 2010 From: marc.fournier at camptocamp.com (Marc Fournier) Date: Tue, 21 Sep 2010 19:19:55 +0200 Subject: Nagios Plugin References: <1460142507.63995.1284755769398.JavaMail.root@mail.dtgmail.com> Message-ID: <20100921191955.1684472f@lonquimay.wrk.lsn.camptocamp.com> Hello, On Fri, 17 Sep 2010 16:36:09 -0400 (EDT) eng wrote: > Currently I am looking to install the varnish nagios plugin into XI > and during the compiling of the plugin I have been running into some > issues as seen below, grabbed source svn branch from 2.1x > varnish-tools/nagios . Tried trunk as well w/ same results. Current > version of varnish running 2.1 all from rpm install on CentOS 5. If > anyone has any advice that would be greatly appreciated . With 2.1.2, I was successful building revision 4009 of the plugin. Same with 2.1.3, but I had to add a couple of options to make: make VARNISHAPI_LIBS='-lvarnishapi -lvarnish -lvarnishcompat Tollef, if you're working on this, maybe this could help you understand the chronology of changes: http://www.varnish-cache.org/trac/ticket/710#comment:3 Thanks for your work ! Cheers, Marc From lbjay at reallywow.com Tue Sep 21 20:38:30 2010 From: lbjay at reallywow.com (Jay Luker) Date: Tue, 21 Sep 2010 16:38:30 -0400 Subject: Dealing with "http format error" In-Reply-To: References: Message-ID: After continuing to debug my problem I've had to reassess what the actual issue is. Again, the request URL I'm trying to proxy is this: http://nedwww.ipac.caltech.edu/cgi-bin/nph-objsearch?z_unit=z&z_value1=&z_value2=&radius=9.99996&zv_breaker=30000.0&hconst=73&omegam=0.27&out_equinox=J2000.0&list_limit=5&in_equinox=J2000.0&lon=05+23+34.6&in_csys=Equatorial&omegav=0.73&obj_sort=Distance+to+search+center&corr_z=1&out_csys=Equatorial&img_stamp=NO&lat=-69+45+22&ot_include=ANY&of=ascii_bar&nmp_op=ANY&z_constraint=Unconstrained&search_type=Near+Position+Search It appears that because this backend does not provide any http headers in the response that varnish is interpreting part of the response itself as the protocol, status code, etc. The log output below indicates that the RxProtocol, RxStatus, RxResponse and RxHeader values actually contain strings from what should be the body of the response. """ 12 TxRequest b GET 12 TxURL b http://nedwww.ipac.caltech.edu/cgi-bin/nph-objsearch?z_unit=z&z_value1=&z_value2=&radius=9.99996 &zv_breaker=30000.0&hconst=73&omegam=0.27&out_equinox=J2000.0&list_limit=5&in_equinox=J2000.0&lon=05+23+34.6&in_csys= Equatorial&omegav=0.73&obj_sort=Distance+t 12 TxProtocol b HTTP/1.1 12 TxHeader b Host: nedwww.ipac.caltech.edu 12 TxHeader b User-agent: Python-urllib/1.16 12 TxHeader b X-Varnish: 1646987917 12 TxHeader b X-Forwarded-For: 131.142.185.41 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1285100629 1.0 12 RxProtocol b Results 12 RxStatus b from 12 RxResponse b query to NASA/IPAC Extragalactic Database (NED), 12 RxHeader b which is operated by the Jet Propulsion Laboratory, California Institute of 12 RxHeader b Technology, under contract with the National Aeronautics and Space Administration. 12 RxHeader b This work was (partially) supported by the US National Virtual Observatory 12 RxHeader b development project, which is funded by the National Science Foundation 12 RxHeader b under cooperative agreement AST0122449 with The Johns Hopkins University. 12 HttpGarbage b Results 11 FetchError c http format error 12 BackendClose b ned 11 VCL_call c error 11 VCL_return c deliver 11 Length c 488 11 VCL_call c deliver 11 VCL_return c deliver 11 TxProtocol c HTTP/1.1 11 TxStatus c 503 """ Can someone tell me if it's possible to correct this response prior to the point varnish tries to interpret it? vcl_fetch appears to be too late in the process. Thanks, --jay On Wed, Sep 15, 2010 at 2:14 PM, Jay Luker wrote: > I have a backend that I do not control that is causing an error in > varnish due (I think) to a lack of http headers in the response. The > URL in question is http://bit.ly/c8AXGI (shortened). If you look > you'll see that the http response consists of "HTTP/1.0 200 OK" and > that's all. No content-type, content-length, etc. This results in the > following varnishlog output: > > Results > ? 11 FetchError ? c http format error > ? 12 BackendClose b ned > ? 11 VCL_call ? ? c error > ? ... > > My question is, since I don't have any control of this backend, is > there some VCL magic I can use to fudge whatever missing headers are > causing varnish to spit the bit on the response? I have already > unsuccessfully tried the only obvious thing I could think of, which > was to add to vcl_fetch: > > ? ?set obj.http.Content-Type = "text/plain; charset=ascii"; > ? ?set obj.http.Content-Length = "100000"; > ? ?set obj.http.Content-Encoding = "none"; > > I'm using v2.06. > > Thanks, > --jay > From gmoniey at gmail.com Wed Sep 22 05:13:27 2010 From: gmoniey at gmail.com (.) Date: Tue, 21 Sep 2010 22:13:27 -0700 Subject: Question about Varnish Purge Message-ID: Hi, I use the Varnish purge command quite heavily (through this ruby gem: http://github.com/schoefmax/klarlack), and I heard that there is a bug with the way Varnish handles it's purge list. I heard that the purge list is never actually cleared, and can grow to be really large over time. I use the purge quite a bit (which looks something like: purge ^/users). I searched the tickets, and didn't see any mention of this, so I was wondering if this is even an issue? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Wed Sep 22 06:40:54 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 22 Sep 2010 08:40:54 +0200 Subject: Dealing with "http format error" In-Reply-To: (Jay Luker's message of "Tue, 21 Sep 2010 16:38:30 -0400") References: Message-ID: <87aana2scp.fsf@qurzaw.linpro.no> ]] Jay Luker | After continuing to debug my problem I've had to reassess what the | actual issue is. Again, the request URL I'm trying to proxy is this: | | http://nedwww.ipac.caltech.edu/cgi-bin/nph-objsearch?z_unit=z&z_value1=&z_value2=&radius=9.99996&zv_breaker=30000.0&hconst=73&omegam=0.27&out_equinox=J2000.0&list_limit=5&in_equinox=J2000.0&lon=05+23+34.6&in_csys=Equatorial&omegav=0.73&obj_sort=Distance+to+search+center&corr_z=1&out_csys=Equatorial&img_stamp=NO&lat=-69+45+22&ot_include=ANY&of=ascii_bar&nmp_op=ANY&z_constraint=Unconstrained&search_type=Near+Position+Search | | It appears that because this backend does not provide any http headers | in the response that varnish is interpreting part of the response | itself as the protocol, status code, etc. The log output below | indicates that the RxProtocol, RxStatus, RxResponse and RxHeader | values actually contain strings from what should be the body of the | response. Seems like your backend is speaking HTTP 0.9, which I'm not sure we support. Is there any chance you could get it to be less archaic? -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From lbjay at reallywow.com Wed Sep 22 13:36:33 2010 From: lbjay at reallywow.com (Jay Luker) Date: Wed, 22 Sep 2010 09:36:33 -0400 Subject: Dealing with "http format error" In-Reply-To: <87aana2scp.fsf@qurzaw.linpro.no> References: <87aana2scp.fsf@qurzaw.linpro.no> Message-ID: On Wed, Sep 22, 2010 at 2:40 AM, Tollef Fog Heen wrote: > | It appears that because this backend does not provide any http headers > | in the response that varnish is interpreting part of the response > | itself as the protocol, status code, etc. The log output below > | indicates that the RxProtocol, RxStatus, RxResponse and RxHeader > | values actually contain strings from what should be the body of the > | response. > > Seems like your backend is speaking HTTP 0.9, which I'm not sure we > support. ?Is there any chance you could get it to be less archaic? Unfortunately, the backend is not something I have any control over. There was an offlist suggestion to add a haproxy instance between varnish and my backend for the purpose of inserting headers. I may have to pursue something like that. --jay From j.vanarragon at lukkien.com Wed Sep 22 14:13:28 2010 From: j.vanarragon at lukkien.com (Jaap van Arragon) Date: Wed, 22 Sep 2010 16:13:28 +0200 Subject: Cachefile full In-Reply-To: Message-ID: Is there a way to see why Varnish stopped accepting and/or serving images from his cache file? Thanks. On 9/21/10 9:39 AM, "Per Buer" wrote: > > On Tue, Sep 21, 2010 at 9:30 AM, Jaap van Arragon > wrote: >> Hello, >> >> Last Friday we?ve experienced a serious problem with Varnish. We cache a lot >> of images for our costumers and last Friday we saw that the varnish server >> wouldn?t serve any images. Is it possible that our 10 GB cachefile is full? >> Doesn?t Varnish, when it reach it?s limit (10GB), just throw out the oldest >> images present in the cache file? > > Yes. It does. There are some counters showing the LRU (last-recently-used, the > algorithm that is used to decide what to kick out)? > at work - "varnishstat -1 |grep lru" shoud show you something. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hc at codecompany.dk Thu Sep 23 06:25:04 2010 From: hc at codecompany.dk (HC Saustrup) Date: Thu, 23 Sep 2010 08:25:04 +0200 Subject: Dealing with "http format error" In-Reply-To: References: <87aana2scp.fsf@qurzaw.linpro.no> Message-ID: <4C9AF2C0.6050003@codecompany.dk> On 9/22/10 3:36 PM, Jay Luker wrote: > On Wed, Sep 22, 2010 at 2:40 AM, Tollef Fog Heen > wrote: > >> | It appears that because this backend does not provide any http headers >> | in the response that varnish is interpreting part of the response >> | itself as the protocol, status code, etc. The log output below >> | indicates that the RxProtocol, RxStatus, RxResponse and RxHeader >> | values actually contain strings from what should be the body of the >> | response. >> >> Seems like your backend is speaking HTTP 0.9, which I'm not sure we >> support. Is there any chance you could get it to be less archaic? >> > I tested the URL on my laptop out of curiosity, and it turns out that my transparent Squid proxy can fill in the missing headers (I kinda forgot I still had it running): $ curl -D - -o /dev/null --silent 'http://nedwww.ipac.caltech.edu/cgi-bin/nph-objsearch?z_unit=z&z_value1=&z_value2=&radius=9.99996&zv_breaker=30000.0&hconst=73&omegam=0.27&out_equinox=J2000.0&list_limit=5&in_equinox=J2000.0&lon=05+23+34.6&in_csys=Equatorial&omegav=0.73&obj_sort=Distance+to+search+center&corr_z=1&out_csys=Equatorial&img_stamp=NO&lat=-69+45+22&ot_include=ANY&of=ascii_bar&nmp_op=ANY&z_constraint=Unconstrained&search_type=Near+Position+Search' HTTP/1.0 200 OK Server: squid/3.1.4 Mime-Version: 1.0 Date: Thu, 23 Sep 2010 06:24:00 GMT X-Transformed-From: HTTP/0.9 X-Cache: MISS from server.lan X-Cache-Lookup: MISS from server.lan:3128 Via: 1.0 server.lan (squid/3.1.4) Connection: close Seems like a lot of hoops to jump through, but apparently a Squid in between will fix the problem. Cheers, HC From Candace.Copper at goAAA.com Thu Sep 23 16:21:43 2010 From: Candace.Copper at goAAA.com (Copper, Candace L) Date: Thu, 23 Sep 2010 09:21:43 -0700 Subject: Varnish w/hardware loadbalancing woes Message-ID: <666C0AC37B26164FB0EFBD83F4631A09560925A2@P01IMW302.ent.rt.csaa.com> I've looked all over for information on how to configure varnish to use a VIP and have not been able to find much - so I'm hoping someone here can assist. Our setup goes like this: Hardware Loadbalancer (sticky sessions enabled) with one VIP(perf-drupal.domain.com) directs to three Apache identical webservers (each running Varnish - (perf-drupal1.domain.com, perf-drupal2.domain.com & perf-drupal3.domain.com)). With the VCL that we currently have, it works like a charm on each individual server, but when we try to use the VIP we get errors (50% of the time) when trying to access the site, stating that it is not available. I've tried: backend newsite { .host = "localhost"; .port = "8080"; } set req.http.host = "perf-drupal.domain.com"; set req.backend = newsite; and using the default backend: backend default { .host = "127.0.0.1"; .port = "8080"; } I've read about the DNS Director, but that's not available in the version I have installed - 2.1.3. But since we are only running one site, I don't know if it will help. I don't have access to any additional hardware, so I cannot split out Varnish from the Apache web server. Any assistance would be greatly appreciated! :) Candace Copper -------------- next part -------------- An HTML attachment was scrubbed... URL: From mloftis at wgops.com Thu Sep 23 17:57:18 2010 From: mloftis at wgops.com (Michael Loftis) Date: Thu, 23 Sep 2010 11:57:18 -0600 Subject: Varnish w/hardware loadbalancing woes In-Reply-To: <666C0AC37B26164FB0EFBD83F4631A09560925A2@P01IMW302.ent.rt.csaa.com> References: <666C0AC37B26164FB0EFBD83F4631A09560925A2@P01IMW302.ent.rt.csaa.com> Message-ID: Sounds more like a load balancer setup problem than a varnish issue. Under Linux atleast with DR setups you must disable ARP for the VIP, unless you're using a tunnel setup or a setup where your VIPs are completely separate VLAN/LAN. It sounds like this is what's happening, that the real servers are receiving the traffic sometimes and the load balancer at others because of the ARP issue. --On Thursday, September 23, 2010 9:21 AM -0700 "Copper, Candace L" wrote: > > > I've looked all over for information on how to configure varnish to use > a VIP and have not been able to find much ? so I'm hoping someone > here can assist. > > > > Our setup goes like this: Hardware Loadbalancer (sticky sessions enabled) > with one VIP(perf-drupal.domain.com) directs to three Apache identical > webservers (each running Varnish ? (perf-drupal1.domain.com, > perf-drupal2.domain.com & perf-drupal3.domain.com)). With the VCL that we > currently have, it works like a charm on each individual server, but when > we try to use the VIP we get errors (50% of the time) when trying to > access the site, stating that it is not available. > > > > I've tried: > > > > backend newsite { > > .host = "localhost"; > > .port = "8080"; > > } > > set req.http.host = "perf-drupal.domain.com"; > > set req.backend = newsite; > > > > and using the default backend: > > > > backend default { > > .host = "127.0.0.1"; > > .port = "8080"; > > } > > > > I've read about the DNS Director, but that's not available in the > version I have installed - 2.1.3. But since we are only running one site, > I don't know if it will help. I don't have access to any additional > hardware, so I cannot split out Varnish from the Apache web server. > > > > Any assistance would be greatly appreciated! J > > > > Candace Copper > > > From Candace.Copper at goAAA.com Thu Sep 23 23:23:00 2010 From: Candace.Copper at goAAA.com (Copper, Candace L) Date: Thu, 23 Sep 2010 16:23:00 -0700 Subject: Varnish w/hardware loadbalancing woes In-Reply-To: References: <666C0AC37B26164FB0EFBD83F4631A09560925A2@P01IMW302.ent.rt.csaa.com> Message-ID: <666C0AC37B26164FB0EFBD83F4631A0956092C1D@P01IMW302.ent.rt.csaa.com> Michael, had the ARP disabled on the VIP and then the site wouldn't load at all with the VIP and still saw the Unable to connect errors. We re-enabled ARP and then our site still wouldn't load, and we couldn't kill any httpd processes and had to reboot. Just weird stuff happened with that. Candace Copper -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Michael Loftis Sent: Thursday, September 23, 2010 10:57 AM To: 'varnish-misc at varnish-cache.org' Subject: Re: Varnish w/hardware loadbalancing woes Sounds more like a load balancer setup problem than a varnish issue. Under Linux atleast with DR setups you must disable ARP for the VIP, unless you're using a tunnel setup or a setup where your VIPs are completely separate VLAN/LAN. It sounds like this is what's happening, that the real servers are receiving the traffic sometimes and the load balancer at others because of the ARP issue. --On Thursday, September 23, 2010 9:21 AM -0700 "Copper, Candace L" wrote: > > > I've looked all over for information on how to configure varnish to use > a VIP and have not been able to find much ? so I'm hoping someone > here can assist. > > > > Our setup goes like this: Hardware Loadbalancer (sticky sessions enabled) > with one VIP(perf-drupal.domain.com) directs to three Apache identical > webservers (each running Varnish ? (perf-drupal1.domain.com, > perf-drupal2.domain.com & perf-drupal3.domain.com)). With the VCL that we > currently have, it works like a charm on each individual server, but when > we try to use the VIP we get errors (50% of the time) when trying to > access the site, stating that it is not available. > > > > I've tried: > > > > backend newsite { > > .host = "localhost"; > > .port = "8080"; > > } > > set req.http.host = "perf-drupal.domain.com"; > > set req.backend = newsite; > > > > and using the default backend: > > > > backend default { > > .host = "127.0.0.1"; > > .port = "8080"; > > } > > > > I've read about the DNS Director, but that's not available in the > version I have installed - 2.1.3. But since we are only running one site, > I don't know if it will help. I don't have access to any additional > hardware, so I cannot split out Varnish from the Apache web server. > > > > Any assistance would be greatly appreciated! J > > > > Candace Copper > > > _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From mloftis at wgops.com Thu Sep 23 23:45:26 2010 From: mloftis at wgops.com (Michael Loftis) Date: Thu, 23 Sep 2010 17:45:26 -0600 Subject: Varnish w/hardware loadbalancing woes In-Reply-To: <666C0AC37B26164FB0EFBD83F4631A0956092C1D@P01IMW302.ent.rt.csaa.com> References: <666C0AC37B26164FB0EFBD83F4631A09560925A2@P01IMW302.ent.rt.csaa.com> <666C0AC37B26164FB0EFBD83F4631A0956092C1D@P01IMW302.ent.rt.csaa.com> Message-ID: <91175C6D206E8D3B723BC0FF@[192.168.66.199]> Well if you could diagram or walk-through your specific setup and configs I can certainly help you (I've deployed probably a couple hundred or so various load balanced applications using both hardware and software LB approaches, and with approaches with proxies like varnish too) The *only* machine directly receiving traffic for a VIP should be the hardware load balancer itself. It will then decide which (real) machine (in your case apparently running varnish on top of ?) to send the traffic on to. If the real machines are answering for the VIP because they're all sharing a VLAN/LAN then you're going to get failures randomly. Generally the VIP is configured on a loopback device as a /32 (or /128 in IPv6) -- return traffic can either be DR or via the LB itself. in the latter you might be doing NAT in which case the real machines won't have the VIP configured at all. In a DR situation the real machines have to have the VIP because they'll send all traffic back to their (default) router, not the load balancer. --On Thursday, September 23, 2010 4:23 PM -0700 "Copper, Candace L" wrote: > Michael, had the ARP disabled on the VIP and then the site wouldn't load > at all with the VIP and still saw the Unable to connect errors. We > re-enabled ARP and then our site still wouldn't load, and we couldn't > kill any httpd processes and had to reboot. > > Just weird stuff happened with that. > > Candace Copper > > > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Michael > Loftis Sent: Thursday, September 23, 2010 10:57 AM > To: 'varnish-misc at varnish-cache.org' > Subject: Re: Varnish w/hardware loadbalancing woes > > Sounds more like a load balancer setup problem than a varnish issue. > Under Linux atleast with DR setups you must disable ARP for the VIP, > unless you're using a tunnel setup or a setup where your VIPs are > completely separate VLAN/LAN. > > It sounds like this is what's happening, that the real servers are > receiving the traffic sometimes and the load balancer at others because > of the ARP issue. > > --On Thursday, September 23, 2010 9:21 AM -0700 "Copper, Candace L" > wrote: > >> >> >> I've looked all over for information on how to configure varnish to use >> a VIP and have not been able to find much ? so I'm hoping someone >> here can assist. >> >> >> >> Our setup goes like this: Hardware Loadbalancer (sticky sessions enabled) >> with one VIP(perf-drupal.domain.com) directs to three Apache identical >> webservers (each running Varnish ? (perf-drupal1.domain.com, >> perf-drupal2.domain.com & perf-drupal3.domain.com)). With the VCL that we >> currently have, it works like a charm on each individual server, but when >> we try to use the VIP we get errors (50% of the time) when trying to >> access the site, stating that it is not available. >> >> >> >> I've tried: >> >> >> >> backend newsite { >> >> .host = "localhost"; >> >> .port = "8080"; >> >> } >> >> set req.http.host = "perf-drupal.domain.com"; >> >> set req.backend = newsite; >> >> >> >> and using the default backend: >> >> >> >> backend default { >> >> .host = "127.0.0.1"; >> >> .port = "8080"; >> >> } >> >> >> >> I've read about the DNS Director, but that's not available in the >> version I have installed - 2.1.3. But since we are only running one site, >> I don't know if it will help. I don't have access to any additional >> hardware, so I cannot split out Varnish from the Apache web server. >> >> >> >> Any assistance would be greatly appreciated! J >> >> >> >> Candace Copper >> >> >> > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From tungcnw at gmail.com Fri Sep 24 04:21:18 2010 From: tungcnw at gmail.com (Dang Tung) Date: Fri, 24 Sep 2010 11:21:18 +0700 Subject: Fixing backend URLs In-Reply-To: <0FAEA775-D78B-4CD6-9C0C-972ADE57287F@gmail.com> References: <2006354608710103182@unknownmsgid> <20100915032231.GC497@grum.quex.org> <0FAEA775-D78B-4CD6-9C0C-972ADE57287F@gmail.com> Message-ID: Dear, It's not work for me, may be my configurations are wrong, could anyone help me please. Here's my configuration: sub vcl_recv { # Host header check. # * Check if we recognize the domain used # * Normalize host headers to limit cache usage # * Rewrite URL for the virtual host monster # Return an error for other requests. # cache multimedia #if (req.request == "GET" && req.url ~ "\.(jpg|jpeg|gif|ico)$") { # lookup; #} # cache CSS and JS files if (req.request == "GET" && req.url ~ "\.(css|js)$") { lookup; } # do not cache POST requests if (req.request == "POST") { pipe; } if (req.request != "GET" && req.request != "HEAD") { # figure out if you are allowed to purge stuff from cache if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } lookup; } # else just pass it along pipe; } if (req.http.Authenticate || req.http.Authorization) { pass; } } sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } } sub vcl_miss { if (req.http.If-Modified-Since) { pass; } if (req.request == "PURGE") { error 404 "Not in cache."; } } sub vcl_fetch { if (!beresp.cacheable) { return (pass); } if (beresp.http.Set-Cookie) { return (pass); } return (deliver); if (beresp.status == 301 || beresp.status == 302) { set beresp.http.Location = regsub(beresp.http.Location, "^(\w+://[^/]+):\d+", "\1"); } } Many thanks and regards. On Wed, Sep 15, 2010 at 11:04 AM, Chris Cook wrote: > Awesome - the regex works perfectly. I think I'll keep it with the > redirect since those were the only area I was observing a suboptimal URI. > Thanks! > > -Chris > > On Sep 14, 2010, at 11:22 PM, Michael Alger wrote: > > > On Tue, Sep 14, 2010 at 07:40:10PM -0400, Chris Cook wrote: > >> I have an application that takes a URI that's submitted and spits back > >> a different URI with a 302 redirect. The backend server translates > >> this and sends back a new URI that contains the backend server port. > >> I want to use Varnish to remove the backend server port before sending > >> the request to the client. > >> > >> Varnish receives - http://domain.com:8080/blah/stuff > >> Varnish makes it - http://domain.com/blah/stuff > >> > >> What's the best way to do this? I assume it involves vcl_fetch and > >> some variants of the beresp, but I haven't been able to figure it out > >> yet. > > > > I think you want something like this in vcl_fetch(): > > > > if (beresp.status == 301 || beresp.status == 302) > > { > > set beresp.http.Location = regsub(beresp.http.Location, > "^(\w+://[^/]+):\d+", "\1"); > > } > > > > Seems to work okay in our UAT environment. The regex is hopefully > > matching any protocol specification (\w+://) and any hostname([^/]+). > > A colon after the hostname followed by any number of digits will > > then get dropped, but the request path should remain intact. > > > > Technically you can use a Location: header in other responses, so if > > that matters to you, you might want to change the if's to just a > > check if beresp.http.Location is present. > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- Nguyen Dang Tung -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at mm.quex.org Fri Sep 24 04:35:15 2010 From: varnish at mm.quex.org (Michael Alger) Date: Fri, 24 Sep 2010 12:35:15 +0800 Subject: Fixing backend URLs In-Reply-To: References: <2006354608710103182@unknownmsgid> <20100915032231.GC497@grum.quex.org> <0FAEA775-D78B-4CD6-9C0C-972ADE57287F@gmail.com> Message-ID: <20100924043515.GA30650@grum.quex.org> On Fri, Sep 24, 2010 at 11:21:18AM +0700, Dang Tung wrote: > > It's not work for me, may be my configurations are wrong, could anyone help > me please. Here's my configuration: > > sub vcl_fetch { > if (!beresp.cacheable) { > return (pass); > } > if (beresp.http.Set-Cookie) { > return (pass); > } > return (deliver); Here you are returning from vcl_fetch(), so the following code will never be executed. You should move that return statement to be the last in the function. > if (beresp.status == 301 || beresp.status == 302) > { > set beresp.http.Location = regsub(beresp.http.Location, "^(\w+://[^/]+):\d+", "\1"); > } > } From tungcnw at gmail.com Fri Sep 24 04:48:16 2010 From: tungcnw at gmail.com (Dang Tung) Date: Fri, 24 Sep 2010 11:48:16 +0700 Subject: Fixing backend URLs In-Reply-To: <20100924043515.GA30650@grum.quex.org> References: <2006354608710103182@unknownmsgid> <20100915032231.GC497@grum.quex.org> <0FAEA775-D78B-4CD6-9C0C-972ADE57287F@gmail.com> <20100924043515.GA30650@grum.quex.org> Message-ID: Thanks for fast reply, but I pasted wrong, actually when I remove these block, it doesn't work Here my configure again: sub vcl_recv { # Host header check. # * Check if we recognize the domain used # * Normalize host headers to limit cache usage # * Rewrite URL for the virtual host monster # Return an error for other requests. # cache multimedia #if (req.request == "GET" && req.url ~ "\.(jpg|jpeg|gif|ico)$") { # lookup; #} # cache CSS and JS files if (req.request == "GET" && req.url ~ "\.(css|js)$") { lookup; } # do not cache POST requests if (req.request == "POST") { pipe; } if (req.request != "GET" && req.request != "HEAD") { # figure out if you are allowed to purge stuff from cache if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } lookup; } # else just pass it along pipe; } if (req.http.Authenticate || req.http.Authorization) { pass; } } sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } } sub vcl_miss { if (req.http.If-Modified-Since) { pass; } if (req.request == "PURGE") { error 404 "Not in cache."; } } sub vcl_fetch { if (beresp.status == 301 || beresp.status == 302) { set beresp.http.Location = regsub(beresp.http.Location, "^(\w+://[^/]+):\d+", "\1"); } } Many thanks and regards On Fri, Sep 24, 2010 at 11:35 AM, Michael Alger wrote: > On Fri, Sep 24, 2010 at 11:21:18AM +0700, Dang Tung wrote: > > > > It's not work for me, may be my configurations are wrong, could anyone > help > > me please. Here's my configuration: > > > > sub vcl_fetch { > > if (!beresp.cacheable) { > > return (pass); > > } > > if (beresp.http.Set-Cookie) { > > return (pass); > > } > > return (deliver); > > Here you are returning from vcl_fetch(), so the following code will > never be executed. You should move that return statement to be the > last in the function. > > > if (beresp.status == 301 || beresp.status == 302) > > { > > set beresp.http.Location = regsub(beresp.http.Location, > "^(\w+://[^/]+):\d+", "\1"); > > } > > } > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- Nguyen Dang Tung -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at mm.quex.org Fri Sep 24 05:12:35 2010 From: varnish at mm.quex.org (Michael Alger) Date: Fri, 24 Sep 2010 13:12:35 +0800 Subject: Fixing backend URLs In-Reply-To: References: <2006354608710103182@unknownmsgid> <20100915032231.GC497@grum.quex.org> <0FAEA775-D78B-4CD6-9C0C-972ADE57287F@gmail.com> <20100924043515.GA30650@grum.quex.org> Message-ID: <20100924051235.GA31378@grum.quex.org> On Fri, Sep 24, 2010 at 11:48:16AM +0700, Dang Tung wrote: > Thanks for fast reply, > > but I pasted wrong, actually when I remove these block, it doesn't work > > sub vcl_fetch { > if (beresp.status == 301 || beresp.status == 302) > { > set beresp.http.Location = regsub(beresp.http.Location, "^(\w+://[^/]+):\d+", "\1"); > } > } Hmm. Can you provide the headers of one of the URLs you're testing, with as little anonymisation as possible? e.g. using wget -S -O /dev/null http://www.example.com/ Is it possible that your test URL is already cached by Varnish? vcl_fetch() is only called when the resource was fetched from the backend, so if it was already cached by Varnish that code will never be called. Either purge the URL or add similar code in vcl_hit(). From cosimo at streppone.it Fri Sep 24 08:08:25 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Fri, 24 Sep 2010 10:08:25 +0200 Subject: Varnish w/hardware loadbalancing woes In-Reply-To: <666C0AC37B26164FB0EFBD83F4631A0956092C1D@P01IMW302.ent.rt.csaa.com> References: <666C0AC37B26164FB0EFBD83F4631A09560925A2@P01IMW302.ent.rt.csaa.com> <666C0AC37B26164FB0EFBD83F4631A0956092C1D@P01IMW302.ent.rt.csaa.com> Message-ID: On Fri, 24 Sep 2010 01:23:00 +0200, Copper, Candace L wrote: > Michael, had the ARP disabled on the VIP Most probably it's not your problem, but I thought I'd mention it anyway. This is for LVS + direct routing: 1) ARP is enabled on the LVS machine, where your VIP is 2) on the backend servers, we apply the following sysctl mantra for direct routing to work properly: net.ipv4.conf.lo.arp_ignore = 1 net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.lo.arp_announce = 2 net.ipv4.conf.all.arp_announce = 2 See also: http://kb.linuxvirtualserver.org/wiki/Using_arp_announce/arp_ignore_to_disable_ARP http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.arp_problem.html -- Cosimo From ffanuel.metanext at spsbetting.com Fri Sep 24 08:54:07 2010 From: ffanuel.metanext at spsbetting.com (=?iso-8859-1?Q?Fran=E7ois_Fanuel?=) Date: Fri, 24 Sep 2010 01:54:07 -0700 Subject: Varnishncsa missing requests Message-ID: <40D93BB183F159428451FD5F2A45D2C54254A1318B@IE2RD2XVS041.red002.local> Hi, We've got a strange behavior with varnishncsa : some requests that we can see in varnishlog are missing in the varnishncsa logs. We can clearly see that with Healthchecks from our BIGIP (We've got 4x Varnish behind 2x BIGIP). The BIGIPs are doing 1 request every 5 seconds that we can see in varnishlog. In the varnishncsa logs, we can't see every healthcheck request. Sometime, we have no log at all for 3 or 4 minutes and then the logs reappears. We've done tests with or withour load, and the behavior is the same. Any idea ? Thanks for your help ! Regards, Fran?ois FANUEL -------------- next part -------------- An HTML attachment was scrubbed... URL: From scaunter at topscms.com Fri Sep 24 14:32:53 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Fri, 24 Sep 2010 10:32:53 -0400 Subject: Varnishncsa missing requests In-Reply-To: <40D93BB183F159428451FD5F2A45D2C54254A1318B@IE2RD2XVS041.red002.local> References: <40D93BB183F159428451FD5F2A45D2C54254A1318B@IE2RD2XVS041.red002.local> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C501BAC@TMG-EVS02.torstar.net> Is the BIGIP health check an http check? We stopped using that in favour of a simple tcp check to port 80. I've looked in my old varnishlog output, we were catching the F5 http health check in vcl_error. Can you send some log examples? Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Fran?ois Fanuel Sent: September-24-10 4:54 AM To: varnish-misc at varnish-cache.org Subject: Varnishncsa missing requests Hi, We've got a strange behavior with varnishncsa : some requests that we can see in varnishlog are missing in the varnishncsa logs. We can clearly see that with Healthchecks from our BIGIP (We've got 4x Varnish behind 2x BIGIP). The BIGIPs are doing 1 request every 5 seconds that we can see in varnishlog. In the varnishncsa logs, we can't see every healthcheck request. Sometime, we have no log at all for 3 or 4 minutes and then the logs reappears. We've done tests with or withour load, and the behavior is the same. Any idea ? Thanks for your help ! Regards, Fran?ois FANUEL -------------- next part -------------- An HTML attachment was scrubbed... URL: From ffanuel.metanext at spsbetting.com Fri Sep 24 14:37:56 2010 From: ffanuel.metanext at spsbetting.com (=?iso-8859-1?Q?Fran=E7ois_Fanuel?=) Date: Fri, 24 Sep 2010 07:37:56 -0700 Subject: Varnishncsa missing requests In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C501BAC@TMG-EVS02.torstar.net> References: <40D93BB183F159428451FD5F2A45D2C54254A1318B@IE2RD2XVS041.red002.local> <7F0AA702B8A85A4A967C4C8EBAD6902C501BAC@TMG-EVS02.torstar.net> Message-ID: <40D93BB183F159428451FD5F2A45D2C54254A13591@IE2RD2XVS041.red002.local> Hi, Yes, it's an http check. We also have a nagios http check with the same behavior : sometimes the request is logged by varnishncsa, sometimes not. We also catch the health check in vcl_error : sub vcl_error { # Serve HealthCheck if (obj.status == 699) { set obj.status = 200; set obj.http.Content-Type = "text/html; charset=utf-8"; synthetic "PASS"; return(deliver); } } Here is an example of a valid request logging in varnishncsa: 10.33.1.201 - - [24/Sep/2010:16:36:45 +0200] "GET /rp_hc/ HTTP/1.1" 200 4 "-" "-" Thanks for your help, Regards, Fran?ois FANUEL From: Caunter, Stefan [mailto:scaunter at topscms.com] Sent: vendredi 24 septembre 2010 16:33 To: Fran?ois Fanuel; varnish-misc at varnish-cache.org Subject: RE: Varnishncsa missing requests Is the BIGIP health check an http check? We stopped using that in favour of a simple tcp check to port 80. I've looked in my old varnishlog output, we were catching the F5 http health check in vcl_error. Can you send some log examples? Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Fran?ois Fanuel Sent: September-24-10 4:54 AM To: varnish-misc at varnish-cache.org Subject: Varnishncsa missing requests Hi, We've got a strange behavior with varnishncsa : some requests that we can see in varnishlog are missing in the varnishncsa logs. We can clearly see that with Healthchecks from our BIGIP (We've got 4x Varnish behind 2x BIGIP). The BIGIPs are doing 1 request every 5 seconds that we can see in varnishlog. In the varnishncsa logs, we can't see every healthcheck request. Sometime, we have no log at all for 3 or 4 minutes and then the logs reappears. We've done tests with or withour load, and the behavior is the same. Any idea ? Thanks for your help ! Regards, Fran?ois FANUEL -------------- next part -------------- An HTML attachment was scrubbed... URL: From hitendra.molleti at itp.com Sun Sep 26 07:38:59 2010 From: hitendra.molleti at itp.com (Hitendra Molleti) Date: Sun, 26 Sep 2010 11:38:59 +0400 Subject: Varnish stress test Message-ID: <070801cb5d4d$e20feab0$a62fc010$@molleti@itp.com> Hi, Not sure if anyone here can help me with this. But we now have fully developed our website with Java using Varnish in the frontend while Tomcat serving from the backend. Now, I am looking to run a stress test for the website to relaunch. There are so many tools online available and not sure which is a good tool to use for stress testing the website. Also, if someone can give us an idea of any final optimizations that can be done at a Varnish level to speed up the delivery of a page. We use cookies hence, can only cache certain widgets on the page. Thanks Hitendra Molleti Chief Technology Officer Direct: +971 4 444 3179 GSM: +971 50 499 7639 Online Business : www.arabianbusiness.com, www.arabianbusiness.com/arabic Property : www.arabianbusiness.com/property Recruitment : www.arabianbusiness.com/jobs Automotive : www.carmiddleeast.com Celebrity : www.ahlanlive.com, www.masala.com Entertainment : www.timeoutdubai.com, www.timeoutabudhabi.com www.timeoutbahrain.com, www.timeoutdoha.com Industry Portals : www.constructionweekonline.com, www.digitalproductionme.com, www.arabiansupplychain.com, www.arabianoilandgas.com Technology : www.itp.net Events : www.itp.net/events Tickets : www.timeouttickets.com This is an e-mail from ITP. Its contents are confidential to the intended recipient. If you are not the intended recipient, be advised that you have received this e-mail in error and that any use, dissemination, forwarding, printing or copying of this e-mail is strictly prohibited. It may not be disclosed to or used by anyone other than its intended recipient, nor may it be copied in any way. If received in error please email a reply to the sender, then delete it from your system. Although this e-mail has been scanned for viruses, ITP cannot accept any responsibility for viruses and it is your responsibility to scan any attachments. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 22207 bytes Desc: not available URL: From jerome.renard at gmail.com Sun Sep 26 09:03:42 2010 From: jerome.renard at gmail.com (Jerome Renard) Date: Sun, 26 Sep 2010 11:03:42 +0200 Subject: Varnish stress test In-Reply-To: <7651974892958231593@unknownmsgid> References: <7651974892958231593@unknownmsgid> Message-ID: Hi Hitendra, On Sun, Sep 26, 2010 at 9:38 AM, Hitendra Molleti wrote: > > Hi, > > > > Not sure if anyone here can help me with this. > > > > But we now have fully developed our website with Java using Varnish in the frontend while Tomcat serving from the backend. > > > > Now, I am looking to run a stress test for the website to relaunch. There are so many tools online available and not sure which is a good tool to use for stress testing the website. How about trying jmeter (http://jakarta.apache.org/jmeter/) or FunkLoad (http://funkload.nuxeo.org/) ? > > > > Also, if someone can give us an idea of any final optimizations that can be done at a Varnish level to speed up the delivery of a page. We use cookies hence, can only cache certain widgets on the page. You can try to setup a per user cache if you want : sub vcl_hash { set req.hash += req.http.cookie; } Ref : - http://www.varnish-cache.org/trac/wiki/VCLExampleCacheCookies - http://www.varnish-cache.org/trac/wiki/VCLExampleCachingLoggedInUsers 'Hope that helps, -- J?r?me Renard http://39web.fr | http://jrenard.info | http://twitter.com/jeromerenard From tungcnw at gmail.com Mon Sep 27 04:26:26 2010 From: tungcnw at gmail.com (Dang Tung) Date: Mon, 27 Sep 2010 11:26:26 +0700 Subject: Fixing backend URLs In-Reply-To: <20100924051235.GA31378@grum.quex.org> References: <2006354608710103182@unknownmsgid> <20100915032231.GC497@grum.quex.org> <0FAEA775-D78B-4CD6-9C0C-972ADE57287F@gmail.com> <20100924043515.GA30650@grum.quex.org> <20100924051235.GA31378@grum.quex.org> Message-ID: here is the header when I used wget: front-end: wget -S -O /dev/null http://myaddress/ HTTP request sent, awaiting response... HTTP/1.1 200 OK Server: Apache/2.2.3 (CentOS) X-Powered-By: PHP/5.2.10 Set-Cookie: symfony=8nrdjbj0q3a5367cr3eg2u7po7; path=/ Last-Modified: Mon, 27 Sep 2010 04:19:22 GMT Cache-Control: max-age=0 Expires: Mon, 27 Sep 2010 04:19:21 GMT Content-Type: text/html; charset=utf-8 Content-Length: 63066 Date: Mon, 27 Sep 2010 04:18:02 GMT X-Varnish: 153748005 Age: 0 Via: 1.1 varnish Connection: keep-alive Length: 63066 (62K) [text/html] Saving to: `/dev/null' 100%[======================================>] 63,066 --.-K/s in 0.1s 2010-09-27 11:17:50 (577 KB/s) - `/dev/null' saved [63066/63066] backend: wget -S -O /dev/null http://myaddress/admin wget -S -O /dev/null http://myaddress/admin --2010-09-27 11:18:35-- http://myaddress/admin Connecting to myaddress:80... connected. HTTP request sent, awaiting response... HTTP/1.1 302 Found Server: Apache/2.2.3 (CentOS) X-Powered-By: PHP/5.2.10 Set-Cookie: symfony=lli9m9oog3flp23ggchhbt5s71; path=/ Location: http://myaddress:8080/admin.php Cache-Control: max-age=0 Expires: Mon, 27 Sep 2010 04:20:08 GMT Content-Type: text/html; charset=utf-8 Content-Length: 106 Date: Mon, 27 Sep 2010 04:18:48 GMT X-Varnish: 153748006 Age: 0 Via: 1.1 varnish Connection: keep-alive Location: http://myaddress:8080/admin.php [following] --2010-09-27 11:18:35-- http://myaddress:8080/admin.php Connecting to myaddress:8080... failed: No route to host. I saw 302 found but I don't know what can I do then. 8080, here's the backend-port. Thanks and regards. Hmm. Can you provide the headers of one of the URLs you're testing, with > as little anonymisation as possible? e.g. using > > wget -S -O /dev/null http://www.example.com/ > > Is it possible that your test URL is already cached by Varnish? > vcl_fetch() is only called when the resource was fetched from the > backend, so if it was already cached by Varnish that code will never be > called. Either purge the URL or add similar code in vcl_hit(). > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- Nguyen Dang Tung -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Mon Sep 27 08:32:03 2010 From: perbu at varnish-software.com (Per Buer) Date: Mon, 27 Sep 2010 10:32:03 +0200 Subject: Question about Varnish Purge In-Reply-To: References: Message-ID: Hi. On Wed, Sep 22, 2010 at 7:13 AM, . wrote: > I heard that the purge list is never actually cleared, and can grow to be > really large over time. I use the purge quite a bit (which looks something > like: purge ^/users). > It's not a bug, it's more a result of how the bans (that what they are called in Varnish 3.0) work. The bans will add to the list and bans are discarded when they are older then the oldest object in the cache. So if you have objects with really long TTLs and a lot of bans this list can get long. If you have a lot of CPU usage this might the thing causing it. Otherwise, don't worry about it. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From bayron.guevara at gmail.com Mon Sep 27 18:07:02 2010 From: bayron.guevara at gmail.com (Bayron Guevara) Date: Mon, 27 Sep 2010 12:07:02 -0600 Subject: Backend load balancing delegation in order to support sticky sessions Message-ID: Hello community, Currently I?ve installed Varnish 2.0.6 and I'm using a cluster of Web Servers. In order to find a work around the session clustering issue, I thought implement session stickiness as part of the load balancing configuration, so all subsequent requests from the same user would go to the same webserver and so ensure the session information consistency. But as I know, Varnish don?t have integrated support to Stickiness, and therefore I have to delegate this function to a Load Balancer. Through this infrastructure the client requests would follow the below path: User Agent --> Load Balancer Public VIP --> One of the Varnish Servers --> (IF cache hit --> deliver cached content to Client ) ELSE ( Load Balancer Private VIP --> One of the Web Servers) To support sticky sessions my Load Balancer uses the Arrow Point cookie mechanism which, to be brief, add a cookie named ARPT that identified the user session. And according to the above path, Varnish sends requests and receive responses from the Load Balancer private VIP which needs that the cookie in question be passed transparently to and from the client. So, Varnish should take care of this cookie in two ways: the Cookie header in the incoming requests and the Set-Cookie in the responses from the Load Balancer. What do you recommend me to do?. I have ideas but I don?t sure. For example, to extract the desired cookie from Cookie header I?m doing the following: ### Code Start #### vcl_receive { ... if (req.http.Cookie ~ "ARPT=") { ????? set req.http.LB-Cookie = regsub(req.http.Cookie, ".*(ARPT=[^;]+).*", "\1"); } unset req.http.Cookie; ... } sub vcl_miss { ?????? set bereq.http.Cookie = req.http.LB-Cookie; } sub vcl_pass { ?????? set bereq.http.Cookie = req.http.LB-Cookie; } ###? Code End #### To keep it in the http response, I guess: ###? Code Start ### sub vcl_fetch { ? if (obj.http.Set-Cookie ~ "ARPT=") { ?????????????? #Save Set-Cookie, so it can be used in vcl_deliver ?????????????? set obj.http.X-Set-Cookie = obj.http.Set-Cookie; } ? } sub vcl_deliver { if (obj.hits > 0) { ... ?????? } else { ?????? #It was a cache miss ?????????????? ... ?????????????? #Set back Set-Cookie header in case it's set ?????????????? if (resp.http.X-Set-Cookie){ ?????????????????????? set resp.http.Set-Cookie = resp.http.X-Set-Cookie; ?????????????????????? unset resp.http.X-Set-Cookie; ?????????????? } ?????? } } ### Code End ### I don?t know how to extract a cookie from a Set-Cookie header, but I tried with this? code: ?????? obj.http.LB-Cookie = regsub(obj.http.Set-Cookie, ?.*(ARPT=[^;]+(;[^=]+=[^;]+)*).*?, ?\1?); Nevertheless, it appears don?t work fine. Am I in the right way? Is there a better solution? Thank you in advance. --------------------------------------------------- Bayron Guevara Web Programmer bayron.guevara at gmail.com From Candace.Copper at goAAA.com Mon Sep 27 18:20:35 2010 From: Candace.Copper at goAAA.com (Copper, Candace L) Date: Mon, 27 Sep 2010 11:20:35 -0700 Subject: Varnish w/hardware loadbalancing woes In-Reply-To: <91175C6D206E8D3B723BC0FF@[192.168.66.199]> References: <666C0AC37B26164FB0EFBD83F4631A09560925A2@P01IMW302.ent.rt.csaa.com> <666C0AC37B26164FB0EFBD83F4631A0956092C1D@P01IMW302.ent.rt.csaa.com> <91175C6D206E8D3B723BC0FF@[192.168.66.199]> Message-ID: <666C0AC37B26164FB0EFBD83F4631A0956138FD9@P01IMW302.ent.rt.csaa.com> Thanks again Michael - I was able to gather some more information regarding our VIP today - here is what was given to me: VIP- server virtual perf-drupal.domain.com 192.168.X.X sym-priority 4 predictor round-robin port http sticky port ssl sticky bind http server1 http server2 http server3 http bind ssl server1 ssl server2 ssl server3 ssl Real Server(s) - server real server1 192.168.X.X port http port http keepalive port http url "HEAD /" port ssl port ssl keepalive ! server real server2 192.168.X.X port http port http keepalive port http url "HEAD /" port ssl port ssl keepalive ! server real server3 192.168.X.X port http port http keepalive port http url "HEAD /" port ssl port ssl keepalive So the way we have it configured now it goes: VIP (hardware based) -> Server1, 2 or 3 -> Varnish on Server1, 2 or 3 (listening on port 80) -> Apache (listening on port 8080). Varnish is running on Server1, 2 and 3 as 80 and it points to apache on 8080. Hope this helps some. Candace Copper -----Original Message----- From: Michael Loftis [mailto:mloftis at wgops.com] Sent: Thursday, September 23, 2010 4:45 PM To: Copper, Candace L; 'varnish-misc at varnish-cache.org' Subject: RE: Varnish w/hardware loadbalancing woes Well if you could diagram or walk-through your specific setup and configs I can certainly help you (I've deployed probably a couple hundred or so various load balanced applications using both hardware and software LB approaches, and with approaches with proxies like varnish too) The *only* machine directly receiving traffic for a VIP should be the hardware load balancer itself. It will then decide which (real) machine (in your case apparently running varnish on top of ?) to send the traffic on to. If the real machines are answering for the VIP because they're all sharing a VLAN/LAN then you're going to get failures randomly. Generally the VIP is configured on a loopback device as a /32 (or /128 in IPv6) -- return traffic can either be DR or via the LB itself. in the latter you might be doing NAT in which case the real machines won't have the VIP configured at all. In a DR situation the real machines have to have the VIP because they'll send all traffic back to their (default) router, not the load balancer. --On Thursday, September 23, 2010 4:23 PM -0700 "Copper, Candace L" wrote: > Michael, had the ARP disabled on the VIP and then the site wouldn't load > at all with the VIP and still saw the Unable to connect errors. We > re-enabled ARP and then our site still wouldn't load, and we couldn't > kill any httpd processes and had to reboot. > > Just weird stuff happened with that. > > Candace Copper > > > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Michael > Loftis Sent: Thursday, September 23, 2010 10:57 AM > To: 'varnish-misc at varnish-cache.org' > Subject: Re: Varnish w/hardware loadbalancing woes > > Sounds more like a load balancer setup problem than a varnish issue. > Under Linux atleast with DR setups you must disable ARP for the VIP, > unless you're using a tunnel setup or a setup where your VIPs are > completely separate VLAN/LAN. > > It sounds like this is what's happening, that the real servers are > receiving the traffic sometimes and the load balancer at others because > of the ARP issue. > > --On Thursday, September 23, 2010 9:21 AM -0700 "Copper, Candace L" > wrote: > >> >> >> I've looked all over for information on how to configure varnish to use >> a VIP and have not been able to find much ? so I'm hoping someone >> here can assist. >> >> >> >> Our setup goes like this: Hardware Loadbalancer (sticky sessions enabled) >> with one VIP(perf-drupal.domain.com) directs to three Apache identical >> webservers (each running Varnish ? (perf-drupal1.domain.com, >> perf-drupal2.domain.com & perf-drupal3.domain.com)). With the VCL that we >> currently have, it works like a charm on each individual server, but when >> we try to use the VIP we get errors (50% of the time) when trying to >> access the site, stating that it is not available. >> >> >> >> I've tried: >> >> >> >> backend newsite { >> >> .host = "localhost"; >> >> .port = "8080"; >> >> } >> >> set req.http.host = "perf-drupal.domain.com"; >> >> set req.backend = newsite; >> >> >> >> and using the default backend: >> >> >> >> backend default { >> >> .host = "127.0.0.1"; >> >> .port = "8080"; >> >> } >> >> >> >> I've read about the DNS Director, but that's not available in the >> version I have installed - 2.1.3. But since we are only running one site, >> I don't know if it will help. I don't have access to any additional >> hardware, so I cannot split out Varnish from the Apache web server. >> >> >> >> Any assistance would be greatly appreciated! J >> >> >> >> Candace Copper >> >> >> > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From scaunter at topscms.com Mon Sep 27 19:39:30 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Mon, 27 Sep 2010 15:39:30 -0400 Subject: Varnish w/hardware loadbalancing woes In-Reply-To: <666C0AC37B26164FB0EFBD83F4631A0956138FD9@P01IMW302.ent.rt.csaa.com> References: <666C0AC37B26164FB0EFBD83F4631A09560925A2@P01IMW302.ent.rt.csaa.com><666C0AC37B26164FB0EFBD83F4631A0956092C1D@P01IMW302.ent.rt.csaa.com><91175C6D206E8D3B723BC0FF@[192.168.66.199]> <666C0AC37B26164FB0EFBD83F4631A0956138FD9@P01IMW302.ent.rt.csaa.com> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C501DFC@TMG-EVS02.torstar.net> -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Copper, Candace L Sent: September-27-10 2:21 PM To: 'Michael Loftis'; 'varnish-misc at varnish-cache.org' Subject: RE: Varnish w/hardware loadbalancing woes > Thanks again Michael - I was able to gather some more information regarding our VIP today - here is what was given to me: > VIP- > server virtual perf-drupal.domain.com 192.168.X.X sym-priority 4 predictor round-robin port http sticky port ssl sticky bind http server1 http server2 http server3 > http bind ssl server1 ssl server2 ssl server3 ssl varnish won't do ssl Do clients expect ssl or plain http? --Stef From Candace.Copper at goAAA.com Mon Sep 27 19:43:58 2010 From: Candace.Copper at goAAA.com (Copper, Candace L) Date: Mon, 27 Sep 2010 12:43:58 -0700 Subject: Varnish w/hardware loadbalancing woes In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C501DFC@TMG-EVS02.torstar.net> References: <666C0AC37B26164FB0EFBD83F4631A09560925A2@P01IMW302.ent.rt.csaa.com><666C0AC37B26164FB0EFBD83F4631A0956092C1D@P01IMW302.ent.rt.csaa.com><91175C6D206E8D3B723BC0FF@[192.168.66.199]> <666C0AC37B26164FB0EFBD83F4631A0956138FD9@P01IMW302.ent.rt.csaa.com> <7F0AA702B8A85A4A967C4C8EBAD6902C501DFC@TMG-EVS02.torstar.net> Message-ID: <666C0AC37B26164FB0EFBD83F4631A0956139111@P01IMW302.ent.rt.csaa.com> Yea, I read that Varnish will not do SSL - the SSL is configured on the VIP though. Our site will need to handle http and https requests. But I want Varnish to work for the http portions - and that is what we have been testing, only http requests. Candace Copper -----Original Message----- From: Caunter, Stefan [mailto:scaunter at topscms.com] Sent: Monday, September 27, 2010 12:40 PM To: Copper, Candace L; Michael Loftis; varnish-misc at varnish-cache.org Subject: RE: Varnish w/hardware loadbalancing woes -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Copper, Candace L Sent: September-27-10 2:21 PM To: 'Michael Loftis'; 'varnish-misc at varnish-cache.org' Subject: RE: Varnish w/hardware loadbalancing woes > Thanks again Michael - I was able to gather some more information regarding our VIP today - here is what was given to me: > VIP- > server virtual perf-drupal.domain.com 192.168.X.X sym-priority 4 predictor round-robin port http sticky port ssl sticky bind http server1 http server2 http server3 > http bind ssl server1 ssl server2 ssl server3 ssl varnish won't do ssl Do clients expect ssl or plain http? --Stef From mloftis at wgops.com Mon Sep 27 20:18:12 2010 From: mloftis at wgops.com (Michael Loftis) Date: Mon, 27 Sep 2010 14:18:12 -0600 Subject: Varnish w/hardware loadbalancing woes In-Reply-To: <666C0AC37B26164FB0EFBD83F4631A0956138FD9@P01IMW302.ent.rt.csaa.com> References: <666C0AC37B26164FB0EFBD83F4631A09560925A2@P01IMW302.ent.rt.csaa.com> <666C0AC37B26164FB0EFBD83F4631A0956092C1D@P01IMW302.ent.rt.csaa.com> <91175C6D206E8D3B723BC0FF@[192.168.66.199]> <666C0AC37B26164FB0EFBD83F4631A0956138FD9@P01IMW302.ent.rt.csaa.com> Message-ID: <2E5D05224A8109C6419E4784@[192.168.66.199]> --On Monday, September 27, 2010 11:20 AM -0700 "Copper, Candace L" wrote: > Thanks again Michael - I was able to gather some more information > regarding our VIP today - here is what was given to me: > As has been pointed out, varnish cant/wont do SSL. Its not clear what your network topology looks like from this. Have you verified sessions are all setting up on the load balancer? And that the real servers are indeed NOT responding to ARP requests for the shared VIP? > VIP- > server virtual perf-drupal.domain.com 192.168.X.X sym-priority 4 > predictor round-robin port http sticky port ssl sticky bind http > server1 http server2 http server3 http bind ssl server1 ssl server2 ssl > server3 ssl > > Real Server(s) - > > server real server1 192.168.X.X > port http > port http keepalive > port http url "HEAD /" > port ssl > port ssl keepalive > ! > server real server2 192.168.X.X > port http > port http keepalive > port http url "HEAD /" > port ssl > port ssl keepalive > ! > server real server3 192.168.X.X > port http > port http keepalive > port http url "HEAD /" > port ssl > port ssl keepalive > > > So the way we have it configured now it goes: VIP (hardware based) -> > Server1, 2 or 3 -> Varnish on Server1, 2 or 3 (listening on port 80) -> > Apache (listening on port 8080). Varnish is running on Server1, 2 and 3 > as 80 and it points to apache on 8080. > > Hope this helps some. > > Candace Copper > > > -----Original Message----- > From: Michael Loftis [mailto:mloftis at wgops.com] > Sent: Thursday, September 23, 2010 4:45 PM > To: Copper, Candace L; 'varnish-misc at varnish-cache.org' > Subject: RE: Varnish w/hardware loadbalancing woes > > Well if you could diagram or walk-through your specific setup and configs > I can certainly help you (I've deployed probably a couple hundred or so > various load balanced applications using both hardware and software LB > approaches, and with approaches with proxies like varnish too) > > The *only* machine directly receiving traffic for a VIP should be the > hardware load balancer itself. It will then decide which (real) machine > (in your case apparently running varnish on top of ?) to send the traffic > on to. If the real machines are answering for the VIP because they're > all sharing a VLAN/LAN then you're going to get failures randomly. > > Generally the VIP is configured on a loopback device as a /32 (or /128 in > IPv6) -- return traffic can either be DR or via the LB itself. in the > latter you might be doing NAT in which case the real machines won't have > the VIP configured at all. In a DR situation the real machines have to > have the VIP because they'll send all traffic back to their (default) > router, not the load balancer. > > --On Thursday, September 23, 2010 4:23 PM -0700 "Copper, Candace L" > wrote: > >> Michael, had the ARP disabled on the VIP and then the site wouldn't load >> at all with the VIP and still saw the Unable to connect errors. We >> re-enabled ARP and then our site still wouldn't load, and we couldn't >> kill any httpd processes and had to reboot. >> >> Just weird stuff happened with that. >> >> Candace Copper >> >> >> -----Original Message----- >> From: varnish-misc-bounces at varnish-cache.org >> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Michael >> Loftis Sent: Thursday, September 23, 2010 10:57 AM >> To: 'varnish-misc at varnish-cache.org' >> Subject: Re: Varnish w/hardware loadbalancing woes >> >> Sounds more like a load balancer setup problem than a varnish issue. >> Under Linux atleast with DR setups you must disable ARP for the VIP, >> unless you're using a tunnel setup or a setup where your VIPs are >> completely separate VLAN/LAN. >> >> It sounds like this is what's happening, that the real servers are >> receiving the traffic sometimes and the load balancer at others because >> of the ARP issue. >> >> --On Thursday, September 23, 2010 9:21 AM -0700 "Copper, Candace L" >> wrote: >> >>> >>> >>> I've looked all over for information on how to configure varnish to use >>> a VIP and have not been able to find much ? so I'm hoping someone >>> here can assist. >>> >>> >>> >>> Our setup goes like this: Hardware Loadbalancer (sticky sessions >>> enabled) with one VIP(perf-drupal.domain.com) directs to three Apache >>> identical webservers (each running Varnish ? (perf-drupal1.domain.com, >>> perf-drupal2.domain.com & perf-drupal3.domain.com)). With the VCL that >>> we currently have, it works like a charm on each individual server, but >>> when we try to use the VIP we get errors (50% of the time) when trying >>> to access the site, stating that it is not available. >>> >>> >>> >>> I've tried: >>> >>> >>> >>> backend newsite { >>> >>> .host = "localhost"; >>> >>> .port = "8080"; >>> >>> } >>> >>> set req.http.host = "perf-drupal.domain.com"; >>> >>> set req.backend = newsite; >>> >>> >>> >>> and using the default backend: >>> >>> >>> >>> backend default { >>> >>> .host = "127.0.0.1"; >>> >>> .port = "8080"; >>> >>> } >>> >>> >>> >>> I've read about the DNS Director, but that's not available in the >>> version I have installed - 2.1.3. But since we are only running one >>> site, I don't know if it will help. I don't have access to any >>> additional hardware, so I cannot split out Varnish from the Apache web >>> server. >>> >>> >>> >>> Any assistance would be greatly appreciated! J >>> >>> >>> >>> Candace Copper >>> >>> >>> >> >> >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > > > From tfheen at varnish-software.com Tue Sep 28 07:04:42 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 28 Sep 2010 09:04:42 +0200 Subject: Backend load balancing delegation in order to support sticky sessions In-Reply-To: (Bayron Guevara's message of "Mon, 27 Sep 2010 12:07:02 -0600") References: Message-ID: <87mxr28i2d.fsf@qurzaw.linpro.no> ]] Bayron Guevara | Currently I?ve installed Varnish 2.0.6 and I'm using a cluster of Web | Servers. In order to find a work around the session clustering issue, | I thought implement session stickiness as part of the load balancing | configuration, so all subsequent requests from the same user would go | to the same webserver and so ensure the session information | consistency. But as I know, Varnish don?t have integrated support to | Stickiness, and therefore I have to delegate this function to a Load | Balancer. Through this infrastructure the client requests would follow | the below path: In 2.1, you can use the client director to get that stickiness. With 2.1.4, you'll also have client.identity which you can use to influence what the client director uses for choosing the backend. [...] | I don?t know how to extract a cookie from a Set-Cookie header, but I | tried with this? code: | ?????? obj.http.LB-Cookie = regsub(obj.http.Set-Cookie, | ?.*(ARPT=[^;]+(;[^=]+=[^;]+)*).*?, ?\1?); | Nevertheless, it appears don?t work fine. The Set-Cookie header doesn't work like all the other headers in RFC2616, so you need to use inline C to poke at it. This is doable, but non-trivial. | Am I in the right way? Is there a better solution? I'd recommend going with 2.1 and the client director, at least if you don't need any other features than reasonably simple load balancing from your load balancer. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tdevelioglu at ebuddy.com Tue Sep 28 17:21:30 2010 From: tdevelioglu at ebuddy.com (Taylan Develioglu) Date: Tue, 28 Sep 2010 19:21:30 +0200 Subject: varnish caches too many object variances Message-ID: <1285694490.1855.65.camel@oasis> Hi, I have some varnish 2.1.3 behavior I cannot explain. Our backend servers are not adding any 'Vary:' header to responses, but objects > object_heads by a great difference. The ratio objects:object_heads is ~ 2:1. The vcl is clean and simple: sub vcl_recv { unset req.http.cookie; if (req.request == "HEAD") { return(pass); } } Is there anything besides 'Vary' that makes varnish cache multiple objects ? What else could be causing this ? From jhayter at manta.com Tue Sep 28 19:06:04 2010 From: jhayter at manta.com (Jim Hayter) Date: Tue, 28 Sep 2010 15:06:04 -0400 Subject: varnish caches too many object variances In-Reply-To: <1285694490.1855.65.camel@oasis> References: <1285694490.1855.65.camel@oasis> Message-ID: I had to look into this recently myself. Take a look at the documentation on how gzip/deflate are handled. For one object varnish can potentially have multiple objects: gzipped, deflated and uncompressed. Jim -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Taylan Develioglu Sent: Tuesday, September 28, 2010 1:22 PM To: varnish-misc at varnish-cache.org Subject: varnish caches too many object variances Hi, I have some varnish 2.1.3 behavior I cannot explain. Our backend servers are not adding any 'Vary:' header to responses, but objects > object_heads by a great difference. The ratio objects:object_heads is ~ 2:1. The vcl is clean and simple: sub vcl_recv { unset req.http.cookie; if (req.request == "HEAD") { return(pass); } } Is there anything besides 'Vary' that makes varnish cache multiple objects ? What else could be causing this ? _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From tdevelioglu at ebuddy.com Tue Sep 28 19:11:17 2010 From: tdevelioglu at ebuddy.com (Taylan Develioglu) Date: Tue, 28 Sep 2010 21:11:17 +0200 Subject: varnish caches too many object variances In-Reply-To: References: <1285694490.1855.65.camel@oasis> Message-ID: Compressed objects are cached according to Vary: Accept-encoding headers. Like I said the backend does not add these headers (nor does compression). -----Original Message----- From: Jim Hayter [mailto:jhayter at manta.com] Sent: Tuesday, September 28, 2010 9:06 PM To: Taylan Develioglu; varnish-misc at varnish-cache.org Subject: RE: varnish caches too many object variances I had to look into this recently myself. Take a look at the documentation on how gzip/deflate are handled. For one object varnish can potentially have multiple objects: gzipped, deflated and uncompressed. Jim -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Taylan Develioglu Sent: Tuesday, September 28, 2010 1:22 PM To: varnish-misc at varnish-cache.org Subject: varnish caches too many object variances Hi, I have some varnish 2.1.3 behavior I cannot explain. Our backend servers are not adding any 'Vary:' header to responses, but objects > object_heads by a great difference. The ratio objects:object_heads is ~ 2:1. The vcl is clean and simple: sub vcl_recv { unset req.http.cookie; if (req.request == "HEAD") { return(pass); } } Is there anything besides 'Vary' that makes varnish cache multiple objects ? What else could be causing this ? _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From fla_torres at yahoo.com.br Tue Sep 28 22:33:17 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Tue, 28 Sep 2010 19:33:17 -0300 Subject: backend_toolate - Backend conn. was closed Message-ID: <4CA26D2D.7060001@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, Please, in varnishstat, what does backend_toolate means ? backend_toolate 826901 3.28 Backend conn. was closed Ok, I can see "Backend conn. was closed", but I would like to know 'why ?'. It's because backend ( varnish's > VIP LoadBalance > nginx farm) closes by: net.ipv4.tcp_keepalive_time, first_byte_timeout; keepalive_timeout (nginx) or: - - Varnish request Page1, the server become slow to respond; varnish request Page1 again and get!. When the first request is delivered it (varnish) responds with the message "too late"? Am I making myself clear? Thank you guys. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyibSAACgkQNRQApncg295lCACfZ8sLlMmDbkvVPRhsDTP3nS/t k+4An3k85kmqcCTyiWxIYOqKq4DUfBqh =sjx8 -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Tue Sep 28 22:51:49 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 28 Sep 2010 22:51:49 +0000 Subject: backend_toolate - Backend conn. was closed In-Reply-To: Your message of "Tue, 28 Sep 2010 19:33:17 -0300." <4CA26D2D.7060001@yahoo.com.br> Message-ID: <77659.1285714309@critter.freebsd.dk> In message <4CA26D2D.7060001 at yahoo.com.br>, Flavio Torres writes: >Please, in varnishstat, what does backend_toolate means ? > > >backend_toolate 826901 3.28 Backend conn. was closed The backend closed an idle connection before we tried to reuse it. Varnish picks or opens another connection. If the number is as high as you have here, it may pay off to increase the idle-timeout on your backend. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Wed Sep 29 13:01:59 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 29 Sep 2010 13:01:59 +0000 Subject: Spring cleaning Message-ID: <20440.1285765319@critter.freebsd.dk> We have held a little bit of a spring cleaning, to dump stuff we have abandonned. We are unsure if anybody cares/works/uses these bits, please report back if you do, or they will be removed at some later date: varnish-tools/autobuild varnish-tools/fetcher varnish-tools/webmin Thanks in advance. Also: The people maintaining the python and perl extensions should start upgrading to the new VSM/VSC stuff in 3.0 soon. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From caltman at matterco.com Wed Sep 29 16:06:25 2010 From: caltman at matterco.com (Chris Altman) Date: Wed, 29 Sep 2010 12:06:25 -0400 Subject: How do you confirm cookies are removed from static files? References: Message-ID: I am new to Varnish and have read the documentation. My goal is to remove all cookies from static files. I am using Varnish 2.1.3 My VCL is: sub vcl_recv { if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { unset req.http.cookie; } } sub vcl_fetch { if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { unset beresp.http.set-cookie; } } Question: I am confused because it appears the static files are served from Varnish but still have a cookie in their Request Headers. How do I remove the Request Headers cookie? Am I still gaining the benefits of Varnish if the Request Headers cookies exist? Thank you, Chris Altman http://my.perqworks.com http://prtclr.com Response Headersview source Server Apache/2.2.14 (Ubuntu) X-Powered-By PHP/5.3.2-1ubuntu4.5 Expires Thu, 29 Sep 2011 15:56:42 GMT Vary Accept-Encoding Last-Modified Thu, 24 Jun 2010 21:16:50 GMT Cache-Control max-age=31536000, public Content-Encoding gzip Content-Type text/css; charset=utf-8 Content-Length 3339 Date Wed, 29 Sep 2010 15:56:42 GMT X-Varnish 2062520895 Age 0 Via 1.1 varnish Connection keep-alive Request Headersview source Host my.perqapp.com User-Agent Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10 Accept text/css,*/*;q=0.1 Accept-Language en-us,en;q=0.5 Accept-Encoding gzip,deflate Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive 115 Connection keep-alive Referer http://my.perqapp.com/login Cookie perq=20; __utma=90567579.1481902958.1285699566.1285768162.1285775102.4; __utmz=90567579.1285699566.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); perqworks=c7jcfkdueap5ggtbds500al703; __utmb=90567579.4.10.1285775102; __utmc=90567579 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tdevelioglu at ebuddy.com Wed Sep 29 16:43:18 2010 From: tdevelioglu at ebuddy.com (Taylan Develioglu) Date: Wed, 29 Sep 2010 18:43:18 +0200 Subject: varnish caches too many object variances In-Reply-To: <1285694490.1855.65.camel@oasis> References: <1285694490.1855.65.camel@oasis> Message-ID: <1285778598.1855.493.camel@oasis> So, nobody has any other idea on why we have more objects than object_heads ? varnishstat : n_object 1239057 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 1241116 . N struct objectcore n_objecthead 640299 . N struct objecthead n_expired 32151 . N expired objects On Tue, 2010-09-28 at 19:21 +0200, Taylan Develioglu wrote: > Hi, > > I have some varnish 2.1.3 behavior I cannot explain. > > Our backend servers are not adding any 'Vary:' header to responses, but > objects > object_heads by a great difference. > > The ratio objects:object_heads is ~ 2:1. > > The vcl is clean and simple: > > sub vcl_recv { > unset req.http.cookie; > > if (req.request == "HEAD") { > return(pass); > } > } > > Is there anything besides 'Vary' that makes varnish cache multiple > objects ? What else could be causing this ? > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From scaunter at topscms.com Wed Sep 29 18:30:39 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Wed, 29 Sep 2010 14:30:39 -0400 Subject: varnish caches too many object variances In-Reply-To: <1285778598.1855.493.camel@oasis> References: <1285694490.1855.65.camel@oasis> <1285778598.1855.493.camel@oasis> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C5021A2@TMG-EVS02.torstar.net> What is the issue? It isn't unusual. 26791 . . N struct object 27548 . . N struct objectcore 20782 . . N struct objecthead Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Taylan Develioglu Sent: September-29-10 12:43 PM To: varnish-misc at varnish-cache.org Subject: Re: varnish caches too many object variances So, nobody has any other idea on why we have more objects than object_heads ? varnishstat : n_object 1239057 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 1241116 . N struct objectcore n_objecthead 640299 . N struct objecthead n_expired 32151 . N expired objects On Tue, 2010-09-28 at 19:21 +0200, Taylan Develioglu wrote: > Hi, > > I have some varnish 2.1.3 behavior I cannot explain. > > Our backend servers are not adding any 'Vary:' header to responses, but > objects > object_heads by a great difference. > > The ratio objects:object_heads is ~ 2:1. > > The vcl is clean and simple: > > sub vcl_recv { > unset req.http.cookie; > > if (req.request == "HEAD") { > return(pass); > } > } > > Is there anything besides 'Vary' that makes varnish cache multiple > objects ? What else could be causing this ? > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From tdevelioglu at ebuddy.com Wed Sep 29 18:46:22 2010 From: tdevelioglu at ebuddy.com (Taylan Develioglu) Date: Wed, 29 Sep 2010 20:46:22 +0200 Subject: varnish caches too many object variances In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C5021A2@TMG-EVS02.torstar.net> References: <1285694490.1855.65.camel@oasis> <1285778598.1855.493.camel@oasis> <7F0AA702B8A85A4A967C4C8EBAD6902C5021A2@TMG-EVS02.torstar.net> Message-ID: So, what you are saying is it's not unusual to have twice as many objects as objects as objectheads when there is no Vary header ? I don't know if it's usual or not, but it struck me as odd as I expected to have one object header per cached object. -----Original Message----- From: Caunter, Stefan [mailto:scaunter at topscms.com] Sent: Wednesday, September 29, 2010 8:31 PM To: Taylan Develioglu; varnish-misc at varnish-cache.org Subject: RE: varnish caches too many object variances What is the issue? It isn't unusual. 26791 . . N struct object 27548 . . N struct objectcore 20782 . . N struct objecthead Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Taylan Develioglu Sent: September-29-10 12:43 PM To: varnish-misc at varnish-cache.org Subject: Re: varnish caches too many object variances So, nobody has any other idea on why we have more objects than object_heads ? varnishstat : n_object 1239057 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 1241116 . N struct objectcore n_objecthead 640299 . N struct objecthead n_expired 32151 . N expired objects On Tue, 2010-09-28 at 19:21 +0200, Taylan Develioglu wrote: > Hi, > > I have some varnish 2.1.3 behavior I cannot explain. > > Our backend servers are not adding any 'Vary:' header to responses, but > objects > object_heads by a great difference. > > The ratio objects:object_heads is ~ 2:1. > > The vcl is clean and simple: > > sub vcl_recv { > unset req.http.cookie; > > if (req.request == "HEAD") { > return(pass); > } > } > > Is there anything besides 'Vary' that makes varnish cache multiple > objects ? What else could be causing this ? > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From tdevelioglu at ebuddy.com Wed Sep 29 20:26:45 2010 From: tdevelioglu at ebuddy.com (Taylan Develioglu) Date: Wed, 29 Sep 2010 22:26:45 +0200 Subject: How do you confirm cookies are removed from static files? In-Reply-To: References: Message-ID: > Am I still gaining the benefits of Varnish if the Request Headers cookies exist? Varnish by default will not cache anything with a cookie. You could verify if the unset is actually reached by placing a log statement in your if clausule. Hope that helps ( a little ) ________________________________ From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Chris Altman Sent: Wednesday, September 29, 2010 6:06 PM To: varnish-misc at varnish-cache.org Subject: How do you confirm cookies are removed from static files? I am new to Varnish and have read the documentation. My goal is to remove all cookies from static files. I am using Varnish 2.1.3 My VCL is: sub vcl_recv { if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { unset req.http.cookie; } } sub vcl_fetch { if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { unset beresp.http.set-cookie; } } Question: I am confused because it appears the static files are served from Varnish but still have a cookie in their Request Headers. How do I remove the Request Headers cookie? Am I still gaining the benefits of Varnish if the Request Headers cookies exist? Thank you, Chris Altman http://my.perqworks.com http://prtclr.com Response Headersview source Server Apache/2.2.14 (Ubuntu) X-Powered-By PHP/5.3.2-1ubuntu4.5 Expires Thu, 29 Sep 2011 15:56:42 GMT Vary Accept-Encoding Last-Modified Thu, 24 Jun 2010 21:16:50 GMT Cache-Control max-age=31536000, public Content-Encoding gzip Content-Type text/css; charset=utf-8 Content-Length 3339 Date Wed, 29 Sep 2010 15:56:42 GMT X-Varnish 2062520895 Age 0 Via 1.1 varnish Connection keep-alive Request Headersview source Host my.perqapp.com User-Agent Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10 Accept text/css,*/*;q=0.1 Accept-Language en-us,en;q=0.5 Accept-Encoding gzip,deflate Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive 115 Connection keep-alive Referer http://my.perqapp.com/login Cookie perq=20; __utma=90567579.1481902958.1285699566.1285768162.1285775102.4; __utmz=90567579.1285699566.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); perqworks=c7jcfkdueap5ggtbds500al703; __utmb=90567579.4.10.1285775102; __utmc=90567579 -------------- next part -------------- An HTML attachment was scrubbed... URL: From felix at seconddrawer.com.au Thu Sep 30 00:17:59 2010 From: felix at seconddrawer.com.au (Felix Hanley) Date: Thu, 30 Sep 2010 07:17:59 +0700 Subject: How do you confirm cookies are removed from static files? In-Reply-To: References: Message-ID: <20100930001759.GB9693@thinkpad.local> The client will request that cookies be set and there is not much you can do about that, except unset them. But they will still be in the request. The line in vcl_recv will remove set-cookie headers from the request. The line in vcl_fetch will remove cookie headers before being stored in the cache. Your code looks fine. To confirm it is being stored, you will need to use varnishlog or something. -felix On Wed, Sep 29, 2010 at 12:06:25PM -0400, Chris Altman wrote: > I am new to Varnish and have read the documentation. My goal is to remove all cookies from static files. I am using Varnish 2.1.3 > > My VCL is: > > sub vcl_recv { > if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { > unset req.http.cookie; > } > } > > sub vcl_fetch { > if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { > unset beresp.http.set-cookie; > } > } > > Question: > I am confused because it appears the static files are served from Varnish but still have a cookie in their Request Headers. How do I remove the Request Headers cookie? Am I still gaining the benefits of Varnish if the Request Headers cookies exist? > > Thank you, > Chris Altman > http://my.perqworks.com > http://prtclr.com > > > Response Headersview source > Server Apache/2.2.14 (Ubuntu) > X-Powered-By PHP/5.3.2-1ubuntu4.5 > Expires Thu, 29 Sep 2011 15:56:42 GMT > Vary Accept-Encoding > Last-Modified Thu, 24 Jun 2010 21:16:50 GMT > Cache-Control max-age=31536000, public > Content-Encoding gzip > Content-Type text/css; charset=utf-8 > Content-Length 3339 > Date Wed, 29 Sep 2010 15:56:42 GMT > X-Varnish 2062520895 > Age 0 > Via 1.1 varnish > Connection keep-alive > > > Request Headersview source > Host my.perqapp.com > User-Agent Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10 > Accept text/css,*/*;q=0.1 > Accept-Language en-us,en;q=0.5 > Accept-Encoding gzip,deflate > Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7 > Keep-Alive 115 > Connection keep-alive > Referer http://my.perqapp.com/login > Cookie perq=20; __utma=90567579.1481902958.1285699566.1285768162.1285775102.4; __utmz=90567579.1285699566.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); perqworks=c7jcfkdueap5ggtbds500al703; __utmb=90567579.4.10.1285775102; __utmc=90567579 > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -- email: felix at seconddrawer.com.au web: http://seconddrawer.com.au/ gpg: E6FC 5BC6 268D B874 E546 8F6F A2BB 220B D5F6 92E3 Please don't send me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html From tungcnw at gmail.com Thu Sep 30 04:52:11 2010 From: tungcnw at gmail.com (Dang Tung) Date: Thu, 30 Sep 2010 11:52:11 +0700 Subject: Fixing backend URLs In-Reply-To: References: <2006354608710103182@unknownmsgid> <20100915032231.GC497@grum.quex.org> <0FAEA775-D78B-4CD6-9C0C-972ADE57287F@gmail.com> <20100924043515.GA30650@grum.quex.org> <20100924051235.GA31378@grum.quex.org> Message-ID: Are there any solutions for my issue, please help, I'm still waiting your reply. Thanks. On Mon, Sep 27, 2010 at 11:26 AM, Dang Tung wrote: > here is the header when I used wget: > > front-end: wget -S -O /dev/null http://myaddress/ > > HTTP request sent, awaiting response... > HTTP/1.1 200 OK > Server: Apache/2.2.3 (CentOS) > X-Powered-By: PHP/5.2.10 > Set-Cookie: symfony=8nrdjbj0q3a5367cr3eg2u7po7; path=/ > Last-Modified: Mon, 27 Sep 2010 04:19:22 GMT > Cache-Control: max-age=0 > Expires: Mon, 27 Sep 2010 04:19:21 GMT > Content-Type: text/html; charset=utf-8 > Content-Length: 63066 > Date: Mon, 27 Sep 2010 04:18:02 GMT > X-Varnish: 153748005 > Age: 0 > Via: 1.1 varnish > Connection: keep-alive > Length: 63066 (62K) [text/html] > Saving to: `/dev/null' > > 100%[======================================>] 63,066 --.-K/s in > 0.1s > > 2010-09-27 11:17:50 (577 KB/s) - `/dev/null' saved [63066/63066] > > > > backend: wget -S -O /dev/null http://myaddress/admin > > wget -S -O /dev/null http://myaddress/admin > --2010-09-27 11:18:35-- http://myaddress/admin > Connecting to myaddress:80... connected. > HTTP request sent, awaiting response... > HTTP/1.1 302 Found > Server: Apache/2.2.3 (CentOS) > X-Powered-By: PHP/5.2.10 > Set-Cookie: symfony=lli9m9oog3flp23ggchhbt5s71; path=/ > Location: http://myaddress:8080/admin.php > Cache-Control: max-age=0 > Expires: Mon, 27 Sep 2010 04:20:08 GMT > Content-Type: text/html; charset=utf-8 > Content-Length: 106 > Date: Mon, 27 Sep 2010 04:18:48 GMT > X-Varnish: 153748006 > Age: 0 > Via: 1.1 varnish > Connection: keep-alive > Location: http://myaddress:8080/admin.php [following] > --2010-09-27 11:18:35-- http://myaddress:8080/admin.php > Connecting to myaddress:8080... failed: No route to host. > > I saw 302 found but I don't know what can I do then. 8080, here's the > backend-port. > > Thanks and regards. > > > > Hmm. Can you provide the headers of one of the URLs you're testing, with >> as little anonymisation as possible? e.g. using >> >> wget -S -O /dev/null http://www.example.com/ >> >> Is it possible that your test URL is already cached by Varnish? >> vcl_fetch() is only called when the resource was fetched from the >> backend, so if it was already cached by Varnish that code will never be >> called. Either purge the URL or add similar code in vcl_hit(). >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > > > > -- > Nguyen Dang Tung > -- Nguyen Dang Tung -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at mm.quex.org Thu Sep 30 05:07:46 2010 From: varnish at mm.quex.org (Michael Alger) Date: Thu, 30 Sep 2010 13:07:46 +0800 Subject: Fixing backend URLs In-Reply-To: References: <2006354608710103182@unknownmsgid> <20100915032231.GC497@grum.quex.org> <0FAEA775-D78B-4CD6-9C0C-972ADE57287F@gmail.com> <20100924043515.GA30650@grum.quex.org> <20100924051235.GA31378@grum.quex.org> Message-ID: <20100930050746.GA5024@grum.quex.org> On Thu, Sep 30, 2010 at 11:52:11AM +0700, Dang Tung wrote: > Are there any solutions for my issue, please help, I'm still > waiting your reply. I don't really have any other ideas; the code should work if it's run, so I think you need to try to determine why the code isn't being run. You can use varnishlog to monitor the request flow and see if you can work it out that way. Or, you can add some debugging statements to log messages or add headers to the response at various points in your VCL so you can determine exactly how far Varnish is getting, and then from that work out why it's not running the code in question. You might also want to see if it's possible to adjust your application's configuration so it emits the 'correct' Location header itself; it's generally better to configure the application to know what its external address is rather than to rewrite things, as rewriting the Location: header won't do anything to fix fully-qualified URLs embedded in the HTML. Alternatively if you're willing to attach your entire Varnish configuration we might be able to work it out from that. > On Mon, Sep 27, 2010 at 11:26 AM, Dang Tung wrote: > > > here is the header when I used wget: > > > > backend: wget -S -O /dev/null http://myaddress/admin > > > > wget -S -O /dev/null http://myaddress/admin > > --2010-09-27 11:18:35-- http://myaddress/admin > > Connecting to myaddress:80... connected. > > HTTP request sent, awaiting response... > > HTTP/1.1 302 Found > > Server: Apache/2.2.3 (CentOS) > > X-Powered-By: PHP/5.2.10 > > Set-Cookie: symfony=lli9m9oog3flp23ggchhbt5s71; path=/ > > Location: http://myaddress:8080/admin.php > > Cache-Control: max-age=0 > > Expires: Mon, 27 Sep 2010 04:20:08 GMT > > Content-Type: text/html; charset=utf-8 > > Content-Length: 106 > > Date: Mon, 27 Sep 2010 04:18:48 GMT > > X-Varnish: 153748006 > > Age: 0 > > Via: 1.1 varnish > > Connection: keep-alive From tfheen at varnish-software.com Thu Sep 30 05:52:25 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 30 Sep 2010 07:52:25 +0200 Subject: How do you confirm cookies are removed from static files? In-Reply-To: (Chris Altman's message of "Wed, 29 Sep 2010 12:06:25 -0400") References: Message-ID: <87d3rvdbhi.fsf@qurzaw.linpro.no> ]] Chris Altman | I am new to Varnish and have read the documentation. My goal is to remove all cookies from static files. I am using Varnish 2.1.3 | | My VCL is: | | sub vcl_recv { | if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { | unset req.http.cookie; | } | } | | sub vcl_fetch { | if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { | unset beresp.http.set-cookie; | } | } This looks fine. | I am confused because it appears the static files are served from | Varnish but still have a cookie in their Request Headers. How do I | remove the Request Headers cookie? Am I still gaining the benefits of | Varnish if the Request Headers cookies exist? If you use varnishlog to look at the request sent to the backend, you'll notice that it does not have a Cookie header sent with it. Varnish obviously can't change what the client sends to Varnish, only what it forwards to the backend. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Thu Sep 30 05:53:51 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 30 Sep 2010 07:53:51 +0200 Subject: varnish caches too many object variances In-Reply-To: <1285778598.1855.493.camel@oasis> (Taylan Develioglu's message of "Wed, 29 Sep 2010 18:43:18 +0200") References: <1285694490.1855.65.camel@oasis> <1285778598.1855.493.camel@oasis> Message-ID: <878w2jdbf4.fsf@qurzaw.linpro.no> ]] Taylan Develioglu | So, nobody has any other idea on why we have more objects than | object_heads ? It could be a bug in 2.1.3, namely the one fixed with r5195. If you try running with the current 2.1 branch from svn, that includes the fix. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tdevelioglu at ebuddy.com Thu Sep 30 12:21:32 2010 From: tdevelioglu at ebuddy.com (Taylan Develioglu) Date: Thu, 30 Sep 2010 14:21:32 +0200 Subject: varnish caches too many object variances In-Reply-To: <878w2jdbf4.fsf@qurzaw.linpro.no> References: <1285694490.1855.65.camel@oasis> <1285778598.1855.493.camel@oasis> <878w2jdbf4.fsf@qurzaw.linpro.no> Message-ID: <1285849292.1855.519.camel@oasis> http://www.varnish-cache.org/trac/changeset/5195 I understand it is a cosmetic issue then, no rush. Thank you Tollef. On Thu, 2010-09-30 at 07:53 +0200, Tollef Fog Heen wrote: > ]] Taylan Develioglu > > | So, nobody has any other idea on why we have more objects than > | object_heads ? > > It could be a bug in 2.1.3, namely the one fixed with r5195. If you try > running with the current 2.1 branch from svn, that includes the fix. > From mrbits.dcf at gmail.com Thu Sep 30 15:08:41 2010 From: mrbits.dcf at gmail.com (MrBiTs) Date: Thu, 30 Sep 2010 12:08:41 -0300 Subject: backend_toolate - Backend conn. was closed In-Reply-To: <77659.1285714309@critter.freebsd.dk> References: <77659.1285714309@critter.freebsd.dk> Message-ID: <4CA4A7F9.3060703@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 09/28/2010 19:51 , Poul-Henning Kamp wrote: > In message <4CA26D2D.7060001 at yahoo.com.br>, Flavio Torres writes: > >> Please, in varnishstat, what does backend_toolate means ? >> >> >> backend_toolate 826901 3.28 Backend conn. was closed > > The backend closed an idle connection before we tried to reuse it. > > Varnish picks or opens another connection. > > If the number is as high as you have here, it may pay off to > increase the idle-timeout on your backend. > Hi, All Let me try to understand. Is backend_toolate with high values good or bad ? AFAIU, if varnish is opening and closing connections, this is bad and I need to change some configuration. What do you mean with "it may pay off to increase the idle-timeout" ? Thanks a lot .0. MrBiTs - mrbits.dcf at gmail.com ..0 GnuPG - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6EC818FC2B3CA5AB 000 http://www.mrbits.com.br -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQEcBAEBCAAGBQJMpKf4AAoJEG7IGPwrPKWrx/oH/2nFQ8A8l8ApHdzJwfCPBheU Uf0cDwxV+0LxGzs4n8Isbx0dhY44wD5H1jIwW2xEwhtQy990r7bNkHAFhQq4dtPE PkqYQIeC2he0rNhXU5sO4bUzF32bVy64WXIFH+zjFJY/3kcUD6ZpktuoyVza+Kcu PtZ7zdfQEDmMFQY9am+4QBKEwMoUGApfOqWzSbBz8YIYCgp0GOxNcWarnKNX1XkA Qg8JftKafJO5hX5rf3x1DKLc5o+KR+u0KUfJXUmdnaQREgHdCU3DbE2lKw/usy+K uIklROIRVY8Imuhq+DA2XfCELAq7groRq+J+XAEOYZ8Sz3VUHx9QD6Y7SoX2b/g= =9Kc/ -----END PGP SIGNATURE----- From bnowacky at competitorgroup.com Thu Sep 30 16:52:58 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Thu, 30 Sep 2010 09:52:58 -0700 Subject: Panic Signal 6. Message-ID: <9C0B05C0-A205-4E41-B906-034F9B4EC2A4@competitorgroup.com> Hey everyone ! Wondering if someone can lend a hand. We've gotten our servers upgraded to 64-bit OS, and seen tremendous stability enhancement with varnish over our 32-bit os, which crashed every 5-10 minutes. We can now get about 2 days of uptime before we have a new child process crash. Logs are listed below.. Wondering if something obvious jumps out at anyone as to what the cause of this may be. Appreciate any insight from the community ! Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) died signal=6 Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) Panic message: Assert error in WS_Release(), cache_ws.c line 193: Condition(bytes <= ws->e - ws->f) not true. errno = 104 (Connection reset by peer) thread = (cache-worker) ident = Linux,2.6.18-194.el5,x86_64,-smalloc,-hcritbit,epoll Backtrace: 0x422616: /usr/sbin/varnishd [0x422616] 0x42d475: /usr/sbin/varnishd(WS_Release+0xf5) [0x42d475] 0x427b33: /usr/sbin/varnishd [0x427b33] 0x42bc45: /usr/sbin/varnishd(VRT_SetHdr+0xf5) [0x42bc45] 0x2aaaab003e76: ./vcl.1P9zoqAU.so [0x2aaaab003e76] 0x427006: /usr/sbin/varnishd(VCL_deliver_method+0x46) [0x427006] 0x4130cf: /usr/sbin/varnishd [0x4130cf] 0x413e49: /usr/sbin/varnishd(CNT_Session+0x369) [0x413e49] 0x424a68: /usr/sbin/varnishd [0x424a68] 0x423d4d: /usr/sbin/varnishd [0x423d4d] sp = 0x2aaac060e008 { fd = 351, id = 351, xid = 1104824170, client = 63.240.230.197:26479, step = STP_DELIVER, handling = deliver, restarts = 0, esis = 0 ws = 0x2aaac060e078 { id = "sess", {s,f Sep 30 16:37:17 srv860 varnishd[16195]: child (32562) Started Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said Child starts -------------- next part -------------- An HTML attachment was scrubbed... URL: From scaunter at topscms.com Thu Sep 30 17:57:47 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Thu, 30 Sep 2010 13:57:47 -0400 Subject: Panic Signal 6. In-Reply-To: <9C0B05C0-A205-4E41-B906-034F9B4EC2A4@competitorgroup.com> References: <9C0B05C0-A205-4E41-B906-034F9B4EC2A4@competitorgroup.com> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C502366@TMG-EVS02.torstar.net> What version of varnish? Have you tried with -s file,/path/to/file ,1075200000 instead of -s malloc? Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky Sent: September-30-10 12:53 PM To: varnish-misc at varnish-cache.org Subject: Panic Signal 6. Hey everyone ! Wondering if someone can lend a hand. We've gotten our servers upgraded to 64-bit OS, and seen tremendous stability enhancement with varnish over our 32-bit os, which crashed every 5-10 minutes. We can now get about 2 days of uptime before we have a new child process crash. Logs are listed below.. Wondering if something obvious jumps out at anyone as to what the cause of this may be. Appreciate any insight from the community ! Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) died signal=6 Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) Panic message: Assert error in WS_Release(), cache_ws.c line 193: Condition(bytes <= ws->e - ws->f) not true. errno = 104 (Connection reset by peer) thread = (cache-worker) ident = Linux,2.6.18-194.el5,x86_64,-smalloc,-hcritbit,epoll Backtrace: 0x422616: /usr/sbin/varnishd [0x422616] 0x42d475: /usr/sbin/varnishd(WS_Release+0xf5) [0x42d475] 0x427b33: /usr/sbin/varnishd [0x427b33] 0x42bc45: /usr/sbin/varnishd(VRT_SetHdr+0xf5) [0x42bc45] 0x2aaaab003e76: ./vcl.1P9zoqAU.so [0x2aaaab003e76] 0x427006: /usr/sbin/varnishd(VCL_deliver_method+0x46) [0x427006] 0x4130cf: /usr/sbin/varnishd [0x4130cf] 0x413e49: /usr/sbin/varnishd(CNT_Session+0x369) [0x413e49] 0x424a68: /usr/sbin/varnishd [0x424a68] 0x423d4d: /usr/sbin/varnishd [0x423d4d] sp = 0x2aaac060e008 { fd = 351, id = 351, xid = 1104824170, client = 63.240.230.197:26479, step = STP_DELIVER, handling = deliver, restarts = 0, esis = 0 ws = 0x2aaac060e078 { id = "sess", {s,f Sep 30 16:37:17 srv860 varnishd[16195]: child (32562) Started Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said Child starts -------------- next part -------------- An HTML attachment was scrubbed... URL: From bnowacky at competitorgroup.com Thu Sep 30 18:02:10 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Thu, 30 Sep 2010 11:02:10 -0700 Subject: Panic Signal 6. In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C502366@TMG-EVS02.torstar.net> References: <9C0B05C0-A205-4E41-B906-034F9B4EC2A4@competitorgroup.com> <7F0AA702B8A85A4A967C4C8EBAD6902C502366@TMG-EVS02.torstar.net> Message-ID: 2.1.2, would prefer to use malloc over file storage if possible, but haven't tried file to see if that's the issue. On Sep 30, 2010, at 10:57 AM, Caunter, Stefan wrote: What version of varnish? Have you tried with ?s file,/path/to/file ,1075200000 instead of ?s malloc? Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky Sent: September-30-10 12:53 PM To: varnish-misc at varnish-cache.org Subject: Panic Signal 6. Hey everyone ! Wondering if someone can lend a hand. We've gotten our servers upgraded to 64-bit OS, and seen tremendous stability enhancement with varnish over our 32-bit os, which crashed every 5-10 minutes. We can now get about 2 days of uptime before we have a new child process crash. Logs are listed below.. Wondering if something obvious jumps out at anyone as to what the cause of this may be. Appreciate any insight from the community ! Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) died signal=6 Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) Panic message: Assert error in WS_Release(), cache_ws.c line 193: Condition(bytes <= ws->e - ws->f) not true. errno = 104 (Connection reset by peer) thread = (cache-worker) ident = Linux,2.6.18-194.el5,x86_64,-smalloc,-hcritbit,epoll Backtrace: 0x422616: /usr/sbin/varnishd [0x422616] 0x42d475: /usr/sbin/varnishd(WS_Release+0xf5) [0x42d475] 0x427b33: /usr/sbin/varnishd [0x427b33] 0x42bc45: /usr/sbin/varnishd(VRT_SetHdr+0xf5) [0x42bc45] 0x2aaaab003e76: ./vcl.1P9zoqAU.so [0x2aaaab003e76] 0x427006: /usr/sbin/varnishd(VCL_deliver_method+0x46) [0x427006] 0x4130cf: /usr/sbin/varnishd [0x4130cf] 0x413e49: /usr/sbin/varnishd(CNT_Session+0x369) [0x413e49] 0x424a68: /usr/sbin/varnishd [0x424a68] 0x423d4d: /usr/sbin/varnishd [0x423d4d] sp = 0x2aaac060e008 { fd = 351, id = 351, xid = 1104824170, client = 63.240.230.197:26479, step = STP_DELIVER, handling = deliver, restarts = 0, esis = 0 ws = 0x2aaac060e078 { id = "sess", {s,f Sep 30 16:37:17 srv860 varnishd[16195]: child (32562) Started Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said Child starts -------------- next part -------------- An HTML attachment was scrubbed... URL: From ross at trademe.co.nz Thu Sep 30 19:06:31 2010 From: ross at trademe.co.nz (Ross Brown) Date: Fri, 1 Oct 2010 08:06:31 +1300 Subject: Panic Signal 6. In-Reply-To: References: <9C0B05C0-A205-4E41-B906-034F9B4EC2A4@competitorgroup.com> <7F0AA702B8A85A4A967C4C8EBAD6902C502366@TMG-EVS02.torstar.net> Message-ID: <1FF67D7369ED1A45832180C7C1109BCA264AAC8BCB@tmmail0.trademe.local> 2 Days? That's certainly.... sub-optimal. What OS are you running? Our 8 x Varnish servers (64-bit Ubuntu 10.04.1 LTS) are rock solid and easily run for weeks without any issues. We run with malloc,24G (32Gb RAM installed) Can you share your startup options and VCL? From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky Sent: Friday, 1 October 2010 7:02 a.m. To: Caunter, Stefan Cc: varnish-misc at varnish-cache.org Subject: Re: Panic Signal 6. 2.1.2, would prefer to use malloc over file storage if possible, but haven't tried file to see if that's the issue. On Sep 30, 2010, at 10:57 AM, Caunter, Stefan wrote: What version of varnish? Have you tried with -s file,/path/to/file ,1075200000 instead of -s malloc? Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky Sent: September-30-10 12:53 PM To: varnish-misc at varnish-cache.orgSubject: Panic Signal 6. Hey everyone ! Wondering if someone can lend a hand. We've gotten our servers upgraded to 64-bit OS, and seen tremendous stability enhancement with varnish over our 32-bit os, which crashed every 5-10 minutes. We can now get about 2 days of uptime before we have a new child process crash. Logs are listed below.. Wondering if something obvious jumps out at anyone as to what the cause of this may be. Appreciate any insight from the community ! Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) died signal=6 Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) Panic message: Assert error in WS_Release(), cache_ws.c line 193: Condition(bytes <= ws->e - ws->f) not true. errno = 104 (Connection reset by peer) thread = (cache-worker) ident = Linux,2.6.18-194.el5,x86_64,-smalloc,-hcritbit,epoll Backtrace: 0x422616: /usr/sbin/varnishd [0x422616] 0x42d475: /usr/sbin/varnishd(WS_Release+0xf5) [0x42d475] 0x427b33: /usr/sbin/varnishd [0x427b33] 0x42bc45: /usr/sbin/varnishd(VRT_SetHdr+0xf5) [0x42bc45] 0x2aaaab003e76: ./vcl.1P9zoqAU.so [0x2aaaab003e76] 0x427006: /usr/sbin/varnishd(VCL_deliver_method+0x46) [0x427006] 0x4130cf: /usr/sbin/varnishd [0x4130cf] 0x413e49: /usr/sbin/varnishd(CNT_Session+0x369) [0x413e49] 0x424a68: /usr/sbin/varnishd [0x424a68] 0x423d4d: /usr/sbin/varnishd [0x423d4d] sp = 0x2aaac060e008 { fd = 351, id = 351, xid = 1104824170, client = 63.240.230.197:26479, step = STP_DELIVER, handling = deliver, restarts = 0, esis = 0 ws = 0x2aaac060e078 { id = "sess", {s,f Sep 30 16:37:17 srv860 varnishd[16195]: child (32562) Started Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said Child starts -------------- next part -------------- An HTML attachment was scrubbed... URL: From bnowacky at competitorgroup.com Thu Sep 30 19:10:21 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Thu, 30 Sep 2010 12:10:21 -0700 Subject: Panic Signal 6. In-Reply-To: <1FF67D7369ED1A45832180C7C1109BCA264AAC8BCB@tmmail0.trademe.local> References: <9C0B05C0-A205-4E41-B906-034F9B4EC2A4@competitorgroup.com> <7F0AA702B8A85A4A967C4C8EBAD6902C502366@TMG-EVS02.torstar.net> <1FF67D7369ED1A45832180C7C1109BCA264AAC8BCB@tmmail0.trademe.local> Message-ID: <804FB091-853B-4D6C-BD56-99743186472A@competitorgroup.com> We're running CentOS 64 bit, 5.5... Here's our VCL, and startup flags: sysconfig/varnish: -u varnish -s malloc,800m -p cli_timeout=1200s -p thread_pools=1 -p thread_pool_min=120 -p thread_pool_max=200 -p listen_depth=1024 p shm_reclen=32768 -p sess_workspace=8192 -p http_headers=32 -p sess_timeout=10 -p session_linger=100 -p between_bytes_timeout=10 -p first_byte_timeout=10 -h critbit VCL: # Default backend definition. Set this to point to your content # server. # backend default { .host = "64.237.104.85"; .port = "8080"; } acl purge { "localhost"; "64.237.104.85"; } sub vcl_recv { # Serve stale content while fresh is retrieved for 2minutes. set req.grace = 6m; if (req.http.host ~ "elite.competitor.com") { if (req.url ~ "^/confcard$") { set req.http.New-Location = regsub(req.url,"$","/"); error 301 "Redirecting you to the slash representation..."; } if (req.url ~ "^/confcard/") { return (pipe); } } if (req.http.host ~ "forums.competitor.com") { return (pipe); } if (req.http.host ~ "svn.competitor.com"){ return (pipe); } if (req.http.host ~ "competitorradio.competitor.com") { if (req.url ~ "^/wp-admin$") { set req.http.New-Location = regsub(req.url,"$","/"); error 301 "Redirecting you to the slash representation..."; } else { return (pipe); } } # always exempt Dev.lan from cache. if (!(req.http.host ~ "competitor.com")) { return (pipe); } # Exempt mobile users from cache... straight to backend. if (req.http.User-Agent ~ "(iPhone|iPod|iPad)|(Android)|(BlackBerry9530)|(webOS)") { return (pipe); } # redirect URL for trailing slash when logging in. if (req.url ~ "^/wp-admin$") { set req.http.New-Location = regsub(req.url,"$","/"); error 301 "Redirecting you to the slash representation..."; } if (req.http.cookie ~ "wordpress_") { return (pipe); } # if wp-admin or wp-login is in the URL, pipe to apache. if (req.url ~ "/wp-(admin|login)") { return (pipe); } # otherwise just unset cookie and cache all pages. else { unset req.http.cookie; } # # cache the home page. # if (req.url ~ "^/$") { # unset req.http.cookie; # } # #category page # if (req.url ~ "^/category(.)*") { # unset req.http.cookie; # } # #post page with date in URL # if (req.url ~ "^/?p=[0-9](.)*") { # unset req.http.cookie; # } # #post page with date in URL # if (req.url ~ "^/[0-9](.)*") { # unset req.http.cookie; # } if (req.http.Accept-Encoding) { if (req.url ~ "^/[^?]+\.(ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.*|)$") { # No point in compressing these remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } elsif (req.http.Accept-Encoding ~ "MSIE 6") { set req.http.Accept-Encoding = "MSIE"; } elsif (req.http.Accept-Encoding ~ "MSIE 7") { set req.http.Accept-Encoding = "MSIE"; } elsif (req.http.Accept-Encoding ~ "MSIE 8") { set req.http.Accept-Encoding = "MSIE"; } elsif (req.http.Accept-Encoding ~ "Mozilla") { set req.http.Accept-Encoding = "mozilla"; } else { # unkown algorithm remove req.http.Accept-Encoding; } } unset req.http.user-agent; if (req.url ~ "^/[^?]+\.(jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.*|)$") { remove req.http.cookie; set req.url = regsub(req.url, "\?.*$", ""); } # don't bother caching large files if (req.url ~ "^/[^?]+\.(mp3|pdf|flv|mov|mp4|mpg|mpeg|avi|dmg|swf)(\?.*|)$") { return (pipe); } # accept purge requests from Wordpress. if (req.request == "PURGE") { if(!client.ip ~ purge) { error 405 "Not allowed."; } purge_url(req.url); error 200 "Purged"; purge("req.url == " req.url); } #################### # End sub_recv # #################### } sub vcl_deliver { #add cache hit data if (obj.hits > 0) { #if hit add hit count set resp.http.X-Served-By = server.hostname; set resp.http.X-Cache = "HIT"; set resp.http.X-Cache-Hits = obj.hits; set resp.http.X-Encoding = req.http.Accept-Encoding; set resp.http.X-Debug = req.http.host; } else { set resp.http.X-Served-By = server.hostname; set resp.http.X-Cache = "MISS"; set resp.http.X-Encoding = req.http.Accept-Encoding; set resp.http.X-Debug = req.http.host; } } sub vcl_error { # for redirect on trailing slash. if (req.http.New-Location) { set obj.http.Location = req.http.New-Location; } } sub vcl_fetch { if (beresp.status != 200 && beresp.status != 403 && beresp.status != 404 && beresp.status != 405 && beresp.status != 301 && beresp.status != 302) { restart; } if (beresp.http.cookie ~ "wordpress_") { return (pass); } # Serve stale content while fresh is retrieved for 2minutes. set beresp.grace = 6m; set beresp.ttl = 720s; if (beresp.ttl < 720s) { set beresp.ttl = 720s; } if (!beresp.cacheable) { set beresp.ttl = 0s; } if (beresp.http.Set-Cookie) { return(deliver); } # make sure cookies aren't set for retrieved static objects. if (req.url ~ "^/[^?]+\.(jpeg|jpg|png|gif|ico)(\?.*|)$") { set beresp.http.Cache-Control = "max-age=31536000, public"; unset beresp.http.set-cookie; } if (req.url ~ "^/[^?]+\.(js|html|htm|txt)(\?.*|)$") { set beresp.http.Cache-Control = "max-age=1728000, public, must-revalidate"; unset beresp.http.set-cookie; } if (req.url ~ "^/[^?]+\.(css)(\?.*|)$") { set beresp.http.Cache-Control = "max-age=1728000, public, must-revalidate"; unset beresp.http.set-cookie; } # cache the home page. if (req.url ~ "^/$") { unset beresp.http.cookie; set beresp.ttl = 600s; } if (req.url ~ "^/category(.)*") { unset beresp.http.cookie; } if (req.url ~ "^(.)/[0-9](.)*") { unset beresp.http.cookie; } if (req.url ~ "^/?p=[0-9](.)*") { unset beresp.http.cookie; } #Change Web Server header because we can. unset beresp.http.Server; set beresp.http.Server = "Mini Con Adventure Team"; #################### # End sub_fetch # #################### } sub vcl_pipe { # # http://www.varnish-cache.org/ticket/451 # # This forces every pipe request to be the first one. set bereq.http.connection = "close"; } sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } } sub vcl_hash { # //hash the object with url+host set req.hash += req.url; set req.hash += req.http.host; set req.hash += req.http.User-Agent; set req.hash += req.http.cookie; return(hash); } On Sep 30, 2010, at 12:06 PM, Ross Brown wrote: 2 Days? That?s certainly?. sub-optimal. What OS are you running? Our 8 x Varnish servers (64-bit Ubuntu 10.04.1 LTS) are rock solid and easily run for weeks without any issues. We run with malloc,24G (32Gb RAM installed) Can you share your startup options and VCL? From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky Sent: Friday, 1 October 2010 7:02 a.m. To: Caunter, Stefan Cc: varnish-misc at varnish-cache.org Subject: Re: Panic Signal 6. 2.1.2, would prefer to use malloc over file storage if possible, but haven't tried file to see if that's the issue. On Sep 30, 2010, at 10:57 AM, Caunter, Stefan wrote: What version of varnish? Have you tried with ?s file,/path/to/file ,1075200000 instead of ?s malloc? Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky Sent: September-30-10 12:53 PM To: varnish-misc at varnish-cache.orgSubject: Panic Signal 6. Hey everyone ! Wondering if someone can lend a hand. We've gotten our servers upgraded to 64-bit OS, and seen tremendous stability enhancement with varnish over our 32-bit os, which crashed every 5-10 minutes. We can now get about 2 days of uptime before we have a new child process crash. Logs are listed below.. Wondering if something obvious jumps out at anyone as to what the cause of this may be. Appreciate any insight from the community ! Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) died signal=6 Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) Panic message: Assert error in WS_Release(), cache_ws.c line 193: Condition(bytes <= ws->e - ws->f) not true. errno = 104 (Connection reset by peer) thread = (cache-worker) ident = Linux,2.6.18-194.el5,x86_64,-smalloc,-hcritbit,epoll Backtrace: 0x422616: /usr/sbin/varnishd [0x422616] 0x42d475: /usr/sbin/varnishd(WS_Release+0xf5) [0x42d475] 0x427b33: /usr/sbin/varnishd [0x427b33] 0x42bc45: /usr/sbin/varnishd(VRT_SetHdr+0xf5) [0x42bc45] 0x2aaaab003e76: ./vcl.1P9zoqAU.so [0x2aaaab003e76] 0x427006: /usr/sbin/varnishd(VCL_deliver_method+0x46) [0x427006] 0x4130cf: /usr/sbin/varnishd [0x4130cf] 0x413e49: /usr/sbin/varnishd(CNT_Session+0x369) [0x413e49] 0x424a68: /usr/sbin/varnishd [0x424a68] 0x423d4d: /usr/sbin/varnishd [0x423d4d] sp = 0x2aaac060e008 { fd = 351, id = 351, xid = 1104824170, client = 63.240.230.197:26479, step = STP_DELIVER, handling = deliver, restarts = 0, esis = 0 ws = 0x2aaac060e078 { id = "sess", {s,f Sep 30 16:37:17 srv860 varnishd[16195]: child (32562) Started Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said Child starts -------------- next part -------------- An HTML attachment was scrubbed... URL: From eli at fiercemarkets.com Thu Sep 30 22:09:06 2010 From: eli at fiercemarkets.com (Eli Dickinson) Date: Thu, 30 Sep 2010 18:09:06 -0400 Subject: Strange problem with truncated responses when POSTing data Message-ID: Greetings, First, thanks for Varnish! It's a great piece of software. Today, though, I've been pulling my hair out with a strange problem and I'm hoping someone here can help. I've got Varnish sitting in front of Apache on one server and I've got a PHP script that POSTs data from another server. If the length of the content I'm posting is over about 1500 bytes, I get an incomplete response from Varnish. Like, literally I only get half the page back. According to Wireshark, it looks like I get 1 or 2 packets of data (it's not consistent) and then there's suddently a RST/ACK. Weirder still, an almost identical request from cURL works fine. So it's either some kind of funny timing issue or it's something very specific to the way PHP makes HTTP requests. The problem disappears if: - I tell varnish to pipe instead of pass -?I run the PHP script on the same server and have it POST to localhost (which makes me think timing is a factor...) - I POST less than 1kb of data I'm running varnish-2.1.3 SVN and I don't see anything unusual in the syslog or when running varnishlog (though I'm not exactly sure what I'm looking for) Here's what my request looks like: POST / HTTP/1.0 Host: web2.servicecenter.fiercemarkets.com Content-Length: 1517 ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX [...snip...] And the response I get back looks like: HTTP/1.1 403 Forbidden Server: Apache/2.2.3 (Red Hat) X-Powered-By: PHP/5.2.14 Expires: Sun, 11 Mar 1984 12:00:00 GMT Last-Modified: Thu, 30 Sep 2010 21:40:44 +0000 Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 ETag: "1285882844" Content-Type: text/html; charset=utf-8 Content-Length: 4104 Date: Thu, 30 Sep 2010 21:40:44 GMT X-Varnish: 598880740 Age: 0 Via: 1.1 varnish Connection: close ?? [...snip...] The response looks fine (content-length is correct), aside from the fact that I don't get all of it. I'm stumped. Any thoughts? -- Eli Dickinson FierceMarkets, Inc. 202-824-5058 From Bayron.Guevara at laprensa.hn Mon Sep 27 18:04:28 2010 From: Bayron.Guevara at laprensa.hn (Bayron Jose Guevara Calderon) Date: Mon, 27 Sep 2010 12:04:28 -0600 Subject: Backend load balancing delegation in order to support sticky sessions Message-ID: <9F55CC3AD20584449418F23472E35437019290A18AB6@EXCHANGE1.GrupoEditorial.hn> Hello community, Currently I've installed Varnish 2.0.6 and I'm using a cluster of Web Servers. In order to find a work around the session clustering issue, I thought implement session stickiness as part of the load balancing configuration, so all subsequent requests from the same user would go to the same webserver and so ensure the session information consistency. But as I know, Varnish don't have integrated support to Stickiness, and therefore I have to delegate this function to a Load Balancer. Through this infrastructure the client requests would follow the below path: User Agent --> Load Balancer Public VIP --> One of the Varnish Servers --> (IF cache hit --> deliver cached content to Client ) ELSE ( Load Balancer Private VIP --> One of the Web Servers) To support sticky sessions my Load Balancer uses the Arrow Point cookie mechanism which, to be brief, add a cookie named ARPT that identified the user session. And according to the above path, Varnish sends requests and receive responses from the Load Balancer private VIP which needs that the cookie in question be passed transparently to and from the client. So, Varnish should take care of this cookie in two ways: the Cookie header in the incoming requests and the Set-Cookie in the responses from the Load Balancer. What do you recommend me to do?. I have ideas but I don't sure. For example, to extract the desired cookie from Cookie header I'm doing the following: ### Code Start #### vcl_receive { ... if (req.http.Cookie ~ "ARPT=") { set req.http.LB-Cookie = regsub(req.http.Cookie, ".*(ARPT=[^;]+).*", "\1"); } unset req.http.Cookie; ... } sub vcl_miss { set bereq.http.Cookie = req.http.LB-Cookie; } sub vcl_pass { set bereq.http.Cookie = req.http.LB-Cookie; } ### Code End #### To keep it in the http response, I guess: ### Code Start ### sub vcl_fetch { ... if (obj.http.Set-Cookie ~ "ARPT=") { #Save Set-Cookie, so it can be used in vcl_deliver set obj.http.X-Set-Cookie = obj.http.Set-Cookie; } ... } sub vcl_deliver { if (obj.hits > 0) { ... } else { #It was a cache miss ... #Set back Set-Cookie header in case it's set if (resp.http.X-Set-Cookie){ set resp.http.Set-Cookie = resp.http.X-Set-Cookie; unset resp.http.X-Set-Cookie; } } } ### Code End ### I don't know how to extract a cookie from a Set-Cookie header, but I tried with this code: obj.http.LB-Cookie = regsub(obj.http.Set-Cookie, ".*(ARPT=[^;]+(;[^=]+=[^;]+)*).*", "\1"); Nevertheless, it appears don't work fine. Am I in the right way? Is there a better solution? Thank you in advance. --------------------------------------------------- Bayron Guevara Web Programmer Bayron.Guevara at laprensa.hn Grupo OPSA Barrio Guamilito 3ra. Avenida 6 y 7 calle, No.34 PBX:(504) 553 3101, Ext:3213, Cel:9987-8669 San Pedro Sula, Honduras, C.A. AVISO DE CONFIDENCIALIDAD: Este e-mail contiene informacion que es confidencial y solo puede ser utilizada por las personas o entidades a la cuales esta dirigida. Si usted no es el destinatario autorizado, cualquier modificacion, retension, difusion o copia total o parcial esta prohibida. CONFIDENTIALITY NOTICE: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. From caltman at matterco.com Wed Sep 29 16:04:08 2010 From: caltman at matterco.com (Chris Altman) Date: Wed, 29 Sep 2010 12:04:08 -0400 Subject: How do you confirm cookies are removed from static files? Message-ID: I am new to Varnish and have read the documentation. My goal is to remove all cookies from static files. My VCL is: sub vcl_recv { if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { unset req.http.cookie; } } sub vcl_fetch { if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { unset beresp.http.set-cookie; } } Question: I am confused because it appears the static files are served from Varnish but still have a cookie in their Request Headers. How do I remove the Request Headers cookie? Am I still gaining the benefits of Varnish if the Request Headers cookies exist? Thank you, Chris Altman http://my.perqworks.com http://prtclr.com Response Headersview source Server Apache/2.2.14 (Ubuntu) X-Powered-By PHP/5.3.2-1ubuntu4.5 Expires Thu, 29 Sep 2011 15:56:42 GMT Vary Accept-Encoding Last-Modified Thu, 24 Jun 2010 21:16:50 GMT Cache-Control max-age=31536000, public Content-Encoding gzip Content-Type text/css; charset=utf-8 Content-Length 3339 Date Wed, 29 Sep 2010 15:56:42 GMT X-Varnish 2062520895 Age 0 Via 1.1 varnish Connection keep-alive Request Headersview source Host my.perqapp.com User-Agent Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10 Accept text/css,*/*;q=0.1 Accept-Language en-us,en;q=0.5 Accept-Encoding gzip,deflate Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive 115 Connection keep-alive Referer http://my.perqapp.com/login Cookie perq=20; __utma=90567579.1481902958.1285699566.1285768162.1285775102.4; __utmz=90567579.1285699566.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); perqworks=c7jcfkdueap5ggtbds500al703; __utmb=90567579.4.10.1285775102; __utmc=90567579 Matter. (404) 961-2065 office (404) 849-0870 mobile www.matterco.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From felipe.ferreira at infoglobo.com.br Fri Sep 3 15:10:26 2010 From: felipe.ferreira at infoglobo.com.br (Felipe Couto Ferreira - Tecnologia - Infoglobo) Date: Fri, 03 Sep 2010 15:10:26 -0000 Subject: How to Stop Varnish the correct way - Drain Connections Message-ID: First, Congratulations for the makers of Varnish. It is a golden code, it improved our website from 10 seconds response to 2 seconds. I would like to know if there is a correctly way to stop the varnish deamon, I notice a memory increase and maybe restarting would solve, but I constantly have about 1000 connections and I dont want to close everyone. Basically, I need a draincommand is it possible?So that I after that I will restart the deamon. My little help to the project: How to monitor Varnish with Nagios thanks in advance, -- Felipe Ferreira http://felipeferreira.net - - - - - - - - - - - - - - - - - - - - AVISO IMPORTANTE / IMPORTANT NOTICE - - - - - - - - - - - - - - - - - - - - - - Esta mensagem pode conter informa??es confidenciais e somente o indiv?duo ou entidade a quem foi destinada pode utiliz?-la. A transmiss?o incorreta da mensagem n?o acarreta a perda de sua confidencialidade. Caso esta mensagem tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada de seu sistema imediatamente. ? vedado a qualquer pessoa que n?o seja o destinat?rio usar, revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de comunica??o sujeito a monitoramento. This message may include confidential information and only the intended addressee have the right to use it as is, or any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or erroneous transmission, please notify the senderdelete it from your system immediately. This communication environment is controlled and monitored. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -------------- next part -------------- An HTML attachment was scrubbed... URL: From wittwerch at gmail.com Wed Sep 8 13:37:08 2010 From: wittwerch at gmail.com (Christian Wittwer) Date: Wed, 08 Sep 2010 13:37:08 -0000 Subject: SSL and Varnish In-Reply-To: References: <4C8621A7.50804@gmail.com> Message-ID: I'm facing the same question at the moment. Why do you guys recommend/use nginx? I had a look at pound, and it looks pretty easy and lightweight for the purpose of a ssl endpoint. Are there any advantages/drawbacks using pound instead of nginx? Cheers, Christian 2010/9/7 Jim Hayter : > We run nginx to accept both SSL and non-SSL requests. ?This allows me to > have some logic to block certain requests at the nginx level and also > for nginx to return a customized error page if varnishd is not running. > Nginx on passes all remaining traffic to varnishd which sits in front of > multiple web servers. > > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Rob S > Sent: Tuesday, September 07, 2010 7:28 AM > To: varnish-misc at varnish-cache.org > Subject: SSL and Varnish > > ?If you've used SSL with varnish in production, can you share any > wisdom on your configuration and approaches? > > > We've been happily using Varnish in production for just over a year, and > > now want to use it for an SSL site so that we can use ESI, load > balancing and other such facilities offered by Varnish. ?I've searched > over the Varnish website, and can't see any examples of a recommended > SSL configuration. > http://plone.org/documentation/kb/plone-behind-varnish-using-pound-for-s > sl > provides some documentation, but I suspect there are other options. > > I'll happily consolidate the answers into a coherent document for the > website so that others can benefit in future. > > > Thanks, > > > Rob > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From marcos.corvalan at mercadolibre.com Tue Sep 14 16:55:54 2010 From: marcos.corvalan at mercadolibre.com (marcos.corvalan at mercadolibre.com) Date: Tue, 14 Sep 2010 16:55:54 -0000 Subject: How to monitor varnish with snmp Message-ID: <005f01cb542d$874b9490$95e2bdb0$@corvalan@mercadolibre.com> Hi everyone: How can I monitor data with snmp from varnish process. Since now thanks a lots. Saludos/Regards Marcos Corvalan Infraestructura IT Mercadolibre.com Tronador 4890 - 8 piso Capital Federal | Buenos Aires Argentina | CP 1430 | Tel +54 11 5352-8000 Ext 8426 email marcos.corvalan| msn marcoscorvalan_04 at hotmail.com Mobile : +54 11 37701933 -------------- next part -------------- An HTML attachment was scrubbed... URL: