From pprocacci at datapipe.com Thu May 2 12:03:16 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Thu, 2 May 2013 07:03:16 -0500 Subject: Error 413 Request Entity Too Large In-Reply-To: References: Message-ID: <20130502120316.GC7976@nat.myhome> Take a look at https://www.varnish-cache.org/trac/ticket/1016 Perhaps your problem is similar. ~Paul On Sun, Feb 03, 2013 at 04:17:43PM -0500, Piper Sponaas wrote: > How do I get rid of it. Its ruining my > life!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From pprocacci at datapipe.com Thu May 2 12:12:29 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Thu, 2 May 2013 07:12:29 -0500 Subject: swf not being delivered In-Reply-To: <60553E926D9B0C4F803B070D24FC3C0D0A032657A5@codvexdsa.codad.cityofdenton.com> References: <60553E926D9B0C4F803B070D24FC3C0D0A032657A5@codvexdsa.codad.cityofdenton.com> Message-ID: <20130502121229.GD7976@nat.myhome> > root at revproxy:~# varnishd -f /etc/varnish/dpdandmedia.vcl -s malloc,1G -T 172.16.64.52:2000 -a :80 > Message from VCC-compiler: > Regexp compilation error: > > nothing to repeat > > ('input' Line 38 Pos 24) > if (bereq.url ~ "\.swf($|?)") { > > -----------------------############---- > > Running VCC-compiler failed, exit 1 > > VCL compilation failed > ================== \.swf($|\?)? Match .swf optionally follow'd by a '$' or a '?' is the above regex. ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From r at roze.lv Thu May 2 12:12:46 2013 From: r at roze.lv (Reinis Rozitis) Date: Thu, 2 May 2013 15:12:46 +0300 Subject: varnish ssl In-Reply-To: References: Message-ID: <60D4EA92FD964AC79C868721D7179D1D@MasterPC> > Public => varnish(x2) => loadbalancer => Web servers (x4) > > We have around 15 domains with ssl support on login/payment pages. Varnish doesn't support ssl / you have to put something before it like nginx / stud / haproxy etc. rr From pprocacci at datapipe.com Thu May 2 12:14:43 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Thu, 2 May 2013 07:14:43 -0500 Subject: 503 Error In-Reply-To: References: Message-ID: <20130502121443.GE7976@nat.myhome> On Mon, Mar 04, 2013 at 07:52:06PM +0100, Michael Finke wrote: > Dear Varnish Community, > > I get an 503 error > Service Unavailable > > Guru Meditation: > XID: 1059580559 > > > -------------------------------------------------------------------------------- > > Varnish cache server > > > > everytime i want to import products in my magentosystem, and even if i delete products. > > Can you tell me how to deal with this problem? Is it possible to deactivate the varnish cache server? > > You can disable varnish by effectively loading a different vcl that simply pipe's requests. You can do this on the fly by using either the management console or varnishadm. ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From pprocacci at datapipe.com Thu May 2 12:18:08 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Thu, 2 May 2013 07:18:08 -0500 Subject: set req.http.Connection varnish In-Reply-To: <001501cddc9a$f57963a0$e06c2ae0$@ainfach.de> References: <001501cddc9a$f57963a0$e06c2ae0$@ainfach.de> Message-ID: <20130502121808.GF7976@nat.myhome> On Mon, Dec 17, 2012 at 10:10:35PM +0100, Tim Glabisch wrote: > Hello, > > > > i try to set the connection setting (having some trouble with cutted content > using iis) to disable keep-alive. > > my main Problem is that varnish doesnt allow me to overwrite the Connection > Header. > > > > set req.http.Connection = "foo"; // nothing changed > > set req.http.Connection2 = "foo"; // works great > > > > everytime i try to set the req.http.Connection, varnish (3.0.0 / 3.0.4) > just ignores it. > Set the connection to close in the response. set beresp.http.connection = "close". You don't want to set it in the request because varnish keeps connections alive for itself. ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From r at roze.lv Thu May 2 12:18:33 2013 From: r at roze.lv (Reinis Rozitis) Date: Thu, 2 May 2013 15:18:33 +0300 Subject: Problem in varnish In-Reply-To: References: Message-ID: > if(req.request=="PURGE") > > here what is PURGE and Where to set PURGE variable . please help me out. PURGE is a http request type (like GET or POST). It?s typically used for cache invalidation https://www.varnish-software.com/static/book/Cache_invalidation.html A simple way to craft a PURGE request is by using curl for example: curl -I -X PURGE http://yoursite.org/yourfile rr From chris.magee at gametheworld.com Thu May 2 12:19:19 2013 From: chris.magee at gametheworld.com (Chris Magee) Date: Thu, 2 May 2013 13:19:19 +0100 Subject: varnish ssl In-Reply-To: <60D4EA92FD964AC79C868721D7179D1D@MasterPC> References: <60D4EA92FD964AC79C868721D7179D1D@MasterPC> Message-ID: If you are using Amazon AWS, their Elastic Load Balancer ( http://aws.amazon.com/elasticloadbalancing/) can handle your HTTPS traffic and then pass it onto your servers as HTTP traffic. It costs money (currently $0.025 / hour / balancer and about $0.008 per GB), but is super simple and quick to setup. Chris. On Thu, May 2, 2013 at 1:12 PM, Reinis Rozitis wrote: > Public => varnish(x2) => loadbalancer => Web servers (x4) >> >> We have around 15 domains with ssl support on login/payment pages. >> > > Varnish doesn't support ssl / you have to put something before it like > nginx / stud / haproxy etc. > > rr > > ______________________________**_________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/**lists/mailman/listinfo/**varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From graham.lyons at bbc.co.uk Thu May 2 12:33:28 2013 From: graham.lyons at bbc.co.uk (Graham Lyons) Date: Thu, 2 May 2013 12:33:28 +0000 Subject: varnish-misc Digest, Vol 86, Issue 1 In-Reply-To: Message-ID: That's great, thanks to Jonathan and everyone else who replied. Some interesting stuff in there. On 01/05/2013 11:00, "varnish-misc-request at varnish-cache.org" wrote: >Send varnish-misc mailing list submissions to > varnish-misc at varnish-cache.org > >To subscribe or unsubscribe via the World Wide Web, visit > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >or, via email, send a message with subject or body 'help' to > varnish-misc-request at varnish-cache.org > >You can reach the person managing the list at > varnish-misc-owner at varnish-cache.org > >When replying, please edit your Subject line so it is more specific >than "Re: Contents of varnish-misc digest..." > > >Today's Topics: > > 1. RE: Varnishlog and Splunk (jonathan.huot at thomsonreuters.com) > > >---------------------------------------------------------------------- > >Message: 1 >Date: Tue, 30 Apr 2013 13:29:30 +0000 >From: >To: >Subject: RE: Varnishlog and Splunk >Message-ID: > <8E656B642592B942AE317E2AFAE0ABA13F9BE324 at UK2P-ERFMMBX10.ERF.thomson.com> > >Content-Type: text/plain; charset="utf-8" > >Hi Graham, > >Splunk didn?t care with separate lines or not, it?s all about regexp. You >can setup your Splunk events by adding any separator you want. It can be >a line feeds or any separator (ReqStart/ReqEnd) >Currently, we?re fetching records (about 10 lines for each record) using >Splunk without any issues. > >However, I will suggest you to use varnishncsa instead of varnishlog >because the main purpose of ncsa is to write one line for each requests. >You can setup the ?-F ? option to add more HTTP headers if needed. > > >Jonathan Huot >Phone: +33(0)1.47.62.78.65 > >From: varnish-misc-bounces at varnish-cache.org >[mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Graham Lyons >Sent: jeudi 25 avril 2013 12:16 >To: varnish-misc at varnish-cache.org >Subject: Varnishlog and Splunk > >Hello, > >Has anyone had any experience of putting output from varnishlog into >Splunk? My experience of Splunk so far has involved access log type >sources with events on separate lines, which is obviously quite different >to what comes out of varnishlog. > >If there's any prior art it would interesting to hear. > >Thanks, >Graham. > > > >---------------------------- > >http://www.bbc.co.uk >This e-mail (and any attachments) is confidential and may contain >personal views which are not the views of the BBC unless specifically >stated. >If you have received it in error, please delete it from your system. >Do not use, copy or disclose the information in any way nor act in >reliance on it and notify the sender immediately. >Please note that the BBC monitors e-mails sent or received. >Further communication will signify your consent to this. > >--------------------- > >This email was sent to you by Thomson Reuters, the global news and >information company. Any views expressed in this message are those of the >individual sender, except where the sender specifically states them to be >the views of Thomson Reuters. >-------------- next part -------------- >An HTML attachment was scrubbed... >URL: >130430/f029e1dd/attachment-0001.html> > >------------------------------ > >_______________________________________________ >varnish-misc mailing list >varnish-misc at varnish-cache.org >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > >End of varnish-misc Digest, Vol 86, Issue 1 >******************************************* ----------------------------- http://www.bbc.co.uk This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this. ----------------------------- From pprocacci at datapipe.com Thu May 2 13:20:25 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Thu, 2 May 2013 08:20:25 -0500 Subject: Problem in varnish In-Reply-To: References: Message-ID: <20130502132025.GG7976@nat.myhome> You make a PURGE request much like you make a GET request. Instead of sending 'GET /url HTTP/1.0' for example, you'd send 'PURGE /url HTTP/1.0'. ~Paul On Mon, Feb 25, 2013 at 09:57:30PM +0530, rahul jain wrote: > Hi, > > > i am getting problem how to send PURGE request Header from HTTP and how to > matche > > if(req.request=="PURGE") > > here what is PURGE and Where to set PURGE variable . please help me out. > > Thanks & regards > Rahul JAin > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From Raul.Rangel at disney.com Thu May 2 14:19:09 2013 From: Raul.Rangel at disney.com (Rangel, Raul) Date: Thu, 2 May 2013 07:19:09 -0700 Subject: add req.http.x-forwarded-for header In-Reply-To: References: Message-ID: <2465AAEEC8B8A242B26ED5F44BCA805F2616319CB6@SM-CALA-VXMB04A.swna.wdpr.disney.com> The default vcl_recv will handle setting the X-Forwarded-For header. You can see the default here: https://www.varnish-cache.org/trac/browser/bin/varnishd/default.vcl?rev=3.0 Do you have a return(XXX); in your vcl_recv? If you do then it will never execute the default vcl_recv which sets the X-Forwarded-For. Your other option is to just copy/paste the X-Forwarded-For code from the default.vcl into your vcl_recv. Raul -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Andreas G?tzfried Sent: Thursday, January 31, 2013 7:02 AM To: varnish-misc at varnish-cache.org Subject: add req.http.x-forwarded-for header Hello, i use varnish (3.0.2) and nginx (1.2.1), and i have a special setup: http: varnish (listens on *.80) -> nginx-backend (127.0.0.1:81) https: nginx (public ip:443) -> proxy_pass to same varnish instance -> nginx-backend (127.0.0.1:81) When varnish receives the requests proxied by nginx (https), varnish sees 127.0.0.1 as source, and there seems to be no solution getting varnish see the real ip. But, as you might guess, i want the public ip (need it for performance reasons) of the user. For https i found a solution. nginx is able to pass the real ip in a header: proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; By customizing the varnishncsa log format i'm able to see the users' ip address passed by nginx. So far, so good. But i'm not able to set this header when nginx isn't involved. I tried this: At the beginning of 'sub vcl_recv': if (!req.http.X-Forwarded-For) { set req.http.X-Forwarded-For = client.ip; } My intention was: if this header isn't set (and it shouldn't when varnish is accessed directly via https), set it with the value of the client ip. I've tried a couple of variations, but in the end the value in the varnishncsa log is always empty. Well... what am i doing wrong? Where's the error? thx Andreas _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From pprocacci at datapipe.com Thu May 2 14:24:40 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Thu, 2 May 2013 09:24:40 -0500 Subject: url rewrite - varnish In-Reply-To: <1360060143.25270.YahooMailNeo@web171505.mail.ir2.yahoo.com> References: <1360060143.25270.YahooMailNeo@web171505.mail.ir2.yahoo.com> Message-ID: <20130502142440.GH7976@nat.myhome> On Tue, Feb 05, 2013 at 10:29:03AM +0000, nic?e lenny bessay wrote: > Hi Sir, > > I'm having problem rewriting the request from varnish to query the backend. > > Here is what varnish receive ==> > > > http://cmsfoqua:8180/vgnExtTemplating/stresource?SecurityKey=lVPNsLcf&SiteName=axabanque&ServiceName=DetailLAB&Language=fr&ResourceName=Entete&TTL=3600&CIBLAGE= > > And I would like varnish to rewrite the url like this (http://cmsfoqua/xml/DetailLABEntete.xml) , then call the backend. > > Here is my rule in vcl_recv subroutine: > > set req.url = regsub(req.url, "^/vgnExtTemplating/stresource.([&?]SecurityKey=([a-zA-Z0-9])|[&?]SiteName=([a-zA-Z])|[&?]ServiceName=([a-zA-Z])|[&?]Language=([a-zA-Z])|[&?]ResourceName=([a-zA-Z])|[&?]TTL=([0-9])|[&?]CIBLAGE=([a-zA-Z]))*", "/xml/\1\2\4.xml"); > > Here is actually what the backend get : /xml/.xml > > Please, Help!!!! > I'm not entirely sure if you can chain regsub's together as I've never done so (you probably can), but the following (or similar) can pull the values you need from the url. if(req.url ~ "^/vgnExtTemplating/stresource\?"){ set req.url = "/xml/" + regsub(req.url, ".*SecurityKey=([^&]+).*", "\1") + regsub(req.url, ".*SiteName=([^&]+).*", "\1") + regsub(req.url, ".*Language=([^&]+).*", "\1") + ".xml"; } If that doesn't work, you can do something similar to the following: if(req.url ~ "^/vgnExtTemplating/stresource\?"){ set req.http.X-SecurityKey = regsub(req.url, ".*SecurityKey=([^&]+).*", "\1"); set req.http.X-SiteName = regsub(req.url, ".*SiteName=([^&]+).*", "\1"); set req.http.X-Language = regsub(req.url, ".*Language=([^&]+).*", "\1"); set req.url = "/xml/" + req.http.X-SecurityKey + req.http.X-SiteName + req.http.X-Language + ".xml"; remove req.http.X-SecurityKey; remove req.http.X-SiteName; remove req.http.X-Language } Once variable negative lookbehind assertions are implemented in PCRE, you'd be able to accomplish this with a one-liner. Hope that helps. ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From Raul.Rangel at disney.com Thu May 2 14:30:30 2013 From: Raul.Rangel at disney.com (Rangel, Raul) Date: Thu, 2 May 2013 07:30:30 -0700 Subject: Error 413 Request Entity Too Large In-Reply-To: References: Message-ID: <2465AAEEC8B8A242B26ED5F44BCA805F2616319CBF@SM-CALA-VXMB04A.swna.wdpr.disney.com> This usually happens because your Cookie header has gotten out of hand. You can configure varnishd and tell it to allow a larger header: http://manpages.ubuntu.com/manpages/precise/man1/varnishd.1.html look at http_req_hdr_len. Raul From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Piper Sponaas Sent: Sunday, February 03, 2013 2:18 PM To: varnish-misc at varnish-cache.org Subject: Re: Error 413 Request Entity Too Large How do I get rid of it. Its ruining my life!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -------------- next part -------------- An HTML attachment was scrubbed... URL: From Raul.Rangel at disney.com Thu May 2 14:31:52 2013 From: Raul.Rangel at disney.com (Rangel, Raul) Date: Thu, 2 May 2013 07:31:52 -0700 Subject: varnish 3 - thread_pools In-Reply-To: <5137b3ca67d00_71b77e144784b6@a4-winter14.tmail> References: <5137b3ca67d00_71b77e144784b6@a4-winter14.tmail> Message-ID: <2465AAEEC8B8A242B26ED5F44BCA805F2616319CC3@SM-CALA-VXMB04A.swna.wdpr.disney.com> I think someone recently commented that 2 thread pools is the ideal value. Anymore and you don?t see a performance increase. Raul From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of lnsano at bol.com.br Sent: Wednesday, March 06, 2013 2:23 PM To: varnish-misc at varnish-cache.org Subject: varnish 3 - thread_pools Hi, By the book, the thread_pools should be 2. However the server is multi-processed(8 cpus) and can handle more thread_pools. This server handles more than 5000 requests/sec. Is it a good ideia increasing the thread_pools to 8 and decreasing the thread_pool_max? regards, Leandro Sano -------------- next part -------------- An HTML attachment was scrubbed... URL: From jefe78 at gmail.com Thu May 2 14:59:40 2013 From: jefe78 at gmail.com (Jeffrey Taylor) Date: Thu, 2 May 2013 10:59:40 -0400 Subject: Configuration of varnish for HTTP streaming of Audio In-Reply-To: References: Message-ID: Did you read the Varnish book at all? It's a very simple concept. Cache the extensions you use. Flush when necessary. Also, your confidentiality warning is a joke, non-binding, and a waste of space. Lovingly, Jeff On Wed, Dec 19, 2012 at 2:27 AM, wrote: > Dear Team,**** > > ** ** > > Please provide some help or document regarding configuration of varnish > for http streaming of audio.**** > > ** ** > > *Warm Regards,* > > *Anish Kumar * > Phone : 022-447-73233** > > ** ** > > > "*Confidentiality Warning*: This message and any attachments are intended > only for the use of the intended recipient(s), are confidential and may be > privileged. If you are not the intended recipient, you are hereby notified > that any review, re-transmission, conversion to hard copy, copying, > circulation or other use of this message and any attachments is strictly > prohibited. If you are not the intended recipient, please notify the sender > immediately by return email and delete this message and any attachments > from your system. > > *Virus Warning:* Although the company has taken reasonable precautions to > ensure no viruses are present in this email. The company cannot accept > responsibility for any loss or damage arising from the use of this email or > attachment." > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From smwood4 at gmail.com Thu May 2 17:16:38 2013 From: smwood4 at gmail.com (Stephen Wood) Date: Thu, 2 May 2013 10:16:38 -0700 Subject: How does varnish cache determine if a resource needs to be refreshed In-Reply-To: References: Message-ID: Generally, you set the TTL on objects based on how frequently they update. You can also manually send a purge or force refresh objects if you know they've been updated. If your application includes a mixture of static and dynamic content, you can read up on ESI . On Mon, Jan 14, 2013 at 7:52 AM, Anton Soradoi wrote: > Hi, > > I've found a link to the site on HN and got very curious in what you have > to offer. > > My only concern is how does Varnish deal with frequent resource updates? > (The company I work for has a website that it is being actively developed, > with it's resources being updated multiple times a week. I.e js files, css > files, images often need refreshing) > > Disclaimer, I have not read up too much about Varnish, so a quick answer > would suffice. > > Thanks! > Anton > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From smwood4 at gmail.com Thu May 2 17:27:00 2013 From: smwood4 at gmail.com (Stephen Wood) Date: Thu, 2 May 2013 10:27:00 -0700 Subject: 503 for CSV File download In-Reply-To: <739CCB2D4BF8FD48985D6E158AD327B83A5ED09010@DC01.tecracerde.local> References: <739CCB2D4BF8FD48985D6E158AD327B83A5ED09010@DC01.tecracerde.local> Message-ID: Can you post your varnish.vcl for us to examine? On Fri, Jan 11, 2013 at 5:30 AM, Alexander Diedler wrote: > Hello, > We have a Typo3 Installation fronted with Varnish 3.0.2 out of the Ubuntu > Packages. If we try to load a CSV File, there ist an error: > Error 503 Service Unavailable > > Service Unavailable > Guru Meditation: > > XID: 1113729206 > > Varnish cache server > > The script create a statistc file, that will be placed in typo3temp. > The header of the file for transfer to the client is : > Pragma: public > Cache-Control: must-revalidate, post-check=0, pre-check=0 > Expires: 0 > Content-Disposition: attachment; filename= > res_GewinnspielDaten_090113-1113.csv > Content-type: x-application/octet-stream > Content-Transfer-Encoding: binary > Content-length:... > > Can anybody help us with this issue? > > Best regards > AlexD1979 > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From smwood4 at gmail.com Thu May 2 17:42:50 2013 From: smwood4 at gmail.com (Stephen Wood) Date: Thu, 2 May 2013 10:42:50 -0700 Subject: Varnish 2.1 - Error 503 during Logrotate In-Reply-To: <01d601ce079b$f3218a70$d9649f50$@crealya.com> References: <01d601ce079b$f3218a70$d9649f50$@crealya.com> Message-ID: Varnish may be marking your backend as "unhealthy" because you're failing too many health probes during this phase. When Varnish does backend health polling, it opens a *new* tcp connection to the backend with a preconfigured request. I would increase the timeout, window, and/or threshold to see if you can rule this out. You can read more about backend polling here . On Sun, Feb 10, 2013 at 6:35 AM, Tristan CHARBONNIER @ Crealya < charbonnier at crealya.com> wrote: > Hello,**** > > ** ** > > First, I want to thanks everyone who is contributing to this great piece > of software! Indeed, it?s working very well and it?s very efficient.**** > > ** ** > > I just encounter a small problem : I use varnish/squeeze v2.1.3-8 on a > Debian Linux 6.0 web server with the kernel v2.6.32-5-amd64.**** > > I use Virtualmin to manage this webserver and I set up Varnish on top of > apache2/squeeze v2.2.16-6+squeeze10.**** > > ** ** > > During the logrotate process, Varnish gives 503 errors. I traced the > problem to the multiple apache2 graceful restart that are done during this > process.**** > > Indeed, I host plenty of websites (more than 100) on my web server and, > for each one, I have a block like this in my ?/etc/logrotate.conf? file:** > ** > > ** ** > > /var/log/virtualmin/mywebsite.fr_access_log > /var/log/virtualmin/mywebsite.fr_error_log {**** > > rotate 5**** > > weekly**** > > compress**** > > postrotate**** > > /usr/sbin/apache2ctl graceful ; sleep 5**** > > endscript**** > > }**** > > ** ** > > So, during the logrotate process (which lasts for 20 minutes), there is > 100+ graceful restart of apache. ?graceful restart? are not supposed to > terminate existing connections so I don?t understand why Varnish fails to > work: probably a mistake on my side!**** > > ** ** > > In my VCL file, I just have ?backend apache { .host = "88.190.19.00"; > .port = "8080"; }? block along with the usual ones (acl purge, vcl_fetch, > ?). I use the default Varnish configuration file, except that I raised the > memory to 4G and use ram to store the cache.**** > > ** ** > > Any clue on how I can solve this small problem?**** > > ** ** > > Thanks in advance,**** > > ** ** > > Tristan CHARBONNIER**** > > ** ** > > ** ** > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From themadindian at yahoo.com Thu May 2 20:49:08 2013 From: themadindian at yahoo.com (Keith Grefski) Date: Thu, 02 May 2013 16:49:08 -0400 Subject: varnish ssl Message-ID: <9mr46ktc995aiau0ewtnuda3.1367527748486@email.android.com> Use nginx to terminate your SSL Ashish wrote: >I am setting up varnish as caching+entry point for public traffic. > >Public => varnish(x2) => loadbalancer => Web servers (x4) > >We have around 15 domains with ssl support on login/payment pages. > >I am not quite getting done here. > >1) i could point all domins to varnish IP and it could route >accordingly, but dont think i can make ssl workout to be sent st. to >loadbalancer and then webserver > question: Does ssl request gets untouched and sent directly to end >server? >2) Can i somehow configure varnish to be stand alone, but point dns to >loadbalancer IP's and somehow still manage to get varnish serve cached >objects? > >Please guide me > >_______________________________________________ >varnish-misc mailing list >varnish-misc at varnish-cache.org >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From perbu at varnish-software.com Thu May 2 21:02:22 2013 From: perbu at varnish-software.com (Per Buer) Date: Thu, 2 May 2013 23:02:22 +0200 Subject: varnish 3 - thread_pools In-Reply-To: <2465AAEEC8B8A242B26ED5F44BCA805F2616319CC3@SM-CALA-VXMB04A.swna.wdpr.disney.com> References: <5137b3ca67d00_71b77e144784b6@a4-winter14.tmail> <2465AAEEC8B8A242B26ED5F44BCA805F2616319CC3@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: That is correct. We haven't been able to see any increase in performance when the number of pools is increased above 2. On Thu, May 2, 2013 at 4:31 PM, Rangel, Raul wrote: > I think someone recently commented that 2 thread pools is the ideal value. > Anymore and you don?t see a performance increase.**** > > ** ** > > Raul**** > > ** ** > > *From:* varnish-misc-bounces at varnish-cache.org [mailto: > varnish-misc-bounces at varnish-cache.org] *On Behalf Of *lnsano at bol.com.br > *Sent:* Wednesday, March 06, 2013 2:23 PM > *To:* varnish-misc at varnish-cache.org > *Subject:* varnish 3 - thread_pools**** > > ** ** > > Hi,**** > > **** > > By the book, the thread_pools should be 2. However the server is > multi-processed(8 cpus) and can handle more thread_pools.**** > > This server handles more than 5000 requests/sec. **** > > **** > > Is it a good ideia increasing the thread_pools to 8 and decreasing the > thread_pool_max?**** > > **** > > regards,**** > > **** > > Leandro Sano**** > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Europe Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From smwood4 at gmail.com Thu May 2 22:34:44 2013 From: smwood4 at gmail.com (Stephen Wood) Date: Thu, 2 May 2013 15:34:44 -0700 Subject: varnish 3 - thread_pools In-Reply-To: References: <5137b3ca67d00_71b77e144784b6@a4-winter14.tmail> <2465AAEEC8B8A242B26ED5F44BCA805F2616319CC3@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: Are there any exceptions to the "2 is best" rule for this? I'm asking out of curiosity. It seems strange to me that Varnish would even provide the option to change it if any other value is detrimental to performance. On Thu, May 2, 2013 at 2:02 PM, Per Buer wrote: > That is correct. We haven't been able to see any increase in performance > when the number of pools is increased above 2. > > > On Thu, May 2, 2013 at 4:31 PM, Rangel, Raul wrote: > >> I think someone recently commented that 2 thread pools is the ideal >> value. Anymore and you don?t see a performance increase.**** >> >> ** ** >> >> Raul**** >> >> ** ** >> >> *From:* varnish-misc-bounces at varnish-cache.org [mailto: >> varnish-misc-bounces at varnish-cache.org] *On Behalf Of *lnsano at bol.com.br >> *Sent:* Wednesday, March 06, 2013 2:23 PM >> *To:* varnish-misc at varnish-cache.org >> *Subject:* varnish 3 - thread_pools**** >> >> ** ** >> >> Hi,**** >> >> **** >> >> By the book, the thread_pools should be 2. However the server is >> multi-processed(8 cpus) and can handle more thread_pools.**** >> >> This server handles more than 5000 requests/sec. **** >> >> **** >> >> Is it a good ideia increasing the thread_pools to 8 and decreasing the >> thread_pool_max?**** >> >> **** >> >> regards,**** >> >> **** >> >> Leandro Sano**** >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > > -- > *Per Buer* > CEO | Varnish Software AS > Phone: +47 958 39 117 | Skype: per.buer > We Make Websites Fly! > > Winner of the Red Herring Top 100 Europe Award 2013 > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at ibotty.net Fri May 3 06:31:04 2013 From: me at ibotty.net (me at ibotty.net) Date: Fri, 03 May 2013 08:31:04 +0200 Subject: varnish ssl In-Reply-To: References: Message-ID: <518359A8.1020107@ibotty.net> hi, i don't know whether i completely understand what you are trying to do. i understood, that your load balancer understands https traffic and passes cleartext traffic to the webservers. if that is right, instead of your proposed stack > Public => varnish(x2) => loadbalancer => Web servers (x4) you might do the stack (your notation) public => loadbalancer => varnish(x2) => web servers (x4) the load balancer won't do as much work, because varnish will load distribute as well. that might or might not work in your case (i'm pretty confident it in fact will work). good luck, tobias florek From aashisn at hotmail.com Fri May 3 06:55:11 2013 From: aashisn at hotmail.com (Ashish Nepal) Date: Fri, 3 May 2013 07:55:11 +0100 Subject: varnish ssl Message-ID: Hi thanks for your reply, I have started using nginx as SSL termination and now my infra is somewhat changed ... public will hit to DNS - pointed towards varnish+nginx(x2) - now varnish box is also doing loadbalancig - which will send traffic in round roubin fashion to 4 web boxes. So far everything works and quite reasonable amount of hits however I am getting some kind of inconsistency on varnish response : I randomly get 503 send back even when there is healthy web boxes and I have been following this list where I saw quite a few having similar issue ... I have changed timeout and checking out if that helps. Secondly can anyone guide me on nginx socket level security tips other than Conn limit. And if there is any additional security tips on varnish layer other than securityvcl? Thank you me at ibotty.net wrote: >hi, > >i don't know whether i completely understand what you are trying to do. > >i understood, that your load balancer understands https traffic and >passes cleartext traffic to the webservers. if that is right, instead of >your proposed stack > >> Public => varnish(x2) => loadbalancer => Web servers (x4) > >you might do the stack (your notation) > >public => loadbalancer => varnish(x2) => web servers (x4) > >the load balancer won't do as much work, because varnish will load >distribute as well. that might or might not work in your case (i'm >pretty confident it in fact will work). > >good luck, > tobias florek > > > >_______________________________________________ >varnish-misc mailing list >varnish-misc at varnish-cache.org >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > From perbu at varnish-software.com Fri May 3 08:18:02 2013 From: perbu at varnish-software.com (Per Buer) Date: Fri, 3 May 2013 10:18:02 +0200 Subject: varnish 3 - thread_pools In-Reply-To: References: <5137b3ca67d00_71b77e144784b6@a4-winter14.tmail> <2465AAEEC8B8A242B26ED5F44BCA805F2616319CC3@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: The option to change the number of thread pools might very well go away in the future. I guess PHK wasn't sure how much locking and CPU affinity would influence the performance of the pools. In order to know you have to implement it. A lot of the tunables are like that, values based on guesses, not necessary exact measurements. On Fri, May 3, 2013 at 12:34 AM, Stephen Wood wrote: > Are there any exceptions to the "2 is best" rule for this? I'm asking out > of curiosity. It seems strange to me that Varnish would even provide the > option to change it if any other value is detrimental to performance. > > > On Thu, May 2, 2013 at 2:02 PM, Per Buer wrote: > >> That is correct. We haven't been able to see any increase in performance >> when the number of pools is increased above 2. >> >> >> On Thu, May 2, 2013 at 4:31 PM, Rangel, Raul wrote: >> >>> I think someone recently commented that 2 thread pools is the ideal >>> value. Anymore and you don?t see a performance increase.**** >>> >>> ** ** >>> >>> Raul**** >>> >>> ** ** >>> >>> *From:* varnish-misc-bounces at varnish-cache.org [mailto: >>> varnish-misc-bounces at varnish-cache.org] *On Behalf Of *lnsano at bol.com.br >>> *Sent:* Wednesday, March 06, 2013 2:23 PM >>> *To:* varnish-misc at varnish-cache.org >>> *Subject:* varnish 3 - thread_pools**** >>> >>> ** ** >>> >>> Hi,**** >>> >>> **** >>> >>> By the book, the thread_pools should be 2. However the server is >>> multi-processed(8 cpus) and can handle more thread_pools.**** >>> >>> This server handles more than 5000 requests/sec. **** >>> >>> **** >>> >>> Is it a good ideia increasing the thread_pools to 8 and decreasing the >>> thread_pool_max?**** >>> >>> **** >>> >>> regards,**** >>> >>> **** >>> >>> Leandro Sano**** >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> >> >> -- >> *Per Buer* >> CEO | Varnish Software AS >> Phone: +47 958 39 117 | Skype: per.buer >> We Make Websites Fly! >> >> Winner of the Red Herring Top 100 Europe Award 2013 >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Europe Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at monkeysnatchbanana.com Fri May 3 19:15:41 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Fri, 3 May 2013 15:15:41 -0400 Subject: I wrote a post about one of the ways we are using varnish at TheLadders Message-ID: Hopefully, some find it helpful. I'm still fairly new to varnish. Improvements, suggestions etc welcomed. http://dev.theladders.com/2013/05/varnish-in-five-acts/ -- Ce n'est pas une signature -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Fri May 3 20:08:50 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Fri, 3 May 2013 22:08:50 +0200 Subject: I wrote a post about one of the ways we are using varnish at TheLadders In-Reply-To: References: Message-ID: Hi, This is very interesting and didactic, and I have a few questions : Why not simply trust the backend? if (beresp.http.x-invalidates && beresp.status >= 200 && beresp.status < 400) { ban("obj.http.x-url ~ " + beresp.http.x-invalidates); } I don't see any handling of the PATCH method in vcl_recv, check it doesn't end up being piped: if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } Best Regards, Dridi On Fri, May 3, 2013 at 9:15 PM, Sean Allen wrote: > Hopefully, some find it helpful. > > I'm still fairly new to varnish. Improvements, suggestions etc welcomed. > > http://dev.theladders.com/2013/05/varnish-in-five-acts/ > > -- > > Ce n'est pas une signature > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From sean at monkeysnatchbanana.com Fri May 3 20:12:28 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Fri, 3 May 2013 16:12:28 -0400 Subject: I wrote a post about one of the ways we are using varnish at TheLadders In-Reply-To: References: Message-ID: On Fri, May 3, 2013 at 4:08 PM, Dridi Boukelmoune < dridi.boukelmoune at zenika.com> wrote: > Hi, > > This is very interesting and didactic, and I have a few questions : > > Why not simply trust the backend? > if (beresp.http.x-invalidates && beresp.status >= 200 && beresp.status < > 400) { > ban("obj.http.x-url ~ " + beresp.http.x-invalidates); > } > > Completeness I suppose. I see your point. Something to think about. > I don't see any handling of the PATCH method in vcl_recv, check it doesn't > end up being piped: > if (req.request != "GET" && > req.request != "HEAD" && > req.request != "PUT" && > req.request != "POST" && > req.request != "TRACE" && > req.request != "OPTIONS" && > req.request != "DELETE") { > /* Non-RFC2616 or CONNECT which is weird. */ > return (pipe); > } > We don't have any clients that use PATCH. But that is a good point, I should add just to be safe. > > Best Regards, > Dridi > > > On Fri, May 3, 2013 at 9:15 PM, Sean Allen > wrote: > > Hopefully, some find it helpful. > > > > I'm still fairly new to varnish. Improvements, suggestions etc welcomed. > > > > http://dev.theladders.com/2013/05/varnish-in-five-acts/ > > > > -- > > > > Ce n'est pas une signature > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Ce n'est pas une signature -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Fri May 3 20:18:01 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Fri, 3 May 2013 22:18:01 +0200 Subject: I wrote a post about one of the ways we are using varnish at TheLadders In-Reply-To: References: Message-ID: One last question then, why do you expect PATCH requests ? (I know I'm curious =) On Fri, May 3, 2013 at 10:12 PM, Sean Allen wrote: > > > > On Fri, May 3, 2013 at 4:08 PM, Dridi Boukelmoune > wrote: >> >> Hi, >> >> This is very interesting and didactic, and I have a few questions : >> >> Why not simply trust the backend? >> if (beresp.http.x-invalidates && beresp.status >= 200 && beresp.status < >> 400) { >> ban("obj.http.x-url ~ " + beresp.http.x-invalidates); >> } >> > > Completeness I suppose. I see your point. > Something to think about. > >> >> I don't see any handling of the PATCH method in vcl_recv, check it doesn't >> end up being piped: >> if (req.request != "GET" && >> req.request != "HEAD" && >> req.request != "PUT" && >> req.request != "POST" && >> req.request != "TRACE" && >> req.request != "OPTIONS" && >> req.request != "DELETE") { >> /* Non-RFC2616 or CONNECT which is weird. */ >> return (pipe); >> } > > > We don't have any clients that use PATCH. But that is a good point, I should > add just to be safe. > >> >> >> Best Regards, >> Dridi >> >> >> On Fri, May 3, 2013 at 9:15 PM, Sean Allen >> wrote: >> > Hopefully, some find it helpful. >> > >> > I'm still fairly new to varnish. Improvements, suggestions etc welcomed. >> > >> > http://dev.theladders.com/2013/05/varnish-in-five-acts/ >> > >> > -- >> > >> > Ce n'est pas une signature >> > >> > _______________________________________________ >> > varnish-misc mailing list >> > varnish-misc at varnish-cache.org >> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > > -- > > Ce n'est pas une signature From sean at monkeysnatchbanana.com Fri May 3 20:27:38 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Fri, 3 May 2013 16:27:38 -0400 Subject: I wrote a post about one of the ways we are using varnish at TheLadders In-Reply-To: References: Message-ID: On Fri, May 3, 2013 at 4:18 PM, Dridi Boukelmoune < dridi.boukelmoune at zenika.com> wrote: > One last question then, why do you expect PATCH requests ? > > (I know I'm curious =) > > > On Fri, May 3, 2013 at 10:12 PM, Sean Allen > wrote: > > > > > > > > On Fri, May 3, 2013 at 4:08 PM, Dridi Boukelmoune > > wrote: > >> > >> Hi, > >> > >> This is very interesting and didactic, and I have a few questions : > >> > >> Why not simply trust the backend? > >> if (beresp.http.x-invalidates && beresp.status >= 200 && beresp.status < > >> 400) { > >> ban("obj.http.x-url ~ " + beresp.http.x-invalidates); > >> } > >> > > > > Completeness I suppose. I see your point. > > Something to think about. > > > >> > >> I don't see any handling of the PATCH method in vcl_recv, check it > doesn't > >> end up being piped: > >> if (req.request != "GET" && > >> req.request != "HEAD" && > >> req.request != "PUT" && > >> req.request != "POST" && > >> req.request != "TRACE" && > >> req.request != "OPTIONS" && > >> req.request != "DELETE") { > >> /* Non-RFC2616 or CONNECT which is weird. */ > >> return (pipe); > >> } > > > > > > We don't have any clients that use PATCH. But that is a good point, I > should > > add just to be safe. > Ah, patch in the normal default. I put it in ours because we plan on switching over to using it instead of our existing hacky 'partial update PUT' -- Ce n'est pas une signature -------------- next part -------------- An HTML attachment was scrubbed... URL: From lnsano at bol.com.br Fri May 3 20:33:36 2013 From: lnsano at bol.com.br (lnsano at bol.com.br) Date: Fri, 3 May 2013 17:33:36 -0300 Subject: varnish 3 - thread_pools In-Reply-To: References: <5137b3ca67d00_71b77e144784b6@a4-winter14.tmail> <2465AAEEC8B8A242B26ED5F44BCA805F2616319CC3@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: <51841f20e4f93_4f7f56a44746ef@a4-winter18.tmail> An HTML attachment was scrubbed... URL: From parsimail2001 at yahoo.se Sat May 4 19:59:29 2013 From: parsimail2001 at yahoo.se (Your Friend) Date: Sat, 4 May 2013 20:59:29 +0100 (BST) Subject: SV: varnish ssl In-Reply-To: References: Message-ID: <1367697569.58812.YahooMailNeo@web171901.mail.ir2.yahoo.com> Hi, Please correct if i'm wrong but I think that your ssl certificate is issued for a specific ip && domain. Pointing your domain to loadbalancer (new different ip) may cause problem for you and demand that you reissue your ssl certificate to make it work. Thanks, Ali ________________________________ Fr?n: Ashish Till: varnish-misc at varnish-cache.org Skickat: s?ndag, 14 april 2013 12:46 ?mne: varnish ssl I am setting up varnish as caching+entry point for public traffic. Public => varnish(x2) => loadbalancer => Web servers (x4) We have around 15 domains with ssl support on login/payment pages. I am not quite getting done here. 1) i could point all domins to varnish IP and it could route accordingly, but dont think i can make ssl workout to be sent st. to loadbalancer and then webserver ? ? question: Does ssl request gets untouched and sent directly to end server? 2) Can i somehow configure varnish to be stand alone, but point dns to loadbalancer IP's and somehow still manage to get varnish serve cached objects? Please guide me _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlevine at iwin.com Sat May 4 21:07:37 2013 From: jlevine at iwin.com (Joshua Levine) Date: Sat, 4 May 2013 14:07:37 -0700 Subject: varnish ssl In-Reply-To: <1367697569.58812.YahooMailNeo@web171901.mail.ir2.yahoo.com> References: <1367697569.58812.YahooMailNeo@web171901.mail.ir2.yahoo.com> Message-ID: In the previous proposal, you would have your DNS refer to the IP address of the virtual server on the load balancer, such that DNS reflects the proper hostname of the SSL certificate in question. Your Load Balancer would be configured with a Virtual Server that terminates SSL for you, and passes traffic to your backend varnish cluster, and varnish passes the traffic to your back end web servers. To take it a step further I might recommend: client -> DNS -> Public IP for the hostname on the Load Balancer (Virtual Server) -> Varnish Cluster -> An internal IP (RFC 1918) on the Load Balancer (Virtual Server) -> Web Server Cluster That will ensure: 1. Valid termination of your SSL traffic and none of the client errors you are concerned about. 2. n+1 management for your varnish cluster 3. n+1 management for your web server cluster Your IP will not need to change, you just want it move to the load balancer, and you can then use whatever you want (preferably internal IPs) for the rest of the hosts. Joshua On May 4, 2013, at 12:59 PM, Your Friend wrote: > Hi, > > Please correct if i'm wrong but I think that your ssl certificate is issued for a specific ip && domain. Pointing your domain to loadbalancer (new different ip) may cause problem for you and demand that you reissue your ssl certificate to make it work. > > Thanks, Ali > > Fr?n: Ashish > Till: varnish-misc at varnish-cache.org > Skickat: s?ndag, 14 april 2013 12:46 > ?mne: varnish ssl > > I am setting up varnish as caching+entry point for public traffic. > > Public => varnish(x2) => loadbalancer => Web servers (x4) > > We have around 15 domains with ssl support on login/payment pages. > > I am not quite getting done here. > > 1) i could point all domins to varnish IP and it could route > accordingly, but dont think i can make ssl workout to be sent st. to > loadbalancer and then webserver > question: Does ssl request gets untouched and sent directly to end > server? > 2) Can i somehow configure varnish to be stand alone, but point dns to > loadbalancer IP's and somehow still manage to get varnish serve cached > objects? > > Please guide me > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailman at haukebruno.de Tue May 7 12:25:16 2013 From: mailman at haukebruno.de (Hauke) Date: Tue, 07 May 2013 14:25:16 +0200 Subject: Keep URL after rewriting Message-ID: <2937497.rsAbnldLi0@notenbuch.haukebruno.de> Hi there, i think I've got a fallacy at the moment. After rewriting my URLs with Varnish I want to keep the origin URL given by a client. We want to use our domain with some followed seperators, e.g. www.example.com/corporate should point to a specific webapp, www.example.com/service should point to another webapp and so on. The backend servers listen to some internal names like corporate- stag.example.local. Here is my current Varnish config: backend web1 { .host="10.7.138.21"; .probe = { .url = "/"; .interval = 5s; .timeout = 1 s; .window = 5; .threshold = 3; } } backend web2 { .host="10.7.136.61"; .probe = { .url = "/"; .interval = 5s; .timeout = 1 s; .window = 5; .threshold = 3; } } director example round-robin { { .backend = web1; } { .backend = web2; } } sub vcl_recv { set req.backend = example; set req.http.host = regsub(req.http.host, "^www\.", ""); if (req.http.host ~ "example.com" && req.url ~ "^/corporate") { set req.http.host = "corporate-stag.example.local"; set req.url = regsub(req.url, "corporate", ""); } if (req.http.host ~ "example.com" && req.url ~ "^/service") { set req.http.host = "service-stag.example.local"; set req.url = regsub(req.url, "service", ""); } } If I hit _www.example.com/corporate_ in my browser, I get the correct webapp. But if I navigate, the hyperlinks are all pointed to _corporate- stag.example.local_ and they never pass Varnish again. Where is my mistake? -- Best regards, Hauke From Raul.Rangel at disney.com Tue May 7 14:12:49 2013 From: Raul.Rangel at disney.com (Rangel, Raul) Date: Tue, 7 May 2013 07:12:49 -0700 Subject: Keep URL after rewriting In-Reply-To: <2937497.rsAbnldLi0@notenbuch.haukebruno.de> References: <2937497.rsAbnldLi0@notenbuch.haukebruno.de> Message-ID: <2465AAEEC8B8A242B26ED5F44BCA805F2616703927@SM-CALA-VXMB04A.swna.wdpr.disney.com> I'm going to make the assumption that your webapp is writing absolute urls. I'm also going to make the assumption that it is using the Host header to write those urls. This is why your gusts never navigate back to the same domain. I see two potential options. 1) Configure your webapp not to write full urls. 2) Some web frameworks support the X-Forwarded-Host header. So you could set that header before you do any manipulations to it. i.e) set req.http.X-Forwarded-Host = req.http.host; Raul -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Hauke Sent: Tuesday, May 07, 2013 6:25 AM To: varnish-misc at varnish-cache.org Subject: Keep URL after rewriting Hi there, i think I've got a fallacy at the moment. After rewriting my URLs with Varnish I want to keep the origin URL given by a client. We want to use our domain with some followed seperators, e.g. www.example.com/corporate should point to a specific webapp, www.example.com/service should point to another webapp and so on. The backend servers listen to some internal names like corporate- stag.example.local. Here is my current Varnish config: backend web1 { .host="10.7.138.21"; .probe = { .url = "/"; .interval = 5s; .timeout = 1 s; .window = 5; .threshold = 3; } } backend web2 { .host="10.7.136.61"; .probe = { .url = "/"; .interval = 5s; .timeout = 1 s; .window = 5; .threshold = 3; } } director example round-robin { { .backend = web1; } { .backend = web2; } } sub vcl_recv { set req.backend = example; set req.http.host = regsub(req.http.host, "^www\.", ""); if (req.http.host ~ "example.com" && req.url ~ "^/corporate") { set req.http.host = "corporate-stag.example.local"; set req.url = regsub(req.url, "corporate", ""); } if (req.http.host ~ "example.com" && req.url ~ "^/service") { set req.http.host = "service-stag.example.local"; set req.url = regsub(req.url, "service", ""); } } If I hit _www.example.com/corporate_ in my browser, I get the correct webapp. But if I navigate, the hyperlinks are all pointed to _corporate- stag.example.local_ and they never pass Varnish again. Where is my mistake? -- Best regards, Hauke _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From smwood4 at gmail.com Wed May 8 00:12:12 2013 From: smwood4 at gmail.com (Stephen Wood) Date: Tue, 7 May 2013 17:12:12 -0700 Subject: varnishncsa silently stops if %D or %T (fulfillment times) are logged Message-ID: I've been unable to get varnishncsa to log response times, either in microseconds or seconds (using the %D or %T flags). All of the other varnishncsa logging options are working fine. For example, I'm able to use %{Varnish:time_firstbyte}x. The varnishncsa process will run until the first request comes in, and then it will simply die. I don't see anything in syslog. Here's my logging options from /etc/init.d/varnishncsa "%{X-Forwarded-For}i %t '%r' %s %b '%D' '%{Varnish:time_firstbyte}x' '%{Varnish:handling}x' '%{X-Row-Count}o' '%{User-agent}i' '%{X-AccessID}o'" Any ideas on why I can't log these metrics? Stephen Wood -------------- next part -------------- An HTML attachment was scrubbed... URL: From itxx00 at gmail.com Wed May 8 04:44:48 2013 From: itxx00 at gmail.com (Gao Yongwei) Date: Wed, 8 May 2013 12:44:48 +0800 Subject: how to flush director dns ttl Message-ID: Hello,lists: I am using varnish in a CentOS6.4 box,bellow is version info: [root at cdn001 ~]# uname -r 2.6.32-358.2.1.el6.x86_64 [root at cdn001 ~]# rpm -q varnish varnish-3.0.3-3.el6.art.x86_64 and in my /etc/varnish/default.vcl I use ' dns director': director dnsdomain dns { .list = { .port = "80"; "10.0.0.0"/24; } .ttl = 12h; } notice that the ttl has been set to 12h because I want to reduse dns resolve time,there is a domain www.example.com which resolved to 10.0.0.2 ,after the dns changed www.example.com to 10.0.0.3, varnish cannot find the real backend as the ttl been set to 12h.so I must do a 'service varnish reload' to update the director. Is there any other way to flush the director ttl without a service reload ? thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pprocacci at datapipe.com Wed May 8 06:00:07 2013 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Wed, 8 May 2013 01:00:07 -0500 Subject: how to flush director dns ttl In-Reply-To: References: Message-ID: <20130508060007.GJ7976@nat.myhome> On Wed, May 08, 2013 at 12:44:48PM +0800, Gao Yongwei wrote: > Hello,lists: > I am using varnish in a CentOS6.4 box,bellow is version info: > > [root at cdn001 ~]# uname -r > 2.6.32-358.2.1.el6.x86_64 > [root at cdn001 ~]# rpm -q varnish > varnish-3.0.3-3.el6.art.x86_64 > > and in my /etc/varnish/default.vcl > I use ' dns director': > > director dnsdomain dns { > .list = { > .port = "80"; > "10.0.0.0"/24; > } > .ttl = 12h; > } > notice that the ttl has been set to 12h because I want to reduse dns > resolve time,there is a domain www.example.com which resolved to 10.0.0.2 > ,after the dns changed www.example.com to 10.0.0.3, varnish cannot find the > real backend as the ttl been set to 12h.so I must do a 'service varnish > reload' to update the director. > > Is there any other way to flush the director ttl without a service reload > ? thanks. I believe the answer is no. Varnish checks the hostname at compile time, and doesn't recheck it until the ttl expires. ~Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From itxx00 at gmail.com Wed May 8 07:41:54 2013 From: itxx00 at gmail.com (Gao Yongwei) Date: Wed, 8 May 2013 15:41:54 +0800 Subject: how to flush director dns ttl In-Reply-To: <20130508060007.GJ7976@nat.myhome> References: <20130508060007.GJ7976@nat.myhome> Message-ID: 2013/5/8 Paul A. Procacci > On Wed, May 08, 2013 at 12:44:48PM +0800, Gao Yongwei wrote: > > Hello,lists: > > I am using varnish in a CentOS6.4 box,bellow is version info: > > > > [root at cdn001 ~]# uname -r > > 2.6.32-358.2.1.el6.x86_64 > > [root at cdn001 ~]# rpm -q varnish > > varnish-3.0.3-3.el6.art.x86_64 > > > > and in my /etc/varnish/default.vcl > > I use ' dns director': > > > > director dnsdomain dns { > > .list = { > > .port = "80"; > > "10.0.0.0"/24; > > } > > .ttl = 12h; > > } > > notice that the ttl has been set to 12h because I want to reduse dns > > resolve time,there is a domain www.example.com which resolved to > 10.0.0.2 > > ,after the dns changed www.example.com to 10.0.0.3, varnish cannot find > the > > real backend as the ttl been set to 12h.so I must do a 'service varnish > > reload' to update the director. > > > > Is there any other way to flush the director ttl without a service > reload > > ? thanks. > > I believe the answer is no. Varnish checks the hostname at compile time, > and doesn't recheck it until the ttl expires. > > ~Paul > See,when the ttl expires, varnish will flush the hostname ,so ,is there anyway we can manually flush the ttl immediately ? thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From apj at mutt.dk Wed May 8 07:50:13 2013 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Wed, 8 May 2013 09:50:13 +0200 Subject: how to flush director dns ttl In-Reply-To: References: <20130508060007.GJ7976@nat.myhome> Message-ID: <20130508075013.GC19694@nerd.dk> On Wed, May 08, 2013 at 03:41:54PM +0800, Gao Yongwei wrote: > > > See,when the ttl expires, varnish will flush the hostname ,so ,is there No. It will never flush it. It is only resolved at VCL compile time. > anyway we can manually flush the ttl immediately ? Reload the VCL. (as you're already doing). -- Andreas From apj at mutt.dk Wed May 8 07:57:27 2013 From: apj at mutt.dk (Andreas Plesner Jacobsen) Date: Wed, 8 May 2013 09:57:27 +0200 Subject: how to flush director dns ttl In-Reply-To: <20130508075013.GC19694@nerd.dk> References: <20130508060007.GJ7976@nat.myhome> <20130508075013.GC19694@nerd.dk> Message-ID: <20130508075727.GD19694@nerd.dk> On Wed, May 08, 2013 at 09:50:13AM +0200, Andreas Plesner Jacobsen wrote: > > > > > See,when the ttl expires, varnish will flush the hostname ,so ,is there > > No. It will never flush it. It is only resolved at VCL compile time. Sorry, my bad, didn't realise this was the DNS director. You're right that it expires. > > anyway we can manually flush the ttl immediately ? > > Reload the VCL. (as you're already doing). However, I believe this is still the only option. Why is that a problem? -- Andreas From perbu at varnish-software.com Wed May 8 08:10:27 2013 From: perbu at varnish-software.com (Per Buer) Date: Wed, 8 May 2013 10:10:27 +0200 Subject: how to flush director dns ttl In-Reply-To: References: Message-ID: Hi Geo, On Wed, May 8, 2013 at 6:44 AM, Gao Yongwei wrote: > notice that the ttl has been set to 12h because I want to reduse dns > resolve time,there is a domain www.example.com which resolved to 10.0.0.2 > ,after the dns changed www.example.com to 10.0.0.3, varnish cannot find > the real backend as the ttl been set to 12h.so I must do a 'service > varnish reload' to update the director. > > Is there any other way to flush the director ttl without a service reload > ? thanks. > The whole point of the DNS director and it's TTL settings is to have Varnish automatically update the backend definitions based on TTL. So the only option you have is AFAIK: 1) To lower the TTL so it won't become a problem (speed up your DNS) 2) To modify the DNS director and implement flushing through VCL Per. -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Europe Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailman at haukebruno.de Wed May 8 08:43:01 2013 From: mailman at haukebruno.de (Hauke) Date: Wed, 08 May 2013 10:43:01 +0200 Subject: Keep URL after rewriting In-Reply-To: <2465AAEEC8B8A242B26ED5F44BCA805F2616703927@SM-CALA-VXMB04A.swna.wdpr.disney.com> References: <2937497.rsAbnldLi0@notenbuch.haukebruno.de> <2465AAEEC8B8A242B26ED5F44BCA805F2616703927@SM-CALA-VXMB04A.swna.wdpr.disney.com> Message-ID: <2637855.SErP7riHFn@notenbuch.haukebruno.de> Hi Raul, thanks for your advices. for wordpress I configured the Site-URL to www.example.com/corporate. That will fix my hyperlinks. But now I have the following problem: As shown in the config if (req.http.host ~ "example.com" && req.url ~ "^/corporate") { set req.http.host = "corporate-stag.example.local"; set req.url = regsub(req.url, "corporate", ""); } I remove the _corporate_ part in req.url. That's because the webapp lies directly under the DocumentRoot of _corporate-stag.example.local_. If I click some hyperlink in the webapp like _http://www.example.com/corporate/foo.php_ this does not work because of removing the _corporate_ from the req.url. Is there any other option for my case except of moving the whole webapp from DocumentRoot to DocumentRoot/corporate and don't delete the _corporate_ part in req.url? I tried this and this will work, but it's a little bit messy at all. Thanks, -- Best regards, Hauke On Tuesday, May 07, 2013 07:12:49 AM Rangel, Raul wrote: > I'm going to make the assumption that your webapp is writing absolute urls. > I'm also going to make the assumption that it is using the Host header to > write those urls. This is why your gusts never navigate back to the same > domain. I see two potential options. > > 1) Configure your webapp not to write full urls. > 2) Some web frameworks support the X-Forwarded-Host header. So you could set > that header before you do any manipulations to it. i.e) > set req.http.X-Forwarded-Host = req.http.host; > > Raul > > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Hauke Sent: > Tuesday, May 07, 2013 6:25 AM > To: varnish-misc at varnish-cache.org > Subject: Keep URL after rewriting > > Hi there, > > i think I've got a fallacy at the moment. > > After rewriting my URLs with Varnish I want to keep the origin URL given by > a client. > > We want to use our domain with some followed seperators, e.g. > www.example.com/corporate should point to a specific webapp, > www.example.com/service should point to another webapp and so on. > > The backend servers listen to some internal names like corporate- > stag.example.local. > > Here is my current Varnish config: > > backend web1 { > .host="10.7.138.21"; > .probe = { > .url = "/"; > .interval = 5s; > .timeout = 1 s; > .window = 5; > .threshold = 3; > } > } > > backend web2 { > .host="10.7.136.61"; > .probe = { > .url = "/"; > .interval = 5s; > .timeout = 1 s; > .window = 5; > .threshold = 3; > } > } > > director example round-robin { > { > .backend = web1; > } > { > .backend = web2; > } > } > > sub vcl_recv { > set req.backend = example; > set req.http.host = regsub(req.http.host, "^www\.", ""); > > if (req.http.host ~ "example.com" && req.url ~ "^/corporate") { > set req.http.host = "corporate-stag.example.local"; > set req.url = regsub(req.url, "corporate", ""); > } > > if (req.http.host ~ "example.com" && req.url ~ "^/service") { > set req.http.host = "service-stag.example.local"; > set req.url = regsub(req.url, "service", ""); > } > } > > If I hit _www.example.com/corporate_ in my browser, I get the correct > webapp. But if I navigate, the hyperlinks are all pointed to _corporate- > stag.example.local_ and they never pass Varnish again. > > Where is my mistake? > > -- > Best regards, > Hauke > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From itxx00 at gmail.com Wed May 8 08:58:09 2013 From: itxx00 at gmail.com (Gao Yongwei) Date: Wed, 8 May 2013 16:58:09 +0800 Subject: how to flush director dns ttl In-Reply-To: References: Message-ID: The whole point of the DNS director and it's TTL settings is to have > Varnish automatically update the backend definitions based on TTL. So the > only option you have is AFAIK: > 1) To lower the TTL so it won't become a problem (speed up your DNS) > 2) To modify the DNS director and implement flushing through VCL > Thanks,perbu ,I can achieve the first method , and could you please teach me more about the second method , how does this vcl look like ? many thanks , :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Wed May 8 09:06:40 2013 From: perbu at varnish-software.com (Per Buer) Date: Wed, 8 May 2013 11:06:40 +0200 Subject: how to flush director dns ttl In-Reply-To: References: Message-ID: Hi Gao, On Wed, May 8, 2013 at 10:58 AM, Gao Yongwei wrote: > > > The whole point of the DNS director and it's TTL settings is to have >> Varnish automatically update the backend definitions based on TTL. So the >> only option you have is AFAIK: >> 1) To lower the TTL so it won't become a problem (speed up your DNS) >> 2) To modify the DNS director and implement flushing through VCL >> > > Thanks,perbu ,I can achieve the first method , and could you please > teach me more about the second method , how does this vcl look like ? > many thanks , :-) > You would need to create a VMOD that kills off the cache in the DNS director. For this to work you would probably need to modify the DNS director in the Varnish Cache source. -- *Per Buer* CEO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Europe Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From itxx00 at gmail.com Wed May 8 09:14:47 2013 From: itxx00 at gmail.com (Gao Yongwei) Date: Wed, 8 May 2013 17:14:47 +0800 Subject: how to flush director dns ttl In-Reply-To: References: Message-ID: > > > You would need to create a VMOD that kills off the cache in the DNS > director. For this to work you would probably need to modify the DNS > director in the Varnish Cache source. > Soga, looks very hard for me now but it may be very interesting ,I'll have a try .Thanks again : ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at monkeysnatchbanana.com Wed May 8 09:22:48 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Wed, 8 May 2013 05:22:48 -0400 Subject: file storage engine overflow and 503 error Message-ID: Varnish 3 series. We recently switched one of our varnish caches from being having a malloc based storage to file based. When the cache overflowed using the malloc based one, an older item was nuked and the requested item was returned. When we overflowed the file storage engine cache yesterday, varnishstat showed the items being evicted but would return a 503 for the requested item and in varnishlog, was showing it couldn't allocate storage during the deliver phase for the requested item. What do I need to do to change to not have a cache overflow result in 503 errors? -- Ce n'est pas une signature -------------- next part -------------- An HTML attachment was scrubbed... URL: From linuxbsdfreak at gmail.com Wed May 8 14:05:28 2013 From: linuxbsdfreak at gmail.com (kevin fernandes) Date: Wed, 8 May 2013 16:05:28 +0200 Subject: Varnish URL rewrite having integers with scientific nos in URL Message-ID: Hi All, I would like to know how can i rewrite/convert a URL containing number is scientific form to normal integer. For eg: I have something like this coming in the URL http://domain.com/public/domain/app/1.024e+06/1028422/1028422_b24db5c8391ec5c5d93cd0ac083acbdf I would like to have to have the URL having a normal integer so the output should be http://domain.com/public/domain/app/1024000/1028422/1028422_b24db5c8391ec5c5d93cd0ac083acbdf Playing around something like this for testing if (req.http.Host ~ "domain.com") { set req.url = regsub(req.url, "^/public/domain/app/[+|-]?\d\.?\d{0,}[E|e|X|x](10)?[\^\*]?[+|-]?\d+/(.*)/(.*)$", "/public/domain/app/\d+/(.*)/(.*)$"); } Any ideas or any clues if the syntax for the matching scientific nos is correct? Regards, Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From Raul.Rangel at disney.com Wed May 8 15:57:58 2013 From: Raul.Rangel at disney.com (Rangel, Raul) Date: Wed, 8 May 2013 08:57:58 -0700 Subject: Keep URL after rewriting In-Reply-To: <2637855.SErP7riHFn@notenbuch.haukebruno.de> References: <2937497.rsAbnldLi0@notenbuch.haukebruno.de> <2465AAEEC8B8A242B26ED5F44BCA805F2616703927@SM-CALA-VXMB04A.swna.wdpr.disney.com> <2637855.SErP7riHFn@notenbuch.haukebruno.de> Message-ID: <2465AAEEC8B8A242B26ED5F44BCA805F2616A1FA00@SM-CALA-VXMB04A.swna.wdpr.disney.com> So you said you set the Site-URL to www.example.com/corporate which I'm assuming makes urls like http://www.example.com/corporate/foo.php. You also said that the DocumentRoot is /. So by removing the corporate you end up with //foo.php. So you are expecting to find foo.php in the DocumentRoot. It might be the double slash that is breaking it. But I think apache will handle that correctly. Change your regsub to replace /corporate and see if that helps. Otherwise use varnishlog to look at your logs. Raul -----Original Message----- From: Hauke [mailto:mailman at haukebruno.de] Sent: Wednesday, May 08, 2013 2:43 AM To: Rangel, Raul; 'varnish-misc at varnish-cache.org' Subject: Re: Keep URL after rewriting Hi Raul, thanks for your advices. for wordpress I configured the Site-URL to www.example.com/corporate. That will fix my hyperlinks. But now I have the following problem: As shown in the config if (req.http.host ~ "example.com" && req.url ~ "^/corporate") { set req.http.host = "corporate-stag.example.local"; set req.url = regsub(req.url, "corporate", ""); } I remove the _corporate_ part in req.url. That's because the webapp lies directly under the DocumentRoot of _corporate-stag.example.local_. If I click some hyperlink in the webapp like _http://www.example.com/corporate/foo.php_ this does not work because of removing the _corporate_ from the req.url. Is there any other option for my case except of moving the whole webapp from DocumentRoot to DocumentRoot/corporate and don't delete the _corporate_ part in req.url? I tried this and this will work, but it's a little bit messy at all. Thanks, -- Best regards, Hauke On Tuesday, May 07, 2013 07:12:49 AM Rangel, Raul wrote: > I'm going to make the assumption that your webapp is writing absolute urls. > I'm also going to make the assumption that it is using the Host header > to write those urls. This is why your gusts never navigate back to the > same domain. I see two potential options. > > 1) Configure your webapp not to write full urls. > 2) Some web frameworks support the X-Forwarded-Host header. So you > could set that header before you do any manipulations to it. i.e) set > req.http.X-Forwarded-Host = req.http.host; > > Raul > > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Hauke Sent: > Tuesday, May 07, 2013 6:25 AM > To: varnish-misc at varnish-cache.org > Subject: Keep URL after rewriting > > Hi there, > > i think I've got a fallacy at the moment. > > After rewriting my URLs with Varnish I want to keep the origin URL > given by a client. > > We want to use our domain with some followed seperators, e.g. > www.example.com/corporate should point to a specific webapp, > www.example.com/service should point to another webapp and so on. > > The backend servers listen to some internal names like corporate- > stag.example.local. > > Here is my current Varnish config: > > backend web1 { > .host="10.7.138.21"; > .probe = { > .url = "/"; > .interval = 5s; > .timeout = 1 s; > .window = 5; > .threshold = 3; > } > } > > backend web2 { > .host="10.7.136.61"; > .probe = { > .url = "/"; > .interval = 5s; > .timeout = 1 s; > .window = 5; > .threshold = 3; > } > } > > director example round-robin { > { > .backend = web1; > } > { > .backend = web2; > } > } > > sub vcl_recv { > set req.backend = example; > set req.http.host = regsub(req.http.host, "^www\.", ""); > > if (req.http.host ~ "example.com" && req.url ~ "^/corporate") { > set req.http.host = "corporate-stag.example.local"; > set req.url = regsub(req.url, "corporate", ""); > } > > if (req.http.host ~ "example.com" && req.url ~ "^/service") { > set req.http.host = "service-stag.example.local"; > set req.url = regsub(req.url, "service", ""); > } > } > > If I hit _www.example.com/corporate_ in my browser, I get the correct > webapp. But if I navigate, the hyperlinks are all pointed to > _corporate- stag.example.local_ and they never pass Varnish again. > > Where is my mistake? > > -- > Best regards, > Hauke > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From spam at onlineconcepts.com Thu May 9 20:47:34 2013 From: spam at onlineconcepts.com (John) Date: Thu, 9 May 2013 16:47:34 -0400 Subject: Does code belong in vcl_recv & vcl_fetch? Message-ID: Using Varnish 3.0.2, I would like to cache a specific URL path. When Varnish sends this path to the backend, Apache gathers data from external services like Twitter. Do I need to place this code in both the vcl_fetch and vcl_recv? I've currently got it in both places, but I do not want to duplicate code unless it's required (or a best practice). =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= # Respond to incoming requests. sub vcl_recv { [...] # Always cache the following URLs to prevent frequent requests to external # services. For example: Twitter at /ext/twittersearch # This code appears twice, once here and again in vcl_fetch so make sure you # edit both and keep them equal. if (req.url ~ "^/ext/") { unset req.http.Cookie; } [...] # Code determining what to do when serving items from the Apache servers. # beresp == Back-end response from the web server. sub vcl_fetch { [...] # Always cache the following URLs to prevent frequent requests to external # services. For example: Twitter at /ext/twittersearch # This code appears twice, once here and again in vcl_recv so make sure you # edit both and keep them equal. if (req.url ~ "^/ext/") { unset req.http.Cookie; } =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Rather than post my entire VCL file, I'll just say that I modeled it on this one. https://fourkitchens.atlassian.net/wiki/display/TECH/Configure+Varnish+3+for+Drupal+7 I've also found this article helpful. http://www.lullabot.com/blog/articles/configuring-varnish-high-availability-multiple-web-servers TIA, John From perbu at varnish-software.com Fri May 10 06:45:27 2013 From: perbu at varnish-software.com (Per Buer) Date: Fri, 10 May 2013 08:45:27 +0200 Subject: Does code belong in vcl_recv & vcl_fetch? In-Reply-To: References: Message-ID: On Thu, May 9, 2013 at 10:47 PM, John wrote: > > Do I need to place this code in both the vcl_fetch and vcl_recv? I've > currently got it in both places, but I do not want to duplicate code > unless it's required (or a best practice). > > You do two totally different things in recv and fetch. In recv you can modify the req object, changing the _request_. In fetch you modify beresp, the backed response. Cookie is a request header, so change it in recv. If you where to alter Set-Cookie OTOH, you would do so in fetch. I would recommend to spend some time on the Varnish Book og the tutorial "Using Varnish". Or, if you are in NYC on the User Group meetup around the end of May there will also be a training session. -- *Per Buer* ?Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at monkeysnatchbanana.com Fri May 10 13:20:58 2013 From: sean at monkeysnatchbanana.com (Sean Allen) Date: Fri, 10 May 2013 09:20:58 -0400 Subject: file storage engine overflow and 503 error In-Reply-To: References: Message-ID: For the edification of future parties, check out the nuke limit option. We were pulling back some objects that were larger than 50x average object size so nuking didn't create enough space. On Wed, May 8, 2013 at 5:22 AM, Sean Allen wrote: > Varnish 3 series. > > We recently switched one of our varnish caches from being having a malloc > based storage to file based. When the cache overflowed using the malloc > based one, an older item was nuked and the requested item was returned. > > When we overflowed the file storage engine cache yesterday, varnishstat > showed the items being evicted but would return a 503 for the requested > item and in varnishlog, was showing it couldn't allocate storage during the > deliver phase for the requested item. > > What do I need to do to change to not have a cache overflow result in 503 > errors? > > > -- > > Ce n'est pas une signature > -- Ce n'est pas une signature -------------- next part -------------- An HTML attachment was scrubbed... URL: From miguel_3_gonzalez at yahoo.es Fri May 10 17:52:21 2013 From: miguel_3_gonzalez at yahoo.es (Miguel Gonzalez) Date: Fri, 10 May 2013 18:52:21 +0100 (BST) Subject: Varnish and mod_pagespeed Message-ID: <1368208341.51251.YahooMailNeo@web171606.mail.ir2.yahoo.com> Dear all, ?? I have Varnish as a front-end and an Apache/PHP running a Symfony application as a backend. ?? I had mod_pagespeed enabled in Apache long time before I decided to put Varnish in front of it. It worked well. ?? Since we decided to put Varnish in front, we are getting many errors logged in error_log file of this kind: ?? Fri May 10 08:29:15 2013] [warn] [mod_pagespeed 1.3.25.4-2941 @17719] Fetch failed for http://www.midomain.com/images//image2.jpg.pagespeed.ic.t1diDHbOw9.jpg, status=404 ? and others like: ?? ?? [warn] [mod_pagespeed 1.1.23.2-2258 @28956] Fetch timed out: http://127.0.0.1:8000/media/cache/thumb_image_user6.jpg (35) waiting for 50 ms ?? the result is that the webpage is not loading as fast as it did without varnish. ?? I have googled around and I found webpages like this one: ?? http://serverfault.com/questions/290776/mod-pagespeed-varnish-and-apache-cache-issues-after-new-code-pushes ?? but since I'm quite new to Varnish I don't really know what I should do. ?? Regards, ?? Miguel here is my default.vcl backend default { ? .host = "127.0.0.1"; ? .port = "8000"; ? .connect_timeout = 600s; ? .first_byte_timeout = 600s; ? .between_bytes_timeout = 600s; } acl purge { ? "localhost"; } acl ban { ? "localhost"; } sub vcl_recv { ? // Strip cookies for static files: ? if (req.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm|xml)$") { ??? unset req.http.Cookie; ? } ? // Remove specific cookies: ? set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", ""); ? set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", ""); ? set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); ? if (req.http.cookie ~ "^ *$") { unset req.http.cookie; } ? if (req.http.Cookie ~ "^\s*$") { unset req.http.Cookie; } ? if (req.http.Cookie == "") { remove req.http.Cookie; } ? // Normalize Accept-Encoding header (straight from the manual: https://www.varnish-cache.org/docs/3.0/tutorial/vary.html) ? if (req.http.Accept-Encoding) { ??? if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { ????? // No point in compressing these ????? remove req.http.Accept-Encoding; ??? } elsif (req.http.Accept-Encoding ~ "gzip") { ????? set req.http.Accept-Encoding = "gzip"; ??? } elsif (req.http.Accept-Encoding ~ "deflate") { ????? set req.http.Accept-Encoding = "deflate"; ??? } else { ????? // unkown algorithm ????? remove req.http.Accept-Encoding; ??? } ? } ? if (req.restarts == 0) { ??? if (req.http.x-forwarded-for) { ????? set req.http.X-Forwarded-For = req.http.X-Forwarded-For; ??? } else { ????? set req.http.X-Forwarded-For = client.ip; ??? } ? } ? if (req.request != "GET" && ??? req.request != "HEAD" && ??? req.request != "PUT" && ??? req.request != "POST" && ??? req.request != "TRACE" && ??? req.request != "OPTIONS" && ??? req.request != "DELETE") { ????? /* Non-RFC2616 or CONNECT which is weird. */ ????? return (pipe); ? } ? if (req.request != "GET" && req.request != "HEAD") { ??? /* We only deal with GET and HEAD by default */ ??? return (pass); ? } ? if (req.http.Authorization || req.http.Cookie) { ??? /* Not cacheable by default */ ??? return (pass); ? } ? if (req.request == "PURGE") { ??? if (!client.ip ~ purge) { ????? error 405 "Not allowed."; ??? } ??? return (lookup); ? } ? if (req.request == "BAN") { ??? if (!client.ip ~ ban) { ????? error 405 "Not allowed."; ??? } ??? ban("req.http.host == " + req.http.host + "&& req.url == " + req.url); ??? error 200 "Ban added"; ? } ? return (lookup); } sub vcl_pipe { ? set bereq.http.connection = "close"; ? return (pipe); } sub vcl_pass { ? return (pass); } sub vcl_hash { ? hash_data(req.url); ? if (req.http.host) { ??? hash_data(req.http.host); ? } else { ??? hash_data(server.ip); ? } ? // If the client supports compression, keep that in a different cache ? if (req.http.Accept-Encoding) { ??? hash_data(req.http.Accept-Encoding); ? } ? return (hash); } sub vcl_hit { ? if (req.request == "PURGE") { ??? purge; ??? error 200 "Purged."; ? } ? //Listen to browser force refresh ? if (req.http.Cache-Control ~ "no-cache") { ??? if (! (req.http.Via || req.http.User-Agent ~ "bot|MSIE")) { ????? set obj.ttl = 0s; ????? return (restart); ??? } ? } ? return (deliver); } sub vcl_miss { ? if (req.request == "PURGE") { ??? purge; ??? error 200 "Purged."; ? } ? return (fetch); } sub vcl_fetch { ? if (beresp.ttl <= 0s || ??? beresp.http.Set-Cookie || ??? beresp.http.Vary == "*") { ????? set beresp.ttl = 120 s; ????? return (hit_for_pass); ? } ? return (deliver); } sub vcl_deliver { ? // Debugging ? if (obj.hits > 0) { ??? set resp.http.X-Cache = "HIT"; ? } else { ??? set resp.http.X-Cache = "MISS"; ? } ? // Remove some headers: PHP version ? unset resp.http.X-Powered-By; ? // Remove some headers: Apache version & OS ? unset resp.http.Server; ? return (deliver); } sub vcl_error { ? set obj.http.Content-Type = "text/html; charset=utf-8"; ? set obj.http.Retry-After = "5"; ? synthetic {" ? ??? "} + obj.status + " " + obj.response + {" ? ? ???

Error "} + obj.status + " " + obj.response + {"

???

"} + obj.response + {"

???

Guru Meditation:

???

XID: "} + req.xid + {"

???
???

Varnish cache server

? "}; ? return (deliver); } sub vcl_init { ? return (ok); } sub vcl_fini { ? return (ok); } -------------- next part -------------- An HTML attachment was scrubbed... URL: From miguel_3_gonzalez at yahoo.es Mon May 13 03:33:46 2013 From: miguel_3_gonzalez at yahoo.es (=?ISO-8859-1?Q?Miguel_Gonz=E1lez?=) Date: Mon, 13 May 2013 05:33:46 +0200 Subject: benchmarking varnish In-Reply-To: References: <1363087572.33460.YahooMailNeo@web171605.mail.ir2.yahoo.com> Message-ID: <3852_1368416025_51905F19_3852_7273_1_51905F1A.2040403@yahoo.es> On 12/03/2013 12:35, Jonathan Matthews wrote: > On 12 March 2013 11:26, Miguel Gonzalez wrote: >> Dear all, >> >> I have a Varnish 3 installation pointing to an Apache Server. Any manual >> of how to tune Varnish, OS, Apache and any manual of how to test Varnish >> considering it's a cache tool? > Have you worked through > https://www.varnish-cache.org/docs/3.0/tutorial/increasing_your_hitrate.html > yet? > > Jonathan Sorry I haven't replied for so long. I was trying to make sense of the docs. From my munin monitoring and varnishstat it seems I'm getting around 80% of hitrate in my Varnish cache. This Varnish cache has a remote Apache backup (Varnish is in a server in Europe while Apache is in a server in the US. How can I pinpoint why is not working fast? My default.vcl and /etc/sysconfig/varnish backend default { .host = "XXX.XXX.XXX.XXX"; .port = "8000"; .connect_timeout = 600s; .first_byte_timeout = 600s; .between_bytes_timeout = 600s; } acl purge { "localhost"; } acl ban { "localhost"; } sub vcl_recv { // Strip cookies for static files: if (req.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm|xml)$") { unset req.http.Cookie; } // Remove specific cookies: set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", ""); set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); if (req.http.cookie ~ "^ *$") { unset req.http.cookie; } if (req.http.Cookie ~ "^\s*$") { unset req.http.Cookie; } if (req.http.Cookie == "") { remove req.http.Cookie; } // Normalize Accept-Encoding header (straight from the manual: https://www.varnish-cache.org/docs/3.0/tutorial/vary.html) if (req.http.Accept-Encoding) { if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { // No point in compressing these remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { // unkown algorithm remove req.http.Accept-Encoding; } } if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For; } else { set req.http.X-Forwarded-For = client.ip; } } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.request != "GET" && req.request != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } if (req.http.Authorization || req.http.Cookie) { /* Not cacheable by default */ return (pass); } if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } return (lookup); } if (req.request == "BAN") { if (!client.ip ~ ban) { error 405 "Not allowed."; } ban("req.http.host == " + req.http.host + "&& req.url == " + req.url); error 200 "Ban added"; } return (lookup); } sub vcl_pipe { set bereq.http.connection = "close"; return (pipe); } sub vcl_pass { return (pass); } sub vcl_hash { hash_data(req.url); if (req.http.host) { hash_data(req.http.host); } else { hash_data(server.ip); } // If the client supports compression, keep that in a different cache if (req.http.Accept-Encoding) { hash_data(req.http.Accept-Encoding); } return (hash); } sub vcl_hit { if (req.request == "PURGE") { purge; error 200 "Purged."; } //Listen to browser force refresh if (req.http.Cache-Control ~ "no-cache") { if (! (req.http.Via || req.http.User-Agent ~ "bot|MSIE")) { set obj.ttl = 0s; return (restart); } } return (deliver); } sub vcl_miss { if (req.request == "PURGE") { purge; error 200 "Purged."; } return (fetch); } sub vcl_fetch { if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") { set beresp.ttl = 120 s; return (hit_for_pass); } return (deliver); } sub vcl_deliver { // Debugging if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } // Remove some headers: PHP version unset resp.http.X-Powered-By; // Remove some headers: Apache version & OS unset resp.http.Server; return (deliver); } sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; set obj.http.Retry-After = "5"; synthetic {" "} + obj.status + " " + obj.response + {"

Error "} + obj.status + " " + obj.response + {"

"} + obj.response + {"

Guru Meditation:

XID: "} + req.xid + {"

Varnish cache server

"}; return (deliver); } sub vcl_init { return (ok); } sub vcl_fini { return (ok); } -------------------------------------- # Configuration file for varnish # # /etc/init.d/varnish expects the variable $DAEMON_OPTS to be set from this # shell script fragment. # # Maximum number of open files (for ulimit -n) NFILES=131072 # Locked shared memory (for ulimit -l) # Default log size is 82MB + header MEMLOCK=82000 # Maximum size of corefile (for ulimit -c). Default in Fedora is 0 # DAEMON_COREFILE_LIMIT="unlimited" # Set this to 1 to make init script reload try to switch vcl without restart. # To make this work, you need to set the following variables # explicit: VARNISH_VCL_CONF, VARNISH_ADMIN_LISTEN_ADDRESS, # VARNISH_ADMIN_LISTEN_PORT, VARNISH_SECRET_FILE, or in short, # use Alternative 3, Advanced configuration, below RELOAD_VCL=1 # This file contains 4 alternatives, please use only one. ## Alternative 1, Minimal configuration, no VCL # # Listen on port 6081, administration on localhost:6082, and forward to # content server on localhost:8080. Use a fixed-size cache file. # #DAEMON_OPTS="-a :6081 \ # -T localhost:6082 \ # -b localhost:8080 \ # -u varnish -g varnish \ # -s file,/var/lib/varnish/varnish_storage.bin,1G" ## Alternative 2, Configuration with VCL # # Listen on port 6081, administration on localhost:6082, and forward to # one content server selected by the vcl file, based on the request. Use a # fixed-size cache file. # #DAEMON_OPTS="-a :6081 \ # -T localhost:6082 \ # -f /etc/varnish/default.vcl \ # -u varnish -g varnish \ # -S /etc/varnish/secret \ # -s file,/var/lib/varnish/varnish_storage.bin,1G" ## Alternative 3, Advanced configuration # # See varnishd(1) for more information. # # # Main configuration file. You probably want to change it :) VARNISH_VCL_CONF=/etc/varnish/default.vcl # # # Default address and port to bind to # # Blank address means all IPv4 and IPv6 interfaces, otherwise specify # # a host name, an IPv4 dotted quad, or an IPv6 address in brackets. # VARNISH_LISTEN_ADDRESS= VARNISH_LISTEN_PORT=80 # # # Telnet admin interface listen address and port VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 VARNISH_ADMIN_LISTEN_PORT=6082 # # # Shared secret file for admin interface VARNISH_SECRET_FILE=/etc/varnish/secret # # # The minimum number of worker threads to start VARNISH_MIN_THREADS=800 # # # The Maximum number of worker threads to start VARNISH_MAX_THREADS=1000 # # # Idle timeout for worker threads VARNISH_THREAD_TIMEOUT=120 VARNISH_CACHE_SIZE=18G VARNISH_CACHE="malloc,${VARNISH_CACHE_SIZE}" # # # Cache file location #VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin # # # Cache file size: in bytes, optionally using k / M / G / T suffix, # # or in percentage of available disk space using the % suffix. #VARNISH_STORAGE_SIZE=1G # # # Backend storage specification #VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}" # # # Default TTL used when the backend does not specify one VARNISH_TTL=120 # # # DAEMON_OPTS is used by the init script. If you add or remove options, make # # sure you update this section, too. DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ -f ${VARNISH_VCL_CONF} \ -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ -t ${VARNISH_TTL} \ -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \ -u varnish -g varnish \ -S ${VARNISH_SECRET_FILE} \ -s ${VARNISH_CACHE}" # ## Alternative 4, Do It Yourself. See varnishd(1) for more information. # # DAEMON_OPTS="" This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by the sender. Please consider the environment before printing this email. From miguel_3_gonzalez at yahoo.es Tue May 14 18:07:58 2013 From: miguel_3_gonzalez at yahoo.es (=?ISO-8859-1?Q?Miguel_Gonz=E1lez?=) Date: Tue, 14 May 2013 20:07:58 +0200 Subject: benchmarking varnish In-Reply-To: <3852_1368416025_51905F19_3852_7273_1_51905F1A.2040403@yahoo.es> References: <1363087572.33460.YahooMailNeo@web171605.mail.ir2.yahoo.com> <3852_1368416025_51905F19_3852_7273_1_51905F1A.2040403@yahoo.es> Message-ID: <51927D7E.5010804@yahoo.es> On 13/05/2013 05:33, Miguel Gonz?lez wrote: > On 12/03/2013 12:35, Jonathan Matthews wrote: >> On 12 March 2013 11:26, Miguel Gonzalez >> wrote: >>> Dear all, >>> >>> I have a Varnish 3 installation pointing to an Apache Server. Any >>> manual >>> of how to tune Varnish, OS, Apache and any manual of how to test >>> Varnish >>> considering it's a cache tool? >> Have you worked through >> https://www.varnish-cache.org/docs/3.0/tutorial/increasing_your_hitrate.html >> >> yet? >> >> Jonathan > Sorry I haven't replied for so long. I was trying to make sense of the > docs. > > From my munin monitoring and varnishstat it seems I'm getting around > 80% of hitrate in my Varnish cache. This Varnish cache has a remote > Apache backup (Varnish is in a server in Europe while Apache is in a > server in the US. > > How can I pinpoint why is not working fast? It seems one of the issues was that pingdom was misleading because It always reported all tests were resulting into a MISS instead of a HIT. Checking with wget -S gave some real information. Also normalizing user agents could help when it comes to caching images and certain files: http://serverfault.com/questions/365056/varnish-only-cache-assets-from-single-session this seems to have increased the hitrate Regards, Miguel From wxz19861013 at gmail.com Wed May 15 02:44:48 2013 From: wxz19861013 at gmail.com (Xianzhe Wang) Date: Wed, 15 May 2013 10:44:48 +0800 Subject: varnish-cache-3.0.3-plus increase %CPU and crashed by "errno = 12(Cannot allocate memory)" Message-ID: Hi, I used varnish-cache-3.0.3-plus(get from https://github.com/mbgrydeland/varnish-cache/tree/3.0.3-plus). I use persistent storage and "set beresp.do_stream = true". The %CPU is increased from 4% to 800% (8 cpu), and varnish crashed sometimes by errno = 12 (Cannot allocate memory). my startup command is: /opt/ace/cache/varnish-cache-3.0.3-plus/sbin/varnishd -f /opt/ace/cache/varnish-stream.vcl -s persistent,/opt/ace/file/varnish_cache_persistent.data,12G -T 127.0.0.1:2000-a 0.0.0.0:8080 -p thread_pool_min 200 -p thread_pool_max 2000 -p thread_pool_add_delay 2 -p session_linger 100 -p thread_pools 2 -p http_req_hdr_len 32768 -p http_resp_hdr_len 32768 -p http_max_hdr 256 Then I change the command to: /opt/ace/cache/varnish-cache-3.0.3-plus/sbin/varnishd -f /opt/ace/cache/varnish-stream.vcl -s persistent,/opt/ace/file/varnish_cache_persistent.data,12G -T 127.0.0.1:2000-a 0.0.0.0:8080 *-h classic,500009* -p thread_pool_min 200 -p thread_pool_max 2000 -p thread_pool_add_delay 2 -p session_linger 100 -p thread_pools 2 -p http_req_hdr_len 32768 -p http_resp_hdr_len 32768 -p http_max_hdr 256 *-p lru_interval 3600* The %CPU is increasing too, but varnish hasn't crashed for 16 hours. Is there somebody has experience about this problem? Any advice will be greatly appreciated. Here is the information about system and varnish. top: top - 09:37:36 up 50 days, 13:06, 3 users, load average: 5.00, 4.97, 4.73 Tasks: 141 total, 1 running, 140 sleeping, 0 stopped, 0 zombie Cpu(s): 62.6%us, 0.0%sy, 0.0%ni, 37.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 20132M total, 19781M used, 351M free, 137M buffers Swap: 4102M total, 3M used, 4098M free, 18517M cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12255 nobody 20 0 14.7g 11g 11g S 502 59.8 2280:28 varnishd 30934 acce 20 0 8772 1212 872 R 0 0.0 0:00.02 top free -m: total used free shared buffers cached Mem: 20132 19782 350 0 137 18517 -/+ buffers/cache: 1127 19004 Swap: 4102 3 4098 ulimit -a: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 156007 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) 17523512 open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 4096 cpu time (seconds, -t) unlimited max user processes (-u) 156007 virtual memory (kbytes, -v) 19853520 file locks (-x) unlimited varnish panic.show: Last panic at: Tue, 14 May 2013 02:56:50 GMT Assert error in CNT_Session(), cache_center.c line 1813: Condition((w->do_stream) == 0) not true. errno = 12 (Cannot allocate memory) thread = (cache-worker) ident = Linux,2.6.32.59-0.7-xen,x86_64,-spersistent,-smalloc,-hcritbit,epoll sp = 0x7f4dcd947008 { fd = 1435, id = 1435, xid = 1304392785, client = xx.xx.xx.xx 2396, step = STP_DONE, handling = hit_for_pass, err_code = 502, err_reason = (null), restarts = 0, esi_level = 0 flags = do_stream is_gunzip bodystatus = 4 ws = 0x7f4dcd947080 { id = "sess", {s,f,r,e} = {0x7f4dcd949610,+344,(nil),+65536}, }, http[req] = { ws = 0x7f4dcd947080[sess] "GET", "http://www.2xk.cn/index.asp", "HTTP/1.1", "Accept: */*", "Referer: http://www.2xk.cn/", "Accept-Language: zh-cn", "User-Agent:", "Host: www.2xk.cn", "Proxy-Connection: Keep-Alive", "Pragma: no-cache", "X-Forwarded-For: xx.xx.xx.xx", "Accept-Encoding: gzip", }, worker = 0x7f4d83a1ca90 { ws = 0x7f4d83a1ccd8 { id = "wrk", {s,f,r,e} = {0x7f4d83a0aa20,+616,(nil),+65536}, }, }, vcl = { srcname = { "input", "Default", }, }, }, /opt/ace/cache/varnish-cache-3.0.3-plus/bin/varnishstat -1 acce at sz-acce-varnish01:~> /opt/huawei/ace/cache/varnish-cache-3.0.3-plus/bin/varnishstat -1 client_conn 2341640 40.03 Client connections accepted client_drop 8 0.00 Connection dropped, no sess/wrk client_req 3031369 51.82 Client requests received cache_hit 164025 2.80 Cache hits cache_hitpass 409263 7.00 Cache hits for pass cache_miss 1857720 31.75 Cache misses backend_conn 500716 8.56 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 8097 0.14 Backend conn. failures backend_reuse 2375031 40.60 Backend conn. reuses backend_toolate 34982 0.60 Backend conn. was closed backend_recycle 2410099 41.20 Backend conn. recycles backend_retry 15375 0.26 Backend conn. retry fetch_head 1051 0.02 Fetch head fetch_length 2185540 37.36 Fetch with Length fetch_chunked 600378 10.26 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_streamed 2792280 47.73 Fetch streamed fetch_bad 0 0.00 Fetch had bad headers fetch_close 10117 0.17 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 37332 0.64 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 2720 0.05 Fetch no body (204) fetch_304 12112 0.21 Fetch no body (304) n_sess_mem 7231 . N struct sess_mem n_sess 93 . N struct sess n_object 86410 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 86666 . N struct objectcore n_objecthead 51679 . N struct objecthead n_waitinglist 22932 . N struct waitinglist n_vbc 24 . N struct vbc n_wrk 400 . N worker threads n_wrk_create 14003 0.24 N worker threads created n_wrk_failed 2553 0.04 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 114487 1.96 N queued work requests n_wrk_drop 8 0.00 N dropped work requests n_backend 4 . N backends n_expired 1427930 . N expired objects n_lru_nuked 42056 . N LRU nuked objects n_lru_moved 0 . N LRU moved objects losthdr 112 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 194021 3.32 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 2341612 40.03 Total Sessions s_req 3031369 51.82 Total Requests s_pipe 7436 0.13 Total pipe s_pass 1003423 17.15 Total pass s_fetch 17926 0.31 Total fetch s_stream 2787698 47.65 Total streamed requests s_hdrbytes 1048771245 17927.10 Total header bytes s_bodybytes 20286532441 346766.48 Total body bytes sess_closed 1781645 30.45 Session Closed sess_pipeline 11352 0.19 Session Pipeline sess_readahead 7111 0.12 Session Read Ahead sess_linger 1375398 23.51 Session Linger sess_herd 874711 14.95 Session herd shm_records 244018270 4171.11 SHM records shm_writes 19725032 337.17 SHM writes shm_flushes 3325 0.06 SHM flushes due to overflow shm_cont 33597 0.57 SHM MTX contention shm_cycles 104 0.00 SHM cycles through buffer sms_nreq 49339 0.84 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 23339027 . SMS bytes allocated sms_bfree 23339027 . SMS bytes freed backend_req 2867335 49.01 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_gone 1 . N total gone bans n_ban_add 6 0.00 N new bans added n_ban_retire 5 0.00 N old bans deleted n_ban_obj_test 23688 0.40 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 5 0.00 N duplicate bans removed hcb_nolock 0 0.00 HCB Lookups without lock hcb_lock 0 0.00 HCB Lookups with lock hcb_insert 0 0.00 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 58502 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 0 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 1266712 21.65 Gunzip operations LCK.sms.creat 1 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 148017 2.53 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 1 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 10868618 185.78 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 1 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 4945521 84.54 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 0 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 0 0.00 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 0 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 0 0.00 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 500009 8.55 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 4918023 84.07 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 2573092 43.98 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 7933355 135.61 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 7932823 135.60 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 142801 2.44 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 4032 0.07 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 2 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 11199698 191.44 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 800591 13.68 Created locks LCK.objhdr.destroy 748913 12.80 Destroyed locks LCK.objhdr.locks 15295089 261.45 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 3544111 60.58 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 9854 0.17 Created locks LCK.lru.destroy 8840 0.15 Destroyed locks LCK.lru.locks 2066424 35.32 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 19486 0.33 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 3842809 65.69 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 7808 0.13 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 1017738 17.40 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 4 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 6346813 108.49 Lock Operations LCK.backend.colls 0 0.00 Collisions LCK.busyobj.creat 13983 0.24 Created locks LCK.busyobj.destroy 13582 0.23 Destroyed locks LCK.busyobj.locks 47081281 804.78 Lock Operations LCK.busyobj.colls 0 0.00 Collisions SMP.s0.c_req 3025767 . Allocater requests SMP.s0.c_fail 352 . Allocater failures SMP.s0.c_bytes 115414281600 . Bytes allocated SMP.s0.c_freed 110868836208 . Bytes freed SMP.s0.g_alloc 158780 . Allocations outstanding SMP.s0.g_bytes 4545445392 . Bytes outstanding SMP.s0.g_free 138490880 . Free space SMP.s0.g_free_pending 0 . Free space pending SMP.s0.g_segments 1012 . In use segments SMP.s0.c_segments_nuked 8506 . Number of nuked segments SMP.s0.g_vampireobjects 0 . Vampire objects SMP.s0.c_resurrection_fail 46 . Failed resurrections SMA.Transient.c_req 2271927 38.84 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 59014304554 1008757.04 Bytes allocated SMA.Transient.c_freed 58939691205 1007481.65 Bytes freed SMA.Transient.g_alloc 18069 . Allocations outstanding SMA.Transient.g_bytes 74613349 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.proxy1(10.21.34.21,,8080).vcls 1 . VCL references VBE.proxy1(10.21.34.21,,8080).happy18446744073709551615 . Happy health probes VBE.proxy2(10.21.34.22,,8080).vcls 1 . VCL references VBE.proxy2(10.21.34.22,,8080).happy18446744056529682431 . Happy health probes VBE.proxy3(10.21.34.23,,8082).vcls 1 . VCL references VBE.proxy3(10.21.34.23,,8082).happy18446744073709551615 . Happy health probes VBE.proxy4(10.21.34.24,,8081).vcls 1 . VCL references VBE.proxy4(10.21.34.24,,8081).happy18446744073709551615 . Happy health probes varnish-stream.vcl: acce at sz-acce-varnish01:~> cat /opt/huawei/ace/cache/varnish-stream.vcl # This is a basic VCL configuration file for varnish. See the vcl(7) # man page for details on VCL syntax and semantics. # # Default backend definition. Set this to point to your content # server. # probe healthcheck { .url = "/"; .interval = 30s; .timeout = 0.5s; .window = 8; .threshold = 3; .initial = 3; } backend proxy1 { .host = "10.21.34.21"; .port = "8080"; .probe = healthcheck; } backend proxy2 { .host = "10.21.34.22"; .port = "8080"; .probe = healthcheck; } backend proxy3 { .host = "10.21.34.23"; .port = "8082"; .probe = healthcheck; } backend proxy4 { .host = "10.21.34.24"; .port = "8081"; .probe = healthcheck; } director proxy client { { .backend = proxy1; .weight = 1; } { .backend = proxy2; .weight = 1; } # { # .backend = proxy3; # .weight = 1; # } # { # .backend = proxy4; # .weight = 1; # } } acl refresh { "10.21.34.31"; } # Below is a commented-out copy of the default VCL logic. If you # redefine any of these subroutines, the built-in logic will be # appended to your code. sub vcl_recv { if(req.http.X-Real-IP){ set client.identity = req.http.X-Real-IP; }else if (req.http.referer) { set client.identity = req.http.referer; }else{ set client.identity = req.url; } # set req.backend = proxy; if (client.ip == "10.21.34.11" || client.ip == "10.21.34.12"){ if (req.http.X-SZ-TEST && req.http.X-SZ-TEST == "1"){ set req.backend = proxy3; } else { set req.backend = proxy; } } else { set req.backend = proxy4; if (client.ip ~ refresh) { set req.hash_always_miss = true; } } #set grace if (req.backend.healthy) { set req.grace = 30s; } else { set req.grace = 30m; } if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { set req.http.X-Forwarded-For = client.ip; } } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { # /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.http.x-pipe && req.restarts > 0) { remove req.http.x-pipe; return (pipe); } if(req.request != "GET" && req.request != "HEAD") { # /* We only deal with GET and HEAD by default */ return (pass); } if (req.http.Cache-Control ~ "no-cache") { return (pass); } if (req.http.Accept-Encoding) { if (req.url ~ "\.(webp|jpeg|png|mid|mp3|gif|sql|jpg|nth|thm|utz|mtf|sdt|hme|tsk|zip|rar|sx|pxl|cab|mbm|app|exe|apk)$") { # No point in compressing these remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { # unknown algorithm remove req.http.Accept-Encoding; } } # if (req.request == "GET" && # req.url ~ "\.(webp|jpeg|png|mid|mp3|gif|sql|jpg|nth|thm|utz|mtf|sdt|hme|tsk|zip|rar|sx|pxl|cab|mbm|app|exe|apk)$"){ # unset req.http.Cookie; # return (lookup); # } # if (req.http.Authorization || req.http.Cookie) { # return (pass); # } if (req.http.Authorization) { return (pass); } return (lookup); } sub vcl_pipe { # # Note that only the first request to the backend will have # # X-Forwarded-For set. If you use X-Forwarded-For and want to # # have it set for all requests, make sure to have: set bereq.http.connection = "close"; # # here. It is not set by default as it might break some broken web # # applications, like IIS with NTLM authentication. return (pipe); } sub vcl_pass { return (pass); } sub vcl_hash { if (req.url ~ ".(jpeg|jpg|png|gif|ico|js|css)\?.*") { hash_data(regsub(req.url, "\?[^\?]*$", "")); } else{ hash_data(req.url); } # hash_data(req.url); if (req.http.host) { hash_data(req.http.host); } else { hash_data(server.ip); } return (hash); } sub vcl_hit { return (deliver); } sub vcl_miss { return (fetch); } sub vcl_fetch { set beresp.grace = 30m; set beresp.do_stream = true; if (beresp.http.Content-Length && beresp.http.Content-Length ~ "[0-9]{8,}") { set req.http.x-pipe = "1"; return (restart); } if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ "no-cache" || beresp.http.Cache-Control ~ "private"){ return (hit_for_pass); } if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") { set beresp.ttl = 120 s; return (hit_for_pass); } if(beresp.status >=300 && beresp.status <= 307){ set beresp.ttl = 120 s; return (hit_for_pass); } return (deliver); } sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Cache = "HIT from varnis"; set resp.http.X-Hits = obj.hits; } else { set resp.http.X-Cache = "MISS from varnish"; } remove resp.http.Via; remove resp.http.X-Varnish; return (deliver); } sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; set obj.http.Retry-After = "5"; synthetic {" "} + obj.status + " " + obj.response + {"

Error "} + obj.status + " " + obj.response + {"

"} + obj.response + {"

Guru Meditation:

XID: "} + req.xid + {"

Varnish cache server

varnish

"}; return (deliver); } sub vcl_init { return (ok); } sub vcl_fini { return (ok); } varnish param.show param.show 200 acceptor_sleep_decay 0.900000 [] acceptor_sleep_incr 0.001000 [s] acceptor_sleep_max 0.050000 [s] auto_restart on [bool] ban_dups on [bool] ban_lurker_sleep 0.010000 [s] between_bytes_timeout 60.000000 [s] cc_command "exec gcc -std=gnu99 -pthread -fpic -shared -Wl,-x -o %o %s" cli_buffer 8192 [bytes] cli_timeout 10 [seconds] clock_skew 10 [s] connect_timeout 0.700000 [s] critbit_cooloff 180.000000 [s] default_grace 10.000000 [seconds] default_keep 0.000000 [seconds] default_ttl 120.000000 [seconds] diag_bitmap 0x0 [bitmap] esi_syntax 0 [bitmap] expiry_sleep 1.000000 [seconds] fetch_chunksize 128 [kilobytes] fetch_maxchunksize 262144 [kilobytes] first_byte_timeout 60.000000 [s] group nobody (65533) gzip_level 6 [] gzip_memlevel 8 [] gzip_stack_buffer 32768 [Bytes] gzip_tmp_space 0 [] gzip_window 15 [] http_gzip_support on [bool] http_max_hdr 256 [header lines] http_range_support on [bool] http_req_hdr_len 32768 [bytes] http_req_size 32768 [bytes] http_resp_hdr_len 32768 [bytes] http_resp_size 32768 [bytes] idle_send_timeout 60 [seconds] listen_address 0.0.0.0:8080 listen_depth 1024 [connections] log_hashstring on [bool] log_local_address off [bool] lru_interval 3600 [seconds] max_esi_depth 5 [levels] max_restarts 4 [restarts] nuke_limit 50 [allocations] pcre_match_limit 10000 [] pcre_match_limit_recursion 10000 [] ping_interval 3 [seconds] pipe_timeout 60 [seconds] prefer_ipv6 off [bool] queue_max 100 [%] rush_exponent 3 [requests per request] saintmode_threshold 10 [objects] send_timeout 600 [seconds] sess_timeout 5 [seconds] sess_workspace 65536 [bytes] session_linger 100 [ms] session_max 100000 [sessions] shm_reclen 255 [bytes] shm_workspace 8192 [bytes] shortlived 10.000000 [s] stream_maxchunksize 256 [kilobytes] stream_tokens 10 [] syslog_cli_traffic on [bool] thread_pool_add_delay 2 [milliseconds] thread_pool_add_threshold 0 [requests] thread_pool_fail_delay 200 [milliseconds] thread_pool_max 2000 [threads] thread_pool_min 200 [threads] thread_pool_purge_delay 1000 [milliseconds] thread_pool_stack unlimited [bytes] thread_pool_timeout 300 [seconds] thread_pool_workspace 65536 [bytes] thread_pools 2 [pools] thread_stats_rate 10 [requests] user nobody (65534) vcc_err_unref on [bool] vcl_dir /opt//ace/cache/varnish-cache-3.0.3-plus/etc/varnish vcl_trace off [bool] vmod_dir /opt/ace/cache/varnish-cache-3.0.3-plus/lib/varnish/vmods waiter default (epoll, poll) Thank you again. Regards, Shawn -------------- next part -------------- An HTML attachment was scrubbed... URL: From wxz19861013 at gmail.com Wed May 15 08:21:22 2013 From: wxz19861013 at gmail.com (Xianzhe Wang) Date: Wed, 15 May 2013 16:21:22 +0800 Subject: varnish-cache-3.0.3-plus increase %CPU and crashed by "errno = 12(Cannot allocate memory)" In-Reply-To: References: Message-ID: Hi, I use the order: ps H -e -o pid,tid,pcpu,cmd --sort=pcpu |grep varnish I get the message: 12255 12280 0.1 /opt/ace/cache/varnish-cache-3.0.3-plus/sbin/varnishd -f /opt/ace/cache/varnish-stream.vcl -s persistent,/opt/ace/file/varnish_cache_persistent.data,12G -T 127.0.0.1:2000-a 0.0.0.0:8080 -h classic,500009 -p thread_pool_min 200 -p thread_pool_max 2000 -p thread_pool_add_delay 2 -p session_linger 100 -p thread_pools 2 -p http_req_hdr_len 32768 -p http_resp_hdr_len 32768 -p http_max_hdr 256 -p lru_interval 3600 12255 30175 22.6 ?? ?omit startup command? 12255 26385 87.2 ?? ?omit startup command? 12255 30478 89.1 ?? ?omit startup command? 12255 21623 96.3 ?? ?omit startup command? 12255 12504 96.7 ?? ?omit startup command? There are several threads consume about 100% CPU. Then I debugged varnish, attach the threads above. I found they lingered in in VGZ_WrwGunzip . I think there may be some *endless loop* or *deadlock* issues. There are some debug log about above threads: attach 21623 ... inflate (strm=0x7fd5652680f8, flush=0) at inflate.c:1222 1222 RESTORE(); (gdb) info thread * 1 process 21623 inflate (strm=0x7fd5652680f8, flush=0) at inflate.c:1222 (gdb) thread 1 [Switching to thread 1 (process 21623)]#0 inflate (strm=0x7fd5652680f8, flush=0) at inflate.c:1222 1222 RESTORE(); (gdb) bt #0 inflate (strm=0x7fd5652680f8, flush=0) at inflate.c:1222 #1 0x0000000000428068 in VGZ_Gunzip (vg=0x7fd5652680c0, pptr=0x7fd55eddf458, plen=0x7fd55eddf460) at cache_gzip.c:290 #2 0x00000000004287e4 in VGZ_WrwGunzip (sp=0x7fd575a13008, vg=0x7fd5652680c0, ibuf=, ibufl=, obuf=0x7fd55eddf4f0 "\n\n