From antonymayi at yahoo.com Thu Mar 6 20:16:45 2014 From: antonymayi at yahoo.com (Antony Mayi) Date: Thu, 6 Mar 2014 20:16:45 +0000 (GMT) Subject: chunked response transfer encoding Message-ID: <1394137005.61529.YahooMailNeo@web133001.mail.ir2.yahoo.com> Hi, my backend is sending chunked response (~100chunks, 100bytes each). it seems varnish then sends it again using chunked encoding but only as one big chunk. the app is sort of a json streaming system and the whole point of sending the responses in chunks is for the client to be able to handle each chunk asap once they are available. unfortunately varnish cumulates it into (which means waits for) one big chunk so the whole advantage of early client processing is gone. is there any way to control this in varnish - ie. some timeouts or buffer sizes etc (not interested in piping)? thanks, Antony. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Paul.McInerney at faredge.com.au Thu Mar 6 22:57:15 2014 From: Paul.McInerney at faredge.com.au (Paul McInerney) Date: Fri, 7 Mar 2014 09:57:15 +1100 Subject: varnish 3.0 + mobile app Message-ID: Hi all, We are currently developing a mobile android/iphone app for our website. When the app hits the varnish frontend, it has the user-agent string set as 'mobileapp'. So I have put in place a subroutine that correctly identifies the app and sets a custom x-device header to send to the backend for rendering the page(s) in the required format. This part is working well with the same URL now serving both formats of the page, however when I issue a purge against the URL, it purges the cached versions of the PC version of the page, and doesn't clear all variants (which should include the mobileapp version afaik ) Here's the subroutine called from vlc_recv sub identify_device { unset req.http.X-UA-Device; set req.http.X-UA-Device = "pc"; if (req.http.User-Agent ~ "mobileapp" ) { set req.http.X-UA-Device = "mobileapp"; } } and the hash being set based on device sub vcl_hash { if (req.http.X-UA-Device) { hash_data(req.http.X-UA-Device); } } What am I doing wrong? or missing? or not understanding? Cheers, :) -- Paul McInerney Senior Systems Administrator Far Edge Technology P: +61 2 8425 1410 F: +61 2 8425 1489 M: +61 411 288 030 E: paul.mcinerney at faredge.com.au -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.jennings at nytimes.com Thu Mar 6 23:06:21 2014 From: raymond.jennings at nytimes.com (Jennings, Raymond) Date: Thu, 6 Mar 2014 18:06:21 -0500 Subject: varnish 3.0 + mobile app In-Reply-To: References: Message-ID: I assume your purge is not hashing properly. When you purge it has to match whatever it hashed on when the obj went into the cache. I have problems using the standard sub vcl_hash so I hash only on the URL and not the hostname so this let's me purge from outside of the "restricted area." In your case I think you probably need to purge all possibilities based on that X-header you are setting. If that gets too difficult I fallback on doing a BAN instead. Ray On Thu, Mar 6, 2014 at 5:57 PM, Paul McInerney < Paul.McInerney at faredge.com.au> wrote: > Hi all, > > > > We are currently developing a mobile android/iphone app for our website. > > When the app hits the varnish frontend, it has the user-agent string set > as 'mobileapp'. > > So I have put in place a subroutine that correctly identifies the app and > sets a > custom x-device header to send to the backend for rendering the page(s) > in the > > required format. > > This part is working well with the same URL now serving both formats of > the page, > > however when I issue a purge against the URL, it purges the cached > versions of the > > PC version of the page, and doesn't clear all variants (which should > include the mobileapp > > version afaik ) > > Here's the subroutine called from vlc_recv > > sub identify_device { > unset req.http.X-UA-Device; > set req.http.X-UA-Device = "pc"; > > if (req.http.User-Agent ~ "mobileapp" ) { > set req.http.X-UA-Device = "mobileapp"; > } > } > > and the hash being set based on device > > sub vcl_hash { > if (req.http.X-UA-Device) { > hash_data(req.http.X-UA-Device); > } > } > > What am I doing wrong? or missing? or not understanding? > > > > Cheers, > > > > J > > > > -- > > *Paul McInerney* > > Senior Systems Administrator > > Far Edge Technology > > > > P: +61 2 8425 1410 > > F: +61 2 8425 1489 > > M: +61 411 288 030 > > E: paul.mcinerney at faredge.com.au > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From javier at casares.org Thu Mar 6 23:26:20 2014 From: javier at casares.org (Javier Casares) Date: Fri, 7 Mar 2014 00:26:20 +0100 Subject: varnish 3.0 + mobile app In-Reply-To: References: Message-ID: Try sub identify_device { if (req.http.User-Agent ~ "mobileapp" ) { set req.http.X-UA-Device = "mobileapp"; } else { set req.http.X-UA-Device = "pc"; } } And sub vcl_hash { if (req.http.X-UA-Device) { hash_data(req.http.X-UA-Device); } } From raymond.jennings at nytimes.com Thu Mar 6 23:26:50 2014 From: raymond.jennings at nytimes.com (Jennings, Raymond) Date: Thu, 6 Mar 2014 18:26:50 -0500 Subject: Anyone use a %D in varnishncsa (like in Apache) Message-ID: I recently tired it from someone else telling me it works. The man pages for varnishncsa do not list it as a valid format option. I used for a day and it seems to work but occasionally I do get some negative floating point response times. Does anyone know if this format is "semi supported" ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Paul.McInerney at faredge.com.au Thu Mar 6 23:50:56 2014 From: Paul.McInerney at faredge.com.au (Paul McInerney) Date: Fri, 7 Mar 2014 10:50:56 +1100 Subject: varnish 3.0 + mobile app In-Reply-To: References: Message-ID: I thought that myself, and have tried purging with the specific header being called - still no luck. Doesn't a purge by default, remove all variants of the specified URL? :) From: Jennings, Raymond [mailto:raymond.jennings at nytimes.com] Sent: Friday, 7 March 2014 10:06 AM To: Paul McInerney Cc: varnish-misc at varnish-cache.org Subject: Re: varnish 3.0 + mobile app I assume your purge is not hashing properly. When you purge it has to match whatever it hashed on when the obj went into the cache. I have problems using the standard sub vcl_hash so I hash only on the URL and not the hostname so this let's me purge from outside of the "restricted area." In your case I think you probably need to purge all possibilities based on that X-header you are setting. If that gets too difficult I fallback on doing a BAN instead. Ray On Thu, Mar 6, 2014 at 5:57 PM, Paul McInerney > wrote: Hi all, We are currently developing a mobile android/iphone app for our website. When the app hits the varnish frontend, it has the user-agent string set as 'mobileapp'. So I have put in place a subroutine that correctly identifies the app and sets a custom x-device header to send to the backend for rendering the page(s) in the required format. This part is working well with the same URL now serving both formats of the page, however when I issue a purge against the URL, it purges the cached versions of the PC version of the page, and doesn't clear all variants (which should include the mobileapp version afaik ) Here's the subroutine called from vlc_recv sub identify_device { unset req.http.X-UA-Device; set req.http.X-UA-Device = "pc"; if (req.http.User-Agent ~ "mobileapp" ) { set req.http.X-UA-Device = "mobileapp"; } } and the hash being set based on device sub vcl_hash { if (req.http.X-UA-Device) { hash_data(req.http.X-UA-Device); } } What am I doing wrong? or missing? or not understanding? Cheers, :) -- Paul McInerney Senior Systems Administrator Far Edge Technology P: +61 2 8425 1410 F: +61 2 8425 1489 M: +61 411 288 030 E: paul.mcinerney at faredge.com.au _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.jennings at nytimes.com Thu Mar 6 23:56:10 2014 From: raymond.jennings at nytimes.com (Jennings, Raymond) Date: Thu, 6 Mar 2014 18:56:10 -0500 Subject: varnish 3.0 + mobile app In-Reply-To: References: Message-ID: No it does not - it has to match the hash. BAN will do the trick but it has the (small) overhead - depending upon your setup. My guess is that you might be doing what I was and trying to do in the beginning which was to purge from another interface. Your sub vcl_hash is not returning so the default sub vcl_hash runs after yours and that also hashes the Host header. You could try to add the url to your sub vcl_hash and the last line put "return(hash);" (to prevent the default handler from running.) Then try the same purge test you were doing with the headers set. On Thu, Mar 6, 2014 at 6:50 PM, Paul McInerney < Paul.McInerney at faredge.com.au> wrote: > I thought that myself, and have tried purging with the specific header > being called - still no luck. > > > > Doesn't a purge by default, remove all variants of the specified URL? > > > > J > > > > *From:* Jennings, Raymond [mailto:raymond.jennings at nytimes.com] > *Sent:* Friday, 7 March 2014 10:06 AM > *To:* Paul McInerney > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: varnish 3.0 + mobile app > > > > I assume your purge is not hashing properly. When you purge it has to > match whatever it hashed on when the obj went into the cache. > > > > I have problems using the standard sub vcl_hash so I hash only on the URL > and not the hostname so this let's me purge from outside of the "restricted > area." > > > > > > In your case I think you probably need to purge all possibilities based on > that X-header you are setting. > > > > If that gets too difficult I fallback on doing a BAN instead. > > > > > > Ray > > > > On Thu, Mar 6, 2014 at 5:57 PM, Paul McInerney < > Paul.McInerney at faredge.com.au> wrote: > > Hi all, > > > > We are currently developing a mobile android/iphone app for our website. > > When the app hits the varnish frontend, it has the user-agent string set > as 'mobileapp'. > > So I have put in place a subroutine that correctly identifies the app and > sets a > custom x-device header to send to the backend for rendering the page(s) > in the > > required format. > > This part is working well with the same URL now serving both formats of > the page, > > however when I issue a purge against the URL, it purges the cached > versions of the > > PC version of the page, and doesn't clear all variants (which should > include the mobileapp > > version afaik ) > > Here's the subroutine called from vlc_recv > > sub identify_device { > unset req.http.X-UA-Device; > set req.http.X-UA-Device = "pc"; > > if (req.http.User-Agent ~ "mobileapp" ) { > set req.http.X-UA-Device = "mobileapp"; > } > } > > and the hash being set based on device > > sub vcl_hash { > if (req.http.X-UA-Device) { > hash_data(req.http.X-UA-Device); > } > } > > What am I doing wrong? or missing? or not understanding? > > > > Cheers, > > > > J > > > > -- > > *Paul McInerney* > > Senior Systems Administrator > > Far Edge Technology > > > > P: +61 2 8425 1410 > > F: +61 2 8425 1489 > > M: +61 411 288 030 > > E: paul.mcinerney at faredge.com.au > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Mar 7 07:30:05 2014 From: perbu at varnish-software.com (Per Buer) Date: Fri, 7 Mar 2014 08:30:05 +0100 Subject: varnish 3.0 + mobile app In-Reply-To: References: Message-ID: Hi Paul, On Thu, Mar 6, 2014 at 11:57 PM, Paul McInerney < Paul.McInerney at faredge.com.au> wrote: > Hi all, > > > > We are currently developing a mobile android/iphone app for our website. > > When the app hits the varnish frontend, it has the user-agent string set > as 'mobileapp'. > > So I have put in place a subroutine that correctly identifies the app and > sets a > custom x-device header to send to the backend for rendering the page(s) > in the > > required format. > > This part is working well with the same URL now serving both formats of > the page, > > however when I issue a purge against the URL, it purges the cached > versions of the > > PC version of the page, and doesn't clear all variants (which should > include the mobileapp > > version afaik ) > It's because you are doing it wrong. There is a very elegant solution to this; HTTP Vary. https://www.varnish-cache.org/docs/3.0/tutorial/vary.html You are trying to manipulate the hash, but it fails as you most likely forget to do the same manipulations when purging. This would not be a problem if you where to use Vary as all the variants would be connected to the same hash value. You can either Vary directly on the custom headers that you are creating or use it on the X-UA-Device Javier suggested. I like that. -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Mar 7 07:32:13 2014 From: perbu at varnish-software.com (Per Buer) Date: Fri, 7 Mar 2014 08:32:13 +0100 Subject: chunked response transfer encoding In-Reply-To: <1394137005.61529.YahooMailNeo@web133001.mail.ir2.yahoo.com> References: <1394137005.61529.YahooMailNeo@web133001.mail.ir2.yahoo.com> Message-ID: Hi On Thu, Mar 6, 2014 at 9:16 PM, Antony Mayi wrote: > Hi, > > my backend is sending chunked response (~100chunks, 100bytes each). it > seems varnish then sends it again using chunked encoding but only as one > big chunk. the app is sort of a json streaming system and the whole point > of sending the responses in chunks is for the client to be able to handle > each chunk asap once they are available. unfortunately varnish cumulates it > into (which means waits for) one big chunk so the whole advantage of early > client processing is gone. > > is there any way to control this in varnish - ie. some timeouts or buffer > sizes etc (not interested in piping)? > This would require streaming. Varnish 3.0 will wait for the whole object to arrive before forwarding it. Use Varnish 4.0 or dig out the streaming patches. Set do_stream = true. -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From smwood4 at gmail.com Fri Mar 7 09:07:19 2014 From: smwood4 at gmail.com (Stephen Wood) Date: Fri, 7 Mar 2014 01:07:19 -0800 Subject: Anyone use a %D in varnishncsa (like in Apache) In-Reply-To: References: Message-ID: It's listed as a valid format for varnishncsa according to here : %D Time taken to serve the request, in microseconds. On Thu, Mar 6, 2014 at 3:26 PM, Jennings, Raymond < raymond.jennings at nytimes.com> wrote: > I recently tired it from someone else telling me it works. > > The man pages for varnishncsa do not list it as a valid format option. > > I used for a day and it seems to work but occasionally I do get some > negative floating point response times. Does anyone know if this format is > "semi supported" ? > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Stephen Wood www.heystephenwood.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From manca.alessio at tiscali.it Mon Mar 10 14:36:54 2014 From: manca.alessio at tiscali.it (Alessio (Personale)) Date: Mon, 10 Mar 2014 15:36:54 +0100 Subject: support to install varnish on a VPS (Plesk on Centos) Message-ID: <004101cf3c6e$2ebad8e0$8c308aa0$@alessio@tiscali.it> Hello, could anyone help me in finding a good howto for the varnish installation? I tried with http://geroldm.com/2013/05/install-varnish-on-plesk-11-server/ but after installation there are no "/etc/sysconfig/varnish" and "/etc/varnish/default.vcl" files Is it because I'm on VPS? Many thanks if someone can help! Alessio -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.langhorn at digital.cabinet-office.gov.uk Mon Mar 10 15:05:07 2014 From: andrew.langhorn at digital.cabinet-office.gov.uk (Andrew Langhorn) Date: Mon, 10 Mar 2014 15:05:07 +0000 Subject: support to install varnish on a VPS (Plesk on Centos) In-Reply-To: <531dcea9.61be700a.7dcc.6354SMTPIN_ADDED_BROKEN@mx.google.com> References: <531dcea9.61be700a.7dcc.6354SMTPIN_ADDED_BROKEN@mx.google.com> Message-ID: > Is it because I'm on VPS? If you have full root access, then no - it should make no difference. Do you have full root access? The howto you've linked to is for Plesk installations - Plesk's a bit of a beast, so you might find something like the official docs (at varnish-cache.org) more useful. They're self-explanatory, and fairly easy to follow. On 10 March 2014 14:36, Alessio (Personale) wrote: > Hello, > > could anyone help me in finding a good howto for the varnish installation? > > I tried with > > http://geroldm.com/2013/05/install-varnish-on-plesk-11-server/ > > but after installation there are no "*/etc/sysconfig/varnish" and * "*/etc/varnish/default.vcl" > files * > > > > Is it because I'm on VPS? > > Many thanks if someone can help! > > Alessio > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Kind regards, Andrew Langhorn Web Operations Government Digital Service e: andrew.langhorn at digital.cabinet-office.gov.uk t: +44 (0)7810 737375 a: 6th Floor, Aviation House, 125 Kingsway, London, WC2B 6NH -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdmls at yahoo.com Mon Mar 10 15:03:04 2014 From: jdmls at yahoo.com (John Doe) Date: Mon, 10 Mar 2014 08:03:04 -0700 (PDT) Subject: support to install varnish on a VPS (Plesk on Centos) In-Reply-To: <004101cf3c6e$2ebad8e0$8c308aa0$@alessio@tiscali.it> References: <004101cf3c6e$2ebad8e0$8c308aa0$@alessio@tiscali.it> Message-ID: <1394463784.60197.YahooMailNeo@web126102.mail.ne1.yahoo.com> From: Alessio (Personale) > could anyone help me in finding a good howto for the > varnish installation? > I tried with > http://geroldm.com/2013/05/install-varnish-on-plesk-11-server/ > but after installation there are no ?/etc/sysconfig/varnish? > and ?/etc/varnish/default.vcl? files Which rpm did it install (if it did)...? ? # rpm -qa | grep varnish The howto you used is based on (the old) CentOS 5. Are you on CentOS 6 or still on CentOS 5...? If you are on 6, you should install the corresponding varnish repository: http://repo.varnish-cache.org/redhat/varnish-3.0/el6/noarch/varnish-release/varnish-release-3.0-1.el6.noarch.rpm JD From manca.alessio at tiscali.it Mon Mar 10 15:43:18 2014 From: manca.alessio at tiscali.it (Alessio (Personale)) Date: Mon, 10 Mar 2014 16:43:18 +0100 Subject: R: support to install varnish on a VPS (Plesk on Centos) In-Reply-To: References: <531dcea9.61be700a.7dcc.6354SMTPIN_ADDED_BROKEN@mx.google.com> Message-ID: <005201cf3c77$75eaf160$61c0d420$@alessio@tiscali.it> Thank you Andrew. I confirm you I have root access (I switch using ?su root? command). The howto I?ve found is good because I had the rollback way J Da: Andrew Langhorn [mailto:andrew.langhorn at digital.cabinet-office.gov.uk] Inviato: luned? 10 marzo 2014 16:05 A: Alessio (Personale) Cc: varnish-misc at varnish-cache.org Oggetto: Re: support to install varnish on a VPS (Plesk on Centos) > Is it because I?m on VPS? If you have full root access, then no - it should make no difference. Do you have full root access? The howto you've linked to is for Plesk installations - Plesk's a bit of a beast, so you might find something like the official docs (at varnish-cache.org) more useful. They're self-explanatory, and fairly easy to follow. On 10 March 2014 14:36, Alessio (Personale) wrote: Hello, could anyone help me in finding a good howto for the varnish installation? I tried with http://geroldm.com/2013/05/install-varnish-on-plesk-11-server/ but after installation there are no ?/etc/sysconfig/varnish? and ?/etc/varnish/default.vcl? files Is it because I?m on VPS? Many thanks if someone can help! Alessio _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Kind regards, Andrew Langhorn Web Operations Government Digital Service e: andrew.langhorn at digital.cabinet-office.gov.uk t: +44 (0)7810 737375 a: 6th Floor, Aviation House, 125 Kingsway, London, WC2B 6NH -------------- next part -------------- An HTML attachment was scrubbed... URL: From manca.alessio at tiscali.it Mon Mar 10 15:46:59 2014 From: manca.alessio at tiscali.it (Alessio (Personale)) Date: Mon, 10 Mar 2014 16:46:59 +0100 Subject: R: support to install varnish on a VPS (Plesk on Centos) In-Reply-To: <1394463784.60197.YahooMailNeo@web126102.mail.ne1.yahoo.com> References: <004101cf3c6e$2ebad8e0$8c308aa0$@alessio@tiscali.it> <1394463784.60197.YahooMailNeo@web126102.mail.ne1.yahoo.com> Message-ID: <005701cf3c77$f8e5ac90$eab105b0$@alessio@tiscali.it> Thanks "John". Using "rpm -qa | grep varnish" I find "varnish-release-3.0-1.noarch" this sould confirm it's installed. I confirm you I'm running on CentOS 6. -----Messaggio originale----- Da: varnish-misc-bounces+manca.alessio=tiscali.it at varnish-cache.org [mailto:varnish-misc-bounces+manca.alessio=tiscali.it at varnish-cache.org] Per conto di John Doe Inviato: luned? 10 marzo 2014 16:03 A: varnish-misc at varnish-cache.org Oggetto: Re: support to install varnish on a VPS (Plesk on Centos) From: Alessio (Personale) > could anyone help me in finding a good howto for the > varnish installation? > I tried with > http://geroldm.com/2013/05/install-varnish-on-plesk-11-server/ > but after installation there are no ?/etc/sysconfig/varnish? > and ?/etc/varnish/default.vcl? files Which rpm did it install (if it did)...? # rpm -qa | grep varnish The howto you used is based on (the old) CentOS 5. Are you on CentOS 6 or still on CentOS 5...? If you are on 6, you should install the corresponding varnish repository: http://repo.varnish-cache.org/redhat/varnish-3.0/el6/noarch/varnish-release/varnish-release-3.0-1.el6.noarch.rpm JD _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From jdmls at yahoo.com Mon Mar 10 16:09:22 2014 From: jdmls at yahoo.com (John Doe) Date: Mon, 10 Mar 2014 09:09:22 -0700 (PDT) Subject: R: support to install varnish on a VPS (Plesk on Centos) In-Reply-To: <005701cf3c77$f8e5ac90$eab105b0$@alessio@tiscali.it> References: <004101cf3c6e$2ebad8e0$8c308aa0$@alessio@tiscali.it> <1394463784.60197.YahooMailNeo@web126102.mail.ne1.yahoo.com> <005701cf3c77$f8e5ac90$eab105b0$@alessio@tiscali.it> Message-ID: <1394467762.94162.YahooMailNeo@web126104.mail.ne1.yahoo.com> From: Alessio (Personale) > conto di John Doe >> Which rpm did it install (if it did)...? >> ? # rpm -qa | grep varnish >> The howto you used is based on (the old) CentOS 5. >> Are you on CentOS 6 or still on CentOS 5...? >> If you are on 6, you should install the corresponding varnish repository: >> http://repo.varnish-cache.org/redhat/varnish-3.0/el6/noarch/varnish-release/varnish-release-3.0-1.el6.noarch.rpm > > Thanks "John". > Using "rpm -qa | grep varnish" > I find "varnish-release-3.0-1.noarch" this sould confirm it's > installed. > I confirm you I'm running on CentOS 6. Yes and no... it confirms a varnish repository is installed, but not which one (el5 or el6?)... If you found only "varnish-release", it means varnish (the application) is not installed. The "yum install varnish" from the howto did not work. That's why you cannot find the files in etc... Remove the varnish repository: ? # rpm -e varnish-release Install the correct one: ? # rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el6/noarch/varnish-release/varnish-release-3.0-1.el6.noarch.rpm Install the varnish application: ? # yum install varnish JD From manca.alessio at tiscali.it Tue Mar 11 09:15:48 2014 From: manca.alessio at tiscali.it (Alessio (Personale)) Date: Tue, 11 Mar 2014 10:15:48 +0100 Subject: R: R: support to install varnish on a VPS (Plesk on Centos) In-Reply-To: <1394467762.94162.YahooMailNeo@web126104.mail.ne1.yahoo.com> References: <004101cf3c6e$2ebad8e0$8c308aa0$@alessio@tiscali.it> <1394463784.60197.YahooMailNeo@web126102.mail.ne1.yahoo.com> <005701cf3c77$f8e5ac90$eab105b0$@alessio@tiscali.it> <1394467762.94162.YahooMailNeo@web126104.mail.ne1.yahoo.com> Message-ID: <000001cf3d0a$7dee4bd0$79cae370$@alessio@tiscali.it> Hi John, just removed and reinstalled with no errors or warnings. Unfortunately I still can't find any "/etc/sysconfig/varnish" file or directory. Many thanks for your support! Alessio -----Messaggio originale----- Da: varnish-misc-bounces+manca.alessio=tiscali.it at varnish-cache.org [mailto:varnish-misc-bounces+manca.alessio=tiscali.it at varnish-cache.org] Per conto di John Doe Inviato: luned? 10 marzo 2014 17:09 A: varnish-misc at varnish-cache.org Oggetto: Re: R: support to install varnish on a VPS (Plesk on Centos) From: Alessio (Personale) > conto di John Doe >> Which rpm did it install (if it did)...? >> ? # rpm -qa | grep varnish >> The howto you used is based on (the old) CentOS 5. >> Are you on CentOS 6 or still on CentOS 5...? >> If you are on 6, you should install the corresponding varnish repository: >> http://repo.varnish-cache.org/redhat/varnish-3.0/el6/noarch/varnish-release/ varnish-release-3.0-1.el6.noarch.rpm > > Thanks "John". > Using "rpm -qa | grep varnish" > I find "varnish-release-3.0-1.noarch" this sould confirm it's > installed. > I confirm you I'm running on CentOS 6. Yes and no... it confirms a varnish repository is installed, but not which one (el5 or el6?)... If you found only "varnish-release", it means varnish (the application) is not installed. The "yum install varnish" from the howto did not work. That's why you cannot find the files in etc... Remove the varnish repository: ? # rpm -e varnish-release Install the correct one: ? # rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el6/noarch/varnish-release/ varnish-release-3.0-1.el6.noarch.rpm Install the varnish application: ? # yum install varnish JD _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From antonymayi at yahoo.com Tue Mar 11 11:14:08 2014 From: antonymayi at yahoo.com (Antony Mayi) Date: Tue, 11 Mar 2014 11:14:08 +0000 (GMT) Subject: chunked response transfer encoding In-Reply-To: <1394536235.17723.YahooMailNeo@web133004.mail.ir2.yahoo.com> References: <1394137005.61529.YahooMailNeo@web133001.mail.ir2.yahoo.com> <1394536235.17723.YahooMailNeo@web133004.mail.ir2.yahoo.com> Message-ID: <1394536448.75884.YahooMailNeo@web133003.mail.ir2.yahoo.com> On Friday, 7 March 2014, 8:32, Per Buer wrote: >Hi? >> >> >> >>On Thu, Mar 6, 2014 at 9:16 PM, Antony Mayi??wrote: >> >>Hi, >>> >>> >>>my backend is sending chunked response (~100chunks, 100bytes each). it seems varnish then sends it again using chunked encoding but only as one big chunk. the app is sort of a json streaming system and the whole point of sending the responses in chunks is for the client to be able to handle each chunk asap once they are available. unfortunately varnish cumulates it into (which means waits for) one big chunk so the whole advantage of early client processing is gone. >>> >>> >>>is there any way to control this in varnish - ie. some timeouts or buffer sizes etc (not interested in piping)? >> >> >>This would require streaming. Varnish 3.0 will wait for the whole object to arrive before forwarding it. Use Varnish 4.0 or dig out the streaming patches. Set do_stream = true. >> >> > > >ok, thanks for explaining. I found this:?https://www.varnish-software.com/blog/streaming-varnish-30 which suggest it should work in Varnish 3.0. I am using the official?3.0.5 from varnish rpm repo. I am setting the do_stream to true but still getting one big chunk. so is the blogpost wrong or has it not been officially released as rpm yet? >thanks, >Antony. -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Tue Mar 11 11:39:12 2014 From: perbu at varnish-software.com (Per Buer) Date: Tue, 11 Mar 2014 12:39:12 +0100 Subject: chunked response transfer encoding In-Reply-To: <1394536448.75884.YahooMailNeo@web133003.mail.ir2.yahoo.com> References: <1394137005.61529.YahooMailNeo@web133001.mail.ir2.yahoo.com> <1394536235.17723.YahooMailNeo@web133004.mail.ir2.yahoo.com> <1394536448.75884.YahooMailNeo@web133003.mail.ir2.yahoo.com> Message-ID: Antony, On Tue, Mar 11, 2014 at 12:14 PM, Antony Mayi wrote: > ok, thanks for explaining. I found this: > https://www.varnish-software.com/blog/streaming-varnish-30 which suggest > it should work in Varnish 3.0. I am using the official 3.0.5 from varnish > rpm repo. I am setting the do_stream to true but still getting one big > chunk. so is the blogpost wrong or has it not been officially released as > rpm yet? > thanks, > Antony. > > As I said. You need to use Varnish 4.0 (which is currently in "tech preview") or dig out the streaming patches (which is a fair bit of work). 3.0.5 vanilla will not enable streaming. -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Paul.McInerney at faredge.com.au Thu Mar 13 00:25:03 2014 From: Paul.McInerney at faredge.com.au (Paul McInerney) Date: Thu, 13 Mar 2014 11:25:03 +1100 Subject: varnish 3.0 + mobile app In-Reply-To: References: Message-ID: Hi Per, Thank you for your quick response. I am currently using 'vary' on the deliver coming back from apache - see http://pastebin.com/AkwWrs62 <-- called as an include at start of default.vcl Looking at the response headers I can see 'Vary: User-Agent' set. Again, varnish is caching all variants but I still cannot purge. If I just execute ` curl -I -X PURGE 'www.site.com/?mode=test' `, it returns a MISS when all variants of that URL are being cached. I am still quite new to varnish so I am sorry if this seems like a novice mistake/misunderstanding. Cheers, Paul From: Per Buer [mailto:perbu at varnish-software.com] Sent: Friday, 7 March 2014 6:30 PM To: Paul McInerney Cc: varnish-misc at varnish-cache.org Subject: Re: varnish 3.0 + mobile app Hi Paul, On Thu, Mar 6, 2014 at 11:57 PM, Paul McInerney > wrote: Hi all, We are currently developing a mobile android/iphone app for our website. When the app hits the varnish frontend, it has the user-agent string set as 'mobileapp'. So I have put in place a subroutine that correctly identifies the app and sets a custom x-device header to send to the backend for rendering the page(s) in the required format. This part is working well with the same URL now serving both formats of the page, however when I issue a purge against the URL, it purges the cached versions of the PC version of the page, and doesn't clear all variants (which should include the mobileapp version afaik ) It's because you are doing it wrong. There is a very elegant solution to this; HTTP Vary. https://www.varnish-cache.org/docs/3.0/tutorial/vary.html You are trying to manipulate the hash, but it fails as you most likely forget to do the same manipulations when purging. This would not be a problem if you where to use Vary as all the variants would be connected to the same hash value. You can either Vary directly on the custom headers that you are creating or use it on the X-UA-Device Javier suggested. I like that. -- [http://www.varnish-software.com/static/media/logo-email.png] Per Buer CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 [https://www.varnish-software.com/sites/default/files/redherring_2013_winner_sml.jpg] -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at tengu.ch Thu Mar 13 08:30:44 2014 From: varnish at tengu.ch (=?UTF-8?B?Q8OpZHJpYyBKZWFubmVyZXQ=?=) Date: Thu, 13 Mar 2014 09:30:44 +0100 Subject: Varnishd stops sending logs to VSM after a while Message-ID: <53216CB4.1000609@tengu.ch> Hello, We have a small problem with 5 of our varnishes (on a total of 6?): it seems every morning, varnishd stops sending stuff to the Shared Memory[1], meaning we don't have any logs. The only thing I found in order to get logs back is to restart varnish, but of course this isn't the best way to solve the problem? Here are some information: Version: varnishd (varnish-3.0.2 revision 55e70a4) ** OS: Debian Squeeze ** System memory: 7468Mo ** CPU: dual E5645 @ 2.40GHz (Note: for those who knows about Amazon AWS instances, it's an m1.large, instance-store AMI.) ** Daemon options: DAEMON_OPTS="-n \ -u varnish -g varnish \ -a :80 \ -T localhost:6082 \ -s malloc,5G \ -f /etc/varnish/.vcl \ -S /path/to/secret \ -p shm_reclen=65535 \ " ** Tasks around the time the logs stop: logrotate for varnishncsa logs, with a varnishncsa restart. This shouldn't break varnishd log system, and it worked fine for months? We didn't detect any problem with memory nor disk I/O during the outage. This morning, it was the third time in a row we detected this issue. For what we know, neither the VCL nor daemon options were changed just before the problem appears (well, VCL was changed, some backend "routing" was updated, but nothing out of the ordinary stuff we do for months now). Symptoms: - /var/lib/varnish//_.vsm isn't updated - running varnishncsa or varnishlog from the shell doesn't show any log entries - restarting the varnishd service bring logs up again (we can see the flow if we keep the varnishncsa up) An "lsof -p " shows this line: varnishd 17948 varnish DEL REG 202,1 264837 /var/lib/varnish//_.vsm I'm not very comfortable with the "DEL" FD: when I do the same command once logs flow, I get: varnishd 22603 varnish mem REG 202,1 84934656 264058 /var/lib/varnish//_.vsm It seems "something" is degrading the shared memory? Any help would be welcome, I'm a bit stuck with the investigations right now :(. Cheers, C. [1] https://www.varnish-cache.org/docs/trunk/reference/vsm.html From perbu at varnish-software.com Thu Mar 13 08:49:03 2014 From: perbu at varnish-software.com (Per Buer) Date: Thu, 13 Mar 2014 09:49:03 +0100 Subject: varnish 3.0 + mobile app In-Reply-To: References: Message-ID: Hi Paul. Have you set up your VCL to handle PURGE requests as described in the documentation? There is no PURGE support in the VCL you posted. PURGE is not available our of the box. You need to add it through VCL. On Thu, Mar 13, 2014 at 1:25 AM, Paul McInerney < Paul.McInerney at faredge.com.au> wrote: > Hi Per, > > > > Thank you for your quick response. > > > > I am currently using 'vary' on the deliver coming back from apache - see > http://pastebin.com/AkwWrs62 ? called as an include at start of > default.vcl > > > > Looking at the response headers I can see 'Vary: User-Agent' set. Again, > varnish is caching all variants but I still cannot purge. > > > > If I just execute ` curl -I -X PURGE 'www.site.com/?mode=test' `, it > returns a MISS when all variants of that URL are being cached. > > > > I am still quite new to varnish so I am sorry if this seems like a novice > mistake/misunderstanding. > > > > Cheers, > > > > Paul > > > > *From:* Per Buer [mailto:perbu at varnish-software.com] > *Sent:* Friday, 7 March 2014 6:30 PM > > *To:* Paul McInerney > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: varnish 3.0 + mobile app > > > > Hi Paul, > > > > > > On Thu, Mar 6, 2014 at 11:57 PM, Paul McInerney < > Paul.McInerney at faredge.com.au> wrote: > > Hi all, > > > > We are currently developing a mobile android/iphone app for our website. > > When the app hits the varnish frontend, it has the user-agent string set > as 'mobileapp'. > > So I have put in place a subroutine that correctly identifies the app and > sets a > custom x-device header to send to the backend for rendering the page(s) > in the > > required format. > > This part is working well with the same URL now serving both formats of > the page, > > however when I issue a purge against the URL, it purges the cached > versions of the > > PC version of the page, and doesn't clear all variants (which should > include the mobileapp > > version afaik ) > > > > It's because you are doing it wrong. There is a very elegant solution to > this; HTTP Vary. > > > > https://www.varnish-cache.org/docs/3.0/tutorial/vary.html > > > > You are trying to manipulate the hash, but it fails as you most likely > forget to do the same manipulations when purging. This would not be a > problem if you where to use Vary as all the variants would be connected to > the same hash value. > > > > You can either Vary directly on the custom headers that you are creating > or use it on the X-UA-Device Javier suggested. I like that. > > > > > > > > -- > > > > *Per Buer* > CTO | Varnish Software > Phone: +47 958 39 117 | Skype: per.buer > > > *We Make Websites Fly!**Winner of the Red Herring Top 100 Global Award > 2013* > > > -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Thu Mar 13 09:14:45 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Thu, 13 Mar 2014 10:14:45 +0100 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: <53216CB4.1000609@tengu.ch> References: <53216CB4.1000609@tengu.ch> Message-ID: Hi, On Thu, Mar 13, 2014 at 9:30 AM, C?dric Jeanneret wrote: > Hello, > > We have a small problem with 5 of our varnishes (on a total of 6?): > it seems every morning, varnishd stops sending stuff to the Shared > Memory[1], meaning we don't have any logs. > > The only thing I found in order to get logs back is to restart varnish, > but of course this isn't the best way to solve the problem? > > Here are some information: > > Version: varnishd (varnish-3.0.2 revision 55e70a4) > ** OS: Debian Squeeze > ** System memory: 7468Mo > ** CPU: dual E5645 @ 2.40GHz (Note: for those who knows about Amazon AWS > instances, it's an m1.large, instance-store AMI.) > ** Daemon options: > DAEMON_OPTS="-n \ > -u varnish -g varnish \ > -a :80 \ > -T localhost:6082 \ > -s malloc,5G \ > -f /etc/varnish/.vcl \ > -S /path/to/secret \ > -p shm_reclen=65535 \ > " > > ** Tasks around the time the logs stop: logrotate for varnishncsa logs, > with a varnishncsa restart. This shouldn't break varnishd log system, > and it worked fine for months? > > We didn't detect any problem with memory nor disk I/O during the outage. > This morning, it was the third time in a row we detected this issue. > > For what we know, neither the VCL nor daemon options were changed just > before the problem appears (well, VCL was changed, some backend > "routing" was updated, but nothing out of the ordinary stuff we do for > months now). > > Symptoms: > - /var/lib/varnish//_.vsm isn't updated > - running varnishncsa or varnishlog from the shell doesn't show any log > entries > - restarting the varnishd service bring logs up again (we can see the > flow if we keep the varnishncsa up) > > An "lsof -p " shows this line: > varnishd 17948 varnish DEL REG 202,1 264837 > /var/lib/varnish//_.vsm I suspect something "unrelated" deleted the log file, but varnishd is probably still logging as usual. Because it still holds a file descriptor of the log file. It's the log-reading tools on the other hand that won't see the file on the filesystem once it's deleted (unless they already hold a fd). That's why it feels like varnishd is not logging anymore. Dridi > I'm not very comfortable with the "DEL" FD: when I do the same command > once logs flow, I get: > varnishd 22603 varnish mem REG 202,1 84934656 264058 > /var/lib/varnish//_.vsm > > It seems "something" is degrading the shared memory? > > Any help would be welcome, I'm a bit stuck with the investigations right > now :(. > > Cheers, > > C. > > > [1] https://www.varnish-cache.org/docs/trunk/reference/vsm.html > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From varnish at tengu.ch Thu Mar 13 09:25:52 2014 From: varnish at tengu.ch (=?UTF-8?B?Q8OpZHJpYyBKZWFubmVyZXQ=?=) Date: Thu, 13 Mar 2014 10:25:52 +0100 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: References: <53216CB4.1000609@tengu.ch> Message-ID: <532179A0.4080502@tengu.ch> Hello, On 03/13/2014 10:14 AM, Dridi Boukelmoune wrote: > Hi, > > On Thu, Mar 13, 2014 at 9:30 AM, C?dric Jeanneret wrote: >> Hello, >> >> We have a small problem with 5 of our varnishes (on a total of 6?): >> it seems every morning, varnishd stops sending stuff to the Shared >> Memory[1], meaning we don't have any logs. >> >> The only thing I found in order to get logs back is to restart varnish, >> but of course this isn't the best way to solve the problem? >> >> Here are some information: >> >> Version: varnishd (varnish-3.0.2 revision 55e70a4) >> ** OS: Debian Squeeze >> ** System memory: 7468Mo >> ** CPU: dual E5645 @ 2.40GHz (Note: for those who knows about Amazon AWS >> instances, it's an m1.large, instance-store AMI.) >> ** Daemon options: >> DAEMON_OPTS="-n \ >> -u varnish -g varnish \ >> -a :80 \ >> -T localhost:6082 \ >> -s malloc,5G \ >> -f /etc/varnish/.vcl \ >> -S /path/to/secret \ >> -p shm_reclen=65535 \ >> " >> >> ** Tasks around the time the logs stop: logrotate for varnishncsa logs, >> with a varnishncsa restart. This shouldn't break varnishd log system, >> and it worked fine for months? >> >> We didn't detect any problem with memory nor disk I/O during the outage. >> This morning, it was the third time in a row we detected this issue. >> >> For what we know, neither the VCL nor daemon options were changed just >> before the problem appears (well, VCL was changed, some backend >> "routing" was updated, but nothing out of the ordinary stuff we do for >> months now). >> >> Symptoms: >> - /var/lib/varnish//_.vsm isn't updated >> - running varnishncsa or varnishlog from the shell doesn't show any log >> entries >> - restarting the varnishd service bring logs up again (we can see the >> flow if we keep the varnishncsa up) >> >> An "lsof -p " shows this line: >> varnishd 17948 varnish DEL REG 202,1 264837 >> /var/lib/varnish//_.vsm > > I suspect something "unrelated" deleted the log file, but varnishd is > probably still logging as usual. Because it still holds a file > descriptor of the log file. > > It's the log-reading tools on the other hand that won't see the file > on the filesystem once it's deleted (unless they already hold a fd). > That's why it feels like varnishd is not logging anymore. > > Dridi Hmm, _.vsm is shown when we perform a simple "ls" in the directory? Else, I would get some errors from either varnishncsa or varnishlog when I start them (I tried that this morning, just to see what would happen)? More over, I saw a small garbage being pushed to the VSM, some unreadable stuff ending with a FREE. It wasn't a "normal" line as we can see in the VSM, it was shorter (like 3-4 garbage,then FREE)? At the same time, nothing appeared in varnishncsa nor log. In order to debug that, I also run a curl on localhost, one request per second hitting the varnish (with correct headers and so on so that it falls on some existing URL). Result: - content is served - no log in varnishncsa - no log in varnishlog - nothing in VSM file Cheers, C. > >> I'm not very comfortable with the "DEL" FD: when I do the same command >> once logs flow, I get: >> varnishd 22603 varnish mem REG 202,1 84934656 264058 >> /var/lib/varnish//_.vsm >> >> It seems "something" is degrading the shared memory? >> >> Any help would be welcome, I'm a bit stuck with the investigations right >> now :(. >> >> Cheers, >> >> C. >> >> >> [1] https://www.varnish-cache.org/docs/trunk/reference/vsm.html >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From dridi.boukelmoune at zenika.com Thu Mar 13 10:20:39 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Thu, 13 Mar 2014 11:20:39 +0100 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: <532179A0.4080502@tengu.ch> References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> Message-ID: On Thu, Mar 13, 2014 at 10:25 AM, C?dric Jeanneret wrote: > Hello, > > On 03/13/2014 10:14 AM, Dridi Boukelmoune wrote: >> Hi, >> >> On Thu, Mar 13, 2014 at 9:30 AM, C?dric Jeanneret wrote: >>> Hello, >>> >>> We have a small problem with 5 of our varnishes (on a total of 6?): >>> it seems every morning, varnishd stops sending stuff to the Shared >>> Memory[1], meaning we don't have any logs. >>> >>> The only thing I found in order to get logs back is to restart varnish, >>> but of course this isn't the best way to solve the problem? >>> >>> Here are some information: >>> >>> Version: varnishd (varnish-3.0.2 revision 55e70a4) >>> ** OS: Debian Squeeze >>> ** System memory: 7468Mo >>> ** CPU: dual E5645 @ 2.40GHz (Note: for those who knows about Amazon AWS >>> instances, it's an m1.large, instance-store AMI.) >>> ** Daemon options: >>> DAEMON_OPTS="-n \ >>> -u varnish -g varnish \ >>> -a :80 \ >>> -T localhost:6082 \ >>> -s malloc,5G \ >>> -f /etc/varnish/.vcl \ >>> -S /path/to/secret \ >>> -p shm_reclen=65535 \ >>> " >>> >>> ** Tasks around the time the logs stop: logrotate for varnishncsa logs, >>> with a varnishncsa restart. This shouldn't break varnishd log system, >>> and it worked fine for months? >>> >>> We didn't detect any problem with memory nor disk I/O during the outage. >>> This morning, it was the third time in a row we detected this issue. >>> >>> For what we know, neither the VCL nor daemon options were changed just >>> before the problem appears (well, VCL was changed, some backend >>> "routing" was updated, but nothing out of the ordinary stuff we do for >>> months now). >>> >>> Symptoms: >>> - /var/lib/varnish//_.vsm isn't updated >>> - running varnishncsa or varnishlog from the shell doesn't show any log >>> entries >>> - restarting the varnishd service bring logs up again (we can see the >>> flow if we keep the varnishncsa up) >>> >>> An "lsof -p " shows this line: >>> varnishd 17948 varnish DEL REG 202,1 264837 >>> /var/lib/varnish//_.vsm >> >> I suspect something "unrelated" deleted the log file, but varnishd is >> probably still logging as usual. Because it still holds a file >> descriptor of the log file. >> >> It's the log-reading tools on the other hand that won't see the file >> on the filesystem once it's deleted (unless they already hold a fd). >> That's why it feels like varnishd is not logging anymore. >> >> Dridi > > Hmm, _.vsm is shown when we perform a simple "ls" in the directory? > Else, I would get some errors from either varnishncsa or varnishlog when > I start them (I tried that this morning, just to see what would happen)? If it's visible on your filesystem, then it got deleted and a new one was created, and varnishd is still using the deleted one. Dridi > More over, I saw a small garbage being pushed to the VSM, some > unreadable stuff ending with a FREE. It wasn't a "normal" line as we can > see in the VSM, it was shorter (like 3-4 garbage,then FREE)? > At the same time, nothing appeared in varnishncsa nor log. > > In order to debug that, I also run a curl on localhost, one request per > second hitting the varnish (with correct headers and so on so that it > falls on some existing URL). Result: > - content is served > - no log in varnishncsa > - no log in varnishlog > - nothing in VSM file > > Cheers, > > C. > >> >>> I'm not very comfortable with the "DEL" FD: when I do the same command >>> once logs flow, I get: >>> varnishd 22603 varnish mem REG 202,1 84934656 264058 >>> /var/lib/varnish//_.vsm >>> >>> It seems "something" is degrading the shared memory? >>> >>> Any help would be welcome, I'm a bit stuck with the investigations right >>> now :(. >>> >>> Cheers, >>> >>> C. >>> >>> >>> [1] https://www.varnish-cache.org/docs/trunk/reference/vsm.html >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From thomas.lecomte at virtual-expo.com Thu Mar 13 10:25:33 2014 From: thomas.lecomte at virtual-expo.com (Thomas Lecomte) Date: Thu, 13 Mar 2014 11:25:33 +0100 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: <532179A0.4080502@tengu.ch> References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> Message-ID: <20140313102533.GA5505@wks140.directindustry.com> On Thu, Mar 13, 2014 at 10:25:52AM +0100, C?dric Jeanneret wrote: > > Hmm, _.vsm is shown when we perform a simple "ls" in the directory? > Else, I would get some errors from either varnishncsa or varnishlog when > I start them (I tried that this morning, just to see what would happen)? Maybe you could compare the inodes using ls -i on the visible _.vsm against the one shown by lsof on the varnish process. -- Thomas Lecomte / +33 4 86 13 48 65 Sysadmin / Virtual Expo / Marseille From raymond.jennings at nytimes.com Thu Mar 13 11:33:03 2014 From: raymond.jennings at nytimes.com (Raymond Jennings) Date: Thu, 13 Mar 2014 07:33:03 -0400 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: <20140313102533.GA5505@wks140.directindustry.com> References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> Message-ID: <8673798013017974179@unknownmsgid> I have what might be a related problem. I can only get varnishlog to write to a file after I stop it and restart it. I am running on ec2. I tried various tricks to wait so many seconds after varnished starts then start varnishlog. Varnishncsa runs perfectly fine right out of the gate. I have had this problem for about two years now. Varnishlog is clearly running and "service varnishlog stop" successfully stops it. "service varnishlog start" and things are good. > On Mar 13, 2014, at 6:27 AM, Thomas Lecomte wrote: > >> On Thu, Mar 13, 2014 at 10:25:52AM +0100, C?dric Jeanneret wrote: >> >> Hmm, _.vsm is shown when we perform a simple "ls" in the directory... >> Else, I would get some errors from either varnishncsa or varnishlog when >> I start them (I tried that this morning, just to see what would happen)... > > Maybe you could compare the inodes using ls -i on the visible _.vsm > against the one shown by lsof on the varnish process. > > -- > Thomas Lecomte / +33 4 86 13 48 65 > Sysadmin / Virtual Expo / Marseille > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From ruben at varnish-software.com Thu Mar 13 14:11:39 2014 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Thu, 13 Mar 2014 15:11:39 +0100 Subject: Want to host VUG9 @ US West Coast in May? Message-ID: Hello varnishers, We are planning the next Varnish User Group (VUG) meeting and are making progress on that. Right now, we need your help as we hope to see this happening in the San Francisco Bay Area or somewhere else in the US West Coast (Portland or Seattle, anyone?). In order to do this we need a company willing to host us. If you are a user of our software and would like to contribute something back to the project, helping organize and host the Varnish User Day is one of the best possible ways to do this (and profile your company at the same time). ***So, please consider hosting our meeting (one day, 80-100 varnishers, food+drinks) and get in touch with me for further details. We are hoping to make this happen the last week of May or during June.*** If you can/want to host us, but you are not in the US West Cost, don't worry our VUGs are a community gathering and we hope to take them global if someone is willing to host us,so get in touch anyway and we will consider your proposal for future meetings. Thanks in advance. I am looking forward to hearing back from you all! Best regards, -- *Rub?n Romero* Community & Sales | Varnish Software AS Cell: +47 95964088 / Office: +47 21989260 Skype, Twitter & IRC: ruben_varnish We Make Websites Fly!Winner of the 2013 Red Herring Top 100 Global Awards -------------- next part -------------- An HTML attachment was scrubbed... URL: From Paul.McInerney at faredge.com.au Thu Mar 13 14:19:26 2014 From: Paul.McInerney at faredge.com.au (Paul McInerney) Date: Fri, 14 Mar 2014 01:19:26 +1100 Subject: varnish 3.0 + mobile app Message-ID: <7ayw7isqjdvgbearji5311pt.1394720355091@email.android.com> Hi Per, Yes I have purge already working with varnish. I can purge all other variants of the url (as well as other objects for other sites). It's just with this particular site, and more specifically for the same url which is rendered differently based on the user-agent string. So I can purge: www.site.com or www.site.com/node etc... but the url www.site.com/?mode=test (which is displayed differently depending on agent) I cannot clear from cache. This is what I am trying to hash to as to be able to clear the mobile app version of that url or the pc version, or both. Hope that majes sense. :) Paul McInerney -------- Original message -------- From: Per Buer Date: To: Paul McInerney Cc: varnish-misc at varnish-cache.org Subject: Re: varnish 3.0 + mobile app Hi Paul. Have you set up your VCL to handle PURGE requests as described in the documentation? There is no PURGE support in the VCL you posted. PURGE is not available our of the box. You need to add it through VCL. On Thu, Mar 13, 2014 at 1:25 AM, Paul McInerney > wrote: Hi Per, Thank you for your quick response. I am currently using ?vary? on the deliver coming back from apache ? see http://pastebin.com/AkwWrs62 <-- called as an include at start of default.vcl Looking at the response headers I can see ?Vary: User-Agent? set. Again, varnish is caching all variants but I still cannot purge. If I just execute ` curl -I -X PURGE ?www.site.com/?mode=test? `, it returns a MISS when all variants of that URL are being cached. I am still quite new to varnish so I am sorry if this seems like a novice mistake/misunderstanding. Cheers, Paul From: Per Buer [mailto:perbu at varnish-software.com] Sent: Friday, 7 March 2014 6:30 PM To: Paul McInerney Cc: varnish-misc at varnish-cache.org Subject: Re: varnish 3.0 + mobile app Hi Paul, On Thu, Mar 6, 2014 at 11:57 PM, Paul McInerney > wrote: Hi all, We are currently developing a mobile android/iphone app for our website. When the app hits the varnish frontend, it has the user-agent string set as 'mobileapp'. So I have put in place a subroutine that correctly identifies the app and sets a custom x-device header to send to the backend for rendering the page(s) in the required format. This part is working well with the same URL now serving both formats of the page, however when I issue a purge against the URL, it purges the cached versions of the PC version of the page, and doesn't clear all variants (which should include the mobileapp version afaik ) It's because you are doing it wrong. There is a very elegant solution to this; HTTP Vary. https://www.varnish-cache.org/docs/3.0/tutorial/vary.html You are trying to manipulate the hash, but it fails as you most likely forget to do the same manipulations when purging. This would not be a problem if you where to use Vary as all the variants would be connected to the same hash value. You can either Vary directly on the custom headers that you are creating or use it on the X-UA-Device Javier suggested. I like that. -- [http://www.varnish-software.com/static/media/logo-email.png] Per Buer CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 [https://www.varnish-software.com/sites/default/files/redherring_2013_winner_sml.jpg] -- [http://www.varnish-software.com/static/media/logo-email.png] Per Buer CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 [https://www.varnish-software.com/sites/default/files/redherring_2013_winner_sml.jpg] -------------- next part -------------- An HTML attachment was scrubbed... URL: From smwood4 at gmail.com Fri Mar 14 21:01:45 2014 From: smwood4 at gmail.com (Stephen Wood) Date: Fri, 14 Mar 2014 14:01:45 -0700 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: <8673798013017974179@unknownmsgid> References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> <8673798013017974179@unknownmsgid> Message-ID: My first thought is that logrotate is running and varishncsa is not properly catching a SIGHUP after a rotate. Therefore the logfile gets rotated but varnishncsa continues writing to the old fd. If you run varnishncsa by itself on the command line during these periods, do you get output? If so then it's probably not a problem with shared memory. If you want to test this, you can simply change the varnishncsa logrotate behavior to use copytruncate and not bother sending a HUP, which will truncate the log without rotating it. For reference here's what my logrotate file looks like: /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { size 1G rotate 7 missingok compress delaycompress postrotate for service in varnishlog varnishncsa; do if /usr/bin/pgrep -P 1 $service >/dev/null; then service $service reload > /dev/null fi done endscript } If you want to try copytruncate: /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { size 1G rotate 7 missingok compress delaycompress copytruncate } Note that you'll get a message in your log file that states it was truncated. It may mess up your log parsing software. On Thu, Mar 13, 2014 at 4:33 AM, Raymond Jennings < raymond.jennings at nytimes.com> wrote: > I have what might be a related problem. I can only get varnishlog to > write to a file after I stop it and restart it. I am running on ec2. > I tried various tricks to wait so many seconds after varnished starts > then start varnishlog. Varnishncsa runs perfectly fine right out of > the gate. I have had this problem for about two years now. > Varnishlog is clearly running and "service varnishlog stop" > successfully stops it. "service varnishlog start" and things are > good. > > > On Mar 13, 2014, at 6:27 AM, Thomas Lecomte < > thomas.lecomte at virtual-expo.com> wrote: > > > >> On Thu, Mar 13, 2014 at 10:25:52AM +0100, C?dric Jeanneret wrote: > >> > >> Hmm, _.vsm is shown when we perform a simple "ls" in the directory... > >> Else, I would get some errors from either varnishncsa or varnishlog when > >> I start them (I tried that this morning, just to see what would > happen)... > > > > Maybe you could compare the inodes using ls -i on the visible _.vsm > > against the one shown by lsof on the varnish process. > > > > -- > > Thomas Lecomte / +33 4 86 13 48 65 > > Sysadmin / Virtual Expo / Marseille > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Stephen Wood www.heystephenwood.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.jennings at nytimes.com Fri Mar 14 21:17:13 2014 From: raymond.jennings at nytimes.com (Raymond Jennings) Date: Fri, 14 Mar 2014 17:17:13 -0400 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> <8673798013017974179@unknownmsgid> Message-ID: <544618721658714945@unknownmsgid> I don't believe this a log rotate problem as I gave logrotate scheduled to run at midnight. I think this might be related to how amazon ec2 instances sometimes change their hostnames - after varnish starts - and if I recall the various varnish binaries make use of the hostname to find the cache (file or memory.) Might be this but then why does varnishd and varnishd work okay and only logrotate does not? On Mar 14, 2014, at 5:02 PM, Stephen Wood wrote: My first thought is that logrotate is running and varishncsa is not properly catching a SIGHUP after a rotate. Therefore the logfile gets rotated but varnishncsa continues writing to the old fd. If you run varnishncsa by itself on the command line during these periods, do you get output? If so then it's probably not a problem with shared memory. If you want to test this, you can simply change the varnishncsa logrotate behavior to use copytruncate and not bother sending a HUP, which will truncate the log without rotating it. For reference here's what my logrotate file looks like: /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { size 1G rotate 7 missingok compress delaycompress postrotate for service in varnishlog varnishncsa; do if /usr/bin/pgrep -P 1 $service >/dev/null; then service $service reload > /dev/null fi done endscript } If you want to try copytruncate: /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { size 1G rotate 7 missingok compress delaycompress copytruncate } Note that you'll get a message in your log file that states it was truncated. It may mess up your log parsing software. On Thu, Mar 13, 2014 at 4:33 AM, Raymond Jennings < raymond.jennings at nytimes.com> wrote: > I have what might be a related problem. I can only get varnishlog to > write to a file after I stop it and restart it. I am running on ec2. > I tried various tricks to wait so many seconds after varnished starts > then start varnishlog. Varnishncsa runs perfectly fine right out of > the gate. I have had this problem for about two years now. > Varnishlog is clearly running and "service varnishlog stop" > successfully stops it. "service varnishlog start" and things are > good. > > > On Mar 13, 2014, at 6:27 AM, Thomas Lecomte < > thomas.lecomte at virtual-expo.com> wrote: > > > >> On Thu, Mar 13, 2014 at 10:25:52AM +0100, C?dric Jeanneret wrote: > >> > >> Hmm, _.vsm is shown when we perform a simple "ls" in the directory... > >> Else, I would get some errors from either varnishncsa or varnishlog when > >> I start them (I tried that this morning, just to see what would > happen)... > > > > Maybe you could compare the inodes using ls -i on the visible _.vsm > > against the one shown by lsof on the varnish process. > > > > -- > > Thomas Lecomte / +33 4 86 13 48 65 > > Sysadmin / Virtual Expo / Marseille > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Stephen Wood www.heystephenwood.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From smwood4 at gmail.com Fri Mar 14 22:02:05 2014 From: smwood4 at gmail.com (Stephen Wood) Date: Fri, 14 Mar 2014 15:02:05 -0700 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: <544618721658714945@unknownmsgid> References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> <8673798013017974179@unknownmsgid> <544618721658714945@unknownmsgid> Message-ID: That's possible, but EC2 instances should only change their hostnames after a physical reboot or after being shut-off and turned on. This has been my experience at least. I believe the default work dir for varnishd is /var/lib/varnish/, so if you have a changing hostname that could potentially cause issues. I'm not sure how that would affect varnishncsa and varnishlog. You can hard-code this directory using the -n option when starting varnishd. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.jennings at nytimes.com Fri Mar 14 22:05:04 2014 From: raymond.jennings at nytimes.com (Raymond Jennings) Date: Fri, 14 Mar 2014 18:05:04 -0400 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> <8673798013017974179@unknownmsgid> <544618721658714945@unknownmsgid> Message-ID: <7274914769920241645@unknownmsgid> We have "another layer" on top of ec2 so it is a little more complicated to explain but the -n option does sound useful. Thanks. On Mar 14, 2014, at 6:02 PM, Stephen Wood wrote: That's possible, but EC2 instances should only change their hostnames after a physical reboot or after being shut-off and turned on. This has been my experience at least. I believe the default work dir for varnishd is /var/lib/varnish/, so if you have a changing hostname that could potentially cause issues. I'm not sure how that would affect varnishncsa and varnishlog. You can hard-code this directory using the -n option when starting varnishd. -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at tengu.ch Mon Mar 17 06:51:34 2014 From: varnish at tengu.ch (=?UTF-8?B?Q8OpZHJpYyBKZWFubmVyZXQ=?=) Date: Mon, 17 Mar 2014 07:51:34 +0100 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> <8673798013017974179@unknownmsgid> Message-ID: <53269B76.3090503@tengu.ch> Hello, Sorry for the delay, but? So, some more information: Logrotate: this shouldn't be a problem, as logrotate doesn't touch the _.vsm file? Varnishncsa/Varnishlog: I don't think the problem is on their side either, as they perform read-only access to the _.vsm (tell me if I'm wrong with this statement) VSM: the four varnishes had the very same problem this weekend (of course, when nobody's here?). I'm keeping one of them at hand with the problem "enabled" in order to be able to debug a bit. Here are what I can tell: Inodes: indeed, the inodes aren't the same for the one used by the process (marked as DEL in lsof, inode 264058) and the one on the system (268725). Creation date of the system one: March 16, 2014, 16:10. After a quick look in the logs, it seems there's a configuration modification at this very time: some filters were added, calling a reload of the VCL. Please see bellow for the "hot-reload" script content. Is there anything weird in this script? Thank you in advance. Cheers, C. #!/bin/bash # Reload a varnish config # Author: Kristian Lyngstol # Original version: http://kly.no/posts/2009_02_18__Easy_reloading_of_varnish__VCL__.html if [ $# -lt 1 -o $# -gt 2 ]; then echo "Usage: $0 vcl_file [secret_file]" exit 1 fi FILE=$1 if [ $# -eq 2 ]; then SECRET_OPT="-S $2" fi # Hostname and management port # (defined in /etc/default/varnish or on startup) HOSTPORT="localhost:6082" NOW=`date +%F_%T` error() { echo 1>&2 "Failed to reload $FILE." exit 1 } echo "@@@ Checking VCL file syntax:" varnishd -d -s malloc -f $FILE < /dev/null || error echo -e "\n@@@ Loading new VCL file:" varnishadm $SECRET_OPT -T $HOSTPORT vcl.load reload$NOW $FILE || error varnishadm $SECRET_OPT -T $HOSTPORT vcl.use reload$NOW || error echo -e "\n@@@ Currently available VCL configs:" varnishadm $SECRET_OPT -T $HOSTPORT vcl.list exit 0 On 03/14/2014 10:01 PM, Stephen Wood wrote: > My first thought is that logrotate is running and varishncsa is not > properly catching a SIGHUP after a rotate. Therefore the logfile gets > rotated but varnishncsa continues writing to the old fd. > > If you run varnishncsa by itself on the command line during these periods, > do you get output? If so then it's probably not a problem with shared > memory. > > If you want to test this, you can simply change the varnishncsa logrotate > behavior to use copytruncate and not bother sending a HUP, which will > truncate the log without rotating it. > > For reference here's what my logrotate file looks like: > > /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { > size 1G > rotate 7 > missingok > compress > delaycompress > postrotate > for service in varnishlog varnishncsa; do > if /usr/bin/pgrep -P 1 $service >/dev/null; then > service $service reload > /dev/null > fi > done > endscript > } > > If you want to try copytruncate: > > /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { > size 1G > rotate 7 > missingok > compress > delaycompress > copytruncate > } > > Note that you'll get a message in your log file that states it was > truncated. It may mess up your log parsing software. > > > On Thu, Mar 13, 2014 at 4:33 AM, Raymond Jennings < > raymond.jennings at nytimes.com> wrote: > >> I have what might be a related problem. I can only get varnishlog to >> write to a file after I stop it and restart it. I am running on ec2. >> I tried various tricks to wait so many seconds after varnished starts >> then start varnishlog. Varnishncsa runs perfectly fine right out of >> the gate. I have had this problem for about two years now. >> Varnishlog is clearly running and "service varnishlog stop" >> successfully stops it. "service varnishlog start" and things are >> good. >> >>> On Mar 13, 2014, at 6:27 AM, Thomas Lecomte < >> thomas.lecomte at virtual-expo.com> wrote: >>> >>>> On Thu, Mar 13, 2014 at 10:25:52AM +0100, C?dric Jeanneret wrote: >>>> >>>> Hmm, _.vsm is shown when we perform a simple "ls" in the directory... >>>> Else, I would get some errors from either varnishncsa or varnishlog when >>>> I start them (I tried that this morning, just to see what would >> happen)... >>> >>> Maybe you could compare the inodes using ls -i on the visible _.vsm >>> against the one shown by lsof on the varnish process. >>> >>> -- >>> Thomas Lecomte / +33 4 86 13 48 65 >>> Sysadmin / Virtual Expo / Marseille >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > From dridi.boukelmoune at zenika.com Mon Mar 17 07:46:24 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Mon, 17 Mar 2014 08:46:24 +0100 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: <53269B76.3090503@tengu.ch> References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> <8673798013017974179@unknownmsgid> <53269B76.3090503@tengu.ch> Message-ID: Hi, On Mon, Mar 17, 2014 at 7:51 AM, C?dric Jeanneret wrote: > Hello, > > Sorry for the delay, but? > > So, some more information: > > Logrotate: > this shouldn't be a problem, as logrotate doesn't touch the _.vsm file? > > Varnishncsa/Varnishlog: > I don't think the problem is on their side either, as they perform > read-only access to the _.vsm (tell me if I'm wrong with this statement) > > VSM: > the four varnishes had the very same problem this weekend (of course, > when nobody's here?). I'm keeping one of them at hand with the problem > "enabled" in order to be able to debug a bit. > > Here are what I can tell: > > Inodes: > indeed, the inodes aren't the same for the one used by the process > (marked as DEL in lsof, inode 264058) and the one on the system (268725). > Creation date of the system one: March 16, 2014, 16:10. > > After a quick look in the logs, it seems there's a configuration > modification at this very time: some filters were added, calling a > reload of the VCL. > > Please see bellow for the "hot-reload" script content. > > Is there anything weird in this script? > > Thank you in advance. > > Cheers, > > C. > > > > #!/bin/bash > # Reload a varnish config > # Author: Kristian Lyngstol > > # Original version: > http://kly.no/posts/2009_02_18__Easy_reloading_of_varnish__VCL__.html > > if [ $# -lt 1 -o $# -gt 2 ]; then > echo "Usage: $0 vcl_file [secret_file]" > exit 1 > fi > FILE=$1 > > if [ $# -eq 2 ]; then > SECRET_OPT="-S $2" > fi > > # Hostname and management port > # (defined in /etc/default/varnish or on startup) > HOSTPORT="localhost:6082" > NOW=`date +%F_%T` > > error() > { > echo 1>&2 "Failed to reload $FILE." > exit 1 > } > echo "@@@ Checking VCL file syntax:" > varnishd -d -s malloc -f $FILE < /dev/null || error Maybe this varnishd instance you bring up deletes the file and creates its own. Btw, why would you need a storage for a simple syntax check ? And isn't your redirection from /dev/null in the wrong direction ? Dridi > echo -e "\n@@@ Loading new VCL file:" > varnishadm $SECRET_OPT -T $HOSTPORT vcl.load reload$NOW $FILE || error > varnishadm $SECRET_OPT -T $HOSTPORT vcl.use reload$NOW || error > > > echo -e "\n@@@ Currently available VCL configs:" > varnishadm $SECRET_OPT -T $HOSTPORT vcl.list > > exit 0 > > > On 03/14/2014 10:01 PM, Stephen Wood wrote: >> My first thought is that logrotate is running and varishncsa is not >> properly catching a SIGHUP after a rotate. Therefore the logfile gets >> rotated but varnishncsa continues writing to the old fd. >> >> If you run varnishncsa by itself on the command line during these periods, >> do you get output? If so then it's probably not a problem with shared >> memory. >> >> If you want to test this, you can simply change the varnishncsa logrotate >> behavior to use copytruncate and not bother sending a HUP, which will >> truncate the log without rotating it. >> >> For reference here's what my logrotate file looks like: >> >> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >> size 1G >> rotate 7 >> missingok >> compress >> delaycompress >> postrotate >> for service in varnishlog varnishncsa; do >> if /usr/bin/pgrep -P 1 $service >/dev/null; then >> service $service reload > /dev/null >> fi >> done >> endscript >> } >> >> If you want to try copytruncate: >> >> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >> size 1G >> rotate 7 >> missingok >> compress >> delaycompress >> copytruncate >> } >> >> Note that you'll get a message in your log file that states it was >> truncated. It may mess up your log parsing software. >> >> >> On Thu, Mar 13, 2014 at 4:33 AM, Raymond Jennings < >> raymond.jennings at nytimes.com> wrote: >> >>> I have what might be a related problem. I can only get varnishlog to >>> write to a file after I stop it and restart it. I am running on ec2. >>> I tried various tricks to wait so many seconds after varnished starts >>> then start varnishlog. Varnishncsa runs perfectly fine right out of >>> the gate. I have had this problem for about two years now. >>> Varnishlog is clearly running and "service varnishlog stop" >>> successfully stops it. "service varnishlog start" and things are >>> good. >>> >>>> On Mar 13, 2014, at 6:27 AM, Thomas Lecomte < >>> thomas.lecomte at virtual-expo.com> wrote: >>>> >>>>> On Thu, Mar 13, 2014 at 10:25:52AM +0100, C?dric Jeanneret wrote: >>>>> >>>>> Hmm, _.vsm is shown when we perform a simple "ls" in the directory... >>>>> Else, I would get some errors from either varnishncsa or varnishlog when >>>>> I start them (I tried that this morning, just to see what would >>> happen)... >>>> >>>> Maybe you could compare the inodes using ls -i on the visible _.vsm >>>> against the one shown by lsof on the varnish process. >>>> >>>> -- >>>> Thomas Lecomte / +33 4 86 13 48 65 >>>> Sysadmin / Virtual Expo / Marseille >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From dridi.boukelmoune at zenika.com Mon Mar 17 07:49:38 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Mon, 17 Mar 2014 08:49:38 +0100 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> <8673798013017974179@unknownmsgid> <53269B76.3090503@tengu.ch> Message-ID: On Mon, Mar 17, 2014 at 8:46 AM, Dridi Boukelmoune wrote: > Hi, > > On Mon, Mar 17, 2014 at 7:51 AM, C?dric Jeanneret wrote: >> Hello, >> >> Sorry for the delay, but? >> >> So, some more information: >> >> Logrotate: >> this shouldn't be a problem, as logrotate doesn't touch the _.vsm file? >> >> Varnishncsa/Varnishlog: >> I don't think the problem is on their side either, as they perform >> read-only access to the _.vsm (tell me if I'm wrong with this statement) >> >> VSM: >> the four varnishes had the very same problem this weekend (of course, >> when nobody's here?). I'm keeping one of them at hand with the problem >> "enabled" in order to be able to debug a bit. >> >> Here are what I can tell: >> >> Inodes: >> indeed, the inodes aren't the same for the one used by the process >> (marked as DEL in lsof, inode 264058) and the one on the system (268725). >> Creation date of the system one: March 16, 2014, 16:10. >> >> After a quick look in the logs, it seems there's a configuration >> modification at this very time: some filters were added, calling a >> reload of the VCL. >> >> Please see bellow for the "hot-reload" script content. >> >> Is there anything weird in this script? >> >> Thank you in advance. >> >> Cheers, >> >> C. >> >> >> >> #!/bin/bash >> # Reload a varnish config >> # Author: Kristian Lyngstol >> >> # Original version: >> http://kly.no/posts/2009_02_18__Easy_reloading_of_varnish__VCL__.html >> >> if [ $# -lt 1 -o $# -gt 2 ]; then >> echo "Usage: $0 vcl_file [secret_file]" >> exit 1 >> fi >> FILE=$1 >> >> if [ $# -eq 2 ]; then >> SECRET_OPT="-S $2" >> fi >> >> # Hostname and management port >> # (defined in /etc/default/varnish or on startup) >> HOSTPORT="localhost:6082" >> NOW=`date +%F_%T` >> >> error() >> { >> echo 1>&2 "Failed to reload $FILE." >> exit 1 >> } >> echo "@@@ Checking VCL file syntax:" >> varnishd -d -s malloc -f $FILE < /dev/null || error > > Maybe this varnishd instance you bring up deletes the file and creates its own. > > Btw, why would you need a storage for a simple syntax check ? Don't mind this question (and the other below). Your echo mislead me, and you're checking the vcl syntax with the wrong arguments. See varnishd(1) for the -C usage. > And isn't your redirection from /dev/null in the wrong direction ? > > Dridi > >> echo -e "\n@@@ Loading new VCL file:" >> varnishadm $SECRET_OPT -T $HOSTPORT vcl.load reload$NOW $FILE || error >> varnishadm $SECRET_OPT -T $HOSTPORT vcl.use reload$NOW || error >> >> >> echo -e "\n@@@ Currently available VCL configs:" >> varnishadm $SECRET_OPT -T $HOSTPORT vcl.list >> >> exit 0 >> >> >> On 03/14/2014 10:01 PM, Stephen Wood wrote: >>> My first thought is that logrotate is running and varishncsa is not >>> properly catching a SIGHUP after a rotate. Therefore the logfile gets >>> rotated but varnishncsa continues writing to the old fd. >>> >>> If you run varnishncsa by itself on the command line during these periods, >>> do you get output? If so then it's probably not a problem with shared >>> memory. >>> >>> If you want to test this, you can simply change the varnishncsa logrotate >>> behavior to use copytruncate and not bother sending a HUP, which will >>> truncate the log without rotating it. >>> >>> For reference here's what my logrotate file looks like: >>> >>> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >>> size 1G >>> rotate 7 >>> missingok >>> compress >>> delaycompress >>> postrotate >>> for service in varnishlog varnishncsa; do >>> if /usr/bin/pgrep -P 1 $service >/dev/null; then >>> service $service reload > /dev/null >>> fi >>> done >>> endscript >>> } >>> >>> If you want to try copytruncate: >>> >>> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >>> size 1G >>> rotate 7 >>> missingok >>> compress >>> delaycompress >>> copytruncate >>> } >>> >>> Note that you'll get a message in your log file that states it was >>> truncated. It may mess up your log parsing software. >>> >>> >>> On Thu, Mar 13, 2014 at 4:33 AM, Raymond Jennings < >>> raymond.jennings at nytimes.com> wrote: >>> >>>> I have what might be a related problem. I can only get varnishlog to >>>> write to a file after I stop it and restart it. I am running on ec2. >>>> I tried various tricks to wait so many seconds after varnished starts >>>> then start varnishlog. Varnishncsa runs perfectly fine right out of >>>> the gate. I have had this problem for about two years now. >>>> Varnishlog is clearly running and "service varnishlog stop" >>>> successfully stops it. "service varnishlog start" and things are >>>> good. >>>> >>>>> On Mar 13, 2014, at 6:27 AM, Thomas Lecomte < >>>> thomas.lecomte at virtual-expo.com> wrote: >>>>> >>>>>> On Thu, Mar 13, 2014 at 10:25:52AM +0100, C?dric Jeanneret wrote: >>>>>> >>>>>> Hmm, _.vsm is shown when we perform a simple "ls" in the directory... >>>>>> Else, I would get some errors from either varnishncsa or varnishlog when >>>>>> I start them (I tried that this morning, just to see what would >>>> happen)... >>>>> >>>>> Maybe you could compare the inodes using ls -i on the visible _.vsm >>>>> against the one shown by lsof on the varnish process. >>>>> >>>>> -- >>>>> Thomas Lecomte / +33 4 86 13 48 65 >>>>> Sysadmin / Virtual Expo / Marseille >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From varnish at tengu.ch Mon Mar 17 08:03:35 2014 From: varnish at tengu.ch (=?UTF-8?B?Q8OpZHJpYyBKZWFubmVyZXQ=?=) Date: Mon, 17 Mar 2014 09:03:35 +0100 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> <8673798013017974179@unknownmsgid> <53269B76.3090503@tengu.ch> Message-ID: <5326AC57.4030101@tengu.ch> Hello Dridi, that was it: the varnishd (run only in order to check the syntax, really, the reload itself is done later) was indeed overriding the _.vsm? I didn't thought about this first, because I wasn't aware it was taking using the already-present instance? Now, I updated the script in order to force the use of a new, temporary instance, forcing the user (-u) as well so that it will work in any cases. This was a bit vicious :]. Cheers, C. On 03/17/2014 08:46 AM, Dridi Boukelmoune wrote: > Hi, > > On Mon, Mar 17, 2014 at 7:51 AM, C?dric Jeanneret wrote: >> Hello, >> >> Sorry for the delay, but? >> >> So, some more information: >> >> Logrotate: >> this shouldn't be a problem, as logrotate doesn't touch the _.vsm file? >> >> Varnishncsa/Varnishlog: >> I don't think the problem is on their side either, as they perform >> read-only access to the _.vsm (tell me if I'm wrong with this statement) >> >> VSM: >> the four varnishes had the very same problem this weekend (of course, >> when nobody's here?). I'm keeping one of them at hand with the problem >> "enabled" in order to be able to debug a bit. >> >> Here are what I can tell: >> >> Inodes: >> indeed, the inodes aren't the same for the one used by the process >> (marked as DEL in lsof, inode 264058) and the one on the system (268725). >> Creation date of the system one: March 16, 2014, 16:10. >> >> After a quick look in the logs, it seems there's a configuration >> modification at this very time: some filters were added, calling a >> reload of the VCL. >> >> Please see bellow for the "hot-reload" script content. >> >> Is there anything weird in this script? >> >> Thank you in advance. >> >> Cheers, >> >> C. >> >> >> >> #!/bin/bash >> # Reload a varnish config >> # Author: Kristian Lyngstol >> >> # Original version: >> http://kly.no/posts/2009_02_18__Easy_reloading_of_varnish__VCL__.html >> >> if [ $# -lt 1 -o $# -gt 2 ]; then >> echo "Usage: $0 vcl_file [secret_file]" >> exit 1 >> fi >> FILE=$1 >> >> if [ $# -eq 2 ]; then >> SECRET_OPT="-S $2" >> fi >> >> # Hostname and management port >> # (defined in /etc/default/varnish or on startup) >> HOSTPORT="localhost:6082" >> NOW=`date +%F_%T` >> >> error() >> { >> echo 1>&2 "Failed to reload $FILE." >> exit 1 >> } >> echo "@@@ Checking VCL file syntax:" >> varnishd -d -s malloc -f $FILE < /dev/null || error > > Maybe this varnishd instance you bring up deletes the file and creates its own. > > Btw, why would you need a storage for a simple syntax check ? > > And isn't your redirection from /dev/null in the wrong direction ? > > Dridi > >> echo -e "\n@@@ Loading new VCL file:" >> varnishadm $SECRET_OPT -T $HOSTPORT vcl.load reload$NOW $FILE || error >> varnishadm $SECRET_OPT -T $HOSTPORT vcl.use reload$NOW || error >> >> >> echo -e "\n@@@ Currently available VCL configs:" >> varnishadm $SECRET_OPT -T $HOSTPORT vcl.list >> >> exit 0 >> >> >> On 03/14/2014 10:01 PM, Stephen Wood wrote: >>> My first thought is that logrotate is running and varishncsa is not >>> properly catching a SIGHUP after a rotate. Therefore the logfile gets >>> rotated but varnishncsa continues writing to the old fd. >>> >>> If you run varnishncsa by itself on the command line during these periods, >>> do you get output? If so then it's probably not a problem with shared >>> memory. >>> >>> If you want to test this, you can simply change the varnishncsa logrotate >>> behavior to use copytruncate and not bother sending a HUP, which will >>> truncate the log without rotating it. >>> >>> For reference here's what my logrotate file looks like: >>> >>> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >>> size 1G >>> rotate 7 >>> missingok >>> compress >>> delaycompress >>> postrotate >>> for service in varnishlog varnishncsa; do >>> if /usr/bin/pgrep -P 1 $service >/dev/null; then >>> service $service reload > /dev/null >>> fi >>> done >>> endscript >>> } >>> >>> If you want to try copytruncate: >>> >>> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >>> size 1G >>> rotate 7 >>> missingok >>> compress >>> delaycompress >>> copytruncate >>> } >>> >>> Note that you'll get a message in your log file that states it was >>> truncated. It may mess up your log parsing software. >>> >>> >>> On Thu, Mar 13, 2014 at 4:33 AM, Raymond Jennings < >>> raymond.jennings at nytimes.com> wrote: >>> >>>> I have what might be a related problem. I can only get varnishlog to >>>> write to a file after I stop it and restart it. I am running on ec2. >>>> I tried various tricks to wait so many seconds after varnished starts >>>> then start varnishlog. Varnishncsa runs perfectly fine right out of >>>> the gate. I have had this problem for about two years now. >>>> Varnishlog is clearly running and "service varnishlog stop" >>>> successfully stops it. "service varnishlog start" and things are >>>> good. >>>> >>>>> On Mar 13, 2014, at 6:27 AM, Thomas Lecomte < >>>> thomas.lecomte at virtual-expo.com> wrote: >>>>> >>>>>> On Thu, Mar 13, 2014 at 10:25:52AM +0100, C?dric Jeanneret wrote: >>>>>> >>>>>> Hmm, _.vsm is shown when we perform a simple "ls" in the directory... >>>>>> Else, I would get some errors from either varnishncsa or varnishlog when >>>>>> I start them (I tried that this morning, just to see what would >>>> happen)... >>>>> >>>>> Maybe you could compare the inodes using ls -i on the visible _.vsm >>>>> against the one shown by lsof on the varnish process. >>>>> >>>>> -- >>>>> Thomas Lecomte / +33 4 86 13 48 65 >>>>> Sysadmin / Virtual Expo / Marseille >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From dridi.boukelmoune at zenika.com Mon Mar 17 16:53:15 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Mon, 17 Mar 2014 17:53:15 +0100 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: <5326AC57.4030101@tengu.ch> References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> <8673798013017974179@unknownmsgid> <53269B76.3090503@tengu.ch> <5326AC57.4030101@tengu.ch> Message-ID: On Mon, Mar 17, 2014 at 9:03 AM, C?dric Jeanneret wrote: > Hello Dridi, > > that was it: the varnishd (run only in order to check the syntax, > really, the reload itself is done later) was indeed overriding the _.vsm? > > I didn't thought about this first, because I wasn't aware it was taking > using the already-present instance? Now, I updated the script in order > to force the use of a new, temporary instance, forcing the user (-u) as > well so that it will work in any cases. Check the varnishd(1) man page, there's a "-C" option you can use to check the VCL syntax. And you can even skip this check, since varnishadm will fail to vcl.load a bogus VCL. Cheers, Dridi > This was a bit vicious :]. > > Cheers, > > C. > > On 03/17/2014 08:46 AM, Dridi Boukelmoune wrote: >> Hi, >> >> On Mon, Mar 17, 2014 at 7:51 AM, C?dric Jeanneret wrote: >>> Hello, >>> >>> Sorry for the delay, but? >>> >>> So, some more information: >>> >>> Logrotate: >>> this shouldn't be a problem, as logrotate doesn't touch the _.vsm file? >>> >>> Varnishncsa/Varnishlog: >>> I don't think the problem is on their side either, as they perform >>> read-only access to the _.vsm (tell me if I'm wrong with this statement) >>> >>> VSM: >>> the four varnishes had the very same problem this weekend (of course, >>> when nobody's here?). I'm keeping one of them at hand with the problem >>> "enabled" in order to be able to debug a bit. >>> >>> Here are what I can tell: >>> >>> Inodes: >>> indeed, the inodes aren't the same for the one used by the process >>> (marked as DEL in lsof, inode 264058) and the one on the system (268725). >>> Creation date of the system one: March 16, 2014, 16:10. >>> >>> After a quick look in the logs, it seems there's a configuration >>> modification at this very time: some filters were added, calling a >>> reload of the VCL. >>> >>> Please see bellow for the "hot-reload" script content. >>> >>> Is there anything weird in this script? >>> >>> Thank you in advance. >>> >>> Cheers, >>> >>> C. >>> >>> >>> >>> #!/bin/bash >>> # Reload a varnish config >>> # Author: Kristian Lyngstol >>> >>> # Original version: >>> http://kly.no/posts/2009_02_18__Easy_reloading_of_varnish__VCL__.html >>> >>> if [ $# -lt 1 -o $# -gt 2 ]; then >>> echo "Usage: $0 vcl_file [secret_file]" >>> exit 1 >>> fi >>> FILE=$1 >>> >>> if [ $# -eq 2 ]; then >>> SECRET_OPT="-S $2" >>> fi >>> >>> # Hostname and management port >>> # (defined in /etc/default/varnish or on startup) >>> HOSTPORT="localhost:6082" >>> NOW=`date +%F_%T` >>> >>> error() >>> { >>> echo 1>&2 "Failed to reload $FILE." >>> exit 1 >>> } >>> echo "@@@ Checking VCL file syntax:" >>> varnishd -d -s malloc -f $FILE < /dev/null || error >> >> Maybe this varnishd instance you bring up deletes the file and creates its own. >> >> Btw, why would you need a storage for a simple syntax check ? >> >> And isn't your redirection from /dev/null in the wrong direction ? >> >> Dridi >> >>> echo -e "\n@@@ Loading new VCL file:" >>> varnishadm $SECRET_OPT -T $HOSTPORT vcl.load reload$NOW $FILE || error >>> varnishadm $SECRET_OPT -T $HOSTPORT vcl.use reload$NOW || error >>> >>> >>> echo -e "\n@@@ Currently available VCL configs:" >>> varnishadm $SECRET_OPT -T $HOSTPORT vcl.list >>> >>> exit 0 >>> >>> >>> On 03/14/2014 10:01 PM, Stephen Wood wrote: >>>> My first thought is that logrotate is running and varishncsa is not >>>> properly catching a SIGHUP after a rotate. Therefore the logfile gets >>>> rotated but varnishncsa continues writing to the old fd. >>>> >>>> If you run varnishncsa by itself on the command line during these periods, >>>> do you get output? If so then it's probably not a problem with shared >>>> memory. >>>> >>>> If you want to test this, you can simply change the varnishncsa logrotate >>>> behavior to use copytruncate and not bother sending a HUP, which will >>>> truncate the log without rotating it. >>>> >>>> For reference here's what my logrotate file looks like: >>>> >>>> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >>>> size 1G >>>> rotate 7 >>>> missingok >>>> compress >>>> delaycompress >>>> postrotate >>>> for service in varnishlog varnishncsa; do >>>> if /usr/bin/pgrep -P 1 $service >/dev/null; then >>>> service $service reload > /dev/null >>>> fi >>>> done >>>> endscript >>>> } >>>> >>>> If you want to try copytruncate: >>>> >>>> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >>>> size 1G >>>> rotate 7 >>>> missingok >>>> compress >>>> delaycompress >>>> copytruncate >>>> } >>>> >>>> Note that you'll get a message in your log file that states it was >>>> truncated. It may mess up your log parsing software. >>>> >>>> >>>> On Thu, Mar 13, 2014 at 4:33 AM, Raymond Jennings < >>>> raymond.jennings at nytimes.com> wrote: >>>> >>>>> I have what might be a related problem. I can only get varnishlog to >>>>> write to a file after I stop it and restart it. I am running on ec2. >>>>> I tried various tricks to wait so many seconds after varnished starts >>>>> then start varnishlog. Varnishncsa runs perfectly fine right out of >>>>> the gate. I have had this problem for about two years now. >>>>> Varnishlog is clearly running and "service varnishlog stop" >>>>> successfully stops it. "service varnishlog start" and things are >>>>> good. >>>>> >>>>>> On Mar 13, 2014, at 6:27 AM, Thomas Lecomte < >>>>> thomas.lecomte at virtual-expo.com> wrote: >>>>>> >>>>>>> On Thu, Mar 13, 2014 at 10:25:52AM +0100, C?dric Jeanneret wrote: >>>>>>> >>>>>>> Hmm, _.vsm is shown when we perform a simple "ls" in the directory... >>>>>>> Else, I would get some errors from either varnishncsa or varnishlog when >>>>>>> I start them (I tried that this morning, just to see what would >>>>> happen)... >>>>>> >>>>>> Maybe you could compare the inodes using ls -i on the visible _.vsm >>>>>> against the one shown by lsof on the varnish process. >>>>>> >>>>>> -- >>>>>> Thomas Lecomte / +33 4 86 13 48 65 >>>>>> Sysadmin / Virtual Expo / Marseille >>>>>> >>>>>> _______________________________________________ >>>>>> varnish-misc mailing list >>>>>> varnish-misc at varnish-cache.org >>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From ruben at varnish-software.com Mon Mar 17 21:05:29 2014 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Mon, 17 Mar 2014 22:05:29 +0100 Subject: March 26th, 2014: What's coming in Varnish 4.0? Hangouts w/Q&A Message-ID: Hello Varnishers, The release of Varnish Cache 4.0 is nearing (planned for Q2) and that means *upgrade*. This Hangout session will cover the key features Varnish 4.0 will introduce and we'll let you know how you can prepare for the release. Post you Varnish 4.0 related questions to Twitter using the #v4rp hashtag or send them to me by replying to this email, should you have any. Cast: * Varnish?s chief architect, Poul-Henning Kamp * Varnish Software?s Founder and CTO, Per Buer * Varnish Release Manager Lasse Karstensen * Varnish Core Developer, Martin Blix Grydeland * Hildur Smaradottir. International Marketing Manager (moderator) * Myself (moderator) Details on our blog: https://www.varnish-software.com/blog/varnish-core-devs-tell-you-what-coming-upcoming-varnish-40-wqa RSVP: https://plus.google.com/events/c56aruu3oevmrdcv9vt7758ofk0 Hope you join us next week. Have a nice Monday! All the best, -- *Rub?n Romero* Community & Sales | Varnish Software AS Cell: +47 95964088 / Office: +47 21989260 Skype, Twitter & IRC: ruben_varnish We Make Websites Fly!Winner of the 2013 Red Herring Top 100 Global Awards -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at tengu.ch Tue Mar 18 08:48:29 2014 From: varnish at tengu.ch (=?UTF-8?B?Q8OpZHJpYyBKZWFubmVyZXQ=?=) Date: Tue, 18 Mar 2014 09:48:29 +0100 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> <8673798013017974179@unknownmsgid> <53269B76.3090503@tengu.ch> <5326AC57.4030101@tengu.ch> Message-ID: <5328085D.1010507@tengu.ch> Hello Dridi On 03/17/2014 05:53 PM, Dridi Boukelmoune wrote: > On Mon, Mar 17, 2014 at 9:03 AM, C?dric Jeanneret wrote: >> Hello Dridi, >> >> that was it: the varnishd (run only in order to check the syntax, >> really, the reload itself is done later) was indeed overriding the _.vsm? >> >> I didn't thought about this first, because I wasn't aware it was taking >> using the already-present instance? Now, I updated the script in order >> to force the use of a new, temporary instance, forcing the user (-u) as >> well so that it will work in any cases. > > Check the varnishd(1) man page, there's a "-C" option you can use to > check the VCL syntax. And you can even skip this check, since > varnishadm will fail to vcl.load a bogus VCL. > > Cheers, > Dridi So it will fail with some error code if there's an error in the VCL? Will it also show the line error, like the current script does? I'll give it a try in order to see if this may replace the current varnishd call. Thanks for your support! Cheers, C. > >> This was a bit vicious :]. >> >> Cheers, >> >> C. >> >> On 03/17/2014 08:46 AM, Dridi Boukelmoune wrote: >>> Hi, >>> >>> On Mon, Mar 17, 2014 at 7:51 AM, C?dric Jeanneret wrote: >>>> Hello, >>>> >>>> Sorry for the delay, but? >>>> >>>> So, some more information: >>>> >>>> Logrotate: >>>> this shouldn't be a problem, as logrotate doesn't touch the _.vsm file? >>>> >>>> Varnishncsa/Varnishlog: >>>> I don't think the problem is on their side either, as they perform >>>> read-only access to the _.vsm (tell me if I'm wrong with this statement) >>>> >>>> VSM: >>>> the four varnishes had the very same problem this weekend (of course, >>>> when nobody's here?). I'm keeping one of them at hand with the problem >>>> "enabled" in order to be able to debug a bit. >>>> >>>> Here are what I can tell: >>>> >>>> Inodes: >>>> indeed, the inodes aren't the same for the one used by the process >>>> (marked as DEL in lsof, inode 264058) and the one on the system (268725). >>>> Creation date of the system one: March 16, 2014, 16:10. >>>> >>>> After a quick look in the logs, it seems there's a configuration >>>> modification at this very time: some filters were added, calling a >>>> reload of the VCL. >>>> >>>> Please see bellow for the "hot-reload" script content. >>>> >>>> Is there anything weird in this script? >>>> >>>> Thank you in advance. >>>> >>>> Cheers, >>>> >>>> C. >>>> >>>> >>>> >>>> #!/bin/bash >>>> # Reload a varnish config >>>> # Author: Kristian Lyngstol >>>> >>>> # Original version: >>>> http://kly.no/posts/2009_02_18__Easy_reloading_of_varnish__VCL__.html >>>> >>>> if [ $# -lt 1 -o $# -gt 2 ]; then >>>> echo "Usage: $0 vcl_file [secret_file]" >>>> exit 1 >>>> fi >>>> FILE=$1 >>>> >>>> if [ $# -eq 2 ]; then >>>> SECRET_OPT="-S $2" >>>> fi >>>> >>>> # Hostname and management port >>>> # (defined in /etc/default/varnish or on startup) >>>> HOSTPORT="localhost:6082" >>>> NOW=`date +%F_%T` >>>> >>>> error() >>>> { >>>> echo 1>&2 "Failed to reload $FILE." >>>> exit 1 >>>> } >>>> echo "@@@ Checking VCL file syntax:" >>>> varnishd -d -s malloc -f $FILE < /dev/null || error >>> >>> Maybe this varnishd instance you bring up deletes the file and creates its own. >>> >>> Btw, why would you need a storage for a simple syntax check ? >>> >>> And isn't your redirection from /dev/null in the wrong direction ? >>> >>> Dridi >>> >>>> echo -e "\n@@@ Loading new VCL file:" >>>> varnishadm $SECRET_OPT -T $HOSTPORT vcl.load reload$NOW $FILE || error >>>> varnishadm $SECRET_OPT -T $HOSTPORT vcl.use reload$NOW || error >>>> >>>> >>>> echo -e "\n@@@ Currently available VCL configs:" >>>> varnishadm $SECRET_OPT -T $HOSTPORT vcl.list >>>> >>>> exit 0 >>>> >>>> >>>> On 03/14/2014 10:01 PM, Stephen Wood wrote: >>>>> My first thought is that logrotate is running and varishncsa is not >>>>> properly catching a SIGHUP after a rotate. Therefore the logfile gets >>>>> rotated but varnishncsa continues writing to the old fd. >>>>> >>>>> If you run varnishncsa by itself on the command line during these periods, >>>>> do you get output? If so then it's probably not a problem with shared >>>>> memory. >>>>> >>>>> If you want to test this, you can simply change the varnishncsa logrotate >>>>> behavior to use copytruncate and not bother sending a HUP, which will >>>>> truncate the log without rotating it. >>>>> >>>>> For reference here's what my logrotate file looks like: >>>>> >>>>> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >>>>> size 1G >>>>> rotate 7 >>>>> missingok >>>>> compress >>>>> delaycompress >>>>> postrotate >>>>> for service in varnishlog varnishncsa; do >>>>> if /usr/bin/pgrep -P 1 $service >/dev/null; then >>>>> service $service reload > /dev/null >>>>> fi >>>>> done >>>>> endscript >>>>> } >>>>> >>>>> If you want to try copytruncate: >>>>> >>>>> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >>>>> size 1G >>>>> rotate 7 >>>>> missingok >>>>> compress >>>>> delaycompress >>>>> copytruncate >>>>> } >>>>> >>>>> Note that you'll get a message in your log file that states it was >>>>> truncated. It may mess up your log parsing software. >>>>> >>>>> >>>>> On Thu, Mar 13, 2014 at 4:33 AM, Raymond Jennings < >>>>> raymond.jennings at nytimes.com> wrote: >>>>> >>>>>> I have what might be a related problem. I can only get varnishlog to >>>>>> write to a file after I stop it and restart it. I am running on ec2. >>>>>> I tried various tricks to wait so many seconds after varnished starts >>>>>> then start varnishlog. Varnishncsa runs perfectly fine right out of >>>>>> the gate. I have had this problem for about two years now. >>>>>> Varnishlog is clearly running and "service varnishlog stop" >>>>>> successfully stops it. "service varnishlog start" and things are >>>>>> good. >>>>>> >>>>>>> On Mar 13, 2014, at 6:27 AM, Thomas Lecomte < >>>>>> thomas.lecomte at virtual-expo.com> wrote: >>>>>>> >>>>>>>> On Thu, Mar 13, 2014 at 10:25:52AM +0100, C?dric Jeanneret wrote: >>>>>>>> >>>>>>>> Hmm, _.vsm is shown when we perform a simple "ls" in the directory... >>>>>>>> Else, I would get some errors from either varnishncsa or varnishlog when >>>>>>>> I start them (I tried that this morning, just to see what would >>>>>> happen)... >>>>>>> >>>>>>> Maybe you could compare the inodes using ls -i on the visible _.vsm >>>>>>> against the one shown by lsof on the varnish process. >>>>>>> >>>>>>> -- >>>>>>> Thomas Lecomte / +33 4 86 13 48 65 >>>>>>> Sysadmin / Virtual Expo / Marseille >>>>>>> >>>>>>> _______________________________________________ >>>>>>> varnish-misc mailing list >>>>>>> varnish-misc at varnish-cache.org >>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>> >>>>>> _______________________________________________ >>>>>> varnish-misc mailing list >>>>>> varnish-misc at varnish-cache.org >>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From manca.alessio at tiscali.it Tue Mar 18 08:56:50 2014 From: manca.alessio at tiscali.it (Alessio (Personale)) Date: Tue, 18 Mar 2014 09:56:50 +0100 Subject: R: R: support to install varnish on a VPS (Plesk on Centos) References: <004101cf3c6e$2ebad8e0$8c308aa0$@alessio@tiscali.it> <1394463784.60197.YahooMailNeo@web126102.mail.ne1.yahoo.com> <005701cf3c77$f8e5ac90$eab105b0$@alessio@tiscali.it> <1394467762.94162.YahooMailNeo@web126104.mail.ne1.yahoo.com> Message-ID: <026a01cf4288$00271320$00753960$@alessio@tiscali.it> Hello! Any other suggestion to check what's wrong with my "missing" installation? Many thanks -----Messaggio originale----- Da: Alessio (Personale) [mailto:manca.alessio at tiscali.it] Inviato: marted? 11 marzo 2014 10:16 A: 'John Doe'; 'varnish-misc at varnish-cache.org' Oggetto: R: R: support to install varnish on a VPS (Plesk on Centos) Hi John, just removed and reinstalled with no errors or warnings. Unfortunately I still can't find any "/etc/sysconfig/varnish" file or directory. Many thanks for your support! Alessio -----Messaggio originale----- Da: varnish-misc-bounces+manca.alessio=tiscali.it at varnish-cache.org [mailto:varnish-misc-bounces+manca.alessio=tiscali.it at varnish-cache.org] Per conto di John Doe Inviato: luned? 10 marzo 2014 17:09 A: varnish-misc at varnish-cache.org Oggetto: Re: R: support to install varnish on a VPS (Plesk on Centos) From: Alessio (Personale) > conto di John Doe >> Which rpm did it install (if it did)...? >> ? # rpm -qa | grep varnish >> The howto you used is based on (the old) CentOS 5. >> Are you on CentOS 6 or still on CentOS 5...? >> If you are on 6, you should install the corresponding varnish repository: >> http://repo.varnish-cache.org/redhat/varnish-3.0/el6/noarch/varnish-release/ varnish-release-3.0-1.el6.noarch.rpm > > Thanks "John". > Using "rpm -qa | grep varnish" > I find "varnish-release-3.0-1.noarch" this sould confirm it's > installed. > I confirm you I'm running on CentOS 6. Yes and no... it confirms a varnish repository is installed, but not which one (el5 or el6?)... If you found only "varnish-release", it means varnish (the application) is not installed. The "yum install varnish" from the howto did not work. That's why you cannot find the files in etc... Remove the varnish repository: ? # rpm -e varnish-release Install the correct one: ? # rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el6/noarch/varnish-release/ varnish-release-3.0-1.el6.noarch.rpm Install the varnish application: ? # yum install varnish JD _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From dridi.boukelmoune at zenika.com Tue Mar 18 09:00:37 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Tue, 18 Mar 2014 10:00:37 +0100 Subject: Varnishd stops sending logs to VSM after a while In-Reply-To: <5328085D.1010507@tengu.ch> References: <53216CB4.1000609@tengu.ch> <532179A0.4080502@tengu.ch> <20140313102533.GA5505@wks140.directindustry.com> <8673798013017974179@unknownmsgid> <53269B76.3090503@tengu.ch> <5326AC57.4030101@tengu.ch> <5328085D.1010507@tengu.ch> Message-ID: On Tue, Mar 18, 2014 at 9:48 AM, C?dric Jeanneret wrote: > Hello Dridi > > On 03/17/2014 05:53 PM, Dridi Boukelmoune wrote: >> On Mon, Mar 17, 2014 at 9:03 AM, C?dric Jeanneret wrote: >>> Hello Dridi, >>> >>> that was it: the varnishd (run only in order to check the syntax, >>> really, the reload itself is done later) was indeed overriding the _.vsm? >>> >>> I didn't thought about this first, because I wasn't aware it was taking >>> using the already-present instance? Now, I updated the script in order >>> to force the use of a new, temporary instance, forcing the user (-u) as >>> well so that it will work in any cases. >> >> Check the varnishd(1) man page, there's a "-C" option you can use to >> check the VCL syntax. And you can even skip this check, since >> varnishadm will fail to vcl.load a bogus VCL. >> >> Cheers, >> Dridi > > So it will fail with some error code if there's an error in the VCL? > Will it also show the line error, like the current script does? I'll > give it a try in order to see if this may replace the current varnishd call. > > Thanks for your support! === $ sudo varnishadm vcl.load test /etc/varnish/test.vcl Message from VCC-compiler: Expected an action, 'if', '{' or '}' ('input' Line 24 Pos 5) i f (req.request != "GET" && ----#--------------------------- Running VCC-compiler failed, exit 1 VCL compilation failed Command failed with error code 106 === Enjoy ;) Dridi > Cheers, > > C. > >> >>> This was a bit vicious :]. >>> >>> Cheers, >>> >>> C. >>> >>> On 03/17/2014 08:46 AM, Dridi Boukelmoune wrote: >>>> Hi, >>>> >>>> On Mon, Mar 17, 2014 at 7:51 AM, C?dric Jeanneret wrote: >>>>> Hello, >>>>> >>>>> Sorry for the delay, but? >>>>> >>>>> So, some more information: >>>>> >>>>> Logrotate: >>>>> this shouldn't be a problem, as logrotate doesn't touch the _.vsm file? >>>>> >>>>> Varnishncsa/Varnishlog: >>>>> I don't think the problem is on their side either, as they perform >>>>> read-only access to the _.vsm (tell me if I'm wrong with this statement) >>>>> >>>>> VSM: >>>>> the four varnishes had the very same problem this weekend (of course, >>>>> when nobody's here?). I'm keeping one of them at hand with the problem >>>>> "enabled" in order to be able to debug a bit. >>>>> >>>>> Here are what I can tell: >>>>> >>>>> Inodes: >>>>> indeed, the inodes aren't the same for the one used by the process >>>>> (marked as DEL in lsof, inode 264058) and the one on the system (268725). >>>>> Creation date of the system one: March 16, 2014, 16:10. >>>>> >>>>> After a quick look in the logs, it seems there's a configuration >>>>> modification at this very time: some filters were added, calling a >>>>> reload of the VCL. >>>>> >>>>> Please see bellow for the "hot-reload" script content. >>>>> >>>>> Is there anything weird in this script? >>>>> >>>>> Thank you in advance. >>>>> >>>>> Cheers, >>>>> >>>>> C. >>>>> >>>>> >>>>> >>>>> #!/bin/bash >>>>> # Reload a varnish config >>>>> # Author: Kristian Lyngstol >>>>> >>>>> # Original version: >>>>> http://kly.no/posts/2009_02_18__Easy_reloading_of_varnish__VCL__.html >>>>> >>>>> if [ $# -lt 1 -o $# -gt 2 ]; then >>>>> echo "Usage: $0 vcl_file [secret_file]" >>>>> exit 1 >>>>> fi >>>>> FILE=$1 >>>>> >>>>> if [ $# -eq 2 ]; then >>>>> SECRET_OPT="-S $2" >>>>> fi >>>>> >>>>> # Hostname and management port >>>>> # (defined in /etc/default/varnish or on startup) >>>>> HOSTPORT="localhost:6082" >>>>> NOW=`date +%F_%T` >>>>> >>>>> error() >>>>> { >>>>> echo 1>&2 "Failed to reload $FILE." >>>>> exit 1 >>>>> } >>>>> echo "@@@ Checking VCL file syntax:" >>>>> varnishd -d -s malloc -f $FILE < /dev/null || error >>>> >>>> Maybe this varnishd instance you bring up deletes the file and creates its own. >>>> >>>> Btw, why would you need a storage for a simple syntax check ? >>>> >>>> And isn't your redirection from /dev/null in the wrong direction ? >>>> >>>> Dridi >>>> >>>>> echo -e "\n@@@ Loading new VCL file:" >>>>> varnishadm $SECRET_OPT -T $HOSTPORT vcl.load reload$NOW $FILE || error >>>>> varnishadm $SECRET_OPT -T $HOSTPORT vcl.use reload$NOW || error >>>>> >>>>> >>>>> echo -e "\n@@@ Currently available VCL configs:" >>>>> varnishadm $SECRET_OPT -T $HOSTPORT vcl.list >>>>> >>>>> exit 0 >>>>> >>>>> >>>>> On 03/14/2014 10:01 PM, Stephen Wood wrote: >>>>>> My first thought is that logrotate is running and varishncsa is not >>>>>> properly catching a SIGHUP after a rotate. Therefore the logfile gets >>>>>> rotated but varnishncsa continues writing to the old fd. >>>>>> >>>>>> If you run varnishncsa by itself on the command line during these periods, >>>>>> do you get output? If so then it's probably not a problem with shared >>>>>> memory. >>>>>> >>>>>> If you want to test this, you can simply change the varnishncsa logrotate >>>>>> behavior to use copytruncate and not bother sending a HUP, which will >>>>>> truncate the log without rotating it. >>>>>> >>>>>> For reference here's what my logrotate file looks like: >>>>>> >>>>>> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >>>>>> size 1G >>>>>> rotate 7 >>>>>> missingok >>>>>> compress >>>>>> delaycompress >>>>>> postrotate >>>>>> for service in varnishlog varnishncsa; do >>>>>> if /usr/bin/pgrep -P 1 $service >/dev/null; then >>>>>> service $service reload > /dev/null >>>>>> fi >>>>>> done >>>>>> endscript >>>>>> } >>>>>> >>>>>> If you want to try copytruncate: >>>>>> >>>>>> /var/log/varnish/varnish.log /var/log/varnish/varnishncsa.log { >>>>>> size 1G >>>>>> rotate 7 >>>>>> missingok >>>>>> compress >>>>>> delaycompress >>>>>> copytruncate >>>>>> } >>>>>> >>>>>> Note that you'll get a message in your log file that states it was >>>>>> truncated. It may mess up your log parsing software. >>>>>> >>>>>> >>>>>> On Thu, Mar 13, 2014 at 4:33 AM, Raymond Jennings < >>>>>> raymond.jennings at nytimes.com> wrote: >>>>>> >>>>>>> I have what might be a related problem. I can only get varnishlog to >>>>>>> write to a file after I stop it and restart it. I am running on ec2. >>>>>>> I tried various tricks to wait so many seconds after varnished starts >>>>>>> then start varnishlog. Varnishncsa runs perfectly fine right out of >>>>>>> the gate. I have had this problem for about two years now. >>>>>>> Varnishlog is clearly running and "service varnishlog stop" >>>>>>> successfully stops it. "service varnishlog start" and things are >>>>>>> good. >>>>>>> >>>>>>>> On Mar 13, 2014, at 6:27 AM, Thomas Lecomte < >>>>>>> thomas.lecomte at virtual-expo.com> wrote: >>>>>>>> >>>>>>>>> On Thu, Mar 13, 2014 at 10:25:52AM +0100, C?dric Jeanneret wrote: >>>>>>>>> >>>>>>>>> Hmm, _.vsm is shown when we perform a simple "ls" in the directory... >>>>>>>>> Else, I would get some errors from either varnishncsa or varnishlog when >>>>>>>>> I start them (I tried that this morning, just to see what would >>>>>>> happen)... >>>>>>>> >>>>>>>> Maybe you could compare the inodes using ls -i on the visible _.vsm >>>>>>>> against the one shown by lsof on the varnish process. >>>>>>>> >>>>>>>> -- >>>>>>>> Thomas Lecomte / +33 4 86 13 48 65 >>>>>>>> Sysadmin / Virtual Expo / Marseille >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> varnish-misc mailing list >>>>>>>> varnish-misc at varnish-cache.org >>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>> >>>>>>> _______________________________________________ >>>>>>> varnish-misc mailing list >>>>>>> varnish-misc at varnish-cache.org >>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> varnish-misc mailing list >>>>>> varnish-misc at varnish-cache.org >>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>> >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From slink at schokola.de Tue Mar 18 16:16:45 2014 From: slink at schokola.de (Nils Goroll) Date: Tue, 18 Mar 2014 17:16:45 +0100 Subject: Supported Platforms for Varnish 4 Message-ID: <5328716D.9010808@schokola.de> Hi, we'd like to inform you about one result of today's Varnish Developer Day, see https://www.varnish-cache.org/trac/wiki/VDD14Q1 (currently undergoing editing, so expect frequent updates until 2014-03-19): We've agreed that the following platforms should be "officially" supported with the next Varnish major release 4.0. Supported in this context means that builds from source tarballs should be possible without error provided that all requirements are met on the respective platform: * Debian wheezy, jessie * Ubuntu 12.04 + 14.04 * RHEL 6 * FreeBSD 9 + 10 Platforms we aim to support * Solaris-descendants: OmniOS, SmartOS Joyent Zones, Oracle Solaris Cheers, Nils From darshak.modi at elitecore.com Wed Mar 19 06:41:12 2014 From: darshak.modi at elitecore.com (Darshak Modi) Date: Wed, 19 Mar 2014 12:11:12 +0530 Subject: Want to have cache server for small ISP Message-ID: <53293C08.2060804@elitecore.com> Hello, I want to deploy a cache server for small ISP. From documents , I understand Varnish can be used for reverse-proxy only. I have to cache all world HTTP traffic, video traffic, updates for end client who are using ISP internet. Cache server to be deployed at ISP router end. I gone through Squid and I think it can work. But I can not understand if Varnish can be used for the same purpose.? Please guide. From perbu at varnish-software.com Wed Mar 19 08:42:12 2014 From: perbu at varnish-software.com (Per Buer) Date: Wed, 19 Mar 2014 09:42:12 +0100 Subject: Want to have cache server for small ISP In-Reply-To: <53293C08.2060804@elitecore.com> References: <53293C08.2060804@elitecore.com> Message-ID: On Wed, Mar 19, 2014 at 7:41 AM, Darshak Modi wrote: > Hello, > > I want to deploy a cache server for small ISP. > From documents , I understand Varnish can be used for reverse-proxy only. > > I have to cache all world HTTP traffic, video traffic, updates for end > client who are using ISP internet. > Cache server to be deployed at ISP router end. I gone through Squid and I > think it can work. > > But I can not understand if Varnish can be used for the same purpose.? > No. Varnish isn't built for this. -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From smwood4 at gmail.com Wed Mar 19 17:50:36 2014 From: smwood4 at gmail.com (Stephen Wood) Date: Wed, 19 Mar 2014 10:50:36 -0700 Subject: Want to have cache server for small ISP In-Reply-To: References: <53293C08.2060804@elitecore.com> Message-ID: Can we clarify exactly why? My understanding is that varnish is a web accelerator for a *website*, and in practice it will only cache for a single domain. It can't be setup to cache for all domains. I can imagine you can get around this by using a varnish + squid pair, but the performance would be unacceptable for an ISP. Not to mention that fact that making varnish highly available and distributed is a lot of work on its own. Can anyone on this list recommend a better solution to the original question? On Wed, Mar 19, 2014 at 1:42 AM, Per Buer wrote: > > > > On Wed, Mar 19, 2014 at 7:41 AM, Darshak Modi wrote: > >> Hello, >> >> I want to deploy a cache server for small ISP. >> From documents , I understand Varnish can be used for reverse-proxy only. >> >> I have to cache all world HTTP traffic, video traffic, updates for end >> client who are using ISP internet. >> Cache server to be deployed at ISP router end. I gone through Squid and >> I think it can work. >> >> But I can not understand if Varnish can be used for the same purpose.? >> > > No. > > Varnish isn't built for this. > > -- > *Per Buer* > CTO | Varnish Software > Phone: +47 958 39 117 | Skype: per.buer > We Make Websites Fly! > > Winner of the Red Herring Top 100 Global Award 2013 > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Stephen Wood www.heystephenwood.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjorn at ruberg.no Wed Mar 19 18:36:16 2014 From: bjorn at ruberg.no (=?ISO-8859-1?Q?Bj=F8rn_Ruberg?=) Date: Wed, 19 Mar 2014 19:36:16 +0100 Subject: Want to have cache server for small ISP In-Reply-To: References: <53293C08.2060804@elitecore.com> Message-ID: <5329E3A0.7040109@ruberg.no> On 03/19/2014 06:50 PM, Stephen Wood wrote: > Can we clarify exactly why? My understanding is that varnish is a web > accelerator for a /website/, and in practice it will only cache for a > single domain. It can't be setup to cache for all domains. Short version: Varnish needs you to *predefine*, as backends, every domain you want to cache for. > I can imagine you can get around this by using a varnish + squid pair, > but the performance would be unacceptable for an ISP. Not to mention > that fact that making varnish highly available and distributed is a > lot of work on its own. Varnish + squid is doable, then the squid installation would be Varnish' one and only backend, which in turn resolves and caches ad hoc. I have no idea how this would perform, but it would give you the combined flexibility of both products. -- Bj?rn -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Wed Mar 19 20:55:51 2014 From: perbu at varnish-software.com (Per Buer) Date: Wed, 19 Mar 2014 21:55:51 +0100 Subject: Want to have cache server for small ISP In-Reply-To: References: <53293C08.2060804@elitecore.com> Message-ID: To supplement Bj?rns answer a bit. Forward caching was never the intended use for Varnish so Varnish as a forward proxy is completely untested. Without knowing too much about the Squid codebase I expect quite a bit of it are workarounds for various bugs/quirks that IE6, Flash 9 and other weird clients have. In additions I'm certain there are optimisations for use cases we simply don't care about. A simple thing such as cookies, which are more or less omnipresent, will completely disable Varnish, which is probably not something you want in a forward proxy. On paper, with a non-caching proxy to serve as a backend, it might work. But then again, there are pieces of software out there that are built specifically for this that will certainly provide a better solution. I think Squid is still seeing active development and it has a lot of relevant features (support for anti-malware plugins, native SSL support...). I think ATS is also meant to work as a forward proxy, but I'm not 100% sure. Per. On Wed, Mar 19, 2014 at 6:50 PM, Stephen Wood wrote: > Can we clarify exactly why? My understanding is that varnish is a web > accelerator for a *website*, and in practice it will only cache for a > single domain. It can't be setup to cache for all domains. > > I can imagine you can get around this by using a varnish + squid pair, but > the performance would be unacceptable for an ISP. Not to mention that fact > that making varnish highly available and distributed is a lot of work on > its own. > > Can anyone on this list recommend a better solution to the original > question? > > > On Wed, Mar 19, 2014 at 1:42 AM, Per Buer wrote: > >> >> >> >> On Wed, Mar 19, 2014 at 7:41 AM, Darshak Modi > > wrote: >> >>> Hello, >>> >>> I want to deploy a cache server for small ISP. >>> From documents , I understand Varnish can be used for reverse-proxy only. >>> >>> I have to cache all world HTTP traffic, video traffic, updates for end >>> client who are using ISP internet. >>> Cache server to be deployed at ISP router end. I gone through Squid and >>> I think it can work. >>> >>> But I can not understand if Varnish can be used for the same purpose.? >>> >> >> No. >> >> Varnish isn't built for this. >> >> -- >> *Per Buer* >> CTO | Varnish Software >> Phone: +47 958 39 117 | Skype: per.buer >> We Make Websites Fly! >> >> Winner of the Red Herring Top 100 Global Award 2013 >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > > -- > Stephen Wood > www.heystephenwood.com > -- *Per Buer* CTO | Varnish Software Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Global Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ticktockhouse at gmail.com Fri Mar 21 10:07:02 2014 From: ticktockhouse at gmail.com (Jerry Steele) Date: Fri, 21 Mar 2014 10:07:02 +0000 Subject: Multiple backends on same server (cPanel) Message-ID: Hello, I seem to be having somewhat confusing issues with Varnish. I've got a simple setup (as I haven't been using it that long, I haven't really got into anything more complex). backend default { .host = "11.22.33.208"; .port = "8081"; } backend default2 { .host = "11.22.33.210"; .port = "8081"; } ## Multiple virtual hosts sub vcl_recv { if (req.http.host ~ "^domain1.com(:[0-9]+)?$";) { set req.backend = default; } elsif (req.http.host ~ "^domain2.co.uk(:[0-9]+)?$";) { set req.backend = default; } elsif (req.http.host ~ "^domain3.com(:[0-9]+)?$";) { set req.backend = default2; } Now, this seems to work well for domain1.com, domain2.co.uk (plus subdomains - I'm not sure why this should be, as the regexes have the ^ anchor tag ?). The problem is, that this doesn't seem to work for domain3.com, which is pointing to a different backend. Well, actually it does work, in that it ends up at the server's default page (it's a cPanel machine), rather than timing out. Fear not, I'm more command line than a panel user :) I've checked the virtualhosts are correct, and indeed the site is visible at domain3.com:8081 I initially had a problem with domain3.com working, and www.domain3.com going to the default page, which is why I've been making changes. Obviously, IPs and domain names have been changed to protect the innocent.. Does anyone have any ideas? Thanks -- --- Jerry Steele Telephone: +44 (0)7920 237105 http://ticktockhouse.co.uk From thierry.magnien at sfr.com Fri Mar 21 10:19:30 2014 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Fri, 21 Mar 2014 10:19:30 +0000 Subject: Multiple backends on same server (cPanel) In-Reply-To: References: Message-ID: <5D103CE839D50E4CBC62C9FD7B83287C8FB4BD10@EXCN015.encara.local.ads> Hi, I would say that none of your regex works and you always end up using the default backend. That why you think it works for domain1 and domain2. Regards, Thierry -----Message d'origine----- De?: varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Jerry Steele Envoy??: vendredi 21 mars 2014 11:07 ??: varnish-misc at varnish-cache.org Objet?: Multiple backends on same server (cPanel) Hello, I seem to be having somewhat confusing issues with Varnish. I've got a simple setup (as I haven't been using it that long, I haven't really got into anything more complex). backend default { .host = "11.22.33.208"; .port = "8081"; } backend default2 { .host = "11.22.33.210"; .port = "8081"; } ## Multiple virtual hosts sub vcl_recv { if (req.http.host ~ "^domain1.com(:[0-9]+)?$";) { set req.backend = default; } elsif (req.http.host ~ "^domain2.co.uk(:[0-9]+)?$";) { set req.backend = default; } elsif (req.http.host ~ "^domain3.com(:[0-9]+)?$";) { set req.backend = default2; } Now, this seems to work well for domain1.com, domain2.co.uk (plus subdomains - I'm not sure why this should be, as the regexes have the ^ anchor tag ?). The problem is, that this doesn't seem to work for domain3.com, which is pointing to a different backend. Well, actually it does work, in that it ends up at the server's default page (it's a cPanel machine), rather than timing out. Fear not, I'm more command line than a panel user :) I've checked the virtualhosts are correct, and indeed the site is visible at domain3.com:8081 I initially had a problem with domain3.com working, and www.domain3.com going to the default page, which is why I've been making changes. Obviously, IPs and domain names have been changed to protect the innocent.. Does anyone have any ideas? Thanks -- --- Jerry Steele Telephone: +44 (0)7920 237105 http://ticktockhouse.co.uk _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From dridi.boukelmoune at zenika.com Fri Mar 21 10:23:31 2014 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Fri, 21 Mar 2014 11:23:31 +0100 Subject: Multiple backends on same server (cPanel) In-Reply-To: <5D103CE839D50E4CBC62C9FD7B83287C8FB4BD10@EXCN015.encara.local.ads> References: <5D103CE839D50E4CBC62C9FD7B83287C8FB4BD10@EXCN015.encara.local.ads> Message-ID: On Fri, Mar 21, 2014 at 11:19 AM, MAGNIEN, Thierry wrote: > Hi, > > I would say that none of your regex works and you always end up using the default backend. That why you think it works for domain1 and domain2. One way to easily check that is to swap the backend declarations: backend default2 { .host = "11.22.33.210"; .port = "8081"; } backend default { .host = "11.22.33.208"; .port = "8081"; } [...] Cheers, Dridi > Regards, > Thierry > > -----Message d'origine----- > De : varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Jerry Steele > Envoy? : vendredi 21 mars 2014 11:07 > ? : varnish-misc at varnish-cache.org > Objet : Multiple backends on same server (cPanel) > > Hello, > > I seem to be having somewhat confusing issues with Varnish. I've got a > simple setup (as I haven't been using it that long, I haven't really > got into anything more complex). > > backend default { > .host = "11.22.33.208"; > .port = "8081"; > } > > backend default2 { > .host = "11.22.33.210"; > .port = "8081"; > } > > > ## Multiple virtual hosts > sub vcl_recv { > if (req.http.host ~ "^domain1.com(:[0-9]+)?$";) { > set req.backend = default; > } > > elsif (req.http.host ~ "^domain2.co.uk(:[0-9]+)?$";) { > set req.backend = default; > } > elsif (req.http.host ~ "^domain3.com(:[0-9]+)?$";) { > set req.backend = default2; > } > > Now, this seems to work well for domain1.com, domain2.co.uk (plus > subdomains - I'm not sure why this should be, as the regexes have the > ^ anchor tag ?). The problem is, that this doesn't seem to work for > domain3.com, which is pointing to a different backend. Well, actually > it does work, in that it ends up at the server's default page (it's a > cPanel machine), rather than timing out. Fear not, I'm more command > line than a panel user :) > > I've checked the virtualhosts are correct, and indeed the site is > visible at domain3.com:8081 > > I initially had a problem with domain3.com working, and > www.domain3.com going to the default page, which is why I've been > making changes. > > Obviously, IPs and domain names have been changed to protect the innocent.. > > Does anyone have any ideas? > > Thanks > > -- > --- > > Jerry Steele > Telephone: +44 (0)7920 237105 > http://ticktockhouse.co.uk > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From ticktockhouse at gmail.com Mon Mar 24 14:37:34 2014 From: ticktockhouse at gmail.com (Jerry Steele) Date: Mon, 24 Mar 2014 14:37:34 +0000 Subject: Multiple backends on same server (cPanel) In-Reply-To: References: <5D103CE839D50E4CBC62C9FD7B83287C8FB4BD10@EXCN015.encara.local.ads> Message-ID: Hello, So, this now appears to work: backend default { .host = "11.22.33.208"; .port = "8081"; } backend default2 { .host = "11.22.33.210"; .port = "8081"; } sub vcl_recv { if (req.http.host ~ "domain3.com$";) { set req.backend = default2; } Is "default" some kind of builtin? The fact that the other virtualhosts are working, to me, says that they're being sent to the default backend, and only requests going to *.domain3.com are being sent to the "default2" backend. varnishlog appears to confirm this. One other issue - I changed this last night, and it appeared not to work, however, I come back to it this morning, and everything is fine. Does this mean that the old configuration (or objects that the new config was looking for) had been cached, and their TTL had to expire? If this is the case, where are these defaults set? Is it necessary to manually clear the cache when changing configs like this? I believe there's a curl command to do this... Thanks -- --- Jerry Steele Telephone: +44 (0)7920 237105 http://ticktockhouse.co.uk From Anton.Kornexl at Uni-Passau.De Mon Mar 24 16:54:54 2014 From: Anton.Kornexl at Uni-Passau.De (Anton Kornexl) Date: Mon, 24 Mar 2014 17:54:54 +0100 Subject: varnish .htaccess IP restrictions Message-ID: <5330716E020000940007EA6E@smtp1.gw.uni-passau.de> Hello, our users can enter IP access restrictions in .htaccess files e.g .allow from 1.2.3.4 deny from 2.3.0.0/16 If a client from 1.2.3.4 accesses a file in the restricted location, it is successfully fetched from varnish and cached in varnish. Later another client from 2.3.4.5 request the same file and receives the cached file without checking the IP address. Is there a way to automagically add and check such restrictions to cached objects? -- Kindly regards Anton Kornexl Rechenzentrum Universit?t Passau Innstr. 33 D-94032 Passau Tel.: 0851/509-1812 Fax: 0851/509-1802 -------------- next part -------------- A non-text attachment was scrubbed... Name: Kornexl, Anton.vcf Type: application/octet-stream Size: 244 bytes Desc: not available URL: From me at ibotty.net Mon Mar 24 17:23:25 2014 From: me at ibotty.net (Tobias Florek) Date: Mon, 24 Mar 2014 18:23:25 +0100 Subject: varnish .htaccess IP restrictions In-Reply-To: <5330716E020000940007EA6E@smtp1.gw.uni-passau.de> References: <5330716E020000940007EA6E@smtp1.gw.uni-passau.de> Message-ID: <53306A0D.9030803@ibotty.net> hi, > our users can enter IP access restrictions in .htaccess files > e.g .allow from 1.2.3.4 > deny from 2.3.0 > [...] > Is there a way to automagically add and check such restrictions to > cached objects? it might be possible to always send a HEAD request first and if it succeeds send the cached copy. that adds additional roundtrips of course. it might be possible to set a header whenever an ip restriction happens and inspect that header within varnish (would need a vmod to do efficiently without edge cases i suppose). finally _if_ you can enumerate the htaccess files it might be possible to parse them and generate (a fragment of) the varnish vcl from a script and reload. use kqueue or inotify to do that automatically, if there is any need. good luck, tobias florek