From shantanu at techblue.co.uk Mon Nov 4 10:18:39 2013 From: shantanu at techblue.co.uk (Kumar Shantanu) Date: Mon, 04 Nov 2013 15:48:39 +0530 Subject: Caching images only Message-ID: <5277747F.5050908@techblue.co.uk> Hello everybody, I am trying to cache all images for my website and it's working fine with simple configuration as given below. > sub vcl_recv { > if (req.url ~ "\.(png|gif|jpg|jpeg|bmp)$"){ > remove req.http.Cookie; > } It's caching image url which is simply ending with filename.jpeg, but this can't cache if the url is bit complex like this > http://test.demo.co.uk/web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=*152.jpeg*&_propertyslider_WAR_propertysliderportlet_img-height=400&_propertyslider_WAR_propertysliderportlet_img-width=960 Any idea on how to achieve this. I tried using reqex to match url but failed :( Thanks Shantanu -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi.boukelmoune at zenika.com Mon Nov 4 10:53:14 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Mon, 4 Nov 2013 11:53:14 +0100 Subject: Caching images only In-Reply-To: <5277747F.5050908@techblue.co.uk> References: <5277747F.5050908@techblue.co.uk> Message-ID: Hi, I maintain a module that might help solve your issue: https://github.com/dridi/libvmod-querystring If you have any question, please do not hesitate. Best Regards, Dridi On Mon, Nov 4, 2013 at 11:18 AM, Kumar Shantanu wrote: > Hello everybody, > > I am trying to cache all images for my website and it's working fine with > simple configuration as given below. > > > sub vcl_recv { > if (req.url ~ "\.(png|gif|jpg|jpeg|bmp)$"){ > remove req.http.Cookie; > } > > > It's caching image url which is simply ending with filename.jpeg, but this > can't cache if the url is bit complex like this > > http://test.demo.co.uk/web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=152.jpeg&_propertyslider_WAR_propertysliderportlet_img-height=400&_propertyslider_WAR_propertysliderportlet_img-width=960 > > > Any idea on how to achieve this. I tried using reqex to match url but failed > :( > > > Thanks > Shantanu > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From shantanu at techblue.co.uk Mon Nov 4 11:04:41 2013 From: shantanu at techblue.co.uk (Kumar Shantanu) Date: Mon, 04 Nov 2013 16:34:41 +0530 Subject: Caching images only In-Reply-To: References: <5277747F.5050908@techblue.co.uk> Message-ID: <52777F49.9080002@techblue.co.uk> Hi Dridi, Thanks for the quick response. Can I install this on my running instance of varnish or will I need to reinstall varnish with this module ? Thanks Shantanu On 11/04/2013 04:23 PM, Dridi Boukelmoune wrote: > Hi, > > I maintain a module that might help solve your issue: > https://github.com/dridi/libvmod-querystring > > If you have any question, please do not hesitate. > > Best Regards, > Dridi > > > On Mon, Nov 4, 2013 at 11:18 AM, Kumar Shantanu wrote: >> Hello everybody, >> >> I am trying to cache all images for my website and it's working fine with >> simple configuration as given below. >> >> >> sub vcl_recv { >> if (req.url ~ "\.(png|gif|jpg|jpeg|bmp)$"){ >> remove req.http.Cookie; >> } >> >> >> It's caching image url which is simply ending with filename.jpeg, but this >> can't cache if the url is bit complex like this >> >> http://test.demo.co.uk/web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=152.jpeg&_propertyslider_WAR_propertysliderportlet_img-height=400&_propertyslider_WAR_propertysliderportlet_img-width=960 >> >> >> Any idea on how to achieve this. I tried using reqex to match url but failed >> :( >> >> >> Thanks >> Shantanu >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- KUMAR SHANTANU Signature TEAM LEAD - IT FOR AND ON BEHALF OF: Technology Blueprint Ltd 23 Clemens Street Royal Leamington Spa Warwickshire CV31 2DW T: 0845 004 7142 (Ext. 5001) E: shantanu at techblue.co.uk W: www.technologyblueprint.co.uk www.propco.co.uk The contents of this email are confidential and may be privileged, and are intended only for the use of the person or company named herein. Any views or opinions presented are solely those of the author and do not necessarily represent those of Technology Blueprint Limited. If you are not the intended recipient of this email or a person responsible for delivering it to the intended recipient, you are hereby notified that any distribution, copying or dissemination of the information herein is strictly prohibited -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: redhat.jpg Type: image/jpeg Size: 2033 bytes Desc: not available URL: From shantanu at techblue.co.uk Mon Nov 4 11:21:39 2013 From: shantanu at techblue.co.uk (Kumar Shantanu) Date: Mon, 04 Nov 2013 16:51:39 +0530 Subject: Caching images only In-Reply-To: <52777F49.9080002@techblue.co.uk> References: <5277747F.5050908@techblue.co.uk> <52777F49.9080002@techblue.co.uk> Message-ID: <52778343.5060902@techblue.co.uk> Hi Dridi, Additionally can you please let me know if what exactly your module will do in my case and how it will cache the images. Thanks Shantanu On 11/04/2013 04:34 PM, Kumar Shantanu wrote: > Hi Dridi, > > Thanks for the quick response. Can I install this on my running > instance of varnish or will I need to reinstall varnish with this module ? > > Thanks > Shantanu > > > On 11/04/2013 04:23 PM, Dridi Boukelmoune wrote: >> Hi, >> >> I maintain a module that might help solve your issue: >> https://github.com/dridi/libvmod-querystring >> >> If you have any question, please do not hesitate. >> >> Best Regards, >> Dridi >> >> >> On Mon, Nov 4, 2013 at 11:18 AM, Kumar Shantanu wrote: >>> Hello everybody, >>> >>> I am trying to cache all images for my website and it's working fine with >>> simple configuration as given below. >>> >>> >>> sub vcl_recv { >>> if (req.url ~ "\.(png|gif|jpg|jpeg|bmp)$"){ >>> remove req.http.Cookie; >>> } >>> >>> >>> It's caching image url which is simply ending with filename.jpeg, but this >>> can't cache if the url is bit complex like this >>> >>> http://test.demo.co.uk/web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=152.jpeg&_propertyslider_WAR_propertysliderportlet_img-height=400&_propertyslider_WAR_propertysliderportlet_img-width=960 >>> >>> >>> Any idea on how to achieve this. I tried using reqex to match url but failed >>> :( >>> >>> >>> Thanks >>> Shantanu >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > -- > > KUMAR SHANTANU > > Signature > > TEAM LEAD - IT > > FOR AND ON BEHALF OF: > > Technology Blueprint Ltd > 23 Clemens Street > Royal Leamington Spa > Warwickshire CV31 2DW > > T: 0845 004 7142 (Ext. 5001) > E: shantanu at techblue.co.uk > W: www.technologyblueprint.co.uk www.propco.co.uk > > The contents of this email are confidential and may be privileged, and > are intended only for the use of the person or company named herein. > Any views or opinions presented are solely those of the author and do > not necessarily represent those of Technology Blueprint Limited. > If you are not the intended recipient of this email or a person > responsible for delivering it to the intended recipient, you are > hereby notified that any distribution, copying or dissemination of the > information herein is strictly prohibited > -- KUMAR SHANTANU Signature TEAM LEAD - IT FOR AND ON BEHALF OF: Technology Blueprint Ltd 23 Clemens Street Royal Leamington Spa Warwickshire CV31 2DW T: 0845 004 7142 (Ext. 5001) E: shantanu at techblue.co.uk W: www.technologyblueprint.co.uk www.propco.co.uk The contents of this email are confidential and may be privileged, and are intended only for the use of the person or company named herein. Any views or opinions presented are solely those of the author and do not necessarily represent those of Technology Blueprint Limited. If you are not the intended recipient of this email or a person responsible for delivering it to the intended recipient, you are hereby notified that any distribution, copying or dissemination of the information herein is strictly prohibited -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 2033 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: redhat.jpg Type: image/jpeg Size: 2033 bytes Desc: not available URL: From gkaragiannidis at dolnet.gr Mon Nov 4 12:21:45 2013 From: gkaragiannidis at dolnet.gr (=?iso-8859-7?B?w+nc7e3n8iDK4fHh4+nh7e3f5Ofy?=) Date: Mon, 4 Nov 2013 12:21:45 +0000 Subject: Caching images only In-Reply-To: <5277747F.5050908@techblue.co.uk> References: <5277747F.5050908@techblue.co.uk> Message-ID: <67ced882581a4a73a23df50d09bce2d0@DB4PR02MB046.eurprd02.prod.outlook.com> Hi, You could always use // remove cookie from these if (req.http.Content-Type ~ "(image|audio|video|pdf|flash|text)") { remove req.http.Cookie; } From: varnish-misc-bounces+gkaragiannidis=dolnet.gr at varnish-cache.org [mailto:varnish-misc-bounces+gkaragiannidis=dolnet.gr at varnish-cache.org] On Behalf Of Kumar Shantanu Sent: Monday, November 04, 2013 12:19 PM To: varnish-misc at varnish-cache.org Subject: Caching images only Hello everybody, I am trying to cache all images for my website and it's working fine with simple configuration as given below. sub vcl_recv { if (req.url ~ "\.(png|gif|jpg|jpeg|bmp)$"){ remove req.http.Cookie; } It's caching image url which is simply ending with filename.jpeg, but this can't cache if the url is bit complex like this http://test.demo.co.uk/web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=152.jpeg&_propertyslider_WAR_propertysliderportlet_img-height=400&_propertyslider_WAR_propertysliderportlet_img-width=960 Any idea on how to achieve this. I tried using reqex to match url but failed :( Thanks Shantanu -------------- next part -------------- An HTML attachment was scrubbed... URL: From gkaragiannidis at dolnet.gr Mon Nov 4 12:23:25 2013 From: gkaragiannidis at dolnet.gr (=?iso-8859-7?B?w+nc7e3n8iDK4fHh4+nh7e3f5Ofy?=) Date: Mon, 4 Nov 2013 12:23:25 +0000 Subject: Caching images only In-Reply-To: <5277747F.5050908@techblue.co.uk> References: <5277747F.5050908@techblue.co.uk> Message-ID: <10971c70fc044a11b16e12989978eaf7@DB4PR02MB046.eurprd02.prod.outlook.com> Hi, You could always use // remove cookie from these if (req.http.Content-Type ~ "(image|audio|video|pdf|flash|text)") { remove req.http.Cookie; } Regards From: varnish-misc-bounces+gkaragiannidis=dolnet.gr at varnish-cache.org [mailto:varnish-misc-bounces+gkaragiannidis=dolnet.gr at varnish-cache.org] On Behalf Of Kumar Shantanu Sent: Monday, November 04, 2013 12:19 PM To: varnish-misc at varnish-cache.org Subject: Caching images only Hello everybody, I am trying to cache all images for my website and it's working fine with simple configuration as given below. ? sub vcl_recv { ? if (req.url ~ "\.(png|gif|jpg|jpeg|bmp)$"){ ??????? remove req.http.Cookie; ?? } It's caching image url which is simply ending with filename.jpeg, but this can't cache if the url is bit complex like this http://test.demo.co.uk/web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=152.jpeg&_propertyslider_WAR_propertysliderportlet_img-height=400&_propertyslider_WAR_propertysliderportlet_img-width=960 Any idea on how to achieve this. I tried using reqex to match url but failed :( Thanks Shantanu From dridi.boukelmoune at zenika.com Mon Nov 4 12:40:27 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Mon, 4 Nov 2013 13:40:27 +0100 Subject: Caching images only In-Reply-To: <52778343.5060902@techblue.co.uk> References: <5277747F.5050908@techblue.co.uk> <52777F49.9080002@techblue.co.uk> <52778343.5060902@techblue.co.uk> Message-ID: On Mon, Nov 4, 2013 at 12:21 PM, Kumar Shantanu wrote: > > Hi Dridi, > > Additionally can you please let me know if what exactly your module will do in my case and how it will cache the images. The module contains utility functions to manipulate the query-string of a URL. If query-params prevent you from caching, there may be a function (or several) that could help you work this out. > Thanks > Shantanu > > > On 11/04/2013 04:34 PM, Kumar Shantanu wrote: > > Hi Dridi, > > Thanks for the quick response. Can I install this on my running instance of varnish or will I need to reinstall varnish with this module ? You need to build the module, install it, and use it in your VCL. You can take a look at the tests for examples on how to use it. > Thanks > Shantanu > > > On 11/04/2013 04:23 PM, Dridi Boukelmoune wrote: > > Hi, > > I maintain a module that might help solve your issue: > https://github.com/dridi/libvmod-querystring > > If you have any question, please do not hesitate. > > Best Regards, > Dridi > > > On Mon, Nov 4, 2013 at 11:18 AM, Kumar Shantanu wrote: > > Hello everybody, > > I am trying to cache all images for my website and it's working fine with > simple configuration as given below. > > > sub vcl_recv { > if (req.url ~ "\.(png|gif|jpg|jpeg|bmp)$"){ > remove req.http.Cookie; > } > > > It's caching image url which is simply ending with filename.jpeg, but this > can't cache if the url is bit complex like this > > http://test.demo.co.uk/web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=152.jpeg&_propertyslider_WAR_propertysliderportlet_img-height=400&_propertyslider_WAR_propertysliderportlet_img-width=960 > > > Any idea on how to achieve this. I tried using reqex to match url but failed > :( > > > Thanks > Shantanu > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > -- > > KUMAR SHANTANU > > TEAM LEAD - IT > > FOR AND ON BEHALF OF: > > Technology Blueprint Ltd > 23 Clemens Street > Royal Leamington Spa > Warwickshire CV31 2DW > > T: 0845 004 7142 (Ext. 5001) > E: shantanu at techblue.co.uk > W: www.technologyblueprint.co.uk www.propco.co.uk > > The contents of this email are confidential and may be privileged, and are intended only for the use of the person or company named herein. Any views or opinions presented are solely those of the author and do not necessarily represent those of Technology Blueprint Limited. > If you are not the intended recipient of this email or a person responsible for delivering it to the intended recipient, you are hereby notified that any distribution, copying or dissemination of the information herein is strictly prohibited > > > > -- > > KUMAR SHANTANU > > TEAM LEAD - IT > > FOR AND ON BEHALF OF: > > Technology Blueprint Ltd > 23 Clemens Street > Royal Leamington Spa > Warwickshire CV31 2DW > > T: 0845 004 7142 (Ext. 5001) > E: shantanu at techblue.co.uk > W: www.technologyblueprint.co.uk www.propco.co.uk > > The contents of this email are confidential and may be privileged, and are intended only for the use of the person or company named herein. Any views or opinions presented are solely those of the author and do not necessarily represent those of Technology Blueprint Limited. > If you are not the intended recipient of this email or a person responsible for delivering it to the intended recipient, you are hereby notified that any distribution, copying or dissemination of the information herein is strictly prohibited From shantanu at techblue.co.uk Mon Nov 4 13:04:22 2013 From: shantanu at techblue.co.uk (Kumar Shantanu) Date: Mon, 04 Nov 2013 18:34:22 +0530 Subject: Caching images only In-Reply-To: <10971c70fc044a11b16e12989978eaf7@DB4PR02MB046.eurprd02.prod.outlook.com> References: <5277747F.5050908@techblue.co.uk> <10971c70fc044a11b16e12989978eaf7@DB4PR02MB046.eurprd02.prod.outlook.com> Message-ID: <52779B56.5090101@techblue.co.uk> Hi, I used this as suggested by you , it's still not caching these images Varnish logs > 11 TxHeader c Content-Type: image/jpeg;charset=UTF-8 > 11 TxHeader c X-Cache: MISS > 13 TxURL b > /web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=168.jpg&_propert > 13 TxProtocol b HTTP/1.1 > 11 RxURL c > /web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=168.jpg&_propert > 11 ObjHeader c Content-Type: image/jpeg;charset=UTF-8 > 11 TxHeader c X-Cache: MISS Thanks Shantanu On 11/04/2013 05:53 PM, ??????? ????????????? wrote: > Hi, > You could always use > > // remove cookie from these > if (req.http.Content-Type ~ "(image|audio|video|pdf|flash|text)") { remove req.http.Cookie; } > > Regards > > From: varnish-misc-bounces+gkaragiannidis=dolnet.gr at varnish-cache.org [mailto:varnish-misc-bounces+gkaragiannidis=dolnet.gr at varnish-cache.org] On Behalf Of Kumar Shantanu > Sent: Monday, November 04, 2013 12:19 PM > To: varnish-misc at varnish-cache.org > Subject: Caching images only > > Hello everybody, > > I am trying to cache all images for my website and it's working fine with simple configuration as given below. > > sub vcl_recv { > if (req.url ~ "\.(png|gif|jpg|jpeg|bmp)$"){ > remove req.http.Cookie; > } > > It's caching image url which is simply ending with filename.jpeg, but this can't cache if the url is bit complex like this > > http://test.demo.co.uk/web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=152.jpeg&_propertyslider_WAR_propertysliderportlet_img-height=400&_propertyslider_WAR_propertysliderportlet_img-width=960 > > Any idea on how to achieve this. I tried using reqex to match url but failed :( > > > Thanks > Shantanu -- KUMAR SHANTANU Signature TEAM LEAD - IT FOR AND ON BEHALF OF: Technology Blueprint Ltd 23 Clemens Street Royal Leamington Spa Warwickshire CV31 2DW T: 0845 004 7142 (Ext. 5001) E: shantanu at techblue.co.uk W: www.technologyblueprint.co.uk www.propco.co.uk The contents of this email are confidential and may be privileged, and are intended only for the use of the person or company named herein. Any views or opinions presented are solely those of the author and do not necessarily represent those of Technology Blueprint Limited. If you are not the intended recipient of this email or a person responsible for delivering it to the intended recipient, you are hereby notified that any distribution, copying or dissemination of the information herein is strictly prohibited -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: redhat.jpg Type: image/jpeg Size: 2033 bytes Desc: not available URL: From stef at scaleengine.com Mon Nov 4 13:15:46 2013 From: stef at scaleengine.com (Stefan Caunter) Date: Mon, 4 Nov 2013 08:15:46 -0500 Subject: Caching images only In-Reply-To: <10971c70fc044a11b16e12989978eaf7@DB4PR02MB046.eurprd02.prod.outlook.com> References: <5277747F.5050908@techblue.co.uk> <10971c70fc044a11b16e12989978eaf7@DB4PR02MB046.eurprd02.prod.outlook.com> Message-ID: On Mon, Nov 4, 2013 at 7:23 AM, ??????? ????????????? wrote: > Hi, > You could always use > > // remove cookie from these > if (req.http.Content-Type ~ "(image|audio|video|pdf|flash|text)") { remove req.http.Cookie; } > > Regards > > > I am trying to cache all images for my website and it's working fine with simple configuration as given below. > > sub vcl_recv { > if (req.url ~ "\.(png|gif|jpg|jpeg|bmp)$"){ > remove req.http.Cookie; > } > > It's caching image url which is simply ending with filename.jpeg, but this can't cache if the url is bit complex like this > > http://test.demo.co.uk/web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=152.jpeg&_propertyslider_WAR_propertysliderportlet_img-height=400&_propertyslider_WAR_propertysliderportlet_img-width=960 > > Any idea on how to achieve this. I tried using reqex to match url but failed :( > the regex does not match; you have told varnish to cache files ending in .jpeg but this test url will fail so you will still have the cookie without knowing how to reverse engineer your webapp (which is what i have found you need to do to use varnish usually), it seems to me you would want to match on the string "_propertyslider_WAR_propertysliderportlet_img-name" as well as strip cookies, and set your cache behaviour in fetch Stefan From james at talkunafraid.co.uk Mon Nov 4 15:45:02 2013 From: james at talkunafraid.co.uk (James Harrison) Date: Mon, 04 Nov 2013 15:45:02 +0000 Subject: varnishstat reporting weird numbers In-Reply-To: References: Message-ID: <5277C0FE.2040705@talkunafraid.co.uk> On 31/10/13 05:50, Hugues Alary wrote: > > Has someone ever run into this problem? > I've seen something similar - no idea on cause but a shutdown and start (as opposed to a restart) cleared it up, iirc. Possibly SHM corruption of some sort? Cheers, James From spaceman at antispaceman.com Wed Nov 6 00:00:19 2013 From: spaceman at antispaceman.com (spaceman) Date: Wed, 6 Nov 2013 00:00:19 +0000 Subject: Caching images only In-Reply-To: <5277747F.5050908@techblue.co.uk> References: <5277747F.5050908@techblue.co.uk> Message-ID: <20131106000019.GA18836@spaceman.antispaceman.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Assuming cookies are the reason it is failing to cache, this should solve the problem: if (req.url ~ "\.(png|gif|jpg|jpeg|bmp)(\?.*|)$"){ remove req.http.Cookie; } Your if statement doesn't apply if there are get parameters on the end of it, hence it doesn't remove the cookie and misses. Regards, spaceman On 15:11 04/11/2013, Kumar Shantanu wrote: > Hello everybody, > > I am trying to cache all images for my website and it's working fine > with simple configuration as given below. > >sub vcl_recv { > > if (req.url ~ "\.(png|gif|jpg|jpeg|bmp)$"){ > > remove req.http.Cookie; > > } > > It's caching image url which is simply ending with filename.jpeg, > but this can't cache if the url is bit complex like this > >http://test.demo.co.uk/web/demo/home?p_p_id=propertyslider_WAR_propertysliderportlet&p_p_lifecycle=2&p_p_state=normal&p_p_mode=view&p_p_cacheability=cacheLevelPage&p_p_col_id=column-1&p_p_col_count=1&_propertyslider_WAR_propertysliderportlet_img-name=*152.jpeg*&_propertyslider_WAR_propertysliderportlet_img-height=400&_propertyslider_WAR_propertysliderportlet_img-width=960 > > Any idea on how to achieve this. I tried using reqex to match url > but failed :( > > > Thanks > Shantanu > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > !DSPAM:5277789736183523517242! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJSeYaTAAoJELI7LCoggkTZHvcIAJPTOt7MghJGVue4DRDbB0HM WZXftVr7YemuLIgUT5Zaqb2FJrYuFKpVE16APpEKpt/I3Y4MEpLkaSiNRf1TMgyj CLqKMU23TpctFzOI9ID9GrA0hiXj+3HsQVlgp3X2jSpXLyQEwV/XG/WlQ61Rlr7A JkfDYfviGtBAK+AUlt7K8fyFPfizSB8r0mZ/vaxd9p5g1i6EJjyk3+sxXss0UC06 V1Bm9gdiwPtIPMNGeBYut4IZ7paO4Ebbs285CkN6Q1gb9wI+Sdii/LqFJbZ7jMSM /XxneIxpwZFqD4bsGPoYQkCLuf+j8zVVr89A3NmnXo56pJf64algbczgaMjHDL4= =K8k7 -----END PGP SIGNATURE----- !DSPAM:5279869436161097714120! From dridi.boukelmoune at zenika.com Tue Nov 12 18:59:29 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Tue, 12 Nov 2013 19:59:29 +0100 Subject: Forcing backend http requests to go thru varnish In-Reply-To: References: Message-ID: Hi, I'm slowly catching up on the mailing list, and obviously I'm a bit late. I haven't seen any response, is your problem solved or do you still need help ? Dridi On Tue, Oct 8, 2013 at 7:05 PM, Payton, Karen J wrote: > Hi all, > > I have some js in a site on my backend that does a jQuery get on an ajax > page within the same site (so in theory all the work takes place in the same > webserver). > The code looks something like this: > > jQuery.get('/ajax?reqtype=providersearch&location='+mylocation+'&maxdistance='+maxdistance+'&cat=,'+mycat); > > Of course I'd like to repopulate the Varnish cache for a long list of zip > codes, so I set up a wget that would hit this ajax page for any number of > location variables. > The only problem is that from within the site, it doesn't know about Varnish > and doesn't go thru Varnish as the front end? so my prepopulated cache is > worthless when my page hits that same ajax from within a web session. At > least I think that's what's going on. > > I thought that forcing a fully qualified url might cause the jQuery.get to > go outside the server and come thru the Varnish front door, so I coded this: > > jQuery.get('http://[my.website.address.com]/programs/roadmap/ajax/?reqtype=providersearch&location='+mylocation+'&maxdistance='+maxdistance+'&cat=,'+mycat); > > Still no dice. Anyone have experience getting an ajaxed chunk of data > cached effectively using Varnish? > > Thanks for your help, I'm new so if this is obvious please forgive my > ignorance! > > > > Joy Payton > > Web Services > > X 67658 > > > > great service ? great discoveries > > Have I provided you with great service that powers your great discoveries? > Tell me about it. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From cdgraff at gmail.com Tue Nov 12 23:47:56 2013 From: cdgraff at gmail.com (Alejandro) Date: Tue, 12 Nov 2013 21:47:56 -0200 Subject: Varnish-Agent question Message-ID: Hi guys, I'll like to know if the version into github https://github.com/varnish/vagent2 is the same that varnish-agent into the oficial repo? I'm trying to setup https://github.com/pbruna/Varnish-Agent-Dashboard but have some issue, with the instructions. For this reason I'll like to know if be the same or not? I have this runing into old server, but wil oldest version of agent and varnish. Thanks Alejandro -------------- next part -------------- An HTML attachment was scrubbed... URL: From daghf at varnish-software.com Wed Nov 13 10:27:41 2013 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Wed, 13 Nov 2013 11:27:41 +0100 Subject: Varnish-Agent question In-Reply-To: References: Message-ID: On Wed, Nov 13, 2013 at 12:47 AM, Alejandro wrote: > > > I'll like to know if the version into github > https://github.com/varnish/vagent2 > is the same that varnish-agent into the oficial repo? No. The one in the offical repos is varnish-agent 1, https://github.com/varnish/varnish-agent (packages for agent2 will be in the repos sometime soon) -- Dag Haavi Finstad Software Developer | Varnish Software AS Mobile: +47 476 64 134 | Office: +47 21 98 92 60 We Make Websites Fly! From PaytonK at email.chop.edu Wed Nov 13 14:52:29 2013 From: PaytonK at email.chop.edu (Payton, Joy J) Date: Wed, 13 Nov 2013 14:52:29 +0000 Subject: Forcing backend http requests to go thru varnish In-Reply-To: References: Message-ID: Solved! It was an ESO (equipment smarter than operator) error on my part... literally a typo. Thanks for checking in. Joy -----Original Message----- From: Dridi Boukelmoune [mailto:dridi.boukelmoune at zenika.com] Sent: Tuesday, November 12, 2013 1:59 PM To: Payton, Joy J Cc: varnish-misc at varnish-cache.org Subject: Re: Forcing backend http requests to go thru varnish Hi, I'm slowly catching up on the mailing list, and obviously I'm a bit late. I haven't seen any response, is your problem solved or do you still need help ? Dridi On Tue, Oct 8, 2013 at 7:05 PM, Payton, Karen J wrote: > Hi all, > > I have some js in a site on my backend that does a jQuery get on an > ajax page within the same site (so in theory all the work takes place > in the same webserver). > The code looks something like this: > > jQuery.get('/ajax?reqtype=providersearch&location='+mylocation+'&maxdi > stance='+maxdistance+'&cat=,'+mycat); > > Of course I'd like to repopulate the Varnish cache for a long list of > zip codes, so I set up a wget that would hit this ajax page for any > number of location variables. > The only problem is that from within the site, it doesn't know about > Varnish and doesn't go thru Varnish as the front end? so my > prepopulated cache is worthless when my page hits that same ajax from > within a web session. At least I think that's what's going on. > > I thought that forcing a fully qualified url might cause the > jQuery.get to go outside the server and come thru the Varnish front door, so I coded this: > > jQuery.get('http://[my.website.address.com]/programs/roadmap/ajax/?req > type=providersearch&location='+mylocation+'&maxdistance='+maxdistance+ > '&cat=,'+mycat); > > Still no dice. Anyone have experience getting an ajaxed chunk of data > cached effectively using Varnish? > > Thanks for your help, I'm new so if this is obvious please forgive my > ignorance! > > > > Joy Payton > > Web Services > > X 67658 > > > > great service ? great discoveries > > Have I provided you with great service that powers your great discoveries? > Tell me about it. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From dridi.boukelmoune at zenika.com Wed Nov 13 16:14:15 2013 From: dridi.boukelmoune at zenika.com (Dridi Boukelmoune) Date: Wed, 13 Nov 2013 17:14:15 +0100 Subject: Varnish Purge - Ban, custom hash In-Reply-To: References: Message-ID: Hi, Have you solved your issue ? I haven't looked closely at the VCL yet but maybe I could help. Dridi On Mon, Oct 28, 2013 at 8:34 PM, Lane, Richard wrote: > I am using multiple hash_data calls to customize the hash key. I need to > cache potentially three different version of the page. > > My problem is how do I purge the items. I have tried implementing purge; in > recv, hit, miss functions like Varnish docs show but still not working. > > I have tried implementing ban using a regex or == on the URL. > > Below is my VCL hash function. How do I purge a URL that is hashed with > key... > > "/mysection/mystory.htmlwww.example.com_preview_mobile" > > I had this working prior to Varnish 3 but seems to have stopped working > after that. > > ##VCL Hash function below > > vcl_hash { > > hash_data(req.url); > if (req.http.host) { > hash_data(req.http.host); > } else { > hash_data(server.ip); > } > > #if(req.http.Accept-Encoding) { > # set req.hash += req.http.Accept-Encoding; > #} > > ## /* Normalize Accept-Encoding to reduce effects of Vary: > Accept-Encoding > ## (cf. http://varnish-cache.org/wiki/FAQ/Compression) > ## Also note that Vary: User-Agent is considered harmful */ > if (req.http.Accept-Encoding) { > if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { > // Don't compress already-compressed files > remove req.http.Accept-Encoding; > } > elsif (req.http.Accept-Encoding ~ "gzip") { > set req.http.Accept-Encoding = "gzip"; > } > elsif (req.http.Accept-Encoding ~ "deflate") { > set req.http.Accept-Encoding = "deflate"; > } > else { > // unknown algorithm > remove req.http.Accept-Encoding; > } > } > > if(req.http.x-preview == "yes" && (req.url ~ ".shtml" || req.url ~ > ".html") ) > { > hash_data("_preview"); > ##Add device view to hash > hash_data("_" + req.http.x-preferred-device); > > }else{ > > ##Add device view to hash > hash_data("_" + req.http.x-device); > } > return (hash); > } > > #Purge pieces > vcl_recv { > > if (req.request == "PURGE") { > if (!client.ip ~ localaddr) { > error 405 "Method not allowed"; > } > ban("obj.http.x-url ~ " + req.url + " && obj.http.x-host == " > + req.http.host + " && obj.http.x-pd ~ .*" ); > std.log("Purge : yes really purge."); > return(lookup); > } > > vcl_hit { > if (req.request == "PURGE") { > purge; > error 200 "Purged."; > } > } > > vcl_miss { > if (req.request == "PURGE") { > purge; > error 200 "Purged."; > } > > vcl_fetch { > #setup for banning > set beresp.http.x-url = req.url; > set beresp.http.x-host = req.http.host; > set beresp.http.x-pd = req.http.x-device; > > } > > vcl_deliver { > #Clean up banning > unset resp.http.x-url; > unset resp.http.x-host; > unset resp.http.x-pd; > > } > > Thanks in advance, > > Richard > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From maillists0 at gmail.com Fri Nov 15 20:52:31 2013 From: maillists0 at gmail.com (maillists0 at gmail.com) Date: Fri, 15 Nov 2013 15:52:31 -0500 Subject: Fail Whale Message-ID: In some load balancers (haproxy, for example) you can create a backend that only gets hit if all the other backends are down. A common enough setup is to have a local apache serve a polite "oops" page on an alternate port on the loopback. Is there some clever way to do this with Varnish? I've read to docs and googled but can't find anything like this. If this is possible, a doc to the right link would be awesome. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stef at scaleengine.com Fri Nov 15 21:11:07 2013 From: stef at scaleengine.com (Stefan Caunter) Date: Fri, 15 Nov 2013 16:11:07 -0500 Subject: Fail Whale In-Reply-To: References: Message-ID: On Fri, Nov 15, 2013 at 3:52 PM, wrote: > In some load balancers (haproxy, for example) you can create a backend that > only gets hit if all the other backends are down. A common enough setup is > to have a local apache serve a polite "oops" page on an alternate port on > the loopback. Is there some clever way to do this with Varnish? I've read to > docs and googled but can't find anything like this. If this is possible, a > doc to the right link would be awesome. I have done this, with some external domain hosting the fail whale sub vcl_error { set obj.http.Location = ?http://oops.scaleengine.com/?; set obj.status = 302; return(deliver); } this is from http://caunter.ca/?p=124 from 3 years ago, the general idea should still work E: stefan.caunter at scaleengine.com Skype: stefan.caunter Toronto Canada: +1 800 224 0192 From smwood4 at gmail.com Fri Nov 15 21:11:12 2013 From: smwood4 at gmail.com (Stephen Wood) Date: Fri, 15 Nov 2013 13:11:12 -0800 Subject: Fail Whale In-Reply-To: References: Message-ID: I don't think it would be very hard to do with the vcl. If the beresp != 200 you would set a fail header and then retry it. Then just check for that header to direct to a "fail whale" cluster that serves a simple error page. I'm sure someone can chime in with a better solution. On Fri, Nov 15, 2013 at 12:52 PM, wrote: > In some load balancers (haproxy, for example) you can create a backend > that only gets hit if all the other backends are down. A common enough > setup is to have a local apache serve a polite "oops" page on an alternate > port on the loopback. Is there some clever way to do this with Varnish? > I've read to docs and googled but can't find anything like this. If this is > possible, a doc to the right link would be awesome. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Stephen Wood www.heystephenwood.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From smwood4 at gmail.com Fri Nov 15 21:13:03 2013 From: smwood4 at gmail.com (Stephen Wood) Date: Fri, 15 Nov 2013 13:13:03 -0800 Subject: Fail Whale In-Reply-To: References: Message-ID: One final thought. Obviously you wouldn't want to redirect everything that's not a 200. Just throwing that in as an example. A failed backend would actually return 503 I believe. On Fri, Nov 15, 2013 at 1:11 PM, Stephen Wood wrote: > I don't think it would be very hard to do with the vcl. If the beresp != > 200 you would set a fail header and then retry it. Then just check for that > header to direct to a "fail whale" cluster that serves a simple error page. > > I'm sure someone can chime in with a better solution. > > > On Fri, Nov 15, 2013 at 12:52 PM, wrote: > >> In some load balancers (haproxy, for example) you can create a backend >> that only gets hit if all the other backends are down. A common enough >> setup is to have a local apache serve a polite "oops" page on an alternate >> port on the loopback. Is there some clever way to do this with Varnish? >> I've read to docs and googled but can't find anything like this. If this is >> possible, a doc to the right link would be awesome. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > > -- > Stephen Wood > www.heystephenwood.com > -- Stephen Wood www.heystephenwood.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From james at ifixit.com Fri Nov 15 21:29:01 2013 From: james at ifixit.com (James Pearson) Date: Fri, 15 Nov 2013 13:29:01 -0800 Subject: Fail Whale In-Reply-To: References: Message-ID: <1384550259-sup-9669@geror.local> Excerpts from maillists0's message of 2013-11-15 12:52:31 -0800: > In some load balancers (haproxy, for example) you can create a backend that > only gets hit if all the other backends are down. A common enough setup is > to have a local apache serve a polite "oops" page on an alternate port on > the loopback. Is there some clever way to do this with Varnish? I've read > to docs and googled but can't find anything like this. If this is possible, > a doc to the right link would be awesome. If all you want to do is serve a static page, you can do that in vcl_error. We read in an external HTML file (that's also used for other things) using std.fileread[0]. Here's the entire thing[1]; perhaps it will be useful. - P [0]: http://comments.gmane.org/gmane.comp.web.varnish.misc/8099 [1]: http://dpaste.com/hold/1465288/ From numard at gmail.com Fri Nov 15 22:20:10 2013 From: numard at gmail.com (Norberto Meijome) Date: Sat, 16 Nov 2013 09:20:10 +1100 Subject: Fail Whale In-Reply-To: References: Message-ID: +1 You could always hard code the error page in your vcl (like the guru meditation error is....) On 16/11/2013 9:13 AM, "Stephen Wood" wrote: > I don't think it would be very hard to do with the vcl. If the beresp != > 200 you would set a fail header and then retry it. Then just check for that > header to direct to a "fail whale" cluster that serves a simple error page. > > I'm sure someone can chime in with a better solution. > > > On Fri, Nov 15, 2013 at 12:52 PM, wrote: > >> In some load balancers (haproxy, for example) you can create a backend >> that only gets hit if all the other backends are down. A common enough >> setup is to have a local apache serve a polite "oops" page on an alternate >> port on the loopback. Is there some clever way to do this with Varnish? >> I've read to docs and googled but can't find anything like this. If this is >> possible, a doc to the right link would be awesome. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > > -- > Stephen Wood > www.heystephenwood.com > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From glenn at squixa.com Sat Nov 16 10:05:18 2013 From: glenn at squixa.com (Glenn Slaven) Date: Sat, 16 Nov 2013 21:05:18 +1100 Subject: Varnish 3.0 Guru Meditation with "FetchError TestGunzip error at the very end" Message-ID: I'm getting intermittent guru meditations in varnish (as in I fire 50 requests off and 3 come back bad). In varnishlog it says 15 VCL_return c hash 15 HitPass c 1394372109 15 VCL_call c pass pass 15 Backend c 17 default default 15 TTL c 1394372164 RFC 0 -1 -1 1384590297 0 1384590291 0 0 15 VCL_call c fetch 15 TTL c 1394372164 VCL 120 -1 -1 1384590297 -0 15 VCL_return c hit_for_pass 15 ObjProtocol c HTTP/1.1 15 ObjResponse c OK 15 ObjHeader c Date: Sat, 16 Nov 2013 08:24:51 GMT 15 ObjHeader c Server: Apache 15 ObjHeader c Accept-Ranges: bytes 15 ObjHeader c Cache-Control: max-age=0, no-cache 15 ObjHeader c Vary: Accept-Encoding 15 ObjHeader c X-Mod-Pagespeed: 1.5.27.2-2912 15 ObjHeader c Content-Encoding: gzip 15 ObjHeader c Content-Type: text/html; charset=utf-8 15 Gzip c u F - 3755 13624 80 0 0 15 FetchError c TestGunzip error at the very end 15 VCL_call c error deliver 15 VCL_call c deliver deliver 15 TxProtocol c HTTP/1.1 15 TxStatus c 503 15 TxResponse c Service Unavailable You can see the 15 FetchError c TestGunzip error at the very end which is the problem. I'm not sure how to interpret the line above Gzip c u F - 3755 13624 80 0 0 and I can't see why this is a problem. The site did not have any reported problem loading pages before we put varnish in front. In an assumption that varnish is just being more strict about gzip than the browsers are, I attempted to turn off the gzip handling, so I set http_gzip_support to off in /etc/defaults/varnish: DAEMON_OPTS="-a :80 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -S /etc/varnish/secret \ -s malloc,256m \ -p http_gzip_support=off" This has failed to make any difference. Any help would be greatly appreciated. Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From gabster at lelutin.ca Sat Nov 16 21:26:12 2013 From: gabster at lelutin.ca (Gabriel Filion) Date: Sat, 16 Nov 2013 16:26:12 -0500 Subject: Fail Whale In-Reply-To: <1384550259-sup-9669@geror.local> References: <1384550259-sup-9669@geror.local> Message-ID: <5287E2F4.20709@lelutin.ca> On 15/11/13 04:29 PM, James Pearson wrote: > Excerpts from maillists0's message of 2013-11-15 12:52:31 -0800: >> In some load balancers (haproxy, for example) you can create a backend that >> only gets hit if all the other backends are down. A common enough setup is >> to have a local apache serve a polite "oops" page on an alternate port on >> the loopback. Is there some clever way to do this with Varnish? I've read >> to docs and googled but can't find anything like this. If this is possible, >> a doc to the right link would be awesome. > > If all you want to do is serve a static page, you can do that in vcl_error. We > read in an external HTML file (that's also used for other things) using > std.fileread[0]. Here's the entire thing[1]; perhaps it will be useful. hmm this is nice. we've been using static html in the vcl up to now, but we also deploy the html as a file for the local lighttpd instance so that we can point arbitrary domains to it for maintenance. with your suggestion, we can maintain only one copy of the html, and then use puppet (for example) to trigger a reload when the contents change. > [0]: http://comments.gmane.org/gmane.comp.web.varnish.misc/8099 > [1]: http://dpaste.com/hold/1465288/ since the paste in [1] above is set to expire in 30 days, it means that ppl who stumble upon this thread in more than one month will not see the example. the paste is relatively small, so for posterity and future users' sake, here's a copy of it: /** * Called when we hit an error, either explicitly or implicitly due to backend * or internal errors. * * The vcl_error subroutine may terminate by calling return with one of the * following keywords: * * deliver * Deliver the error object to the client. * restart * Restart the transaction. Increases the restart counter. If the number of * restarts is higher than max_restarts varnish emits a guru meditation * error. */ sub vcl_error { # Allow the use of the error directive to do redirects in VCL. if (obj.status == 301 || obj.status == 302) { set obj.http.Location = obj.response; return (deliver); } set obj.http.Content-Type = "text/html; charset=utf-8"; set obj.http.Retry-After = "5"; # This file appears to only be read on a Varnish restart, or reload if the # VCL changes. set obj.http.errorHTML = std.fileread("/path/to/down.html"); # Custom error message? if (req.http.X-Error-Message) { set obj.http.errorHTML = regsub( obj.http.errorHTML, "We're down for scheduled maintenance\.", req.http.X-Error-Message); } synthetic obj.http.errorHTML; unset obj.http.errorHTML; return (deliver); } -- Gabriel Filion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 291 bytes Desc: OpenPGP digital signature URL: From viktor.villafuerte at optusnet.com.au Sun Nov 17 22:27:05 2013 From: viktor.villafuerte at optusnet.com.au (Viktor Villafuerte) Date: Mon, 18 Nov 2013 09:27:05 +1100 Subject: Backend content location Message-ID: <20131117222705.GB4777@optusnet.com.au> Hi all, I'd like to ask if it's possible to get the content location, from a pool of backend servers, instead of the content itself. Eg: backends are grouped by 'director hash' therefore each of them will have unique content. When my request is not found in the cache it then proceeds onto the backends and gets the content. I'd like to find the location such as server1.domain.com/content1 instead of actually getting the content. Is this possible? v..v -- Regards Viktor Villafuerte Optus Internet Engineering t: 02 808-25265 From tfheen at varnish-software.com Mon Nov 18 08:11:37 2013 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 18 Nov 2013 09:11:37 +0100 Subject: Varnish 3.0.5 rc 1 Message-ID: <20131118081137.GA10243@err.no> Hi all, we're in the process of getting 3.0.5 out the door and have a release candidate. Please take a look at the packages from one of: https://jenkins.varnish-software.com/view/varnish-3.0/job/varnish-3.0-rpm-el6/ https://jenkins.varnish-software.com/view/varnish-3.0/job/varnish-3.0-rpm-el5/ https://jenkins.varnish-software.com/view/varnish-3.0/job/varnish-3.0-deb-debian-wheezy-amd64/lastSuccessfulBuild/artifact/deb-build/ or the source from https://jenkins.varnish-software.com/view/varnish-3.0/job/varnish-3.0-src/ Feedback on this most welcome. Since this is a release candidate in the stable release series, no changes to your configuration should be necessary. -- Tollef Fog Heen Technical lead | Varnish Software AS ??: +47 21 98 92 64 We Make Websites Fly! From guillaume.quintard at smartjog.com Mon Nov 18 08:54:38 2013 From: guillaume.quintard at smartjog.com (Guillaume Quintard) Date: Mon, 18 Nov 2013 09:54:38 +0100 Subject: Backend content location In-Reply-To: <20131117222705.GB4777@optusnet.com.au> References: <20131117222705.GB4777@optusnet.com.au> Message-ID: <5289D5CE.3090801@smartjog.com> On 11/17/2013 11:27 PM, Viktor Villafuerte wrote: > Hi all, > > I'd like to ask if it's possible to get the content location, from a > pool of backend servers, instead of the content itself. > > Eg: backends are grouped by 'director hash' therefore each of them will > have unique content. When my request is not found in the cache it then > proceeds onto the backends and gets the content. I'd like to find the > location such as server1.domain.com/content1 instead of actually getting > the content. > > Is this possible? > > v..v > Would set beresp.http.be_name = beresp.backend.name; Be what you are looking for? (taken from bin/varnishtest/tests/v00028.vtc) -- Guillaume Quintard From viktor.villafuerte at optusnet.com.au Mon Nov 18 23:38:59 2013 From: viktor.villafuerte at optusnet.com.au (Viktor Villafuerte) Date: Tue, 19 Nov 2013 10:38:59 +1100 Subject: Backend content location In-Reply-To: <5289D5CE.3090801@smartjog.com> References: <20131117222705.GB4777@optusnet.com.au> <5289D5CE.3090801@smartjog.com> Message-ID: <20131118233859.GC4777@optusnet.com.au> Thanks for the suggestion. I just got to work so I'm gonna try that out. One side question though the 'beresp.http.be_name' - where can I find all these options? I just searched thru the Varnish manual (pdf) and vcl man page but there's nothing about 'be_name'. How do you find these? v.v. On Mon 18 Nov 2013 09:54:38, Guillaume Quintard wrote: > On 11/17/2013 11:27 PM, Viktor Villafuerte wrote: > >Hi all, > > > >I'd like to ask if it's possible to get the content location, from a > >pool of backend servers, instead of the content itself. > > > >Eg: backends are grouped by 'director hash' therefore each of them will > >have unique content. When my request is not found in the cache it then > >proceeds onto the backends and gets the content. I'd like to find the > >location such as server1.domain.com/content1 instead of actually getting > >the content. > > > >Is this possible? > > > >v..v > > > Would > set beresp.http.be_name = beresp.backend.name; > Be what you are looking for? (taken from bin/varnishtest/tests/v00028.vtc) > > -- > Guillaume Quintard -- Regards Viktor Villafuerte Optus Internet Engineering t: 02 808-25265 From james at ifixit.com Tue Nov 19 01:49:49 2013 From: james at ifixit.com (James Pearson) Date: Mon, 18 Nov 2013 17:49:49 -0800 Subject: Backend content location In-Reply-To: <20131118233859.GC4777@optusnet.com.au> References: <20131117222705.GB4777@optusnet.com.au> <5289D5CE.3090801@smartjog.com> <20131118233859.GC4777@optusnet.com.au> Message-ID: <1384825731-sup-2435@geror.local> Excerpts from Viktor Villafuerte's message of 2013-11-18 15:38:59 -0800: > Thanks for the suggestion. I just got to work so I'm gonna try that out. > One side question though the 'beresp.http.be_name' - where can I find > all these options? I just searched thru the Varnish manual (pdf) and vcl > man page but there's nothing about 'be_name'. How do you find these? https://www.varnish-cache.org/docs/3.0/reference/vcl.html#variables Although, things in beresp.http are just HTTP headers, so I'm not sure where the idea of be_name doing something special comes in. - P From Andy.Lightfoot at ig.com Tue Nov 19 13:31:22 2013 From: Andy.Lightfoot at ig.com (Andy Lightfoot) Date: Tue, 19 Nov 2013 13:31:22 +0000 Subject: Space in Cache but n_lru_nuked statistic growing Message-ID: Hi there, We have recently increase the size of our file based cache to 50Gb to cope with a significant increase in the content we need to cache. However, even though there is plenty of free space in the cache the n_lru_nuked stat is growing. I thought that this would only be the case if there was not sufficient space in the cache (but there appears to be over 14GB free in the cache and we have plenty of disk space on the partition that contains the cache). We are using Varnish 3.0.3 on RHEL6. Any ideas why the n_lru_nuked stat is rising? This is the output of varnishstat client_conn 4411215 13.94 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 11657777 36.84 Client requests received cache_hit 5692984 17.99 Cache hits cache_hitpass 267 0.00 Cache hits for pass cache_miss 4606906 14.56 Cache misses backend_conn 180865 0.57 Backend conn. success backend_unhealthy 4298 0.01 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 0 0.00 Backend conn. failures backend_reuse 5316479 16.80 Backend conn. reuses backend_toolate 158949 0.50 Backend conn. was closed backend_recycle 5475442 17.30 Backend conn. recycles backend_retry 314 0.00 Backend conn. retry fetch_head 1 0.00 Fetch head fetch_length 2763177 8.73 Fetch with Length fetch_chunked 2733682 8.64 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 0 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 10 0.00 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 0 0.00 Fetch no body (204) fetch_304 17 0.00 Fetch no body (304) n_sess_mem 141 . N struct sess_mem n_sess 52 . N struct sess n_object 707545 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 707577 . N struct objectcore n_objecthead 278199 . N struct objecthead n_waitinglist 509915 . N struct waitinglist n_vbc 14 . N struct vbc n_wrk 100 . N worker threads n_wrk_create 102 0.00 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 0 0.00 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 3 . N backends n_expired 2015 . N expired objects n_lru_nuked 3893293 . N LRU nuked objects n_lru_moved 1569381 . N LRU moved objects losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 10710522 33.84 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 4411215 13.94 Total Sessions s_req 11657777 36.84 Total Requests s_pipe 0 0.00 Total pipe s_pass 894489 2.83 Total pass s_fetch 5496867 17.37 Total fetch s_hdrbytes 3296529604 10416.13 Total header bytes s_bodybytes 148093689394 467935.68 Total body bytes sess_closed 1178668 3.72 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 11189500 35.36 Session Linger sess_herd 9372758 29.62 Session herd shm_records 814659170 2574.10 SHM records shm_writes 46456055 146.79 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 26661 0.08 SHM MTX contention shm_cycles 341 0.00 SHM cycles through buffer sms_nreq 4528 0.01 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 1897232 . SMS bytes allocated sms_bfree 1897232 . SMS bytes freed backend_req 5497344 17.37 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_gone 1 . N total gone bans n_ban_add 1 0.00 N new bans added n_ban_retire 0 0.00 N old bans deleted n_ban_obj_test 0 0.00 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 0 0.00 N duplicate bans removed hcb_nolock 10300157 32.55 HCB Lookups without lock hcb_lock 1174513 3.71 HCB Lookups with lock hcb_insert 1174512 3.71 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 316483 1.00 Client uptime dir_dns_lookups 1051 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 1869574 5.91 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 0 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 12123851 38.31 Gunzip operations LCK.sms.creat 1 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 13584 0.04 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 1 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 3009407 9.51 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 1 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 26428567 83.51 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 1 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 2073113 6.55 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 629200 1.99 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 4411304 13.94 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 4493280 14.20 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 638013 2.02 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 1 0.00 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 2 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 22757276 71.91 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 1174563 3.71 Created locks LCK.objhdr.destroy 896364 2.83 Destroyed locks LCK.objhdr.locks 52930423 167.25 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 8814491 27.85 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 8496146 26.85 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 105440 0.33 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 8814517 27.85 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 315703 1.00 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 361716 1.14 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 3 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 11497749 36.33 Lock Operations LCK.backend.colls 0 0.00 Collisions SMF.s0.c_req 13566962 42.87 Allocator requests SMF.s0.c_fail 3893293 12.30 Allocator failures SMF.s0.c_bytes 639411638272 2020366.46 Bytes allocated SMF.s0.c_freed 624779067392 1974131.52 Bytes freed SMF.s0.g_alloc 1414918 . Allocations outstanding SMF.s0.g_bytes 14632570880 . Bytes outstanding SMF.s0.g_space 14229925888 . Bytes available SMF.s0.g_smf 1542019 . N struct smf SMF.s0.g_smf_frag 127100 . N small free smf SMF.s0.g_smf_large 1 . N large free smf SMA.Transient.c_req 1304195 4.12 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 54287399829 171533.38 Bytes allocated SMA.Transient.c_freed 54287399829 171533.38 Bytes freed SMA.Transient.g_alloc 0 . Allocations outstanding SMA.Transient.g_bytes 0 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available Thanks Andy ________________________________ The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andy.Lightfoot at ig.com Wed Nov 20 16:03:35 2013 From: Andy.Lightfoot at ig.com (Andy Lightfoot) Date: Wed, 20 Nov 2013 16:03:35 +0000 Subject: Logging to file with varnishlog and applying filters Message-ID: Hi, Has anyone had success with logging to file with varnishlog (using the -w argument) whilst also using filters on the data it is logging? Whenever, I log the varnishlog data to file it appears to ignore any filters I am applying and log the complete data (which is a huge amount of data!). Any hints on where I am going wrong or if this is possible would be gratefully received! Thanks Andy ________________________________ The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean.respen at gmail.com Wed Nov 20 19:25:29 2013 From: jean.respen at gmail.com (Jean Respen) Date: Wed, 20 Nov 2013 14:25:29 -0500 Subject: Varnish 503 error Message-ID: Hi all, I sometimes have the following error in my logs: 12 VCL_return c hash 12 VCL_call c miss fetch 12 Backend c 15 default default 12 FetchError c http first read error: -1 0 (No error recorded) 12 Backend c 11 default default 12 FetchError c http first read error: -1 0 (No error recorded) 12 VCL_call c error deliver 12 VCL_call c deliver deliver 12 TxProtocol c HTTP/1.1 12 TxStatus c 503 12 TxResponse c Service Unavailable 12 TxHeader c Server: Varnish Does anyone know what could happen? It is boring, as it gives a 503 error on around 1 over 5 requests. Thanks, Cheers, -- Jean Respen From ingvar at redpill-linpro.com Thu Nov 21 08:13:42 2013 From: ingvar at redpill-linpro.com (Ingvar Hagelund) Date: Thu, 21 Nov 2013 09:13:42 +0100 Subject: Varnish 3.0.5 rc 1 In-Reply-To: <20131118081137.GA10243@err.no> References: <20131118081137.GA10243@err.no> Message-ID: <528DC0B6.20107@redpill-linpro.com> Den 18. nov. 2013 09:11, skrev Tollef Fog Heen: > Hi all, > > we're in the process of getting 3.0.5 out the door and have a release > candidate. Please take a look at the packages from one of: > > https://jenkins.varnish-software.com/view/varnish-3.0/job/varnish-3.0-rpm-el6/ > https://jenkins.varnish-software.com/view/varnish-3.0/job/varnish-3.0-rpm-el5/ > https://jenkins.varnish-software.com/view/varnish-3.0/job/varnish-3.0-deb-debian-wheezy-amd64/lastSuccessfulBuild/artifact/deb-build/ > > or the source from > https://jenkins.varnish-software.com/view/varnish-3.0/job/varnish-3.0-src/ > > Feedback on this most welcome. Since this is a release candidate in the > stable release series, no changes to your configuration should be > necessary. > Rebuilt source with fedora spec. At first glance, this works fine on fedora 19. Ingvar From ruben at varnish-software.com Thu Nov 21 14:12:48 2013 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Thu, 21 Nov 2013 15:12:48 +0100 Subject: Reminder: Eight Varnish User Group Meeting in Berlin next week #VUG8 Message-ID: Hallo, Alles gut? Sch?n. Das ist gut. ** Please help us to spread the word to all Varnish & Web Performance enthusiasts you know ** In behalf of the Varnish Team, it is my pleasure to announce to you the next Varnish User Group Meeting, the eight of its kind, which is to be held in Berlin, Germany. #VUG8 is possible thanks to the sponsorship of Axel Springer, Fastly, Uplex and Varnish Software. * Varnish User Day is open for everyone and it will be at Axel Springer HQ on Thursday, November 28th, 2013. Agenda and general information is up [1] and there are a few free tickets left. Get one now! [2] * Varnish (Core) Developer meeting: will happen on Wednesday, November 27th, 2013. This event is invite only. See and use the wiki to see the venue, add discussion topics and other details [3] * Hacking Day / Community Roundtable: New for #VUG8! It's open for everyone (we have a few free seats left) and takes place on Friday, November 29th, 2013. Discussion topics are Better documentation, V4 Release & Parties, Varnish Local User Groups, Varnish Security Firewall, VMODs, Utilities, improving http://varnish.org and more. We are very much looking forward to meet the Varnish community in Berlin and spending time with you while making Varnish Cache better. Before I let you go: Do not forget to let your colleagues and friends know about the event AND use the #VUG8 tag in social media. Thanks! Wir sehen uns in Berlin! Links: [1] [2] [3] [4] [5] All the best, -- *Rub?n Romero* Community & Sales | Varnish Software AS Cell: +47 95964088 / Office: +47 21989260 Skype & Twitter: ruben_varnish We Make Websites Fly!Winner of the 2013 Red Herring Top 100 Global Awards -------------- next part -------------- An HTML attachment was scrubbed... URL: From theoverblue at gmail.com Sat Nov 23 18:31:15 2013 From: theoverblue at gmail.com (Mr OverBlue) Date: Sat, 23 Nov 2013 18:31:15 +0000 Subject: Compiling varnish cache on Mac OS 10.8.5 Message-ID: <5290F473.7050909@gmail.com> Hi, I have been trying to compile varnish source on Mac OS without sucess. At first i had a few errors because the AUTOMAKE_OPTIONS=subdir-objects wasn't present on Makefile.am. After adding that autogen _seems_ to work correctly but make is failing and i guess there is something wrong with autoheader... (CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh /opt/varnish-cache/build-aux/missing autoheader) (..) Making all in libvarnishapi python2.7 ./generate.py . ../.. make[3]: *** No rule to make target `vsl-tags.rst', needed by `all'. Stop. make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 I'm trying to compile the latest source from git (i have pulled 2 or 3 commits since i started this journey). Thanks in advance, Miguel Lopes -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- + glibtoolize --copy --force glibtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `build-aux'. glibtoolize: copying file `build-aux/ltmain.sh' glibtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'. glibtoolize: copying file `m4/libtool.m4' glibtoolize: copying file `m4/ltoptions.m4' glibtoolize: copying file `m4/ltsugar.m4' glibtoolize: copying file `m4/ltversion.m4' glibtoolize: copying file `m4/lt~obsolete.m4' + aclocal -I m4 + autoheader + automake --add-missing --copy --foreign + autoconf checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether gcc understands -c and -o together... yes checking how to run the C preprocessor... gcc -E checking for grep that handles long lines and -e... /usr/bin/grep checking for egrep... /usr/bin/grep -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking minix/config.h usability... no checking minix/config.h presence... no checking for minix/config.h... no checking whether it is safe to define __EXTENSIONS__... yes checking build system type... x86_64-apple-darwin12.5.0 checking host system type... x86_64-apple-darwin12.5.0 checking target system type... x86_64-apple-darwin12.5.0 checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... build-aux/install-sh -c -d checking for gawk... no checking for mawk... no checking for nawk... no checking for awk... awk checking whether make sets $(MAKE)... yes checking for style of include used by make... GNU checking whether make supports nested variables... yes checking dependency style of gcc... gcc3 checking whether make supports nested variables... (cached) yes checking how to print strings... printf checking for a sed that does not truncate output... /usr/bin/sed checking for fgrep... /usr/bin/grep -F checking for ld used by gcc... /usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld checking if the linker (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) is GNU ld... no checking for BSD- or MS-compatible name lister (nm)... /opt/local/bin/nm checking the name lister (/opt/local/bin/nm) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 196608 checking whether the shell understands some XSI constructs... yes checking whether the shell understands "+="... yes checking how to convert x86_64-apple-darwin12.5.0 file names to x86_64-apple-darwin12.5.0 format... func_convert_file_noop checking how to convert x86_64-apple-darwin12.5.0 file names to toolchain format... func_convert_file_noop checking for /usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld option to reload object files... -r checking for objdump... no checking how to recognize dependent libraries... pass_all checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for ar... ar checking for archiver @FILE support... no checking for strip... strip checking for ranlib... ranlib checking command to parse /opt/local/bin/nm output from gcc object... ok checking for sysroot... no checking for mt... no checking if : is a manifest tool... no checking for dsymutil... dsymutil checking for nmedit... nmedit checking for lipo... lipo checking for otool... otool checking for otool64... no checking for -single_module linker flag... yes checking for -exported_symbols_list linker flag... yes checking for -force_load linker flag... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fno-common -DPIC checking if gcc PIC flag -fno-common -DPIC works... yes checking if gcc static flag -static works... no checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) supports shared libraries... yes checking dynamic linker characteristics... darwin12.5.0 dyld checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... no checking for gcc... (cached) gcc checking whether we are using the GNU C compiler... (cached) yes checking whether gcc accepts -g... (cached) yes checking for gcc option to accept ISO C89... (cached) none needed checking whether gcc understands -c and -o together... (cached) yes checking for gcc option to accept ISO C99... -std=gnu99 checking for gcc -std=gnu99 option to accept ISO Standard C... (cached) -std=gnu99 checking how to run the C preprocessor... gcc -E checking for the pthreads library -lpthreads... no checking whether pthreads work without any flags... yes checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE checking if more special flags are required for pthreads... -D_THREAD_SAFE checking whether make sets $(MAKE)... (cached) yes checking for rst2man... no checking for rst2man.py... no configure: WARNING: rst2man not found - not building man pages checking for rst2html... no checking for rst2html.py... no configure: WARNING: rst2html not found - not building changelog checking for clock_gettime in -lrt... no checking for dlopen in -ldl... yes checking for NcursesW wide-character library... no checking for Ncurses library... yes checking for working ncurses/curses.h... no checking for working ncurses.h... yes checking for library containing pthread_create... none required checking for socket in -lsocket... no checking for getaddrinfo in -lnsl... no checking for pkg-config... /opt/local/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for PCRE... yes checking for LIBEDIT... yes checking for ANSI C header files... (cached) yes checking for sys/wait.h that is POSIX.1 compatible... yes checking whether time.h and sys/time.h may both be included... yes checking sys/param.h usability... yes checking sys/param.h presence... yes checking for sys/param.h... yes checking for sys/types.h... (cached) yes checking sys/endian.h usability... no checking sys/endian.h presence... no checking for sys/endian.h... no checking sys/filio.h usability... yes checking sys/filio.h presence... yes checking for sys/filio.h... yes checking for sys/mount.h... yes checking sys/socket.h usability... yes checking sys/socket.h presence... yes checking for sys/socket.h... yes checking sys/statvfs.h usability... yes checking sys/statvfs.h presence... yes checking for sys/statvfs.h... yes checking sys/vfs.h usability... no checking sys/vfs.h presence... no checking for sys/vfs.h... no checking endian.h usability... no checking endian.h presence... no checking for endian.h... no checking execinfo.h usability... yes checking execinfo.h presence... yes checking for execinfo.h... yes checking netinet/in.h usability... yes checking netinet/in.h presence... yes checking for netinet/in.h... yes checking for pthread_np.h... no checking stddef.h usability... yes checking stddef.h presence... yes checking for stddef.h... yes checking for stdlib.h... (cached) yes checking for unistd.h... (cached) yes checking priv.h usability... no checking priv.h presence... no checking for priv.h... no checking for an ANSI C-conforming const... yes checking for struct sockaddr.sa_len... yes checking return type of signal handlers... void checking for size_t... yes checking for vprintf... yes checking for _doprnt... no checking for strerror... yes checking whether strerror_r is declared... yes checking for strerror_r... yes checking whether strerror_r returns char *... no checking for dladdr... yes checking for socket... yes checking for strptime... yes checking for fmtcheck... yes checking for getdtablesize... yes checking for timegm... yes checking for nanosleep... yes checking for setppriv... no checking for pthread_set_name_np... no checking for pthread_mutex_isowned_np... no checking for pthread_timedjoin_np... no checking whether we have support for visibility attributes... yes checking for setproctitle... no checking for srandomdev... yes checking for library containing backtrace... none required checking whether daemon() works... no checking for special C compiler options needed for large files... no checking for _FILE_OFFSET_BITS value needed for large files... no checking for clock_gettime... no checking for gethrtime... no checking for kqueue... yes checking for epoll_ctl... no checking for port_create... no checking for python3... no checking for python3.1... no checking for python3.2... no checking for python2.7... python2.7 checking whether SO_ACCEPTFILTER is declared... no checking whether SO_RCVTIMEO works... yes checking whether SO_SNDTIMEO works... yes checking for TCP_KEEP(CNT|IDLE|INTVL) socket options... no checking if LD -Wl,--version-script works... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating bin/Makefile config.status: creating bin/varnishadm/Makefile config.status: creating bin/varnishd/Makefile config.status: creating bin/varnishlog/Makefile config.status: creating bin/varnishstat/Makefile config.status: creating bin/varnishtest/Makefile config.status: creating bin/varnishncsa/Makefile config.status: creating doc/Makefile config.status: creating doc/sphinx/Makefile config.status: creating doc/sphinx/conf.py config.status: creating etc/Makefile config.status: creating include/Makefile config.status: creating lib/Makefile config.status: creating lib/libvarnish/Makefile config.status: creating lib/libvarnishapi/Makefile config.status: creating lib/libvarnishtools/Makefile config.status: creating lib/libvarnishcompat/Makefile config.status: creating lib/libvcc/Makefile config.status: creating lib/libvgz/Makefile config.status: creating lib/libvmod_debug/Makefile config.status: creating lib/libvmod_std/Makefile config.status: creating lib/libvmod_directors/Makefile config.status: creating man/Makefile config.status: creating redhat/Makefile config.status: creating varnishapi.pc config.status: creating varnishapi-uninstalled.pc config.status: creating config.h config.status: executing depfiles commands config.status: executing libtool commands (CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh /opt/varnish-cache/build-aux/missing autoheader) rm -f stamp-h1 touch config.h.in cd . && /bin/sh ./config.status config.h config.status: creating config.h config.status: config.h is unchanged make all-recursive Making all in include make all-am mkdir -p tbl python2.7 ../lib/libvcc/generate.py .. .. Making all in lib Making all in libvarnishcompat CC daemon.lo CC execinfo.lo CC srandomdev.lo CCLD libvarnishcompat.la Making all in libvarnish CC libvarnish_la-vav.lo CC libvarnish_la-vas.lo CC libvarnish_la-binary_heap.lo CC libvarnish_la-vsub.lo CC libvarnish_la-cli_auth.lo CC libvarnish_la-cli_common.lo CC libvarnish_la-cli_serve.lo CC libvarnish_la-flopen.lo CC libvarnish_la-vnum.lo CC libvarnish_la-vtim.lo CC libvarnish_la-vtcp.lo CC libvarnish_la-vct.lo CC libvarnish_la-version.lo CC libvarnish_la-vev.lo CC libvarnish_la-vfil.lo CC libvarnish_la-vin.lo CC libvarnish_la-vlu.lo CC libvarnish_la-vmb.lo CC libvarnish_la-vpf.lo CC libvarnish_la-vre.lo CC libvarnish_la-vsa.lo CC libvarnish_la-vsb.lo CC libvarnish_la-vsha256.lo CC libvarnish_la-vss.lo CCLD libvarnish.la Making all in libvarnishapi python2.7 ./generate.py . ../.. make[3]: *** No rule to make target `vsl-tags.rst', needed by `all'. Stop. make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 From japrice at gmail.com Sun Nov 24 01:59:15 2013 From: japrice at gmail.com (Jason Price) Date: Sat, 23 Nov 2013 20:59:15 -0500 Subject: vmod_redis Message-ID: I'm working with vmod_redis to do some metrics gathering and sundry. But I think I've got a memory leak in there. Is there a recent, canonical implementation of vmod_redis? I've worked my way through github a couple of times trying to find 'the one', but to no avail. (ideally, the vmod_redis implementation would support the pipeline call) My symptoms: Varnish, vmod_redis, and redis all work beautifully, but after several hours, the redis connections start returning "Out of memory" errors. The server has stacks (10+GB) of unused ram, so that's not the problem. I believe my threads are bumping into 'thread_pool_workspace" or a similar limit, but I'm having challenges proving that assertion. I know redis isn't the problem because other clients can connect to it without issue at the same time varnish can't. Thank you for any pointers; Jason From japrice at gmail.com Sun Nov 24 03:24:53 2013 From: japrice at gmail.com (Jason Price) Date: Sat, 23 Nov 2013 22:24:53 -0500 Subject: vmod_redis In-Reply-To: References: Message-ID: Followup question: How would someone measure the current use of thread_pool_workspace? On Sat, Nov 23, 2013 at 8:59 PM, Jason Price wrote: > I'm working with vmod_redis to do some metrics gathering and sundry. > But I think I've got a memory leak in there. > > Is there a recent, canonical implementation of vmod_redis? I've > worked my way through github a couple of times trying to find 'the > one', but to no avail. (ideally, the vmod_redis implementation would > support the pipeline call) > > My symptoms: Varnish, vmod_redis, and redis all work beautifully, but > after several hours, the redis connections start returning "Out of > memory" errors. The server has stacks (10+GB) of unused ram, so > that's not the problem. I believe my threads are bumping into > 'thread_pool_workspace" or a similar limit, but I'm having challenges > proving that assertion. I know redis isn't the problem because other > clients can connect to it without issue at the same time varnish > can't. > > Thank you for any pointers; > Jason From tobias.eichelbroenner at lamp-solutions.de Sun Nov 24 11:48:35 2013 From: tobias.eichelbroenner at lamp-solutions.de (=?ISO-8859-1?Q?Tobias_Eichelbr=F6nner?=) Date: Sun, 24 Nov 2013 12:48:35 +0100 Subject: vmod_redis In-Reply-To: References: Message-ID: <5291E793.6030607@lamp-solutions.de> Hi Jason, > I'm working with vmod_redis to do some metrics gathering and sundry. > But I think I've got a memory leak in there. I didn?t had the time to have some deeper thoughts on the problems you described, yet. But we use vmod_redis on several servers with quite a workload (150 req/s) and never recognized any memory issues. Sincerely, Tobias -- LAMP solutions GmbH Gostenhofer Hauptstrasse 35 90443 Nuernberg Amtsgericht Nuernberg: HRB 22366 Geschaeftsfuehrer: Heiko Schubert Es gelten unsere allgemeinen Geschaeftsbedingungen. http://www.lamp-solutions.de/agbs/ Telefon : 0911 / 376 516 0 Fax : 0911 / 376 516 11 E-Mail : support at lamp-solutions.de Web : www.lamp-solutions.de Facebook : http://www.facebook.com/LAMPsolutions Twitter : http://twitter.com/#!/lampsolutions From japrice at gmail.com Sun Nov 24 14:35:14 2013 From: japrice at gmail.com (Jason Price) Date: Sun, 24 Nov 2013 09:35:14 -0500 Subject: vmod_redis In-Reply-To: <5291E793.6030607@lamp-solutions.de> References: <5291E793.6030607@lamp-solutions.de> Message-ID: Which implementation of vmod_redis do you use? Link? On Nov 24, 2013 7:09 AM, "Tobias Eichelbr?nner" < tobias.eichelbroenner at lamp-solutions.de> wrote: > Hi Jason, > > > I'm working with vmod_redis to do some metrics gathering and sundry. > > But I think I've got a memory leak in there. > > I didn?t had the time to have some deeper thoughts on the problems you > described, yet. But we use vmod_redis on several servers with quite a > workload (150 req/s) and never recognized any memory issues. > > Sincerely, > > Tobias > > -- > LAMP solutions GmbH > Gostenhofer Hauptstrasse 35 > 90443 Nuernberg > > Amtsgericht Nuernberg: HRB 22366 > Geschaeftsfuehrer: Heiko Schubert > > Es gelten unsere allgemeinen Geschaeftsbedingungen. > http://www.lamp-solutions.de/agbs/ > > Telefon : 0911 / 376 516 0 > Fax : 0911 / 376 516 11 > E-Mail : support at lamp-solutions.de > Web : www.lamp-solutions.de > Facebook : http://www.facebook.com/LAMPsolutions > Twitter : http://twitter.com/#!/lampsolutions > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobias.eichelbroenner at lamp-solutions.de Sun Nov 24 18:53:58 2013 From: tobias.eichelbroenner at lamp-solutions.de (=?ISO-8859-1?Q?Tobias_Eichelbr=F6nner?=) Date: Sun, 24 Nov 2013 19:53:58 +0100 Subject: vmod_redis In-Reply-To: References: <5291E793.6030607@lamp-solutions.de> Message-ID: <52924B46.8080903@lamp-solutions.de> Hi Jason, > Which implementation of vmod_redis do you use? Link? https://github.com/zephirworks/libvmod-redis Sincerely, Tobias -- LAMP solutions GmbH Gostenhofer Hauptstrasse 35 90443 Nuernberg Amtsgericht Nuernberg: HRB 22366 Geschaeftsfuehrer: Heiko Schubert Es gelten unsere allgemeinen Geschaeftsbedingungen. http://www.lamp-solutions.de/agbs/ Telefon : 0911 / 376 516 0 Fax : 0911 / 376 516 11 E-Mail : support at lamp-solutions.de Web : www.lamp-solutions.de Facebook : http://www.facebook.com/LAMPsolutions Twitter : http://twitter.com/#!/lampsolutions From t.honacker at googlemail.com Wed Nov 27 10:35:22 2013 From: t.honacker at googlemail.com (Tobias Honacker) Date: Wed, 27 Nov 2013 11:35:22 +0100 Subject: Varnish - set backend if statuscode is 404 Message-ID: Hey guys, i?ve got one problem and don?t know how to solve this problem. I have to use another backend if the req.url matches AND HTTP Status is 404. sub vcl_recv { if (req.url ~ ?^/path/to/images") { set req.backend = p1_backend; set req.http.host = "www.mysite.de?; } How to solve the 404 Status? varnishd -V varnishd (varnish-3.0.2 revision 55e70a4) Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2011 Varnish Software AS Thanks for any information. Best regards, Tobias -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Wed Nov 27 15:17:56 2013 From: perbu at varnish-software.com (Per Buer) Date: Wed, 27 Nov 2013 16:17:56 +0100 Subject: Varnish - set backend if statuscode is 404 In-Reply-To: References: Message-ID: Hi Tobias. Check the status in vcl_fetch. If it is then enable saint mode and call restart. Make sure saintmode_threshold is 0 so it wont disable a backend based on the number of 404s. Per. On Wed, Nov 27, 2013 at 11:35 AM, Tobias Honacker wrote: > Hey guys, > > i?ve got one problem and don?t know how to solve this problem. > I have to use another backend if the req.url matches AND HTTP Status is > 404. > > sub vcl_recv { > if (req.url ~ ?^/path/to/images") { > set req.backend = p1_backend; > set req.http.host = "www.mysite.de?; > } > > How to solve the 404 Status? > > > varnishd -V > varnishd (varnish-3.0.2 revision 55e70a4) > Copyright (c) 2006 Verdens Gang AS > Copyright (c) 2006-2011 Varnish Software AS > > > Thanks for any information. > > > Best regards, > Tobias > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- *Per Buer* CTO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Europe Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From duarte.meneses at cern.ch Wed Nov 27 17:20:52 2013 From: duarte.meneses at cern.ch (Duarte Bacelar De Begonha De Meneses) Date: Wed, 27 Nov 2013 17:20:52 +0000 Subject: Streaming of requests with Range Message-ID: Hi, I need to cache requests of chunks of files using the "Range" HTTP header. The problem is that in some cases the files are very large, and Varnish downloads the entire file from the backend before passing the requested part to the client. I've seen that some developments have been made to add support to streaming in a separate branch, since some time ago. Does currently this feature will return the requested range as soon as it gets downloaded, and then keeps on transferring the rest, to have then the entire file on cache? This would be a behavior very similar to what Squid offers. After checking the change logs of the latest releases (up to 3.0.4), I suppose this branch hasn't been merged with the main branch. Is the 3.0.2s the latest release supporting streaming? Also, going a bit offtopic, I feel a bit confused about how Varnish processes HTTP cache-related headers, such has If-Modified-Since, Cache-control, etc. For example, I saw that it respects the "If-Modified-Since" in a transparent way, and returns 304 if needed. What would be the best way to override this behavior and ignore the header? Would it be to simply unset the header in vcl_recv? Thanks and best regards. Duarte -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu Nov 28 15:15:35 2013 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 28 Nov 2013 16:15:35 +0100 Subject: Varnish - set backend if statuscode is 404 In-Reply-To: References: Message-ID: <20131128151535.GA27307@mushkin.tanso.net> If you manage to do this, I would much appreciate you post a recipie :-) I've been struggeling trying to achieve the same thing, without success.. Actually my setup is a bit different, we have a director, and just want to check the other backend if the first backend gives 404, but the same solution might apply. backend backend1 { .host = "backend1.example.com"; .probe = { .url = "/alive.html"; .timeout = 34 ms; .interval = 2s; .window = 10; .threshold = 8; } } backend backend2 { .host = "backend2.example.com"; .probe = { .url = "/alive.html"; .timeout = 34 ms; .interval = 2s; .window = 10; .threshold = 8; } } director dir1 client { { .backend = backend1; .weight = 1; } { .backend = backend2; .weight = 1; } } sub vcl_recv { if ( req.http.host ~ "(?i)www.example.com$" ) { set req.backend = dir1; return(lookup); } } -jf From t.honacker at googlemail.com Thu Nov 28 15:30:14 2013 From: t.honacker at googlemail.com (Tobias Honacker) Date: Thu, 28 Nov 2013 16:30:14 +0100 Subject: Varnish - set backend if statuscode is 404 In-Reply-To: <20131128151535.GA27307@mushkin.tanso.net> References: <20131128151535.GA27307@mushkin.tanso.net> Message-ID: Hi Jan, This is my solution: sub vcl_recv { // Redirect 404er Pictures - vcl_fetch will restart request and set new header to match with this rule if (req.http.host == "new-header.de") { if (req.url ~ "^/var/storage/images") { set req.backend = new_backend; } } } sub vcl_fetch { if ((beresp.status == 404) && (req.url ~ "^/var/storage/images")) { set req.http.Host = "new-header.de"; return(restart); } } You have to set a new header and restart the request thats it. Best regards, Tobias Am 28.11.13 16:15 schrieb "Jan-Frode Myklebust" unter : >If you manage to do this, I would much appreciate you post a recipie :-) >I've been struggeling trying to achieve the same thing, without >success.. > >Actually my setup is a bit different, we have a director, and just want >to check the other backend if the first backend gives 404, but the same >solution might apply. > >backend backend1 { > .host = "backend1.example.com"; > .probe = { > .url = "/alive.html"; > .timeout = 34 ms; > .interval = 2s; > .window = 10; > .threshold = 8; > } >} >backend backend2 { > .host = "backend2.example.com"; > .probe = { > .url = "/alive.html"; > .timeout = 34 ms; > .interval = 2s; > .window = 10; > .threshold = 8; > } >} >director dir1 client { > { .backend = backend1; .weight = 1; } > { .backend = backend2; .weight = 1; } >} >sub vcl_recv { > if ( req.http.host ~ "(?i)www.example.com$" ) { > set req.backend = dir1; > return(lookup); > } >} > > > > -jf From infos at opendoc.net Thu Nov 28 17:29:59 2013 From: infos at opendoc.net (Alexandre) Date: Thu, 28 Nov 2013 18:29:59 +0100 Subject: Cache invalidation with regex Message-ID: <52977D97.4040104@opendoc.net> Hi all, I will wish to invalidate my cache using a regex. example: ^/image.php?a=1&b=2* I would like to disable all url starting with ^/image.php?a=1&b=2 Do you have an idea for my problem? Thanks for any information Alexandre From numard at gmail.com Fri Nov 29 04:04:42 2013 From: numard at gmail.com (Norberto Meijome) Date: Fri, 29 Nov 2013 15:04:42 +1100 Subject: Varnish - set backend if statuscode is 404 In-Reply-To: References: <20131128151535.GA27307@mushkin.tanso.net> Message-ID: +1, this is how we do this ( and similar tricks) @per, you suggested saint mode. Why? What would it do better than Tobias' solution? Thx, On 29/11/2013 3:10 AM, "Tobias Honacker" wrote: > Hi Jan, > > This is my solution: > > sub vcl_recv { > > // Redirect 404er Pictures - vcl_fetch will restart request and set new > header to match with this rule > if (req.http.host == "new-header.de") { > if (req.url ~ "^/var/storage/images") { > set req.backend = new_backend; > } > } > } > > > sub vcl_fetch { > > if ((beresp.status == 404) && (req.url ~ "^/var/storage/images")) { > set req.http.Host = "new-header.de"; > return(restart); > } > } > > > You have to set a new header and restart the request thats it. > > > Best regards, > Tobias > > Am 28.11.13 16:15 schrieb "Jan-Frode Myklebust" unter >: > > >If you manage to do this, I would much appreciate you post a recipie :-) > >I've been struggeling trying to achieve the same thing, without > >success.. > > > >Actually my setup is a bit different, we have a director, and just want > >to check the other backend if the first backend gives 404, but the same > >solution might apply. > > > >backend backend1 { > > .host = "backend1.example.com"; > > .probe = { > > .url = "/alive.html"; > > .timeout = 34 ms; > > .interval = 2s; > > .window = 10; > > .threshold = 8; > > } > >} > >backend backend2 { > > .host = "backend2.example.com"; > > .probe = { > > .url = "/alive.html"; > > .timeout = 34 ms; > > .interval = 2s; > > .window = 10; > > .threshold = 8; > > } > >} > >director dir1 client { > > { .backend = backend1; .weight = 1; } > > { .backend = backend2; .weight = 1; } > >} > >sub vcl_recv { > > if ( req.http.host ~ "(?i)www.example.com$" ) { > > set req.backend = dir1; > > return(lookup); > > } > >} > > > > > > > > -jf > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Nov 29 08:06:15 2013 From: perbu at varnish-software.com (Per Buer) Date: Fri, 29 Nov 2013 09:06:15 +0100 Subject: Varnish - set backend if statuscode is 404 In-Reply-To: References: <20131128151535.GA27307@mushkin.tanso.net> Message-ID: On Fri, Nov 29, 2013 at 5:04 AM, Norberto Meijome wrote: > +1, this is how we do this ( and similar tricks) > > @per, you suggested saint mode. Why? What would it do better than Tobias' > solution? > Saint mode works if you have two backends in the same director. Then it blacklists that object from that backend for a certain time. A restart would then direct the request to the other backend. Tobias solution is to have two backends. The other one is only used when the first one fails. Per. > Thx, > On 29/11/2013 3:10 AM, "Tobias Honacker" > wrote: > >> Hi Jan, >> >> This is my solution: >> >> sub vcl_recv { >> >> // Redirect 404er Pictures - vcl_fetch will restart request and set new >> header to match with this rule >> if (req.http.host == "new-header.de") { >> if (req.url ~ "^/var/storage/images") { >> set req.backend = new_backend; >> } >> } >> } >> >> >> sub vcl_fetch { >> >> if ((beresp.status == 404) && (req.url ~ "^/var/storage/images")) >> { >> set req.http.Host = "new-header.de"; >> return(restart); >> } >> } >> >> >> You have to set a new header and restart the request thats it. >> >> >> Best regards, >> Tobias >> >> Am 28.11.13 16:15 schrieb "Jan-Frode Myklebust" unter > >: >> >> >If you manage to do this, I would much appreciate you post a recipie :-) >> >I've been struggeling trying to achieve the same thing, without >> >success.. >> > >> >Actually my setup is a bit different, we have a director, and just want >> >to check the other backend if the first backend gives 404, but the same >> >solution might apply. >> > >> >backend backend1 { >> > .host = "backend1.example.com"; >> > .probe = { >> > .url = "/alive.html"; >> > .timeout = 34 ms; >> > .interval = 2s; >> > .window = 10; >> > .threshold = 8; >> > } >> >} >> >backend backend2 { >> > .host = "backend2.example.com"; >> > .probe = { >> > .url = "/alive.html"; >> > .timeout = 34 ms; >> > .interval = 2s; >> > .window = 10; >> > .threshold = 8; >> > } >> >} >> >director dir1 client { >> > { .backend = backend1; .weight = 1; } >> > { .backend = backend2; .weight = 1; } >> >} >> >sub vcl_recv { >> > if ( req.http.host ~ "(?i)www.example.com$" ) { >> > set req.backend = dir1; >> > return(lookup); >> > } >> >} >> > >> > >> > >> > -jf >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- *Per Buer* CTO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Europe Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Fri Nov 29 10:01:19 2013 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 29 Nov 2013 11:01:19 +0100 Subject: Cache invalidation with regex In-Reply-To: <52977D97.4040104@opendoc.net> References: <52977D97.4040104@opendoc.net> Message-ID: <529865EF.90309@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/28/2013 06:29 PM, Alexandre wrote: > > I will wish to invalidate my cache using a regex. > > example: ^/image.php?a=1&b=2* > > I would like to disable all url starting with ^/image.php?a=1&b=2 > > Do you have an idea for my problem? The '*' in a regular expression is not like a file glob, it means that the character before it, in your case the '2', can appear 0 or more times. So your regex matches anything starting with: /image.php?a=1&b= If you don't care what comes after 'b=2', you can just end the regex there -- a regex doesn't have to describe a string all the way from beginning to end. ^/image.php?a=1&b=2 ... should do what you want. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJSmGXkAAoJEOUwvh9pJNURmLkQAIxu/g4T27Fr+LcYoJGFlAPE cGSKwaVGVaKKnIWTIjgGIsG3s45sAk40y1KxR1LqyYHnLa9bkz4U0+ifCVthT2wM +SNo7U7qkbywwJNbkMJjRHGXi+sIOf6EnkR67SBfynvmkVtQhOE0X0HJZgHQ6hvA h7wcMz2t/lPuESGs1mt1y60+HFFL76/lJOQXm7JbA9GBKlA3ibsMbW2f90ZUDQRG AOumsa8O8BeoGLn4dYfdmWwhG+wpIo3ztAPy7cZx9u1eY4KulhYJ90K3ErBvvr1b CAXbk3BMvCFH/HPbHe2NJtyk26MksQ90yoYYxN/F8rCTFhUCHrlD2iVjrHGNndaN sPCuSMEDiQR95VYWXdT81IkXBLBMYTJvLx0n0p95XH40dkhUf5hV1gS6akuiq9nC d6MeoX1qWOYFIIJ1gEGiJRf38Nx0yr/+I+zS118FfmgMZeGccjogMdgpQqpSU3jo IPNISPIwZlIRuCUn0Da/6Sykvosr9lhET2kVhDlKZ1b0Q4c8GEFIWQCOJpBGd6Mp gcqne0oRW3arOUwaeEkpTTibqFZdSCUp24tJKvgJUhowfzCFlOWUYziiVq4FvRo3 qFqp8ZT0ZqnCid7bnV7xvZ43rOOnBVb+mwYeUgsa4FpYxtvfZQznich7uKH/pnQQ kVXVQ+X7ML3NVOzzurvU =I/IM -----END PGP SIGNATURE----- From infos at opendoc.net Fri Nov 29 10:08:46 2013 From: infos at opendoc.net (Alexandre) Date: Fri, 29 Nov 2013 11:08:46 +0100 Subject: Cache invalidation with regex In-Reply-To: <529865EF.90309@uplex.de> References: <52977D97.4040104@opendoc.net> <529865EF.90309@uplex.de> Message-ID: <529867AE.2050101@opendoc.net> Hi, Thank you for your information, but what is the relation with my question ? This is an example. Do you have an idea for the cache invalidation ? Thx Alex. On 29/11/13 11:01, Geoff Simmons wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 11/28/2013 06:29 PM, Alexandre wrote: >> >> I will wish to invalidate my cache using a regex. >> >> example: ^/image.php?a=1&b=2* >> >> I would like to disable all url starting with ^/image.php?a=1&b=2 >> >> Do you have an idea for my problem? > > The '*' in a regular expression is not like a file glob, it means that > the character before it, in your case the '2', can appear 0 or more times. > > So your regex matches anything starting with: > > /image.php?a=1&b= > > If you don't care what comes after 'b=2', you can just end the regex > there -- a regex doesn't have to describe a string all the way from > beginning to end. > > ^/image.php?a=1&b=2 > > ... should do what you want. > > > Best, > Geoff > - -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQIcBAEBCAAGBQJSmGXkAAoJEOUwvh9pJNURmLkQAIxu/g4T27Fr+LcYoJGFlAPE > cGSKwaVGVaKKnIWTIjgGIsG3s45sAk40y1KxR1LqyYHnLa9bkz4U0+ifCVthT2wM > +SNo7U7qkbywwJNbkMJjRHGXi+sIOf6EnkR67SBfynvmkVtQhOE0X0HJZgHQ6hvA > h7wcMz2t/lPuESGs1mt1y60+HFFL76/lJOQXm7JbA9GBKlA3ibsMbW2f90ZUDQRG > AOumsa8O8BeoGLn4dYfdmWwhG+wpIo3ztAPy7cZx9u1eY4KulhYJ90K3ErBvvr1b > CAXbk3BMvCFH/HPbHe2NJtyk26MksQ90yoYYxN/F8rCTFhUCHrlD2iVjrHGNndaN > sPCuSMEDiQR95VYWXdT81IkXBLBMYTJvLx0n0p95XH40dkhUf5hV1gS6akuiq9nC > d6MeoX1qWOYFIIJ1gEGiJRf38Nx0yr/+I+zS118FfmgMZeGccjogMdgpQqpSU3jo > IPNISPIwZlIRuCUn0Da/6Sykvosr9lhET2kVhDlKZ1b0Q4c8GEFIWQCOJpBGd6Mp > gcqne0oRW3arOUwaeEkpTTibqFZdSCUp24tJKvgJUhowfzCFlOWUYziiVq4FvRo3 > qFqp8ZT0ZqnCid7bnV7xvZ43rOOnBVb+mwYeUgsa4FpYxtvfZQznich7uKH/pnQQ > kVXVQ+X7ML3NVOzzurvU > =I/IM > -----END PGP SIGNATURE----- > From geoff at uplex.de Fri Nov 29 10:40:32 2013 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 29 Nov 2013 11:40:32 +0100 Subject: Cache invalidation with regex In-Reply-To: <529867AE.2050101@opendoc.net> References: <52977D97.4040104@opendoc.net> <529865EF.90309@uplex.de> <529867AE.2050101@opendoc.net> Message-ID: <52986F20.1010205@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/29/2013 11:08 AM, Alexandre wrote: > > Thank you for your information, but what is the relation with my > question ? This is an example. Do you have an idea for the cache > invalidation ? Since your regex didn't match your URL example, I thought that was your problem. What is your problem then? If you question is about how to go about cache invalidation at all, you can use the ban command either with varnishadm or in VCL, as described here: https://www.varnish-cache.org/docs/3.0/tutorial/purging.html#bans Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJSmG8gAAoJEOUwvh9pJNURC4kP/1RmmoV3NCiwbxbHiJauI/eI kpNSg79ZZHBYVu2htyMLJJ3AQ3jiJDGPuhbvu6bu7N0LS5JEjohs4vqQ79W5514G aWvyj9tAeFdIPGAjdApgPAfB6pKsH876XRjHq6I4lxPC9kbBlOaLXYmjofAN5szs Y7nqHcA8f05V3H+pCpPrh//606vIKFlZwy+jk02lOXW6bc9ccSUu5wcF+WcEmKDW GgnI7vE8R8n13JmLwtcEgcGaVlUqG+yIq1v+rxggY7EbdkkqiQMOf3KUwz0a3Y4o wJHclPETyqBOVEWjGKQNtv7pGG1pbl+4ctCQvcomHv4f5g+AVwH4FV0RXl6InmQf lwIoE5YZ+ZSJSkNcx3OhKukjgrOt86YGQjDhtzDwZXvfQVYEDB32oqPyEM6hzAVM VMkqKlXDGItdZeUt74ysPV9fI6gz8aAo/daulty0XeuG6AWgsgGZkT9R4YCoj5aJ Icl8xgtvzVJDGbZsPEmXU+6nEXPkTbzidgzqW0FTk2MwX66/4XOQRsOJA4enFaqU 5O3FGTOC6C4lUaPNZ6D3OMBwAFqQ0eSEpGsr2Ddatu1rjpLNxDJMbjKdUL0GgFeU PEroFYewozOY25uQgPzXicwAoAvwjq1y/9vMOEE8dQoFHAKOkywpjnFf2edZaO+5 WdIpcofF2M+TnGclTcwV =bP65 -----END PGP SIGNATURE----- From janfrode at tanso.net Fri Nov 29 12:15:47 2013 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 29 Nov 2013 13:15:47 +0100 Subject: Varnish - set backend if statuscode is 404 In-Reply-To: References: <20131128151535.GA27307@mushkin.tanso.net> Message-ID: <20131129121547.GA17178@mushkin.tanso.net> On Fri, Nov 29, 2013 at 09:06:15AM +0100, Per Buer wrote: > > > > @per, you suggested saint mode. Why? What would it do better than Tobias' > > solution? > > Saint mode works if you have two backends in the same director. Then it > blacklists that object from that backend for a certain time. A restart > would then direct the request to the other backend. > > Tobias solution is to have two backends. The other one is only used when > the first one fails. > Ideally we want both. Our two hosts in the director are fully functional, but one of them might not have received a given file yet. So we want to try the other backend in the director, not mark it as faulty when it gets the 404. Do you see any way to maybe combine Tobias' solution with a director? -jf From perbu at varnish-software.com Fri Nov 29 12:24:59 2013 From: perbu at varnish-software.com (Per Buer) Date: Fri, 29 Nov 2013 13:24:59 +0100 Subject: Varnish - set backend if statuscode is 404 In-Reply-To: <20131129121547.GA17178@mushkin.tanso.net> References: <20131128151535.GA27307@mushkin.tanso.net> <20131129121547.GA17178@mushkin.tanso.net> Message-ID: On Fri, Nov 29, 2013 at 1:15 PM, Jan-Frode Myklebust wrote: > On Fri, Nov 29, 2013 at 09:06:15AM +0100, Per Buer wrote: > > > > > > @per, you suggested saint mode. Why? What would it do better than > Tobias' > > > solution? > > > > Saint mode works if you have two backends in the same director. Then it > > blacklists that object from that backend for a certain time. A restart > > would then direct the request to the other backend. > > > > Tobias solution is to have two backends. The other one is only used when > > the first one fails. > > > > Ideally we want both. Our two hosts in the director are fully > functional, but one of them might not have received a given file yet. So > we want to try the other backend in the director, not mark it as faulty > when it gets the 404. > I'm pretty sure you didn't read my reply properly. :-P Note that the blacklist is temporary. If you know that you spend up to five seconds updating your server you set it to 5s. -- *Per Buer* CTO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Europe Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From cosimo at streppone.it Fri Nov 29 12:49:26 2013 From: cosimo at streppone.it (Cosimo Streppone) Date: Fri, 29 Nov 2013 13:49:26 +0100 Subject: Cache invalidation with regex In-Reply-To: <529865EF.90309@uplex.de> References: <52977D97.4040104@opendoc.net> <529865EF.90309@uplex.de> Message-ID: <52988D56.2050101@streppone.it> On 11/29/2013 11:01 AM, Geoff Simmons wrote: > > On 11/28/2013 06:29 PM, Alexandre wrote: >> >> I will wish to invalidate my cache using a regex. >> >> example: ^/image.php?a=1&b=2* >> >> I would like to disable all url starting with ^/image.php?a=1&b=2 >> >> Do you have an idea for my problem? > > The '*' in a regular expression is not like a file glob, it means that > the character before it, in your case the '2', can appear 0 or more times. > > So your regex matches anything starting with: > > /image.php?a=1&b= I made a simple shell script wrapper to the varnish admin. It is available here: https://gist.github.com/cosimo/890217 If you run it with: $ ./purge-cache.sh -h it will show some usage instructions. In particular: $ ./purge-cache -r '^/image\.php\?a=1&b=2.*' should purge the URLs you asked for. It worked for me at least, and I added to my varnish puppet module, to always have it available alongside reload-vcl etc... -- Cosimo From infos at opendoc.net Fri Nov 29 13:18:17 2013 From: infos at opendoc.net (Alexandre) Date: Fri, 29 Nov 2013 14:18:17 +0100 Subject: Cache invalidation with regex In-Reply-To: <52988D56.2050101@streppone.it> References: <52977D97.4040104@opendoc.net> <529865EF.90309@uplex.de> <52988D56.2050101@streppone.it> Message-ID: <52989419.5000607@opendoc.net> Thank you very much. This is exactly what I want. I'll try to test the script in the day. Good day. Alex. On 29/11/13 13:49, Cosimo Streppone wrote: > On 11/29/2013 11:01 AM, Geoff Simmons wrote: >> >> On 11/28/2013 06:29 PM, Alexandre wrote: >>> >>> I will wish to invalidate my cache using a regex. >>> >>> example: ^/image.php?a=1&b=2* >>> >>> I would like to disable all url starting with ^/image.php?a=1&b=2 >>> >>> Do you have an idea for my problem? >> >> The '*' in a regular expression is not like a file glob, it means that >> the character before it, in your case the '2', can appear 0 or more times. >> >> So your regex matches anything starting with: >> >> /image.php?a=1&b= > > I made a simple shell script wrapper to the varnish admin. > It is available here: > > https://gist.github.com/cosimo/890217 > > If you run it with: > > $ ./purge-cache.sh -h > > it will show some usage instructions. > In particular: > > $ ./purge-cache -r '^/image\.php\?a=1&b=2.*' > > should purge the URLs you asked for. > > It worked for me at least, and I added > to my varnish puppet module, to always have it available > alongside reload-vcl etc... > From stef at scaleengine.com Fri Nov 29 13:29:39 2013 From: stef at scaleengine.com (Stefan Caunter) Date: Fri, 29 Nov 2013 08:29:39 -0500 Subject: Varnish - set backend if statuscode is 404 In-Reply-To: <20131129121547.GA17178@mushkin.tanso.net> References: <20131128151535.GA27307@mushkin.tanso.net> <20131129121547.GA17178@mushkin.tanso.net> Message-ID: On Fri, Nov 29, 2013 at 7:15 AM, Jan-Frode Myklebust wrote: > On Fri, Nov 29, 2013 at 09:06:15AM +0100, Per Buer wrote: >> > >> > @per, you suggested saint mode. Why? What would it do better than Tobias' >> > solution? >> >> Saint mode works if you have two backends in the same director. Then it >> blacklists that object from that backend for a certain time. A restart >> would then direct the request to the other backend. >> >> Tobias solution is to have two backends. The other one is only used when >> the first one fails. >> > > Ideally we want both. Our two hosts in the director are fully > functional, but one of them might not have received a given file yet. So > we want to try the other backend in the director, not mark it as faulty > when it gets the 404. > > Do you see any way to maybe combine Tobias' solution with a director? > in fetch if (beresp.status == 404) { set beresp.http.X-Some-Edge-Restart = "True " beresp.status; set beresp.ttl = 0s; restart; } in recv if (req.restarts == 0) { set req.backend = b1; } if (req.restarts == 1) { set req.backend = b2; } if (req.restarts == 2) { set req.backend = b3; } if (req.restarts == 3) { set req.backend = b4; } } we use this for a particular case where each backend has unique content, and we want varnish to be point of access for application varnish goes and gets the file from any of the possible locations which is faster than trying to create an authoritative archive with rsync or some other silly solution ---- Stefan Caunter E: stefan.caunter at scaleengine.com Toronto Canada From perbu at varnish-software.com Fri Nov 29 13:51:23 2013 From: perbu at varnish-software.com (Per Buer) Date: Fri, 29 Nov 2013 14:51:23 +0100 Subject: Varnish - set backend if statuscode is 404 In-Reply-To: References: <20131128151535.GA27307@mushkin.tanso.net> <20131129121547.GA17178@mushkin.tanso.net> Message-ID: I've written a short blog post on the subject since this seems to come up from time to time. https://www.varnish-software.com/blog/404-handling-varnish-cache I need to figure out how we'll deal with this in 4.0. Per. On Fri, Nov 29, 2013 at 1:24 PM, Per Buer wrote: > On Fri, Nov 29, 2013 at 1:15 PM, Jan-Frode Myklebust wrote: > >> On Fri, Nov 29, 2013 at 09:06:15AM +0100, Per Buer wrote: >> > > >> > > @per, you suggested saint mode. Why? What would it do better than >> Tobias' >> > > solution? >> > >> > Saint mode works if you have two backends in the same director. Then it >> > blacklists that object from that backend for a certain time. A restart >> > would then direct the request to the other backend. >> > >> > Tobias solution is to have two backends. The other one is only used when >> > the first one fails. >> > >> >> Ideally we want both. Our two hosts in the director are fully >> functional, but one of them might not have received a given file yet. So >> we want to try the other backend in the director, not mark it as faulty >> when it gets the 404. >> > > I'm pretty sure you didn't read my reply properly. :-P > > Note that the blacklist is temporary. If you know that you spend up to > five seconds updating your server you set it to 5s. > > > -- > *Per Buer* > CTO | Varnish Software AS > Phone: +47 958 39 117 | Skype: per.buer > We Make Websites Fly! > > Winner of the Red Herring Top 100 Europe Award 2013 > > > -- *Per Buer* CTO | Varnish Software AS Phone: +47 958 39 117 | Skype: per.buer We Make Websites Fly! Winner of the Red Herring Top 100 Europe Award 2013 -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Fri Nov 29 14:11:27 2013 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Fri, 29 Nov 2013 15:11:27 +0100 Subject: Varnish - set backend if statuscode is 404 In-Reply-To: References: <20131128151535.GA27307@mushkin.tanso.net> <20131129121547.GA17178@mushkin.tanso.net> Message-ID: <20131129141127.GA24630@mushkin.tanso.net> On Fri, Nov 29, 2013 at 02:51:23PM +0100, Per Buer wrote: > I've written a short blog post on the subject since this seems to come up > from time to time. > > https://www.varnish-software.com/blog/404-handling-varnish-cache > Perfect, thanks! -jf