From Niklas.Donath at lvdigital.de Thu Mar 2 09:18:13 2017 From: Niklas.Donath at lvdigital.de (Donath, Niklas) Date: Thu, 2 Mar 2017 09:18:13 +0000 Subject: HTTP Digest Authentication in varnish4 Message-ID: <805A0E95-1802-4B45-8E7F-6F0949DF5990@lv.de> Hi there, i am currently trying to put varnish in front of a restful webservice with an existing http digest authentication. Does anyone already had this use case and could give me a hint or an example code on how to set this up properly? I didn?t find useful information in the varnish docs. The e-book ?Getting started with Varnish Cache? only scratches that topic. Examples I found about basic authentication do not really match because the authentication is implemented on varnish side only. I would like to pass through the authentication to the backend because I don?t want to add user handling in varnish itself. In my understanding, the authentication works as follows: - 1st request: no authentication, passed to the backend, backend response with nonce in header - 2nd request: client sends user credentials, passed to the backend, server gives proof (200) or fail (401) - 3rd request: probably cache hit, request with auth credentials is ?forked? as HEAD request, sent to the backend, server gives proof (200) or fail (401) Am I getting this right, am I probably missing something? Thank you in advance for any input and/or feedback! Greetings, Niklas -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Mar 2 09:39:19 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 2 Mar 2017 10:39:19 +0100 Subject: HTTP Digest Authentication in varnish4 In-Reply-To: <805A0E95-1802-4B45-8E7F-6F0949DF5990@lv.de> References: <805A0E95-1802-4B45-8E7F-6F0949DF5990@lv.de> Message-ID: You should be able to accomplish this with restarts (the wayezpublish does it: https://github.com/ezsystems/ezpublish-community/blob/master/doc/varnish/vcl/varnish4.vcl), or you can use vmod-curl. -- Guillaume Quintard On Thu, Mar 2, 2017 at 10:18 AM, Donath, Niklas wrote: > Hi there, > > > > i am currently trying to put varnish in front of a restful webservice with > an existing http digest authentication. > > Does anyone already had this use case and could give me a hint or an > example code on how to set this up properly? > > I didn?t find useful information in the varnish docs. The e-book ?Getting > started with Varnish Cache? only scratches that topic. > > Examples I found about basic authentication do not really match because > the authentication is implemented on varnish side only. > > I would like to pass through the authentication to the backend because I > don?t want to add user handling in varnish itself. > > In my understanding, the authentication works as follows: > > - 1st request: no authentication, passed to the backend, backend > response with nonce in header > > - 2nd request: client sends user credentials, passed to the > backend, server gives proof (200) or fail (401) > > - 3rd request: probably cache hit, request with auth credentials > is ?forked? as HEAD request, sent to the backend, server gives proof (200) > or fail (401) > > Am I getting this right, am I probably missing something? > > Thank you in advance for any input and/or feedback! > > > > Greetings, > > > > Niklas > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leonfauster at googlemail.com Sat Mar 4 19:04:37 2017 From: leonfauster at googlemail.com (Leon Fauster) Date: Sat, 4 Mar 2017 20:04:37 +0100 Subject: Varnish5 / h2 Support Message-ID: <2FE9E296-BD63-490C-8CF3-0BE73FB22C42@googlemail.com> All, just joined this list a couple of minutes ago. Saying that also to acknowledge that varnish for me as a tool is being used since short time. In general i am requesting for patience. Thanks. I'm starting in whole. Basically directly with varnish5 and http/2. Therefore some question to clarify my big picture. How experimental is this h2 support (as stated in the docs)? Should I deploy my prod system with varnish5 and h2 support enabled or not? Is the h2 support limited to the browser2varnish interaction? What about varnish2backendwebserver (the logs shows that some requests reaches the last layer (webserver) as http/1.1 and a few as http/2 mainly 404-requests -> its confusing)? Sorry for the many questions. As stated before I'm starting right now ... Currently I have a setup like this Browser -> TLSTerm -> Varnish5 -> ApacheWithH2/H2C working. What kind of stresstest (tool) do you suggest (with h2 support)? -- Thanks in advance! LF From phk at phk.freebsd.dk Sat Mar 4 19:24:15 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 04 Mar 2017 19:24:15 +0000 Subject: Varnish5 / h2 Support In-Reply-To: <2FE9E296-BD63-490C-8CF3-0BE73FB22C42@googlemail.com> References: <2FE9E296-BD63-490C-8CF3-0BE73FB22C42@googlemail.com> Message-ID: <36934.1488655455@critter.freebsd.dk> -------- In message <2FE9E296-BD63-490C-8CF3-0BE73FB22C42 at googlemail.com>, Leon Fauster writes: >How experimental is this h2 support (as stated in the docs)? >Should I deploy my prod system with varnish5 and h2 support enabled or not? V5 is fine for production, H2 is not (yet). V5.1 will be out in two weeks, and it will have better H2 support, but exactly how good is too early to say yet. >Is the h2 support limited to the browser2varnish interaction? Yes, for now we are only implementing H2 on the client side. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From guillaume at varnish-software.com Mon Mar 6 10:17:08 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 6 Mar 2017 11:17:08 +0100 Subject: Huge system load when starting Varnish Cache In-Reply-To: References: Message-ID: Maybe your machine is a bit slow to start threads, would you mind changing thread_pool_min to 4000 ? Also, could you try 4.1 and see if you still have the problem? -- Guillaume Quintard On Sat, Feb 18, 2017 at 11:53 AM, Pentium100 wrote: > /usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 > -t 120 -p thread_pool_min=256 -p thread_pool_max=4095 -p > thread_pool_timeout=120 -u varnish -g varnish -S /etc/varnish/secret -s > malloc,6G -P /var/run/varnish.pid > > On Sat, Feb 18, 2017 at 11:44 AM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> On a hunch: can you post your Varnish command line please (the v4 one)? >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Tue Mar 7 09:03:40 2017 From: lagged at gmail.com (Andrei) Date: Tue, 7 Mar 2017 11:03:40 +0200 Subject: Captcha vmod? Message-ID: Hi everyone, Has anyone run across a captcha vmod, or possible implementation methods? I'm trying to implement a function similar to how Cloudflare has to challenge certain requests with a captcha before allowing the request to go to the backend, but I'm not having much luck. Any thoughts or suggestions are greatly appreciated! Andrei -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Tue Mar 7 12:54:55 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 7 Mar 2017 13:54:55 +0100 Subject: Captcha vmod? In-Reply-To: References: Message-ID: No experience with that, but if you are using a service la reCaptcha, it seems you should be able to build something using vmod-curl and/or restarts. -- Guillaume Quintard On Tue, Mar 7, 2017 at 10:03 AM, Andrei wrote: > Hi everyone, > > Has anyone run across a captcha vmod, or possible implementation methods? > I'm trying to implement a function similar to how Cloudflare has to > challenge certain requests with a captcha before allowing the request to go > to the backend, but I'm not having much luck. Any thoughts or suggestions > are greatly appreciated! > > Andrei > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From MAlberghini at habitat.org Wed Mar 8 21:38:11 2017 From: MAlberghini at habitat.org (Mike Alberghini) Date: Wed, 8 Mar 2017 21:38:11 +0000 Subject: Setting a region cookie Message-ID: We?re trying to get a varnish setup running that sets a regional cookie. I?m having some troubles getting it to work, so any advice will be appreciated. We?re running Drupal 8 and Varnish 4.1.5 with the geoip2 vmod. IP Geolocation is working fine, and our X-GeoIP headers are being set correctly. What I would like to do is have varnish set a "country? cookie when requests to our home page come in. I?ve got it manipulating the req.http.Cookie header, but it does not seem to be actually setting a cookie in the browser. Any suggestions? -- Mike Alberghini Software Developer, Habitat for Humanity International 270 Peachtree Street NW, Suite 1300, Atlanta, GA 30303 office phone: (404) 420-6751 malberghini at habitat.org ? habitat.org | Habitat. We build. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mloftis at wgops.com Wed Mar 8 22:13:22 2017 From: mloftis at wgops.com (Michael Loftis) Date: Wed, 8 Mar 2017 14:13:22 -0800 Subject: Setting a region cookie In-Reply-To: References: Message-ID: You're setting it in the wrong place/way. To set a cookie in the browser you must set it in the response sent to the browser inside of say vcl_deliver. There's no variables so if you need to propogate data from e/g vcl_recv you can set an "internal" header on the req....see say https://www.fastly.com/blog/vcl-cookie-monster for an example. On Wed, Mar 8, 2017 at 1:38 PM, Mike Alberghini wrote: > We?re trying to get a varnish setup running that sets a regional cookie. > I?m having some troubles getting it to work, so any advice will be > appreciated. > > We?re running Drupal 8 and Varnish 4.1.5 with the geoip2 vmod. IP > Geolocation is working fine, and our X-GeoIP headers are being set > correctly. What I would like to do is have varnish set a "country? cookie > when requests to our home page come in. I?ve got it manipulating the > req.http.Cookie header, but it does not seem to be actually setting a cookie > in the browser. Any suggestions? > > -- > > Mike Alberghini > > Software Developer, Habitat for Humanity International > > 270 Peachtree Street NW, Suite 1300, Atlanta, GA 30303 > > office phone: (404) 420-6751 > > malberghini at habitat.org ? habitat.org | Habitat. We build. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler From colas.delmas at gmail.com Thu Mar 9 11:40:14 2017 From: colas.delmas at gmail.com (Nicolas Delmas) Date: Thu, 9 Mar 2017 12:40:14 +0100 Subject: Setting a region cookie In-Reply-To: References: Message-ID: Hi, Be careful the method in the link given by Michael doesn't work. To set a cookie, you must use the vmod header. And in the subroutines *vcl_deliver* you must add sotmehting like this (just change the name of the cookie and the value) header.append(resp.http.Set-Cookie, > "VarnishCache="+resp.http.X-Cache+"path:/"); Because using the set resp.http.Set-Cookie = overwrite all Cookie written by the backend. I just implement this method in my website to get Analytics stats of my Varnish *Nicolas Delmas* http://tutoandco.colas-delmas.fr/ 2017-03-08 23:13 GMT+01:00 Michael Loftis : > You're setting it in the wrong place/way. To set a cookie in the > browser you must set it in the response sent to the browser inside of > say vcl_deliver. There's no variables so if you need to propogate > data from e/g vcl_recv you can set an "internal" header on the > req....see say https://www.fastly.com/blog/vcl-cookie-monster for an > example. > > On Wed, Mar 8, 2017 at 1:38 PM, Mike Alberghini > wrote: > > We?re trying to get a varnish setup running that sets a regional cookie. > > I?m having some troubles getting it to work, so any advice will be > > appreciated. > > > > We?re running Drupal 8 and Varnish 4.1.5 with the geoip2 vmod. IP > > Geolocation is working fine, and our X-GeoIP headers are being set > > correctly. What I would like to do is have varnish set a "country? > cookie > > when requests to our home page come in. I?ve got it manipulating the > > req.http.Cookie header, but it does not seem to be actually setting a > cookie > > in the browser. Any suggestions? > > > > -- > > > > Mike Alberghini > > > > Software Developer, Habitat for Humanity International > > > > 270 Peachtree Street NW, Suite 1300, Atlanta, GA 30303 > > > > office phone: (404) 420-6751 > > > > malberghini at habitat.org ? habitat.org | Habitat. We build. > > > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > -- > > "Genius might be described as a supreme capacity for getting its possessors > into trouble of all kinds." > -- Samuel Butler > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From y.karayiannidis at stoiximan.gr Thu Mar 9 21:15:27 2017 From: y.karayiannidis at stoiximan.gr (Yiannis Karayiannidis) Date: Thu, 9 Mar 2017 23:15:27 +0200 Subject: Varnish Panic Message-ID: Hi all, We had a stragne panic problem with our varnish server i attach some log lines any help will be more than appriciated. *Mar 9 20:13:03 lin-varnish03 varnishd[5295]: Child (5297) not responding to CLI, killed it.* *Mar 9 20:13:03 lin-varnish03 varnishd[5295]: Unexpected reply from ping: 400 CLI communication error (hdr)* *Mar 9 20:13:04 lin-varnish03 varnishd[5295]: Child (5297) died signal=6* *Mar 9 20:13:04 lin-varnish03 varnishd[5295]: Child (5297) Last panic at: Thu, 09 Mar 2017 20:13:04 GMT#012"Assert error in mpl_alloc(), cache/cache_mempool.c line 79:#012 Condition((mi) != 0) not true.#012errno = 12 (Cannot allocate memory)#012thread = (cache-worker)#012version = varnish-4.1.5 revision 2c82b1c#012ident = Linux,3.10.0-514.6.1.el7.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll*#012now = 1254528.822470 (mono), 1489090356.283966 (real)#012Backtrace:#012req = 0x7f6d185f6020 {#012 vxid = 1027769622, step = R_STP_PASS,#012 req_body = R_BODY_NONE,#012 restarts = 0, esi_level = 0,#012 sp = 0x7f6b465dd220 {#012 fd = 46366, vxid = 1027769621,#012 client = 127.0.0.1 45267,#012 t_open = 1489090356.283755,#012 t_idle = 1489090356.283755,#012 step = S_STP_H1PROC,#012 },#012 worker = 0x7f6e12a0ac40 {#012 stack = {0x7f6e12a0b000 -> 0x7f6e129ff000},#012 ws = 0x7f6e12a0ae38 {#012 id = \"wrk\",#012 {s,f,r,e} = {0x7f6e12a0a3e0,0x7f6e12a0a3e0,(nil),+2040},#012 },#012 VCL::method = PASS,#012 VCL::return = fetch,#012 VCL::methods = {RECV, PASS, HASH},#012 },#012 ws = 0x7f6d185f6200 {#012 id = \"req\",#012 {s,f,r,e} = {0x7f6d185f8000,+5024,(nil),+57336},#012 },#012 http_conn = 0x7f6d185f6128 {#012 fd = 46366,#012 doclose = NULL,#012 ws = 0x7f6d185f6200,#012 {rxbuf_b, rxbuf_e} = {0x7f6d185f8000, 0x7f6d185f8774},#012 {pipeline_b, pipeline_e} = {(nil), (nil)},#012 content_length = -1,#012 body_status = none,#012 first_byte_timeout = 0.000000,#012 between_bytes_timeout = 0.000000,#012 },#012 http[req] = 0x7f6d185f6298 {#012 ws[req] = 0x7f6d185f6200,#012 hdrs {#012 \"GET\",#012 \"/out?only=true&_=1489089262776\",#012 \"HTTP/1.1\",#012 \"Host: www.xxx.com\",#012 \"X-Forwarded-Proto: https\",#012 \"Connection: close\",#012 \"Accept-Encoding: gzip\",#012 \"CF-IPCountry: EN\",#012 \"CF-RAY: 33d0b0a4dae65cb7-ATH\",#012 \"CF-Visitor: {\"scheme\":\"https\"}\",#012 \"Accept: application/json, text/javascript, */*; q=0.01\",#012 \"X-If-None-Match: RtbjjaE-8Jrgzu9Yp9qIPQ2\",#012 \"X-Requested-With: XMLHttpRequest\",#012 \"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\",#012 \"Referer: https://www.xxx.com/ntlive/\",#012 \"Accept-Language: en-GR,en;q=0.8,hu;q=0.6,en;q=0.4\",#012 \"CF-Connecting-IP: 122.85.108.207\",#012 \"X-Forwarded-For: 122.85.108.207, 122.85.108.207, 127.0.0.1\",#012 \"sticky: stx98.141\",#012 \"Cookie: sticky=stx89.746; _ga=GA1.2.17060266.132423773; _tz=120; uth1=F9A34D852B9715121425DB5B1393F63D30003234A8AC5811C9209E59; AUTH=Yyg1I2QAZ7dsiHxSrs6cX4g9pxcOvLh2iPaF2NoEjcnoVa2p6fmNA5oyhfueuWMoMI1Kr3dR8Voi0er_Bm5nR6aNa3UQ251tg=\",#012 \"X-defHash: /out?liveonly=true&_=1489089262776 + www.xxx.com\",#012 },#012 },#012 vcl = {#012 busy = 47089#012 discard = 0,#012 state = auto,#012 temp = warm,#012 conf = {#012 srcname = {#012 \"/etc/varnish/default.vcl\",#012 \"Builtin\",#012 \"/etc/varnish/origins2.vcl\",#012 \"/etc/varnish/origins1.com.vcl\",#012 \"/etc/varnish/origins.vcl\",#012 \"/etc/varnish/origins-stagwqwqi-trick.vcl\",#012 \"/etc/varnish/bwewanlist.vcl\",#012 },#012 },#012 },#012 objcore[REQ] = 0x7f6b2d7b3080 {#012 refcnt = 1,#012 flags = 0x10a,#012 exp_flags = 0x0,#012 exp = { 0.000000, 0.000000, 0.000000, 0.000000 }#012 objhead = 0x7f6e3f4f03e0,#012 stevedore = (nil),#012 },#012 flags = {#012 },#012},#012#012" Mar 9 20:13:04 lin-varnish03 varnishd[5295]: Child cleanup complete Mar 9 20:13:04 lin-varnish03 varnishd[5295]: Child (5892) Started Mar 9 20:13:04 lin-varnish03 varnishd[5295]: Child (5892) said Child starts *Mar 9 20:13:54 lin-varnish03 varnishd[5295]: Child (5892) not responding to CLI, killed it.* *Mar 9 20:13:54 lin-varnish03 varnishd[5295]: Unexpected reply from ping: 400 CLI communication error (hdr)* *Mar 9 20:13:55 lin-varnish03 varnishd[5295]: Child (5892) died signal=6* *Mar 9 20:13:55 lin-varnish03 varnishd[5295]: Child (5892) Last panic at: Thu, 09 Mar 2017 20:13:55 GMT#012"Missing errorhandling code in hsh_NewObjCore(), cache/cache_hash.c line 76:#012 Condition((oc) != 0) not true.#012errno = 12 (Cannot allocate memory)#012thread = (cache-worker)#012version = varnish-4.1.5 revision 2c82b1c#012ident = Linux,3.10.0-514.6.1.el7.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#*012now = 1254528.822470 (mono), 1489090356.283966 (real)#012Backtrace:#012req = 0x7f6d185f6020 {#012 vxid = 1027769622, step = R_STP_PASS,#012 req_body = R_BODY_NONE,#012 restarts = 0, esi_level = 0,#012 sp = 0x7f6b465dd220 {#012 fd = 46366, vxid = 1027769621,#012 client = 127.0.0.1 45267,#012 t_open = 1489090356.283755,#012 t_idle = 1489090356.283755,#012 step = S_STP_H1PROC,#012 },#012 worker = 0x7f6e12a0ac40 {#012 stack = {0x7f6e12a0b000 -> 0x7f6e129ff000},#012 ws = 0x7f6e12a0ae38 {#012 id = \"wrk\",#012 {s,f,r,e} = {0x7f6e12a0a3e0,0x7f6e12a0a3e0,(nil),+2040},#012 },#012 VCL::method = PASS,#012 VCL::return = fetch,#012 VCL::methods = {RECV, PASS, HASH},#012 },#012 ws = 0x7f6d185f6200 {#012 id = \"req\",#012 {s,f,r,e} = {0x7f6d185f8000,+5024,(nil),+57336},#012 },#012 http_conn = 0x7f6d185f6128 {#012 fd = 46366,#012 doclose = NULL,#012 ws = 0x7f6d185f6200,#012 {rxbuf_b, rxbuf_e} = {0x7f6d185f8000, 0x7f6d185f8774},#012 {pipeline_b, pipeline_e} = {(nil), (nil)},#012 content_length = -1,#012 body_status = none,#012 first_byte_timeout = 0.000000,#012 between_bytes_timeout = 0.000000,#012 },#012 http[req] = 0x7f6d185f6298 {#012 ws[req] = 0x7f6d185f6200,#012 hdrs {#012 \"GET\",#012 \"/out?only=true&_=1489089262776\",#012 \"HTTP/1.1\",#012 \"Host: www.xxx.com\",#012 \"X-Forwarded-Proto: https\",#012 \"Connection: close\",#012 \"Accept-Encoding: gzip\",#012 \"CF-IPCountry: EN\",#012 \"CF-RAY: 33d0b0a4dae65cb7-ATH\",#012 \"CF-Visitor: {\"scheme\":\"https\"}\",#012 \"Accept: application/json, text/javascript, */*; q=0.01\",#012 \"X-If-None-Match: RtbjjaE-8Jrgzu9Yp9qIPQ2\",#012 \"X-Requested-With: XMLHttpRequest\",#012 \"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\",#012 \"Referer: https://www.xxx.com/ntlive/\",#012 \"Accept-Language: en-GR,en;q=0.8,hu;q=0.6,en;q=0.4\",#012 \"CF-Connecting-IP: 122.85.108.207\",#012 \"X-Forwarded-For: 122.85.108.207, 122.85.108.207, 127.0.0.1\",#012 \"sticky: stx98.141\",#012 \"Cookie: sticky=stx89.746; _ga=GA1.2.17060266.132423773; _tz=120; uth1=D30A30ABC4B7D4AA7A0639A37F1C84B572B396E51335C456A710F72B1411990C4CF9A34D852B9715121425DB5B1393F63D30003234A8AC5811C9209E59; AUTH=Yyg1I2QAZ7dsiHxSrs6cX4g9pxcOvLh2MI1Kr3dR8Voi0er_Bm5nR6aNa3UQ251tg=\",#012 \"X-defHash: /out?only=true&_=1489089262776 + www.xxx.com\",#012 },#012 },#012 vcl = {#012 busy = 47089#012 discard = 0,#012 state = auto,#012 temp = warm,#012 conf = {#012 srcname = {#012 \"/etc/varnish/default.vcl\",#012 \"Builtin\",#012 \"/etc/varnish/origins2.vcl\",#012 \"/etc/varnish/origins1.com.vcl\",#012 \"/etc/varnish/origins.vcl\",#012 \"/etc/varnish/origins-stagwqwqi-trick.vcl\",#012 \"/etc/varnish/bwewanlist.vcl\",#012 },#012 },#012 },#012 objcore[REQ] = 0x7f6b2d7b3080 {#012 refcnt = 1,#012 flags = 0x10a,#012 exp_flags = 0x0,#012 exp = { 0.000000, 0.000000, 0.000000, 0.000000 }#012 objhead = 0x7f6e3f4f03e0,#012 stevedore = (nil),#012 },#012 flags = {#012 },#012},#012#012" Mar 9 20:13:55 lin-varnish03 varnishd[5295]: Child cleanup complete Mar 9 20:13:55 lin-varnish03 varnishd[5295]: Child (2878) Started Mar 9 20:13:56 lin-varnish03 varnishd[5295]: Child (2878) said Child starts Mar 9 20:13:56 lin-varnish03 varnishd[5295]: CLI telnet 127.0.0.1 35892 127.0.0.1 6082 Rd auth 9a804ba5663b2525c372565714aedbb2a11b899cd24bf3d7d33d4969c440a05c Mar 9 20:13:56 lin-varnish03 varnishd[5295]: CLI telnet 127.0.0.1 35892 127.0.0.1 6082 Wr 200 -----------------------------#012Varnish Cache CLI 1.0#012-----------------------------#012Linux,3.10.0-514.6.1.el7.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit#012varnish-4.1.5 revision 2c82b1c#012#012Type 'help' for command list.#012Type 'quit' to close CLI session. Mar 9 20:14:36 lin-varnish03 varnishd[5295]: Child (2878) not responding to CLI, killed it. Mar 9 20:14:36 lin-varnish03 varnishd[5295]: Unexpected reply from ping: 400 CLI communication error (hdr) Mar 9 20:14:36 lin-varnish03 varnishd[5295]: Child (2878) not responding to CLI, killed it. Mar 9 20:14:36 lin-varnish03 varnishd[5295]: Unexpected reply from ping: 400 CLI communication error Mar 9 20:14:37 lin-varnish03 varnishd[5295]: Child (2878) died signal=11 Mar 9 20:14:37 lin-varnish03 varnishd[5295]: Child (2878) Last panic at: Thu, 09 Mar 2017 20:14:37 GMT#012"Missing errorhandling code in hsh_NewObjCore(), cache/cache_hash.c line 76:#012 Condition((oc) != 0) not true.errno = 12 (Cannot allocate memory)#012thread = (cache-worker)#012version = varnish-4.1.5 revision 2c82b1c#012ident = Linux,3.10.0-514.6.1.el7.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#012now = 1254622.398113 (mono), 1489090449.859610 (real)#012" Mar 9 20:14:37 lin-varnish03 varnishd[5295]: Child cleanup complete Mar 9 20:14:37 lin-varnish03 varnishd[5295]: Child (35949) Started Mar 9 20:14:37 lin-varnish03 varnishd[5295]: Child (35949) said Child starts Regards Yianni -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Fri Mar 10 06:36:33 2017 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 10 Mar 2017 07:36:33 +0100 Subject: Varnish Panic In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 03/09/2017 10:15 PM, Yiannis Karayiannidis wrote: > > errno = 12 (Cannot allocate memory) > errno = 12 (Cannot allocate memory) Your system's memory was evidently full. Varnish attempted internal allocations, but they failed, and so the panics were thrown. Best, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYwklxAAoJEOUwvh9pJNURZZsP/00OKF0dPF3+CHaCR9GkZ/P+ ltERvS+5JDkSvjME6tSIP3ykQ2sI6upYs9mqnsPa51rzhqkyW6lNNE6IeRI1n83B Rtr0F+J/PZRYT+sgZ4uBbSt9KESVhtyGoCf04oZuqQ9QOlCuSzxXKA7dtgYCnpp5 e69txYxnDeC+bkNDw504/uMTCb27TuFYXATkRejGxEwWdLZVscrIlIi+hVK8HSW1 T0qvIIFLP7OEo5LlE1wpSWDLPbiBLUi17IU80/Q1zF8cVgybB3oEZCyIyuCAiVrd H58SojE2r/XP9jDHN4maiXSHNCbRBnDLYGNUpLc9T0bSurEQuyDu9H4Rm/FHkWJ5 HfBiaqMPdnSRh2rDTdnj7tkBG+c8rPouwAjgUJEmEyHtZQ+e9GEIxiyMSPn9efgm 5MjBVZPfHk2XS98loy09v00xgHTJyrl9QX1aKXrzsB2W/bB2tQ7WQRfESp/3oVBf mo1MrsTTXrSNuJYUUVp4Y3+MPTeGi5pdjWixZmNfCArYLhtlcxAmj+9969/rYHio Rdm/Sw0G9DMv/TPViFZvl+fEDua57U1+3Xe+dclqWwBLFbtYnU0dwNOqB8Qm1Fy2 az3SxxW8Is/v/qA0DHfwC9wxY84OGICKJL3IUklWzZHbA9On4ur9M7V/Pk1hcDgg 0Hxu7CeoL+hwuS9HRxhC =itRS -----END PGP SIGNATURE----- From lagged at gmail.com Fri Mar 10 09:53:12 2017 From: lagged at gmail.com (Andrei) Date: Fri, 10 Mar 2017 03:53:12 -0600 Subject: Setting a region cookie In-Reply-To: References: Message-ID: So by copying the cloudflare/google analytics cookies for example to a custom header before stripping them for a possible cache hit, we can later add it back to the client response cookies? Is it even worth bothering for? I strip all cloudflare/google analytics cookies and haven't had any complaints yet. Just wondering what the use case might be. On Thu, Mar 9, 2017 at 5:40 AM, Nicolas Delmas wrote: > Hi, > > Be careful the method in the link given by Michael doesn't work. > To set a cookie, you must use the vmod header. > > And in the subroutines *vcl_deliver* you must add sotmehting like this > (just change the name of the cookie and the value) > > header.append(resp.http.Set-Cookie, "VarnishCache="+resp.http.X- >> Cache+"path:/"); > > > > Because using the set resp.http.Set-Cookie = overwrite all Cookie written > by the backend. > > > I just implement this method in my website to get Analytics stats of my > Varnish > > > *Nicolas Delmas* > http://tutoandco.colas-delmas.fr/ > > > > > > > > 2017-03-08 23:13 GMT+01:00 Michael Loftis : > >> You're setting it in the wrong place/way. To set a cookie in the >> browser you must set it in the response sent to the browser inside of >> say vcl_deliver. There's no variables so if you need to propogate >> data from e/g vcl_recv you can set an "internal" header on the >> req....see say https://www.fastly.com/blog/vcl-cookie-monster for an >> example. >> >> On Wed, Mar 8, 2017 at 1:38 PM, Mike Alberghini >> wrote: >> > We?re trying to get a varnish setup running that sets a regional cookie. >> > I?m having some troubles getting it to work, so any advice will be >> > appreciated. >> > >> > We?re running Drupal 8 and Varnish 4.1.5 with the geoip2 vmod. IP >> > Geolocation is working fine, and our X-GeoIP headers are being set >> > correctly. What I would like to do is have varnish set a "country? >> cookie >> > when requests to our home page come in. I?ve got it manipulating the >> > req.http.Cookie header, but it does not seem to be actually setting a >> cookie >> > in the browser. Any suggestions? >> > >> > -- >> > >> > Mike Alberghini >> > >> > Software Developer, Habitat for Humanity International >> > >> > 270 Peachtree Street NW, Suite 1300, Atlanta, GA 30303 >> > >> > office phone: (404) 420-6751 >> > >> > malberghini at habitat.org ? habitat.org | Habitat. We build. >> > >> > >> > _______________________________________________ >> > varnish-misc mailing list >> > varnish-misc at varnish-cache.org >> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> >> >> -- >> >> "Genius might be described as a supreme capacity for getting its >> possessors >> into trouble of all kinds." >> -- Samuel Butler >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jllach at agilecontents.com Fri Mar 10 09:58:33 2017 From: jllach at agilecontents.com (Jordi Llach) Date: Fri, 10 Mar 2017 10:58:33 +0100 Subject: About choosing the right instance type for Varnish in AmazonEC2 Message-ID: Hi guys, any suggestion choosing an Amazon EC2 instance type for Varnish ? I stumbled upon this doubt/question because we are currently using "m3.medium" which only has 1 CPU and as stated in the docs Varnish assumes at least 2 CPU https://varnish-cache.org/docs/4.1/reference/varnishd.html#thread-pools The performance feeling that we have is exactly the one cited in the docs "...more than one pool for each CPU is most likely detrimental to performance" Should I decrease this value to 1 or better use another instance type. In the latter we could upgrade to m3.large(same kind of instance type but with 2 CPU) or switch to other types : more CPU optimized and less RAM optimized (c3.large or c4.large) Any experience/suggestion will be highly appreciated Thanks Jordi -------------- next part -------------- An HTML attachment was scrubbed... URL: From albert.tollkuci at gmail.com Fri Mar 10 10:02:39 2017 From: albert.tollkuci at gmail.com (=?UTF-8?Q?Albert_Tollku=C3=A7i?=) Date: Fri, 10 Mar 2017 11:02:39 +0100 Subject: version `LIBVARNISHAPI_1.4' not found (required by varnishadm) Message-ID: Hi everyone, I have been running varnish the last few months without problems. Recently I did install AWS CLI tools to manage snapshots (server is running AWS) and now I can't run varnishadm. The error I'm getting is: *varnishadm: /usr/local/lib/libvarnishapi.so.1: version `LIBVARNISHAPI_1.4' not found (required by varnishadm)* I'm running varnish version 4.1.3 in Ubuntu Ubuntu 15.10. Any idea how to fix it? Albert -- Web: http://www.tollkuci.com Follow me on: LinkedIn Google+ Facebook Twitter Career 2.0 ------------------------------ Imagination is more important than knowledge *Albert Einstein* -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul at krischer.nl Fri Mar 10 11:02:33 2017 From: paul at krischer.nl (Paul Krischer) Date: Fri, 10 Mar 2017 12:02:33 +0100 Subject: About choosing the right instance type for Varnish in AmazonEC2 In-Reply-To: References: Message-ID: Hi Jordi, It's hard to make this determination without any data on the traffic your site is receiving and what peak traffic you're expecting.Varnish will surely perform better on an instance with multiple CPUs and that's why we generally standardize on c3.large instances as the initial size for our Varnish instances. You will have to determine if that's an economical choice for you. Paul K On Fri, Mar 10, 2017 at 10:58 AM, Jordi Llach wrote: > Hi guys, > any suggestion choosing an Amazon EC2 instance type for Varnish ? > > I stumbled upon this doubt/question because we are currently using > "m3.medium" which only has 1 CPU and as stated in the docs Varnish assumes > at least 2 CPU > > https://varnish-cache.org/docs/4.1/reference/varnishd.html#thread-pools > > The performance feeling that we have is exactly the one cited in the docs > "...more than one pool for each CPU is most likely detrimental to > performance" > > Should I decrease this value to 1 or better use another instance type. > In the latter we could upgrade to m3.large(same kind of instance type but > with 2 CPU) or switch to other types : more CPU optimized and less RAM > optimized (c3.large or c4.large) > > Any experience/suggestion will be highly appreciated > > Thanks > > Jordi > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From albert.tollkuci at gmail.com Fri Mar 10 12:02:05 2017 From: albert.tollkuci at gmail.com (=?UTF-8?Q?Albert_Tollku=C3=A7i?=) Date: Fri, 10 Mar 2017 13:02:05 +0100 Subject: About choosing the right instance type for Varnish in AmazonEC2 In-Reply-To: References: Message-ID: Hi Jordi, I'm running it on t2.medium for several websites totaling 200k+ pageviews per day and it's running great. Albert On Fri, Mar 10, 2017 at 10:58 AM, Jordi Llach wrote: > Hi guys, > any suggestion choosing an Amazon EC2 instance type for Varnish ? > > I stumbled upon this doubt/question because we are currently using > "m3.medium" which only has 1 CPU and as stated in the docs Varnish assumes > at least 2 CPU > > https://varnish-cache.org/docs/4.1/reference/varnishd.html#thread-pools > > The performance feeling that we have is exactly the one cited in the docs > "...more than one pool for each CPU is most likely detrimental to > performance" > > Should I decrease this value to 1 or better use another instance type. > In the latter we could upgrade to m3.large(same kind of instance type but > with 2 CPU) or switch to other types : more CPU optimized and less RAM > optimized (c3.large or c4.large) > > Any experience/suggestion will be highly appreciated > > Thanks > > Jordi > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Web: http://www.tollkuci.com Follow me on: LinkedIn Google+ Facebook Twitter Career 2.0 ------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From pentium100 at gmail.com Fri Mar 10 13:59:27 2017 From: pentium100 at gmail.com (Pentium100) Date: Fri, 10 Mar 2017 15:59:27 +0200 Subject: Huge system load when starting Varnish Cache In-Reply-To: References: Message-ID: Thank you, I will try it next time I need to restart Varnish on that server. On Mon, Mar 6, 2017 at 12:17 PM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > Maybe your machine is a bit slow to start threads, would you mind changing thread_pool_min > to 4000 ? > > Also, could you try 4.1 and see if you still have the problem? > > -- > Guillaume Quintard > > On Sat, Feb 18, 2017 at 11:53 AM, Pentium100 wrote: > >> /usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 >> -t 120 -p thread_pool_min=256 -p thread_pool_max=4095 -p >> thread_pool_timeout=120 -u varnish -g varnish -S /etc/varnish/secret -s >> malloc,6G -P /var/run/varnish.pid >> >> On Sat, Feb 18, 2017 at 11:44 AM, Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> On a hunch: can you post your Varnish command line please (the v4 one)? >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Fri Mar 10 14:22:20 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 10 Mar 2017 15:22:20 +0100 Subject: Huge system load when starting Varnish Cache In-Reply-To: References: Message-ID: On Fri, Mar 10, 2017 at 2:59 PM, Pentium100 wrote: > Thank you, I will try it next time I need to restart Varnish on that server. Hello, This is a runtime parameter, you don't need to restart Varnish to tune that one. Simply use varnishadm and update your service configuration to persist it for future restarts. Dridi From rbizzell at measinc.com Fri Mar 10 16:20:30 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Fri, 10 Mar 2017 16:20:30 +0000 Subject: Configuring Varnish to cache IIS Message-ID: <0160784d19c94260ba4ac98465dea468@mbx2serv.meas-inc.com> I have varnish up and running but I am not sure how to configure varnish to work with an external IIS server. I have the server as the content server in default.vcl this is all for proof of concept everything is internal on port 80. Do I just need to add the url under sub vcl_recv. Any help would be greatly appreciated This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | -------------- next part -------------- An HTML attachment was scrubbed... URL: From albert.tollkuci at gmail.com Fri Mar 10 16:58:11 2017 From: albert.tollkuci at gmail.com (=?UTF-8?Q?Albert_Tollku=C3=A7i?=) Date: Fri, 10 Mar 2017 17:58:11 +0100 Subject: Configuring Varnish to cache IIS In-Reply-To: <0160784d19c94260ba4ac98465dea468@mbx2serv.meas-inc.com> References: <0160784d19c94260ba4ac98465dea468@mbx2serv.meas-inc.com> Message-ID: Basically varnish doesn't know or cares what's behind. It works the same with Apache, IIS, etc. Configuration is the same. Albert On Fri, Mar 10, 2017 at 5:20 PM, Rodney Bizzell wrote: > I have varnish up and running but I am not sure how to configure varnish > to work with an external IIS server. I have the server as the content > server in default.vcl this is all for proof of concept everything is > internal on port 80. Do I just need to add the url under sub vcl_recv. Any > help would be greatly appreciated > > > This email (including any attachments) may contain confidential > information intended solely for acknowledged recipients. If you think you > have received this information in error, please reply to the sender and > delete all copies from your system. Please note that unauthorized use, > disclosure, or further distribution of this information is prohibited by > the sender. Note also that we may monitor email directed to or originating > from our network. Thank you for your consideration and assistance. | > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Web: http://www.tollkuci.com Follow me on: LinkedIn Google+ Facebook Twitter Career 2.0 ------------------------------ Imagination is more important than knowledge *Albert Einstein* The three chief virtues of a programmer are: Laziness, Impatience and Hubris *Larry Wall* Men are basically smart or dumb and lazy or ambitious. The dumb and ambitious ones are dangerous and I get rid of them. The dumb and lazy ones I give mundane duties. The smart ambitious ones I put on my staff. The smart and lazy ones I make my commanders *Erwin Rommel* The best programmers are not marginally better than merely good ones. They are an order-of-magnitude better, measured by whatever standard: conceptual creativity, speed, ingenuity of design, or problem-solving ability. *Randall E. Stross* Measuring programming progress by lines of code is like measuring aircraft building progress by weight. *Bill Gates* -------------- next part -------------- An HTML attachment was scrubbed... URL: From cservin-varnish at cromagnon.com Fri Mar 10 20:16:42 2017 From: cservin-varnish at cromagnon.com (Craig Servin) Date: Fri, 10 Mar 2017 14:16:42 -0600 Subject: About choosing the right instance type for Varnish in AmazonEC2 In-Reply-To: References: Message-ID: We found that we run into the network IO limits before we had CPU issues. You have to use trial and error to figure out how much IO you can get from each instance type as we couldn't find good documentation defining that. However it seemed that the network limit goes up as you step up machine sizes in each family. You could also put your varnish into an ASG and add instances on CPU or Network IO. The cache wouldn't be shared between them, but depending on your traffic that may not matter. At our busiest time we were running 6 c4.4xlarge instances 3 east 3 west. Now, we run 4 r4.xlarge instances. The nice thing about them being virtual is you can load test them and then load test another instance type fairly easily. Cheers, Craig On 2017-03-10 03:58, Jordi Llach wrote: > Hi guys, > any suggestion choosing an Amazon EC2 instance type for Varnish ? From reza at varnish-software.com Fri Mar 10 21:54:15 2017 From: reza at varnish-software.com (Reza Naghibi) Date: Fri, 10 Mar 2017 16:54:15 -0500 Subject: About choosing the right instance type for Varnish in AmazonEC2 In-Reply-To: References: Message-ID: Just to chime in: - 1 CPU is a good starting point, as others have said, Varnish does not need much CPU. If you max out the CPU, then move to 2 CPUs, repeat... - Do not change any of the thread settings, the defaults will work nice on a single CPU or more. -- Reza Naghibi Varnish Software On Fri, Mar 10, 2017 at 3:16 PM, Craig Servin wrote: > We found that we run into the network IO limits before we had CPU issues. > You have to use trial and error to figure out how much IO you can get from > each instance type as we couldn't find good documentation defining that. > However it seemed that the network limit goes up as you step up machine > sizes in each family. > > You could also put your varnish into an ASG and add instances on CPU or > Network IO. The cache wouldn't be shared between them, but depending on > your traffic that may not matter. > > At our busiest time we were running 6 c4.4xlarge instances 3 east 3 west. > Now, we run 4 r4.xlarge instances. The nice thing about them being virtual > is you can load test them and then load test another instance type fairly > easily. > > Cheers, > > Craig > > > On 2017-03-10 03:58, Jordi Llach wrote: > >> Hi guys, >> any suggestion choosing an Amazon EC2 instance type for Varnish ? >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From reza at varnish-software.com Fri Mar 10 21:56:50 2017 From: reza at varnish-software.com (Reza Naghibi) Date: Fri, 10 Mar 2017 16:56:50 -0500 Subject: version `LIBVARNISHAPI_1.4' not found (required by varnishadm) In-Reply-To: References: Message-ID: The easiest thing to do is reinstall. Start by uninstalling whatever version you are using now and make sure there are no varnish binaries or libs on the system before starting the install again. -- Reza Naghibi Varnish Software On Fri, Mar 10, 2017 at 5:02 AM, Albert Tollku?i wrote: > Hi everyone, > I have been running varnish the last few months without problems. Recently > I did install AWS CLI tools to manage snapshots (server is running AWS) and > now I can't run varnishadm. The error I'm getting is: > > *varnishadm: /usr/local/lib/libvarnishapi.so.1: version > `LIBVARNISHAPI_1.4' not found (required by varnishadm)* > > I'm running varnish version 4.1.3 in Ubuntu Ubuntu 15.10. > > Any idea how to fix it? > > Albert > > -- > Web: http://www.tollkuci.com > Follow me on: LinkedIn Google+ > Facebook > Twitter > Career 2.0 > > ------------------------------ > > Imagination is more important than knowledge > *Albert Einstein* > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Fri Mar 10 22:05:20 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 10 Mar 2017 23:05:20 +0100 Subject: About choosing the right instance type for Varnish in AmazonEC2 In-Reply-To: References: Message-ID: On Fri, Mar 10, 2017 at 9:16 PM, Craig Servin wrote: > We found that we run into the network IO limits before we had CPU issues. > You have to use trial and error to figure out how much IO you can get from > each instance type as we couldn't find good documentation defining that. > However it seemed that the network limit goes up as you step up machine > sizes in each family. CPU usage tends to grow with features like compression or ESI, VMODs may also burn cycles depending on their purpose. Dridi From japrice at gmail.com Sat Mar 11 18:44:50 2017 From: japrice at gmail.com (Jason Price) Date: Sat, 11 Mar 2017 13:44:50 -0500 Subject: About choosing the right instance type for Varnish in AmazonEC2 In-Reply-To: References: Message-ID: In my experience, the R3/R4 family works best for Varnish. Varnish isn't great at spreading the load across multiple cores, but it will fully utilize all the ram you give it. I like to run an ASG of size 2 as the front door to the side. This maximizes availability, and keeps cache fragmentation to a minimum, and protects you from most AZ issues and single instance hardware issues. -Jason On Fri, Mar 10, 2017 at 5:05 PM, Dridi Boukelmoune wrote: > On Fri, Mar 10, 2017 at 9:16 PM, Craig Servin > wrote: > > We found that we run into the network IO limits before we had CPU issues. > > You have to use trial and error to figure out how much IO you can get > from > > each instance type as we couldn't find good documentation defining that. > > However it seemed that the network limit goes up as you step up machine > > sizes in each family. > > CPU usage tends to grow with features like compression or ESI, VMODs > may also burn cycles depending on their purpose. > > Dridi > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.huot at thomsonreuters.com Mon Mar 13 13:01:18 2017 From: jonathan.huot at thomsonreuters.com (jonathan.huot at thomsonreuters.com) Date: Mon, 13 Mar 2017 13:01:18 +0000 Subject: Persistent "admin" state/backend health Message-ID: <8E656B642592B942AE317E2AFAE0ABA18BF0CAB2@C111WBSLMBX06.ERF.thomson.com> Hi everyone, We have some clusters of varnishd and we're using varnish CLI to trigger failover by setting the "admin health" of a backend to sick. It works well, until: - we upgrade varnish (occurs a restart) - we change VCL (occurs a reload) - varnishd crashes (notably because of 3rd party VMODs) So, we're trying to find a way of preserving the "admin" states across restart/reload/crashes. We're thinking about reusing vagent2, or creating another intermediate above the Child, but I'm wondering first if this is not better suited in Varnish Core as a basic feature? But then, where? Thanks for your time, -- Jonathan Thomson Reuters ________________________________ This e-mail is for the sole use of the intended recipient and contains information that may be privileged and/or confidential. If you are not an intended recipient, please notify the sender by return e-mail and delete this e-mail and any attachments. Certain required legal entity disclosures can be accessed on our website. From dridi at varni.sh Mon Mar 13 13:16:03 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 13 Mar 2017 14:16:03 +0100 Subject: Persistent "admin" state/backend health In-Reply-To: <8E656B642592B942AE317E2AFAE0ABA18BF0CAB2@C111WBSLMBX06.ERF.thomson.com> References: <8E656B642592B942AE317E2AFAE0ABA18BF0CAB2@C111WBSLMBX06.ERF.thomson.com> Message-ID: > So, we're trying to find a way of preserving the "admin" states across restart/reload/crashes. > We're thinking about reusing vagent2, or creating another intermediate above the Child, but I'm wondering first if this is not better suited in Varnish Core as a basic feature? But then, where? Since 4.1 backends are owned by VCLs so it's no longer possible to preserve the state across reloads. You could however make your own reload script that could collect the backends state from the active VCL and apply it to the new VCL between vcl.load and vcl.use. Dridi From myles at magicalwonders.com Tue Mar 14 18:47:19 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Tue, 14 Mar 2017 18:47:19 +0000 Subject: Which version of Varnish to install? In-Reply-To: References: Message-ID: <7e1b8433-5024-5930-147e-969b286a1166@magicalwonders.com> Hello everybody, I'm looking at installing Varnish on my VPS, but would appreciate confirmation as to which version to use. My machine is running CENTOS 6.8 x86_64 virtuozzo. From searching around it looks like version 4.1 of Varnish would be the one to install? Hope someone can confirm or advise. Many thanks, Myles -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Tue Mar 14 18:58:45 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 14 Mar 2017 18:58:45 +0000 Subject: Which version of Varnish to install? In-Reply-To: <7e1b8433-5024-5930-147e-969b286a1166@magicalwonders.com> References: <7e1b8433-5024-5930-147e-969b286a1166@magicalwonders.com> Message-ID: <24527.1489517925@critter.freebsd.dk> -------- In message <7e1b8433-5024-5930-147e-969b286a1166 at magicalwonders.com>, Magical W onders writes: >My machine is running CENTOS 6.8 x86_64 virtuozzo. From searching around >it looks like version 4.1 of Varnish would be the one to install? I'm obviously biased, but I would tell you to wait a couple of days and then install the 5.1 release we're rolling tomorrow. (Or clone our github repos and get a headstart! :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From myles at magicalwonders.com Tue Mar 14 19:51:05 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Tue, 14 Mar 2017 19:51:05 +0000 Subject: Which version of Varnish to install? In-Reply-To: <24527.1489517925@critter.freebsd.dk> References: <7e1b8433-5024-5930-147e-969b286a1166@magicalwonders.com> <24527.1489517925@critter.freebsd.dk> Message-ID: On 14/03/2017 18:58, Poul-Henning Kamp wrote:-------- > In message <7e1b8433-5024-5930-147e-969b286a1166 at magicalwonders.com>, Magical W > onders writes: > >> My machine is running CENTOS 6.8 x86_64 virtuozzo. From searching around >> it looks like version 4.1 of Varnish would be the one to install? > I'm obviously biased, but I would tell you to wait a couple of days > and then install the 5.1 release we're rolling tomorrow. > > (Or clone our github repos and get a headstart! :-) Hi Poul, Thanks for the advice on that. I'm happy to wait a couple of days! :-) I'll probably be asking my host if they will install it for me. I'm OK with the front end stuff, but not so knowledgeable with the server side of things. Having said that, from what I can see, installation looks fairly straightforward? :-) I have a couple of other questions before proceeding. Is it preferable to ask them using separate subject lines? -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Tue Mar 14 20:07:08 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 14 Mar 2017 20:07:08 +0000 Subject: Which version of Varnish to install? In-Reply-To: References: <7e1b8433-5024-5930-147e-969b286a1166@magicalwonders.com> <24527.1489517925@critter.freebsd.dk> Message-ID: <59180.1489522028@critter.freebsd.dk> -------- In message , Magical W onders writes: >I have a couple of other questions before proceeding. Is it preferable >to ask them using separate subject lines? If they're totally unrelated, that's probably a good idea, but we're not that strict here on the -misc list. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From myles at magicalwonders.com Tue Mar 14 20:19:42 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Tue, 14 Mar 2017 20:19:42 +0000 Subject: Which version of Varnish to install? In-Reply-To: <59180.1489522028@critter.freebsd.dk> References: <7e1b8433-5024-5930-147e-969b286a1166@magicalwonders.com> <24527.1489517925@critter.freebsd.dk> <59180.1489522028@critter.freebsd.dk> Message-ID: <06438f48-7d75-8b3d-dd87-edb6ffdc8ce1@magicalwonders.com> On 14/03/2017 20:07, Poul-Henning Kamp wrote: > -------- > In message , Magical W > onders writes: > >> I have a couple of other questions before proceeding. Is it preferable >> to ask them using separate subject lines? > If they're totally unrelated, that's probably a good idea, but we're > not that strict here on the -misc list. Ok, great. Thanks. I'll make them separate.:-) Myles -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Tue Mar 14 20:46:51 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Tue, 14 Mar 2017 20:46:51 +0000 Subject: Redirecting traffic to Varnish Cache Server Message-ID: <5f327260f6f74112a42d4b9582bee475@mbx1serv.meas-inc.com> So I have setup a basic default.vcl and my question how do I get the Varnish server to answer web request before going to the backend drupal servers not sure if I am stating this correctly. Our DNS admin point the urls to Varnish server but I am getting page not found. Am I missing a config in the default.vcl? Thanks! I researched and I am not seeing any good information. backend drupal { .host = "drupal.miat.co"; .port = "80"; .connect_timeout = 6000s; .first_byte_timeout = 6000s; .between_bytes_timeout = 6000s; } backend ncwrite { .host = "ncwrite.miat.co"; .port = "80"; .connect_timeout = 6000s; .first_byte_timeout = 6000s; .between_bytes_timeout = 6000s; } sub vcl_recv { if (req.http.host == "drupal.miat.co"){ set req.backend_hint = drupal; } elsif (req.http.host == "ncwrite.miat.co"){ set req.backend_hint = ncwrite; return (hash); } } This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | -------------- next part -------------- An HTML attachment was scrubbed... URL: From myles at magicalwonders.com Wed Mar 15 09:28:21 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Wed, 15 Mar 2017 09:28:21 +0000 Subject: Custom Permalinks Setting in Wordpress Message-ID: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> I'm working my way through the Wiki guide for configuring Varnish to work with Wordpress - https://www.varnish-software.com/wiki/content/tutorials/wordpress/wp_step_by_step.html Two questions immediately spring up - Step 3 says to enable custom permalinks in Wordpress typing /%year%/%monthnum%/%post_id% - My existing Wordpress sites all have custom Permalinks. However they are mostly in the format of /%postname%/ I don't really want to have to change all the permalinks. Will Varnish still work ok for sites with custom Permalinks set at /%postname%/ ? The last part of Step 3 is to open a command prompt and run "ta2enmod rewrite" as root. Mod_rewrite must already be enabled on my server so I guess I can skip that step? Hope someone can advise. Myles -------------- next part -------------- An HTML attachment was scrubbed... URL: From colas.delmas at gmail.com Wed Mar 15 12:37:00 2017 From: colas.delmas at gmail.com (Nicolas Delmas) Date: Wed, 15 Mar 2017 13:37:00 +0100 Subject: Custom Permalinks Setting in Wordpress In-Reply-To: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> References: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> Message-ID: Hi, Will Varnish still work ok for sites with custom Permalinks set at > /%postname%/ ? Yes Varnish will still works with your custom Permalinks. so I guess I can skip that step? Indeed you don't need to run the step 3 You could find a more complete configuration of Varnish for Wordpress here : https://github.com/colas31/varnish/blob/master/v4.1/wordpress.vcl it's the one I use on my personal wordpress. Best regards *Nicolas Delmas* http://tutoandco.colas-delmas.fr/ 2017-03-15 10:28 GMT+01:00 Magical Wonders : > I'm working my way through the Wiki guide for configuring Varnish to work > with Wordpress - https://www.varnish-software.com/wiki/content/tutorials/ > wordpress/wp_step_by_step.html > > Two questions immediately spring up - > > Step 3 says to enable custom permalinks in Wordpress typing > /%year%/%monthnum%/%post_id% - My existing Wordpress sites all have custom > Permalinks. However they are mostly in the format of /%postname%/ I don't > really want to have to change all the permalinks. Will Varnish still work > ok for sites with custom Permalinks set at /%postname%/ ? > > The last part of Step 3 is to open a command prompt and run "ta2enmod > rewrite" as root. Mod_rewrite must already be enabled on my server so I > guess I can skip that step? > > Hope someone can advise. > > Myles > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From myles at magicalwonders.com Wed Mar 15 12:56:00 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Wed, 15 Mar 2017 12:56:00 +0000 Subject: Custom Permalinks Setting in Wordpress In-Reply-To: References: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> Message-ID: <1efc66a8-27e8-9523-6e07-85e68ad3f530@magicalwonders.com> Hi Nicolas, Thanks for the confirmation. :-) Also, thanks for the link showing VCL settings for Wordpress. One of my sites is using WooCommerce and it looks like it's got that covered! I have a couple of other possible items I think I'll need to exclude but need to check further. Best wishes, Myles On 15/03/2017 12:37, Nicolas Delmas wrote: > Hi, > > Will Varnish still work ok for sites with custom Permalinks set at > /%postname%/ ? > > > Yes Varnish will still works with your custom Permalinks. > > so I guess I can skip that step? > > Indeed you don't need to run the step 3 > > > You could find a more complete configuration of Varnish for Wordpress > here : > https://github.com/colas31/varnish/blob/master/v4.1/wordpress.vcl it's > the one I use on my personal wordpress. > > Best regards > > > *Nicolas Delmas* > http://tutoandco.colas-delmas.fr/ > > > > > > > > 2017-03-15 10:28 GMT+01:00 Magical Wonders >: > > I'm working my way through the Wiki guide for configuring Varnish > to work with Wordpress - > https://www.varnish-software.com/wiki/content/tutorials/wordpress/wp_step_by_step.html > > > Two questions immediately spring up - > > Step 3 says to enable custom permalinks in Wordpress typing > /%year%/%monthnum%/%post_id% - My existing Wordpress sites all > have custom Permalinks. However they are mostly in the format of > /%postname%/ I don't really want to have to change all the > permalinks. Will Varnish still work ok for sites with custom > Permalinks set at /%postname%/ ? > > The last part of Step 3 is to open a command prompt and run > "ta2enmod rewrite" as root. Mod_rewrite must already be enabled on > my server so I guess I can skip that step? > > Hope someone can advise. > > Myles > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Mar 15 13:07:57 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 15 Mar 2017 14:07:57 +0100 Subject: Custom Permalinks Setting in Wordpress In-Reply-To: References: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> Message-ID: > Yes Varnish will still works with your custom Permalinks. > >> so I guess I can skip that step? > > Indeed you don't need to run the step 3 Hello, There's a "Show on GitHub" link on all wiki pages, don't hesitate to open an issue or even submit a pull request to help improve the docs. https://github.com/varnish/varnish-wiki/blob/master/source/content/tutorials/wordpress/wp_step_by_step.rst In this case, the wiki should probably suggest setting up permalinks unless it's already the case. And the pattern should clearly appear as a recommendation. I will open an issue this time, but don't hesitate to participate to the wiki on github. Cheers From myles at magicalwonders.com Wed Mar 15 15:22:07 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Wed, 15 Mar 2017 15:22:07 +0000 Subject: Custom Permalinks Setting in Wordpress In-Reply-To: References: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> Message-ID: <6f7bc510-0230-ca9d-0976-788ffb375acd@magicalwonders.com> Ok Dridi, thanks. Step 3 is a bit ambiguous in its present form. I'm guessing that Wordpress default permalinks are not recommended in conjunction with Varnish? If that's the case, maybe the instruction would be clearer mentioning that and recommending to "change Permalinks to any valid custom structure" ? Or something like that. Best wishes, Myles On 15/03/2017 13:07, Dridi Boukelmoune wrote: >> Yes Varnish will still works with your custom Permalinks. >> >>> so I guess I can skip that step? >> Indeed you don't need to run the step 3 > Hello, > > There's a "Show on GitHub" link on all wiki pages, don't hesitate to > open an issue or even submit a pull request to help improve the docs. > > https://github.com/varnish/varnish-wiki/blob/master/source/content/tutorials/wordpress/wp_step_by_step.rst > > In this case, the wiki should probably suggest setting up permalinks > unless it's already the case. And the pattern should clearly appear as > a recommendation. > > I will open an issue this time, but don't hesitate to participate to > the wiki on github. > > Cheers > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Mar 15 15:25:29 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 15 Mar 2017 16:25:29 +0100 Subject: Custom Permalinks Setting in Wordpress In-Reply-To: <6f7bc510-0230-ca9d-0976-788ffb375acd@magicalwonders.com> References: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> <6f7bc510-0230-ca9d-0976-788ffb375acd@magicalwonders.com> Message-ID: On Wed, Mar 15, 2017 at 4:22 PM, Magical Wonders wrote: > Ok Dridi, thanks. > > Step 3 is a bit ambiguous in its present form. I'm guessing that Wordpress > default permalinks are not recommended in conjunction with Varnish? If > that's the case, maybe the instruction would be clearer mentioning that and > recommending to "change Permalinks to any valid custom structure" ? Or > something like that. Hello, Why don't you directly comment on the github issue? I'm not a wordpress user myself, so you're probably in a better position to explain what's confusing you. https://github.com/varnish/varnish-wiki/issues/3 Cheers From myles at magicalwonders.com Wed Mar 15 16:01:57 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Wed, 15 Mar 2017 16:01:57 +0000 Subject: Custom Permalinks Setting in Wordpress In-Reply-To: References: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> <6f7bc510-0230-ca9d-0976-788ffb375acd@magicalwonders.com> Message-ID: <1ab0935a-b8e0-d8ea-ac40-ee86f88d2c99@magicalwonders.com> Ok, I've added a comment now. I had to create a Github account and wait for a confirmation email before I could post. Myles :-) On 15/03/2017 15:25, Dridi Boukelmoune wrote: > On Wed, Mar 15, 2017 at 4:22 PM, Magical Wonders > wrote: >> Ok Dridi, thanks. >> >> Step 3 is a bit ambiguous in its present form. I'm guessing that Wordpress >> default permalinks are not recommended in conjunction with Varnish? If >> that's the case, maybe the instruction would be clearer mentioning that and >> recommending to "change Permalinks to any valid custom structure" ? Or >> something like that. > Hello, > > Why don't you directly comment on the github issue? I'm not a > wordpress user myself, so you're probably in a better position to > explain what's confusing you. > > https://github.com/varnish/varnish-wiki/issues/3 > > Cheers > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlouis at peoplenetonline.com Wed Mar 15 17:30:45 2017 From: jlouis at peoplenetonline.com (Jim Louis) Date: Wed, 15 Mar 2017 12:30:45 -0500 Subject: varnish caching with jsessionid being set Message-ID: Hello, I'm fairly new to varnish and have used it successfully in the past. Here they would like to use it but they have jsessionid tracking thru multiple VIPs. It seems that these jsessionids are being called cookies and I'm not able to cache anything. Is there a way to use the jsessionids and accomplish the caching? Thanks, Jim -- *James Louis* *Lead Systems Engineer* 4400 Baker Road, Minnetonka, MN 55343
CELL 612.203.2631 TOLL FREE 888-346-3486 x 622 | FAX 952-908-6129 http://www.peoplenetonline.com PeopleNet is the leading provider of fleet mobility systems to the transportation industry, including truckload, LTL, private, and energy service fleets. -------------- next part -------------- An HTML attachment was scrubbed... URL: From colas.delmas at gmail.com Wed Mar 15 19:43:42 2017 From: colas.delmas at gmail.com (Nicolas Delmas) Date: Wed, 15 Mar 2017 20:43:42 +0100 Subject: Custom Permalinks Setting in Wordpress In-Reply-To: <1ab0935a-b8e0-d8ea-ac40-ee86f88d2c99@magicalwonders.com> References: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> <6f7bc510-0230-ca9d-0976-788ffb375acd@magicalwonders.com> <1ab0935a-b8e0-d8ea-ac40-ee86f88d2c99@magicalwonders.com> Message-ID: I'm guessing that Wordpress default permalinks are not recommended in > conjunction with Varnish? If that's the case, maybe the instruction would > be clearer mentioning that and recommending to "change Permalinks to any > valid custom structure" ? Or something like that. The structure of URL doesn't impact Varnish. It doesn't care about it. For exemple on the same serveur I host 3 Wordpress and all have a different structure of link. I read again the step 3. And they just suggest to use permalink instead of this kind of link ?id_post=x . And you could use all kind of permalink : - %category_name%/%article_name/ - %article_name% - %date%/article_name% - ... *Nicolas Delmas* http://tutoandco.colas-delmas.fr/ 2017-03-15 17:01 GMT+01:00 Magical Wonders : > Ok, I've added a comment now. I had to create a Github account and wait > for a confirmation email before I could post. > > Myles :-) > > On 15/03/2017 15:25, Dridi Boukelmoune wrote: > > On Wed, Mar 15, 2017 at 4:22 PM, Magical Wonders wrote: > > Ok Dridi, thanks. > > Step 3 is a bit ambiguous in its present form. I'm guessing that Wordpress > default permalinks are not recommended in conjunction with Varnish? If > that's the case, maybe the instruction would be clearer mentioning that and > recommending to "change Permalinks to any valid custom structure" ? Or > something like that. > > Hello, > > Why don't you directly comment on the github issue? I'm not a > wordpress user myself, so you're probably in a better position to > explain what's confusing you. > https://github.com/varnish/varnish-wiki/issues/3 > > Cheers > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From myles at magicalwonders.com Wed Mar 15 20:17:13 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Wed, 15 Mar 2017 20:17:13 +0000 Subject: Custom Permalinks Setting in Wordpress In-Reply-To: References: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> <6f7bc510-0230-ca9d-0976-788ffb375acd@magicalwonders.com> <1ab0935a-b8e0-d8ea-ac40-ee86f88d2c99@magicalwonders.com> Message-ID: The way it's been written though suggests it has to be the specific custom structure mentioned. As it transpires the type of Permalinks used in Wordpress have no actual impact on Varnish there's not that much point in even referencing them? Unless I'm missing something? Myles :-) On 15/03/2017 19:43, Nicolas Delmas wrote: > > > I'm guessing that Wordpress default permalinks are not > recommended in conjunction with Varnish? If that's the case, maybe > the instruction would be clearer mentioning that and recommending > to "change Permalinks to any valid custom structure" ? Or > something like that. > > The structure of URL doesn't impact Varnish. It doesn't care about it. > For exemple on the same serveur I host 3 Wordpress and all have a > different structure of link. > I read again the step 3. And they just suggest to use permalink > instead of this kind of link ?id_post=x . And you could use all kind > of permalink : > > * %category_name%/%article_name/ > * %article_name% > * %date%/article_name% > * ... > > > > *Nicolas Delmas* > http://tutoandco.colas-delmas.fr/ > > > > > > > > 2017-03-15 17:01 GMT+01:00 Magical Wonders >: > > Ok, I've added a comment now. I had to create a Github account and > wait for a confirmation email before I could post. > > Myles :-) > > > On 15/03/2017 15:25, Dridi Boukelmoune wrote: >> On Wed, Mar 15, 2017 at 4:22 PM, Magical Wonders >> wrote: >>> Ok Dridi, thanks. >>> >>> Step 3 is a bit ambiguous in its present form. I'm guessing that Wordpress >>> default permalinks are not recommended in conjunction with Varnish? If >>> that's the case, maybe the instruction would be clearer mentioning that and >>> recommending to "change Permalinks to any valid custom structure" ? Or >>> something like that. >> Hello, >> >> Why don't you directly comment on the github issue? I'm not a >> wordpress user myself, so you're probably in a better position to >> explain what's confusing you. >> >> https://github.com/varnish/varnish-wiki/issues/3 >> >> >> Cheers >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From colas.delmas at gmail.com Thu Mar 16 07:03:56 2017 From: colas.delmas at gmail.com (Nicolas Delmas) Date: Thu, 16 Mar 2017 08:03:56 +0100 Subject: Custom Permalinks Setting in Wordpress In-Reply-To: References: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> <6f7bc510-0230-ca9d-0976-788ffb375acd@magicalwonders.com> <1ab0935a-b8e0-d8ea-ac40-ee86f88d2c99@magicalwonders.com> Message-ID: Of course, the type of Permalinks is important for SEO. I'm not enought good in this domain to suggest one more than the other. *Nicolas Delmas* http://tutoandco.colas-delmas.fr/ 2017-03-15 21:17 GMT+01:00 Magical Wonders : > The way it's been written though suggests it has to be the specific custom > structure mentioned. As it transpires the type of Permalinks used in > Wordpress have no actual impact on Varnish there's not that much point in > even referencing them? Unless I'm missing something? > > Myles :-) > > On 15/03/2017 19:43, Nicolas Delmas wrote: > > > I'm guessing that Wordpress default permalinks are not recommended in >> conjunction with Varnish? If that's the case, maybe the instruction would >> be clearer mentioning that and recommending to "change Permalinks to any >> valid custom structure" ? Or something like that. > > The structure of URL doesn't impact Varnish. It doesn't care about it. For > exemple on the same serveur I host 3 Wordpress and all have a different > structure of link. > I read again the step 3. And they just suggest to use permalink instead of > this kind of link ?id_post=x . And you could use all kind of permalink : > > - %category_name%/%article_name/ > - %article_name% > - %date%/article_name% > - ... > > > > > *Nicolas Delmas* > http://tutoandco.colas-delmas.fr/ > > > > > > > > 2017-03-15 17:01 GMT+01:00 Magical Wonders : > >> Ok, I've added a comment now. I had to create a Github account and wait >> for a confirmation email before I could post. >> >> Myles :-) >> >> On 15/03/2017 15:25, Dridi Boukelmoune wrote: >> >> On Wed, Mar 15, 2017 at 4:22 PM, Magical Wonders wrote: >> >> Ok Dridi, thanks. >> >> Step 3 is a bit ambiguous in its present form. I'm guessing that Wordpress >> default permalinks are not recommended in conjunction with Varnish? If >> that's the case, maybe the instruction would be clearer mentioning that and >> recommending to "change Permalinks to any valid custom structure" ? Or >> something like that. >> >> Hello, >> >> Why don't you directly comment on the github issue? I'm not a >> wordpress user myself, so you're probably in a better position to >> explain what's confusing you. >> https://github.com/varnish/varnish-wiki/issues/3 >> >> Cheers >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Mar 16 08:15:30 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 16 Mar 2017 09:15:30 +0100 Subject: Redirecting traffic to Varnish Cache Server In-Reply-To: <5f327260f6f74112a42d4b9582bee475@mbx1serv.meas-inc.com> References: <5f327260f6f74112a42d4b9582bee475@mbx1serv.meas-inc.com> Message-ID: Hi, "elsif" needs to be "else if", and I would add: else { return(synth(404)); } at the end to make sure you only serve content from these two domains. -- Guillaume Quintard On Tue, Mar 14, 2017 at 9:46 PM, Rodney Bizzell wrote: > So I have setup a basic default.vcl and my question how do I get the > Varnish server to answer web request before going to the backend drupal > servers not sure if I am stating this correctly. Our DNS admin point the > urls to Varnish server but I am getting page not found. Am I missing a > config in the default.vcl? Thanks! I researched and I am not seeing any > good information. > > > > backend drupal { > > .host = "drupal.miat.co"; > > .port = "80"; > > .connect_timeout = 6000s; > > .first_byte_timeout = 6000s; > > .between_bytes_timeout = 6000s; > > } > > > > backend ncwrite { > > .host = "ncwrite.miat.co"; > > .port = "80"; > > .connect_timeout = 6000s; > > .first_byte_timeout = 6000s; > > .between_bytes_timeout = 6000s; > > } > > > > sub vcl_recv { > > if (req.http.host == "drupal.miat.co"){ > > set req.backend_hint = drupal; > > } elsif (req.http.host == "ncwrite.miat.co"){ > > set req.backend_hint = ncwrite; > > return (hash); > > } > > } > > > > > This email (including any attachments) may contain confidential > information intended solely for acknowledged recipients. If you think you > have received this information in error, please reply to the sender and > delete all copies from your system. Please note that unauthorized use, > disclosure, or further distribution of this information is prohibited by > the sender. Note also that we may monitor email directed to or originating > from our network. Thank you for your consideration and assistance. | > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From myles at magicalwonders.com Thu Mar 16 08:42:28 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Thu, 16 Mar 2017 08:42:28 +0000 Subject: Custom Permalinks Setting in Wordpress In-Reply-To: References: <39d10de7-1cc6-0b9a-0c37-f38338f51849@magicalwonders.com> <6f7bc510-0230-ca9d-0976-788ffb375acd@magicalwonders.com> <1ab0935a-b8e0-d8ea-ac40-ee86f88d2c99@magicalwonders.com> Message-ID: <60d4f815-41d6-3a01-0e67-30ad837bc31f@magicalwonders.com> Yes, they are relevant for SEO, and opinions from experts vary as to which is the most effective. Yoast and many others suggest keeping it simple by using /%postname%/. However, others argue against that and suggest different structures! It's hard to know who is right sometimes! :-) Personally I like to keep things simple, but I can see why other structures may be more appropriate depending on the site involved. As far as SEO goes, switching from the default settings to a custom structure is definitely a good idea. :-) Best wishes, Myles On 16/03/2017 07:03, Nicolas Delmas wrote: > Of course, the type of Permalinks is important for SEO. I'm not > enought good in this domain to suggest one more than the other. > > *Nicolas Delmas* > http://tutoandco.colas-delmas.fr/ > > > > > > > > 2017-03-15 21:17 GMT+01:00 Magical Wonders >: > > The way it's been written though suggests it has to be the > specific custom structure mentioned. As it transpires the type of > Permalinks used in Wordpress have no actual impact on Varnish > there's not that much point in even referencing them? Unless I'm > missing something? > > Myles :-) > > > On 15/03/2017 19:43, Nicolas Delmas wrote: >> >> >> I'm guessing that Wordpress default permalinks are not >> recommended in conjunction with Varnish? If that's the case, >> maybe the instruction would be clearer mentioning that and >> recommending to "change Permalinks to any valid custom >> structure" ? Or something like that. >> >> The structure of URL doesn't impact Varnish. It doesn't care >> about it. For exemple on the same serveur I host 3 Wordpress and >> all have a different structure of link. >> I read again the step 3. And they just suggest to use permalink >> instead of this kind of link ?id_post=x . And you could use all >> kind of permalink : >> >> * %category_name%/%article_name/ >> * %article_name% >> * %date%/article_name% >> * ... >> >> >> >> *Nicolas Delmas* >> http://tutoandco.colas-delmas.fr/ >> >> >> >> >> >> >> >> 2017-03-15 17:01 GMT+01:00 Magical Wonders >> >: >> >> Ok, I've added a comment now. I had to create a Github >> account and wait for a confirmation email before I could post. >> >> Myles :-) >> >> >> On 15/03/2017 15:25, Dridi Boukelmoune wrote: >>> On Wed, Mar 15, 2017 at 4:22 PM, Magical Wonders >>> wrote: >>>> Ok Dridi, thanks. >>>> >>>> Step 3 is a bit ambiguous in its present form. I'm guessing that Wordpress >>>> default permalinks are not recommended in conjunction with Varnish? If >>>> that's the case, maybe the instruction would be clearer mentioning that and >>>> recommending to "change Permalinks to any valid custom structure" ? Or >>>> something like that. >>> Hello, >>> >>> Why don't you directly comment on the github issue? I'm not a >>> wordpress user myself, so you're probably in a better position to >>> explain what's confusing you. >>> >>> https://github.com/varnish/varnish-wiki/issues/3 >>> >>> >>> Cheers >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From myles at magicalwonders.com Thu Mar 16 12:05:33 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Thu, 16 Mar 2017 12:05:33 +0000 Subject: Server requirements for installing Varnish In-Reply-To: References: Message-ID: <0f51581a-42ba-0079-852e-bdab1f19ef79@magicalwonders.com> Hello Guys, I'm searching for some step by step instructions for the installation of Varnish, but I'm finding so many variations, it's making my head spin! I have a VPS running CENTOS 6.8 x86_64 virtuozzo. I found this resource which looks fairly straightforward as I do have cPanel/WHM - https://support.qualityunit.com/496090-How-to-install-Varnish-with-CPanel-and-CentOS-to-cache-static-content-on-server I know I would need to change the "rpm -Uvh http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish-release-3.0-1.noarch.rpm" to get the latest version, but not sure what the correct syntax for that would be? However, I've now seen this reference - https://www.saotn.org/install-varnish-cache-on-centos-6-7/ which says that Varnish relies on Jemalloc and says it should be installed! What the heck is Jemalloc? I can't see it mentioned anywhere else online in regards to installing Varnish. I found a mention of it in the Wiki though - https://www.varnish-software.com/wiki/search.html?q=Jemalloc&check_keywords=yes&area=default Looking at the FAQ in Wiki I can see that the following packages are also required in order for Varnish to work - autoconf automake jemalloc-devel libedit-devel libtool ncurses-devel pcre-devel pkgconfig python-docutils python-sphinx graphviz Navigation on a server is not my strong point. Are the packages likely to be referenced in one location? In other words, how do I find out if I've already got them? Hope someone can advise. Myles -------------- next part -------------- An HTML attachment was scrubbed... URL: From ciapnz at gmail.com Thu Mar 16 12:29:34 2017 From: ciapnz at gmail.com (Danila Vershinin) Date: Thu, 16 Mar 2017 15:29:34 +0300 Subject: Server requirements for installing Varnish In-Reply-To: <0f51581a-42ba-0079-852e-bdab1f19ef79@magicalwonders.com> References: <0f51581a-42ba-0079-852e-bdab1f19ef79@magicalwonders.com> Message-ID: <44D50CAB-431B-4815-A9E9-29069D5B6C81@gmail.com> Those are for building / compiling Varnish on your server. If you?re installing Varnish from its official repository, you don?t need any extras. > autoconf > automake > jemalloc-devel > libedit-devel > libtool > ncurses-devel > pcre-devel > pkgconfig > python-docutils > python-sphinx > graphviz From guillaume at varnish-software.com Thu Mar 16 12:37:15 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 16 Mar 2017 13:37:15 +0100 Subject: Server requirements for installing Varnish In-Reply-To: <0f51581a-42ba-0079-852e-bdab1f19ef79@magicalwonders.com> References: <0f51581a-42ba-0079-852e-bdab1f19ef79@magicalwonders.com> Message-ID: Doesn't "yum install varnish" just work? The package manager should pull all the required packages for you. -- Guillaume Quintard On Thu, Mar 16, 2017 at 1:05 PM, Magical Wonders wrote: > Hello Guys, > > I'm searching for some step by step instructions for the installation of > Varnish, but I'm finding so many variations, it's making my head spin! > > I have a VPS running CENTOS 6.8 x86_64 virtuozzo. I found this resource > which looks fairly straightforward as I do have cPanel/WHM - > https://support.qualityunit.com/496090-How-to-install- > Varnish-with-CPanel-and-CentOS-to-cache-static-content-on-server > > I know I would need to change the "rpm -Uvh http://repo.varnish-cache.org/ > redhat/varnish-3.0/el5/noarch/varnish-release-3.0-1.noarch.rpm" to get > the latest version, but not sure what the correct syntax for that would be? > > However, I've now seen this reference - https://www.saotn.org/install- > varnish-cache-on-centos-6-7/ which says that Varnish relies on Jemalloc > and says it should be installed! What the heck is Jemalloc? I can't see it > mentioned anywhere else online in regards to installing Varnish. I found a > mention of it in the Wiki though - https://www.varnish-software. > com/wiki/search.html?q=Jemalloc&check_keywords=yes&area=default > > Looking at the FAQ in Wiki I can see that the following packages are also > required in order for Varnish to work - > > autoconf > automake > jemalloc-devel > libedit-devel > libtool > ncurses-devel > pcre-devel > pkgconfig > python-docutils > python-sphinx > graphviz > > Navigation on a server is not my strong point. Are the packages likely to be referenced in one location? In other words, how do I find out if I've already got them? > > Hope someone can advise. > > Myles > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Mar 16 12:56:46 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 16 Mar 2017 13:56:46 +0100 Subject: varnish caching with jsessionid being set In-Reply-To: References: Message-ID: On Wed, Mar 15, 2017 at 6:30 PM, Jim Louis wrote: > > Hello, > > I'm fairly new to varnish and have used it successfully in the past. Here they would like to use it but they have jsessionid tracking thru multiple VIPs. It seems that these jsessionids are being called cookies and I'm not able to cache anything. > > Is there a way to use the jsessionids and accomplish the caching? Hello, Yes, there is. This is by far my "favorite" topic [1] so I wrote a blog post [2] about it. I would recommend carefulness and a solid understanding of HTTP and cookies first. It also depends a lot on whether your backend speaks proper HTTP and won't (mis)lead Varnish to do TheWrongThing(tm). Cheers, Dridi [1] just in case it wasn't clear, it's not [2] https://info.varnish-software.com/blog/yet-another-post-on-caching-vs-cookies From dridi at varni.sh Thu Mar 16 13:02:57 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 16 Mar 2017 14:02:57 +0100 Subject: Server requirements for installing Varnish In-Reply-To: References: <0f51581a-42ba-0079-852e-bdab1f19ef79@magicalwonders.com> Message-ID: On Thu, Mar 16, 2017 at 1:37 PM, Guillaume Quintard wrote: > Doesn't "yum install varnish" just work? The package manager should pull all > the required packages for you. Yes, only after you set up the repo, the rpm command does that but starting with 5.1 that won't be the case. I'd wait a bit until 5.1 is officially announced to get the new repos details. Dridi From myles at magicalwonders.com Thu Mar 16 13:33:49 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Thu, 16 Mar 2017 13:33:49 +0000 Subject: Server requirements for installing Varnish In-Reply-To: References: <0f51581a-42ba-0079-852e-bdab1f19ef79@magicalwonders.com> Message-ID: <8a0c5c52-201f-22d9-1c2e-7e178b7961f6@magicalwonders.com> Ok, I think I understand. So installing 5.1 when the new details are available, all the required packages will automatically be installed? Is this the page that will give the new details? - https://varnish-cache.org/releases/install_redhat.html#install-redhat Myles On 16/03/2017 13:02, Dridi Boukelmoune wrote: > On Thu, Mar 16, 2017 at 1:37 PM, Guillaume Quintard > wrote: >> Doesn't "yum install varnish" just work? The package manager should pull all >> the required packages for you. > Yes, only after you set up the repo, the rpm command does that but > starting with 5.1 that won't be the case. I'd wait a bit until 5.1 is > officially announced to get the new repos details. > > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Mar 16 13:47:15 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 16 Mar 2017 14:47:15 +0100 Subject: Server requirements for installing Varnish In-Reply-To: <8a0c5c52-201f-22d9-1c2e-7e178b7961f6@magicalwonders.com> References: <0f51581a-42ba-0079-852e-bdab1f19ef79@magicalwonders.com> <8a0c5c52-201f-22d9-1c2e-7e178b7961f6@magicalwonders.com> Message-ID: On Thu, Mar 16, 2017 at 2:33 PM, Magical Wonders wrote: > Ok, I think I understand. So installing 5.1 when the new details are > available, all the required packages will automatically be installed? > > Is this the page that will give the new details? - > https://varnish-cache.org/releases/install_redhat.html#install-redhat Yes, it is. You can also register to the varnish-announce mailing list to know as soon as it's available, it's very low traffic. Cheers From myles at magicalwonders.com Thu Mar 16 14:05:07 2017 From: myles at magicalwonders.com (Magical Wonders) Date: Thu, 16 Mar 2017 14:05:07 +0000 Subject: Server requirements for installing Varnish In-Reply-To: References: <0f51581a-42ba-0079-852e-bdab1f19ef79@magicalwonders.com> <8a0c5c52-201f-22d9-1c2e-7e178b7961f6@magicalwonders.com> Message-ID: <1281712a-99dc-81fc-e65f-af59fb3a3486@magicalwonders.com> Great, thank you. I've signed up to the varnish-announce mailing list now. :-) Best wishes, Myles On 16/03/2017 13:47, Dridi Boukelmoune wrote: > On Thu, Mar 16, 2017 at 2:33 PM, Magical Wonders > wrote: >> Ok, I think I understand. So installing 5.1 when the new details are >> available, all the required packages will automatically be installed? >> >> Is this the page that will give the new details? - >> https://varnish-cache.org/releases/install_redhat.html#install-redhat > Yes, it is. You can also register to the varnish-announce mailing list > to know as soon as it's available, it's very low traffic. > > Cheers > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Mar 16 14:38:27 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 16 Mar 2017 15:38:27 +0100 Subject: Redirecting traffic to Varnish Cache Server In-Reply-To: References: <5f327260f6f74112a42d4b9582bee475@mbx1serv.meas-inc.com> <2f6f97026ff84d6bb480734be8207dd9@mbx2serv.meas-inc.com> Message-ID: Don't forget to reply to the ML too :-) No need to return hash, because you already returned synthetic in that case. By the way, Dridi told me that vcl actually support "else if", "elsif" and "elif", so you can disregard my remark about it. On Mar 16, 2017 15:28, "Rodney Bizzell" wrote: Appreciate your help. One question the else { Return(synth(404)); } Do I add the return hash underneath it } } *From:* Guillaume Quintard [mailto:guillaume at varnish-software.com] *Sent:* Thursday, March 16, 2017 4:16 AM *To:* Rodney Bizzell *Cc:* varnish-misc at varnish-cache.org *Subject:* Re: Redirecting traffic to Varnish Cache Server Hi, "elsif" needs to be "else if", and I would add: else { return(synth(404)); } at the end to make sure you only serve content from these two domains. -- Guillaume Quintard On Tue, Mar 14, 2017 at 9:46 PM, Rodney Bizzell wrote: So I have setup a basic default.vcl and my question how do I get the Varnish server to answer web request before going to the backend drupal servers not sure if I am stating this correctly. Our DNS admin point the urls to Varnish server but I am getting page not found. Am I missing a config in the default.vcl? Thanks! I researched and I am not seeing any good information. backend drupal { .host = "drupal.miat.co"; .port = "80"; .connect_timeout = 6000s; .first_byte_timeout = 6000s; .between_bytes_timeout = 6000s; } backend ncwrite { .host = "ncwrite.miat.co"; .port = "80"; .connect_timeout = 6000s; .first_byte_timeout = 6000s; .between_bytes_timeout = 6000s; } sub vcl_recv { if (req.http.host == "drupal.miat.co"){ set req.backend_hint = drupal; } elsif (req.http.host == "ncwrite.miat.co"){ set req.backend_hint = ncwrite; return (hash); } } This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Thu Mar 16 15:06:33 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Thu, 16 Mar 2017 15:06:33 +0000 Subject: Redirecting traffic to Varnish Cache Server In-Reply-To: References: <5f327260f6f74112a42d4b9582bee475@mbx1serv.meas-inc.com> <2f6f97026ff84d6bb480734be8207dd9@mbx2serv.meas-inc.com> Message-ID: Ok thanks I actually changed it no big deal From: Guillaume Quintard [mailto:guillaume at varnish-software.com] Sent: Thursday, March 16, 2017 10:38 AM To: Rodney Bizzell ; varnish-misc Subject: RE: Redirecting traffic to Varnish Cache Server Don't forget to reply to the ML too :-) No need to return hash, because you already returned synthetic in that case. By the way, Dridi told me that vcl actually support "else if", "elsif" and "elif", so you can disregard my remark about it. On Mar 16, 2017 15:28, "Rodney Bizzell" > wrote: Appreciate your help. One question the else { Return(synth(404)); } Do I add the return hash underneath it } } From: Guillaume Quintard [mailto:guillaume at varnish-software.com] Sent: Thursday, March 16, 2017 4:16 AM To: Rodney Bizzell > Cc: varnish-misc at varnish-cache.org Subject: Re: Redirecting traffic to Varnish Cache Server Hi, "elsif" needs to be "else if", and I would add: else { return(synth(404)); } at the end to make sure you only serve content from these two domains. -- Guillaume Quintard On Tue, Mar 14, 2017 at 9:46 PM, Rodney Bizzell > wrote: So I have setup a basic default.vcl and my question how do I get the Varnish server to answer web request before going to the backend drupal servers not sure if I am stating this correctly. Our DNS admin point the urls to Varnish server but I am getting page not found. Am I missing a config in the default.vcl? Thanks! I researched and I am not seeing any good information. backend drupal { .host = "drupal.miat.co"; .port = "80"; .connect_timeout = 6000s; .first_byte_timeout = 6000s; .between_bytes_timeout = 6000s; } backend ncwrite { .host = "ncwrite.miat.co"; .port = "80"; .connect_timeout = 6000s; .first_byte_timeout = 6000s; .between_bytes_timeout = 6000s; } sub vcl_recv { if (req.http.host == "drupal.miat.co"){ set req.backend_hint = drupal; } elsif (req.http.host == "ncwrite.miat.co"){ set req.backend_hint = ncwrite; return (hash); } } This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Thu Mar 16 15:19:12 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Thu, 16 Mar 2017 15:19:12 +0000 Subject: Setting stale content Message-ID: Hello, I wanted to know what subroutine should I set the stale content before after vcl_recv. This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | -------------- next part -------------- An HTML attachment was scrubbed... URL: From maillist-varnish at iamafreeman.com Thu Mar 16 15:34:10 2017 From: maillist-varnish at iamafreeman.com (varnish list) Date: Thu, 16 Mar 2017 15:34:10 +0000 Subject: Server requirements for installing Varnish In-Reply-To: <1281712a-99dc-81fc-e65f-af59fb3a3486@magicalwonders.com> References: <0f51581a-42ba-0079-852e-bdab1f19ef79@magicalwonders.com> <8a0c5c52-201f-22d9-1c2e-7e178b7961f6@magicalwonders.com> <1281712a-99dc-81fc-e65f-af59fb3a3486@magicalwonders.com> Message-ID: Hello And if you want to use another rpm then yum localinstall rpmfile_or_url On 16 Mar 2017 14:07, "Magical Wonders" wrote: > Great, thank you. I've signed up to the varnish-announce mailing list now. > :-) > > Best wishes, > > Myles > > On 16/03/2017 13:47, Dridi Boukelmoune wrote: > > On Thu, Mar 16, 2017 at 2:33 PM, Magical Wonders wrote: > > Ok, I think I understand. So installing 5.1 when the new details are > available, all the required packages will automatically be installed? > > Is this the page that will give the new details? -https://varnish-cache.org/releases/install_redhat.html#install-redhat > > > Yes, it is. You can also register to the varnish-announce mailing list > to know as soon as it's available, it's very low traffic. > > Cheers > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From colas.delmas at gmail.com Thu Mar 16 15:41:56 2017 From: colas.delmas at gmail.com (Nicolas Delmas) Date: Thu, 16 Mar 2017 16:41:56 +0100 Subject: Setting stale content In-Reply-To: References: Message-ID: Hi, There the link to the documentation : https://varnish-cache.org/docs/4.1/users-guide/vcl-grace.html Hope it's what you're looking for . *Nicolas Delmas* http://tutoandco.colas-delmas.fr/ 2017-03-16 16:19 GMT+01:00 Rodney Bizzell : > Hello, > > I wanted to know what subroutine should I set the stale content before > after vcl_recv. > > > This email (including any attachments) may contain confidential > information intended solely for acknowledged recipients. If you think you > have received this information in error, please reply to the sender and > delete all copies from your system. Please note that unauthorized use, > disclosure, or further distribution of this information is prohibited by > the sender. Note also that we may monitor email directed to or originating > from our network. Thank you for your consideration and assistance. | > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.evonosky at gmail.com Thu Mar 16 17:28:35 2017 From: alex.evonosky at gmail.com (Alex Evonosky) Date: Thu, 16 Mar 2017 13:28:35 -0400 Subject: 5.1.1 compile Message-ID: Hello all- Quick question: I compiled 5.1.1 on two servers without issue, yet on another I get the following: make[3]: Nothing to be done for 'all'. make[3]: Leaving directory '/root/varnish/varnish-5.1.1/doc/graphviz' Making all in sphinx make[3]: Entering directory '/root/varnish/varnish-5.1.1/doc/sphinx' ../../bin/varnishd/varnishd -x parameter > include/params.rst Error: FAILED to set maximum for param critbit_cooloff: 254.000 Must be less than or equal to 254.000 FAILED to set maximum for param vsm_free_cooldown: 600.000 Must be less than or equal to 600.000 Makefile:621: recipe for target 'include/params.rst' failed make[3]: *** [include/params.rst] Error 2 make[3]: Leaving directory '/root/varnish/varnish-5.1.1/doc/sphinx' Makefile:396: recipe for target 'all-recursive' failed make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory '/root/varnish/varnish-5.1.1/doc' Makefile:543: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/root/varnish/varnish-5.1.1' Makefile:428: recipe for target 'all' failed make: *** [all] Error 2 Has anyone seen this before? Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Mar 16 17:39:45 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 16 Mar 2017 18:39:45 +0100 Subject: 5.1.1 compile In-Reply-To: References: Message-ID: > Has anyone seen this before? No, I believe you're the first to report that. Which platform are you building Varnish on? And how are you building Varnish? Dridi From phk at phk.freebsd.dk Thu Mar 16 17:41:02 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 16 Mar 2017 17:41:02 +0000 Subject: 5.1.1 compile In-Reply-To: References: Message-ID: <49577.1489686062@critter.freebsd.dk> -------- In message , Alex Evonosky writes: >Error: >FAILED to set maximum for param critbit_cooloff: 254.000 >Must be less than or equal to 254.000 > >FAILED to set maximum for param vsm_free_cooldown: 600.000 >Must be less than or equal to 600.000 >Has anyone seen this before? Yes. This is an incredibly silly floating-point rounding issue caused by a questionable decision Intel made back in 1977 :-/ See: https://github.com/varnishcache/varnish-cache/issues/1875#issuecomment-204020993 -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From alex.evonosky at gmail.com Thu Mar 16 18:06:12 2017 From: alex.evonosky at gmail.com (Alex Evonosky) Date: Thu, 16 Mar 2017 14:06:12 -0400 Subject: 5.1.1 compile In-Reply-To: <49577.1489686062@critter.freebsd.dk> References: <49577.1489686062@critter.freebsd.dk> Message-ID: thank you. with that referenced, varnish compiled. Thank you. On Thu, Mar 16, 2017 at 1:41 PM, Poul-Henning Kamp wrote: > -------- > In message 9vwA at mail.gmail.com> > , Alex Evonosky writes: > > >Error: > >FAILED to set maximum for param critbit_cooloff: 254.000 > >Must be less than or equal to 254.000 > > > >FAILED to set maximum for param vsm_free_cooldown: 600.000 > >Must be less than or equal to 600.000 > > >Has anyone seen this before? > > Yes. This is an incredibly silly floating-point rounding issue > caused by a questionable decision Intel made back in 1977 :-/ > > See: > > https://github.com/varnishcache/varnish-cache/ > issues/1875#issuecomment-204020993 > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hernan at cmsmedios.com Thu Mar 16 21:46:32 2017 From: hernan at cmsmedios.com (=?UTF-8?Q?Hern=C3=A1n_Marsili?=) Date: Thu, 16 Mar 2017 21:46:32 +0000 Subject: varnish with apache mod_auth Message-ID: Hi, We are having an issue with VARNISH and apache mod_auth. Varnish is on port 80 serving users and Apache is the backend. We have servers restricting access only to authenticated users or certain IP addresses. Since we installed Varnish the issue is that we need to enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can fetch content. The problem, is that the real IP is not used and all the other rules does not apply. Bottom line, how can we still control who is requesting using MOD_AUTH and having Varnish? Regards Hern?n. -------------- next part -------------- An HTML attachment was scrubbed... URL: From japrice at gmail.com Thu Mar 16 23:32:23 2017 From: japrice at gmail.com (Jason Price) Date: Thu, 16 Mar 2017 19:32:23 -0400 Subject: varnish with apache mod_auth In-Reply-To: References: Message-ID: I don't believe there's a trivial way to do this. Varnish will return the cached response to any IP address that comes calling. Even if the first request comes from a valid IP, which gets passed through via X-Forward or similar, and mod_auth is tweaked to respond to that, any subsequent request will not be seen by either apache or mod_auth at all. You have a few options: 1) IP Whitelists are a rather poor means of authentication. Moving to something else might be prudent. But that's not easy. 2) There are probably VMODs that do something similar. If not and if the list of IPs isn't too long, you could limit the IPs in VCL rather than mod_auth. 3) Push the list of IP addresses that can connect to the external port down to IPTables or similar. 4) Push the list of IP addresses to external Firewall, or Security Group or whatever. On Thu, Mar 16, 2017 at 5:46 PM, Hern?n Marsili wrote: > Hi, > > We are having an issue with VARNISH and apache mod_auth. Varnish is on > port 80 serving users and Apache is the backend. > > We have servers restricting access only to authenticated users or certain > IP addresses. Since we installed Varnish the issue is that we need to > enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can > fetch content. The problem, is that the real IP is not used and all the > other rules does not apply. > > Bottom line, how can we still control who is requesting using MOD_AUTH and > having Varnish? > > Regards > Hern?n. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leonfauster at googlemail.com Fri Mar 17 01:58:33 2017 From: leonfauster at googlemail.com (Leon Fauster) Date: Fri, 17 Mar 2017 02:58:33 +0100 Subject: Varnish5 / h2 Support In-Reply-To: <36934.1488655455@critter.freebsd.dk> References: <2FE9E296-BD63-490C-8CF3-0BE73FB22C42@googlemail.com> <36934.1488655455@critter.freebsd.dk> Message-ID: <4ED15FCA-33BA-4A7F-B6F7-ABDAE4A406C5@googlemail.com> > Am 04.03.2017 um 20:24 schrieb Poul-Henning Kamp : > > In message <2FE9E296-BD63-490C-8CF3-0BE73FB22C42 at googlemail.com>, Leon Fauster > writes: > >> How experimental is this h2 support (as stated in the docs)? >> Should I deploy my prod system with varnish5 and h2 support enabled or not? > > V5 is fine for production, H2 is not (yet). > > V5.1 will be out in two weeks, and it will have better H2 support, > but exactly how good is too early to say yet. thanks for the new 5.1.1 release. I got it packaged on CentOS 6. The webstack with hitch and httpd24 is working with h2. Any suggestion to stress test this setup (specially h2), next to h2load -c 100 -n 1000 https://server1/ and more prod traffic ...? Thanks, LF From lagged at gmail.com Fri Mar 17 07:43:29 2017 From: lagged at gmail.com (Andrei) Date: Fri, 17 Mar 2017 09:43:29 +0200 Subject: varnish with apache mod_auth In-Reply-To: References: Message-ID: Authenticated requests should typically bypass cache, unless you want to hash the related session id(s), however that can get "interesting". I suggest using an Apache module such as rpaf or remoteip in order for Apache to set the client IP from the X-Forwarded-For header set by Varnish. This way, you will not need to worry about whitelisting localhost, or other cucumbersome iptables rules, and your IP restrictions will work as intended. On Fri, Mar 17, 2017 at 1:32 AM, Jason Price wrote: > I don't believe there's a trivial way to do this. > > Varnish will return the cached response to any IP address that comes > calling. Even if the first request comes from a valid IP, which gets > passed through via X-Forward or similar, and mod_auth is tweaked to respond > to that, any subsequent request will not be seen by either apache or > mod_auth at all. > > You have a few options: > 1) IP Whitelists are a rather poor means of authentication. Moving to > something else might be prudent. But that's not easy. > 2) There are probably VMODs that do something similar. If not and if the > list of IPs isn't too long, you could limit the IPs in VCL rather than > mod_auth. > 3) Push the list of IP addresses that can connect to the external port > down to IPTables or similar. > 4) Push the list of IP addresses to external Firewall, or Security Group > or whatever. > > > > On Thu, Mar 16, 2017 at 5:46 PM, Hern?n Marsili > wrote: > >> Hi, >> >> We are having an issue with VARNISH and apache mod_auth. Varnish is on >> port 80 serving users and Apache is the backend. >> >> We have servers restricting access only to authenticated users or certain >> IP addresses. Since we installed Varnish the issue is that we need to >> enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can >> fetch content. The problem, is that the real IP is not used and all the >> other rules does not apply. >> >> Bottom line, how can we still control who is requesting using MOD_AUTH >> and having Varnish? >> >> Regards >> Hern?n. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Fri Mar 17 12:11:00 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Fri, 17 Mar 2017 12:11:00 +0000 Subject: Setting stale content In-Reply-To: References: Message-ID: Thanks! From: Nicolas Delmas [mailto:colas.delmas at gmail.com] Sent: Thursday, March 16, 2017 11:42 AM To: Rodney Bizzell Cc: varnish-misc at varnish-cache.org Subject: Re: Setting stale content Hi, There the link to the documentation : https://varnish-cache.org/docs/4.1/users-guide/vcl-grace.html Hope it's what you're looking for . Nicolas Delmas http://tutoandco.colas-delmas.fr/ 2017-03-16 16:19 GMT+01:00 Rodney Bizzell >: Hello, I wanted to know what subroutine should I set the stale content before after vcl_recv. This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hernan at cmsmedios.com Fri Mar 17 12:33:27 2017 From: hernan at cmsmedios.com (=?UTF-8?Q?Hern=C3=A1n_Marsili?=) Date: Fri, 17 Mar 2017 12:33:27 +0000 Subject: varnish with apache mod_auth In-Reply-To: References: Message-ID: Thank you! so, I figure I can parse the x-forwarded-for in which I have 3 ips. The first one is the customer, the second one is the one 1 need (the CDN) and the third I think is the load balancer. I can assign it to a new header x-cdn-ip and use apache_remoteip to use that ip as the connecting ip. What do you think? Only problem here is to parse the second iP. I have something like this: set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "^([^,]+),?.*$", "\1"); I was able to get the first IP but not the second only which is the one I need. Any one can point me in the right direction with the regsub? Thank you! On Fri, Mar 17, 2017 at 4:43 AM Andrei wrote: > Authenticated requests should typically bypass cache, unless you want to > hash the related session id(s), however that can get "interesting". I > suggest using an Apache module such as rpaf or remoteip in order for Apache > to set the client IP from the X-Forwarded-For header set by Varnish. This > way, you will not need to worry about whitelisting localhost, or other > cucumbersome iptables rules, and your IP restrictions will work as intended. > > On Fri, Mar 17, 2017 at 1:32 AM, Jason Price wrote: > > I don't believe there's a trivial way to do this. > > Varnish will return the cached response to any IP address that comes > calling. Even if the first request comes from a valid IP, which gets > passed through via X-Forward or similar, and mod_auth is tweaked to respond > to that, any subsequent request will not be seen by either apache or > mod_auth at all. > > You have a few options: > 1) IP Whitelists are a rather poor means of authentication. Moving to > something else might be prudent. But that's not easy. > 2) There are probably VMODs that do something similar. If not and if the > list of IPs isn't too long, you could limit the IPs in VCL rather than > mod_auth. > 3) Push the list of IP addresses that can connect to the external port > down to IPTables or similar. > 4) Push the list of IP addresses to external Firewall, or Security Group > or whatever. > > > > On Thu, Mar 16, 2017 at 5:46 PM, Hern?n Marsili > wrote: > > Hi, > > We are having an issue with VARNISH and apache mod_auth. Varnish is on > port 80 serving users and Apache is the backend. > > We have servers restricting access only to authenticated users or certain > IP addresses. Since we installed Varnish the issue is that we need to > enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can > fetch content. The problem, is that the real IP is not used and all the > other rules does not apply. > > Bottom line, how can we still control who is requesting using MOD_AUTH and > having Varnish? > > Regards > Hern?n. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Fri Mar 17 12:48:23 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Fri, 17 Mar 2017 12:48:23 +0000 Subject: load-balancing Message-ID: Hello, In the documentation is there a section on setting-up load-balancing between two varnish servers. This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Fri Mar 17 13:31:08 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 17 Mar 2017 14:31:08 +0100 Subject: varnish with apache mod_auth In-Reply-To: References: Message-ID: If you have the ability to compile a vmod, you can use split() from vmod-str (disclaimer: I wrote that) https://github.com/gquintard/libvmod-str/blob/master/src/vmod_str.vcc otherwise, to get the second ip, something like : regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2") should work. Fell free to test, using regex101.com for example. or better, a Varnish Test case Case: https://gist.github.com/gquintard/ee47432bb8b5c97b615d973b57b6338e test it using: varnishtest foo.vtc -- Guillaume Quintard On Fri, Mar 17, 2017 at 1:33 PM, Hern?n Marsili wrote: > Thank you! so, I figure I can parse the x-forwarded-for in which I have 3 > ips. The first one is the customer, the second one is the one 1 need (the > CDN) and the third I think is the load balancer. > > I can assign it to a new header x-cdn-ip and use apache_remoteip to use > that ip as the connecting ip. > > What do you think? > > Only problem here is to parse the second iP. I have something like this: > > set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "^([^,]+),?.*$", > "\1"); > > I was able to get the first IP but not the second only which is the one I > need. Any one can point me in the right direction with the regsub? > > Thank you! > > On Fri, Mar 17, 2017 at 4:43 AM Andrei wrote: > >> Authenticated requests should typically bypass cache, unless you want to >> hash the related session id(s), however that can get "interesting". I >> suggest using an Apache module such as rpaf or remoteip in order for Apache >> to set the client IP from the X-Forwarded-For header set by Varnish. This >> way, you will not need to worry about whitelisting localhost, or other >> cucumbersome iptables rules, and your IP restrictions will work as intended. >> >> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price wrote: >> >> I don't believe there's a trivial way to do this. >> >> Varnish will return the cached response to any IP address that comes >> calling. Even if the first request comes from a valid IP, which gets >> passed through via X-Forward or similar, and mod_auth is tweaked to respond >> to that, any subsequent request will not be seen by either apache or >> mod_auth at all. >> >> You have a few options: >> 1) IP Whitelists are a rather poor means of authentication. Moving to >> something else might be prudent. But that's not easy. >> 2) There are probably VMODs that do something similar. If not and if the >> list of IPs isn't too long, you could limit the IPs in VCL rather than >> mod_auth. >> 3) Push the list of IP addresses that can connect to the external port >> down to IPTables or similar. >> 4) Push the list of IP addresses to external Firewall, or Security Group >> or whatever. >> >> >> >> On Thu, Mar 16, 2017 at 5:46 PM, Hern?n Marsili >> wrote: >> >> Hi, >> >> We are having an issue with VARNISH and apache mod_auth. Varnish is on >> port 80 serving users and Apache is the backend. >> >> We have servers restricting access only to authenticated users or certain >> IP addresses. Since we installed Varnish the issue is that we need to >> enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can >> fetch content. The problem, is that the real IP is not used and all the >> other rules does not apply. >> >> Bottom line, how can we still control who is requesting using MOD_AUTH >> and having Varnish? >> >> Regards >> Hern?n. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> >> > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Fri Mar 17 13:32:19 2017 From: lagged at gmail.com (Andrei) Date: Fri, 17 Mar 2017 15:32:19 +0200 Subject: varnish with apache mod_auth In-Reply-To: References: Message-ID: Does the CDN not provide the IP you want in a separate header? Typically CDN's have custom headers for just that which you can use as well On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > If you have the ability to compile a vmod, you can use split() from > vmod-str (disclaimer: I wrote that) https://github.com/ > gquintard/libvmod-str/blob/master/src/vmod_str.vcc > > otherwise, to get the second ip, something like : > > regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2") > > should work. Fell free to test, using regex101.com for example. or > better, a Varnish Test case Case: https://gist.github.com/gquintard/ > ee47432bb8b5c97b615d973b57b6338e > test it using: varnishtest foo.vtc > > -- > Guillaume Quintard > > On Fri, Mar 17, 2017 at 1:33 PM, Hern?n Marsili > wrote: > >> Thank you! so, I figure I can parse the x-forwarded-for in which I have 3 >> ips. The first one is the customer, the second one is the one 1 need (the >> CDN) and the third I think is the load balancer. >> >> I can assign it to a new header x-cdn-ip and use apache_remoteip to use >> that ip as the connecting ip. >> >> What do you think? >> >> Only problem here is to parse the second iP. I have something like this: >> >> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, >> "^([^,]+),?.*$", "\1"); >> >> I was able to get the first IP but not the second only which is the one I >> need. Any one can point me in the right direction with the regsub? >> >> Thank you! >> >> On Fri, Mar 17, 2017 at 4:43 AM Andrei wrote: >> >>> Authenticated requests should typically bypass cache, unless you want to >>> hash the related session id(s), however that can get "interesting". I >>> suggest using an Apache module such as rpaf or remoteip in order for Apache >>> to set the client IP from the X-Forwarded-For header set by Varnish. This >>> way, you will not need to worry about whitelisting localhost, or other >>> cucumbersome iptables rules, and your IP restrictions will work as intended. >>> >>> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price wrote: >>> >>> I don't believe there's a trivial way to do this. >>> >>> Varnish will return the cached response to any IP address that comes >>> calling. Even if the first request comes from a valid IP, which gets >>> passed through via X-Forward or similar, and mod_auth is tweaked to respond >>> to that, any subsequent request will not be seen by either apache or >>> mod_auth at all. >>> >>> You have a few options: >>> 1) IP Whitelists are a rather poor means of authentication. Moving to >>> something else might be prudent. But that's not easy. >>> 2) There are probably VMODs that do something similar. If not and if >>> the list of IPs isn't too long, you could limit the IPs in VCL rather than >>> mod_auth. >>> 3) Push the list of IP addresses that can connect to the external port >>> down to IPTables or similar. >>> 4) Push the list of IP addresses to external Firewall, or Security Group >>> or whatever. >>> >>> >>> >>> On Thu, Mar 16, 2017 at 5:46 PM, Hern?n Marsili >>> wrote: >>> >>> Hi, >>> >>> We are having an issue with VARNISH and apache mod_auth. Varnish is on >>> port 80 serving users and Apache is the backend. >>> >>> We have servers restricting access only to authenticated users or >>> certain IP addresses. Since we installed Varnish the issue is that we need >>> to enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can >>> fetch content. The problem, is that the real IP is not used and all the >>> other rules does not apply. >>> >>> Bottom line, how can we still control who is requesting using MOD_AUTH >>> and having Varnish? >>> >>> Regards >>> Hern?n. >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >>> >>> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Fri Mar 17 13:34:20 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 17 Mar 2017 14:34:20 +0100 Subject: load-balancing In-Reply-To: References: Message-ID: The docs have this: http://varnish-cache.org/docs/5.0/reference/vcl.html (you are interested in bereq.backend and req.backend_hint) and http://varnish-cache.org/docs/5.0/reference/vmod_directors.generated.html And there is a couple of articles about it: https://info.varnish-software.com/blog/backends-load-balancing https://info.varnish-software.com/blog/backends-load-balancing-part-2 -- Guillaume Quintard On Fri, Mar 17, 2017 at 1:48 PM, Rodney Bizzell wrote: > Hello, > > In the documentation is there a section on setting-up load-balancing > between two varnish servers. > > > This email (including any attachments) may contain confidential > information intended solely for acknowledged recipients. If you think you > have received this information in error, please reply to the sender and > delete all copies from your system. Please note that unauthorized use, > disclosure, or further distribution of this information is prohibited by > the sender. Note also that we may monitor email directed to or originating > from our network. Thank you for your consideration and assistance. | > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Fri Mar 17 13:47:21 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Fri, 17 Mar 2017 13:47:21 +0000 Subject: load-balancing In-Reply-To: References: Message-ID: <172c9f21a64b46fe9fd22531dc9f1458@mbx2serv.meas-inc.com> Thanks! From: Guillaume Quintard [mailto:guillaume at varnish-software.com] Sent: Friday, March 17, 2017 9:34 AM To: Rodney Bizzell Cc: varnish-misc at varnish-cache.org Subject: Re: load-balancing The docs have this: http://varnish-cache.org/docs/5.0/reference/vcl.html (you are interested in bereq.backend and req.backend_hint) and http://varnish-cache.org/docs/5.0/reference/vmod_directors.generated.html And there is a couple of articles about it: https://info.varnish-software.com/blog/backends-load-balancing https://info.varnish-software.com/blog/backends-load-balancing-part-2 -- Guillaume Quintard On Fri, Mar 17, 2017 at 1:48 PM, Rodney Bizzell > wrote: Hello, In the documentation is there a section on setting-up load-balancing between two varnish servers. This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From cbj at touristonline.dk Fri Mar 17 16:32:29 2017 From: cbj at touristonline.dk (=?UTF-8?Q?Christian_Bj=C3=B8rnbak?=) Date: Fri, 17 Mar 2017 17:32:29 +0100 Subject: h2 in 5.1.1 and jsessionid cookies ? Message-ID: Hi, I setup h2 with Hitch 1.4.4 and Varnish 5.1.1 following the guide in the release announcement. It works for stateless pages but if I try to log into our extranet where we use an jsessionid cookie varnish seems to discard the cookie from the request. When I enter the login page I receive a jsessionid cookie in the browser. When I submit the login form I am redirected to the login page with a new jsessionid cookie. If I disable alpn in the Hitch config everything works at it use to but of course without h2... >From what I can find on google h2 is suppose to support h1 style cookies: http://unrestful.io/2015/06/21/cookies.html Do I need to change something in the VCL to support (jsessionid) cookies with h2? Med venlig hilsen / Kind regards, Christian Bj?rnbak Chefudvikler / Lead Developer TouristOnline A/S Islands Brygge 43 2300 K?benhavn S Denmark TLF: +45 32888230 Dir. TLF: +45 32888235 -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Fri Mar 17 16:47:28 2017 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 17 Mar 2017 16:47:28 +0000 Subject: h2 in 5.1.1 and jsessionid cookies ? In-Reply-To: References: Message-ID: <58671.1489769248@critter.freebsd.dk> -------- Hej Christian, Can you capture a varnishlog please ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From rbizzell at measinc.com Fri Mar 17 17:22:01 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Fri, 17 Mar 2017 17:22:01 +0000 Subject: only cache Get and Head Message-ID: I wanted to know what is the correct syntax to only want to cache Get and Head if req.method. I used this and O got and error when I checked the syntax. If (req.method ! = "GET" && req.method ! = "HEAD") { Return (pass); } Does this need to inserted in a particular place This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | -------------- next part -------------- An HTML attachment was scrubbed... URL: From kokoniimasu at gmail.com Fri Mar 17 17:26:06 2017 From: kokoniimasu at gmail.com (kokoniimasu) Date: Sat, 18 Mar 2017 02:26:06 +0900 Subject: h2 in 5.1.1 and jsessionid cookies ? In-Reply-To: References: Message-ID: Hi,Christian. #sorry I forgot add ml-list... Are you manipulating cookies in Varnish?(set, get...) Some browser send several cookie header by H/2. Probably in order to make HPACK compression more effective. you may want to use std.collect. I added the this VCL in my environment. sub vcl_recv{ if(req.proto ~ "HTTP/2"){ if(req.http.cookie){ std.collect(req.http.cookie); set req.http.cookie = regsuball(req.http.cookie,", ","; "); } if(req.http.content-length){ // temporary... // https://github.com/varnishcache/varnish-cache/issues/2247 unset req.http.content-length; } } } I hope to reference. -- Shohei Tanaka(@xcir) http://blog.xcir.net/ 2017-03-18 1:32 GMT+09:00 Christian Bj?rnbak : > Hi, > > I setup h2 with Hitch 1.4.4 and Varnish 5.1.1 following the guide in the > release announcement. > > It works for stateless pages but if I try to log into our extranet where we > use an jsessionid cookie varnish seems to discard the cookie from the > request. > > When I enter the login page I receive a jsessionid cookie in the browser. > > When I submit the login form I am redirected to the login page with a new > jsessionid cookie. > > If I disable alpn in the Hitch config everything works at it use to but of > course without h2... > > > From what I can find on google h2 is suppose to support h1 style cookies: > http://unrestful.io/2015/06/21/cookies.html > > > Do I need to change something in the VCL to support (jsessionid) cookies > with h2? > > > > Med venlig hilsen / Kind regards, > > Christian Bj?rnbak > > Chefudvikler / Lead Developer > TouristOnline A/S > Islands Brygge 43 > 2300 K?benhavn S > Denmark > TLF: +45 32888230 > Dir. TLF: +45 32888235 > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From geoff at uplex.de Fri Mar 17 17:40:03 2017 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 17 Mar 2017 18:40:03 +0100 Subject: only cache Get and Head In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 03/17/2017 06:22 PM, Rodney Bizzell wrote: > I wanted to know what is the correct syntax to only want to cache > Get and Head if req.method. I used this and O got and error when I > checked the syntax. > > If (req.method ! = ?GET? && req.method ! = ?HEAD?) { > > Return (pass); > > } The "if" and "return" keywords are all lower-case. If there's a syntax error, you get an error message from the VCL compiler. The messages are not always very explanatory, but mostly they are, and you should always send along the contents of the error message when you're asking for help. (You might find that it explains the problem well enough that you don't have to ask in the first place.) Also: this exact code is in builtin.vcl for vcl_recv (of course with the correct syntax for the keywords). Can you just let your own vcl_recv drop into the built-in version? If your own VCL subroutines end at any point without calling return(), then Varnish goes on to execute the built-in version. If you can let that happen, then you don't need to do this yourself at all. If this is *only* thing you want in vcl_recv, then you don't need to write your own vcl_recv, just let builtin.vcl do the job. HTH, Geoff - -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYzB9zAAoJEOUwvh9pJNURYD4P/3CNzqUv0Xs4+RIDrO/7raPo Oi9+BHSdbZoFLtzo1YXtm2AKiyPj64KnrGmj6s/rJqTlbkJF8rDXpCY5M4onF93a y0knZP5O30Sn/8gSkH8ClcRtN0qIuj87KoukrEetvN3TC/TWKa8ibBS/zcJSjv7L d5nmz4RQD3kuBNVfuEy5ALFTdzOw+G/C+ORynE1TTg0DUNr+PZIXCpRytsXy3HKe 4bIpIaWTZS167TgFpIK4nZTA5ZriXGJj8yK9BU0m21SONxKLm6T4M2PZi3N2OJ/J Bot2UAhXzf8onIf98JM+yhd7C8hBSajvpxAusb3vms/bKDhxOpvbXUZSkeUfBMsv mXfDPM/MKB5T9NPPH21wwmdS+ZyiKCd7XBjRkKBxHlhocSr/9eweGcnJCGDTe+GN yDlWbYX9rOpGwcCo+5P2HXHO/qRTo8xKjkAgXAxNDDUeNwp19GLd2dvCN0/mSrb7 LAjuBuVYdWqavJRv0y/17zdwD2blG6k6Bxa10xzpEGj64s+fS077f3YcOLq0lZbA Ux3rd0T7mKYwK8UofVtftmPsrxkJ3VPPh24gjWR1tPpy0Sss6qo92pzMWyK2lOey BQqZ+FuNFOtb7H0cPDqB73m/xGYVe1X7wG5Pj4ZzR6GSS/mfw2iXKWWMYK1J4GsH Pk6XzObDga+mHW5lHsF1 =NOFE -----END PGP SIGNATURE----- From cbj at touristonline.dk Fri Mar 17 18:10:28 2017 From: cbj at touristonline.dk (=?UTF-8?Q?Christian_Bj=C3=B8rnbak?=) Date: Fri, 17 Mar 2017 19:10:28 +0100 Subject: h2 in 5.1.1 and jsessionid cookies ? In-Reply-To: References: Message-ID: Hi kokoniimasu, if(req.http.cookie){ std.collect(req.http.cookie); set req.http.cookie = regsuball(req.http.cookie,", ","; "); } did the trick for me.. Thanks for your help. The other if only seems to be relevant and valid if the backend is a varnish too? My backend is Apache. Med venlig hilsen / Kind regards, Christian Bj?rnbak Chefudvikler / Lead Developer TouristOnline A/S Islands Brygge 43 2300 K?benhavn S Denmark TLF: +45 32888230 Dir. TLF: +45 32888235 2017-03-17 18:26 GMT+01:00 kokoniimasu : > Hi,Christian. > > #sorry I forgot add ml-list... > > Are you manipulating cookies in Varnish?(set, get...) > Some browser send several cookie header by H/2. > Probably in order to make HPACK compression more effective. > you may want to use std.collect. > I added the this VCL in my environment. > > > sub vcl_recv{ > if(req.proto ~ "HTTP/2"){ > if(req.http.cookie){ > std.collect(req.http.cookie); > set req.http.cookie = regsuball(req.http.cookie,", ","; "); > } > if(req.http.content-length){ > // temporary... > // https://github.com/varnishcache/varnish-cache/issues/2247 > unset req.http.content-length; > } > } > } > > I hope to reference. > > -- > Shohei Tanaka(@xcir) > http://blog.xcir.net/ > > 2017-03-18 1:32 GMT+09:00 Christian Bj?rnbak : > > Hi, > > > > I setup h2 with Hitch 1.4.4 and Varnish 5.1.1 following the guide in the > > release announcement. > > > > It works for stateless pages but if I try to log into our extranet where > we > > use an jsessionid cookie varnish seems to discard the cookie from the > > request. > > > > When I enter the login page I receive a jsessionid cookie in the browser. > > > > When I submit the login form I am redirected to the login page with a new > > jsessionid cookie. > > > > If I disable alpn in the Hitch config everything works at it use to but > of > > course without h2... > > > > > > From what I can find on google h2 is suppose to support h1 style cookies: > > http://unrestful.io/2015/06/21/cookies.html > > > > > > Do I need to change something in the VCL to support (jsessionid) cookies > > with h2? > > > > > > > > Med venlig hilsen / Kind regards, > > > > Christian Bj?rnbak > > > > Chefudvikler / Lead Developer > > TouristOnline A/S > > Islands Brygge 43 > > 2300 K?benhavn S > > Denmark > > TLF: +45 32888230 > > Dir. TLF: +45 32888235 > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hernan at cmsmedios.com Fri Mar 17 18:23:29 2017 From: hernan at cmsmedios.com (=?UTF-8?Q?Hern=C3=A1n_Marsili?=) Date: Fri, 17 Mar 2017 18:23:29 +0000 Subject: varnish with apache mod_auth In-Reply-To: References: Message-ID: Ok, so I finally make it work with the suggested rule. On the vcl_recv I have: if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^ ,]+)[ ,]?.*", "\2"); } else { set req.http.X-Forwarded-For = client.ip; set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^ ,]+)[ ,]?.*", "\2"); } I then use Apache remote_ip to listen to x-cd-ip with this: RemoteIPHeader x-cdn-ip RemoteIPTrustedProxy 127.0.0.1 172.31.29.204 I don't probable need the IF but since this was in place for some reason, I just leave it. It seems to be working just fine. What do you think? On Fri, Mar 17, 2017 at 10:32 AM Andrei wrote: > Does the CDN not provide the IP you want in a separate header? Typically > CDN's have custom headers for just that which you can use as well > > On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > > If you have the ability to compile a vmod, you can use split() from > vmod-str (disclaimer: I wrote that) > https://github.com/gquintard/libvmod-str/blob/master/src/vmod_str.vcc > > otherwise, to get the second ip, something like : > > regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2") > > should work. Fell free to test, using regex101.com for example. or > better, a Varnish Test case Case: > https://gist.github.com/gquintard/ee47432bb8b5c97b615d973b57b6338e > test it using: varnishtest foo.vtc > > -- > Guillaume Quintard > > On Fri, Mar 17, 2017 at 1:33 PM, Hern?n Marsili > wrote: > > Thank you! so, I figure I can parse the x-forwarded-for in which I have 3 > ips. The first one is the customer, the second one is the one 1 need (the > CDN) and the third I think is the load balancer. > > I can assign it to a new header x-cdn-ip and use apache_remoteip to use > that ip as the connecting ip. > > What do you think? > > Only problem here is to parse the second iP. I have something like this: > > set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "^([^,]+),?.*$", > "\1"); > > I was able to get the first IP but not the second only which is the one I > need. Any one can point me in the right direction with the regsub? > > Thank you! > > On Fri, Mar 17, 2017 at 4:43 AM Andrei wrote: > > Authenticated requests should typically bypass cache, unless you want to > hash the related session id(s), however that can get "interesting". I > suggest using an Apache module such as rpaf or remoteip in order for Apache > to set the client IP from the X-Forwarded-For header set by Varnish. This > way, you will not need to worry about whitelisting localhost, or other > cucumbersome iptables rules, and your IP restrictions will work as intended. > > On Fri, Mar 17, 2017 at 1:32 AM, Jason Price wrote: > > I don't believe there's a trivial way to do this. > > Varnish will return the cached response to any IP address that comes > calling. Even if the first request comes from a valid IP, which gets > passed through via X-Forward or similar, and mod_auth is tweaked to respond > to that, any subsequent request will not be seen by either apache or > mod_auth at all. > > You have a few options: > 1) IP Whitelists are a rather poor means of authentication. Moving to > something else might be prudent. But that's not easy. > 2) There are probably VMODs that do something similar. If not and if the > list of IPs isn't too long, you could limit the IPs in VCL rather than > mod_auth. > 3) Push the list of IP addresses that can connect to the external port > down to IPTables or similar. > 4) Push the list of IP addresses to external Firewall, or Security Group > or whatever. > > > > On Thu, Mar 16, 2017 at 5:46 PM, Hern?n Marsili > wrote: > > Hi, > > We are having an issue with VARNISH and apache mod_auth. Varnish is on > port 80 serving users and Apache is the backend. > > We have servers restricting access only to authenticated users or certain > IP addresses. Since we installed Varnish the issue is that we need to > enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can > fetch content. The problem, is that the real IP is not used and all the > other rules does not apply. > > Bottom line, how can we still control who is requesting using MOD_AUTH and > having Varnish? > > Regards > Hern?n. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kokoniimasu at gmail.com Fri Mar 17 18:48:40 2017 From: kokoniimasu at gmail.com (kokoniimasu) Date: Sat, 18 Mar 2017 03:48:40 +0900 Subject: h2 in 5.1.1 and jsessionid cookies ? In-Reply-To: References: Message-ID: Hi,Christian. Yes, my backend is Varnish. Apache did not error by POST request in my test env too. Probably, C-L is ignored in apache.(I'm not read apache source.) rfc7230 is written If a message is received with both a Transfer-Encoding and a Content-Length header field, the Transfer-Encoding overrides the Content-Length. Such a message might indicate an attempt to perform request smuggling (Section 9.5) or response splitting (Section 9.4) and ought to be handled as an error. A sender MUST remove the received Content-Length field prior to forwarding such a message downstream. https://tools.ietf.org/html/rfc7230#section-3.3.3 2017-03-18 3:10 GMT+09:00 Christian Bj?rnbak : > Hi kokoniimasu, > > if(req.http.cookie){ > std.collect(req.http.cookie); > set req.http.cookie = regsuball(req.http.cookie,", ","; "); > } > > did the trick for me.. > > Thanks for your help. > > The other if only seems to be relevant and valid if the backend is a varnish > too? My backend is Apache. > > > > Med venlig hilsen / Kind regards, > > Christian Bj?rnbak > > Chefudvikler / Lead Developer > TouristOnline A/S > Islands Brygge 43 > 2300 K?benhavn S > Denmark > TLF: +45 32888230 > Dir. TLF: +45 32888235 > > 2017-03-17 18:26 GMT+01:00 kokoniimasu : >> >> Hi,Christian. >> >> #sorry I forgot add ml-list... >> >> Are you manipulating cookies in Varnish?(set, get...) >> Some browser send several cookie header by H/2. >> Probably in order to make HPACK compression more effective. >> you may want to use std.collect. >> I added the this VCL in my environment. >> >> >> sub vcl_recv{ >> if(req.proto ~ "HTTP/2"){ >> if(req.http.cookie){ >> std.collect(req.http.cookie); >> set req.http.cookie = regsuball(req.http.cookie,", ","; "); >> } >> if(req.http.content-length){ >> // temporary... >> // https://github.com/varnishcache/varnish-cache/issues/2247 >> unset req.http.content-length; >> } >> } >> } >> >> I hope to reference. >> >> -- >> Shohei Tanaka(@xcir) >> http://blog.xcir.net/ >> >> 2017-03-18 1:32 GMT+09:00 Christian Bj?rnbak : >> > Hi, >> > >> > I setup h2 with Hitch 1.4.4 and Varnish 5.1.1 following the guide in the >> > release announcement. >> > >> > It works for stateless pages but if I try to log into our extranet where >> > we >> > use an jsessionid cookie varnish seems to discard the cookie from the >> > request. >> > >> > When I enter the login page I receive a jsessionid cookie in the >> > browser. >> > >> > When I submit the login form I am redirected to the login page with a >> > new >> > jsessionid cookie. >> > >> > If I disable alpn in the Hitch config everything works at it use to but >> > of >> > course without h2... >> > >> > >> > From what I can find on google h2 is suppose to support h1 style >> > cookies: >> > http://unrestful.io/2015/06/21/cookies.html >> > >> > >> > Do I need to change something in the VCL to support (jsessionid) cookies >> > with h2? >> > >> > >> > >> > Med venlig hilsen / Kind regards, >> > >> > Christian Bj?rnbak >> > >> > Chefudvikler / Lead Developer >> > TouristOnline A/S >> > Islands Brygge 43 >> > 2300 K?benhavn S >> > Denmark >> > TLF: +45 32888230 >> > Dir. TLF: +45 32888235 >> > >> > _______________________________________________ >> > varnish-misc mailing list >> > varnish-misc at varnish-cache.org >> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > From jlouis at peoplenetonline.com Fri Mar 17 19:46:15 2017 From: jlouis at peoplenetonline.com (Jim Louis) Date: Fri, 17 Mar 2017 14:46:15 -0500 Subject: varnish caching with jsessionid being set In-Reply-To: References: Message-ID: Dridi, I must not be understanding your example on [2] as I'm still getting everything passed. The BereqHeader in varnishlog -b shows: BereqHeader Cookie: JSESSIONID=4A1158EB11C9E93D6AD2A101BB9FA204; pfmhelp=1; _ga=GA1.2.1694534677.1489779146; TrackJS=f2465051-cb66-4cfa-8a47-5c7a5d07aab4; navigation=520533110%7C4130 and I'm seeing: BerespHeader Cache-Control: no-cache, no-store, must-revalidate BerespHeader Pragma: no-cache Also, was there a part 2 to that blog? Thanks, Jim On Thu, Mar 16, 2017 at 7:56 AM, Dridi Boukelmoune wrote: > On Wed, Mar 15, 2017 at 6:30 PM, Jim Louis > wrote: > > > > Hello, > > > > I'm fairly new to varnish and have used it successfully in the past. > Here they would like to use it but they have jsessionid tracking thru > multiple VIPs. It seems that these jsessionids are being called cookies and > I'm not able to cache anything. > > > > Is there a way to use the jsessionids and accomplish the caching? > > Hello, > > Yes, there is. This is by far my "favorite" topic [1] so I wrote a > blog post [2] about it. I would recommend carefulness and a solid > understanding of HTTP and cookies first. It also depends a lot on > whether your backend speaks proper HTTP and won't (mis)lead > Varnish to do TheWrongThing(tm). > > Cheers, > Dridi > > [1] just in case it wasn't clear, it's not > [2] https://info.varnish-software.com/blog/yet-another-post-on- > caching-vs-cookies > -- *James Louis* *Lead Systems Engineer* 4400 Baker Road, Minnetonka, MN 55343
CELL 612.203.2631 TOLL FREE 888-346-3486 x 622 | FAX 952-908-6129 http://www.peoplenetonline.com PeopleNet is the leading provider of fleet mobility systems to the transportation industry, including truckload, LTL, private, and energy service fleets. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Fri Mar 17 19:48:17 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 17 Mar 2017 20:48:17 +0100 Subject: varnish with apache mod_auth In-Reply-To: References: Message-ID: Actually, Varnish should set the XFF header even before you enter vcl_recv. -- Guillaume Quintard On Mar 17, 2017 19:23, "Hern?n Marsili" wrote: > Ok, so I finally make it work with the suggested rule. > > On the vcl_recv I have: > > if (req.http.x-forwarded-for) { > > set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + > client.ip; > > set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^ > ,]+)[ ,]?.*", "\2"); > > } else { > > set req.http.X-Forwarded-For = client.ip; > > set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^ > ,]+)[ ,]?.*", "\2"); > > } > > I then use Apache remote_ip to listen to x-cd-ip with this: > > RemoteIPHeader x-cdn-ip > > RemoteIPTrustedProxy 127.0.0.1 172.31.29.204 > > I don't probable need the IF but since this was in place for some reason, > I just leave it. > > It seems to be working just fine. What do you think? > > On Fri, Mar 17, 2017 at 10:32 AM Andrei wrote: > >> Does the CDN not provide the IP you want in a separate header? Typically >> CDN's have custom headers for just that which you can use as well >> >> On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >> If you have the ability to compile a vmod, you can use split() from >> vmod-str (disclaimer: I wrote that) https://github.com/ >> gquintard/libvmod-str/blob/master/src/vmod_str.vcc >> >> otherwise, to get the second ip, something like : >> >> regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2") >> >> should work. Fell free to test, using regex101.com for example. or >> better, a Varnish Test case Case: https://gist.github.com/gquintard/ >> ee47432bb8b5c97b615d973b57b6338e >> test it using: varnishtest foo.vtc >> >> -- >> Guillaume Quintard >> >> On Fri, Mar 17, 2017 at 1:33 PM, Hern?n Marsili >> wrote: >> >> Thank you! so, I figure I can parse the x-forwarded-for in which I have 3 >> ips. The first one is the customer, the second one is the one 1 need (the >> CDN) and the third I think is the load balancer. >> >> I can assign it to a new header x-cdn-ip and use apache_remoteip to use >> that ip as the connecting ip. >> >> What do you think? >> >> Only problem here is to parse the second iP. I have something like this: >> >> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, >> "^([^,]+),?.*$", "\1"); >> >> I was able to get the first IP but not the second only which is the one I >> need. Any one can point me in the right direction with the regsub? >> >> Thank you! >> >> On Fri, Mar 17, 2017 at 4:43 AM Andrei wrote: >> >> Authenticated requests should typically bypass cache, unless you want to >> hash the related session id(s), however that can get "interesting". I >> suggest using an Apache module such as rpaf or remoteip in order for Apache >> to set the client IP from the X-Forwarded-For header set by Varnish. This >> way, you will not need to worry about whitelisting localhost, or other >> cucumbersome iptables rules, and your IP restrictions will work as intended. >> >> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price wrote: >> >> I don't believe there's a trivial way to do this. >> >> Varnish will return the cached response to any IP address that comes >> calling. Even if the first request comes from a valid IP, which gets >> passed through via X-Forward or similar, and mod_auth is tweaked to respond >> to that, any subsequent request will not be seen by either apache or >> mod_auth at all. >> >> You have a few options: >> 1) IP Whitelists are a rather poor means of authentication. Moving to >> something else might be prudent. But that's not easy. >> 2) There are probably VMODs that do something similar. If not and if the >> list of IPs isn't too long, you could limit the IPs in VCL rather than >> mod_auth. >> 3) Push the list of IP addresses that can connect to the external port >> down to IPTables or similar. >> 4) Push the list of IP addresses to external Firewall, or Security Group >> or whatever. >> >> >> >> On Thu, Mar 16, 2017 at 5:46 PM, Hern?n Marsili >> wrote: >> >> Hi, >> >> We are having an issue with VARNISH and apache mod_auth. Varnish is on >> port 80 serving users and Apache is the backend. >> >> We have servers restricting access only to authenticated users or certain >> IP addresses. Since we installed Varnish the issue is that we need to >> enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can >> fetch content. The problem, is that the real IP is not used and all the >> other rules does not apply. >> >> Bottom line, how can we still control who is requesting using MOD_AUTH >> and having Varnish? >> >> Regards >> Hern?n. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Sat Mar 18 06:11:35 2017 From: lagged at gmail.com (Andrei) Date: Sat, 18 Mar 2017 01:11:35 -0500 Subject: varnish with apache mod_auth In-Reply-To: References: Message-ID: Out of curiosity, has anyone done a CDN of Varnish servers? I have 4 Varnish servers in different datacenters around the world, and use anycast IPs to direct traffic based on the region. I managed to do cache replication using a "fanout" method for new cache hits to be replicated through an intermediary server to the related group of Varnish servers, but was wondering if anyone had a better method. On Fri, Mar 17, 2017 at 2:48 PM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > Actually, Varnish should set the XFF header even before you enter > vcl_recv. > > -- > Guillaume Quintard > > On Mar 17, 2017 19:23, "Hern?n Marsili" wrote: > >> Ok, so I finally make it work with the suggested rule. >> >> On the vcl_recv I have: >> >> if (req.http.x-forwarded-for) { >> >> set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + >> client.ip; >> >> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^ >> ,]+)[ ,]?.*", "\2"); >> >> } else { >> >> set req.http.X-Forwarded-For = client.ip; >> >> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^ >> ,]+)[ ,]?.*", "\2"); >> >> } >> >> I then use Apache remote_ip to listen to x-cd-ip with this: >> >> RemoteIPHeader x-cdn-ip >> >> RemoteIPTrustedProxy 127.0.0.1 172.31.29.204 >> >> I don't probable need the IF but since this was in place for some reason, >> I just leave it. >> >> It seems to be working just fine. What do you think? >> >> On Fri, Mar 17, 2017 at 10:32 AM Andrei wrote: >> >>> Does the CDN not provide the IP you want in a separate header? Typically >>> CDN's have custom headers for just that which you can use as well >>> >>> On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>> If you have the ability to compile a vmod, you can use split() from >>> vmod-str (disclaimer: I wrote that) https://github.com/gquin >>> tard/libvmod-str/blob/master/src/vmod_str.vcc >>> >>> otherwise, to get the second ip, something like : >>> >>> regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2") >>> >>> should work. Fell free to test, using regex101.com for example. or >>> better, a Varnish Test case Case: https://gist.github.com/ >>> gquintard/ee47432bb8b5c97b615d973b57b6338e >>> test it using: varnishtest foo.vtc >>> >>> -- >>> Guillaume Quintard >>> >>> On Fri, Mar 17, 2017 at 1:33 PM, Hern?n Marsili >>> wrote: >>> >>> Thank you! so, I figure I can parse the x-forwarded-for in which I have >>> 3 ips. The first one is the customer, the second one is the one 1 need (the >>> CDN) and the third I think is the load balancer. >>> >>> I can assign it to a new header x-cdn-ip and use apache_remoteip to use >>> that ip as the connecting ip. >>> >>> What do you think? >>> >>> Only problem here is to parse the second iP. I have something like this: >>> >>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, >>> "^([^,]+),?.*$", "\1"); >>> >>> I was able to get the first IP but not the second only which is the one >>> I need. Any one can point me in the right direction with the regsub? >>> >>> Thank you! >>> >>> On Fri, Mar 17, 2017 at 4:43 AM Andrei wrote: >>> >>> Authenticated requests should typically bypass cache, unless you want to >>> hash the related session id(s), however that can get "interesting". I >>> suggest using an Apache module such as rpaf or remoteip in order for Apache >>> to set the client IP from the X-Forwarded-For header set by Varnish. This >>> way, you will not need to worry about whitelisting localhost, or other >>> cucumbersome iptables rules, and your IP restrictions will work as intended. >>> >>> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price wrote: >>> >>> I don't believe there's a trivial way to do this. >>> >>> Varnish will return the cached response to any IP address that comes >>> calling. Even if the first request comes from a valid IP, which gets >>> passed through via X-Forward or similar, and mod_auth is tweaked to respond >>> to that, any subsequent request will not be seen by either apache or >>> mod_auth at all. >>> >>> You have a few options: >>> 1) IP Whitelists are a rather poor means of authentication. Moving to >>> something else might be prudent. But that's not easy. >>> 2) There are probably VMODs that do something similar. If not and if >>> the list of IPs isn't too long, you could limit the IPs in VCL rather than >>> mod_auth. >>> 3) Push the list of IP addresses that can connect to the external port >>> down to IPTables or similar. >>> 4) Push the list of IP addresses to external Firewall, or Security Group >>> or whatever. >>> >>> >>> >>> On Thu, Mar 16, 2017 at 5:46 PM, Hern?n Marsili >>> wrote: >>> >>> Hi, >>> >>> We are having an issue with VARNISH and apache mod_auth. Varnish is on >>> port 80 serving users and Apache is the backend. >>> >>> We have servers restricting access only to authenticated users or >>> certain IP addresses. Since we installed Varnish the issue is that we need >>> to enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can >>> fetch content. The problem, is that the real IP is not used and all the >>> other rules does not apply. >>> >>> Bottom line, how can we still control who is requesting using MOD_AUTH >>> and having Varnish? >>> >>> Regards >>> Hern?n. >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Sat Mar 18 06:37:47 2017 From: lagged at gmail.com (Andrei) Date: Sat, 18 Mar 2017 01:37:47 -0500 Subject: varnish with apache mod_auth In-Reply-To: References: Message-ID: whoops, I totally meant to reply to the load balancing thread :| On Sat, Mar 18, 2017 at 1:11 AM, Andrei wrote: > Out of curiosity, has anyone done a CDN of Varnish servers? I have 4 > Varnish servers in different datacenters around the world, and use anycast > IPs to direct traffic based on the region. I managed to do cache > replication using a "fanout" method for new cache hits to be replicated > through an intermediary server to the related group of Varnish servers, but > was wondering if anyone had a better method. > > On Fri, Mar 17, 2017 at 2:48 PM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> Actually, Varnish should set the XFF header even before you enter >> vcl_recv. >> >> -- >> Guillaume Quintard >> >> On Mar 17, 2017 19:23, "Hern?n Marsili" wrote: >> >>> Ok, so I finally make it work with the suggested rule. >>> >>> On the vcl_recv I have: >>> >>> if (req.http.x-forwarded-for) { >>> >>> set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " >>> + client.ip; >>> >>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^ >>> ,]+)[ ,]?.*", "\2"); >>> >>> } else { >>> >>> set req.http.X-Forwarded-For = client.ip; >>> >>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^ >>> ,]+)[ ,]?.*", "\2"); >>> >>> } >>> >>> I then use Apache remote_ip to listen to x-cd-ip with this: >>> >>> RemoteIPHeader x-cdn-ip >>> >>> RemoteIPTrustedProxy 127.0.0.1 172.31.29.204 >>> >>> I don't probable need the IF but since this was in place for some >>> reason, I just leave it. >>> >>> It seems to be working just fine. What do you think? >>> >>> On Fri, Mar 17, 2017 at 10:32 AM Andrei wrote: >>> >>>> Does the CDN not provide the IP you want in a separate header? >>>> Typically CDN's have custom headers for just that which you can use as well >>>> >>>> On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard < >>>> guillaume at varnish-software.com> wrote: >>>> >>>> If you have the ability to compile a vmod, you can use split() from >>>> vmod-str (disclaimer: I wrote that) https://github.com/gquin >>>> tard/libvmod-str/blob/master/src/vmod_str.vcc >>>> >>>> otherwise, to get the second ip, something like : >>>> >>>> regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2") >>>> >>>> should work. Fell free to test, using regex101.com for example. or >>>> better, a Varnish Test case Case: https://gist.github.com/ >>>> gquintard/ee47432bb8b5c97b615d973b57b6338e >>>> test it using: varnishtest foo.vtc >>>> >>>> -- >>>> Guillaume Quintard >>>> >>>> On Fri, Mar 17, 2017 at 1:33 PM, Hern?n Marsili >>>> wrote: >>>> >>>> Thank you! so, I figure I can parse the x-forwarded-for in which I have >>>> 3 ips. The first one is the customer, the second one is the one 1 need (the >>>> CDN) and the third I think is the load balancer. >>>> >>>> I can assign it to a new header x-cdn-ip and use apache_remoteip to use >>>> that ip as the connecting ip. >>>> >>>> What do you think? >>>> >>>> Only problem here is to parse the second iP. I have something like this: >>>> >>>> set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, >>>> "^([^,]+),?.*$", "\1"); >>>> >>>> I was able to get the first IP but not the second only which is the one >>>> I need. Any one can point me in the right direction with the regsub? >>>> >>>> Thank you! >>>> >>>> On Fri, Mar 17, 2017 at 4:43 AM Andrei wrote: >>>> >>>> Authenticated requests should typically bypass cache, unless you want >>>> to hash the related session id(s), however that can get "interesting". I >>>> suggest using an Apache module such as rpaf or remoteip in order for Apache >>>> to set the client IP from the X-Forwarded-For header set by Varnish. This >>>> way, you will not need to worry about whitelisting localhost, or other >>>> cucumbersome iptables rules, and your IP restrictions will work as intended. >>>> >>>> On Fri, Mar 17, 2017 at 1:32 AM, Jason Price wrote: >>>> >>>> I don't believe there's a trivial way to do this. >>>> >>>> Varnish will return the cached response to any IP address that comes >>>> calling. Even if the first request comes from a valid IP, which gets >>>> passed through via X-Forward or similar, and mod_auth is tweaked to respond >>>> to that, any subsequent request will not be seen by either apache or >>>> mod_auth at all. >>>> >>>> You have a few options: >>>> 1) IP Whitelists are a rather poor means of authentication. Moving to >>>> something else might be prudent. But that's not easy. >>>> 2) There are probably VMODs that do something similar. If not and if >>>> the list of IPs isn't too long, you could limit the IPs in VCL rather than >>>> mod_auth. >>>> 3) Push the list of IP addresses that can connect to the external port >>>> down to IPTables or similar. >>>> 4) Push the list of IP addresses to external Firewall, or Security >>>> Group or whatever. >>>> >>>> >>>> >>>> On Thu, Mar 16, 2017 at 5:46 PM, Hern?n Marsili >>>> wrote: >>>> >>>> Hi, >>>> >>>> We are having an issue with VARNISH and apache mod_auth. Varnish is on >>>> port 80 serving users and Apache is the backend. >>>> >>>> We have servers restricting access only to authenticated users or >>>> certain IP addresses. Since we installed Varnish the issue is that we need >>>> to enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can >>>> fetch content. The problem, is that the real IP is not used and all the >>>> other rules does not apply. >>>> >>>> Bottom line, how can we still control who is requesting using MOD_AUTH >>>> and having Varnish? >>>> >>>> Regards >>>> Hern?n. >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>>> >>>> >>>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Sat Mar 18 08:28:08 2017 From: lagged at gmail.com (Andrei) Date: Sat, 18 Mar 2017 03:28:08 -0500 Subject: load-balancing In-Reply-To: <172c9f21a64b46fe9fd22531dc9f1458@mbx2serv.meas-inc.com> References: <172c9f21a64b46fe9fd22531dc9f1458@mbx2serv.meas-inc.com> Message-ID: (dubbing this over from the mod_auth thread for relevance due to my mistake earlier) Out of curiosity, has anyone done a CDN of Varnish servers? I have 4 Varnish servers in different datacenters around the world, and use anycast IPs to direct traffic based on the region. I managed to do cache replication using a "fanout" method for new cache hits to be replicated through an intermediary server to the related group of Varnish servers, but was wondering if anyone had a better method of load balancing with cache replication. On Fri, Mar 17, 2017 at 8:47 AM, Rodney Bizzell wrote: > Thanks! > > > > *From:* Guillaume Quintard [mailto:guillaume at varnish-software.com] > *Sent:* Friday, March 17, 2017 9:34 AM > *To:* Rodney Bizzell > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: load-balancing > > > > The docs have this: http://varnish-cache.org/docs/5.0/reference/vcl.html > (you are interested in bereq.backend and req.backend_hint) and > http://varnish-cache.org/docs/5.0/reference/vmod_directors.generated.html > > > > And there is a couple of articles about it: > > https://info.varnish-software.com/blog/backends-load-balancing > > https://info.varnish-software.com/blog/backends-load-balancing-part-2 > > > -- > > Guillaume Quintard > > > > On Fri, Mar 17, 2017 at 1:48 PM, Rodney Bizzell > wrote: > > Hello, > > In the documentation is there a section on setting-up load-balancing > between two varnish servers. > > > > This email (including any attachments) may contain confidential > information intended solely for acknowledged recipients. If you think you > have received this information in error, please reply to the sender and > delete all copies from your system. Please note that unauthorized use, > disclosure, or further distribution of this information is prohibited by > the sender. Note also that we may monitor email directed to or originating > from our network. Thank you for your consideration and assistance. | > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hernan at cmsmedios.com Sat Mar 18 11:42:24 2017 From: hernan at cmsmedios.com (=?UTF-8?Q?Hern=C3=A1n_Marsili?=) Date: Sat, 18 Mar 2017 11:42:24 +0000 Subject: varnish with apache mod_auth In-Reply-To: References: Message-ID: Well, this finally did not work since 0,7% of the users end up with a 401 ACCESS DENIED. The problem seems to be users behind a proxy in which the structure of the IPs on the xff is changed. ;( On Fri, Mar 17, 2017 at 4:48 PM Guillaume Quintard < guillaume at varnish-software.com> wrote: > Actually, Varnish should set the XFF header even before you enter > vcl_recv. > > > -- > Guillaume Quintard > > On Mar 17, 2017 19:23, "Hern?n Marsili" wrote: > > Ok, so I finally make it work with the suggested rule. > > On the vcl_recv I have: > > if (req.http.x-forwarded-for) { > > set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + > client.ip; > > set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^ > ,]+)[ ,]?.*", "\2"); > > } else { > > set req.http.X-Forwarded-For = client.ip; > > set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "([^,]+), *([^ > ,]+)[ ,]?.*", "\2"); > > } > > I then use Apache remote_ip to listen to x-cd-ip with this: > > RemoteIPHeader x-cdn-ip > > RemoteIPTrustedProxy 127.0.0.1 172.31.29.204 > > I don't probable need the IF but since this was in place for some reason, > I just leave it. > > It seems to be working just fine. What do you think? > > On Fri, Mar 17, 2017 at 10:32 AM Andrei wrote: > > Does the CDN not provide the IP you want in a separate header? Typically > CDN's have custom headers for just that which you can use as well > > On Fri, Mar 17, 2017 at 3:31 PM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > > If you have the ability to compile a vmod, you can use split() from > vmod-str (disclaimer: I wrote that) > https://github.com/gquintard/libvmod-str/blob/master/src/vmod_str.vcc > > otherwise, to get the second ip, something like : > > regsub(req.http.xff, "([^,]+), *([^ ,]+)[ ,]?.*", "\2") > > should work. Fell free to test, using regex101.com for example. or > better, a Varnish Test case Case: > https://gist.github.com/gquintard/ee47432bb8b5c97b615d973b57b6338e > test it using: varnishtest foo.vtc > > -- > Guillaume Quintard > > On Fri, Mar 17, 2017 at 1:33 PM, Hern?n Marsili > wrote: > > Thank you! so, I figure I can parse the x-forwarded-for in which I have 3 > ips. The first one is the customer, the second one is the one 1 need (the > CDN) and the third I think is the load balancer. > > I can assign it to a new header x-cdn-ip and use apache_remoteip to use > that ip as the connecting ip. > > What do you think? > > Only problem here is to parse the second iP. I have something like this: > > set req.http.x-cdn-ip = regsub(req.http.X-Forwarded-For, "^([^,]+),?.*$", > "\1"); > > I was able to get the first IP but not the second only which is the one I > need. Any one can point me in the right direction with the regsub? > > Thank you! > > On Fri, Mar 17, 2017 at 4:43 AM Andrei wrote: > > Authenticated requests should typically bypass cache, unless you want to > hash the related session id(s), however that can get "interesting". I > suggest using an Apache module such as rpaf or remoteip in order for Apache > to set the client IP from the X-Forwarded-For header set by Varnish. This > way, you will not need to worry about whitelisting localhost, or other > cucumbersome iptables rules, and your IP restrictions will work as intended. > > On Fri, Mar 17, 2017 at 1:32 AM, Jason Price wrote: > > I don't believe there's a trivial way to do this. > > Varnish will return the cached response to any IP address that comes > calling. Even if the first request comes from a valid IP, which gets > passed through via X-Forward or similar, and mod_auth is tweaked to respond > to that, any subsequent request will not be seen by either apache or > mod_auth at all. > > You have a few options: > 1) IP Whitelists are a rather poor means of authentication. Moving to > something else might be prudent. But that's not easy. > 2) There are probably VMODs that do something similar. If not and if the > list of IPs isn't too long, you could limit the IPs in VCL rather than > mod_auth. > 3) Push the list of IP addresses that can connect to the external port > down to IPTables or similar. > 4) Push the list of IP addresses to external Firewall, or Security Group > or whatever. > > > > On Thu, Mar 16, 2017 at 5:46 PM, Hern?n Marsili > wrote: > > Hi, > > We are having an issue with VARNISH and apache mod_auth. Varnish is on > port 80 serving users and Apache is the backend. > > We have servers restricting access only to authenticated users or certain > IP addresses. Since we installed Varnish the issue is that we need to > enable 127.0.0.1 as a permitted IP (required ip rule) so the Varnish can > fetch content. The problem, is that the real IP is not used and all the > other rules does not apply. > > Bottom line, how can we still control who is requesting using MOD_AUTH and > having Varnish? > > Regards > Hern?n. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Mar 20 08:09:27 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 20 Mar 2017 09:09:27 +0100 Subject: varnish caching with jsessionid being set In-Reply-To: References: Message-ID: On Fri, Mar 17, 2017 at 8:46 PM, Jim Louis wrote: > > Dridi, > > I must not be understanding your example on [2] as I'm still getting everything passed. Hi Jim, Thanks for letting me know, I will try to clarify this part. > The BereqHeader in varnishlog -b shows: > BereqHeader Cookie: JSESSIONID=4A1158EB11C9E93D6AD2A101BB9FA204; pfmhelp=1; _ga=GA1.2.1694534677.1489779146; TrackJS=f2465051-cb66-4cfa-8a47-5c7a5d07aab4; navigation=520533110%7C4130 > > and I'm seeing: > BerespHeader Cache-Control: no-cache, no-store, must-revalidate > BerespHeader Pragma: no-cache The backend is telling Varnish, and ultimately the client, that the response cannot be cached. Depending on your version of Varnish it will indeed result in subsequent passed transaction. > Also, was there a part 2 to that blog? Part 2 is written, but still an early draft so not published yet. Cheers, Dridi From jlouis at peoplenetonline.com Mon Mar 20 15:34:33 2017 From: jlouis at peoplenetonline.com (Jim Louis) Date: Mon, 20 Mar 2017 10:34:33 -0500 Subject: varnish caching with jsessionid being set In-Reply-To: References: Message-ID: Thanks Dridi! On Mon, Mar 20, 2017 at 3:09 AM, Dridi Boukelmoune wrote: > On Fri, Mar 17, 2017 at 8:46 PM, Jim Louis > wrote: > > > > Dridi, > > > > I must not be understanding your example on [2] as I'm still getting > everything passed. > > Hi Jim, > > Thanks for letting me know, I will try to clarify this part. > > > The BereqHeader in varnishlog -b shows: > > BereqHeader Cookie: JSESSIONID=4A1158EB11C9E93D6AD2A101BB9FA204; > pfmhelp=1; _ga=GA1.2.1694534677.1489779146; TrackJS=f2465051-cb66-4cfa-8a47-5c7a5d07aab4; > navigation=520533110%7C4130 > > > > and I'm seeing: > > BerespHeader Cache-Control: no-cache, no-store, must-revalidate > > BerespHeader Pragma: no-cache > > The backend is telling Varnish, and ultimately the client, that the > response cannot be cached. Depending on your version of Varnish > it will indeed result in subsequent passed transaction. > > > Also, was there a part 2 to that blog? > > Part 2 is written, but still an early draft so not published yet. > > Cheers, > Dridi > -- *James Louis* *Lead Systems Engineer* 4400 Baker Road, Minnetonka, MN 55343
CELL 612.203.2631 TOLL FREE 888-346-3486 x 622 | FAX 952-908-6129 http://www.peoplenetonline.com PeopleNet is the leading provider of fleet mobility systems to the transportation industry, including truckload, LTL, private, and energy service fleets. -------------- next part -------------- An HTML attachment was scrubbed... URL: From japrice at gmail.com Tue Mar 21 01:45:26 2017 From: japrice at gmail.com (Jason Price) Date: Mon, 20 Mar 2017 21:45:26 -0400 Subject: load-balancing In-Reply-To: References: <172c9f21a64b46fe9fd22531dc9f1458@mbx2serv.meas-inc.com> Message-ID: Andrei: Why do you care that the cache is synchronized between each remote DC? The simple way would be to do your any-cast'd IPs be your front end, and have them all talk to a centralized 'private' varnish which fronts the actual service. This would allow the individual servers to have a more relevant to their queries cache, and the centralized version would handle any duplication of requests from making it back to the true backend. On Sat, Mar 18, 2017 at 4:28 AM, Andrei wrote: > (dubbing this over from the mod_auth thread for relevance due to my > mistake earlier) > > Out of curiosity, has anyone done a CDN of Varnish servers? I have 4 > Varnish servers in different datacenters around the world, and use anycast > IPs to direct traffic based on the region. I managed to do cache > replication using a "fanout" method for new cache hits to be replicated > through an intermediary server to the related group of Varnish servers, but > was wondering if anyone had a better method of load balancing with cache > replication. > > On Fri, Mar 17, 2017 at 8:47 AM, Rodney Bizzell > wrote: > >> Thanks! >> >> >> >> *From:* Guillaume Quintard [mailto:guillaume at varnish-software.com] >> *Sent:* Friday, March 17, 2017 9:34 AM >> *To:* Rodney Bizzell >> *Cc:* varnish-misc at varnish-cache.org >> *Subject:* Re: load-balancing >> >> >> >> The docs have this: http://varnish-cache.org/docs/5.0/reference/vcl.html >> (you are interested in bereq.backend and req.backend_hint) and >> http://varnish-cache.org/docs/5.0/reference/vmod_directors.generated.html >> >> >> >> And there is a couple of articles about it: >> >> https://info.varnish-software.com/blog/backends-load-balancing >> >> https://info.varnish-software.com/blog/backends-load-balancing-part-2 >> >> >> -- >> >> Guillaume Quintard >> >> >> >> On Fri, Mar 17, 2017 at 1:48 PM, Rodney Bizzell >> wrote: >> >> Hello, >> >> In the documentation is there a section on setting-up load-balancing >> between two varnish servers. >> >> >> >> This email (including any attachments) may contain confidential >> information intended solely for acknowledged recipients. If you think you >> have received this information in error, please reply to the sender and >> delete all copies from your system. Please note that unauthorized use, >> disclosure, or further distribution of this information is prohibited by >> the sender. Note also that we may monitor email directed to or originating >> from our network. Thank you for your consideration and assistance. | >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Tue Mar 21 05:54:08 2017 From: lagged at gmail.com (Andrei) Date: Tue, 21 Mar 2017 00:54:08 -0500 Subject: load-balancing In-Reply-To: References: <172c9f21a64b46fe9fd22531dc9f1458@mbx2serv.meas-inc.com> Message-ID: On Mon, Mar 20, 2017 at 8:45 PM, Jason Price wrote: > Andrei: > > Why do you care that the cache is synchronized between each remote DC? > Because that's the whole point of having a CDN, with High Availability. There's no reason not to keep cache consistent across the board, and have low latency per region regardless of requests from that particular region - aka cache priming. For example, if the origin server in North America receives a cacheable request from a client, that data is then instantly made available to all regions. So when client2 from Europe requests the same page, they get consistent data without the added 100ms+ latency spike per origin fetch. This is just good common practice for optimizing large traffic loads and keeping data consistent - imo. Afaik, Varnish Plus actually offers something along these lines, for a good reason - https://www.varnish-software.com/plus/varnish-high-availability/ > > The simple way would be to do your any-cast'd IPs be your front end, and > have them all talk to a centralized 'private' varnish which fronts the > actual service. > > This would allow the individual servers to have a more relevant to their > queries cache, and the centralized version would handle any duplication of > requests from making it back to the true backend. > > The anycast IPs are "in front" of the Varnish boxes. However, there is no "centralized" Varnish to handle the requests as that wouldn't be optimal for a CDN. The closest by comparison would be the origin server, in terms of a "centralized version". Think of this setup as a decentralized mesh which shares data among a set group of Varnish servers for High Availability and low latency regardless of region. > On Sat, Mar 18, 2017 at 4:28 AM, Andrei wrote: > >> (dubbing this over from the mod_auth thread for relevance due to my >> mistake earlier) >> >> Out of curiosity, has anyone done a CDN of Varnish servers? I have 4 >> Varnish servers in different datacenters around the world, and use anycast >> IPs to direct traffic based on the region. I managed to do cache >> replication using a "fanout" method for new cache hits to be replicated >> through an intermediary server to the related group of Varnish servers, but >> was wondering if anyone had a better method of load balancing with cache >> replication. >> >> On Fri, Mar 17, 2017 at 8:47 AM, Rodney Bizzell >> wrote: >> >>> Thanks! >>> >>> >>> >>> *From:* Guillaume Quintard [mailto:guillaume at varnish-software.com] >>> *Sent:* Friday, March 17, 2017 9:34 AM >>> *To:* Rodney Bizzell >>> *Cc:* varnish-misc at varnish-cache.org >>> *Subject:* Re: load-balancing >>> >>> >>> >>> The docs have this: http://varnish-cache.org/docs/5.0/reference/vcl.html >>> (you are interested in bereq.backend and req.backend_hint) and >>> http://varnish-cache.org/docs/5.0/reference/vmod_directors.g >>> enerated.html >>> >>> >>> >>> And there is a couple of articles about it: >>> >>> https://info.varnish-software.com/blog/backends-load-balancing >>> >>> https://info.varnish-software.com/blog/backends-load-balancing-part-2 >>> >>> >>> -- >>> >>> Guillaume Quintard >>> >>> >>> >>> On Fri, Mar 17, 2017 at 1:48 PM, Rodney Bizzell >>> wrote: >>> >>> Hello, >>> >>> In the documentation is there a section on setting-up load-balancing >>> between two varnish servers. >>> >>> >>> >>> This email (including any attachments) may contain confidential >>> information intended solely for acknowledged recipients. If you think you >>> have received this information in error, please reply to the sender and >>> delete all copies from your system. Please note that unauthorized use, >>> disclosure, or further distribution of this information is prohibited by >>> the sender. Note also that we may monitor email directed to or originating >>> from our network. Thank you for your consideration and assistance. | >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rzarouali at gmail.com Tue Mar 21 10:09:06 2017 From: rzarouali at gmail.com (Rachid Zarouali) Date: Tue, 21 Mar 2017 10:09:06 +0000 Subject: dynamic varnish vhost Message-ID: Hi here, We are trying to make varnish loads dynamically vcls contained in a directory. the idea behind is: - drop a new vcl in the directory - varnish will reload it's configuration something like: for URL: http://test.myserver.local:6081 varnish will load vcl from /home/myserver/test/default.vcl for URL: http://test2.myserver.local:6081 varnish will load vcl from /home/myserver/test2/default.vcl any idea on how to achieve that maybe ? Thanks a lot ! Regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Mar 21 10:34:35 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 21 Mar 2017 11:34:35 +0100 Subject: dynamic varnish vhost In-Reply-To: References: Message-ID: > any idea on how to achieve that maybe ? Hello, You could do that with VCL labels, but don't expect Varnish to lazy load/include VCLs during an HTTP transaction. http://varnish.org/docs/5.1/users-guide/vcl-separate.html Dridi From jlouis at peoplenetonline.com Tue Mar 21 15:11:11 2017 From: jlouis at peoplenetonline.com (Jim Louis) Date: Tue, 21 Mar 2017 10:11:11 -0500 Subject: varnish caching with jsessionid being set In-Reply-To: References: Message-ID: Dridi, Supposedly, nginx has the ability to ignore the caching headers. Does Varnish provide a similar mechanism? Thanks, Jim On Mon, Mar 20, 2017 at 3:09 AM, Dridi Boukelmoune wrote: > On Fri, Mar 17, 2017 at 8:46 PM, Jim Louis > wrote: > > > > Dridi, > > > > I must not be understanding your example on [2] as I'm still getting > everything passed. > > Hi Jim, > > Thanks for letting me know, I will try to clarify this part. > > > The BereqHeader in varnishlog -b shows: > > BereqHeader Cookie: JSESSIONID=4A1158EB11C9E93D6AD2A101BB9FA204; > pfmhelp=1; _ga=GA1.2.1694534677.1489779146; TrackJS=f2465051-cb66-4cfa-8a47-5c7a5d07aab4; > navigation=520533110%7C4130 > > > > and I'm seeing: > > BerespHeader Cache-Control: no-cache, no-store, must-revalidate > > BerespHeader Pragma: no-cache > > The backend is telling Varnish, and ultimately the client, that the > response cannot be cached. Depending on your version of Varnish > it will indeed result in subsequent passed transaction. > > > Also, was there a part 2 to that blog? > > Part 2 is written, but still an early draft so not published yet. > > Cheers, > Dridi > -- *James Louis* *Lead Systems Engineer* 4400 Baker Road, Minnetonka, MN 55343
CELL 612.203.2631 TOLL FREE 888-346-3486 x 622 | FAX 952-908-6129 http://www.peoplenetonline.com PeopleNet is the leading provider of fleet mobility systems to the transportation industry, including truckload, LTL, private, and energy service fleets. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ciapnz at gmail.com Tue Mar 21 18:11:54 2017 From: ciapnz at gmail.com (Danila Vershinin) Date: Tue, 21 Mar 2017 21:11:54 +0300 Subject: Log slow backend requests. VSL-query? Message-ID: Hello, Trying to log only the slow backend requests: varnishncsa -b -q 'Timestamp:Process[2] > 0.1? does not produce any output. In fact, even varnishncsa -b -q 'Timestamp:Process[2] > 0? does not work. What am I doing wrong? :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Tue Mar 21 18:28:10 2017 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 21 Mar 2017 19:28:10 +0100 Subject: Log slow backend requests. VSL-query? In-Reply-To: References: Message-ID: <2bc056f1-430b-3644-12f6-d4d1369a70b8@uplex.de> On 03/21/2017 07:11 PM, Danila Vershinin wrote: > > Trying to log only the slow backend requests: > > varnishncsa -b -q 'Timestamp:Process[2] > 0.1? Timestamp:Process only appears in the client-side logs, but with -b you're filtering for the backend logs. The "Backend fetch timestamps" section at the bottom of vsl(7) tells you the names of the timestamps for backends: http://www.varnish-cache.org/docs/5.1/reference/vsl.html#backend-fetch-timestamps The one you're probably looking for is Timestamp:Beresp[3]. Beresp is the timestamp set after receiving backend response headers (the first timestamp recorded after the backend starts sending a response). Field 3 is the time elapsed since the most recent timestamp, which would be Bereq (backend request sent), so this is the best measurement of how fast or slow a backend responds after receiving a request. One thing to watch out for: if there is a fetch error such as a timeout that results in no backend response being received at all, then Timestamp:Beresp isn't recorded in the log. Instead of that you get Timestamp:Error, so you might want to query for Timestamp:Error[3] as well. HTH, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ciapnz at gmail.com Tue Mar 21 18:39:47 2017 From: ciapnz at gmail.com (Danila Vershinin) Date: Tue, 21 Mar 2017 21:39:47 +0300 Subject: Log slow backend requests. VSL-query? In-Reply-To: <2bc056f1-430b-3644-12f6-d4d1369a70b8@uplex.de> References: <2bc056f1-430b-3644-12f6-d4d1369a70b8@uplex.de> Message-ID: Thank you very much, Geoff. > On 21 Mar 2017, at 21:28, Geoff Simmons wrote: > > On 03/21/2017 07:11 PM, Danila Vershinin wrote: >> >> Trying to log only the slow backend requests: >> >> varnishncsa -b -q 'Timestamp:Process[2] > 0.1? > > Timestamp:Process only appears in the client-side logs, but with -b > you're filtering for the backend logs. > > The "Backend fetch timestamps" section at the bottom of vsl(7) tells > you the names of the timestamps for backends: > > http://www.varnish-cache.org/docs/5.1/reference/vsl.html#backend-fetch-timestamps > > The one you're probably looking for is Timestamp:Beresp[3]. Beresp is > the timestamp set after receiving backend response headers (the first > timestamp recorded after the backend starts sending a response). Field > 3 is the time elapsed since the most recent timestamp, which would be > Bereq (backend request sent), so this is the best measurement of how > fast or slow a backend responds after receiving a request. > > One thing to watch out for: if there is a fetch error such as a > timeout that results in no backend response being received at all, > then Timestamp:Beresp isn't recorded in the log. Instead of that you > get Timestamp:Error, so you might want to query for Timestamp:Error[3] > as well. > > > HTH, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From rbizzell at measinc.com Tue Mar 21 20:14:11 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Tue, 21 Mar 2017 20:14:11 +0000 Subject: order of operations Message-ID: Hello, I just wanted to know what is the recommendations for order of operations. Is using the 4.0 config from github a good model to follow This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Mar 22 08:38:49 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 22 Mar 2017 09:38:49 +0100 Subject: order of operations In-Reply-To: References: Message-ID: Hi Rodney, I'm really unsure about what you mean. A minimal VCL is in my opinion the best way to start, ie. just use the default vcl provided by your package and build on that. Also, 4.0 is old, try using 4.1, or even 5.1 when it's packaged. -- Guillaume Quintard On Tue, Mar 21, 2017 at 9:14 PM, Rodney Bizzell wrote: > Hello, > > I just wanted to know what is the recommendations for order of operations. > Is using the 4.0 config from github a good model to follow > > > This email (including any attachments) may contain confidential > information intended solely for acknowledged recipients. If you think you > have received this information in error, please reply to the sender and > delete all copies from your system. Please note that unauthorized use, > disclosure, or further distribution of this information is prohibited by > the sender. Note also that we may monitor email directed to or originating > from our network. Thank you for your consideration and assistance. | > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Wed Mar 22 13:45:11 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Wed, 22 Mar 2017 13:45:11 +0000 Subject: order of operations In-Reply-To: References: Message-ID: Can you put vcl_init anywhere in the config file or any other subroutine? Does it look at that file first then goes to vcl_recv and then goes to vcl_pass vcl_hit etc etc? I have a small simple config and I wanted to add a little at a time. From: Guillaume Quintard [mailto:guillaume at varnish-software.com] Sent: Wednesday, March 22, 2017 4:39 AM To: Rodney Bizzell Cc: varnish-misc at varnish-cache.org Subject: Re: order of operations Hi Rodney, I'm really unsure about what you mean. A minimal VCL is in my opinion the best way to start, ie. just use the default vcl provided by your package and build on that. Also, 4.0 is old, try using 4.1, or even 5.1 when it's packaged. -- Guillaume Quintard On Tue, Mar 21, 2017 at 9:14 PM, Rodney Bizzell > wrote: Hello, I just wanted to know what is the recommendations for order of operations. Is using the 4.0 config from github a good model to follow This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Mar 22 13:59:43 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 22 Mar 2017 14:59:43 +0100 Subject: order of operations In-Reply-To: References: Message-ID: Ah, then yes, you can put it nearly anywhere you want. the vcl_* subroutines are callbacks, called by varnish, it doesn't matter where they are declared. However, know that routines with the same names are concatenated ( http://www.varnish-cache.org/docs/4.0/reference/vcl.html#multiple-subroutines ) -- Guillaume Quintard On Wed, Mar 22, 2017 at 2:45 PM, Rodney Bizzell wrote: > Can you put vcl_init anywhere in the config file or any other > subroutine? Does it look at that file first then goes to vcl_recv and then > goes to vcl_pass vcl_hit etc etc? I have a small simple config and I > wanted to add a little at a time. > > > > *From:* Guillaume Quintard [mailto:guillaume at varnish-software.com] > *Sent:* Wednesday, March 22, 2017 4:39 AM > *To:* Rodney Bizzell > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: order of operations > > > > Hi Rodney, > > > > I'm really unsure about what you mean. A minimal VCL is in my opinion the > best way to start, ie. just use the default vcl provided by your package > and build on that. > > > > Also, 4.0 is old, try using 4.1, or even 5.1 when it's packaged. > > > -- > > Guillaume Quintard > > > > On Tue, Mar 21, 2017 at 9:14 PM, Rodney Bizzell > wrote: > > Hello, > > I just wanted to know what is the recommendations for order of operations. > Is using the 4.0 config from github a good model to follow > > > > This email (including any attachments) may contain confidential > information intended solely for acknowledged recipients. If you think you > have received this information in error, please reply to the sender and > delete all copies from your system. Please note that unauthorized use, > disclosure, or further distribution of this information is prohibited by > the sender. Note also that we may monitor email directed to or originating > from our network. Thank you for your consideration and assistance. | > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Mar 22 14:07:47 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 22 Mar 2017 15:07:47 +0100 Subject: order of operations In-Reply-To: References: Message-ID: 4.0 -> 4.1/5.0/5.1 should be just a matter of upgrading the packages, the VCL shouldn't break (too much). Varnish doesn't need to be compatible with Drupal, only with HTTP :-) -- Guillaume Quintard On Wed, Mar 22, 2017 at 3:02 PM, Rodney Bizzell wrote: > Okay thanks appreciated sorry that I wasn?t clear. Do you know if 5.0 is > compatible with Drupal 8 yet. I know you suggested that I upgrade from 4.0 > to 4.1 or 5.1 what about 5.0 > > > > *From:* Guillaume Quintard [mailto:guillaume at varnish-software.com] > *Sent:* Wednesday, March 22, 2017 10:00 AM > > *To:* Rodney Bizzell > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: order of operations > > > > Ah, then yes, you can put it nearly anywhere you want. > > > > the vcl_* subroutines are callbacks, called by varnish, it doesn't matter > where they are declared. > > > > However, know that routines with the same names are concatenated ( > http://www.varnish-cache.org/docs/4.0/reference/vcl.html# > multiple-subroutines) > > > -- > > Guillaume Quintard > > > > On Wed, Mar 22, 2017 at 2:45 PM, Rodney Bizzell > wrote: > > Can you put vcl_init anywhere in the config file or any other > subroutine? Does it look at that file first then goes to vcl_recv and then > goes to vcl_pass vcl_hit etc etc? I have a small simple config and I > wanted to add a little at a time. > > > > *From:* Guillaume Quintard [mailto:guillaume at varnish-software.com] > *Sent:* Wednesday, March 22, 2017 4:39 AM > *To:* Rodney Bizzell > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: order of operations > > > > Hi Rodney, > > > > I'm really unsure about what you mean. A minimal VCL is in my opinion the > best way to start, ie. just use the default vcl provided by your package > and build on that. > > > > Also, 4.0 is old, try using 4.1, or even 5.1 when it's packaged. > > > -- > > Guillaume Quintard > > > > On Tue, Mar 21, 2017 at 9:14 PM, Rodney Bizzell > wrote: > > Hello, > > I just wanted to know what is the recommendations for order of operations. > Is using the 4.0 config from github a good model to follow > > > > This email (including any attachments) may contain confidential > information intended solely for acknowledged recipients. If you think you > have received this information in error, please reply to the sender and > delete all copies from your system. Please note that unauthorized use, > disclosure, or further distribution of this information is prohibited by > the sender. Note also that we may monitor email directed to or originating > from our network. Thank you for your consideration and assistance. | > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Mar 22 15:28:39 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 22 Mar 2017 16:28:39 +0100 Subject: varnish caching with jsessionid being set In-Reply-To: References: Message-ID: Sure, you can override them: set vcl_backend_response { set beresp.ttl = 5m; } -- Guillaume Quintard On Tue, Mar 21, 2017 at 4:11 PM, Jim Louis wrote: > Dridi, > > Supposedly, nginx has the ability to ignore the caching headers. Does > Varnish provide a similar mechanism? > > Thanks, > Jim > > On Mon, Mar 20, 2017 at 3:09 AM, Dridi Boukelmoune wrote: > >> On Fri, Mar 17, 2017 at 8:46 PM, Jim Louis >> wrote: >> > >> > Dridi, >> > >> > I must not be understanding your example on [2] as I'm still getting >> everything passed. >> >> Hi Jim, >> >> Thanks for letting me know, I will try to clarify this part. >> >> > The BereqHeader in varnishlog -b shows: >> > BereqHeader Cookie: JSESSIONID=4A1158EB11C9E93D6AD2A101BB9FA204; >> pfmhelp=1; _ga=GA1.2.1694534677.1489779146; >> TrackJS=f2465051-cb66-4cfa-8a47-5c7a5d07aab4; navigation=520533110%7C4130 >> > >> > and I'm seeing: >> > BerespHeader Cache-Control: no-cache, no-store, must-revalidate >> > BerespHeader Pragma: no-cache >> >> The backend is telling Varnish, and ultimately the client, that the >> response cannot be cached. Depending on your version of Varnish >> it will indeed result in subsequent passed transaction. >> >> > Also, was there a part 2 to that blog? >> >> Part 2 is written, but still an early draft so not published yet. >> >> Cheers, >> Dridi >> > > > > -- > *James Louis* > *Lead Systems Engineer* > > > > 4400 Baker Road, Minnetonka, MN 55343
> CELL 612.203.2631 <(612)%20203-2631> > TOLL FREE 888-346-3486 x 622 <(888)%20346-3486> | FAX 952-908-6129 > <(952)%20908-6129> > http://www.peoplenetonline.com > > PeopleNet is the leading provider of fleet mobility systems to the > transportation industry, including truckload, LTL, private, and energy > service fleets. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Wed Mar 22 16:19:09 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Wed, 22 Mar 2017 16:19:09 +0000 Subject: varnish caching with jsessionid being set In-Reply-To: References: Message-ID: <0d193e417392443590edea72a3f09469@mbx1serv.meas-inc.com> Can you also add the set beresp.grace parameter as well right after the ttl From: varnish-misc-bounces+rbizzell=measinc.com at varnish-cache.org [mailto:varnish-misc-bounces+rbizzell=measinc.com at varnish-cache.org] On Behalf Of Guillaume Quintard Sent: Wednesday, March 22, 2017 11:29 AM To: Jim Louis Cc: varnish-misc Subject: Re: varnish caching with jsessionid being set Sure, you can override them: set vcl_backend_response { set beresp.ttl = 5m; } -- Guillaume Quintard On Tue, Mar 21, 2017 at 4:11 PM, Jim Louis > wrote: Dridi, Supposedly, nginx has the ability to ignore the caching headers. Does Varnish provide a similar mechanism? Thanks, Jim On Mon, Mar 20, 2017 at 3:09 AM, Dridi Boukelmoune > wrote: On Fri, Mar 17, 2017 at 8:46 PM, Jim Louis > wrote: > > Dridi, > > I must not be understanding your example on [2] as I'm still getting everything passed. Hi Jim, Thanks for letting me know, I will try to clarify this part. > The BereqHeader in varnishlog -b shows: > BereqHeader Cookie: JSESSIONID=4A1158EB11C9E93D6AD2A101BB9FA204; pfmhelp=1; _ga=GA1.2.1694534677.1489779146; TrackJS=f2465051-cb66-4cfa-8a47-5c7a5d07aab4; navigation=520533110%7C4130 > > and I'm seeing: > BerespHeader Cache-Control: no-cache, no-store, must-revalidate > BerespHeader Pragma: no-cache The backend is telling Varnish, and ultimately the client, that the response cannot be cached. Depending on your version of Varnish it will indeed result in subsequent passed transaction. > Also, was there a part 2 to that blog? Part 2 is written, but still an early draft so not published yet. Cheers, Dridi -- James Louis Lead Systems Engineer [Image removed by sender.] 4400 Baker Road, Minnetonka, MN 55343
CELL 612.203.2631 TOLL FREE 888-346-3486 x 622 | FAX 952-908-6129 http://www.peoplenetonline.com PeopleNet is the leading provider of fleet mobility systems to the transportation industry, including truckload, LTL, private, and energy service fleets. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ~WRD000.jpg Type: image/jpeg Size: 823 bytes Desc: ~WRD000.jpg URL: From dridi at varni.sh Fri Mar 24 09:18:19 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 24 Mar 2017 10:18:19 +0100 Subject: varnish caching with jsessionid being set In-Reply-To: References: Message-ID: On Wed, Mar 22, 2017 at 4:28 PM, Guillaume Quintard wrote: > > Sure, you can override them: > > set vcl_backend_response { > set beresp.ttl = 5m; > } Hello Jim, This kind of "yes you can" should always come with a mandatory "but you shouldn't" :) C allows you to easily shoot yourself in the foot, C++ too but it will blow off your whole leg. Those are well known facts. It is the same for VCL: it allows you to shoot yourself in the foot, blow off your leg, and leak sensitive information. That is true with any caching solution that allows you to overrule the origin server. And that applies to any origin server that doesn't do proper caching. If you're backend isn't good at conveying caching intent, fix the backend. Otherwise how can you know when you (and I quote) "ignore the caching headers" that you aren't caching something private? Know what thou art doing and proceed with care. Dridi From rbizzell at measinc.com Fri Mar 24 12:58:07 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Fri, 24 Mar 2017 12:58:07 +0000 Subject: varnish caching with jsessionid being set In-Reply-To: References: <97ecaf5a9ba1431d9c7e4ffe7a1486b8@mbx2serv.meas-inc.com> Message-ID: <5a79903608c045f696868bf4a9fef90f@mbx2serv.meas-inc.com> I want to make sure if my backend goes down that the content will still be served until the backend comes back up -----Original Message----- From: Dridi Boukelmoune [mailto:dridi at varni.sh] Sent: Friday, March 24, 2017 8:41 AM To: Rodney Bizzell Subject: Re: varnish caching with jsessionid being set On Fri, Mar 24, 2017 at 1:35 PM, Rodney Bizzell wrote: > Is it okay to set the set beresp.grace after the set beresp.ttl = 5m; It is, but it mainly depends on what you are trying to achieve. Don't forget to CC the mailing list next time. This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | From devin at pabstatencio.com Sun Mar 26 04:15:02 2017 From: devin at pabstatencio.com (Devin Acosta) Date: Sat, 25 Mar 2017 21:15:02 -0700 Subject: Method to block IPs in DB with Varnish? Message-ID: I have been testing out Varnish and simply have been impressed with what I have seen so far. The only issue I am trying to figure out so that I can implement it in my environment is currently we use OSSEC and have it blocking traffic for periods of time if it notices questionable behavior. What would be the best/easiest way for me to have OSSEC say update some type of database (memcache/redis/mongo) and add IPs to the list for a period of time and if the remote IP matches the IP address block the traffic for a period of time in Varnish? I see it has quite a powerful VCL language but not quite sure what my code would look like to accomplish this task? Anyone help/suggestions on this topic? -- Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin at linuxguru.co -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Mon Mar 27 13:44:49 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Mon, 27 Mar 2017 13:44:49 +0000 Subject: conflict with website names Message-ID: <4e69c640ae2e40ab8d7840c7a0e31a03@mbx2serv.meas-inc.com> Hello, Is there a way to differentiate websites that have the same domain name but different sub-domain www/support. In the backend name you can't use "." character This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Mon Mar 27 14:32:31 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Mon, 27 Mar 2017 14:32:31 +0000 Subject: Domain Message-ID: <11a161139d16497099d0d7347981a52a@mbx2serv.meas-inc.com> Let me rephrase the question. I have a domain name www.peg.com and then there is a support website called support.peg.com. To create the backend server it doesn't allow for the period is there a way to allow for this in the backend server name This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | -------------- next part -------------- An HTML attachment was scrubbed... URL: From mblissett at gbif.org Mon Mar 27 17:05:26 2017 From: mblissett at gbif.org (Matthew Blissett) Date: Mon, 27 Mar 2017 19:05:26 +0200 Subject: Domain In-Reply-To: <11a161139d16497099d0d7347981a52a@mbx2serv.meas-inc.com> References: <11a161139d16497099d0d7347981a52a@mbx2serv.meas-inc.com> Message-ID: <0dff611c-a501-31d5-93b8-e5e81280984a@gbif.org> See https://www.varnish-cache.org/docs/5.1/users-guide/vcl-backends.html#multiple-backends Your backends might look like backend www{ .host = "192.0.2.0"; .port = "80"; } backend support{ .host = "192.0.2.1"; .port = "80"; } The names are just labels. To use backends like this, see https://www.varnish-cache.org/docs/5.1/users-guide/vcl-backends.html#backends-and-virtual-hosts-in-varnish Matt On 27/03/17 16:32, Rodney Bizzell wrote: > > Let me rephrase the question. I have a domain name www.peg.com > and then there is a support website called > support.peg.com. To create the backend server it doesn?t allow for the > period is there a way to allow for this in the backend server name > > > > This email (including any attachments) may contain confidential > information intended solely for acknowledged recipients. If you think > you have received this information in error, please reply to the > sender and delete all copies from your system. Please note that > unauthorized use, disclosure, or further distribution of this > information is prohibited by the sender. Note also that we may monitor > email directed to or originating from our network. Thank you for your > consideration and assistance. | > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Mon Mar 27 17:12:18 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Mon, 27 Mar 2017 17:12:18 +0000 Subject: backend config Message-ID: <037030a730f04275b511e96e3099c322@mbx2serv.meas-inc.com> Hello, I would like to know if this config will work with the naming of my different backend servers I have several servers that have www. and support domain names. For example. Will this config work. Thanks! backend wpponline { .host = "www.wpponline.com"; .port = "80"; .connect_timeout = 6000s; .first_byte_timeout = 6000s; .between_bytes_timeout = 6000s; } backend support { .host = "support.wpponline.com"; .port = "80"; .connect_timeout = 6000s; .first_byte_timeout = 6000s; .between_bytes_timeout = 6000s; } } else if (req.http.host == "www.wpponline.com"){ set req.backend_hint = wpponline; } else if (req.http.host == "support.wpponline.com"){ set req.backend_hint = support.wpponline; This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Mon Mar 27 17:30:16 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Mon, 27 Mar 2017 17:30:16 +0000 Subject: Domain In-Reply-To: <0dff611c-a501-31d5-93b8-e5e81280984a@gbif.org> References: <11a161139d16497099d0d7347981a52a@mbx2serv.meas-inc.com> <0dff611c-a501-31d5-93b8-e5e81280984a@gbif.org> Message-ID: <30e0fbefc37645e5922743277f2d6f9e@mbx2serv.meas-inc.com> Thanks! From: varnish-misc-bounces+rbizzell=measinc.com at varnish-cache.org [mailto:varnish-misc-bounces+rbizzell=measinc.com at varnish-cache.org] On Behalf Of Matthew Blissett Sent: Monday, March 27, 2017 1:05 PM To: varnish-misc at varnish-cache.org Subject: Re: Domain See https://www.varnish-cache.org/docs/5.1/users-guide/vcl-backends.html#multiple-backends Your backends might look like backend www { .host = "192.0.2.0"; .port = "80"; } backend support { .host = "192.0.2.1"; .port = "80"; } The names are just labels. To use backends like this, see https://www.varnish-cache.org/docs/5.1/users-guide/vcl-backends.html#backends-and-virtual-hosts-in-varnish Matt On 27/03/17 16:32, Rodney Bizzell wrote: Let me rephrase the question. I have a domain name www.peg.com and then there is a support website called support.peg.com. To create the backend server it doesn't allow for the period is there a way to allow for this in the backend server name This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Mon Mar 27 17:40:16 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Mon, 27 Mar 2017 17:40:16 +0000 Subject: Domain In-Reply-To: <0dff611c-a501-31d5-93b8-e5e81280984a@gbif.org> References: <11a161139d16497099d0d7347981a52a@mbx2serv.meas-inc.com> <0dff611c-a501-31d5-93b8-e5e81280984a@gbif.org> Message-ID: <0f596c6aad6b4376b4d66094a349e0ee@mbx2serv.meas-inc.com> Also can I use regex to identify support.wpp.com. I have several support backend servers that correlate to specific websites so say that I just wanted to use the name instead of the IP? From: varnish-misc-bounces+rbizzell=measinc.com at varnish-cache.org [mailto:varnish-misc-bounces+rbizzell=measinc.com at varnish-cache.org] On Behalf Of Matthew Blissett Sent: Monday, March 27, 2017 1:05 PM To: varnish-misc at varnish-cache.org Subject: Re: Domain See https://www.varnish-cache.org/docs/5.1/users-guide/vcl-backends.html#multiple-backends Your backends might look like backend www { .host = "192.0.2.0"; .port = "80"; } backend support { .host = "192.0.2.1"; .port = "80"; } The names are just labels. To use backends like this, see https://www.varnish-cache.org/docs/5.1/users-guide/vcl-backends.html#backends-and-virtual-hosts-in-varnish Matt On 27/03/17 16:32, Rodney Bizzell wrote: Let me rephrase the question. I have a domain name www.peg.com and then there is a support website called support.peg.com. To create the backend server it doesn't allow for the period is there a way to allow for this in the backend server name This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Mon Mar 27 17:43:23 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Mon, 27 Mar 2017 17:43:23 +0000 Subject: Domain References: <11a161139d16497099d0d7347981a52a@mbx2serv.meas-inc.com> <0dff611c-a501-31d5-93b8-e5e81280984a@gbif.org> Message-ID: So have something like this set req.backend_hint = wpponline; } else if (req.http.host == "<^(?=.*?\bsupport\b)(?=.*?\bwpponline\b)(?=.*?\bcom\b).*$>"){ set req.backend_hint = support; I have a totally different backend that support.peg.com and I also have www.peg.com and then I have a support.ncwrite.com and then I also have www.ncwrite.com From: Rodney Bizzell Sent: Monday, March 27, 2017 1:42 PM To: 'Matthew Blissett' ; varnish-misc at varnish-cache.org Subject: RE: Domain Also can I use regex to identify support.wpp.com. I have several support backend servers that correlate to specific websites so say that I just wanted to use the name instead of the IP? From: varnish-misc-bounces+rbizzell=measinc.com at varnish-cache.org [mailto:varnish-misc-bounces+rbizzell=measinc.com at varnish-cache.org] On Behalf Of Matthew Blissett Sent: Monday, March 27, 2017 1:05 PM To: varnish-misc at varnish-cache.org Subject: Re: Domain See https://www.varnish-cache.org/docs/5.1/users-guide/vcl-backends.html#multiple-backends Your backends might look like backend www { .host = "192.0.2.0"; .port = "80"; } backend support { .host = "192.0.2.1"; .port = "80"; } The names are just labels. To use backends like this, see https://www.varnish-cache.org/docs/5.1/users-guide/vcl-backends.html#backends-and-virtual-hosts-in-varnish Matt On 27/03/17 16:32, Rodney Bizzell wrote: Let me rephrase the question. I have a domain name www.peg.com and then there is a support website called support.peg.com. To create the backend server it doesn't allow for the period is there a way to allow for this in the backend server name This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbizzell at measinc.com Mon Mar 27 18:35:18 2017 From: rbizzell at measinc.com (Rodney Bizzell) Date: Mon, 27 Mar 2017 18:35:18 +0000 Subject: Domain In-Reply-To: <0dff611c-a501-31d5-93b8-e5e81280984a@gbif.org> References: <11a161139d16497099d0d7347981a52a@mbx2serv.meas-inc.com> <0dff611c-a501-31d5-93b8-e5e81280984a@gbif.org> Message-ID: I just named the backends support 1 2 3 etc From: varnish-misc-bounces+rbizzell=measinc.com at varnish-cache.org [mailto:varnish-misc-bounces+rbizzell=measinc.com at varnish-cache.org] On Behalf Of Matthew Blissett Sent: Monday, March 27, 2017 1:05 PM To: varnish-misc at varnish-cache.org Subject: Re: Domain See https://www.varnish-cache.org/docs/5.1/users-guide/vcl-backends.html#multiple-backends Your backends might look like backend www { .host = "192.0.2.0"; .port = "80"; } backend support { .host = "192.0.2.1"; .port = "80"; } The names are just labels. To use backends like this, see https://www.varnish-cache.org/docs/5.1/users-guide/vcl-backends.html#backends-and-virtual-hosts-in-varnish Matt On 27/03/17 16:32, Rodney Bizzell wrote: Let me rephrase the question. I have a domain name www.peg.com and then there is a support website called support.peg.com. To create the backend server it doesn't allow for the period is there a way to allow for this in the backend server name This email (including any attachments) may contain confidential information intended solely for acknowledged recipients. If you think you have received this information in error, please reply to the sender and delete all copies from your system. Please note that unauthorized use, disclosure, or further distribution of this information is prohibited by the sender. Note also that we may monitor email directed to or originating from our network. Thank you for your consideration and assistance. | _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Tue Mar 28 07:01:35 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 28 Mar 2017 09:01:35 +0200 Subject: conflict with website names In-Reply-To: <4e69c640ae2e40ab8d7840c7a0e31a03@mbx2serv.meas-inc.com> References: <4e69c640ae2e40ab8d7840c7a0e31a03@mbx2serv.meas-inc.com> Message-ID: Marked as duplicate: https://www.varnish-cache.org/lists/pipermail/varnish-misc/2017-March/025758.html -- Guillaume Quintard On Mon, Mar 27, 2017 at 3:44 PM, Rodney Bizzell wrote: > Hello, > > Is there a way to differentiate websites that have the same domain name > but different sub-domain www/support. In the backend name you can?t use ?.? > character > > > This email (including any attachments) may contain confidential > information intended solely for acknowledged recipients. If you think you > have received this information in error, please reply to the sender and > delete all copies from your system. Please note that unauthorized use, > disclosure, or further distribution of this information is prohibited by > the sender. Note also that we may monitor email directed to or originating > from our network. Thank you for your consideration and assistance. | > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at hanfordonline.co.uk Tue Mar 28 09:52:09 2017 From: mark at hanfordonline.co.uk (Mark Hanford) Date: Tue, 28 Mar 2017 10:52:09 +0100 Subject: Upgrading from v3 to v5 - XFF and client.identity Message-ID: Hi folks. I'm in the process of trying to upgrade our current v3 config to v4+ for upgrading to Varnish 5. In my old v3 config, I use the XFF header to change the client.identity if it is present. # For session-persistence, set a client identity. # We're using the client's IP for this to ensure server-persistence. if (req.http.X-Forwarded-For) { set client.identity = req.http.X-Forwarded-For; } else { set client.identity = client.ip; } It is important that a request from one remote client goes to the same backend every time, when possible. Do I still need to deal with this, or will this be handled transparently in some way? I suspect I'll have more questions as I go along, so I apologise in advance for the spam :) thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Tue Mar 28 11:28:14 2017 From: lagged at gmail.com (Andrei) Date: Tue, 28 Mar 2017 06:28:14 -0500 Subject: Upgrading from v3 to v5 - XFF and client.identity In-Reply-To: References: Message-ID: Hi Mark, I suggest going over the following blog post for the changes you're looking for. Good luck moving forward :D https://info.varnish-software.com/blog/proper-sticky-session-load-balancing-varnish On Tue, Mar 28, 2017 at 4:52 AM, Mark Hanford wrote: > Hi folks. I'm in the process of trying to upgrade our current v3 config to > v4+ for upgrading to Varnish 5. > > In my old v3 config, I use the XFF header to change the client.identity if > it is present. > > # For session-persistence, set a client identity. > # We're using the client's IP for this to ensure server-persistence. > if (req.http.X-Forwarded-For) { > set client.identity = req.http.X-Forwarded-For; > } else { > set client.identity = client.ip; > } > > It is important that a request from one remote client goes to the same > backend every time, when possible. Do I still need to deal with this, or > will this be handled transparently in some way? > > I suspect I'll have more questions as I go along, so I apologise in > advance for the spam :) > > thanks, > > Mark > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Tue Mar 28 13:11:36 2017 From: lagged at gmail.com (Andrei) Date: Tue, 28 Mar 2017 08:11:36 -0500 Subject: Upgrading from v3 to v5 - XFF and client.identity In-Reply-To: References: Message-ID: Oh yeah, Guillaume also has a great post on it @ https://info.varnish-software.com/blog/sticky-session-with-cookies :D On Tue, Mar 28, 2017 at 6:28 AM, Andrei wrote: > Hi Mark, > > I suggest going over the following blog post for the changes you're > looking for. Good luck moving forward :D > > https://info.varnish-software.com/blog/proper-sticky- > session-load-balancing-varnish > > > On Tue, Mar 28, 2017 at 4:52 AM, Mark Hanford > wrote: > >> Hi folks. I'm in the process of trying to upgrade our current v3 config >> to v4+ for upgrading to Varnish 5. >> >> In my old v3 config, I use the XFF header to change the client.identity >> if it is present. >> >> # For session-persistence, set a client identity. >> # We're using the client's IP for this to ensure server-persistence. >> if (req.http.X-Forwarded-For) { >> set client.identity = req.http.X-Forwarded-For; >> } else { >> set client.identity = client.ip; >> } >> >> It is important that a request from one remote client goes to the same >> backend every time, when possible. Do I still need to deal with this, or >> will this be handled transparently in some way? >> >> I suspect I'll have more questions as I go along, so I apologise in >> advance for the spam :) >> >> thanks, >> >> Mark >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Tue Mar 28 21:00:36 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 28 Mar 2017 23:00:36 +0200 Subject: Upgrading from v3 to v5 - XFF and client.identity In-Reply-To: References: Message-ID: Thanks Andrei! Mark, one simple solution would be to simply use the hash director, and give it the client.ip as string. This way, the same ip will always go to the same backend. -- Guillaume Quintard On Tue, Mar 28, 2017 at 3:11 PM, Andrei wrote: > Oh yeah, Guillaume also has a great post on it @ https://info.varnish- > software.com/blog/sticky-session-with-cookies :D > > On Tue, Mar 28, 2017 at 6:28 AM, Andrei wrote: > >> Hi Mark, >> >> I suggest going over the following blog post for the changes you're >> looking for. Good luck moving forward :D >> >> https://info.varnish-software.com/blog/proper-sticky-session >> -load-balancing-varnish >> >> >> On Tue, Mar 28, 2017 at 4:52 AM, Mark Hanford >> wrote: >> >>> Hi folks. I'm in the process of trying to upgrade our current v3 config >>> to v4+ for upgrading to Varnish 5. >>> >>> In my old v3 config, I use the XFF header to change the client.identity >>> if it is present. >>> >>> # For session-persistence, set a client identity. >>> # We're using the client's IP for this to ensure server-persistence. >>> if (req.http.X-Forwarded-For) { >>> set client.identity = req.http.X-Forwarded-For; >>> } else { >>> set client.identity = client.ip; >>> } >>> >>> It is important that a request from one remote client goes to the same >>> backend every time, when possible. Do I still need to deal with this, or >>> will this be handled transparently in some way? >>> >>> I suspect I'll have more questions as I go along, so I apologise in >>> advance for the spam :) >>> >>> thanks, >>> >>> Mark >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at hanfordonline.co.uk Tue Mar 28 22:01:08 2017 From: mark at hanfordonline.co.uk (Mark Hanford) Date: Tue, 28 Mar 2017 23:01:08 +0100 Subject: Upgrading from v3 to v5 - XFF and client.identity In-Reply-To: References: Message-ID: (Resending because I forgot to reply to the list earlier...) But what's wrong with using the client IP? These solutions using cookies are a little dirty, regardless of the blog title. This bit for example, is completely unworkable: if (req.http.server == "s1") { set req.backend_hint = s1; } else if (req.http.server == "s2") { set req.backend_hint = s2; } else { if (std.rand(0, 100) < 50) { req.backend_hint = s1; } else { req.backend_hint = s2; } } Because we have at least 15 backends configured in 5 different Director pools, I'd end up having to build an enormous set of conditionals to work out which backend to use. Why is _this_ the preferred solution over determining the client IP and selecting a backend based on that? thanks, Mark On 28 Mar 2017 2:11 pm, "Andrei" wrote: > Oh yeah, Guillaume also has a great post on it @ https://info.varnish- > software.com/blog/sticky-session-with-cookies :D > > On Tue, Mar 28, 2017 at 6:28 AM, Andrei wrote: > >> Hi Mark, >> >> I suggest going over the following blog post for the changes you're >> looking for. Good luck moving forward :D >> >> https://info.varnish-software.com/blog/proper-sticky-session >> -load-balancing-varnish >> >> >> On Tue, Mar 28, 2017 at 4:52 AM, Mark Hanford >> wrote: >> >>> Hi folks. I'm in the process of trying to upgrade our current v3 config >>> to v4+ for upgrading to Varnish 5. >>> >>> In my old v3 config, I use the XFF header to change the client.identity >>> if it is present. >>> >>> # For session-persistence, set a client identity. >>> # We're using the client's IP for this to ensure server-persistence. >>> if (req.http.X-Forwarded-For) { >>> set client.identity = req.http.X-Forwarded-For; >>> } else { >>> set client.identity = client.ip; >>> } >>> >>> It is important that a request from one remote client goes to the same >>> backend every time, when possible. Do I still need to deal with this, or >>> will this be handled transparently in some way? >>> >>> I suspect I'll have more questions as I go along, so I apologise in >>> advance for the spam :) >>> >>> thanks, >>> >>> Mark >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From devin at pabstatencio.com Wed Mar 29 03:43:20 2017 From: devin at pabstatencio.com (Devin Acosta) Date: Tue, 28 Mar 2017 20:43:20 -0700 Subject: Varnish HIT/MISS Web Statistics Question Message-ID: I am trying to get to where I can launch Varnish Cache in my environment. One of the challenges I guess that I am trying to figure out is that currently if a request is a HIT it never logs to the backend server the requests that it processed, therefore it messes up my Web Statistics. I see that I can use "varnishncsa" which will cause it log onto a file on the local machine that Varnish is running on, however is there a cleaner way to get my web statistics so that it's accurate, other than trying to pull logs from both the backend server and the varnish server and combine them together? -- Devin Acosta -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Wed Mar 29 05:00:00 2017 From: lagged at gmail.com (Andrei) Date: Wed, 29 Mar 2017 00:00:00 -0500 Subject: Varnish HIT/MISS Web Statistics Question In-Reply-To: References: Message-ID: Hi Devin, The easiest method would be to use external analytics services for your site(s), such as Google Analytics. However, if you do not wish to use external services then I suggest using something like splitlogs, and having both Apache and varnishncsa cache hits piped to it, which in return will output all requests to your access logs as expected. If you're in a cPanel environment, I wrote a script that runs as a daemon, and that does just that - https://github.com/AndreiG6/vscp On Tue, Mar 28, 2017 at 10:43 PM, Devin Acosta wrote: > > I am trying to get to where I can launch Varnish Cache in my environment. > One of the challenges I guess that I am trying to figure out is that > currently if a request is a HIT it never logs to the backend server the > requests that it processed, therefore it messes up my Web Statistics. I see > that I can use "varnishncsa" which will cause it log onto a file on the > local machine that Varnish is running on, however is there a cleaner way to > get my web statistics so that it's accurate, other than trying to pull logs > from both the backend server and the varnish server and combine them > together? > > -- > > Devin Acosta > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hazarguney at gmail.com Wed Mar 29 05:45:57 2017 From: hazarguney at gmail.com (=?UTF-8?B?SGF6YXIgR8O8bmV5?=) Date: Wed, 29 Mar 2017 08:45:57 +0300 Subject: =?UTF-8?Q?Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= Message-ID: I see sporadic fetch errors (http first read error: EOF) from backend but backend is normally healthy. It seems that timeout is not the root issue in this case. What could be the reason? * << BeReq >> 98808229 - Begin bereq 98808228 fetch - Timestamp Start: 1490683823.763272 0.000000 0.000000 - BereqMethod GET - BereqURL XXXX - BereqProtocol HTTP/1.1 - BereqHeader Pragma: no-cache - BereqHeader Accept: */* - BereqHeader From: bingbot(at)microsoft.com - BereqHeader Host: XXXX - BereqHeader User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 98808229 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 38 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 172.17.0.2 48388 - BackendStart 10.35.78.11 80 - Timestamp Bereq: 1490683823.763758 0.000487 0.000487 - *FetchError* *http first read error: EOF* - BackendClose 38 reload_2017-03-20T11:32:44.st2 - Timestamp Beresp: 1490683823.764271 0.000999 0.000513 - Timestamp Error: 1490683823.764277 0.001005 0.000005 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Service Unavailable - BerespReason Backend fetch failed - BerespHeader Date: Tue, 28 Mar 2017 06:50:23 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR - BereqHeader X-Varnish-Backend-5xx: 1 - VCL_return retry - Timestamp Retry: 1490683823.764294 0.001022 0.000017 - Link bereq 97940444 retry - End -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Mar 29 07:15:49 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 29 Mar 2017 09:15:49 +0200 Subject: Upgrading from v3 to v5 - XFF and client.identity In-Reply-To: References: Message-ID: Mark, to avoid conditionnal, you can avoid vmod-stendahl: https://github.com/gquintard/libvmod-stendhal, the std.rand() part can just use the random director. But, as I said, for this one, I'd go with the hash director and just use the client.ip, since you don't care about the actual session. -- Guillaume Quintard On Wed, Mar 29, 2017 at 12:01 AM, Mark Hanford wrote: > (Resending because I forgot to reply to the list earlier...) > > But what's wrong with using the client IP? These solutions using cookies > are a little dirty, regardless of the blog title. > > This bit for example, is completely unworkable: > > if (req.http.server == "s1") { > set req.backend_hint = s1; > } else if (req.http.server == "s2") { > set req.backend_hint = s2; > } else { > if (std.rand(0, 100) < 50) { > req.backend_hint = s1; > } else { > req.backend_hint = s2; > } > } > > Because we have at least 15 backends configured in 5 different Director > pools, I'd end up having to build an enormous set of conditionals to work > out which backend to use. > > Why is _this_ the preferred solution over determining the client IP and > selecting a backend based on that? > > thanks, > > Mark > > > On 28 Mar 2017 2:11 pm, "Andrei" wrote: > >> Oh yeah, Guillaume also has a great post on it @ >> https://info.varnish-software.com/blog/sticky-session-with-cookies :D >> >> On Tue, Mar 28, 2017 at 6:28 AM, Andrei wrote: >> >>> Hi Mark, >>> >>> I suggest going over the following blog post for the changes you're >>> looking for. Good luck moving forward :D >>> >>> https://info.varnish-software.com/blog/proper-sticky-session >>> -load-balancing-varnish >>> >>> >>> On Tue, Mar 28, 2017 at 4:52 AM, Mark Hanford >>> wrote: >>> >>>> Hi folks. I'm in the process of trying to upgrade our current v3 config >>>> to v4+ for upgrading to Varnish 5. >>>> >>>> In my old v3 config, I use the XFF header to change the client.identity >>>> if it is present. >>>> >>>> # For session-persistence, set a client identity. >>>> # We're using the client's IP for this to ensure server-persistence. >>>> if (req.http.X-Forwarded-For) { >>>> set client.identity = req.http.X-Forwarded-For; >>>> } else { >>>> set client.identity = client.ip; >>>> } >>>> >>>> It is important that a request from one remote client goes to the same >>>> backend every time, when possible. Do I still need to deal with this, or >>>> will this be handled transparently in some way? >>>> >>>> I suspect I'll have more questions as I go along, so I apologise in >>>> advance for the spam :) >>>> >>>> thanks, >>>> >>>> Mark >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> >>> >> > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Mar 29 07:32:24 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 29 Mar 2017 09:32:24 +0200 Subject: Varnish HIT/MISS Web Statistics Question In-Reply-To: References: Message-ID: Hello Devin, varnishncsa is able to return all the client requests it received (-c) as well as all the backend requests it sent (-b), combining those to two would give you the complete picture. -- Guillaume Quintard On Wed, Mar 29, 2017 at 7:00 AM, Andrei wrote: > Hi Devin, > > The easiest method would be to use external analytics services for your > site(s), such as Google Analytics. However, if you do not wish to use > external services then I suggest using something like splitlogs, and having > both Apache and varnishncsa cache hits piped to it, which in return will > output all requests to your access logs as expected. If you're in a cPanel > environment, I wrote a script that runs as a daemon, and that does just > that - https://github.com/AndreiG6/vscp > > On Tue, Mar 28, 2017 at 10:43 PM, Devin Acosta > wrote: > >> >> I am trying to get to where I can launch Varnish Cache in my environment. >> One of the challenges I guess that I am trying to figure out is that >> currently if a request is a HIT it never logs to the backend server the >> requests that it processed, therefore it messes up my Web Statistics. I see >> that I can use "varnishncsa" which will cause it log onto a file on the >> local machine that Varnish is running on, however is there a cleaner way to >> get my web statistics so that it's accurate, other than trying to pull logs >> from both the backend server and the varnish server and combine them >> together? >> >> -- >> >> Devin Acosta >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Mar 29 07:34:22 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 29 Mar 2017 09:34:22 +0200 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: References: Message-ID: Indeed, looking at the Timestamp lines, it's not a timeout. What's your backend? -- Guillaume Quintard On Wed, Mar 29, 2017 at 7:45 AM, Hazar G?ney wrote: > I see sporadic fetch errors (http first read error: EOF) from backend but > backend is normally healthy. It seems that timeout is not the root issue in > this case. What could be the reason? > > * << BeReq >> 98808229 > - Begin bereq 98808228 fetch > - Timestamp Start: 1490683823.763272 0.000000 0.000000 > - BereqMethod GET > - BereqURL XXXX > - BereqProtocol HTTP/1.1 > - BereqHeader Pragma: no-cache > - BereqHeader Accept: */* > - BereqHeader From: bingbot(at)microsoft.com > - BereqHeader Host: XXXX > - BereqHeader User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) > - BereqHeader Accept-Encoding: gzip > - BereqHeader X-Varnish: 98808229 > - VCL_call BACKEND_FETCH > - VCL_return fetch > - BackendOpen 38 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 172.17.0.2 48388 > - BackendStart 10.35.78.11 80 > - Timestamp Bereq: 1490683823.763758 0.000487 0.000487 > - *FetchError* *http first read error: EOF* > - BackendClose 38 reload_2017-03-20T11:32:44.st2 > - Timestamp Beresp: 1490683823.764271 0.000999 0.000513 > - Timestamp Error: 1490683823.764277 0.001005 0.000005 > - BerespProtocol HTTP/1.1 > - BerespStatus 503 > - BerespReason Service Unavailable > - BerespReason Backend fetch failed > - BerespHeader Date: Tue, 28 Mar 2017 06:50:23 GMT > - BerespHeader Server: Varnish > - VCL_call BACKEND_ERROR > - BereqHeader X-Varnish-Backend-5xx: 1 > - VCL_return retry > - Timestamp Retry: 1490683823.764294 0.001022 0.000017 > - Link bereq 97940444 retry > - End > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hazarguney at gmail.com Wed Mar 29 08:12:52 2017 From: hazarguney at gmail.com (=?UTF-8?B?SGF6YXIgR8O8bmV5?=) Date: Wed, 29 Mar 2017 11:12:52 +0300 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: References: Message-ID: Backend is Apache. On Wed, Mar 29, 2017 at 10:34 AM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > Indeed, looking at the Timestamp lines, it's not a timeout. What's your > backend? > > -- > Guillaume Quintard > > On Wed, Mar 29, 2017 at 7:45 AM, Hazar G?ney wrote: > >> I see sporadic fetch errors (http first read error: EOF) from backend >> but backend is normally healthy. It seems that timeout is not the root >> issue in this case. What could be the reason? >> >> * << BeReq >> 98808229 >> - Begin bereq 98808228 fetch >> - Timestamp Start: 1490683823.763272 0.000000 0.000000 >> - BereqMethod GET >> - BereqURL XXXX >> - BereqProtocol HTTP/1.1 >> - BereqHeader Pragma: no-cache >> - BereqHeader Accept: */* >> - BereqHeader From: bingbot(at)microsoft.com >> - BereqHeader Host: XXXX >> - BereqHeader User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) >> - BereqHeader Accept-Encoding: gzip >> - BereqHeader X-Varnish: 98808229 >> - VCL_call BACKEND_FETCH >> - VCL_return fetch >> - BackendOpen 38 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 172.17.0.2 48388 >> - BackendStart 10.35.78.11 80 >> - Timestamp Bereq: 1490683823.763758 0.000487 0.000487 >> - *FetchError* *http first read error: EOF* >> - BackendClose 38 reload_2017-03-20T11:32:44.st2 >> - Timestamp Beresp: 1490683823.764271 0.000999 0.000513 >> - Timestamp Error: 1490683823.764277 0.001005 0.000005 >> - BerespProtocol HTTP/1.1 >> - BerespStatus 503 >> - BerespReason Service Unavailable >> - BerespReason Backend fetch failed >> - BerespHeader Date: Tue, 28 Mar 2017 06:50:23 GMT >> - BerespHeader Server: Varnish >> - VCL_call BACKEND_ERROR >> - BereqHeader X-Varnish-Backend-5xx: 1 >> - VCL_return retry >> - Timestamp Retry: 1490683823.764294 0.001022 0.000017 >> - Link bereq 97940444 retry >> - End >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Mar 29 08:18:19 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 29 Mar 2017 10:18:19 +0200 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: References: Message-ID: I'm suspecting Apache's closing a keep-alive connection because it and Varnish didn't agree on the timeout. Can you check? (in varnish, the param is called backend_idle_timeout) -- Guillaume Quintard On Wed, Mar 29, 2017 at 10:12 AM, Hazar G?ney wrote: > Backend is Apache. > > On Wed, Mar 29, 2017 at 10:34 AM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> Indeed, looking at the Timestamp lines, it's not a timeout. What's your >> backend? >> >> -- >> Guillaume Quintard >> >> On Wed, Mar 29, 2017 at 7:45 AM, Hazar G?ney >> wrote: >> >>> I see sporadic fetch errors (http first read error: EOF) from backend >>> but backend is normally healthy. It seems that timeout is not the root >>> issue in this case. What could be the reason? >>> >>> * << BeReq >> 98808229 >>> - Begin bereq 98808228 fetch >>> - Timestamp Start: 1490683823.763272 0.000000 0.000000 >>> - BereqMethod GET >>> - BereqURL XXXX >>> - BereqProtocol HTTP/1.1 >>> - BereqHeader Pragma: no-cache >>> - BereqHeader Accept: */* >>> - BereqHeader From: bingbot(at)microsoft.com >>> - BereqHeader Host: XXXX >>> - BereqHeader User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) >>> - BereqHeader Accept-Encoding: gzip >>> - BereqHeader X-Varnish: 98808229 >>> - VCL_call BACKEND_FETCH >>> - VCL_return fetch >>> - BackendOpen 38 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 172.17.0.2 48388 >>> - BackendStart 10.35.78.11 80 >>> - Timestamp Bereq: 1490683823.763758 0.000487 0.000487 >>> - *FetchError* *http first read error: EOF* >>> - BackendClose 38 reload_2017-03-20T11:32:44.st2 >>> - Timestamp Beresp: 1490683823.764271 0.000999 0.000513 >>> - Timestamp Error: 1490683823.764277 0.001005 0.000005 >>> - BerespProtocol HTTP/1.1 >>> - BerespStatus 503 >>> - BerespReason Service Unavailable >>> - BerespReason Backend fetch failed >>> - BerespHeader Date: Tue, 28 Mar 2017 06:50:23 GMT >>> - BerespHeader Server: Varnish >>> - VCL_call BACKEND_ERROR >>> - BereqHeader X-Varnish-Backend-5xx: 1 >>> - VCL_return retry >>> - Timestamp Retry: 1490683823.764294 0.001022 0.000017 >>> - Link bereq 97940444 retry >>> - End >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hazarguney at gmail.com Wed Mar 29 08:48:46 2017 From: hazarguney at gmail.com (=?UTF-8?B?SGF6YXIgR8O8bmV5?=) Date: Wed, 29 Mar 2017 11:48:46 +0300 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: References: Message-ID: It is default value which is 60: param.show backend_idle_timeout 200 backend_idle_timeout Value is: 60.000 [seconds] (default) Minimum is: 1.000 Timeout before we close unused backend connections. On Wed, Mar 29, 2017 at 11:18 AM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > I'm suspecting Apache's closing a keep-alive connection because it and > Varnish didn't agree on the timeout. Can you check? (in varnish, the param > is called backend_idle_timeout) > > -- > Guillaume Quintard > > On Wed, Mar 29, 2017 at 10:12 AM, Hazar G?ney > wrote: > >> Backend is Apache. >> >> On Wed, Mar 29, 2017 at 10:34 AM, Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> Indeed, looking at the Timestamp lines, it's not a timeout. What's your >>> backend? >>> >>> -- >>> Guillaume Quintard >>> >>> On Wed, Mar 29, 2017 at 7:45 AM, Hazar G?ney >>> wrote: >>> >>>> I see sporadic fetch errors (http first read error: EOF) from backend >>>> but backend is normally healthy. It seems that timeout is not the root >>>> issue in this case. What could be the reason? >>>> >>>> * << BeReq >> 98808229 >>>> - Begin bereq 98808228 fetch >>>> - Timestamp Start: 1490683823.763272 0.000000 0.000000 >>>> - BereqMethod GET >>>> - BereqURL XXXX >>>> - BereqProtocol HTTP/1.1 >>>> - BereqHeader Pragma: no-cache >>>> - BereqHeader Accept: */* >>>> - BereqHeader From: bingbot(at)microsoft.com >>>> - BereqHeader Host: XXXX >>>> - BereqHeader User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) >>>> - BereqHeader Accept-Encoding: gzip >>>> - BereqHeader X-Varnish: 98808229 >>>> - VCL_call BACKEND_FETCH >>>> - VCL_return fetch >>>> - BackendOpen 38 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 172.17.0.2 48388 >>>> - BackendStart 10.35.78.11 80 >>>> - Timestamp Bereq: 1490683823.763758 0.000487 0.000487 >>>> - *FetchError* *http first read error: EOF* >>>> - BackendClose 38 reload_2017-03-20T11:32:44.st2 >>>> - Timestamp Beresp: 1490683823.764271 0.000999 0.000513 >>>> - Timestamp Error: 1490683823.764277 0.001005 0.000005 >>>> - BerespProtocol HTTP/1.1 >>>> - BerespStatus 503 >>>> - BerespReason Service Unavailable >>>> - BerespReason Backend fetch failed >>>> - BerespHeader Date: Tue, 28 Mar 2017 06:50:23 GMT >>>> - BerespHeader Server: Varnish >>>> - VCL_call BACKEND_ERROR >>>> - BereqHeader X-Varnish-Backend-5xx: 1 >>>> - VCL_return retry >>>> - Timestamp Retry: 1490683823.764294 0.001022 0.000017 >>>> - Link bereq 97940444 retry >>>> - End >>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Mar 29 08:51:36 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 29 Mar 2017 10:51:36 +0200 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: References: Message-ID: And what's the Apache value? -- Guillaume Quintard On Wed, Mar 29, 2017 at 10:48 AM, Hazar G?ney wrote: > It is default value which is 60: > > param.show backend_idle_timeout > 200 > backend_idle_timeout > Value is: 60.000 [seconds] (default) > Minimum is: 1.000 > > Timeout before we close unused backend connections. > > On Wed, Mar 29, 2017 at 11:18 AM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> I'm suspecting Apache's closing a keep-alive connection because it and >> Varnish didn't agree on the timeout. Can you check? (in varnish, the param >> is called backend_idle_timeout) >> >> -- >> Guillaume Quintard >> >> On Wed, Mar 29, 2017 at 10:12 AM, Hazar G?ney >> wrote: >> >>> Backend is Apache. >>> >>> On Wed, Mar 29, 2017 at 10:34 AM, Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> Indeed, looking at the Timestamp lines, it's not a timeout. What's your >>>> backend? >>>> >>>> -- >>>> Guillaume Quintard >>>> >>>> On Wed, Mar 29, 2017 at 7:45 AM, Hazar G?ney >>>> wrote: >>>> >>>>> I see sporadic fetch errors (http first read error: EOF) from backend >>>>> but backend is normally healthy. It seems that timeout is not the root >>>>> issue in this case. What could be the reason? >>>>> >>>>> * << BeReq >> 98808229 >>>>> - Begin bereq 98808228 fetch >>>>> - Timestamp Start: 1490683823.763272 0.000000 0.000000 >>>>> - BereqMethod GET >>>>> - BereqURL XXXX >>>>> - BereqProtocol HTTP/1.1 >>>>> - BereqHeader Pragma: no-cache >>>>> - BereqHeader Accept: */* >>>>> - BereqHeader From: bingbot(at)microsoft.com >>>>> - BereqHeader Host: XXXX >>>>> - BereqHeader User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) >>>>> - BereqHeader Accept-Encoding: gzip >>>>> - BereqHeader X-Varnish: 98808229 >>>>> - VCL_call BACKEND_FETCH >>>>> - VCL_return fetch >>>>> - BackendOpen 38 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 172.17.0.2 48388 >>>>> - BackendStart 10.35.78.11 80 >>>>> - Timestamp Bereq: 1490683823.763758 0.000487 0.000487 >>>>> - *FetchError* *http first read error: EOF* >>>>> - BackendClose 38 reload_2017-03-20T11:32:44.st2 >>>>> - Timestamp Beresp: 1490683823.764271 0.000999 0.000513 >>>>> - Timestamp Error: 1490683823.764277 0.001005 0.000005 >>>>> - BerespProtocol HTTP/1.1 >>>>> - BerespStatus 503 >>>>> - BerespReason Service Unavailable >>>>> - BerespReason Backend fetch failed >>>>> - BerespHeader Date: Tue, 28 Mar 2017 06:50:23 GMT >>>>> - BerespHeader Server: Varnish >>>>> - VCL_call BACKEND_ERROR >>>>> - BereqHeader X-Varnish-Backend-5xx: 1 >>>>> - VCL_return retry >>>>> - Timestamp Retry: 1490683823.764294 0.001022 0.000017 >>>>> - Link bereq 97940444 retry >>>>> - End >>>>> >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>> >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattias at nucleus.be Wed Mar 29 09:03:28 2017 From: mattias at nucleus.be (Mattias Geniar) Date: Wed, 29 Mar 2017 09:03:28 +0000 Subject: =?utf-8?B?UmU6IFJhbmRvbSDigJxodHRwIGZpcnN0IHJlYWQgZXJyb3I6IEVPRuKAnSBl?= =?utf-8?Q?rrors?= In-Reply-To: References: Message-ID: <01F55BD6-0E30-444D-9F7D-470DDA52F329@nucleus.be> > Backend is Apache. In older Varnish versions, you could sometimes see a similar error; > 11 FetchError c straight insufficient bytes The error message you?re seeing might be related, as it mentions the EOF. This happens when the backend sends a Content-Length header that doesn?t match the _actual_ content length it?s sending. In Apache, this was commonly caused by a mod_deflate misconfiguration. For testing, could you try disabling Gzip either in your backend or strip the Accept-Encoding header in Varnish to force a plain text response? Mattias From mark at hanfordonline.co.uk Wed Mar 29 16:12:19 2017 From: mark at hanfordonline.co.uk (Mark Hanford) Date: Wed, 29 Mar 2017 17:12:19 +0100 Subject: Converting 3-to-5 - strange "No backend" errors Message-ID: Hi folks. I'm in the process of migrating to from v3 to v5, and have finally got the config compiling at least. Now I'm having some problems with the backends that I can't seem to work out. Apologies, this is a bit of a wordy one... Let's say I have two backends, web01 and web02, and these are put into a director called admin_director for load-balancing. This is all in "backends.vcl": import directors; probe healthcheck { .request = "GET / HTTP/1.1" "Host: www.mydomain.co.uk" "Connection: close"; .timeout = 30s; .interval = 15s; .window = 10; .threshold = 8; .expected_response = 302; } backend web01 { .host = "192.168.1.11"; .port = "80"; .first_byte_timeout = 600s; .probe = healthcheck; } backend web02 { .host = "192.168.1.12"; .port = "80"; .first_byte_timeout = 600s; .probe = healthcheck; } sub vcl_init { new admin_director = directors.shard(); admin_director.add_backend(web01); admin_director.add_backend(web02); } I've now removed all my main VCL for clarification, so my vcl_recv.vcl file contains just: import std; sub vcl_recv { set req.backend_hint = admin_director.backend(); std.log("Backend hint: " + req.backend_hint); } I then have my main "default.vcl": vcl 4.0; include "backends.vcl"; include "vcl_recv.vcl"; Now to the problem. If I try to access a URL, I always get a "FetchError: No backend" * << BeReq >> 3 - Begin bereq 2 fetch - Timestamp Start: 1490802880.486355 0.000000 0.000000 - BereqMethod GET - BereqURL /Common/Images/NewLoginImages/training.png - BereqProtocol HTTP/1.1 - BereqHeader Host: www.mydomain.com - BereqHeader User-Agent: curl/7.35.0 - BereqHeader Accept: */* - BereqHeader X-Forwarded-Proto: https - BereqHeader X-Real-Ip: 1.2.3.4 - BereqHeader Accept-Encoding: gzip - BereqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23 - BereqHeader X-Varnish: 3 - VCL_call BACKEND_FETCH - VCL_return fetch - FetchError No backend - Timestamp Beresp: 1490802880.486447 0.000092 0.000092 - Timestamp Error: 1490802880.486452 0.000098 0.000006 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Service Unavailable - BerespReason Backend fetch failed - BerespHeader Date: Wed, 29 Mar 2017 15:54:40 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR - BerespHeader Content-Type: text/html; charset=utf-8 - BerespHeader Retry-After: 5 - VCL_return deliver - Storage malloc Transient - ObjProtocol HTTP/1.1 - ObjStatus 503 - ObjReason Backend fetch failed - ObjHeader Date: Wed, 29 Mar 2017 15:54:40 GMT - ObjHeader Server: Varnish - ObjHeader Content-Type: text/html; charset=utf-8 - ObjHeader Retry-After: 5 - Length 278 - BereqAcct 0 0 0 0 0 0 - End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1490803780.494500 0.000000 0.000000 - Timestamp Req: 1490803780.494500 0.000000 0.000000 - ReqStart 192.168.1.23 36896 - ReqMethod GET - ReqURL /Common/Images/NewLoginImages/training.png - ReqProtocol HTTP/1.1 - ReqHeader Host: www.mydomain.com - ReqHeader User-Agent: curl/7.35.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: 1.2.3.4 - ReqHeader X-Forwarded-Proto: https - ReqHeader X-Real-Ip: 1.2.3.4 - ReqHeader Accept-Encoding: gzip - ReqUnset X-Forwarded-For: 1.2.3.4 - ReqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23 - VCL_call RECV - Error shard admin_director: no backends - VCL_Log Backend hint: - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32771 fetch - Timestamp Fetch: 1490803780.495705 0.001204 0.001204 - RespProtocol HTTP/1.1 - RespStatus 503 - RespReason Backend fetch failed - RespHeader Date: Wed, 29 Mar 2017 16:09:40 GMT - RespHeader Server: Varnish - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - RespHeader X-Varnish: 32770 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/5.1) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1490803780.495723 0.001223 0.000019 - RespHeader Content-Length: 282 - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1490803780.495806 0.001305 0.000082 - ReqAcct 238 0 238 250 282 532 - End But my backend.list seems to always be healthy: boot.web01 probe Healthy 10/10 Wed, 29 Mar 2017 15:54:18 GMT boot.web02 probe Healthy 10/10 Wed, 29 Mar 2017 15:54:18 GMT >From the varnish server, I can reach the backends successfully manually: curl --resolve www.mydomain.com:192.168.1.11 http://www.mydomain.com/Common/Images/NewLoginImages/training.png Not sure what it all means. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at hanfordonline.co.uk Thu Mar 30 10:11:07 2017 From: mark at hanfordonline.co.uk (Mark Hanford) Date: Thu, 30 Mar 2017 11:11:07 +0100 Subject: Correct usage of the Shard director Message-ID: Hi folks. I am transitioning to v5 VCL, and the shard directors look useful. In our case, we have several products behind Varnish, so I create a bunch of directors: sub vcl_init { new product_a_director = directors.shard(); product_a_director.add_backend(web01); product_a_director.add_backend(web02); product_a_director.reconfigure(); new product_b_director = directors.shard(); product_b_director.add_backend(web03); product_b_director.add_backend(web04); product_b_director.reconfigure(); } Then, later on in my vcl_recv, I have to make a bunch of decisions based on the host and URL to determine which set of backends to send the request to: if (req.http.host == "product_a.mydomain.com") { set req.backend_hint = product_a_director.backend(); } if (req.url ~ "^/login.*") { set req.backend_hint = product_b_director.backend(); } In reality, there are thirteen directors for various products and product areas, which direct traffic to various combinations of the 15 backends to separate workloads or to route to servers with particular configuration. It is important for us that traffic from a particular IP goes to the same backend in each director whenever possible, so do I have to use the manual "key" property of the shard director? That would change the above to: if (req.http.host == "product_a.mydomain.com") { set req.backend_hint = product_a_director.backend(KEY, product_a_director.key(client.ip)); } if (req.url ~ "^/login.*") { set req.backend_hint = product_b_director.backend(KEY, product_b_director.key(client_ip)); } Is that the correct way to lock a client to a backend? I do not want to use session cookies, this should transcend sessions, so all traffic from one IP goes to the same backend (if healthy/available/possible). Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From hazarguney at gmail.com Thu Mar 30 11:04:07 2017 From: hazarguney at gmail.com (=?UTF-8?B?SGF6YXIgR8O8bmV5?=) Date: Thu, 30 Mar 2017 14:04:07 +0300 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: <01F55BD6-0E30-444D-9F7D-470DDA52F329@nucleus.be> References: <01F55BD6-0E30-444D-9F7D-470DDA52F329@nucleus.be> Message-ID: MaxKeepAliveRequests 20 KeepAliveTimeout 2 Version is "4.1.3 revision 5e3b6d2". We have also seen "straight insufficient bytes" error with POST requests to a specific php script hosted by another backend and fixed it by using "pipe" instead of "pass" but this specific backend gives "http first read error: EOF" error. Another example from today: * << BeReq >> 126635444 - Begin bereq 126635443 fetch - Timestamp Start: 1490870598.921499 0.000000 0.000000 - BereqMethod GET - BereqURL XXXX - BereqProtocol HTTP/1.1 - BereqHeader Host: XXXX - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 - BereqHeader Accept: image/webp,image/*,*/*;q=0.8 - BereqHeader Referer: XXXX - BereqHeader Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4 - BereqHeader RIP: XXXX - BereqHeader X-Forwarded-For: XXXX - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 126635444 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 35 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 172.17.0.2 48896 - BackendStart 10.35.78.11 80 - Timestamp Bereq: 1490870598.922050 0.000552 0.000552 *- FetchError http first read error: EOF* - BackendClose 35 reload_2017-03-20T11:32:44.st2 - Timestamp Beresp: 1490870598.922622 0.001124 0.000572 - Timestamp Error: 1490870598.922627 0.001129 0.000005 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Service Unavailable - BerespReason Backend fetch failed - BerespHeader Date: Thu, 30 Mar 2017 10:43:18 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR - BereqHeader X-Varnish-Backend-5xx: 1 - VCL_return retry - Timestamp Retry: 1490870598.922657 0.001159 0.000030 - Link bereq 126832283 retry - End On Wed, Mar 29, 2017 at 12:03 PM, Mattias Geniar wrote: > > Backend is Apache. > > In older Varnish versions, you could sometimes see a similar error; > > > 11 FetchError c straight insufficient bytes > > The error message you?re seeing might be related, as it mentions the EOF. > > This happens when the backend sends a Content-Length header that doesn?t > match the _actual_ content length it?s sending. In Apache, this was > commonly caused by a mod_deflate misconfiguration. > > For testing, could you try disabling Gzip either in your backend or strip > the Accept-Encoding header in Varnish to force a plain text response? > > Mattias > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Mar 30 11:08:32 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 30 Mar 2017 13:08:32 +0200 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: References: <01F55BD6-0E30-444D-9F7D-470DDA52F329@nucleus.be> Message-ID: Can you try something: add 'set bereq.http.connection = "Close"; ' at the beginning of vcl_backend_fetch and see if that helps? -- Guillaume Quintard On Thu, Mar 30, 2017 at 1:04 PM, Hazar G?ney wrote: > MaxKeepAliveRequests 20 > KeepAliveTimeout 2 > > Version is "4.1.3 revision 5e3b6d2". We have also seen "straight > insufficient bytes" error with POST requests to a specific php script > hosted by another backend and fixed it by using "pipe" instead of "pass" > but this specific backend gives "http first read error: EOF" error. Another > example from today: > > * << BeReq >> 126635444 > - Begin bereq 126635443 fetch > - Timestamp Start: 1490870598.921499 0.000000 0.000000 > - BereqMethod GET > - BereqURL XXXX > - BereqProtocol HTTP/1.1 > - BereqHeader Host: XXXX > - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 > - BereqHeader Accept: image/webp,image/*,*/*;q=0.8 > - BereqHeader Referer: XXXX > - BereqHeader Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4 > - BereqHeader RIP: XXXX > - BereqHeader X-Forwarded-For: XXXX > - BereqHeader Accept-Encoding: gzip > - BereqHeader X-Varnish: 126635444 > - VCL_call BACKEND_FETCH > - VCL_return fetch > - BackendOpen 35 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 > 172.17.0.2 48896 > - BackendStart 10.35.78.11 80 > - Timestamp Bereq: 1490870598.922050 0.000552 0.000552 > *- FetchError http first read error: EOF* > - BackendClose 35 reload_2017-03-20T11:32:44.st2 > - Timestamp Beresp: 1490870598.922622 0.001124 0.000572 > - Timestamp Error: 1490870598.922627 0.001129 0.000005 > - BerespProtocol HTTP/1.1 > - BerespStatus 503 > - BerespReason Service Unavailable > - BerespReason Backend fetch failed > - BerespHeader Date: Thu, 30 Mar 2017 10:43:18 GMT > - BerespHeader Server: Varnish > - VCL_call BACKEND_ERROR > - BereqHeader X-Varnish-Backend-5xx: 1 > - VCL_return retry > - Timestamp Retry: 1490870598.922657 0.001159 0.000030 > - Link bereq 126832283 retry > - End > > On Wed, Mar 29, 2017 at 12:03 PM, Mattias Geniar > wrote: > >> > Backend is Apache. >> >> In older Varnish versions, you could sometimes see a similar error; >> >> > 11 FetchError c straight insufficient bytes >> >> The error message you?re seeing might be related, as it mentions the EOF. >> >> This happens when the backend sends a Content-Length header that doesn?t >> match the _actual_ content length it?s sending. In Apache, this was >> commonly caused by a mod_deflate misconfiguration. >> >> For testing, could you try disabling Gzip either in your backend or strip >> the Accept-Encoding header in Varnish to force a plain text response? >> >> Mattias >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hazarguney at gmail.com Thu Mar 30 11:17:52 2017 From: hazarguney at gmail.com (=?UTF-8?B?SGF6YXIgR8O8bmV5?=) Date: Thu, 30 Mar 2017 14:17:52 +0300 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: References: <01F55BD6-0E30-444D-9F7D-470DDA52F329@nucleus.be> Message-ID: "Connection: close" supersedes keep-alive behavior, is that correct? On Thu, Mar 30, 2017 at 2:08 PM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > Can you try something: add 'set bereq.http.connection = "Close"; ' at the > beginning of vcl_backend_fetch and see if that helps? > > -- > Guillaume Quintard > > On Thu, Mar 30, 2017 at 1:04 PM, Hazar G?ney wrote: > >> MaxKeepAliveRequests 20 >> KeepAliveTimeout 2 >> >> Version is "4.1.3 revision 5e3b6d2". We have also seen "straight >> insufficient bytes" error with POST requests to a specific php script >> hosted by another backend and fixed it by using "pipe" instead of "pass" >> but this specific backend gives "http first read error: EOF" error. Another >> example from today: >> >> * << BeReq >> 126635444 >> - Begin bereq 126635443 fetch >> - Timestamp Start: 1490870598.921499 0.000000 0.000000 >> - BereqMethod GET >> - BereqURL XXXX >> - BereqProtocol HTTP/1.1 >> - BereqHeader Host: XXXX >> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 >> - BereqHeader Accept: image/webp,image/*,*/*;q=0.8 >> - BereqHeader Referer: XXXX >> - BereqHeader Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4 >> - BereqHeader RIP: XXXX >> - BereqHeader X-Forwarded-For: XXXX >> - BereqHeader Accept-Encoding: gzip >> - BereqHeader X-Varnish: 126635444 >> - VCL_call BACKEND_FETCH >> - VCL_return fetch >> - BackendOpen 35 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 >> 172.17.0.2 48896 >> - BackendStart 10.35.78.11 80 >> - Timestamp Bereq: 1490870598.922050 0.000552 0.000552 >> *- FetchError http first read error: EOF* >> - BackendClose 35 reload_2017-03-20T11:32:44.st2 >> - Timestamp Beresp: 1490870598.922622 0.001124 0.000572 >> - Timestamp Error: 1490870598.922627 0.001129 0.000005 >> - BerespProtocol HTTP/1.1 >> - BerespStatus 503 >> - BerespReason Service Unavailable >> - BerespReason Backend fetch failed >> - BerespHeader Date: Thu, 30 Mar 2017 10:43:18 GMT >> - BerespHeader Server: Varnish >> - VCL_call BACKEND_ERROR >> - BereqHeader X-Varnish-Backend-5xx: 1 >> - VCL_return retry >> - Timestamp Retry: 1490870598.922657 0.001159 0.000030 >> - Link bereq 126832283 retry >> - End >> >> On Wed, Mar 29, 2017 at 12:03 PM, Mattias Geniar >> wrote: >> >>> > Backend is Apache. >>> >>> In older Varnish versions, you could sometimes see a similar error; >>> >>> > 11 FetchError c straight insufficient bytes >>> >>> The error message you?re seeing might be related, as it mentions the EOF. >>> >>> This happens when the backend sends a Content-Length header that doesn?t >>> match the _actual_ content length it?s sending. In Apache, this was >>> commonly caused by a mod_deflate misconfiguration. >>> >>> For testing, could you try disabling Gzip either in your backend or >>> strip the Accept-Encoding header in Varnish to force a plain text response? >>> >>> Mattias >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Mar 30 11:34:27 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 30 Mar 2017 13:34:27 +0200 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: References: <01F55BD6-0E30-444D-9F7D-470DDA52F329@nucleus.be> Message-ID: It does, I'm suspecting that the connection reuse is creating some issues, probably because Apache is doing some non-standard stuff (protip: always blame Apache). -- Guillaume Quintard On Thu, Mar 30, 2017 at 1:17 PM, Hazar G?ney wrote: > "Connection: close" supersedes keep-alive behavior, is that correct? > > On Thu, Mar 30, 2017 at 2:08 PM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> Can you try something: add 'set bereq.http.connection = "Close"; ' at the >> beginning of vcl_backend_fetch and see if that helps? >> >> -- >> Guillaume Quintard >> >> On Thu, Mar 30, 2017 at 1:04 PM, Hazar G?ney >> wrote: >> >>> MaxKeepAliveRequests 20 >>> KeepAliveTimeout 2 >>> >>> Version is "4.1.3 revision 5e3b6d2". We have also seen "straight >>> insufficient bytes" error with POST requests to a specific php script >>> hosted by another backend and fixed it by using "pipe" instead of "pass" >>> but this specific backend gives "http first read error: EOF" error. Another >>> example from today: >>> >>> * << BeReq >> 126635444 >>> - Begin bereq 126635443 fetch >>> - Timestamp Start: 1490870598.921499 0.000000 0.000000 >>> - BereqMethod GET >>> - BereqURL XXXX >>> - BereqProtocol HTTP/1.1 >>> - BereqHeader Host: XXXX >>> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) >>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 >>> - BereqHeader Accept: image/webp,image/*,*/*;q=0.8 >>> - BereqHeader Referer: XXXX >>> - BereqHeader Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4 >>> - BereqHeader RIP: XXXX >>> - BereqHeader X-Forwarded-For: XXXX >>> - BereqHeader Accept-Encoding: gzip >>> - BereqHeader X-Varnish: 126635444 >>> - VCL_call BACKEND_FETCH >>> - VCL_return fetch >>> - BackendOpen 35 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 >>> 172.17.0.2 48896 >>> - BackendStart 10.35.78.11 80 >>> - Timestamp Bereq: 1490870598.922050 0.000552 0.000552 >>> *- FetchError http first read error: EOF* >>> - BackendClose 35 reload_2017-03-20T11:32:44.st2 >>> - Timestamp Beresp: 1490870598.922622 0.001124 0.000572 >>> - Timestamp Error: 1490870598.922627 0.001129 0.000005 >>> - BerespProtocol HTTP/1.1 >>> - BerespStatus 503 >>> - BerespReason Service Unavailable >>> - BerespReason Backend fetch failed >>> - BerespHeader Date: Thu, 30 Mar 2017 10:43:18 GMT >>> - BerespHeader Server: Varnish >>> - VCL_call BACKEND_ERROR >>> - BereqHeader X-Varnish-Backend-5xx: 1 >>> - VCL_return retry >>> - Timestamp Retry: 1490870598.922657 0.001159 0.000030 >>> - Link bereq 126832283 retry >>> - End >>> >>> On Wed, Mar 29, 2017 at 12:03 PM, Mattias Geniar >>> wrote: >>> >>>> > Backend is Apache. >>>> >>>> In older Varnish versions, you could sometimes see a similar error; >>>> >>>> > 11 FetchError c straight insufficient bytes >>>> >>>> The error message you?re seeing might be related, as it mentions the >>>> EOF. >>>> >>>> This happens when the backend sends a Content-Length header that >>>> doesn?t match the _actual_ content length it?s sending. In Apache, this was >>>> commonly caused by a mod_deflate misconfiguration. >>>> >>>> For testing, could you try disabling Gzip either in your backend or >>>> strip the Accept-Encoding header in Varnish to force a plain text response? >>>> >>>> Mattias >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hazarguney at gmail.com Thu Mar 30 12:41:25 2017 From: hazarguney at gmail.com (=?UTF-8?B?SGF6YXIgR8O8bmV5?=) Date: Thu, 30 Mar 2017 15:41:25 +0300 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: References: <01F55BD6-0E30-444D-9F7D-470DDA52F329@nucleus.be> Message-ID: It did not work either: * << BeReq >> 127418176 - Begin bereq 127418175 fetch - Timestamp Start: 1490877149.450124 0.000000 0.000000 - BereqMethod GET - BereqURL XXXX - BereqProtocol HTTP/1.1 - BereqHeader Accept: text/css,*/*;q=0.1 - BereqHeader User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0 Mobile/14C92 Safari/602.1 - BereqHeader Accept-Language: tr-tr - BereqHeader Referer: XXXX - BereqHeader Host: XXXX - BereqHeader RIP: XXXX - BereqHeader X-Forwarded-For: XXXX - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 127418176 - VCL_call BACKEND_FETCH - BereqHeader connection: Close - VCL_return fetch - BackendOpen 25 reload_2017-03-30T14:53:46.st2 10.35.78.11 80 172.17.0.2 59152 - BackendStart 10.35.78.11 80 - Timestamp Bereq: 1490877149.450594 0.000470 0.000470 - FetchError http first read error: EOF - BackendClose 25 reload_2017-03-30T14:53:46.st2 - Timestamp Beresp: 1490877149.451184 0.001060 0.000590 - Timestamp Error: 1490877149.451189 0.001065 0.000005 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Service Unavailable - BerespReason Backend fetch failed - BerespHeader Date: Thu, 30 Mar 2017 12:32:29 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR - BereqHeader X-Varnish-Backend-5xx: 1 - VCL_return retry - Timestamp Retry: 1490877149.451205 0.001081 0.000016 - Link bereq 127298071 retry - End On Thu, Mar 30, 2017 at 2:34 PM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > It does, I'm suspecting that the connection reuse is creating some issues, > probably because Apache is doing some non-standard stuff (protip: always > blame Apache). > > -- > Guillaume Quintard > > On Thu, Mar 30, 2017 at 1:17 PM, Hazar G?ney wrote: > >> "Connection: close" supersedes keep-alive behavior, is that correct? >> >> On Thu, Mar 30, 2017 at 2:08 PM, Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> Can you try something: add 'set bereq.http.connection = "Close"; ' at >>> the beginning of vcl_backend_fetch and see if that helps? >>> >>> -- >>> Guillaume Quintard >>> >>> On Thu, Mar 30, 2017 at 1:04 PM, Hazar G?ney >>> wrote: >>> >>>> MaxKeepAliveRequests 20 >>>> KeepAliveTimeout 2 >>>> >>>> Version is "4.1.3 revision 5e3b6d2". We have also seen "straight >>>> insufficient bytes" error with POST requests to a specific php script >>>> hosted by another backend and fixed it by using "pipe" instead of "pass" >>>> but this specific backend gives "http first read error: EOF" error. Another >>>> example from today: >>>> >>>> * << BeReq >> 126635444 >>>> - Begin bereq 126635443 fetch >>>> - Timestamp Start: 1490870598.921499 0.000000 0.000000 >>>> - BereqMethod GET >>>> - BereqURL XXXX >>>> - BereqProtocol HTTP/1.1 >>>> - BereqHeader Host: XXXX >>>> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; >>>> x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 >>>> Safari/537.36 >>>> - BereqHeader Accept: image/webp,image/*,*/*;q=0.8 >>>> - BereqHeader Referer: XXXX >>>> - BereqHeader Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4 >>>> - BereqHeader RIP: XXXX >>>> - BereqHeader X-Forwarded-For: XXXX >>>> - BereqHeader Accept-Encoding: gzip >>>> - BereqHeader X-Varnish: 126635444 >>>> - VCL_call BACKEND_FETCH >>>> - VCL_return fetch >>>> - BackendOpen 35 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 >>>> 172.17.0.2 48896 >>>> - BackendStart 10.35.78.11 80 >>>> - Timestamp Bereq: 1490870598.922050 0.000552 0.000552 >>>> *- FetchError http first read error: EOF* >>>> - BackendClose 35 reload_2017-03-20T11:32:44.st2 >>>> - Timestamp Beresp: 1490870598.922622 0.001124 0.000572 >>>> - Timestamp Error: 1490870598.922627 0.001129 0.000005 >>>> - BerespProtocol HTTP/1.1 >>>> - BerespStatus 503 >>>> - BerespReason Service Unavailable >>>> - BerespReason Backend fetch failed >>>> - BerespHeader Date: Thu, 30 Mar 2017 10:43:18 GMT >>>> - BerespHeader Server: Varnish >>>> - VCL_call BACKEND_ERROR >>>> - BereqHeader X-Varnish-Backend-5xx: 1 >>>> - VCL_return retry >>>> - Timestamp Retry: 1490870598.922657 0.001159 0.000030 >>>> - Link bereq 126832283 retry >>>> - End >>>> >>>> On Wed, Mar 29, 2017 at 12:03 PM, Mattias Geniar >>>> wrote: >>>> >>>>> > Backend is Apache. >>>>> >>>>> In older Varnish versions, you could sometimes see a similar error; >>>>> >>>>> > 11 FetchError c straight insufficient bytes >>>>> >>>>> The error message you?re seeing might be related, as it mentions the >>>>> EOF. >>>>> >>>>> This happens when the backend sends a Content-Length header that >>>>> doesn?t match the _actual_ content length it?s sending. In Apache, this was >>>>> commonly caused by a mod_deflate misconfiguration. >>>>> >>>>> For testing, could you try disabling Gzip either in your backend or >>>>> strip the Accept-Encoding header in Varnish to force a plain text response? >>>>> >>>>> Mattias >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Mar 30 12:43:45 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 30 Mar 2017 14:43:45 +0200 Subject: Converting 3-to-5 - strange "No backend" errors In-Reply-To: References: Message-ID: I think you forgot to commit the backend addtions. -- Guillaume Quintard On Wed, Mar 29, 2017 at 6:12 PM, Mark Hanford wrote: > Hi folks. I'm in the process of migrating to from v3 to v5, and have > finally got the config compiling at least. Now I'm having some problems > with the backends that I can't seem to work out. Apologies, this is a bit > of a wordy one... > > Let's say I have two backends, web01 and web02, and these are put into a > director called admin_director for load-balancing. This is all in > "backends.vcl": > > import directors; > > probe healthcheck { > .request = > "GET / HTTP/1.1" > "Host: www.mydomain.co.uk" > "Connection: close"; > .timeout = 30s; > .interval = 15s; > .window = 10; > .threshold = 8; > .expected_response = 302; > } > > > backend web01 { .host = "192.168.1.11"; .port = "80"; .first_byte_timeout > = 600s; .probe = healthcheck; } > backend web02 { .host = "192.168.1.12"; .port = "80"; .first_byte_timeout > = 600s; .probe = healthcheck; } > > sub vcl_init { > new admin_director = directors.shard(); > admin_director.add_backend(web01); > admin_director.add_backend(web02); > } > > > I've now removed all my main VCL for clarification, so my vcl_recv.vcl > file contains just: > > > import std; > > sub vcl_recv { > set req.backend_hint = admin_director.backend(); > std.log("Backend hint: " + req.backend_hint); > } > > > > I then have my main "default.vcl": > > > vcl 4.0; > include "backends.vcl"; > include "vcl_recv.vcl"; > > > Now to the problem. If I try to access a URL, I always get a "FetchError: > No backend" > > * << BeReq >> 3 > - Begin bereq 2 fetch > - Timestamp Start: 1490802880.486355 0.000000 0.000000 > - BereqMethod GET > - BereqURL /Common/Images/NewLoginImages/training.png > - BereqProtocol HTTP/1.1 > - BereqHeader Host: www.mydomain.com > - BereqHeader User-Agent: curl/7.35.0 > - BereqHeader Accept: */* > - BereqHeader X-Forwarded-Proto: https > - BereqHeader X-Real-Ip: 1.2.3.4 > - BereqHeader Accept-Encoding: gzip > - BereqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23 > - BereqHeader X-Varnish: 3 > - VCL_call BACKEND_FETCH > - VCL_return fetch > - FetchError No backend > - Timestamp Beresp: 1490802880.486447 0.000092 0.000092 > - Timestamp Error: 1490802880.486452 0.000098 0.000006 > - BerespProtocol HTTP/1.1 > - BerespStatus 503 > - BerespReason Service Unavailable > - BerespReason Backend fetch failed > - BerespHeader Date: Wed, 29 Mar 2017 15:54:40 GMT > - BerespHeader Server: Varnish > - VCL_call BACKEND_ERROR > - BerespHeader Content-Type: text/html; charset=utf-8 > - BerespHeader Retry-After: 5 > - VCL_return deliver > - Storage malloc Transient > - ObjProtocol HTTP/1.1 > - ObjStatus 503 > - ObjReason Backend fetch failed > - ObjHeader Date: Wed, 29 Mar 2017 15:54:40 GMT > - ObjHeader Server: Varnish > - ObjHeader Content-Type: text/html; charset=utf-8 > - ObjHeader Retry-After: 5 > - Length 278 > - BereqAcct 0 0 0 0 0 0 > - End > > * << Request >> 32770 > - Begin req 32769 rxreq > - Timestamp Start: 1490803780.494500 0.000000 0.000000 > - Timestamp Req: 1490803780.494500 0.000000 0.000000 > - ReqStart 192.168.1.23 36896 > - ReqMethod GET > - ReqURL /Common/Images/NewLoginImages/training.png > - ReqProtocol HTTP/1.1 > - ReqHeader Host: www.mydomain.com > - ReqHeader User-Agent: curl/7.35.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: 1.2.3.4 > - ReqHeader X-Forwarded-Proto: https > - ReqHeader X-Real-Ip: 1.2.3.4 > - ReqHeader Accept-Encoding: gzip > - ReqUnset X-Forwarded-For: 1.2.3.4 > - ReqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23 > - VCL_call RECV > - Error shard admin_director: no backends > - VCL_Log Backend hint: > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 32771 fetch > - Timestamp Fetch: 1490803780.495705 0.001204 0.001204 > - RespProtocol HTTP/1.1 > - RespStatus 503 > - RespReason Backend fetch failed > - RespHeader Date: Wed, 29 Mar 2017 16:09:40 GMT > - RespHeader Server: Varnish > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - RespHeader X-Varnish: 32770 > - RespHeader Age: 0 > - RespHeader Via: 1.1 varnish (Varnish/5.1) > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1490803780.495723 0.001223 0.000019 > - RespHeader Content-Length: 282 > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1490803780.495806 0.001305 0.000082 > - ReqAcct 238 0 238 250 282 532 > - End > > But my backend.list seems to always be healthy: > > boot.web01 probe Healthy 10/10 > Wed, 29 Mar 2017 15:54:18 GMT > boot.web02 probe Healthy 10/10 > Wed, 29 Mar 2017 15:54:18 GMT > > From the varnish server, I can reach the backends successfully manually: > > curl --resolve www.mydomain.com:192.168.1.11 http://www.mydomain.com/ > Common/Images/NewLoginImages/training.png > > Not sure what it all means. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at hanfordonline.co.uk Thu Mar 30 12:57:41 2017 From: mark at hanfordonline.co.uk (Mark Hanford) Date: Thu, 30 Mar 2017 13:57:41 +0100 Subject: Converting 3-to-5 - strange "No backend" errors In-Reply-To: References: Message-ID: Aha, yes that was it. I literally just spotted the fact that the shard director needs a ".reconfigure()" after making changes. Thanks, that seems to have fixed it. Mark On 30 March 2017 at 13:43, Guillaume Quintard < guillaume at varnish-software.com> wrote: > I think you forgot to commit the backend addtions. > > -- > Guillaume Quintard > > On Wed, Mar 29, 2017 at 6:12 PM, Mark Hanford > wrote: > >> Hi folks. I'm in the process of migrating to from v3 to v5, and have >> finally got the config compiling at least. Now I'm having some problems >> with the backends that I can't seem to work out. Apologies, this is a bit >> of a wordy one... >> >> Let's say I have two backends, web01 and web02, and these are put into a >> director called admin_director for load-balancing. This is all in >> "backends.vcl": >> >> import directors; >> >> probe healthcheck { >> .request = >> "GET / HTTP/1.1" >> "Host: www.mydomain.co.uk" >> "Connection: close"; >> .timeout = 30s; >> .interval = 15s; >> .window = 10; >> .threshold = 8; >> .expected_response = 302; >> } >> >> >> backend web01 { .host = "192.168.1.11"; .port = "80"; .first_byte_timeout >> = 600s; .probe = healthcheck; } >> backend web02 { .host = "192.168.1.12"; .port = "80"; .first_byte_timeout >> = 600s; .probe = healthcheck; } >> >> sub vcl_init { >> new admin_director = directors.shard(); >> admin_director.add_backend(web01); >> admin_director.add_backend(web02); >> } >> >> >> I've now removed all my main VCL for clarification, so my vcl_recv.vcl >> file contains just: >> >> >> import std; >> >> sub vcl_recv { >> set req.backend_hint = admin_director.backend(); >> std.log("Backend hint: " + req.backend_hint); >> } >> >> >> >> I then have my main "default.vcl": >> >> >> vcl 4.0; >> include "backends.vcl"; >> include "vcl_recv.vcl"; >> >> >> Now to the problem. If I try to access a URL, I always get a "FetchError: >> No backend" >> >> * << BeReq >> 3 >> - Begin bereq 2 fetch >> - Timestamp Start: 1490802880.486355 0.000000 0.000000 >> - BereqMethod GET >> - BereqURL /Common/Images/NewLoginImages/training.png >> - BereqProtocol HTTP/1.1 >> - BereqHeader Host: www.mydomain.com >> - BereqHeader User-Agent: curl/7.35.0 >> - BereqHeader Accept: */* >> - BereqHeader X-Forwarded-Proto: https >> - BereqHeader X-Real-Ip: 1.2.3.4 >> - BereqHeader Accept-Encoding: gzip >> - BereqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23 >> - BereqHeader X-Varnish: 3 >> - VCL_call BACKEND_FETCH >> - VCL_return fetch >> - FetchError No backend >> - Timestamp Beresp: 1490802880.486447 0.000092 0.000092 >> - Timestamp Error: 1490802880.486452 0.000098 0.000006 >> - BerespProtocol HTTP/1.1 >> - BerespStatus 503 >> - BerespReason Service Unavailable >> - BerespReason Backend fetch failed >> - BerespHeader Date: Wed, 29 Mar 2017 15:54:40 GMT >> - BerespHeader Server: Varnish >> - VCL_call BACKEND_ERROR >> - BerespHeader Content-Type: text/html; charset=utf-8 >> - BerespHeader Retry-After: 5 >> - VCL_return deliver >> - Storage malloc Transient >> - ObjProtocol HTTP/1.1 >> - ObjStatus 503 >> - ObjReason Backend fetch failed >> - ObjHeader Date: Wed, 29 Mar 2017 15:54:40 GMT >> - ObjHeader Server: Varnish >> - ObjHeader Content-Type: text/html; charset=utf-8 >> - ObjHeader Retry-After: 5 >> - Length 278 >> - BereqAcct 0 0 0 0 0 0 >> - End >> >> * << Request >> 32770 >> - Begin req 32769 rxreq >> - Timestamp Start: 1490803780.494500 0.000000 0.000000 >> - Timestamp Req: 1490803780.494500 0.000000 0.000000 >> - ReqStart 192.168.1.23 36896 >> - ReqMethod GET >> - ReqURL /Common/Images/NewLoginImages/training.png >> - ReqProtocol HTTP/1.1 >> - ReqHeader Host: www.mydomain.com >> - ReqHeader User-Agent: curl/7.35.0 >> - ReqHeader Accept: */* >> - ReqHeader X-Forwarded-For: 1.2.3.4 >> - ReqHeader X-Forwarded-Proto: https >> - ReqHeader X-Real-Ip: 1.2.3.4 >> - ReqHeader Accept-Encoding: gzip >> - ReqUnset X-Forwarded-For: 1.2.3.4 >> - ReqHeader X-Forwarded-For: 1.2.3.4, 192.168.1.23 >> - VCL_call RECV >> - Error shard admin_director: no backends >> - VCL_Log Backend hint: >> - VCL_return hash >> - VCL_call HASH >> - VCL_return lookup >> - VCL_call MISS >> - VCL_return fetch >> - Link bereq 32771 fetch >> - Timestamp Fetch: 1490803780.495705 0.001204 0.001204 >> - RespProtocol HTTP/1.1 >> - RespStatus 503 >> - RespReason Backend fetch failed >> - RespHeader Date: Wed, 29 Mar 2017 16:09:40 GMT >> - RespHeader Server: Varnish >> - RespHeader Content-Type: text/html; charset=utf-8 >> - RespHeader Retry-After: 5 >> - RespHeader X-Varnish: 32770 >> - RespHeader Age: 0 >> - RespHeader Via: 1.1 varnish (Varnish/5.1) >> - VCL_call DELIVER >> - VCL_return deliver >> - Timestamp Process: 1490803780.495723 0.001223 0.000019 >> - RespHeader Content-Length: 282 >> - Debug "RES_MODE 2" >> - RespHeader Connection: keep-alive >> - Timestamp Resp: 1490803780.495806 0.001305 0.000082 >> - ReqAcct 238 0 238 250 282 532 >> - End >> >> But my backend.list seems to always be healthy: >> >> boot.web01 probe Healthy 10/10 >> Wed, 29 Mar 2017 15:54:18 GMT >> boot.web02 probe Healthy 10/10 >> Wed, 29 Mar 2017 15:54:18 GMT >> >> From the varnish server, I can reach the backends successfully manually: >> >> curl --resolve www.mydomain.com:192.168.1.11 >> http://www.mydomain.com/Common/Images/NewLoginImages/training.png >> >> Not sure what it all means. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From devin at pabstatencio.com Thu Mar 30 22:17:07 2017 From: devin at pabstatencio.com (Devin Acosta) Date: Thu, 30 Mar 2017 15:17:07 -0700 Subject: Looking for Varnish Consultant? Message-ID: This may not be the appropriate place but hoping it's OK, we are trying to get Varnish configured on our network to cache websites that are mostly Wordpress related. We are looking for it to provide faster page loads, with loading from cache if the backend isn't responding. We are looking for someone who wants to help us out for some hours and help us ensure we have a valid working config. Please e-mail me if your interested. -- Devin Acosta -------------- next part -------------- An HTML attachment was scrubbed... URL: From hazarguney at gmail.com Fri Mar 31 06:25:09 2017 From: hazarguney at gmail.com (=?UTF-8?B?SGF6YXIgR8O8bmV5?=) Date: Fri, 31 Mar 2017 09:25:09 +0300 Subject: "http first read error: EOF" errors from WordPress backend Message-ID: Hello, >From a backend with Apache+WordPress, we occasionally see "http first read error: EOF" errors. This situation is different then my first question because this time it seems that Varnish waits for 60 seconds from backend. According to Apache access log, backend responds to the request from Varnish less than 1 second which is very strange. (We are 3 hours ahead of GMT) *Backend:* [31/Mar/2017:02:25:29 +0300] "GET XXXX HTTP/1.1" 200 22575 *Varnish:* * << BeReq >> 3684412 - Begin bereq 3684411 fetch - Timestamp Start: 1490916329.664410 0.000000 0.000000 - BereqMethod GET - BereqURL XXXX - BereqProtocol HTTP/1.1 - BereqHeader Host: XXXX - BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - BereqHeader From: googlebot(at)googlebot.com - BereqHeader User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; + http://www.google.com/bot.html) - BereqHeader RIP: 66.249.66.227 - BereqHeader X-Forwarded-For: XXXX - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 3684412 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 21 reload_2017-03-21T100643.default 10.35.78.153 80 172.17.0.2 56104 - BackendStart XXXX 80 - Timestamp Bereq: 1490916329.664853 0.000443 0.000443 *- FetchError http first read error: EOF* - BackendClose 21 reload_2017-03-21T100643.default - Timestamp Beresp: 1490916389.664967 *60.000557* 60.000114 - Timestamp Error: 1490916389.664978 *60.000567 *0.000011 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Service Unavailable - BerespReason Backend fetch failed - BerespHeader Date: Thu, 30 Mar 2017 23:26:29 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR - BerespHeader Content-Type: text/html; charset=utf-8 - BerespHeader Retry-After: 5 - VCL_return deliver - Storage malloc Transient - ObjProtocol HTTP/1.1 - ObjStatus 503 - ObjReason Backend fetch failed - ObjHeader Date: Thu, 30 Mar 2017 23:26:29 GMT - ObjHeader Server: Varnish - ObjHeader Content-Type: text/html; charset=utf-8 - ObjHeader Retry-After: 5 - Length 284 - BereqAcct 509 0 509 0 0 0 - End * << Request >> 3684411 - Begin req 3684410 rxreq - Timestamp Start: 1490916329.664345 0.000000 0.000000 - Timestamp Req: 1490916329.664345 0.000000 0.000000 - ReqStart XXXX 45415 - ReqMethod GET - ReqURL XXXX - ReqProtocol HTTP/1.1 - ReqHeader Host: XXXX - ReqHeader Connection: Keep-alive - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - ReqHeader From: googlebot(at)googlebot.com - ReqHeader User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; + http://www.google.com/bot.html) - ReqHeader RIP: 66.249.66.227 - ReqHeader X-Forwarded-For: XXXX - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3684412 fetch - Timestamp Fetch: 1490916389.665129 60.000784 60.000784 - RespProtocol HTTP/1.1 - RespStatus 503 - RespReason Backend fetch failed - RespHeader Date: Thu, 30 Mar 2017 23:26:29 GMT - RespHeader Server: Varnish - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - RespHeader X-Varnish: 3684411 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_acl NO_MATCH hitpass - RespUnset Via: 1.1 varnish-v4 - RespUnset X-Varnish: 3684411 - VCL_return deliver - Timestamp Process: 1490916389.665181 60.000836 0.000052 - RespHeader Content-Length: 284 - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1490916389.665241 60.000896 0.000060 - ReqAcct 461 0 461 200 284 484 - End -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattias at nucleus.be Fri Mar 31 08:11:24 2017 From: mattias at nucleus.be (Mattias Geniar) Date: Fri, 31 Mar 2017 08:11:24 +0000 Subject: "http first read error: EOF" errors from WordPress backend In-Reply-To: References: Message-ID: <9FC546D0-BB08-49F5-93B8-7A3C9F5A49E7@nucleus.be> > - ? FetchError ? ? http first read error: EOF > - ? BackendClose ? 21 reload_2017-03-21T100643.default > - ? Timestamp ? ? ?Beresp: 1490916389.664967 60.000557 60.000114 > - ? Timestamp ? ? ?Error: 1490916389.664978 60.000567 0.000011 At the risk of repeating myself: try to disable gzip & any wordpress plugins that might be trying to gzip on their own (aka: output buffering in PHP). To me, this seems like Varnish is waiting for the backend to send more data, because it replied with a certain Content-Length header but sent a few bytes less than it advertised, and Varnish is waiting for those missing bytes. Mattias From mark at hanfordonline.co.uk Fri Mar 31 09:44:32 2017 From: mark at hanfordonline.co.uk (Mark Hanford) Date: Fri, 31 Mar 2017 10:44:32 +0100 Subject: Using ACL with non-IP fields Message-ID: Hi folks. Because my varnish nodes are behind two different proxies, I can't really use client.ip within my VCL. What I have is a header "X-Real-Ip" instead, which is populated automatically by one proxy, and by me derived from the "X-Forwarded-For" for the other. What this means is that where I would usually use ACL to block access to a resource: if (req.http.host == "test.mydomain.com") { if (client.ip ~ trustedips) { # allow access } else { return (synth(405, "Not allowed"); } } But this doesn't work if I replace client.ip with a non-IP typed field. Message from VCC-compiler: Expected CSTR got 'purgers' (program line 1193), at ('default.vcl' Line 339 Pos 34) if (req.http.X-Real-Ip ~ trustedips) { ---------------------------------##########--- Is there any way I can get the same result as this but without using client.ip? thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Fri Mar 31 09:59:58 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 31 Mar 2017 11:59:58 +0200 Subject: Using ACL with non-IP fields In-Reply-To: References: Message-ID: Hi Mark, you need to use std.ip() from the std vmod. Also, I highly recommend using the PROXY protocol (Varnish, HAProxy and Nginx, at least support it), it will allow you to use client.ip directly and not bother with this. -- Guillaume Quintard On Fri, Mar 31, 2017 at 11:44 AM, Mark Hanford wrote: > Hi folks. > > Because my varnish nodes are behind two different proxies, I can't really > use client.ip within my VCL. What I have is a header "X-Real-Ip" instead, > which is populated automatically by one proxy, and by me derived from the > "X-Forwarded-For" for the other. > > What this means is that where I would usually use ACL to block access to a > resource: > > if (req.http.host == "test.mydomain.com") { > if (client.ip ~ trustedips) { > # allow access > } else { > return (synth(405, "Not allowed"); > } > } > > But this doesn't work if I replace client.ip with a non-IP typed field. > > Message from VCC-compiler: > Expected CSTR got 'purgers' > (program line 1193), at > ('default.vcl' Line 339 Pos 34) > if (req.http.X-Real-Ip ~ trustedips) { > ---------------------------------##########--- > > Is there any way I can get the same result as this but without using > client.ip? > > thanks, > > Mark > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From colas.delmas at gmail.com Fri Mar 31 10:01:01 2017 From: colas.delmas at gmail.com (Colas Delmas) Date: Fri, 31 Mar 2017 12:01:01 +0200 Subject: Varnish HIT/MISS Web Statistics Question In-Reply-To: References: Message-ID: Hey, As Andrei said, you could use Google Analytics and it works fine. I'm using this method for my own website. I wrote an article about, but it's in french. I still share this link https://tutoandco.colas-delmas.fr/software/varnish/statistiques-varnish-hit-vs-miss-google-analytics/ There the english version : https://tutoandco.colas-delmas.fr/en/software-en/varnish-en/statistics-varnish-hit-vs-miss-google-analytics/ (it's a "private" page since the site is being translated) *Nicolas Delmas* http://tutoandco.colas-delmas.fr/ 2017-03-29 5:43 GMT+02:00 Devin Acosta : > > I am trying to get to where I can launch Varnish Cache in my environment. > One of the challenges I guess that I am trying to figure out is that > currently if a request is a HIT it never logs to the backend server the > requests that it processed, therefore it messes up my Web Statistics. I see > that I can use "varnishncsa" which will cause it log onto a file on the > local machine that Varnish is running on, however is there a cleaner way to > get my web statistics so that it's accurate, other than trying to pull logs > from both the backend server and the varnish server and combine them > together? > > -- > > Devin Acosta > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Fri Mar 31 10:05:49 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 31 Mar 2017 12:05:49 +0200 Subject: Using ACL with non-IP fields In-Reply-To: References: Message-ID: > Is there any way I can get the same result as this but without using > client.ip? Yes: std.ip(string, ip_fallback) if (std.ip(req.http.X-Real-Ip, "some_address") ~ trustedips) { See man vmod_std, or search the online docs. Dridi From colas.delmas at gmail.com Fri Mar 31 10:07:14 2017 From: colas.delmas at gmail.com (Colas Delmas) Date: Fri, 31 Mar 2017 12:07:14 +0200 Subject: Looking for Varnish Consultant? In-Reply-To: References: Message-ID: Hy, You could take a look of my varnish conf I use on my website : https://github.com/colas31/varnish/tree/master/v4.1 I use Varnish 4.1 + Apache *Nicolas Delmas* http://tutoandco.colas-delmas.fr/ 2017-03-31 0:17 GMT+02:00 Devin Acosta : > > This may not be the appropriate place but hoping it's OK, we are trying to > get Varnish configured on our network to cache websites that are mostly > Wordpress related. We are looking for it to provide faster page loads, with > loading from cache if the backend isn't responding. We are looking for > someone who wants to help us out for some hours and help us ensure we have > a valid working config. > > Please e-mail me if your interested. > > -- > > Devin Acosta > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at hanfordonline.co.uk Fri Mar 31 10:14:04 2017 From: mark at hanfordonline.co.uk (Mark Hanford) Date: Fri, 31 Mar 2017 11:14:04 +0100 Subject: Using ACL with non-IP fields In-Reply-To: References: Message-ID: Yeah, I'm looking into PROXY. We have up to two proxies in the way, the first that all traffic has to go through is a Cisco Ace we use for load balancing and SSL offloading, and the second is a CaddyServer that some traffic will go through for LetsEncrypt certificates. It's getting both of those setup to present similar data to the Varnish nodes that's the trick - we don't have direct control of the Cisco gear either. Both should be able to do it, I guess I just need to work out the wrinkles. -- Mark On 31 March 2017 at 10:59, Guillaume Quintard < guillaume at varnish-software.com> wrote: > Hi Mark, you need to use std.ip() from the std vmod. > > Also, I highly recommend using the PROXY protocol (Varnish, HAProxy and > Nginx, at least support it), it will allow you to use client.ip directly > and not bother with this. > > -- > Guillaume Quintard > > On Fri, Mar 31, 2017 at 11:44 AM, Mark Hanford > wrote: > >> Hi folks. >> >> Because my varnish nodes are behind two different proxies, I can't really >> use client.ip within my VCL. What I have is a header "X-Real-Ip" instead, >> which is populated automatically by one proxy, and by me derived from the >> "X-Forwarded-For" for the other. >> >> What this means is that where I would usually use ACL to block access to >> a resource: >> >> if (req.http.host == "test.mydomain.com") { >> if (client.ip ~ trustedips) { >> # allow access >> } else { >> return (synth(405, "Not allowed"); >> } >> } >> >> But this doesn't work if I replace client.ip with a non-IP typed field. >> >> Message from VCC-compiler: >> Expected CSTR got 'purgers' >> (program line 1193), at >> ('default.vcl' Line 339 Pos 34) >> if (req.http.X-Real-Ip ~ trustedips) { >> ---------------------------------##########--- >> >> Is there any way I can get the same result as this but without using >> client.ip? >> >> thanks, >> >> Mark >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at hanfordonline.co.uk Fri Mar 31 10:14:46 2017 From: mark at hanfordonline.co.uk (Mark Hanford) Date: Fri, 31 Mar 2017 11:14:46 +0100 Subject: Using ACL with non-IP fields In-Reply-To: References: Message-ID: Aah, thanks for that. I thought I'd looked through std, but must've missed that bit. Still getting the hang of the post-v3 way of doing things :) thanks -- Mark On 31 March 2017 at 11:05, Dridi Boukelmoune wrote: > > Is there any way I can get the same result as this but without using > > client.ip? > > Yes: std.ip(string, ip_fallback) > > if (std.ip(req.http.X-Real-Ip, "some_address") ~ trustedips) { > > See man vmod_std, or search the online docs. > > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hazarguney at gmail.com Fri Mar 31 11:58:48 2017 From: hazarguney at gmail.com (=?UTF-8?B?SGF6YXIgR8O8bmV5?=) Date: Fri, 31 Mar 2017 14:58:48 +0300 Subject: Using ACL with non-IP fields In-Reply-To: References: Message-ID: >From one of my production servers: import std; sub vcl_deliver { if (std.ip(req.http.RIP,"0.0.0.0") ~ hitpass) { .... } } "RIP" is the client ip value in header. On Fri, Mar 31, 2017 at 1:05 PM, Dridi Boukelmoune wrote: > > Is there any way I can get the same result as this but without using > > client.ip? > > Yes: std.ip(string, ip_fallback) > > if (std.ip(req.http.X-Real-Ip, "some_address") ~ trustedips) { > > See man vmod_std, or search the online docs. > > Dridi > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hazarguney at gmail.com Fri Mar 31 12:25:40 2017 From: hazarguney at gmail.com (=?UTF-8?B?SGF6YXIgR8O8bmV5?=) Date: Fri, 31 Mar 2017 15:25:40 +0300 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: References: <01F55BD6-0E30-444D-9F7D-470DDA52F329@nucleus.be> Message-ID: Any idea? On Thu, Mar 30, 2017 at 3:41 PM, Hazar G?ney wrote: > It did not work either: > > * << BeReq >> 127418176 > - Begin bereq 127418175 fetch > - Timestamp Start: 1490877149.450124 0.000000 0.000000 > - BereqMethod GET > - BereqURL XXXX > - BereqProtocol HTTP/1.1 > - BereqHeader Accept: text/css,*/*;q=0.1 > - BereqHeader User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 > like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0 > Mobile/14C92 Safari/602.1 > - BereqHeader Accept-Language: tr-tr > - BereqHeader Referer: XXXX > - BereqHeader Host: XXXX > - BereqHeader RIP: XXXX > - BereqHeader X-Forwarded-For: XXXX > - BereqHeader Accept-Encoding: gzip > - BereqHeader X-Varnish: 127418176 > - VCL_call BACKEND_FETCH > - BereqHeader connection: Close > - VCL_return fetch > - BackendOpen 25 reload_2017-03-30T14:53:46.st2 10.35.78.11 80 > 172.17.0.2 59152 > - BackendStart 10.35.78.11 80 > - Timestamp Bereq: 1490877149.450594 0.000470 0.000470 > - FetchError http first read error: EOF > - BackendClose 25 reload_2017-03-30T14:53:46.st2 > - Timestamp Beresp: 1490877149.451184 0.001060 0.000590 > - Timestamp Error: 1490877149.451189 0.001065 0.000005 > - BerespProtocol HTTP/1.1 > - BerespStatus 503 > - BerespReason Service Unavailable > - BerespReason Backend fetch failed > - BerespHeader Date: Thu, 30 Mar 2017 12:32:29 GMT > - BerespHeader Server: Varnish > - VCL_call BACKEND_ERROR > - BereqHeader X-Varnish-Backend-5xx: 1 > - VCL_return retry > - Timestamp Retry: 1490877149.451205 0.001081 0.000016 > - Link bereq 127298071 retry > - End > > On Thu, Mar 30, 2017 at 2:34 PM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> It does, I'm suspecting that the connection reuse is creating some >> issues, probably because Apache is doing some non-standard stuff (protip: >> always blame Apache). >> >> -- >> Guillaume Quintard >> >> On Thu, Mar 30, 2017 at 1:17 PM, Hazar G?ney >> wrote: >> >>> "Connection: close" supersedes keep-alive behavior, is that correct? >>> >>> On Thu, Mar 30, 2017 at 2:08 PM, Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> Can you try something: add 'set bereq.http.connection = "Close"; ' at >>>> the beginning of vcl_backend_fetch and see if that helps? >>>> >>>> -- >>>> Guillaume Quintard >>>> >>>> On Thu, Mar 30, 2017 at 1:04 PM, Hazar G?ney >>>> wrote: >>>> >>>>> MaxKeepAliveRequests 20 >>>>> KeepAliveTimeout 2 >>>>> >>>>> Version is "4.1.3 revision 5e3b6d2". We have also seen "straight >>>>> insufficient bytes" error with POST requests to a specific php script >>>>> hosted by another backend and fixed it by using "pipe" instead of "pass" >>>>> but this specific backend gives "http first read error: EOF" error. Another >>>>> example from today: >>>>> >>>>> * << BeReq >> 126635444 >>>>> - Begin bereq 126635443 fetch >>>>> - Timestamp Start: 1490870598.921499 0.000000 0.000000 >>>>> - BereqMethod GET >>>>> - BereqURL XXXX >>>>> - BereqProtocol HTTP/1.1 >>>>> - BereqHeader Host: XXXX >>>>> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; >>>>> x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 >>>>> Safari/537.36 >>>>> - BereqHeader Accept: image/webp,image/*,*/*;q=0.8 >>>>> - BereqHeader Referer: XXXX >>>>> - BereqHeader Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.6,en; >>>>> q=0.4 >>>>> - BereqHeader RIP: XXXX >>>>> - BereqHeader X-Forwarded-For: XXXX >>>>> - BereqHeader Accept-Encoding: gzip >>>>> - BereqHeader X-Varnish: 126635444 >>>>> - VCL_call BACKEND_FETCH >>>>> - VCL_return fetch >>>>> - BackendOpen 35 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 >>>>> 172.17.0.2 48896 >>>>> - BackendStart 10.35.78.11 80 >>>>> - Timestamp Bereq: 1490870598.922050 0.000552 0.000552 >>>>> *- FetchError http first read error: EOF* >>>>> - BackendClose 35 reload_2017-03-20T11:32:44.st2 >>>>> - Timestamp Beresp: 1490870598.922622 0.001124 0.000572 >>>>> - Timestamp Error: 1490870598.922627 0.001129 0.000005 >>>>> - BerespProtocol HTTP/1.1 >>>>> - BerespStatus 503 >>>>> - BerespReason Service Unavailable >>>>> - BerespReason Backend fetch failed >>>>> - BerespHeader Date: Thu, 30 Mar 2017 10:43:18 GMT >>>>> - BerespHeader Server: Varnish >>>>> - VCL_call BACKEND_ERROR >>>>> - BereqHeader X-Varnish-Backend-5xx: 1 >>>>> - VCL_return retry >>>>> - Timestamp Retry: 1490870598.922657 0.001159 0.000030 >>>>> - Link bereq 126832283 retry >>>>> - End >>>>> >>>>> On Wed, Mar 29, 2017 at 12:03 PM, Mattias Geniar >>>>> wrote: >>>>> >>>>>> > Backend is Apache. >>>>>> >>>>>> In older Varnish versions, you could sometimes see a similar error; >>>>>> >>>>>> > 11 FetchError c straight insufficient bytes >>>>>> >>>>>> The error message you?re seeing might be related, as it mentions the >>>>>> EOF. >>>>>> >>>>>> This happens when the backend sends a Content-Length header that >>>>>> doesn?t match the _actual_ content length it?s sending. In Apache, this was >>>>>> commonly caused by a mod_deflate misconfiguration. >>>>>> >>>>>> For testing, could you try disabling Gzip either in your backend or >>>>>> strip the Accept-Encoding header in Varnish to force a plain text response? >>>>>> >>>>>> Mattias >>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hazarguney at gmail.com Fri Mar 31 12:31:58 2017 From: hazarguney at gmail.com (=?UTF-8?B?SGF6YXIgR8O8bmV5?=) Date: Fri, 31 Mar 2017 15:31:58 +0300 Subject: "http first read error: EOF" errors from WordPress backend In-Reply-To: <9FC546D0-BB08-49F5-93B8-7A3C9F5A49E7@nucleus.be> References: <9FC546D0-BB08-49F5-93B8-7A3C9F5A49E7@nucleus.be> Message-ID: But "Content-Length" header is not available at all. I'm afraid we cannot keep gzip disabled even if it solves the issue, Varnish has to be able to handle gzipped inputs from the backend. On Fri, Mar 31, 2017 at 11:11 AM, Mattias Geniar wrote: > > - FetchError http first read error: EOF > > - BackendClose 21 reload_2017-03-21T100643.default > > - Timestamp Beresp: 1490916389.664967 60.000557 60.000114 > > - Timestamp Error: 1490916389.664978 60.000567 0.000011 > > At the risk of repeating myself: try to disable gzip & any wordpress > plugins that might be trying to gzip on their own (aka: output buffering in > PHP). > > To me, this seems like Varnish is waiting for the backend to send more > data, because it replied with a certain Content-Length header but sent a > few bytes less than it advertised, and Varnish is waiting for those missing > bytes. > > Mattias > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Fri Mar 31 13:17:56 2017 From: lagged at gmail.com (Andrei) Date: Fri, 31 Mar 2017 08:17:56 -0500 Subject: =?UTF-8?Q?Re=3A_Random_=E2=80=9Chttp_first_read_error=3A_EOF=E2=80=9D_errors?= In-Reply-To: References: <01F55BD6-0E30-444D-9F7D-470DDA52F329@nucleus.be> Message-ID: Can you provide a tcpdump/ngrep of the requests between Client/Varnish/Apache along with the varnishlog entry to see if that uncovers anything? On Fri, Mar 31, 2017 at 7:25 AM, Hazar G?ney wrote: > Any idea? > > On Thu, Mar 30, 2017 at 3:41 PM, Hazar G?ney wrote: > >> It did not work either: >> >> * << BeReq >> 127418176 >> - Begin bereq 127418175 fetch >> - Timestamp Start: 1490877149.450124 0.000000 0.000000 >> - BereqMethod GET >> - BereqURL XXXX >> - BereqProtocol HTTP/1.1 >> - BereqHeader Accept: text/css,*/*;q=0.1 >> - BereqHeader User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS >> 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0 >> Mobile/14C92 Safari/602.1 >> - BereqHeader Accept-Language: tr-tr >> - BereqHeader Referer: XXXX >> - BereqHeader Host: XXXX >> - BereqHeader RIP: XXXX >> - BereqHeader X-Forwarded-For: XXXX >> - BereqHeader Accept-Encoding: gzip >> - BereqHeader X-Varnish: 127418176 >> - VCL_call BACKEND_FETCH >> - BereqHeader connection: Close >> - VCL_return fetch >> - BackendOpen 25 reload_2017-03-30T14:53:46.st2 10.35.78.11 80 >> 172.17.0.2 59152 >> - BackendStart 10.35.78.11 80 >> - Timestamp Bereq: 1490877149.450594 0.000470 0.000470 >> - FetchError http first read error: EOF >> - BackendClose 25 reload_2017-03-30T14:53:46.st2 >> - Timestamp Beresp: 1490877149.451184 0.001060 0.000590 >> - Timestamp Error: 1490877149.451189 0.001065 0.000005 >> - BerespProtocol HTTP/1.1 >> - BerespStatus 503 >> - BerespReason Service Unavailable >> - BerespReason Backend fetch failed >> - BerespHeader Date: Thu, 30 Mar 2017 12:32:29 GMT >> - BerespHeader Server: Varnish >> - VCL_call BACKEND_ERROR >> - BereqHeader X-Varnish-Backend-5xx: 1 >> - VCL_return retry >> - Timestamp Retry: 1490877149.451205 0.001081 0.000016 >> - Link bereq 127298071 retry >> - End >> >> On Thu, Mar 30, 2017 at 2:34 PM, Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> It does, I'm suspecting that the connection reuse is creating some >>> issues, probably because Apache is doing some non-standard stuff (protip: >>> always blame Apache). >>> >>> -- >>> Guillaume Quintard >>> >>> On Thu, Mar 30, 2017 at 1:17 PM, Hazar G?ney >>> wrote: >>> >>>> "Connection: close" supersedes keep-alive behavior, is that correct? >>>> >>>> On Thu, Mar 30, 2017 at 2:08 PM, Guillaume Quintard < >>>> guillaume at varnish-software.com> wrote: >>>> >>>>> Can you try something: add 'set bereq.http.connection = "Close"; ' at >>>>> the beginning of vcl_backend_fetch and see if that helps? >>>>> >>>>> -- >>>>> Guillaume Quintard >>>>> >>>>> On Thu, Mar 30, 2017 at 1:04 PM, Hazar G?ney >>>>> wrote: >>>>> >>>>>> MaxKeepAliveRequests 20 >>>>>> KeepAliveTimeout 2 >>>>>> >>>>>> Version is "4.1.3 revision 5e3b6d2". We have also seen "straight >>>>>> insufficient bytes" error with POST requests to a specific php script >>>>>> hosted by another backend and fixed it by using "pipe" instead of "pass" >>>>>> but this specific backend gives "http first read error: EOF" error. Another >>>>>> example from today: >>>>>> >>>>>> * << BeReq >> 126635444 >>>>>> - Begin bereq 126635443 fetch >>>>>> - Timestamp Start: 1490870598.921499 0.000000 0.000000 >>>>>> - BereqMethod GET >>>>>> - BereqURL XXXX >>>>>> - BereqProtocol HTTP/1.1 >>>>>> - BereqHeader Host: XXXX >>>>>> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; >>>>>> x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 >>>>>> Safari/537.36 >>>>>> - BereqHeader Accept: image/webp,image/*,*/*;q=0.8 >>>>>> - BereqHeader Referer: XXXX >>>>>> - BereqHeader Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.6,en; >>>>>> q=0.4 >>>>>> - BereqHeader RIP: XXXX >>>>>> - BereqHeader X-Forwarded-For: XXXX >>>>>> - BereqHeader Accept-Encoding: gzip >>>>>> - BereqHeader X-Varnish: 126635444 >>>>>> - VCL_call BACKEND_FETCH >>>>>> - VCL_return fetch >>>>>> - BackendOpen 35 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 >>>>>> 172.17.0.2 48896 >>>>>> - BackendStart 10.35.78.11 80 >>>>>> - Timestamp Bereq: 1490870598.922050 0.000552 0.000552 >>>>>> *- FetchError http first read error: EOF* >>>>>> - BackendClose 35 reload_2017-03-20T11:32:44.st2 >>>>>> - Timestamp Beresp: 1490870598.922622 0.001124 0.000572 >>>>>> - Timestamp Error: 1490870598.922627 0.001129 0.000005 >>>>>> - BerespProtocol HTTP/1.1 >>>>>> - BerespStatus 503 >>>>>> - BerespReason Service Unavailable >>>>>> - BerespReason Backend fetch failed >>>>>> - BerespHeader Date: Thu, 30 Mar 2017 10:43:18 GMT >>>>>> - BerespHeader Server: Varnish >>>>>> - VCL_call BACKEND_ERROR >>>>>> - BereqHeader X-Varnish-Backend-5xx: 1 >>>>>> - VCL_return retry >>>>>> - Timestamp Retry: 1490870598.922657 0.001159 0.000030 >>>>>> - Link bereq 126832283 retry >>>>>> - End >>>>>> >>>>>> On Wed, Mar 29, 2017 at 12:03 PM, Mattias Geniar >>>>>> wrote: >>>>>> >>>>>>> > Backend is Apache. >>>>>>> >>>>>>> In older Varnish versions, you could sometimes see a similar error; >>>>>>> >>>>>>> > 11 FetchError c straight insufficient bytes >>>>>>> >>>>>>> The error message you?re seeing might be related, as it mentions the >>>>>>> EOF. >>>>>>> >>>>>>> This happens when the backend sends a Content-Length header that >>>>>>> doesn?t match the _actual_ content length it?s sending. In Apache, this was >>>>>>> commonly caused by a mod_deflate misconfiguration. >>>>>>> >>>>>>> For testing, could you try disabling Gzip either in your backend or >>>>>>> strip the Accept-Encoding header in Varnish to force a plain text response? >>>>>>> >>>>>>> Mattias >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: