From rednasy at gmail.com Thu Jul 1 20:54:39 2010 From: rednasy at gmail.com (Andrey Nepomnyaschih) Date: Thu, 1 Jul 2010 15:54:39 -0500 Subject: Accept-Language Issue Message-ID: Hello, Can somebody give me a hint for what am I doing wrong with my setup. I have Zope instance that is proxied by nginx (doing address rewriting) that I want to be cached by varnish, but as soon as I configure Zope to add Accept-Language to Vary header, varnish starts closing connection when asked for object from Zope without returning any data. I'm using default.vcl Using tcpdump doesn't reveal anything useful, the only difference is Accept-Language value in Vary header. Request that works: # GET -H "Host: hostname.domain.tld" -Used http://hostname.domain.tld GET http://hostname.domain.tld Host: hostname.domain.tld User-Agent: lwp-request/5.834 libwww-perl/5.836 200 OK Wide character in print at /usr/local/bin/GET line 439. Cache-Control: max-age=0, s-maxage=0, private, must-revalidate Connection: close Date: Thu, 01 Jul 2010 20:30:22 GMT Via: 1.1 varnish Age: 0 ETag: ||Plone Default||0|181||||355004 Server: nginx/0.7.67 Vary: User-Agent Content-Language: en Content-Length: 5485 Content-Type: text/html;charset=utf-8 Expires: Mon, 03 Jul 2000 20:30:22 GMT Client-Date: Thu, 01 Jul 2010 20:30:22 GMT Client-Peer: 82.103.128.123:80 Client-Response-Num: 1 Content-Base: http://hostname.domain.tld/plonesite/front-page Imagetoolbar: no Link: ; rel="kss-base-url" Link: < http://hostname.domain.tld/plonesite/portal_kss/Plone%20Default/at-cachekey6690.kss>; rel="kinetic-stylesheet"; type="text/css" Link: ; rel="shortcut icon"; type="image/x-icon" Link: ; rel="home"; title="Front page" Link: ; rel="contents"; title="Site Map" Link: ; rel="search"; title="Search this site" Title: Plone Site! X-Cache-Headers-Set-By: CachingPolicyManager: /plonesite/caching_policy_manager X-Caching-Rule-Id: plone-content-types X-Header-Set-Id: cache-in-memory X-Meta-Generator: Plone - http://plone.org X-Pagecache: MISS X-UA-Compatible: IE=edge X-Varnish: 1366563744 Results in the following request to nginx: 2010/07/01 22:30:22.196300 127.0.0.1:25266 -> 127.0.0.1:8080 [AP] GET / HTTP/1.1. Host: hostname.domain.tld. User-Agent: lwp-request/5.834 libwww-perl/5.836. X-Forwarded-For: 82.103.128.123. X-Varnish: 1366563744. And the following reply from nginx: T 2010/07/01 22:30:22.267204 127.0.0.1:8080 -> 127.0.0.1:25266 [AP] HTTP/1.1 200 OK. Server: nginx/0.7.67. Date: Thu, 01 Jul 2010 20:30:22 GMT. Content-Type: text/html;charset=utf-8. Connection: keep-alive. X-Pagecache: MISS. Content-Length: 5485. Content-Language: en. X-Cache-Headers-Set-By: CachingPolicyManager: /plonesite/caching_policy_manager. Expires: Mon, 03 Jul 2000 20:30:22 GMT. Vary: User-Agent. ETag: ||Plone Default||0|181||||355004. X-Caching-Rule-Id: plone-content-types. Cache-Control: max-age=0, s-maxage=0, private, must-revalidate. X-Header-Set-Id: cache-in-memory. Request that doesn't work: [root at rain /usr/local/etc/varnish]# GET -H "Host: hostname.domain.tld" -Used http://hostname.domain.tld GET http://hostname.domain.tld Host: hostname.domain.tld User-Agent: lwp-request/5.834 libwww-perl/5.836 500 Server closed connection without sending any data back Content-Type: text/plain Client-Date: Thu, 01 Jul 2010 20:27:54 GMT Client-Warning: Internal response Results in the following request to nginx: 2010/07/01 22:27:54.507294 127.0.0.1:50795 -> 127.0.0.1:8080 [AP] GET / HTTP/1.1. Host: hostname.domain.tld. User-Agent: lwp-request/5.834 libwww-perl/5.836. X-Forwarded-For: 82.103.128.123. X-Varnish: 1656652273. And the following reply from nginx: 2010/07/01 22:27:54.573671 127.0.0.1:8080 -> 127.0.0.1:50795 [AP] HTTP/1.1 200 OK. Server: nginx/0.7.67. Date: Thu, 01 Jul 2010 20:27:54 GMT. Content-Type: text/html;charset=utf-8. Connection: keep-alive. X-Pagecache: MISS. Content-Length: 5485. Content-Language: en. X-Cache-Headers-Set-By: CachingPolicyManager: /plonesite/caching_policy_manager. Expires: Mon, 03 Jul 2000 20:27:54 GMT. Vary: User-Agent Accept-Language. ETag: ||Plone Default||0|180||||355004. X-Caching-Rule-Id: plone-content-types. Cache-Control: max-age=0, s-maxage=0, private, must-revalidate. X-Header-Set-Id: cache-in-memory. Kind Regards, Andrey Nepomnyaschih -------------- next part -------------- An HTML attachment was scrubbed... URL: From kb+varnish at slide.com Thu Jul 1 21:04:35 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Thu, 1 Jul 2010 14:04:35 -0700 Subject: Accept-Language Issue In-Reply-To: References: Message-ID: I do notice that the Vary header is malformed: Vary: User-Agent Accept-Language It should be comma-separated AFAIK. Also, all of the lines from your last header dump have appended periods... not sure that's the actual output or just a paste glitch. Perhaps Varnish is being very strict about malformed headers and dumping the connection immediately. Which is probably a good thing, if true. -- Ken On Jul 1, 2010, at 1:54 PM, Andrey Nepomnyaschih wrote: > Hello, > > Can somebody give me a hint for what am I doing wrong with my setup. I have Zope instance that is proxied by nginx (doing address rewriting) that I want to be cached by varnish, but as soon as I configure Zope to add Accept-Language to Vary header, varnish starts closing connection when asked for object from Zope without returning any data. I'm using default.vcl > > Using tcpdump doesn't reveal anything useful, the only difference is Accept-Language value in Vary header. > > Request that works: > > # GET -H "Host: hostname.domain.tld" -Used http://hostname.domain.tld > GET http://hostname.domain.tld > Host: hostname.domain.tld > User-Agent: lwp-request/5.834 libwww-perl/5.836 > > 200 OK > Wide character in print at /usr/local/bin/GET line 439. > Cache-Control: max-age=0, s-maxage=0, private, must-revalidate > Connection: close > Date: Thu, 01 Jul 2010 20:30:22 GMT > Via: 1.1 varnish > Age: 0 > ETag: ||Plone Default||0|181||||355004 > Server: nginx/0.7.67 > Vary: User-Agent > Content-Language: en > Content-Length: 5485 > Content-Type: text/html;charset=utf-8 > Expires: Mon, 03 Jul 2000 20:30:22 GMT > Client-Date: Thu, 01 Jul 2010 20:30:22 GMT > Client-Peer: 82.103.128.123:80 > Client-Response-Num: 1 > Content-Base: http://hostname.domain.tld/plonesite/front-page > Imagetoolbar: no > Link: ; rel="kss-base-url" > Link: ; rel="kinetic-stylesheet"; type="text/css" > Link: ; rel="shortcut icon"; type="image/x-icon" > Link: ; rel="home"; title="Front page" > Link: ; rel="contents"; title="Site Map" > Link: ; rel="search"; title="Search this site" > Title: Plone Site! > X-Cache-Headers-Set-By: CachingPolicyManager: /plonesite/caching_policy_manager > X-Caching-Rule-Id: plone-content-types > X-Header-Set-Id: cache-in-memory > X-Meta-Generator: Plone - http://plone.org > X-Pagecache: MISS > X-UA-Compatible: IE=edge > X-Varnish: 1366563744 > > Results in the following request to nginx: > > 2010/07/01 22:30:22.196300 127.0.0.1:25266 -> 127.0.0.1:8080 [AP] > GET / HTTP/1.1. > Host: hostname.domain.tld. > User-Agent: lwp-request/5.834 libwww-perl/5.836. > X-Forwarded-For: 82.103.128.123. > X-Varnish: 1366563744. > > And the following reply from nginx: > > T 2010/07/01 22:30:22.267204 127.0.0.1:8080 -> 127.0.0.1:25266 [AP] > HTTP/1.1 200 OK. > Server: nginx/0.7.67. > Date: Thu, 01 Jul 2010 20:30:22 GMT. > Content-Type: text/html;charset=utf-8. > Connection: keep-alive. > X-Pagecache: MISS. > Content-Length: 5485. > Content-Language: en. > X-Cache-Headers-Set-By: CachingPolicyManager: /plonesite/caching_policy_manager. > Expires: Mon, 03 Jul 2000 20:30:22 GMT. > Vary: User-Agent. > ETag: ||Plone Default||0|181||||355004. > X-Caching-Rule-Id: plone-content-types. > Cache-Control: max-age=0, s-maxage=0, private, must-revalidate. > X-Header-Set-Id: cache-in-memory. > > Request that doesn't work: > > [root at rain /usr/local/etc/varnish]# GET -H "Host: hostname.domain.tld" -Used http://hostname.domain.tld > GET http://hostname.domain.tld > Host: hostname.domain.tld > User-Agent: lwp-request/5.834 libwww-perl/5.836 > > 500 Server closed connection without sending any data back > Content-Type: text/plain > Client-Date: Thu, 01 Jul 2010 20:27:54 GMT > Client-Warning: Internal response > > Results in the following request to nginx: > > 2010/07/01 22:27:54.507294 127.0.0.1:50795 -> 127.0.0.1:8080 [AP] > GET / HTTP/1.1. > Host: hostname.domain.tld. > User-Agent: lwp-request/5.834 libwww-perl/5.836. > X-Forwarded-For: 82.103.128.123. > X-Varnish: 1656652273. > > And the following reply from nginx: > > 2010/07/01 22:27:54.573671 127.0.0.1:8080 -> 127.0.0.1:50795 [AP] > HTTP/1.1 200 OK. > Server: nginx/0.7.67. > Date: Thu, 01 Jul 2010 20:27:54 GMT. > Content-Type: text/html;charset=utf-8. > Connection: keep-alive. > X-Pagecache: MISS. > Content-Length: 5485. > Content-Language: en. > X-Cache-Headers-Set-By: CachingPolicyManager: /plonesite/caching_policy_manager. > Expires: Mon, 03 Jul 2000 20:27:54 GMT. > Vary: User-Agent Accept-Language. > ETag: ||Plone Default||0|180||||355004. > X-Caching-Rule-Id: plone-content-types. > Cache-Control: max-age=0, s-maxage=0, private, must-revalidate. > X-Header-Set-Id: cache-in-memory. > > Kind Regards, > Andrey Nepomnyaschih > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From l at lrowe.co.uk Thu Jul 1 22:37:41 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Thu, 1 Jul 2010 23:37:41 +0100 Subject: Accept-Language Issue In-Reply-To: References: Message-ID: Check that you do have the comma in Plone's caching control panel. I can imagine it being an easy typo to make. There are many unique variations of Accept-Language headers, so be careful. I've used this before in vcl_recv to normalise the incoming request: # We only care about the language in the I18N_LANGUAGE cookie if (req.http.Cookie ~ "(^|.*; )I18N_LANGUAGE=") { set req.http.Accept-Language = regsub(req.http.Cookie, "(^|.*; )I18N_LANGUAGE=([^;]*)(; .*|$)", "\2"); # XXX need to work out the proper way to match " here, e.g. "en" set req.http.Accept-Language = regsub(req.http.Accept-Language, "^.(.*).$", "\1"); } else { set req.http.Accept-Language = "en"; } Laurence On 1 July 2010 22:04, Ken Brownfield > wrote: > I do notice that the Vary header is malformed: > > Vary: User-Agent Accept-Language > > It should be comma-separated AFAIK. Also, all of the lines from your last > header dump have appended periods... not sure that's the actual output or > just a paste glitch. > > Perhaps Varnish is being very strict about malformed headers and dumping > the connection immediately. Which is probably a good thing, if true. > -- > Ken > > On Jul 1, 2010, at 1:54 PM, Andrey Nepomnyaschih wrote: > > Hello, > > Can somebody give me a hint for what am I doing wrong with my setup. I have > Zope instance that is proxied by nginx (doing address rewriting) that I want > to be cached by varnish, but as soon as I configure Zope to add > Accept-Language to Vary header, varnish starts closing connection when asked > for object from Zope without returning any data. I'm using default.vcl > > Using tcpdump doesn't reveal anything useful, the only difference is > Accept-Language value in Vary header. > > Request that works: > > # GET -H "Host: hostname.domain.tld" -Used http://hostname.domain.tld > GET http://hostname.domain.tld > Host: hostname.domain.tld > User-Agent: lwp-request/5.834 libwww-perl/5.836 > > 200 OK > Wide character in print at /usr/local/bin/GET line 439. > Cache-Control: max-age=0, s-maxage=0, private, must-revalidate > Connection: close > Date: Thu, 01 Jul 2010 20:30:22 GMT > Via: 1.1 varnish > Age: 0 > ETag: ||Plone Default||0|181||||355004 > Server: nginx/0.7.67 > Vary: User-Agent > Content-Language: en > Content-Length: 5485 > Content-Type: text/html;charset=utf-8 > Expires: Mon, 03 Jul 2000 20:30:22 GMT > Client-Date: Thu, 01 Jul 2010 20:30:22 GMT > Client-Peer: 82.103.128.123:80 > Client-Response-Num: 1 > Content-Base: http://hostname.domain.tld/plonesite/front-page > Imagetoolbar: no > Link: ; > rel="kss-base-url" > Link: < > http://hostname.domain.tld/plonesite/portal_kss/Plone%20Default/at-cachekey6690.kss>; > rel="kinetic-stylesheet"; type="text/css" > Link: ; rel="shortcut > icon"; type="image/x-icon" > Link: ; rel="home"; title="Front > page" > Link: ; rel="contents"; > title="Site Map" > Link: ; rel="search"; > title="Search this site" > Title: Plone Site! > X-Cache-Headers-Set-By: CachingPolicyManager: > /plonesite/caching_policy_manager > X-Caching-Rule-Id: plone-content-types > X-Header-Set-Id: cache-in-memory > X-Meta-Generator: Plone - http://plone.org > X-Pagecache: MISS > X-UA-Compatible: IE=edge > X-Varnish: 1366563744 > > Results in the following request to nginx: > > 2010/07/01 22:30:22.196300 127.0.0.1:25266 -> 127.0.0.1:8080 [AP] > GET / HTTP/1.1. > Host: hostname.domain.tld. > User-Agent: lwp-request/5.834 libwww-perl/5.836. > X-Forwarded-For: 82.103.128.123. > X-Varnish: 1366563744. > > And the following reply from nginx: > > T 2010/07/01 22:30:22.267204 127.0.0.1:8080 -> 127.0.0.1:25266 [AP] > HTTP/1.1 200 OK. > Server: nginx/0.7.67. > Date: Thu, 01 Jul 2010 20:30:22 GMT. > Content-Type: text/html;charset=utf-8. > Connection: keep-alive. > X-Pagecache: MISS. > Content-Length: 5485. > Content-Language: en. > X-Cache-Headers-Set-By: CachingPolicyManager: > /plonesite/caching_policy_manager. > Expires: Mon, 03 Jul 2000 20:30:22 GMT. > Vary: User-Agent. > ETag: ||Plone Default||0|181||||355004. > X-Caching-Rule-Id: plone-content-types. > Cache-Control: max-age=0, s-maxage=0, private, must-revalidate. > X-Header-Set-Id: cache-in-memory. > > Request that doesn't work: > > [root at rain /usr/local/etc/varnish]# GET -H "Host: hostname.domain.tld" > -Used http://hostname.domain.tld > GET http://hostname.domain.tld > Host: hostname.domain.tld > User-Agent: lwp-request/5.834 libwww-perl/5.836 > > 500 Server closed connection without sending any data back > Content-Type: text/plain > Client-Date: Thu, 01 Jul 2010 20:27:54 GMT > Client-Warning: Internal response > > Results in the following request to nginx: > > 2010/07/01 22:27:54.507294 127.0.0.1:50795 -> 127.0.0.1:8080 [AP] > GET / HTTP/1.1. > Host: hostname.domain.tld. > User-Agent: lwp-request/5.834 libwww-perl/5.836. > X-Forwarded-For: 82.103.128.123. > X-Varnish: 1656652273. > > And the following reply from nginx: > > 2010/07/01 22:27:54.573671 127.0.0.1:8080 -> 127.0.0.1:50795 [AP] > HTTP/1.1 200 OK. > Server: nginx/0.7.67. > Date: Thu, 01 Jul 2010 20:27:54 GMT. > Content-Type: text/html;charset=utf-8. > Connection: keep-alive. > X-Pagecache: MISS. > Content-Length: 5485. > Content-Language: en. > X-Cache-Headers-Set-By: CachingPolicyManager: > /plonesite/caching_policy_manager. > Expires: Mon, 03 Jul 2000 20:27:54 GMT. > Vary: User-Agent Accept-Language. > ETag: ||Plone Default||0|180||||355004. > X-Caching-Rule-Id: plone-content-types. > Cache-Control: max-age=0, s-maxage=0, private, must-revalidate. > X-Header-Set-Id: cache-in-memory. > > Kind Regards, > Andrey Nepomnyaschih > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cosimo at streppone.it Fri Jul 2 05:59:32 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Fri, 02 Jul 2010 07:59:32 +0200 Subject: Accept-Language Issue In-Reply-To: References: Message-ID: In data 02 luglio 2010 alle ore 00:37:41, Laurence Rowe ha scritto: > Check that you do have the comma in Plone's caching control panel. I can > imagine it being an easy typo to make. > > There are many unique variations of Accept-Language headers, so be > careful. > I've used this before in vcl_recv to normalise the incoming request: > > # We only care about the language in the I18N_LANGUAGE cookie > if (req.http.Cookie ~ "(^|.*; )I18N_LANGUAGE=") { > set req.http.Accept-Language = regsub(req.http.Cookie, "(^|.*; > )I18N_LANGUAGE=([^;]*)(; .*|$)", "\2"); > # XXX need to work out the proper way to match " here, e.g. "en" > set req.http.Accept-Language = regsub(req.http.Accept-Language, > "^.(.*).$", "\1"); > } else { > set req.http.Accept-Language = "en"; > } This is fine, as long as you always have a cookie. We weren't in this situation, and we wanted to honour Accept-Language properly, so I wrote this VCL: http://github.com/cosimo/varnish-accept-language/ -- Cosimo From alex at bengler.no Fri Jul 2 11:53:39 2010 From: alex at bengler.no (Alexander Staubo) Date: Fri, 2 Jul 2010 13:53:39 +0200 Subject: Varnish constantly restarting Message-ID: Varnish (2.0.6 on Ubuntu Hardy, also tested 2.1.2 from source; Linux x64) restarts many times per day with the following messages being logged: varnishd[9010]: child (9011) Started varnishd[9010]: Child (9011) said Closed fds: 3 4 5 9 10 12 13 varnishd[9010]: Child (9011) said Child starts varnishd[9010]: Child (9011) said Ready varnishd[9010]: Child (9011) not responding to ping, killing it. varnishd[9010]: Child (9011) not responding to ping, killing it. varnishd[9010]: Child (9011) died signal=3 varnishd[9010]: Child cleanup complete varnishd[9010]: child (9315) Started varnishd[9010]: Child (9315) said Closed fds: 3 4 5 10 11 13 14 varnishd[9010]: Child (9315) said Child starts varnishd[9010]: Child (9315) said Ready Tried modifying defaults and switching from file-based cache to malloc-based; it seems to be hanging less often now, but it still happens a couple of times a day. Here is my current command line: varnishd -P /var/run/varnishd.pid \ -a :6081 \ -f /etc/varnish/default.vcl \ -T 127.0.0.1:6082 \ -t 120 \ -w 4,30,120 \ -s malloc,1G \ -p thread_pool_max=300 \ -p ping_interval=5 \ -p cli_timeout=120s \ -u varnish \ -g varnish At the point when the Varnish child stops responding to ping, CPU usage blows through the roof. I started a screen running on the box that logs varnishstat every 10s; this is the last output just abefore it dies: uptime 71589 . Child uptime client_conn 4751731 66.38 Client connections accepted client_drop 0 0.00 Connection dropped, no sess client_req 4747499 66.32 Client requests received cache_hit 4647642 64.92 Cache hits cache_hitpass 775 0.01 Cache hits for pass cache_miss 86706 1.21 Cache misses backend_conn 87236 1.22 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 247 0.00 Backend conn. failures backend_reuse 0 0.00 Backend conn. reuses backend_toolate 0 0.00 Backend conn. was closed backend_recycle 0 0.00 Backend conn. recycles backend_unused 0 0.00 Backend conn. unused fetch_head 2 0.00 Fetch head fetch_length 87222 1.22 Fetch with Length fetch_chunked 0 0.00 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 6 0.00 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 0 0.00 Fetch failed n_srcaddr 0 . N struct srcaddr n_srcaddr_act 0 . N active struct srcaddr n_sess_mem 129 . N struct sess_mem n_sess 1899 . N struct sess n_object 28283 . N struct object n_objecthead 28288 . N struct objecthead n_smf 0 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 18446744073709551598 . N struct vbe_conn n_bereq 30 . N struct bereq n_wrk 9 . N worker threads n_wrk_create 682 0.01 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_queue 119 0.00 N queued work requests n_wrk_overflow 5138 0.07 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 1 . N backends n_expired 39742 . N expired objects n_lru_nuked 19046 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 2610711 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 3055715 42.68 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 4751602 66.37 Total Sessions s_req 4751602 66.37 Total Requests s_pipe 0 0.00 Total pipe s_pass 777 0.01 Total pass s_fetch 87230 1.22 Total fetch s_hdrbytes 2066945543 28872.39 Total header bytes s_bodybytes 16985467831 237263.66 Total body bytes sess_closed 4751602 66.37 Session Closed sess_pipeline 0 0.00 Session Pipeline sess_readahead 0 0.00 Session Read Ahead sess_linger 0 0.00 Session Linger sess_herd 0 0.00 Session herd shm_records 215002154 3003.28 SHM records shm_writes 19146311 267.45 SHM writes shm_flushes 112 0.00 SHM flushes due to overflow shm_cont 105277 1.47 SHM MTX contention shm_cycles 92 0.00 SHM cycles through buffer sm_nreq 0 0.00 allocator requests sm_nobj 0 . outstanding allocations sm_balloc 0 . bytes allocated sm_bfree 0 . bytes free sma_nreq 208247 2.91 SMA allocator requests sma_nobj 55341 . SMA outstanding allocations sma_nbytes 1073656763 . SMA outstanding bytes sma_balloc 2560359776 . SMA bytes allocated sma_bfree 1486703013 . SMA bytes free sms_nreq 14533 0.20 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 6363722 . SMS bytes allocated sms_bfree 6363722 . SMS bytes freed backend_req 87236 1.22 Backend requests made n_vcl 1 0.00 N vcl total n_vcl_avail 1 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_purge 1 . N total active purges n_purge_add 1 0.00 N new purges added n_purge_retire 0 0.00 N old purges deleted n_purge_obj_test 0 0.00 N objects tested n_purge_re_test 0 0.00 N regexps tested against n_purge_dups 0 0.00 N duplicate purges removed hcb_nolock 0 0.00 HCB Lookups without lock hcb_lock 0 0.00 HCB Lookups with lock hcb_insert 0 0.00 HCB Inserts esi_parse 0 0.00 Objects ESI parsed (unlock) esi_errors 0 0.00 ESI parse errors (unlock) The odd one out is n_vbe_conn, don't know if this is benign. From jdixon at omniti.com Fri Jul 2 18:39:36 2010 From: jdixon at omniti.com (Jason Dixon) Date: Fri, 2 Jul 2010 14:39:36 -0400 Subject: CFP for Surge Scalability Conference 2010 Message-ID: <20100702183936.GU29381@omniti.com> A quick reminder that there's one week left to submit your abstract for this year's Surge Scalability Conference. The event is taking place on Sept 30 and Oct 1, 2010 in Baltimore, MD. Surge focuses on case studies that address production failures and the re-engineering efforts that led to victory in Web Applications or Internet Architectures. Our Keynote speakers include John Allspaw and Theo Schlossnagle. We are currently accepting submissions for the Call For Papers through July 9th. You can find more information, including suggested topics and our current list of speakers, online: http://omniti.com/surge/2010 I'd also like to urge folks who are planning to attend, to get your session passes sooner rather than later. We have limited seating and we are on track to sell out early. For more information, including the CFP, sponsorship of the event, or participating as an exhibitor, please visit the Surge website or contact us at surge at omniti.com. Thanks, -- Jason Dixon OmniTI Computer Consulting, Inc. jdixon at omniti.com 443.325.1357 x.241 From fla_torres at yahoo.com.br Sat Jul 3 17:01:40 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Sat, 03 Jul 2010 14:01:40 -0300 Subject: Varnish constantly restarting In-Reply-To: References: Message-ID: <4C2F6CF4.3030705@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Em 2/7/2010 08:53, Alexander Staubo escreveu: > varnishd[9010]: Child (9011) said Ready > varnishd[9010]: Child (9011) not responding to ping, killing it. > varnishd[9010]: Child (9011) not responding to ping, killing it. > varnishd[9010]: Child (9011) died signal=3 Hi Alexander, Have you seen this thread http://lists.varnish-cache.org/pipermail/varnish-misc/2010-June/004358.html ? Also, you are having some failure connections to the backend (1). How is your backend health ? []'s - -- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (MingW32) iEYEARECAAYFAkwvbPQACgkQNRQApncg294ZQACgn5jatwv1+cVsYKfua6jQ1lM8 nx4An21oNmnjYihg2P1tTtcZLVP8Osnn =4NXl -----END PGP SIGNATURE----- From alex at bengler.no Sat Jul 3 17:59:32 2010 From: alex at bengler.no (Alexander Staubo) Date: Sat, 3 Jul 2010 19:59:32 +0200 Subject: Varnish constantly restarting In-Reply-To: <4C2F6CF4.3030705@yahoo.com.br> References: <4C2F6CF4.3030705@yahoo.com.br> Message-ID: On Sat, Jul 3, 2010 at 7:01 PM, Flavio Torres wrote: > Hi Alexander, > > Have you seen this thread > http://lists.varnish-cache.org/pipermail/varnish-misc/2010-June/004358.html > ? Yes. That threads seems to be about a different problem. In my case I don't have any panic messages in the log. Curiously, though, the child is dying with SIGQUIT. Should it not be producing a core dump somewhere then? > Also, you are having some failure connections to the backend (1). How > is your backend health ? The backend is HAProxy, which is pretty solid. Might be worth looking into, but I'm not too concerned about a %0.002 failure rate in this case. From perbu at varnish-software.com Sat Jul 3 21:04:09 2010 From: perbu at varnish-software.com (Per Buer) Date: Sat, 3 Jul 2010 23:04:09 +0200 Subject: Varnish constantly restarting In-Reply-To: References: <4C2F6CF4.3030705@yahoo.com.br> Message-ID: On Sat, Jul 3, 2010 at 7:59 PM, Alexander Staubo wrote: > On Sat, Jul 3, 2010 at 7:01 PM, Flavio Torres wrote: >> Hi Alexander, >> >> Have you seen this thread >> http://lists.varnish-cache.org/pipermail/varnish-misc/2010-June/004358.html >> ? > > Yes. That threads seems to be about a different problem. I don't think so. Try it. > In my case I don't have any panic messages in the log. You're sure you're looking in the right log? > Curiously, though, the child is dying with SIGQUIT. Are you setting diag_bitmap? That would make the master send QUIT. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From alex at bengler.no Sat Jul 3 21:16:10 2010 From: alex at bengler.no (Alexander Staubo) Date: Sat, 3 Jul 2010 23:16:10 +0200 Subject: Varnish constantly restarting In-Reply-To: References: <4C2F6CF4.3030705@yahoo.com.br> Message-ID: On Sat, Jul 3, 2010 at 11:04 PM, Per Buer wrote: > On Sat, Jul 3, 2010 at 7:59 PM, Alexander Staubo wrote: >> On Sat, Jul 3, 2010 at 7:01 PM, Flavio Torres wrote: >>> Hi Alexander, >>> >>> Have you seen this thread >>> http://lists.varnish-cache.org/pipermail/varnish-misc/2010-June/004358.html >>> ? >> >> Yes. That threads seems to be about a different problem. > > I don't think so. Try it. I do think so: (1) The child isn't segfaulting -- it's dying with signal 3 (SIGQUIT), and only after the parent sends SIGKILL. (2) I have already increased cli_timeout and decreased the maximum number of threads, and it's not helping. (3) The number of threads at the time that the child dies is not high. After switching to malloc and increasing the cli_timeout, Varnish does stay up for longer periods of time, but it's still dying at least once a day. >> In my case I don't have any panic messages in the log. > > You're sure you're looking in the right log? Yes. >> Curiously, though, the child is dying with SIGQUIT. > > Are you setting diag_bitmap? That would make the master send QUIT. I'm not. From alex at bengler.no Sun Jul 4 03:35:10 2010 From: alex at bengler.no (Alexander Staubo) Date: Sun, 4 Jul 2010 05:35:10 +0200 Subject: Varnish constantly restarting In-Reply-To: References: <4C2F6CF4.3030705@yahoo.com.br> Message-ID: On Sat, Jul 3, 2010 at 11:04 PM, Per Buer wrote: > On Sat, Jul 3, 2010 at 7:59 PM, Alexander Staubo wrote: >> Curiously, though, the child is dying with SIGQUIT. > > Are you setting diag_bitmap? That would make the master send QUIT. Actually, it looks like Varnish *will* use SIGQUIT ordinarily, as diag_bitmap seems to be 0 by default. From mgt_child.c (apparently 0x1000 means "do not core-dump child process"): if (params->diag_bitmap & 0x1000) (void)kill(child_pid, SIGKILL); else (void)kill(child_pid, SIGQUIT); That's an odd choice of default behaviour. I would expect core-dumping to be something you would explicitly turn on, especially given that the name of the variable is diag_bitmap. From bnowacky at competitorgroup.com Mon Jul 5 01:48:58 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Sun, 4 Jul 2010 18:48:58 -0700 Subject: varnishstat/top/hist not working Message-ID: <705F7CF8-6705-4E0E-86BA-C4BF4B323AB3@competitorgroup.com> So after having some instability with varnish, I tried re-installing from trunk instead of from Sourceforge... I've got Varnish working now (and seems to be stable... finally), but having some new issue. Varnishstat / hist / top / etc... don't seem to be working anymore. I'm deleting the old directory, but whenever I do : varnishstat I get: Cannot open (null): Bad address Any ideas how I can fix this? The only thing I care about is varnishstat... I'm sure it's something I'm doing wrong, but any help would be appreciated! Thanks ! From perbu at varnish-software.com Mon Jul 5 07:08:29 2010 From: perbu at varnish-software.com (Per Buer) Date: Mon, 5 Jul 2010 09:08:29 +0200 Subject: Varnish constantly restarting In-Reply-To: References: <4C2F6CF4.3030705@yahoo.com.br> Message-ID: On Sat, Jul 3, 2010 at 11:16 PM, Alexander Staubo wrote: > On Sat, Jul 3, 2010 at 11:04 PM, Per Buer wrote: >> On Sat, Jul 3, 2010 at 7:59 PM, Alexander Staubo wrote: >>> On Sat, Jul 3, 2010 at 7:01 PM, Flavio Torres wrote: >>>> Hi Alexander, >>>> >>>> Have you seen this thread >>>> http://lists.varnish-cache.org/pipermail/varnish-misc/2010-June/004358.html >>>> ? >>> >>> Yes. That threads seems to be about a different problem. >> >> I don't think so. Try it. > > I do think so: There is only one place where SIGQUIT is used and that is where you're hitting it. I'd try to disable the whole check by setting the cli_timeout ridiculously high. If you have proper monitoring the risk isn't high. You might be having issues with your io scheduler (cfq can be a real disaster) or a bigger working set then your memory can handle (swap on Linux doesn't work very well). -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From alex at bengler.no Mon Jul 5 11:46:51 2010 From: alex at bengler.no (Alexander Staubo) Date: Mon, 5 Jul 2010 13:46:51 +0200 Subject: Varnish constantly restarting In-Reply-To: References: <4C2F6CF4.3030705@yahoo.com.br> Message-ID: On Mon, Jul 5, 2010 at 9:08 AM, Per Buer wrote: > There is only one place where SIGQUIT is used and that is where you're > hitting it. I'd try to disable the whole check by setting the > cli_timeout ridiculously high. If you have proper monitoring the risk > isn't high. That makes sense. I was slow in catching onto the fact that Varnish was simply killing its child because it was slow in responding to the pings; from my point of view it looked more like Varnish was hanging. > You might be having issues ?with your io scheduler (cfq > can be a real disaster) or a bigger working set then your memory can > handle (swap on Linux doesn't work very well). How do I go about determining the underlying cause? As far as I can see, it's not caused by any increase in traffic. I am tracking every varnishstat metric using Munin, and not seeing anything out of the ordinary at the instances when Varnish kills itself -- nor with any of the other system metrics tracked with Munin. There should be plenty of memory available, and I'm not seeing any swapping. Linux 2.6.24 + CFQ reports weird CPU usage [1] that screws up our graphs, but that's hardly an indication of anything. However, we should probably try to switch away from CFQ. [1] http://grab.by/grabs/af573a30e5be0774edcfffc77294f4d8.png From pahudnet at gmail.com Mon Jul 5 13:17:37 2010 From: pahudnet at gmail.com (Pahud) Date: Mon, 5 Jul 2010 21:17:37 +0800 Subject: how can I make varnish to just cache but don't deliver? Message-ID: Hell list, I am running some varnish boxes for my CDN deployment. I am gonna parse the varnish log to see its recent received requests and ask other varnish boxes to cache those content as well. It is very easy to write a script with curl to make it possible, but If I curl on other varnish boxes they well return the content back to me, which I don't want it to do so. I am thinking maybe I can send a "Cache-Control: just-cache" header to other boxes and when they see this header they will not 'deliver' in theyr vcl_deliver(). So I tried this configuration sub vcl_deliver { if (req.http.Cache-Control ~ "just-cache") { error 200 "Cached"; return (pass); } return (deliver); } But no good luck. I guess vcl_deliver just can't get the value of req.http.Cache-Control. Is it still possible to achieve this just with VCL? pahud -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at varnish-software.com Mon Jul 5 14:11:24 2010 From: kristian at varnish-software.com (=?UTF-8?Q?Kristian_Lyngst=C3=B8l?=) Date: Mon, 5 Jul 2010 16:11:24 +0200 Subject: how can I make varnish to just cache but don't deliver? In-Reply-To: References: Message-ID: On Mon, Jul 5, 2010 at 3:17 PM, Pahud wrote: > vcl_deliver(). So I tried this configuration > sub vcl_deliver { > ?? ? ? ? if (req.http.Cache-Control ~ "just-cache") { > ?? ? ? ? ? ? ? error 200 "Cached"; > ?? ? ? ? ? ? ? return (pass); return (pass); doesn't work from vcl_deliver, nor is it needed after an error-statement. > ?? ? ? ? } > ?? ?return (deliver); > } > But no good luck. I guess vcl_deliver just can't get the value of > ?req.http.Cache-Control. > Is it still possible to achieve this just with VCL? Hmm. What you could do is: In vcl_fetch: if (req.http.Cache-Control ~ "just-cache") { set obj.http.X-Cache-Only = "true"; } In vcl_deliver: if (resp.http.X-Cache-Only == "true") { if (obj.hits == 0) { error 795 "Cached"; } else { remove resp.http.X-Cache-Only; } } I used the error code 795 to differentiate from 200, so you can catch it in vcl_error and strip most of the content of the error message if you want to. You may use whatever error code you want, of course. (I haven't tested this, but I don't see why it shouldn't work, except perhaps syntax typos). - Kristian From pahudnet at gmail.com Mon Jul 5 14:49:45 2010 From: pahudnet at gmail.com (Pahud) Date: Mon, 5 Jul 2010 22:49:45 +0800 Subject: how can I make varnish to just cache but don't deliver? In-Reply-To: References: Message-ID: Thanks for your quick reply. I tried your config and varnish started with no error but if I send http request in this case, varnish will have empty response. And if I comment off the vcl_deliver it responds as normal. Can you help me a little bit more? Thanks again. pahud On Mon, Jul 5, 2010 at 10:11 PM, Kristian Lyngst?l < kristian at varnish-software.com> wrote: > On Mon, Jul 5, 2010 at 3:17 PM, Pahud wrote: > > vcl_deliver(). So I tried this configuration > > sub vcl_deliver { > > if (req.http.Cache-Control ~ "just-cache") { > > error 200 "Cached"; > > return (pass); > > return (pass); doesn't work from vcl_deliver, nor is it needed after > an error-statement. > > > } > > return (deliver); > > } > > But no good luck. I guess vcl_deliver just can't get the value of > > req.http.Cache-Control. > > Is it still possible to achieve this just with VCL? > > Hmm. What you could do is: > > In vcl_fetch: if (req.http.Cache-Control ~ "just-cache") { set > obj.http.X-Cache-Only = "true"; } > In vcl_deliver: > > if (resp.http.X-Cache-Only == "true") { > if (obj.hits == 0) { > error 795 "Cached"; > } else { > remove resp.http.X-Cache-Only; > } > } > > I used the error code 795 to differentiate from 200, so you can catch > it in vcl_error and strip most of the content of the error message if > you want to. You may use whatever error code you want, of course. > > (I haven't tested this, but I don't see why it shouldn't work, except > perhaps syntax typos). > > - Kristian > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fla_torres at yahoo.com.br Tue Jul 6 03:00:26 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Tue, 06 Jul 2010 00:00:26 -0300 Subject: varnishstat/top/hist not working In-Reply-To: <705F7CF8-6705-4E0E-86BA-C4BF4B323AB3@competitorgroup.com> References: <705F7CF8-6705-4E0E-86BA-C4BF4B323AB3@competitorgroup.com> Message-ID: <4C329C4A.9090203@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Em 4/7/2010 22:48, Ben Nowacky escreveu: > I'm deleting the old directory, but whenever I do : > > varnishstat hmm, check if ur (echo $PATH) environment variable is pointing to your new varnish directory. regards - -- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (MingW32) iEYEARECAAYFAkwynEoACgkQNRQApncg296cUACg8OtKiGZbTbdvbbJ0mgreP/BX JZsAn3V1Bl0p0xxodjm08UmVkoWXjzk8 =Xjnm -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From fla_torres at yahoo.com.br Tue Jul 6 03:41:49 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Tue, 06 Jul 2010 00:41:49 -0300 Subject: Varnish constantly restarting In-Reply-To: References: <4C2F6CF4.3030705@yahoo.com.br> Message-ID: <4C32A5FD.6030202@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Em 3/7/2010 14:59, Alexander Staubo escreveu: > Yes. That threads seems to be about a different problem. In my case I > don't have any panic messages in the log. Curiously, though, the child > is dying with SIGQUIT. Alexander, We had the same problem with the 2.0.6 version running on a VMBox, it was fixed upgrading to the 2.1.2. Are you still with the 2.1 version ? I saw you Munin graph, 200% of CPU Usage means your load? What is your resource ? Did you tune your settings ? []'s - -- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (MingW32) iEYEARECAAYFAkwypf0ACgkQNRQApncg295w/wCgyO2hn21rj0g09jxdjh1qZNHJ cTUAnjF0MyRzhiySJQkWWc/NY5+ZYgQM =MPJz -----END PGP SIGNATURE----- From checker at d6.com Tue Jul 6 05:10:08 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 05 Jul 2010 22:10:08 -0700 Subject: Varnish constantly restarting In-Reply-To: <4C32A5FD.6030202@yahoo.com.br> References: <4C2F6CF4.3030705@yahoo.com.br> <4C32A5FD.6030202@yahoo.com.br> Message-ID: <4C32BAB0.7020800@d6.com> > We had the same problem with the 2.0.6 version Ugh, this is the version that's installed on my CentOS box that I was about to start using...are there known problems with this version? Chris Flavio Torres wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Em 3/7/2010 14:59, Alexander Staubo escreveu: >> Yes. That threads seems to be about a different problem. In my case I >> don't have any panic messages in the log. Curiously, though, the child >> is dying with SIGQUIT. > > Alexander, > > We had the same problem with the 2.0.6 version running on a VMBox, it > was fixed upgrading to the 2.1.2. Are you still with the 2.1 version ? > > I saw you Munin graph, 200% of CPU Usage means your load? What is your > resource ? Did you tune your settings ? > > > []'s > > > > - -- > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.8 (MingW32) > > iEYEARECAAYFAkwypf0ACgkQNRQApncg295w/wCgyO2hn21rj0g09jxdjh1qZNHJ > cTUAnjF0MyRzhiySJQkWWc/NY5+ZYgQM > =MPJz > -----END PGP SIGNATURE----- > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From tfheen at varnish-software.com Tue Jul 6 06:35:48 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 06 Jul 2010 08:35:48 +0200 Subject: Varnish & ClamAV on FreeBSD 8 STABLE In-Reply-To: <20100615081204.GA16478@heitec.de> (Holger Rauch's message of "Tue, 15 Jun 2010 10:12:04 +0200") References: <20100615081204.GA16478@heitec.de> Message-ID: <87ocelxgl7.fsf@qurzaw.linpro.no> ]] Holger Rauch Hi, | has anybody tried setting up Varnish in conjunction with ClamAV on | FreeBSD 8 STABLE or, more generally asked, does Varnish support some | kind of "interface" to virus scanners? There's currently no (easy) way to access the contents of objects you are serving, so no. regards, -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Tue Jul 6 06:39:28 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 06 Jul 2010 08:39:28 +0200 Subject: Byte range and varnish In-Reply-To: <33C23F91-FFD0-4E14-8ACD-FF07F694DB93@rts.ch> (Jean-Francois Laurens's message of "Thu, 17 Jun 2010 15:00:16 +0200") References: <33C23F91-FFD0-4E14-8ACD-FF07F694DB93@rts.ch> Message-ID: <87k4p9xgf3.fsf@qurzaw.linpro.no> ]] Jean-Francois Laurens Hi, | I'm running varnish 2.0.6 under centos 5.4: [...] | mp3: progressive download is used in order to read mp3 from the | webpages, is it foreseen to have a function to would allow providing | the media before it gets completely retrieved by varnish from the | backend ? It's on the roadmap, but it's not implemented yet. | mp4: how is handled Accept-range field in the header, does varnish supports it ? Varnish 2.1.2 has experimental support for the Range header, so if you upgrade, you can try it out there. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Tue Jul 6 06:52:41 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 06 Jul 2010 08:52:41 +0200 Subject: purge files from cache In-Reply-To: <20100621132930.C5D162F7573@smtp1.acasa.ro> (Alex F.'s message of "Mon, 21 Jun 2010 16:29:27 +0300") References: <20100621132930.C5D162F7573@smtp1.acasa.ro> Message-ID: <87fwzxxft2.fsf@qurzaw.linpro.no> ]] Alex F | Prior to version 2.1.2 there used to be hash.purge to which I would | pass the file and the url to be purged. | Now I see that it has been removed. | How can I achieve the same functionality? | eg: hash.purge style.css example.com Try: purge req.url == "style.css" && req.http.host == "example.com" I think the quotes are needed, but you'll find out quickly enough yourself. :-) cheers, -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From jean-francois.laurens at rts.ch Tue Jul 6 07:01:11 2010 From: jean-francois.laurens at rts.ch (Jean-Francois Laurens) Date: Tue, 6 Jul 2010 09:01:11 +0200 Subject: Byte range and varnish In-Reply-To: <87k4p9xgf3.fsf@qurzaw.linpro.no> References: <33C23F91-FFD0-4E14-8ACD-FF07F694DB93@rts.ch> <87k4p9xgf3.fsf@qurzaw.linpro.no> Message-ID: <21CAC691-E714-43BA-8617-BAAD429CEE68@rts.ch> Hi, Thank you and Per for the answers provided. I'll be happy to try out those new functionality when they'll be implemented, please don't hesitate to keep me informed. I did not have time yet to install 2.1.2 and try the range header. What we basically do now is just pipe all the content that is media file, which is working pretty well till now. We have a problem though with mp3 progressive download: Some of our users are complaining about download that stops at some random time in our embedded flash player. They need to download the file again from scratch by make it play from the beginning from the player. I don't believe varnish creates the problem but more the NFS storage that could be slow to response. I can see some long delays for certain mp3 on the backend apache logs but I don't know how to get some more informations from varnish (if there is any !). Our hosting center can't provide us with some figures on the NFS usage yet so I'm a bit stuck in my investigations. And I did not succeed in reproducing the problem .... Cheers, Jef On 6 juil. 2010, at 08:39, Tollef Fog Heen wrote: > ]] Jean-Francois Laurens > > Hi, > > | I'm running varnish 2.0.6 under centos 5.4: > > [...] > > | mp3: progressive download is used in order to read mp3 from the > | webpages, is it foreseen to have a function to would allow providing > | the media before it gets completely retrieved by varnish from the > | backend ? > > It's on the roadmap, but it's not implemented yet. > > | mp4: how is handled Accept-range field in the header, does varnish supports it ? > > Varnish 2.1.2 has experimental support for the Range header, so if you > upgrade, you can try it out there. > > Regards, > -- > Tollef Fog Heen > Varnish Software > t: +47 21 54 41 73 Jean-Fran?ois Laurens Ing?nieur Syst?me Unix RTS - radio t?l?vision suisse Quai Ernest-Ansermet 20 Case Postale 234 CH-1211 Gen?ve 8 T +41 (0)22 708 8163 -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex at acasa.ro Tue Jul 6 07:16:42 2010 From: alex at acasa.ro (Alex F) Date: Tue, 06 Jul 2010 10:16:42 +0300 Subject: purge files from cache In-Reply-To: <87fwzxxft2.fsf@qurzaw.linpro.no> References: <20100621132930.C5D162F7573@smtp1.acasa.ro> <87fwzxxft2.fsf@qurzaw.linpro.no> Message-ID: <20100706071640.514352F75DB@smtp1.acasa.ro> Hi, I have been using almost the same command as yours: purge req.http.host == example.com && req.url ~ ^/.*style.css (no quotes needed :P) Thank you & cheers. On 6.7.2010 9:52, Tollef Fog Heen wrote: > ]] Alex F > > | Prior to version 2.1.2 there used to be hash.purge to which I would > | pass the file and the url to be purged. > | Now I see that it has been removed. > | How can I achieve the same functionality? > | eg: hash.purge style.css example.com > > Try: > > purge req.url == "style.css"&& req.http.host == "example.com" > > I think the quotes are needed, but you'll find out quickly enough > yourself. :-) > > cheers, > From bedis9 at gmail.com Tue Jul 6 07:45:16 2010 From: bedis9 at gmail.com (Bedis 9) Date: Tue, 6 Jul 2010 09:45:16 +0200 Subject: varnishstat/top/hist not working In-Reply-To: <4C329C4A.9090203@yahoo.com.br> References: <705F7CF8-6705-4E0E-86BA-C4BF4B323AB3@competitorgroup.com> <4C329C4A.9090203@yahoo.com.br> Message-ID: Hi, and check if libncurses-dev was installed before compiling from source. cheers On Tue, Jul 6, 2010 at 5:00 AM, Flavio Torres wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Em 4/7/2010 22:48, Ben Nowacky escreveu: >>? I'm deleting the old directory, but whenever I do : >> >> varnishstat > > > hmm, check if ur (echo $PATH) environment variable is pointing to your > new varnish directory. > > regards > > > - -- > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.8 (MingW32) > > iEYEARECAAYFAkwynEoACgkQNRQApncg296cUACg8OtKiGZbTbdvbbJ0mgreP/BX > JZsAn3V1Bl0p0xxodjm08UmVkoWXjzk8 > =Xjnm > -----END PGP SIGNATURE----- > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From pahudnet at gmail.com Tue Jul 6 10:23:15 2010 From: pahudnet at gmail.com (Pahud) Date: Tue, 6 Jul 2010 18:23:15 +0800 Subject: how can I make varnish to just cache but don't deliver? In-Reply-To: References: Message-ID: I tried varnishd (varnish-2.1.2 SVN 4769:4772) on Gentoo Linux today. And the VCL compilation failed: Message from VCC-compiler: Variable 'obj.http.X-Cache-Only' not accessible in method 'vcl_fetch'. At: (input Line 106 Pos 21) set obj.http.X-Cache-Only = "true"; --------------------#####################---------- Running VCC-compiler failed, exit 1 VCL compilation failed I also tried set resp.http.X-Cache-Only = "true"; but still failed. :( pahud On Mon, Jul 5, 2010 at 10:49 PM, Pahud wrote: > Thanks for your quick reply. I tried your config and varnish started with > no error but if I send http request in this case, varnish will have empty > response. And if I comment off the vcl_deliver it responds as normal. Can > you help me a little bit more? Thanks again. > > pahud > > > > > > > On Mon, Jul 5, 2010 at 10:11 PM, Kristian Lyngst?l < > kristian at varnish-software.com> wrote: > >> On Mon, Jul 5, 2010 at 3:17 PM, Pahud wrote: >> > vcl_deliver(). So I tried this configuration >> > sub vcl_deliver { >> > if (req.http.Cache-Control ~ "just-cache") { >> > error 200 "Cached"; >> > return (pass); >> >> return (pass); doesn't work from vcl_deliver, nor is it needed after >> an error-statement. >> >> > } >> > return (deliver); >> > } >> > But no good luck. I guess vcl_deliver just can't get the value of >> > req.http.Cache-Control. >> > Is it still possible to achieve this just with VCL? >> >> Hmm. What you could do is: >> >> In vcl_fetch: if (req.http.Cache-Control ~ "just-cache") { set >> obj.http.X-Cache-Only = "true"; } >> In vcl_deliver: >> >> if (resp.http.X-Cache-Only == "true") { >> if (obj.hits == 0) { >> error 795 "Cached"; >> } else { >> remove resp.http.X-Cache-Only; >> } >> } >> >> I used the error code 795 to differentiate from 200, so you can catch >> it in vcl_error and strip most of the content of the error message if >> you want to. You may use whatever error code you want, of course. >> >> (I haven't tested this, but I don't see why it shouldn't work, except >> perhaps syntax typos). >> >> - Kristian >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pahudnet at gmail.com Tue Jul 6 11:35:25 2010 From: pahudnet at gmail.com (Pahud) Date: Tue, 6 Jul 2010 19:35:25 +0800 Subject: how can I make varnish to just cache but don't deliver? In-Reply-To: References: Message-ID: OK I think this is because 'error' is not allowed in vcl_deliver. After some google I found this guy had the same problem with me in 2008. http://www.mail-archive.com/varnish-dev at projects.linpro.no/msg00372.html But this thread ended up with no solution. Please help. pahud On Tue, Jul 6, 2010 at 6:23 PM, Pahud wrote: > I tried varnishd (varnish-2.1.2 SVN 4769:4772) on Gentoo Linux today. And > the VCL compilation failed: > > Message from VCC-compiler: > Variable 'obj.http.X-Cache-Only' not accessible in method 'vcl_fetch'. > At: (input Line 106 Pos 21) > set obj.http.X-Cache-Only = "true"; > --------------------#####################---------- > Running VCC-compiler failed, exit 1 > VCL compilation failed > > I also tried > set resp.http.X-Cache-Only = "true"; > but still failed. > > :( > > pahud > > > On Mon, Jul 5, 2010 at 10:49 PM, Pahud wrote: > >> Thanks for your quick reply. I tried your config and varnish started with >> no error but if I send http request in this case, varnish will have empty >> response. And if I comment off the vcl_deliver it responds as normal. Can >> you help me a little bit more? Thanks again. >> >> pahud >> >> >> >> >> >> >> On Mon, Jul 5, 2010 at 10:11 PM, Kristian Lyngst?l < >> kristian at varnish-software.com> wrote: >> >>> On Mon, Jul 5, 2010 at 3:17 PM, Pahud wrote: >>> > vcl_deliver(). So I tried this configuration >>> > sub vcl_deliver { >>> > if (req.http.Cache-Control ~ "just-cache") { >>> > error 200 "Cached"; >>> > return (pass); >>> >>> return (pass); doesn't work from vcl_deliver, nor is it needed after >>> an error-statement. >>> >>> > } >>> > return (deliver); >>> > } >>> > But no good luck. I guess vcl_deliver just can't get the value of >>> > req.http.Cache-Control. >>> > Is it still possible to achieve this just with VCL? >>> >>> Hmm. What you could do is: >>> >>> In vcl_fetch: if (req.http.Cache-Control ~ "just-cache") { set >>> obj.http.X-Cache-Only = "true"; } >>> In vcl_deliver: >>> >>> if (resp.http.X-Cache-Only == "true") { >>> if (obj.hits == 0) { >>> error 795 "Cached"; >>> } else { >>> remove resp.http.X-Cache-Only; >>> } >>> } >>> >>> I used the error code 795 to differentiate from 200, so you can catch >>> it in vcl_error and strip most of the content of the error message if >>> you want to. You may use whatever error code you want, of course. >>> >>> (I haven't tested this, but I don't see why it shouldn't work, except >>> perhaps syntax typos). >>> >>> - Kristian >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bedis9 at gmail.com Tue Jul 6 11:45:52 2010 From: bedis9 at gmail.com (Bedis 9) Date: Tue, 6 Jul 2010 13:45:52 +0200 Subject: how can I make varnish to just cache but don't deliver? In-Reply-To: References: Message-ID: Why not running your curl script on the loopback of your slave Varnish? cheers On Tue, Jul 6, 2010 at 1:35 PM, Pahud wrote: > OK I think this is because 'error' is not allowed in vcl_deliver. ?After > some google I found this guy had the same problem with me in 2008. > http://www.mail-archive.com/varnish-dev at projects.linpro.no/msg00372.html > But this thread ended up with no solution. > Please help. > pahud > > On Tue, Jul 6, 2010 at 6:23 PM, Pahud wrote: >> >> I tried?varnishd (varnish-2.1.2 SVN 4769:4772) on Gentoo Linux today. And >> the VCL compilation failed: >> Message from VCC-compiler: >> Variable 'obj.http.X-Cache-Only' not accessible in method 'vcl_fetch'. >> At: (input Line 106 Pos 21) >> ?? ? ? ? ? ? ? ?set obj.http.X-Cache-Only = "true"; >> --------------------#####################---------- >> Running VCC-compiler failed, exit 1 >> VCL compilation failed >> I also tried >> set resp.http.X-Cache-Only = "true"; >> but still failed. >> :( >> pahud >> >> On Mon, Jul 5, 2010 at 10:49 PM, Pahud wrote: >>> >>> Thanks for your quick reply. I tried your config and varnish started with >>> no error but if I send http request in this case, varnish will have empty >>> response. And if I comment off the vcl_deliver it responds as normal. ?Can >>> you help me a little bit more? Thanks again. >>> pahud >>> >>> >>> >>> >>> >>> On Mon, Jul 5, 2010 at 10:11 PM, Kristian Lyngst?l >>> wrote: >>>> >>>> On Mon, Jul 5, 2010 at 3:17 PM, Pahud wrote: >>>> > vcl_deliver(). So I tried this configuration >>>> > sub vcl_deliver { >>>> > ?? ? ? ? if (req.http.Cache-Control ~ "just-cache") { >>>> > ?? ? ? ? ? ? ? error 200 "Cached"; >>>> > ?? ? ? ? ? ? ? return (pass); >>>> >>>> return (pass); doesn't work from vcl_deliver, nor is it needed after >>>> an error-statement. >>>> >>>> > ?? ? ? ? } >>>> > ?? ?return (deliver); >>>> > } >>>> > But no good luck. I guess vcl_deliver just can't get the value of >>>> > ?req.http.Cache-Control. >>>> > Is it still possible to achieve this just with VCL? >>>> >>>> Hmm. What you could do is: >>>> >>>> In vcl_fetch: if (req.http.Cache-Control ~ "just-cache") { set >>>> obj.http.X-Cache-Only = "true"; } >>>> In vcl_deliver: >>>> >>>> if (resp.http.X-Cache-Only == "true") { >>>> ? ?if (obj.hits == 0) { >>>> ? ? ? ?error 795 "Cached"; >>>> ? ?} else { >>>> ? ? ? ?remove resp.http.X-Cache-Only; >>>> ? ?} >>>> } >>>> >>>> I used the error code 795 to differentiate from 200, so you can catch >>>> it in vcl_error and strip most of the content of the error message if >>>> you want to. You may use whatever error code you want, of course. >>>> >>>> (I haven't tested this, but I don't see why it shouldn't work, except >>>> perhaps syntax typos). >>>> >>>> - Kristian >>> >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From alex at bengler.no Tue Jul 6 12:05:29 2010 From: alex at bengler.no (Alexander Staubo) Date: Tue, 6 Jul 2010 14:05:29 +0200 Subject: Varnish constantly restarting In-Reply-To: <4C32A5FD.6030202@yahoo.com.br> References: <4C2F6CF4.3030705@yahoo.com.br> <4C32A5FD.6030202@yahoo.com.br> Message-ID: On Tue, Jul 6, 2010 at 5:41 AM, Flavio Torres wrote: > We had the same problem with the 2.0.6 version running on a VMBox, it > was fixed upgrading to the 2.1.2. Are you still with the 2.1 version ? This is 2.0.6 as we are on Ubuntu Hardy, but I ran 2.1.2 from source for a while and saw the exact same behaviour then. > I saw you Munin graph, 200% of CPU Usage means your load? What is your > resource ? Did you tune your settings ? I included the Munin graph to show how weird it was (look at the idle percentages), and I think the weirdness is related to the scheduler. What do you mean by "resource"? From tfheen at varnish-software.com Tue Jul 6 12:29:00 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 06 Jul 2010 14:29:00 +0200 Subject: how can I make varnish to just cache but don't deliver? In-Reply-To: (Pahud's message of "Tue, 6 Jul 2010 19:35:25 +0800") References: Message-ID: <87wrt8x08j.fsf@qurzaw.linpro.no> ]] Pahud Hi, | OK I think this is because 'error' is not allowed in vcl_deliver. Given that the message from VCC is: | Message from VCC-compiler: | Variable 'obj.http.X-Cache-Only' not accessible in method 'vcl_fetch'. | At: (input Line 106 Pos 21) | set obj.http.X-Cache-Only = "true"; | --------------------#####################---------- Why do you think the problem is in vcl_deliver? You want to use beresp rather than obj in vcl_fetch. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From fla_torres at yahoo.com.br Tue Jul 6 14:52:30 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Tue, 06 Jul 2010 11:52:30 -0300 Subject: Varnish constantly restarting In-Reply-To: References: <4C2F6CF4.3030705@yahoo.com.br> <4C32A5FD.6030202@yahoo.com.br> Message-ID: <4C33432E.7040007@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/06/2010 09:05 AM, Alexander Staubo wrote: > On Tue, Jul 6, 2010 at 5:41 AM, Flavio Torres > wrote: >> We had the same problem with the 2.0.6 version running on a >> VMBox, it was fixed upgrading to the 2.1.2. Are you still with >> the 2.1 version ? > > This is 2.0.6 as we are on Ubuntu Hardy, but I ran 2.1.2 from > source for a while and saw the exact same behaviour then. > Would be nice keep 2.1.2 and tune it (varnish settings and SO - sysctl) - [1] > What do you mean by "resource"? CPU, Memory, disk. Is your varnish server a DomU ? [1] http://varnish-cache.org/wiki/Performance, http://lists.varnish-cache.org/pipermail/varnish-misc/2008-April/001763.html []'s -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwzQysACgkQNRQApncg297RVACfXu3TvYLnvEWgD0wrdOd99MB9 Ln4An3Na0/BLGTTTdWo5m+Lt4mlzO+g4 =d5By -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From bnowacky at competitorgroup.com Tue Jul 6 15:55:14 2010 From: bnowacky at competitorgroup.com (Ben Nowacky) Date: Tue, 6 Jul 2010 08:55:14 -0700 Subject: Varnish restarting sporadically... losing entire cache... Message-ID: <55B5D77D-5AC5-453D-9CCE-9F317AFCC9D0@competitorgroup.com> Okay, so there has to be something I'm doing wrong still.. I know I'm on a 32 bit system (centos), but even lowering threads, ulimit -s to all values between 32 and 1024, and even trying to adjust thread_pool_stack, I can't keep varnish running for more than 10 minutes at a time under load. There /has/ to be something I"m doing wrong. For some reason, even though they are running, i don't have anything in syslogs. Unfortunately, I'm not the linux pro i should be to enable and get a coredump of the daemon running. - Is anyone else running varnish on a 32 bit OS? (centos / redhat) - Is seg fault and stop / start every 4-10 minutes normal in this case and should i just deal? - If not, what startup commands are you using to keep things under control? Just doesn't make sense that varnish is /this/ unstable in a 32 bit os... Seems that if it were this bad for everyone, it would be completely unusable on anything BUT 64 bit architecture. (our servers are all remote, or we would have upgraded already. Doing a remote upgrade on live production servers is a bit scary as we're concerned about other libraries not functions with a new kernel, and how much debugging and fixing are we going to have to be doing!). Thanks in advance ! -------------- next part -------------- An HTML attachment was scrubbed... URL: From zabrane3 at gmail.com Tue Jul 6 22:10:13 2010 From: zabrane3 at gmail.com (zabrane Mikael) Date: Wed, 7 Jul 2010 00:10:13 +0200 Subject: Can Varnish do this? Message-ID: Hi List, I'm new to Varnish, so please apologize if the question was already asked before. I've a homemade web server running on my intranet. It works like this: 1. If an HTTP request arrives without a special header "X-RAWDATA: ...", this web server forward the request to the live Web, get the response, and send it back (i.e it plays a role of a basic proxy). 2. Now, if a request contains the special header "X-RAWDATA: ...", then the web server will search the response in its internal database (complex computations), and send it back. Point 2 works very very well. But not point 1 as my web server isn't a true HTTP proxy. So, my question is simple: Could I tell Varnish to inspect the HTTP headers, and decide: a. If the special header "X-RAWDATA: ..." doesn't exists, let the request go to the live Web directly. b. Otherwise, forward the request to my web server. Hope I'm clear guys. Thanks in advance. -- Zab -------------- next part -------------- An HTML attachment was scrubbed... URL: From whshub at gmail.com Wed Jul 7 02:46:25 2010 From: whshub at gmail.com (Jacques) Date: Tue, 6 Jul 2010 19:46:25 -0700 Subject: Troubleshooting: Setting up varnish 2.1.2 on OpenSolaris Message-ID: Hello, I'm trying to setup Solaris on OpenSolaris b134. I'm doing a setup in a brand new zone. I have reviewed the instructions on http://www.varnish-cache.org/wiki/Installation#OpenSolaris and am having problems. With a brand new zone, I've found that in addition to installing the list of packages provided, I also need to install: SUNWpcre & SUNWlibm. Once installed, I can configure & build correctly with: ---- VCC_CC="cc -Kpic -G -m64 -o %o %s" CC=/usr/bin/gcc CFLAGS="-O3 -L/opt/extra/lib/amd64 \ -pthreads -m64 -fomit-frame-pointer" LDFLAGS="-lumem -pthreads" ./configure --prefix=/opt/extra make && make install ---- I start by running: # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 >Compiled VCL program failed to load: > ld.so.1: varnishd: fatal: relocation error: R_AMD64_PC32: file ./ vcl.ORk8t3RP.so: symbol main: value 0x2800dcfbd44 does not fit >VCL compilation failed Not unexpected, I run: # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 -p cc_command='gcc -G -m64 -o %o %s' >Compiled VCL program failed to load: > ld.so.1: varnishd: fatal: relocation error: R_AMD64_PC32: file ./ vcl.ORk8t3RP.so: symbol main: value 0x2800dcdbd44 does not fit >VCL compilation failed So I install the sunstudioexpress compiler: # pkg install sunstudioexpress # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 -p cc_command=" /opt/SunStudioExpress/bin/cc -c -Kpic -m64 -o %o %s" >"./vcl.ORk8t3RP.c", line 356: warning: initializer will be sign-extended: -1 >storage_file: filename: ./varnish.BAaWXR size 461096 MB. >Using old SHMFILE >Varnish on -sfile,-hcritbit >200 193 >----------------------------- >Varnish HTTP accelerator CLI. >----------------------------- >Type 'help' for command list. >Type 'quit' to close CLI session. >Type 'start' to launch worker process. Varnish appears to be running. However, I getting 503s when I try to Varnish. I saw that the warning was mentioned in a closed bug: http://varnish-cache.org/ticket/649 So, can someone giving me some pointers? I really would like to use just the gcc compiler and not have to load all the SunStudioExpress stuff. But either way, I just want to make it work. I've reviewed the following resources and haven't solved my problem: http://blogs.everycity.co.uk/alasdair/2009/05/64bit-varnish-on-solaris/ http://letsgetdugg.com/2009/12/04/varnish-on-solaris/ http://src.opensolaris.org/source/xref/webstack/varnish/ Thanks for any help, Jacques Compiler Info: [/usr/bin/gcc -v] gcc version 3.4.3 (csl-sol210-3_4-20050802) [/usr/gnu/bin/cc] gcc version 3.4.3 (csl-sol210-3_4-20050802) [symlinked to gcc] [/opt/SunStudioExpress/bin/cc -V] cc: Sun Ceres C 5.10 SunOS_i386 2009/03/06 -------------- next part -------------- An HTML attachment was scrubbed... URL: From checker at d6.com Wed Jul 7 05:27:19 2010 From: checker at d6.com (Chris Hecker) Date: Tue, 06 Jul 2010 22:27:19 -0700 Subject: Varnish restarting sporadically... losing entire cache... In-Reply-To: <55B5D77D-5AC5-453D-9CCE-9F317AFCC9D0@competitorgroup.com> References: <55B5D77D-5AC5-453D-9CCE-9F317AFCC9D0@competitorgroup.com> Message-ID: <4C341037.60009@d6.com> > - Is anyone else running varnish on a 32 bit OS? (centos / redhat) I'm not yet, but I'm about to, and these recent threads have me terrified, and wondering whether I should have spent time learning squid, even though varnish seems so much better. Chris Ben Nowacky wrote: > Okay, so there has to be something I'm doing wrong still.. I know I'm on > a 32 bit system (centos), but even lowering threads, ulimit -s to all > values between 32 and 1024, and even trying to adjust thread_pool_stack, > I can't keep varnish running for more than 10 minutes at a time under load. > > There /has/ to be something I"m doing wrong. For some reason, even > though they are running, i don't have anything in syslogs. > Unfortunately, I'm not the linux pro i should be to enable and get a > coredump of the daemon running. > > - Is anyone else running varnish on a 32 bit OS? (centos / redhat) > - Is seg fault and stop / start every 4-10 minutes normal in this case > and should i just deal? > - If not, what startup commands are you using to keep things under control? > > Just doesn't make sense that varnish is /this/ unstable in a 32 bit > os... Seems that if it were this bad for everyone, it would be > completely unusable on anything BUT 64 bit architecture. > > (our servers are all remote, or we would have upgraded already. Doing a > remote upgrade on live production servers is a bit scary as we're > concerned about other libraries not functions with a new kernel, and how > much debugging and fixing are we going to have to be doing!). > > Thanks in advance ! > > > ------------------------------------------------------------------------ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From bedis9 at gmail.com Wed Jul 7 06:00:18 2010 From: bedis9 at gmail.com (Bedis 9) Date: Wed, 7 Jul 2010 08:00:18 +0200 Subject: Can Varnish do this? In-Reply-To: References: Message-ID: Hi, Varnish is a pure reverse-proxy. So I doubt you can do what you're saying. rgs On Wed, Jul 7, 2010 at 12:10 AM, zabrane Mikael wrote: > Hi List, > I'm new to Varnish, so please apologize if the question was already asked > before. > I've a homemade web server running on my intranet. It works like this: > 1. If an HTTP request arrives without a special header "X-RAWDATA: ...", > this web server forward the request to the live Web, get the response, and > send it back (i.e it plays a role of a basic proxy). > 2. Now, if a request contains the special header "X-RAWDATA: ...", then the > web server will search the response in > its internal database (complex computations), and send it back. > Point 2 works very very well. But not point 1 as my web server isn't a true > HTTP proxy. > So, my question is simple: > Could I tell Varnish to inspect the HTTP headers, and decide: > a. If the special header "X-RAWDATA: ..." doesn't exists, let the request go > to the live Web directly. > b. Otherwise, forward the request to my web server. > Hope I'm clear guys. > Thanks in advance. > -- > Zab > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From A.Hongens at netmatch.nl Wed Jul 7 06:10:25 2010 From: A.Hongens at netmatch.nl (=?iso-8859-1?Q?Angelo_H=F6ngens?=) Date: Wed, 7 Jul 2010 06:10:25 +0000 Subject: Can Varnish do this? In-Reply-To: References: Message-ID: > On Wed, Jul 7, 2010 at 12:10 AM, zabrane Mikael > wrote: > > Hi List, > > I'm new to Varnish, so please apologize if the question was already > asked > > before. > > I've a homemade web server running on my intranet. It works like > this: > > 1. If an HTTP request arrives without a special header "X-RAWDATA: > ...", > > this web server forward the request to the live Web, get the > response, and > > send it back (i.e it plays a role of a basic proxy). > > 2. Now, if a request contains the special header "X-RAWDATA: ...", > then the > > web server will search the response in > > its internal database (complex computations), and send it back. > > Point 2 works very very well. But not point 1 as my web server isn't > a true > > HTTP proxy. > > So, my question is simple: > > Could I tell Varnish to inspect the HTTP headers, and decide: > > a. If the special header "X-RAWDATA: ..." doesn't exists, let the > request go > > to the live Web directly. > > b. Otherwise, forward the request to my web server. > > Hope I'm clear guys. > > Thanks in advance. So basically what you want to do is define two backends, and if your custom header is present in the request, you want it to go to one backend, and if not, you want to go to the other backend. I'm not a varnish guru, but I think it should be possible quite easily. I googled, and this page shows you can do something with custom headers: http://www.lovelysystems.com/configuring-varnish-to-use-custom-http-headers/ Angelo. From rtshilston at gmail.com Wed Jul 7 06:49:58 2010 From: rtshilston at gmail.com (Rob S) Date: Wed, 07 Jul 2010 07:49:58 +0100 Subject: Can Varnish do this? In-Reply-To: References: Message-ID: <4C342396.8080105@gmail.com> Angelo H?ngens wrote: >> On Wed, Jul 7, 2010 at 12:10 AM, zabrane Mikael >> wrote: >> >>> Hi List, >>> I'm new to Varnish,...Thanks in advance. >>> > > So basically what you want to do is define two backends, and if your custom header is present in the request, you want it to go to one backend, and if not, you want to go to the other backend. > You'd write something like: backend normalhttp { .host = "a.b.c.d"; .port="80"; } backend customserver { .host = "w.x.y.z"; .port="80"; } sub vcl_recv { // Set the default backend: set req.backend = normalhttp; // But override it if this is a request containing X-RAWDATA if (req.http.X-RAWDATA!="") { set req.backend = customserver; } ... This code above isn't tested, but we use a similar approach happily in production. We inspect the http host, and redirect certain hosts to different backend director pools. Good luck! From bjorn at ruberg.no Wed Jul 7 07:02:57 2010 From: bjorn at ruberg.no (=?ISO-8859-1?Q?Bj=F8rn_Ruberg?=) Date: Wed, 07 Jul 2010 09:02:57 +0200 Subject: Can Varnish do this? In-Reply-To: References: Message-ID: <4C3426A1.20209@ruberg.no> On 07/07/2010 12:10 AM, zabrane Mikael wrote: > Hi List, > > I'm new to Varnish, so please apologize if the question was already > asked before. > > I've a homemade web server running on my intranet. It works like this: > > 1. If an HTTP request arrives without a special header "X-RAWDATA: ...", > this web server forward the request to the live Web, get the response, > and send it back (i.e it plays a role of a basic proxy). > 2. Now, if a request contains the special header "X-RAWDATA: ...", > then the web server will search the response in > its internal database (complex computations), and send it back. > > Point 2 works very very well. But not point 1 as my web server isn't a > true HTTP proxy. > > So, my question is simple: > Could I tell Varnish to inspect the HTTP headers, and decide: > > a. If the special header "X-RAWDATA: ..." doesn't exists, let the > request go to the live Web directly. > b. Otherwise, forward the request to my web server. > Varnish won't be able to search random, arbitrary web sites as you will need to predefine all backend. Each external web server will need to be defined as its own backend. If the live web requests are limited to a fixed number of external sites, you may define each of them in the VCL. If that is not the case, an elegant solution for your live web requests may be installing a regular proxy and define that as one backend; see http://ingvar.blog.linpro.no/2010/05/26/accelerating-the-internet-or-actually-squid-with-varnish/ for inspiration and details :-) Then, as others have also explained, you'll define two backends - one towards the regular proxy (e.g. Squid) and one towards your web server - and then make a VCL rule to define the backend based on whether the "X-RAWDATA" header is present or not. -- Bj?rn From perbu at varnish-software.com Wed Jul 7 07:22:36 2010 From: perbu at varnish-software.com (Per Buer) Date: Wed, 7 Jul 2010 09:22:36 +0200 Subject: Varnish constantly restarting In-Reply-To: <4C32BAB0.7020800@d6.com> References: <4C2F6CF4.3030705@yahoo.com.br> <4C32A5FD.6030202@yahoo.com.br> <4C32BAB0.7020800@d6.com> Message-ID: On Tue, Jul 6, 2010 at 7:10 AM, Chris Hecker wrote: > >> We had the same problem with the 2.0.6 version > > Ugh, this is the version that's installed on my CentOS box that I was about > to start using...are there known problems with this version? Not really. 2.0.6 is a very good release still. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From zabrane3 at gmail.com Wed Jul 7 08:28:30 2010 From: zabrane3 at gmail.com (zabrane Mikael) Date: Wed, 7 Jul 2010 10:28:30 +0200 Subject: Can Varnish do this? In-Reply-To: <4C3426A1.20209@ruberg.no> References: <4C3426A1.20209@ruberg.no> Message-ID: Wow guys ;-) Thanks for all these advices. I'll test them and report any progress. 2010/7/7 Bj?rn Ruberg > On 07/07/2010 12:10 AM, zabrane Mikael wrote: > >> Hi List, >> >> I'm new to Varnish, so please apologize if the question was already asked >> before. >> >> I've a homemade web server running on my intranet. It works like this: >> >> 1. If an HTTP request arrives without a special header "X-RAWDATA: ...", >> this web server forward the request to the live Web, get the response, and >> send it back (i.e it plays a role of a basic proxy). >> 2. Now, if a request contains the special header "X-RAWDATA: ...", then >> the web server will search the response in >> its internal database (complex computations), and send it back. >> >> Point 2 works very very well. But not point 1 as my web server isn't a >> true HTTP proxy. >> >> So, my question is simple: >> Could I tell Varnish to inspect the HTTP headers, and decide: >> >> a. If the special header "X-RAWDATA: ..." doesn't exists, let the request >> go to the live Web directly. >> b. Otherwise, forward the request to my web server. >> >> > Varnish won't be able to search random, arbitrary web sites as you will > need to predefine all backend. Each external web server will need to be > defined as its own backend. If the live web requests are limited to a fixed > number of external sites, you may define each of them in the VCL. > > If that is not the case, an elegant solution for your live web requests may > be installing a regular proxy and define that as one backend; see > http://ingvar.blog.linpro.no/2010/05/26/accelerating-the-internet-or-actually-squid-with-varnish/for inspiration and details :-) > > Then, as others have also explained, you'll define two backends - one > towards the regular proxy (e.g. Squid) and one towards your web server - and > then make a VCL rule to define the backend based on whether the "X-RAWDATA" > header is present or not. > > -- > Bj?rn > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- Regards Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From wrighty+varnishmisc at gmail.com Wed Jul 7 08:57:35 2010 From: wrighty+varnishmisc at gmail.com (Paul Wright) Date: Wed, 7 Jul 2010 09:57:35 +0100 Subject: Troubleshooting: Setting up varnish 2.1.2 on OpenSolaris In-Reply-To: References: Message-ID: On 7 July 2010 03:46, Jacques wrote: > Hello, > I'm trying to setup Solaris on OpenSolaris b134. ?I'm doing a setup in a > brand new zone. ?I have reviewed the instructions > on?http://www.varnish-cache.org/wiki/Installation#OpenSolaris and am having > problems. > With a brand new zone, I've found that in addition to installing the list of > packages provided, I also need to install:?SUNWpcre &?SUNWlibm. > Once installed, I can configure & build correctly with: > ---- > > VCC_CC="cc -Kpic -G -m64 -o %o %s" CC=/usr/bin/gcc CFLAGS="-O3 > -L/opt/extra/lib/amd64 \ > ? -pthreads -m64 -fomit-frame-pointer" LDFLAGS="-lumem -pthreads" > ./configure --prefix=/opt/extra > > make && make install > > ---- > I start by running: > # ?/opt/extra/sbin/varnishd -d?-a:80 -b 87.238.47.204:80 >>Compiled VCL program failed to load: >> ?ld.so.1: varnishd: fatal: relocation error: R_AMD64_PC32: file >> ./vcl.ORk8t3RP.so: symbol main: value 0x2800dcfbd44 does not fit >>VCL compilation failed > Not unexpected, I run: > # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80?-p cc_command='gcc > -G -m64 -o %o %s' >>Compiled VCL program failed to load: >> ?ld.so.1: varnishd: fatal: relocation error: R_AMD64_PC32: file >> ./vcl.ORk8t3RP.so: symbol main: value 0x2800dcdbd44 does not fit >>VCL compilation failed Hi Jacques, We're running r4516 on OpenSolaris snv_111b, I freely admit to cargo-culting here, my configure command similarly looks like this: CC=/usr/bin/gcc CFLAGS="-O3 -L/lib/amd64 -pthreads -m64 -fomit-frame-pointer" LDFLAGS="-lumem -pthreads" ./configure --prefix=/opt And then to run I pass through this option to varnishd when starting: -p 'cc_command=/usr/bin/gcc -fpic -shared -m64 -o %o %s' (I think the -fpic argument is what you're missing.) That mostly works for us with just gcc installed - I've tried later varnish revisions but the varnish tools segfault with "Cannot open" errors and it's not been a priority to investigate the cause. Hope that helps, Paul. From jdzstz at gmail.com Wed Jul 7 09:31:38 2010 From: jdzstz at gmail.com (=?ISO-8859-1?Q?Jorge_D=EDaz?=) Date: Wed, 7 Jul 2010 11:31:38 +0200 Subject: Troubleshooting: Setting up varnish 2.1.2 on OpenSolaris In-Reply-To: References: Message-ID: Hello Jacques, I have installed Varnish 2.1.2 in Solaris 10 (not OpenSolaris) and I also had some issues at the begining with GCC compiler, the VCL compilation and some varnish tools. With GCC 3.4.3 I had lot of problems compiling and I upgraded to GCC 3.4.6 (see http://varnish-cache.org/ticket/711 ) About ticket 649, LINGER crash [ http://varnish-cache.org/ticket/649 ] I was able to detect that is a issue of Solaris itself and can be fixed modifing some "asserts" in sources ( *S**olaris setsockopt returns sometimes EINVAL* ) the fixes are in http://varnish-cache.org/changeset/4868 and I think in future version 2.1.3 I also had some problems with varnishncsa logging (we have to keep a track of all request). Varnishncsa coredumped because a duplicated ReqEnd the solution is in http://varnish-cache.org/ticket/709 and also in http://varnish-cache.org/ticket/633 About your problem " ld.so.1: varnishd: fatal: relocation error: R_AMD64_PC32: " in google I have found this: - http://opensolaris.org/jive/thread.jspa?messageID=128240 - http://www.mail-archive.com/tools-discuss at opensolaris.org/msg02012.html - http://www.mail-archive.com/tools-discuss at opensolaris.org/msg02015.html They say to compile with -Kpic parameter, try changing cc_command After changing GCC, and applying solutions of ticket 649, 709, 711 and 633, Varnish is working OK in production. I think version 2.1.3 will fix all problems. Jorge 2010/7/7 Jacques > Hello, > > I'm trying to setup Solaris on OpenSolaris b134. I'm doing a setup in a > brand new zone. I have reviewed the instructions on > http://www.varnish-cache.org/wiki/Installation#OpenSolaris and am having > problems. > > With a brand new zone, I've found that in addition to installing the list > of packages provided, I also need to install: SUNWpcre & SUNWlibm. > > Once installed, I can configure & build correctly with: > ---- > > VCC_CC="cc -Kpic -G -m64 -o %o %s" CC=/usr/bin/gcc CFLAGS="-O3 > -L/opt/extra/lib/amd64 \ > -pthreads -m64 -fomit-frame-pointer" LDFLAGS="-lumem -pthreads" > ./configure --prefix=/opt/extra > > make && make install > ---- > > I start by running: > # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 > >Compiled VCL program failed to load: > > ld.so.1: varnishd: fatal: relocation error: R_AMD64_PC32: file ./ > vcl.ORk8t3RP.so: symbol main: value 0x2800dcfbd44 does not fit > >VCL compilation failed > > Not unexpected, I run: > # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 -p cc_command='gcc > -G -m64 -o %o %s' > >Compiled VCL program failed to load: > > ld.so.1: varnishd: fatal: relocation error: R_AMD64_PC32: file ./ > vcl.ORk8t3RP.so: symbol main: value 0x2800dcdbd44 does not fit > >VCL compilation failed > > So I install the sunstudioexpress compiler: > # pkg install sunstudioexpress > # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 -p cc_command=" > /opt/SunStudioExpress/bin/cc -c > -Kpic -m64 -o %o %s" > > >"./vcl.ORk8t3RP.c", line 356: warning: initializer will be sign-extended: > -1 > >storage_file: filename: ./varnish.BAaWXR size 461096 MB. > >Using old SHMFILE > >Varnish on -sfile,-hcritbit > >200 193 > >----------------------------- > >Varnish HTTP accelerator CLI. > >----------------------------- > >Type 'help' for command list. > >Type 'quit' to close CLI session. > >Type 'start' to launch worker process. > > Varnish appears to be running. However, I getting 503s when I try to > Varnish. > I saw that the warning was mentioned in a closed bug: > http://varnish-cache.org/ticket/649 > > So, can someone giving me some pointers? I really would like to use just > the gcc compiler and not have to load all the SunStudioExpress stuff. But > either way, I just want to make it work. > > I've reviewed the following resources and haven't solved my problem: > http://blogs.everycity.co.uk/alasdair/2009/05/64bit-varnish-on-solaris/ > http://letsgetdugg.com/2009/12/04/varnish-on-solaris/ > http://src.opensolaris.org/source/xref/webstack/varnish/ > > Thanks for any help, > Jacques > > Compiler Info: > [/usr/bin/gcc -v] gcc version 3.4.3 (csl-sol210-3_4-20050802) > [/usr/gnu/bin/cc] gcc version 3.4.3 (csl-sol210-3_4-20050802) [symlinked > to gcc] > [/opt/SunStudioExpress/bin/cc -V] cc: Sun Ceres C 5.10 SunOS_i386 > 2009/03/06 > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zabrane3 at gmail.com Wed Jul 7 14:49:42 2010 From: zabrane3 at gmail.com (zabrane Mikael) Date: Wed, 7 Jul 2010 16:49:42 +0200 Subject: Can Varnish do this? In-Reply-To: <4C342396.8080105@gmail.com> References: <4C342396.8080105@gmail.com> Message-ID: Hi again, > You'd write something like: > backend normalhttp { .host = "a.b.c.d"; .port="80"; } > backend customserver { .host = "w.x.y.z"; .port="80"; } > > sub vcl_recv { > > // Set the default backend: > set req.backend = normalhttp; > > // But override it if this is a request containing X-RAWDATA > if (req.http.X-RAWDATA!="") { > set req.backend = customserver; > } > ... > > This code above isn't tested, but we use a similar approach happily in > production. We inspect the http host, and redirect certain hosts to > different backend director pools. > Believe it or not, the solution works "like a charm" !!! Regards Zab -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger.rauch at empic.de Wed Jul 7 15:30:23 2010 From: holger.rauch at empic.de (Holger Rauch) Date: Wed, 7 Jul 2010 17:30:23 +0200 Subject: Varnish & ClamAV on FreeBSD 8 STABLE In-Reply-To: <87ocelxgl7.fsf@qurzaw.linpro.no> References: <20100615081204.GA16478@heitec.de><87ocelxgl7.fsf@qurzaw.linpro.no> Message-ID: <20100707153023.GA3664@heitec.de> Hi, thanks for your reply. Maybe I didn't clearly state my question: Does Varnish support ICAP (Internet Content Adaptation Protocol) at this time? Thanks in advance & kind regards, Holger On Tue, 06 Jul 2010, Tollef Fog Heen wrote: > ]] Holger Rauch > > Hi, > > | has anybody tried setting up Varnish in conjunction with ClamAV on > | FreeBSD 8 STABLE or, more generally asked, does Varnish support some > | kind of "interface" to virus scanners? > > There's currently no (easy) way to access the contents of objects you > are serving, so no. > [...] -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: From perbu at varnish-software.com Wed Jul 7 18:01:24 2010 From: perbu at varnish-software.com (Per Buer) Date: Wed, 7 Jul 2010 20:01:24 +0200 Subject: Varnish & ClamAV on FreeBSD 8 STABLE In-Reply-To: <20100707153023.GA3664@heitec.de> References: <20100615081204.GA16478@heitec.de> <87ocelxgl7.fsf@qurzaw.linpro.no> <20100707153023.GA3664@heitec.de> Message-ID: No, sorry. Not on the roadmap either at this time. Regards, Per. On Wed, Jul 7, 2010 at 5:30 PM, Holger Rauch wrote: > Hi, > > thanks for your reply. Maybe I didn't clearly state my question: Does > Varnish support ICAP (Internet Content Adaptation Protocol) at this > time? > > Thanks in advance & kind regards, > > ? ?Holger > > On Tue, 06 Jul 2010, Tollef Fog Heen wrote: > >> ]] Holger Rauch >> >> Hi, >> >> | has anybody tried setting up Varnish in conjunction with ClamAV on >> | FreeBSD 8 STABLE or, more generally asked, does Varnish support some >> | kind of "interface" to virus scanners? >> >> There's currently no (easy) way to access the contents of objects you >> are serving, so no. >> [...] > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > > iEYEARECAAYFAkw0nY8ACgkQbiVtWpZdKQLTPACbBlRg42w0O8STR8DQaUjo0ecx > tD0AoI6n7lZMB7CoGI1ROChNDD8Xtuxe > =X5fN > -----END PGP SIGNATURE----- > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From whshub at gmail.com Wed Jul 7 20:47:21 2010 From: whshub at gmail.com (Jacques) Date: Wed, 7 Jul 2010 13:47:21 -0700 Subject: Troubleshooting: Setting up varnish 2.1.2 on OpenSolaris In-Reply-To: References: Message-ID: So the fpic solved my problem with getting the -1 warning. However, I'm getting nothing but 503 responses from varnish. I've updated to using this for configure: VCC_CC="cc -fpic -shared -G -m64 -o %o %s" \ CC=/usr/bin/gcc \ CFLAGS="-O3 -pthreads -m64 -fomit-frame-pointer"\ LDFLAGS="-lumem -pthreads" \ ./configure --prefix=/opt/extra The updated VCC_CC line means I don't have to pass it every time I start varnish. The simplest run: /opt/extra/sbin/varnishd -F -a:80 -b 87.238.47.204:80 I always get 503s. When I check the vmlog, I see backend server not available. To further test, I upped the timeout values in a basic vcl just in case that was the problem. No change. My question is, how can I figure out why I'm getting backend server not available? If I add a probe, I still see no outbound requests and backends start at healthy and then become sick shortly after (where they stay). Running tcpdump, I see no communication with the backend server during any of this. Where is my error? Thanks, Jacques On Wed, Jul 7, 2010 at 2:31 AM, Jorge D?az wrote: > Hello Jacques, > > I have installed Varnish 2.1.2 in Solaris 10 (not OpenSolaris) and I also > had some issues at the begining with GCC compiler, the VCL compilation and > some varnish tools. > > With GCC 3.4.3 I had lot of problems compiling and I upgraded to GCC 3.4.6 > (see http://varnish-cache.org/ticket/711 ) > > About ticket 649, LINGER crash [ http://varnish-cache.org/ticket/649 ] I > was able to detect that is a issue of Solaris itself and can be fixed > modifing some "asserts" in sources ( *S**olaris setsockopt returns > sometimes EINVAL* ) the fixes are in > http://varnish-cache.org/changeset/4868 and I think in future version > 2.1.3 > > I also had some problems with varnishncsa logging (we have to keep a track > of all request). Varnishncsa coredumped because a duplicated ReqEnd the > solution is in http://varnish-cache.org/ticket/709 and also in > http://varnish-cache.org/ticket/633 > > About your problem " ld.so.1: varnishd: fatal: relocation error: > R_AMD64_PC32: " in google I have found this: > > > - http://opensolaris.org/jive/thread.jspa?messageID=128240 > - > http://www.mail-archive.com/tools-discuss at opensolaris.org/msg02012.html > - > http://www.mail-archive.com/tools-discuss at opensolaris.org/msg02015.html > > > They say to compile with -Kpic parameter, try changing cc_command > > After changing GCC, and applying solutions of ticket 649, 709, 711 and 633, > Varnish is working OK in production. I think version 2.1.3 will fix all > problems. > > > Jorge > > > 2010/7/7 Jacques > >> Hello, >> >> I'm trying to setup Solaris on OpenSolaris b134. I'm doing a setup in a >> brand new zone. I have reviewed the instructions on >> http://www.varnish-cache.org/wiki/Installation#OpenSolaris and am having >> problems. >> >> With a brand new zone, I've found that in addition to installing the list >> of packages provided, I also need to install: SUNWpcre & SUNWlibm. >> >> Once installed, I can configure & build correctly with: >> ---- >> >> VCC_CC="cc -Kpic -G -m64 -o %o %s" CC=/usr/bin/gcc CFLAGS="-O3 >> -L/opt/extra/lib/amd64 \ >> -pthreads -m64 -fomit-frame-pointer" LDFLAGS="-lumem -pthreads" >> ./configure --prefix=/opt/extra >> >> make && make install >> ---- >> >> I start by running: >> # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 >> >Compiled VCL program failed to load: >> > ld.so.1: varnishd: fatal: relocation error: R_AMD64_PC32: file ./ >> vcl.ORk8t3RP.so: symbol main: value 0x2800dcfbd44 does not fit >> >VCL compilation failed >> >> Not unexpected, I run: >> # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 -p >> cc_command='gcc -G -m64 -o %o %s' >> >Compiled VCL program failed to load: >> > ld.so.1: varnishd: fatal: relocation error: R_AMD64_PC32: file ./ >> vcl.ORk8t3RP.so: symbol main: value 0x2800dcdbd44 does not fit >> >VCL compilation failed >> >> So I install the sunstudioexpress compiler: >> # pkg install sunstudioexpress >> # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 -p cc_command=" >> /opt/SunStudioExpress/bin/cc -c >> -Kpic -m64 -o %o %s" >> >> >"./vcl.ORk8t3RP.c", line 356: warning: initializer will be sign-extended: >> -1 >> >storage_file: filename: ./varnish.BAaWXR size 461096 MB. >> >Using old SHMFILE >> >Varnish on -sfile,-hcritbit >> >200 193 >> >----------------------------- >> >Varnish HTTP accelerator CLI. >> >----------------------------- >> >Type 'help' for command list. >> >Type 'quit' to close CLI session. >> >Type 'start' to launch worker process. >> >> Varnish appears to be running. However, I getting 503s when I try to >> Varnish. >> I saw that the warning was mentioned in a closed bug: >> http://varnish-cache.org/ticket/649 >> >> So, can someone giving me some pointers? I really would like to use just >> the gcc compiler and not have to load all the SunStudioExpress stuff. But >> either way, I just want to make it work. >> >> I've reviewed the following resources and haven't solved my problem: >> http://blogs.everycity.co.uk/alasdair/2009/05/64bit-varnish-on-solaris/ >> http://letsgetdugg.com/2009/12/04/varnish-on-solaris/ >> http://src.opensolaris.org/source/xref/webstack/varnish/ >> >> Thanks for any help, >> Jacques >> >> Compiler Info: >> [/usr/bin/gcc -v] gcc version 3.4.3 (csl-sol210-3_4-20050802) >> [/usr/gnu/bin/cc] gcc version 3.4.3 (csl-sol210-3_4-20050802) [symlinked >> to gcc] >> [/opt/SunStudioExpress/bin/cc -V] cc: Sun Ceres C 5.10 SunOS_i386 >> 2009/03/06 >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From whshub at gmail.com Wed Jul 7 21:45:38 2010 From: whshub at gmail.com (Jacques) Date: Wed, 7 Jul 2010 14:45:38 -0700 Subject: Troubleshooting: Setting up varnish 2.1.2 on OpenSolaris In-Reply-To: References: Message-ID: As an aside, I saw this exact same behavior using 4516, 2.1.2 with the linger patch, and trunk. On Wed, Jul 7, 2010 at 1:47 PM, Jacques wrote: > So the fpic solved my problem with getting the -1 warning. However, I'm > getting nothing but 503 responses from varnish. > > I've updated to using this for configure: > VCC_CC="cc -fpic -shared -G -m64 -o %o %s" \ > CC=/usr/bin/gcc \ > CFLAGS="-O3 -pthreads -m64 -fomit-frame-pointer"\ > LDFLAGS="-lumem -pthreads" \ > ./configure --prefix=/opt/extra > > The updated VCC_CC line means I don't have to pass it every time I start > varnish. > > The simplest run: > /opt/extra/sbin/varnishd -F -a:80 -b 87.238.47.204:80 > > I always get 503s. When I check the vmlog, I see backend server not > available. To further test, I upped the timeout values in a basic vcl just > in case that was the problem. No change. > > My question is, how can I figure out why I'm getting backend server not > available? If I add a probe, I still see no outbound requests and backends > start at healthy and then become sick shortly after (where they stay). > > Running tcpdump, I see no communication with the backend server during any > of this. > > Where is my error? > > Thanks, > Jacques > > > On Wed, Jul 7, 2010 at 2:31 AM, Jorge D?az wrote: > >> Hello Jacques, >> >> I have installed Varnish 2.1.2 in Solaris 10 (not OpenSolaris) and I also >> had some issues at the begining with GCC compiler, the VCL compilation and >> some varnish tools. >> >> With GCC 3.4.3 I had lot of problems compiling and I upgraded to GCC >> 3.4.6 (see http://varnish-cache.org/ticket/711 ) >> >> About ticket 649, LINGER crash [ http://varnish-cache.org/ticket/649 ] >> I was able to detect that is a issue of Solaris itself and can be fixed >> modifing some "asserts" in sources ( *S**olaris setsockopt returns >> sometimes EINVAL* ) the fixes are in >> http://varnish-cache.org/changeset/4868 and I think in future version >> 2.1.3 >> >> I also had some problems with varnishncsa logging (we have to keep a track >> of all request). Varnishncsa coredumped because a duplicated ReqEnd the >> solution is in http://varnish-cache.org/ticket/709 and also in >> http://varnish-cache.org/ticket/633 >> >> About your problem " ld.so.1: varnishd: fatal: relocation error: >> R_AMD64_PC32: " in google I have found this: >> >> >> - http://opensolaris.org/jive/thread.jspa?messageID=128240 >> - >> http://www.mail-archive.com/tools-discuss at opensolaris.org/msg02012.html >> - >> http://www.mail-archive.com/tools-discuss at opensolaris.org/msg02015.html >> >> >> They say to compile with -Kpic parameter, try changing cc_command >> >> After changing GCC, and applying solutions of ticket 649, 709, 711 and >> 633, Varnish is working OK in production. I think version 2.1.3 will fix all >> problems. >> >> >> Jorge >> >> >> 2010/7/7 Jacques >> >>> Hello, >>> >>> I'm trying to setup Solaris on OpenSolaris b134. I'm doing a setup in a >>> brand new zone. I have reviewed the instructions on >>> http://www.varnish-cache.org/wiki/Installation#OpenSolaris and am having >>> problems. >>> >>> With a brand new zone, I've found that in addition to installing the list >>> of packages provided, I also need to install: SUNWpcre & SUNWlibm. >>> >>> Once installed, I can configure & build correctly with: >>> ---- >>> >>> VCC_CC="cc -Kpic -G -m64 -o %o %s" CC=/usr/bin/gcc CFLAGS="-O3 >>> -L/opt/extra/lib/amd64 \ >>> -pthreads -m64 -fomit-frame-pointer" LDFLAGS="-lumem -pthreads" >>> ./configure --prefix=/opt/extra >>> >>> make && make install >>> ---- >>> >>> I start by running: >>> # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 >>> >Compiled VCL program failed to load: >>> > ld.so.1: varnishd: fatal: relocation error: R_AMD64_PC32: file ./ >>> vcl.ORk8t3RP.so: symbol main: value 0x2800dcfbd44 does not fit >>> >VCL compilation failed >>> >>> Not unexpected, I run: >>> # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 -p >>> cc_command='gcc -G -m64 -o %o %s' >>> >Compiled VCL program failed to load: >>> > ld.so.1: varnishd: fatal: relocation error: R_AMD64_PC32: file ./ >>> vcl.ORk8t3RP.so: symbol main: value 0x2800dcdbd44 does not fit >>> >VCL compilation failed >>> >>> So I install the sunstudioexpress compiler: >>> # pkg install sunstudioexpress >>> # /opt/extra/sbin/varnishd -d -a:80 -b 87.238.47.204:80 -p cc_command=" >>> /opt/SunStudioExpress/bin/cc -c >>> -Kpic -m64 -o %o %s" >>> >>> >"./vcl.ORk8t3RP.c", line 356: warning: initializer will be >>> sign-extended: -1 >>> >storage_file: filename: ./varnish.BAaWXR size 461096 MB. >>> >Using old SHMFILE >>> >Varnish on -sfile,-hcritbit >>> >200 193 >>> >----------------------------- >>> >Varnish HTTP accelerator CLI. >>> >----------------------------- >>> >Type 'help' for command list. >>> >Type 'quit' to close CLI session. >>> >Type 'start' to launch worker process. >>> >>> Varnish appears to be running. However, I getting 503s when I try to >>> Varnish. >>> I saw that the warning was mentioned in a closed bug: >>> http://varnish-cache.org/ticket/649 >>> >>> So, can someone giving me some pointers? I really would like to use just >>> the gcc compiler and not have to load all the SunStudioExpress stuff. But >>> either way, I just want to make it work. >>> >>> I've reviewed the following resources and haven't solved my problem: >>> http://blogs.everycity.co.uk/alasdair/2009/05/64bit-varnish-on-solaris/ >>> http://letsgetdugg.com/2009/12/04/varnish-on-solaris/ >>> http://src.opensolaris.org/source/xref/webstack/varnish/ >>> >>> Thanks for any help, >>> Jacques >>> >>> Compiler Info: >>> [/usr/bin/gcc -v] gcc version 3.4.3 (csl-sol210-3_4-20050802) >>> [/usr/gnu/bin/cc] gcc version 3.4.3 (csl-sol210-3_4-20050802) [symlinked >>> to gcc] >>> [/opt/SunStudioExpress/bin/cc -V] cc: Sun Ceres C 5.10 SunOS_i386 >>> 2009/03/06 >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rednasy at gmail.com Thu Jul 8 01:45:06 2010 From: rednasy at gmail.com (Andrey Nepomnyaschih) Date: Wed, 7 Jul 2010 20:45:06 -0500 Subject: Accept-Language Issue In-Reply-To: References: Message-ID: Ken, Laurence thank you. It was indeed my typo. On Thu, Jul 1, 2010 at 5:37 PM, Laurence Rowe wrote: > Check that you do have the comma in Plone's caching control panel. I can > imagine it being an easy typo to make. > > There are many unique variations of Accept-Language headers, so be careful. > I've used this before in vcl_recv to normalise the incoming request: > > # We only care about the language in the I18N_LANGUAGE cookie > if (req.http.Cookie ~ "(^|.*; )I18N_LANGUAGE=") { > set req.http.Accept-Language = regsub(req.http.Cookie, "(^|.*; > )I18N_LANGUAGE=([^;]*)(; .*|$)", "\2"); > # XXX need to work out the proper way to match " here, e.g. "en" > set req.http.Accept-Language = regsub(req.http.Accept-Language, > "^.(.*).$", "\1"); > } else { > set req.http.Accept-Language = "en"; > } > > Laurence > > On 1 July 2010 22:04, Ken Brownfield > > wrote: > >> I do notice that the Vary header is malformed: >> >> Vary: User-Agent Accept-Language >> >> It should be comma-separated AFAIK. Also, all of the lines from your last >> header dump have appended periods... not sure that's the actual output or >> just a paste glitch. >> >> Perhaps Varnish is being very strict about malformed headers and dumping >> the connection immediately. Which is probably a good thing, if true. >> -- >> Ken >> >> On Jul 1, 2010, at 1:54 PM, Andrey Nepomnyaschih wrote: >> >> Hello, >> >> Can somebody give me a hint for what am I doing wrong with my setup. I >> have Zope instance that is proxied by nginx (doing address rewriting) that I >> want to be cached by varnish, but as soon as I configure Zope to add >> Accept-Language to Vary header, varnish starts closing connection when asked >> for object from Zope without returning any data. I'm using default.vcl >> >> Using tcpdump doesn't reveal anything useful, the only difference is >> Accept-Language value in Vary header. >> >> Request that works: >> >> # GET -H "Host: hostname.domain.tld" -Used http://hostname.domain.tld >> GET http://hostname.domain.tld >> Host: hostname.domain.tld >> User-Agent: lwp-request/5.834 libwww-perl/5.836 >> >> 200 OK >> Wide character in print at /usr/local/bin/GET line 439. >> Cache-Control: max-age=0, s-maxage=0, private, must-revalidate >> Connection: close >> Date: Thu, 01 Jul 2010 20:30:22 GMT >> Via: 1.1 varnish >> Age: 0 >> ETag: ||Plone Default||0|181||||355004 >> Server: nginx/0.7.67 >> Vary: User-Agent >> Content-Language: en >> Content-Length: 5485 >> Content-Type: text/html;charset=utf-8 >> Expires: Mon, 03 Jul 2000 20:30:22 GMT >> Client-Date: Thu, 01 Jul 2010 20:30:22 GMT >> Client-Peer: 82.103.128.123:80 >> Client-Response-Num: 1 >> Content-Base: http://hostname.domain.tld/plonesite/front-page >> Imagetoolbar: no >> Link: ; >> rel="kss-base-url" >> Link: < >> http://hostname.domain.tld/plonesite/portal_kss/Plone%20Default/at-cachekey6690.kss>; >> rel="kinetic-stylesheet"; type="text/css" >> Link: ; rel="shortcut >> icon"; type="image/x-icon" >> Link: ; rel="home"; title="Front >> page" >> Link: ; rel="contents"; >> title="Site Map" >> Link: ; rel="search"; >> title="Search this site" >> Title: Plone Site! >> X-Cache-Headers-Set-By: CachingPolicyManager: >> /plonesite/caching_policy_manager >> X-Caching-Rule-Id: plone-content-types >> X-Header-Set-Id: cache-in-memory >> X-Meta-Generator: Plone - http://plone.org >> X-Pagecache: MISS >> X-UA-Compatible: IE=edge >> X-Varnish: 1366563744 >> >> Results in the following request to nginx: >> >> 2010/07/01 22:30:22.196300 127.0.0.1:25266 -> 127.0.0.1:8080 [AP] >> GET / HTTP/1.1. >> Host: hostname.domain.tld. >> User-Agent: lwp-request/5.834 libwww-perl/5.836. >> X-Forwarded-For: 82.103.128.123. >> X-Varnish: 1366563744. >> >> And the following reply from nginx: >> >> T 2010/07/01 22:30:22.267204 127.0.0.1:8080 -> 127.0.0.1:25266 [AP] >> HTTP/1.1 200 OK. >> Server: nginx/0.7.67. >> Date: Thu, 01 Jul 2010 20:30:22 GMT. >> Content-Type: text/html;charset=utf-8. >> Connection: keep-alive. >> X-Pagecache: MISS. >> Content-Length: 5485. >> Content-Language: en. >> X-Cache-Headers-Set-By: CachingPolicyManager: >> /plonesite/caching_policy_manager. >> Expires: Mon, 03 Jul 2000 20:30:22 GMT. >> Vary: User-Agent. >> ETag: ||Plone Default||0|181||||355004. >> X-Caching-Rule-Id: plone-content-types. >> Cache-Control: max-age=0, s-maxage=0, private, must-revalidate. >> X-Header-Set-Id: cache-in-memory. >> >> Request that doesn't work: >> >> [root at rain /usr/local/etc/varnish]# GET -H "Host: hostname.domain.tld" >> -Used http://hostname.domain.tld >> GET http://hostname.domain.tld >> Host: hostname.domain.tld >> User-Agent: lwp-request/5.834 libwww-perl/5.836 >> >> 500 Server closed connection without sending any data back >> Content-Type: text/plain >> Client-Date: Thu, 01 Jul 2010 20:27:54 GMT >> Client-Warning: Internal response >> >> Results in the following request to nginx: >> >> 2010/07/01 22:27:54.507294 127.0.0.1:50795 -> 127.0.0.1:8080 [AP] >> GET / HTTP/1.1. >> Host: hostname.domain.tld. >> User-Agent: lwp-request/5.834 libwww-perl/5.836. >> X-Forwarded-For: 82.103.128.123. >> X-Varnish: 1656652273. >> >> And the following reply from nginx: >> >> 2010/07/01 22:27:54.573671 127.0.0.1:8080 -> 127.0.0.1:50795 [AP] >> HTTP/1.1 200 OK. >> Server: nginx/0.7.67. >> Date: Thu, 01 Jul 2010 20:27:54 GMT. >> Content-Type: text/html;charset=utf-8. >> Connection: keep-alive. >> X-Pagecache: MISS. >> Content-Length: 5485. >> Content-Language: en. >> X-Cache-Headers-Set-By: CachingPolicyManager: >> /plonesite/caching_policy_manager. >> Expires: Mon, 03 Jul 2000 20:27:54 GMT. >> Vary: User-Agent Accept-Language. >> ETag: ||Plone Default||0|180||||355004. >> X-Caching-Rule-Id: plone-content-types. >> Cache-Control: max-age=0, s-maxage=0, private, must-revalidate. >> X-Header-Set-Id: cache-in-memory. >> >> Kind Regards, >> Andrey Nepomnyaschih >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zabrane3 at gmail.com Thu Jul 8 05:37:19 2010 From: zabrane3 at gmail.com (zabrane Mikael) Date: Thu, 8 Jul 2010 07:37:19 +0200 Subject: New challenge for Varnish! Message-ID: Hi List, Yesterday with Varnish users help (thanks again guys), I succeeded with an extreme simplicity to solve the problem I described here: http://lists.varnish-cache.org/pipermail/varnish-misc/2010-July/004428.html Today, I've a new challenge for Varnish to extend the capabilties of my customserver !!! My current "customserver.vcl" contains 2 backends: backend squid { .host = "127.0.0.1"; .port = "3128"; } backend customserver { .host = "127.0.0.1"; .port = "80"; } sub vcl_recv { // set default backend set req.backend = squid; if (req.http.X-RAWDATA != "") { set req.backend = customserver; } } Challenge: 1. If an HTTP request arrives with a special header "X-STORE-IN-FILE: UUID", I'd like Varnish to get the response data from "squid" backend, save the "raw HTTP response" on disk under filename "UUID" before sending it back to the client. 2. If (1) is doable, I'd like Varnish to save both "raw HTTP request and response" (by "raw", I mean HTTP hearders and payload) in the file "UUID". Any pointer, hint, or piece of VCL will be very welcome! Thanks in advance. NOTE: UUID is a Universally Unique Identifier used in my context to have unique filenames on disk. -- Regards Zab -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtshilston at gmail.com Thu Jul 8 06:28:40 2010 From: rtshilston at gmail.com (Rob S) Date: Thu, 08 Jul 2010 07:28:40 +0100 Subject: New challenge for Varnish! In-Reply-To: References: Message-ID: <4C357018.6000803@gmail.com> Zab, I don't think that's going to be possible with a pure VCL approach. Can you give us a better idea of exactly what you're trying to achieve, as there might be a simpler / more Varnish-friendly way of achieving what you're after. You can certainly use varnishlog to see the full headers for a given UUID, but the body content is more difficult. Rob zabrane Mikael wrote: > Hi List, > > Yesterday with Varnish users help (thanks again guys), I succeeded > with an extreme simplicity to solve the problem I described here: > http://lists.varnish-cache.org/pipermail/varnish-misc/2010-July/004428.html > > Today, I've a new challenge for Varnish to extend the capabilties of > my customserver !!! > > My current "customserver.vcl" contains 2 backends: > backend squid { > .host = "127.0.0.1"; > .port = "3128"; > } > > backend customserver { > .host = "127.0.0.1"; > .port = "80"; > } > > sub vcl_recv { > // set default backend > set req.backend = squid; > > if (req.http.X-RAWDATA != "") { > set req.backend = customserver; > } > } > > Challenge: > > 1. If an HTTP request arrives with a special header "X-STORE-IN-FILE: > UUID", > I'd like Varnish to get the response data from "squid" backend, save > the "raw HTTP response" on disk under filename "UUID" before sending > it back to the client. > > 2. If (1) is doable, I'd like Varnish to save both "raw HTTP request > and response" (by "raw", I mean HTTP hearders and payload) in the > file "UUID". > > Any pointer, hint, or piece of VCL will be very welcome! > Thanks in advance. > > NOTE: UUID is a Universally Unique Identifier used in my context to > have unique filenames on disk. > > -- > Regards > Zab > ------------------------------------------------------------------------ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From zabrane3 at gmail.com Thu Jul 8 07:11:21 2010 From: zabrane3 at gmail.com (zabrane Mikael) Date: Thu, 8 Jul 2010 09:11:21 +0200 Subject: New challenge for Varnish! In-Reply-To: <4C357018.6000803@gmail.com> References: <4C357018.6000803@gmail.com> Message-ID: Hi Rob, I don't think that's going to be possible with a pure VCL approach. Can you > give us a better idea of exactly what you're trying to achieve, as there > might be a simpler / more Varnish-friendly way of achieving what you're > after. > To make it simple, I'd like to store for a document A, all its raw headers (both request and response) for later processing. A cron job will collect them (i.e these stored files) and send them to a complex analysis system. You can certainly use varnishlog to see the full headers for a given UUID I'll give it try! but the body content is more difficult. Is there another way to achieve that? Maybe by using some C to implement a callback for handling raw "request" and "response". Regards Zab -------------- next part -------------- An HTML attachment was scrubbed... URL: From rtshilston at gmail.com Thu Jul 8 07:44:17 2010 From: rtshilston at gmail.com (Rob S) Date: Thu, 08 Jul 2010 08:44:17 +0100 Subject: New challenge for Varnish! In-Reply-To: References: <4C357018.6000803@gmail.com> Message-ID: <4C3581D1.2030101@gmail.com> zabrane Mikael wrote: > Hi Rob, > > I don't think that's going to be possible with a pure VCL > approach. Can you give us a better idea of exactly what you're > trying to achieve, as there might be a simpler / more > Varnish-friendly way of achieving what you're after. > > > To make it simple, I'd like to store for a document A, all its raw > headers (both request and response) for later processing. > A cron job will collect them (i.e these stored files) and send them to > a complex analysis system. > > You can certainly use varnishlog to see the full headers for a > given UUID > > > I'll give it try! > > but the body content is more difficult. > > > Is there another way to achieve that? > Maybe by using some C to implement a callback for handling raw > "request" and "response". You could use some C, but I don't know enough to comment about race conditions between concurrent requests if the same UUID came in at about the same time, or how to work gracefully if Varnish queued some concurrent requests to the same resource. If the UUID isn't that important, I'd consider doing the following: Set up additional backends, on 127.0.0.2, called 'loggedsquid' and 'loggedcustomerserver'. Then, approximately have: // set default backend set req.backend = squid; if (req.http.X-RAWDATA != "") { set req.backend = customserver; } if (req.http.X-STORE!= "") { set req.backend = loggedsquid; if (req.http.X-RAWDATA != "") { set req.backend = loggedcustomserver; } } THEN, on the varnish server, run (approximately) " tcpdump -s 0 -w loggedtraffic.pcap -d lo ip and host 127.0.0.2 " Finally, on the cron, you could parse the pcap file, as that'd have the full headers and body, all in one coherent place. Indeed, if the UUID is essential, there's no reason why you couldn't parse it out of the pcap file, work out the tcp port number for that request, then reparse the pcap to write out the packet contents to a specific file for that UUID. I suppose it all depends on how regularly this logging would happen. Rob From zabrane3 at gmail.com Thu Jul 8 08:05:41 2010 From: zabrane3 at gmail.com (zabrane Mikael) Date: Thu, 8 Jul 2010 10:05:41 +0200 Subject: New challenge for Varnish! In-Reply-To: <4C3581D1.2030101@gmail.com> References: <4C357018.6000803@gmail.com> <4C3581D1.2030101@gmail.com> Message-ID: Thanks Rob. Let me give it a try! 2010/7/8 Rob S > zabrane Mikael wrote: > >> Hi Rob, >> >> I don't think that's going to be possible with a pure VCL >> approach. Can you give us a better idea of exactly what you're >> trying to achieve, as there might be a simpler / more >> Varnish-friendly way of achieving what you're after. >> >> >> To make it simple, I'd like to store for a document A, all its raw headers >> (both request and response) for later processing. >> A cron job will collect them (i.e these stored files) and send them to a >> complex analysis system. >> >> You can certainly use varnishlog to see the full headers for a >> given UUID >> >> >> I'll give it try! >> >> but the body content is more difficult. >> >> >> Is there another way to achieve that? >> Maybe by using some C to implement a callback for handling raw "request" >> and "response". >> > You could use some C, but I don't know enough to comment about race > conditions between concurrent requests if the same UUID came in at about the > same time, or how to work gracefully if Varnish queued some concurrent > requests to the same resource. > If the UUID isn't that important, I'd consider doing the following: > > Set up additional backends, on 127.0.0.2, called 'loggedsquid' and > 'loggedcustomerserver'. Then, approximately have: > > > > // set default backend > set req.backend = squid; > > if (req.http.X-RAWDATA != "") { > set req.backend = customserver; > } > > if (req.http.X-STORE!= "") { > set req.backend = loggedsquid; > > > if (req.http.X-RAWDATA != "") { > set req.backend = loggedcustomserver; > } > } > > THEN, on the varnish server, run (approximately) " tcpdump -s 0 -w > loggedtraffic.pcap -d lo ip and host 127.0.0.2 " > > Finally, on the cron, you could parse the pcap file, as that'd have the > full headers and body, all in one coherent place. Indeed, if the UUID is > essential, there's no reason why you couldn't parse it out of the pcap file, > work out the tcp port number for that request, then reparse the pcap to > write out the packet contents to a specific file for that UUID. > > I suppose it all depends on how regularly this logging would happen. > > > Rob > -- Regards Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From whshub at gmail.com Fri Jul 9 05:08:50 2010 From: whshub at gmail.com (Jacques) Date: Thu, 8 Jul 2010 22:08:50 -0700 Subject: What are the steps to problem solve when all I can get are 503s? Message-ID: I am trying to use Varnish on OpenSolaris. Due the help of others, I have versions compiling and not crashing that are based on three different versions of Varnish. I'm using 4516, 2.1.2 with the 649 patch, and trunk. I've also tried compiling with gcc 4.3.3 in addition to the OpenSolaris standard of 3.4.3. That changed nothing. All compiled copies of varnish seem to start just fine. I can use varnishlog and see my requests arriving at Varnish. However, responses are always 503 with " FetchError: no backend connection". Using tcpdump, I see no calls to the backend server from Varnish, ever. For simplicity, I'm using the basic start of: varnishd -a :80 -b 87.238.47.204:80 What are right steps to determine what is wrong? I see nothing in syslogs. Varnish doesn't complain. Using all the same steps on ubuntu, I get exactly what I expect, a cached version of the backend server. Any pointers about how I should figure out what is failing would be helpful. Thanks, Jacques I've also noticed that the cache is starting at 500GB. Changing the cache size to 512MB didn't have any impact on the 503s. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bedis9 at gmail.com Fri Jul 9 06:27:00 2010 From: bedis9 at gmail.com (Bedis 9) Date: Fri, 9 Jul 2010 08:27:00 +0200 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: Hey, Can you share your VCL conf? cheers On Fri, Jul 9, 2010 at 7:08 AM, Jacques wrote: > I am trying to use Varnish on OpenSolaris. Due the help of others, I have > versions compiling and not crashing that are based on three different > versions of Varnish. > > I'm using 4516, 2.1.2 with the 649 patch, and trunk. I've also tried > compiling with gcc 4.3.3 in addition to the OpenSolaris standard of 3.4.3. > That changed nothing. > > All compiled copies of varnish seem to start just fine. I can use > varnishlog and see my requests arriving at Varnish. > > However, responses are always 503 with " FetchError: no backend > connection". Using tcpdump, I see no calls to the backend server from > Varnish, ever. > > For simplicity, I'm using the basic start of: > > varnishd -a :80 -b 87.238.47.204:80 > > What are right steps to determine what is wrong? I see nothing in syslogs. > Varnish doesn't complain. Using all the same steps on ubuntu, I get > exactly what I expect, a cached version of the backend server. > > Any pointers about how I should figure out what is failing would be > helpful. > > Thanks, > Jacques > > I've also noticed that the cache is starting at 500GB. Changing the cache > size to 512MB didn't have any impact on the 503s. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdzstz at gmail.com Fri Jul 9 07:58:20 2010 From: jdzstz at gmail.com (=?ISO-8859-1?Q?Jorge_D=EDaz?=) Date: Fri, 9 Jul 2010 09:58:20 +0200 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: ?There is communication between Varnish host and 87.238.47.204 machine?? Try first making " telnet 87.238.47.204 80 " in order testing there is no firewall or a rule configured in webserver. If comunication is fine, the problem must be in VCL configuration. Try executing varnishlog while you are testing and look for errors in output. In Solaris is also important to run Varnish with proper parameter that are listen in: - http://letsgetdugg.com/2009/12/04/varnish-on-solaris/ I had a lot of comunication problems without parameter: * -p connect_timeout=0s # Important bug work around for Solaris* 2010/7/9 Bedis 9 > Hey, > > Can you share your VCL conf? > > cheers > > On Fri, Jul 9, 2010 at 7:08 AM, Jacques wrote: > >> I am trying to use Varnish on OpenSolaris. Due the help of others, I have >> versions compiling and not crashing that are based on three different >> versions of Varnish. >> >> I'm using 4516, 2.1.2 with the 649 patch, and trunk. I've also tried >> compiling with gcc 4.3.3 in addition to the OpenSolaris standard of 3.4.3. >> That changed nothing. >> >> All compiled copies of varnish seem to start just fine. I can use >> varnishlog and see my requests arriving at Varnish. >> >> However, responses are always 503 with " FetchError: no backend >> connection". Using tcpdump, I see no calls to the backend server from >> Varnish, ever. >> >> For simplicity, I'm using the basic start of: >> >> varnishd -a :80 -b 87.238.47.204:80 >> >> What are right steps to determine what is wrong? I see nothing in >> syslogs. Varnish doesn't complain. Using all the same steps on ubuntu, I >> get exactly what I expect, a cached version of the backend server. >> >> Any pointers about how I should figure out what is failing would be >> helpful. >> >> Thanks, >> Jacques >> >> I've also noticed that the cache is starting at 500GB. Changing the cache >> size to 512MB didn't have any impact on the 503s. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From checker at d6.com Fri Jul 9 11:01:42 2010 From: checker at d6.com (Chris Hecker) Date: Fri, 09 Jul 2010 04:01:42 -0700 Subject: http purging multiple accept-encoding entries Message-ID: <4C370196.9050106@d6.com> I'm trying to put varnish in front of mediawiki, but there are a couple things going wrong with the purging support built into mediawiki (which supposedly supports varnish and squid). 1. Mediawiki was sending back an HTTP/1.0 PURGE command, which had no Host: header, just the full url after the PURGE, so the hash function wasn't matching with the normal HTTP/1.1 requests. I patched mediawiki to fix this and send an HTTP/1.1 PURGE with a Host: header, not sure what a better way to do this would be. 2. The PURGE does not have an Accept-encoding: on it, so it doesn't match the cached object, which is almost always A-e: gzip. I hacked a gzip header onto the PURGE, but what I really want is to purge all the records associated with an URL. Is there a way to do this? Thanks, Chris From l at lrowe.co.uk Fri Jul 9 11:56:36 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Fri, 9 Jul 2010 12:56:36 +0100 Subject: http purging multiple accept-encoding entries In-Reply-To: <4C370196.9050106@d6.com> References: <4C370196.9050106@d6.com> Message-ID: On 9 July 2010 12:01, Chris Hecker wrote: > > I'm trying to put varnish in front of mediawiki, but there are a couple > things going wrong with the purging support built into mediawiki (which > supposedly supports varnish and squid). > > 1. ?Mediawiki was sending back an HTTP/1.0 PURGE command, which had no Host: > header, just the full url after the PURGE, so the hash function wasn't > matching with the normal HTTP/1.1 requests. ?I patched mediawiki to fix this > and send an HTTP/1.1 PURGE with a Host: header, not sure what a better way > to do this would be. > > 2. ?The PURGE does not have an Accept-encoding: on it, so it doesn't match > the cached object, which is almost always A-e: gzip. ?I hacked a gzip header > onto the PURGE, but what I really want is to purge all the records > associated with an URL. ?Is there a way to do this? See: http://varnish-cache.org/wiki/VCLExamplePurging Using purge("req.url == " req.url); should fix both problems. Laurence From holger.rauch at empic.de Fri Jul 9 15:05:33 2010 From: holger.rauch at empic.de (Holger Rauch) Date: Fri, 9 Jul 2010 17:05:33 +0200 Subject: Transparent Proxying with Varnish and PF Message-ID: <20100709150533.GA16002@heitec.de> Hi to everybody, is transparent proxying with/via PF supported in Varnish? Thanks in advance & kind regards, Holger -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: From jdixon at omniti.com Fri Jul 9 15:08:01 2010 From: jdixon at omniti.com (Jason Dixon) Date: Fri, 9 Jul 2010 11:08:01 -0400 Subject: Last day to submit your Surge 2010 CFP! Message-ID: <20100709150801.GC4133@omniti.com> Today is your last chance to submit a CFP abstract for the 2010 Surge Scalability Conference. The event is taking place on Sept 30 and Oct 1, 2010 in Baltimore, MD. Surge focuses on case studies that address production failures and the re-engineering efforts that led to victory in Web Applications or Internet Architectures. You can find more information, including suggested topics and our current list of speakers, online: http://omniti.com/surge/2010 The final lineup should be available on the conference website next week. If you have questions about the CFP, attending Surge, or having your business sponsor/exhibit at Surge 2010, please contact us at surge at omniti.com. Thanks! -- Jason Dixon OmniTI Computer Consulting, Inc. jdixon at omniti.com 443.325.1357 x.241 From whshub at gmail.com Fri Jul 9 15:18:20 2010 From: whshub at gmail.com (Jacques) Date: Fri, 9 Jul 2010 08:18:20 -0700 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: I'm not currently using a vcl, just configuring via the command line for simplicity. Adding the connect_timeout parameter below had no impact on the 503s. Nor did adding all the parameters listed at letsdugg.com. Communication with the external server works perfectly. (Telnet and via lynx.) It's actually the varnish-cache.org site just to introduce a minimum number of variables. The only thing that I see in the varnish log is the FetchError, no backend connection. Here is a startup, request, shutdown cycle of varnish log: # /opt/extra/bin/varnishlog 0 WorkThread - fffffd7ff9dc0d20 start 0 CLI - Rd vcl.load "boot" ./vcl.ORk8t3RP.so 0 CLI - Wr 200 Loaded "./vcl.ORk8t3RP.so" as "boot" 0 CLI - Rd vcl.use "boot" 0 CLI - Wr 200 0 CLI - Rd start 0 Debug - "Acceptor poll space increased to 512" 0 Debug - "Acceptor is poll" 0 CLI - Wr 200 0 WorkThread - fffffd7ff93c5d20 start 0 WorkThread - fffffd7ff91c6d20 start 0 WorkThread - fffffd7ff8fc7d20 start 0 WorkThread - fffffd7ff8dc8d20 start 0 WorkThread - fffffd7ff8bc9d20 start 0 WorkThread - fffffd7ff89cad20 start 0 WorkThread - fffffd7ff87cbd20 start 0 WorkThread - fffffd7ff85ccd20 start 0 WorkThread - fffffd7ff83cdd20 start 0 CLI - Rd ping 0 CLI - Wr 200 PONG 1278687217 1.0 0 CLI - Rd ping 0 CLI - Wr 200 PONG 1278687220 1.0 9 SessionOpen c 10.100.200.1 64024 :80 9 ReqStart c 10.100.200.1 64024 1455837036 9 RxRequest c GET 9 RxURL c / 9 RxProtocol c HTTP/1.0 9 RxHeader c Host: 10.100.200.2 9 RxHeader c Accept: text/html, text/plain, text/css, text/sgml, */*;q=0.01 9 RxHeader c Accept-Encoding: gzip, compress, bzip2 9 RxHeader c Accept-Language: en 9 RxHeader c User-Agent: Lynx/2.8.6rel.5 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.8l 9 VCL_call c recv 9 VCL_return c lookup 9 VCL_call c hash 9 VCL_return c hash 9 VCL_call c miss 9 VCL_return c fetch 9 FetchError c no backend connection 9 VCL_call c error 9 VCL_return c deliver 9 Length c 488 9 VCL_call c deliver 9 VCL_return c deliver 9 TxProtocol c HTTP/1.1 9 TxStatus c 503 9 TxResponse c Service Unavailable 9 TxHeader c Server: Varnish 9 TxHeader c Retry-After: 0 9 TxHeader c Content-Type: text/html; charset=utf-8 9 TxHeader c Content-Length: 488 9 TxHeader c Date: Fri, 09 Jul 2010 14:53:42 GMT 9 TxHeader c X-Varnish: 1455837036 9 TxHeader c Age: 0 9 TxHeader c Via: 1.1 varnish 9 TxHeader c Connection: close 9 ReqEnd c 1455837036 1278687222.201183081 1278687222.201552391 0.000498056 0.000254393 0.000114918 9 SessionClose c error 9 StatSess c 10.100.200.1 64024 0 1 1 0 0 0 235 488 0 CLI - Rd ping 0 CLI - Wr 200 PONG 1278687223 1.0 0 CLI - Rd ping 0 CLI - Wr 200 PONG 1278687226 1.0 0 CLI - Rd ping 0 CLI - Wr 200 PONG 1278687229 1.0 On Fri, Jul 9, 2010 at 12:58 AM, Jorge D?az wrote: > ?There is communication between Varnish host and 87.238.47.204 machine?? > > Try first making " telnet 87.238.47.204 80 " in order testing there is no > firewall or a rule configured in webserver. > > If comunication is fine, the problem must be in VCL configuration. Try > executing varnishlog while you are testing and look for errors in output. > > In Solaris is also important to run Varnish with proper parameter that are > listen in: > > - http://letsgetdugg.com/2009/12/04/varnish-on-solaris/ > > > I had a lot of comunication problems without parameter: > > * -p connect_timeout=0s # Important bug work around for Solaris* > > > > > > 2010/7/9 Bedis 9 > > Hey, >> >> Can you share your VCL conf? >> >> cheers >> >> On Fri, Jul 9, 2010 at 7:08 AM, Jacques wrote: >> >>> I am trying to use Varnish on OpenSolaris. Due the help of others, I >>> have versions compiling and not crashing that are based on three different >>> versions of Varnish. >>> >>> I'm using 4516, 2.1.2 with the 649 patch, and trunk. I've also tried >>> compiling with gcc 4.3.3 in addition to the OpenSolaris standard of 3.4.3. >>> That changed nothing. >>> >>> All compiled copies of varnish seem to start just fine. I can use >>> varnishlog and see my requests arriving at Varnish. >>> >>> However, responses are always 503 with " FetchError: no backend >>> connection". Using tcpdump, I see no calls to the backend server from >>> Varnish, ever. >>> >>> For simplicity, I'm using the basic start of: >>> >>> varnishd -a :80 -b 87.238.47.204:80 >>> >>> What are right steps to determine what is wrong? I see nothing in >>> syslogs. Varnish doesn't complain. Using all the same steps on ubuntu, I >>> get exactly what I expect, a cached version of the backend server. >>> >>> Any pointers about how I should figure out what is failing would be >>> helpful. >>> >>> Thanks, >>> Jacques >>> >>> I've also noticed that the cache is starting at 500GB. Changing the >>> cache size to 512MB didn't have any impact on the 503s. >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkferreira at gmail.com Fri Jul 9 16:39:10 2010 From: rkferreira at gmail.com (Rodrigo K. Ferreira) Date: Fri, 9 Jul 2010 13:39:10 -0300 Subject: About saintmode_threshold behavior Message-ID: Hi, About the error counters what is compared with saintmode_threshold, when it counter is back to zero ? Just when that backend server are penalized ? Or always after one backend probe ? This questions is why is a bit normal dinamic backends servers returns few 5XX errors, for client reqs bad formed or other reasons. And if isnt back to zero, backend servers will be labeled sick in some time. Thanks, Rodrigo Kellermann Ferreira -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdzstz at gmail.com Sat Jul 10 07:47:42 2010 From: jdzstz at gmail.com (=?ISO-8859-1?Q?Jorge_D=EDaz?=) Date: Sat, 10 Jul 2010 09:47:42 +0200 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: It seems for some reason Varnish cannot connect to backend. I have never used parameter " -b 87.238.47.204:80 " for backend configuration. You sould try creating a little VCL configuration file. Modify default.vcl adding the backend in it, and start varnish with parameter -f -f /opt/extra/etc/varnish/default.vcl Inside VCL, configure the backed with: backend default { .host = "87.238.47.204"; .port = "80"; } If the problem persist, a good idea is to install a Apache webserver in localhost and try It as backend, for example in 127.0.0.1 port 8000 in order to see if the problem is varnish connecting to a remote IP or is varnish that not connect to nobody. My varnish installation is in same host of Apache webserver and works OK. Regards. 2010/7/9 Jacques > I'm not currently using a vcl, just configuring via the command line for > simplicity. Adding the connect_timeout parameter below had no impact on the > 503s. Nor did adding all the parameters listed at letsdugg.com. > > Communication with the external server works perfectly. (Telnet and via > lynx.) It's actually the varnish-cache.org site just to introduce a > minimum number of variables. > > The only thing that I see in the varnish log is the FetchError, no backend > connection. > > Here is a startup, request, shutdown cycle of varnish log: > > # /opt/extra/bin/varnishlog > 0 WorkThread - fffffd7ff9dc0d20 start > 0 CLI - Rd vcl.load "boot" ./vcl.ORk8t3RP.so > 0 CLI - Wr 200 Loaded "./vcl.ORk8t3RP.so" as "boot" > 0 CLI - Rd vcl.use "boot" > 0 CLI - Wr 200 > 0 CLI - Rd start > 0 Debug - "Acceptor poll space increased to 512" > 0 Debug - "Acceptor is poll" > 0 CLI - Wr 200 > 0 WorkThread - fffffd7ff93c5d20 start > 0 WorkThread - fffffd7ff91c6d20 start > 0 WorkThread - fffffd7ff8fc7d20 start > 0 WorkThread - fffffd7ff8dc8d20 start > 0 WorkThread - fffffd7ff8bc9d20 start > 0 WorkThread - fffffd7ff89cad20 start > 0 WorkThread - fffffd7ff87cbd20 start > 0 WorkThread - fffffd7ff85ccd20 start > 0 WorkThread - fffffd7ff83cdd20 start > 0 CLI - Rd ping > 0 CLI - Wr 200 PONG 1278687217 1.0 > 0 CLI - Rd ping > 0 CLI - Wr 200 PONG 1278687220 1.0 > 9 SessionOpen c 10.100.200.1 64024 :80 > 9 ReqStart c 10.100.200.1 64024 1455837036 > 9 RxRequest c GET > 9 RxURL c / > 9 RxProtocol c HTTP/1.0 > 9 RxHeader c Host: 10.100.200.2 > 9 RxHeader c Accept: text/html, text/plain, text/css, text/sgml, > */*;q=0.01 > 9 RxHeader c Accept-Encoding: gzip, compress, bzip2 > 9 RxHeader c Accept-Language: en > 9 RxHeader c User-Agent: Lynx/2.8.6rel.5 libwww-FM/2.14 > SSL-MM/1.4.1 OpenSSL/0.9.8l > 9 VCL_call c recv > 9 VCL_return c lookup > 9 VCL_call c hash > 9 VCL_return c hash > 9 VCL_call c miss > 9 VCL_return c fetch > 9 FetchError c no backend connection > 9 VCL_call c error > 9 VCL_return c deliver > 9 Length c 488 > 9 VCL_call c deliver > 9 VCL_return c deliver > 9 TxProtocol c HTTP/1.1 > 9 TxStatus c 503 > 9 TxResponse c Service Unavailable > 9 TxHeader c Server: Varnish > 9 TxHeader c Retry-After: 0 > 9 TxHeader c Content-Type: text/html; charset=utf-8 > 9 TxHeader c Content-Length: 488 > 9 TxHeader c Date: Fri, 09 Jul 2010 14:53:42 GMT > 9 TxHeader c X-Varnish: 1455837036 > 9 TxHeader c Age: 0 > 9 TxHeader c Via: 1.1 varnish > 9 TxHeader c Connection: close > 9 ReqEnd c 1455837036 1278687222.201183081 1278687222.201552391 > 0.000498056 0.000254393 0.000114918 > 9 SessionClose c error > 9 StatSess c 10.100.200.1 64024 0 1 1 0 0 0 235 488 > 0 CLI - Rd ping > 0 CLI - Wr 200 PONG 1278687223 1.0 > 0 CLI - Rd ping > 0 CLI - Wr 200 PONG 1278687226 1.0 > 0 CLI - Rd ping > 0 CLI - Wr 200 PONG 1278687229 1.0 > > > > > > > On Fri, Jul 9, 2010 at 12:58 AM, Jorge D?az wrote: > >> ?There is communication between Varnish host and 87.238.47.204 machine?? >> >> Try first making " telnet 87.238.47.204 80 " in order testing there is no >> firewall or a rule configured in webserver. >> >> If comunication is fine, the problem must be in VCL configuration. Try >> executing varnishlog while you are testing and look for errors in output. >> >> In Solaris is also important to run Varnish with proper parameter that are >> listen in: >> >> - http://letsgetdugg.com/2009/12/04/varnish-on-solaris/ >> >> >> I had a lot of comunication problems without parameter: >> >> * -p connect_timeout=0s # Important bug work around for Solaris* >> >> >> >> >> >> 2010/7/9 Bedis 9 >> >> Hey, >>> >>> Can you share your VCL conf? >>> >>> cheers >>> >>> On Fri, Jul 9, 2010 at 7:08 AM, Jacques wrote: >>> >>>> I am trying to use Varnish on OpenSolaris. Due the help of others, I >>>> have versions compiling and not crashing that are based on three different >>>> versions of Varnish. >>>> >>>> I'm using 4516, 2.1.2 with the 649 patch, and trunk. I've also tried >>>> compiling with gcc 4.3.3 in addition to the OpenSolaris standard of 3.4.3. >>>> That changed nothing. >>>> >>>> All compiled copies of varnish seem to start just fine. I can use >>>> varnishlog and see my requests arriving at Varnish. >>>> >>>> However, responses are always 503 with " FetchError: no backend >>>> connection". Using tcpdump, I see no calls to the backend server from >>>> Varnish, ever. >>>> >>>> For simplicity, I'm using the basic start of: >>>> >>>> varnishd -a :80 -b 87.238.47.204:80 >>>> >>>> What are right steps to determine what is wrong? I see nothing in >>>> syslogs. Varnish doesn't complain. Using all the same steps on ubuntu, I >>>> get exactly what I expect, a cached version of the backend server. >>>> >>>> Any pointers about how I should figure out what is failing would be >>>> helpful. >>>> >>>> Thanks, >>>> Jacques >>>> >>>> I've also noticed that the cache is starting at 500GB. Changing the >>>> cache size to 512MB didn't have any impact on the 503s. >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Sat Jul 10 07:04:51 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 10 Jul 2010 07:04:51 +0000 Subject: Transparent Proxying with Varnish and PF In-Reply-To: Your message of "Fri, 09 Jul 2010 17:05:33 +0200." <20100709150533.GA16002@heitec.de> Message-ID: <3130.1278745491@critter.freebsd.dk> In message <20100709150533.GA16002 at heitec.de>, Holger Rauch writes: >is transparent proxying with/via PF supported in Varnish? Server-side: yes, client-side: no. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From checker at d6.com Sat Jul 10 08:24:42 2010 From: checker at d6.com (Chris Hecker) Date: Sat, 10 Jul 2010 01:24:42 -0700 Subject: http purging multiple accept-encoding entries In-Reply-To: References: <4C370196.9050106@d6.com> Message-ID: <4C382E4A.5020508@d6.com> Awesome, this worked great, thanks! The one caveat that others might run into was that sometimes the backend would sent / as the url, and I was using the regexp ~ version from the link you sent, so it would clear the whole cache. I changed it to this: purge("req.url ~ " req.url "$"); # use regexp to handle all Accept-Encoding's error 200 "Purged in recv."; and now it works perfectly. Chris On 2010/07/09 04:56, Laurence Rowe wrote: > On 9 July 2010 12:01, Chris Hecker wrote: >> >> I'm trying to put varnish in front of mediawiki, but there are a couple >> things going wrong with the purging support built into mediawiki (which >> supposedly supports varnish and squid). >> >> 1. Mediawiki was sending back an HTTP/1.0 PURGE command, which had no Host: >> header, just the full url after the PURGE, so the hash function wasn't >> matching with the normal HTTP/1.1 requests. I patched mediawiki to fix this >> and send an HTTP/1.1 PURGE with a Host: header, not sure what a better way >> to do this would be. >> >> 2. The PURGE does not have an Accept-encoding: on it, so it doesn't match >> the cached object, which is almost always A-e: gzip. I hacked a gzip header >> onto the PURGE, but what I really want is to purge all the records >> associated with an URL. Is there a way to do this? > > See: http://varnish-cache.org/wiki/VCLExamplePurging > > Using purge("req.url == " req.url); should fix both problems. > > Laurence > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From checker at d6.com Sat Jul 10 08:58:04 2010 From: checker at d6.com (Chris Hecker) Date: Sat, 10 Jul 2010 01:58:04 -0700 Subject: misc questions Message-ID: <4C38361C.9040105@d6.com> 1. Is there a way to set the default ttl inside the vcl? 2. What's the recommended way of "disabling" varnish quickly for testing? Can I just return pass at the top of recv and not worry about other stuff, or do I need to change all the functions? Or, should I make a separate vcl file that's just a passthrough and not muck with my real one? 3. If I've got mediawiki and wordpress (my two backends) purging correctly on edit, what should the ttl be on the content pages? It seems like I should set it to something high, right? mediawiki sets it to 18000s, but wp doesn't set it as far as I can tell from the log. 4. If I've got varnish working in front of wordpress, I assume I don't need any of the caching plugins or any other hacks like those, right? Chris PS. I'm on 32 bit CentOS 5.5 and varnish 2.0.6, and varnish seems to be working fine so far. In fact, it's blazing fast compared to httpd + APC, like hundreds of times faster in my tests so far. It took forever for me to get my stuff set up the way I want, and I still think I have wordpress cookie issues, but in siege tests it's just incredibly fast and uses very little cpu/mem. The true test will be when I get redditted again, I guess. From phk at phk.freebsd.dk Sat Jul 10 09:04:37 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 10 Jul 2010 09:04:37 +0000 Subject: misc questions In-Reply-To: Your message of "Sat, 10 Jul 2010 01:58:04 MST." <4C38361C.9040105@d6.com> Message-ID: <3646.1278752677@critter.freebsd.dk> In message <4C38361C.9040105 at d6.com>, Chris Hecker writes: > >1. Is there a way to set the default ttl inside the vcl? No, only as a parameter. You can however, set the ttl of any object you want. >2. What's the recommended way of "disabling" varnish quickly for >testing? Can I just return pass at the top of recv Yes. >3. If I've got mediawiki and wordpress (my two backends) purging >correctly on edit, what should the ttl be on the content pages? It >seems like I should set it to something high, right? mediawiki sets it >to 18000s, but wp doesn't set it as far as I can tell from the log. As I mentioned above, you can set the ttl in your VCL file, it all depends on your traffic pattern really, but of course the longer you cache, the better your hitrate. The question is how much longer and how much better. >4. If I've got varnish working in front of wordpress, I assume I don't >need any of the caching plugins or any other hacks like those, right? Well, depends if you have many private pages that varnish decides to "pass" etc. >PS. I'm on 32 bit CentOS 5.5 and varnish 2.0.6, and varnish seems to be >working fine so far. If you expect high loads, 32 bit may cramp your style. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From checker at d6.com Sat Jul 10 09:49:21 2010 From: checker at d6.com (Chris Hecker) Date: Sat, 10 Jul 2010 02:49:21 -0700 Subject: misc questions In-Reply-To: <3646.1278752677@critter.freebsd.dk> References: <3646.1278752677@critter.freebsd.dk> Message-ID: <4C384221.6070301@d6.com> > No, only as a parameter. You can however, set the ttl of any object > you want. Yeah, but I couldn't figure out how to test if one had already been set. "if (obj.ttl)" didn't work. >> Can I just return pass at the top of recv > Yes. Awesome, thanks. That works perfectly. > you cache, the better your hitrate. The question is how much longer > and how much better. Yeah, my use case is a single page getting slashdotted, not general high traffic all the time distributed over all the pages, so I guess it doesn't matter too much. I have the default set to 600s right now, which makes it easier to test than 120s. > Well, depends if you have many private pages that varnish decides to > "pass" etc. Hmm, can you explain this, or tell me how I can detect this case? Should I look in the log for Cache-Control with private? > If you expect high loads, 32 bit may cramp your style. I'll have to see. It's such a gigantic improvement over the previous situation it's hard to even compare. Now that I have the "pass;" trick, I can compare both cases even more easily. Here is with pass;, so no caching, hitting the MediaWiki backend: HTTP/1.1 200 6.67 secs: 68579 bytes ==> /Sandbox/TestSiege Lifting the server siege... done. Transactions: 190 hits Availability: 48.10 % Elapsed time: 23.73 secs Data transferred: 12.43 MB Response time: 5.53 secs Transaction rate: 8.01 trans/sec Throughput: 0.52 MB/sec Concurrency: 44.31 Successful transactions: 190 Failed transactions: 205 Longest transaction: 11.49 Shortest transaction: 0.29 This is similar to what I was seeing without varnish, so I think the pass; is not costing much at all. CPU is at 99%, and the machine is completely unresponsive, with top filled with httpd processes. And here is varnish in the way: HTTP/1.1 200 0.06 secs: 68579 bytes ==> /Sandbox/TestSiege Lifting the server siege... done. Transactions: 1531 hits Availability: 100.00 % Elapsed time: 9.84 secs Data transferred: 100.13 MB Response time: 0.12 secs Transaction rate: 155.59 trans/sec Throughput: 10.18 MB/sec Concurrency: 19.18 Successful transactions: 1531 Failed transactions: 0 Longest transaction: 0.76 Shortest transaction: 0.06 It's a 100mbps link, so varnish is saturating it. And cpu is at something like 2%. You can't even tell the machine is being sieged. On a smaller page: HTTP/1.1 200 3.91 secs: 12189 bytes ==> /Homepage Lifting the server siege... done. Transactions: 122 hits Availability: 17.97 % Elapsed time: 16.50 secs Data transferred: 1.42 MB Response time: 4.23 secs Transaction rate: 7.39 trans/sec Throughput: 0.09 MB/sec Concurrency: 31.30 Successful transactions: 122 Failed transactions: 557 Longest transaction: 9.32 Shortest transaction: 0.00 HTTP/1.1 200 0.05 secs: 12189 bytes ==> /Homepage Lifting the server siege... done. Transactions: 3861 hits Availability: 100.00 % Elapsed time: 7.79 secs Data transferred: 45.93 MB Response time: 0.15 secs Transaction rate: 495.64 trans/sec Throughput: 5.90 MB/sec Concurrency: 75.27 Successful transactions: 3951 Failed transactions: 0 Longest transaction: 0.74 Shortest transaction: 0.05 I think this is bound on the siege side, on a faster machine sieging I got ~800 txn/sec and 8MBps on this page. This is a crappy 2.8ghz p4 with two cores and 1gb of ram. Do these numbers look reasonable to people for that machine type? The cache is file,1G Chris On 2010/07/10 02:04, Poul-Henning Kamp wrote: > In message<4C38361C.9040105 at d6.com>, Chris Hecker writes: >> >> 1. Is there a way to set the default ttl inside the vcl? > > No, only as a parameter. You can however, set the ttl of any > object you want. > >> 2. What's the recommended way of "disabling" varnish quickly for >> testing? Can I just return pass at the top of recv > > Yes. > >> 3. If I've got mediawiki and wordpress (my two backends) purging >> correctly on edit, what should the ttl be on the content pages? It >> seems like I should set it to something high, right? mediawiki sets it >> to 18000s, but wp doesn't set it as far as I can tell from the log. > > As I mentioned above, you can set the ttl in your VCL file, it all > depends on your traffic pattern really, but of course the longer > you cache, the better your hitrate. The question is how much longer > and how much better. > >> 4. If I've got varnish working in front of wordpress, I assume I don't >> need any of the caching plugins or any other hacks like those, right? > > Well, depends if you have many private pages that varnish decides to > "pass" etc. > >> PS. I'm on 32 bit CentOS 5.5 and varnish 2.0.6, and varnish seems to be >> working fine so far. > > If you expect high loads, 32 bit may cramp your style. > From phk at phk.freebsd.dk Sat Jul 10 10:28:55 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 10 Jul 2010 10:28:55 +0000 Subject: misc questions In-Reply-To: Your message of "Sat, 10 Jul 2010 02:49:21 MST." <4C384221.6070301@d6.com> Message-ID: <4028.1278757735@critter.freebsd.dk> In message <4C384221.6070301 at d6.com>, Chris Hecker writes: >Yeah, but I couldn't figure out how to test if one had already been set. > "if (obj.ttl)" didn't work. By the time you get to the VCL code, a TTL will have been set, if nothing else the default ttl. You can check for the default ttl, by setting it to a magic value and testing against that. >> you cache, the better your hitrate. The question is how much longer >> and how much better. > >Yeah, my use case is a single page getting slashdotted, not general high >traffic all the time distributed over all the pages, so I guess it >doesn't matter too much. I have the default set to 600s right now, >which makes it easier to test than 120s. Consider enabling "grace" with a few seconds, to cover any updates to that page. >> Well, depends if you have many private pages that varnish decides to >> "pass" etc. > >Hmm, can you explain this, or tell me how I can detect this case? >Should I look in the log for Cache-Control with private? Varnish will pass anything with cookies or Authorization headers. >This is a crappy 2.8ghz p4 with two cores and 1gb of ram. Do these >numbers look reasonable to people for that machine type? The cache is >file,1G Absolutely reasonable, 100mbit/s should never be a problem for varnish. Try running "varnishhist" and see the responstime difference between hits and misses... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From l at lrowe.co.uk Sat Jul 10 14:22:52 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Sat, 10 Jul 2010 15:22:52 +0100 Subject: http purging multiple accept-encoding entries In-Reply-To: <4C382E4A.5020508@d6.com> References: <4C370196.9050106@d6.com> <4C382E4A.5020508@d6.com> Message-ID: That example is only for handling wildcard purges. The one above uses a direct comparison, which is what you want: purge("req.url == " req.url); Your regexp is not doing what you think it's doing either, purging "/" will purge "/foo/" and "/foo/bar/" as well. You need to anchor it to the beginning of the string too: purge("req.url ~ ^" req.url "$"); This should allow wild card purges to work if you need them. Laurence On 10 July 2010 09:24, Chris Hecker wrote: > > Awesome, this worked great, thanks! > > The one caveat that others might run into was that sometimes the backend > would sent / as the url, and I was using the regexp ~ version from the link > you sent, so it would clear the whole cache. ?I changed it to this: > > ? ? ? ? ? ? ? ?purge("req.url ~ " req.url "$"); ?# use regexp to handle all > Accept-Encoding's > ? ? ? ? ? ? ? ?error 200 "Purged in recv."; > > and now it works perfectly. > > Chris > > > On 2010/07/09 04:56, Laurence Rowe wrote: >> >> On 9 July 2010 12:01, Chris Hecker ?wrote: >>> >>> I'm trying to put varnish in front of mediawiki, but there are a couple >>> things going wrong with the purging support built into mediawiki (which >>> supposedly supports varnish and squid). >>> >>> 1. ?Mediawiki was sending back an HTTP/1.0 PURGE command, which had no >>> Host: >>> header, just the full url after the PURGE, so the hash function wasn't >>> matching with the normal HTTP/1.1 requests. ?I patched mediawiki to fix >>> this >>> and send an HTTP/1.1 PURGE with a Host: header, not sure what a better >>> way >>> to do this would be. >>> >>> 2. ?The PURGE does not have an Accept-encoding: on it, so it doesn't >>> match >>> the cached object, which is almost always A-e: gzip. ?I hacked a gzip >>> header >>> onto the PURGE, but what I really want is to purge all the records >>> associated with an URL. ?Is there a way to do this? >> >> See: http://varnish-cache.org/wiki/VCLExamplePurging >> >> Using purge("req.url == " req.url); should fix both problems. >> >> Laurence >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > From checker at d6.com Sat Jul 10 23:39:17 2010 From: checker at d6.com (Chris Hecker) Date: Sat, 10 Jul 2010 16:39:17 -0700 Subject: http purging multiple accept-encoding entries In-Reply-To: References: <4C370196.9050106@d6.com> <4C382E4A.5020508@d6.com> Message-ID: <4C3904A5.2050308@d6.com> Ack, you're totally right. Brain was addled from 10 hours of server nonsense! I will do the ==. Somebody should update the page. Oh, wait, the reason I didn't do the == was because one backend sends a full HTTP/1.0 style url in the PURGE, and the other sends a Host: and just the relative path name. Hmm. Chris On 2010/07/10 07:22, Laurence Rowe wrote: > That example is only for handling wildcard purges. The one above uses > a direct comparison, which is what you want: > > purge("req.url == " req.url); > > Your regexp is not doing what you think it's doing either, purging "/" > will purge "/foo/" and "/foo/bar/" as well. You need to anchor it to > the beginning of the string too: > > purge("req.url ~ ^" req.url "$"); > > This should allow wild card purges to work if you need them. > > Laurence > > On 10 July 2010 09:24, Chris Hecker wrote: >> >> Awesome, this worked great, thanks! >> >> The one caveat that others might run into was that sometimes the backend >> would sent / as the url, and I was using the regexp ~ version from the link >> you sent, so it would clear the whole cache. I changed it to this: >> >> purge("req.url ~ " req.url "$"); # use regexp to handle all >> Accept-Encoding's >> error 200 "Purged in recv."; >> >> and now it works perfectly. >> >> Chris >> >> >> On 2010/07/09 04:56, Laurence Rowe wrote: >>> >>> On 9 July 2010 12:01, Chris Hecker wrote: >>>> >>>> I'm trying to put varnish in front of mediawiki, but there are a couple >>>> things going wrong with the purging support built into mediawiki (which >>>> supposedly supports varnish and squid). >>>> >>>> 1. Mediawiki was sending back an HTTP/1.0 PURGE command, which had no >>>> Host: >>>> header, just the full url after the PURGE, so the hash function wasn't >>>> matching with the normal HTTP/1.1 requests. I patched mediawiki to fix >>>> this >>>> and send an HTTP/1.1 PURGE with a Host: header, not sure what a better >>>> way >>>> to do this would be. >>>> >>>> 2. The PURGE does not have an Accept-encoding: on it, so it doesn't >>>> match >>>> the cached object, which is almost always A-e: gzip. I hacked a gzip >>>> header >>>> onto the PURGE, but what I really want is to purge all the records >>>> associated with an URL. Is there a way to do this? >>> >>> See: http://varnish-cache.org/wiki/VCLExamplePurging >>> >>> Using purge("req.url == " req.url); should fix both problems. >>> >>> Laurence >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> >> > From checker at d6.com Sun Jul 11 05:48:07 2010 From: checker at d6.com (Chris Hecker) Date: Sat, 10 Jul 2010 22:48:07 -0700 Subject: http purging multiple accept-encoding entries In-Reply-To: <4C3904A5.2050308@d6.com> References: <4C370196.9050106@d6.com> <4C382E4A.5020508@d6.com> <4C3904A5.2050308@d6.com> Message-ID: <4C395B17.2090904@d6.com> Hmm, okay, I switched to purge("req.url == " req.url) and it seems to be working, but I'm slightly confused as to why... My two purges look like this: mediawiki: 15 RxRequest c PURGE 15 RxURL c http://www.chrishecker.com/Homepage 15 RxProtocol c HTTP/1.0 15 RxHeader c Connection: Keep-Alive wordpress + wp-varnish: 15 RxRequest c PURGE 15 RxURL c /2010/07/09/test-post/ 15 RxProtocol c HTTP/1.0 15 RxHeader c Host: spyparty.com 15 RxHeader c Connection: Close Am I correct in assuming req.url is the same as RxURL? If so, how are both the full http://blah url and the /blah url matching the url in the cache exactly (which is what == means, I assume). Or, is req.url expanded to the full url (meaning with the host added), not verbatim whatever comes in on the header? My version of varnish (2.0.6) doesn't have the log statement, so I can't figure out how to put req.url into a header that will get printed to the log in the purge request so I can find out for myself. req is not available in vcl_deliver either, and purge bypasses everything else, etc. Thanks, Chris On 2010/07/10 16:39, Chris Hecker wrote: > > Ack, you're totally right. Brain was addled from 10 hours of server > nonsense! I will do the ==. Somebody should update the page. > > Oh, wait, the reason I didn't do the == was because one backend sends a > full HTTP/1.0 style url in the PURGE, and the other sends a Host: and > just the relative path name. Hmm. > > Chris > > > On 2010/07/10 07:22, Laurence Rowe wrote: >> That example is only for handling wildcard purges. The one above uses >> a direct comparison, which is what you want: >> >> purge("req.url == " req.url); >> >> Your regexp is not doing what you think it's doing either, purging "/" >> will purge "/foo/" and "/foo/bar/" as well. You need to anchor it to >> the beginning of the string too: >> >> purge("req.url ~ ^" req.url "$"); >> >> This should allow wild card purges to work if you need them. >> >> Laurence >> >> On 10 July 2010 09:24, Chris Hecker wrote: >>> >>> Awesome, this worked great, thanks! >>> >>> The one caveat that others might run into was that sometimes the backend >>> would sent / as the url, and I was using the regexp ~ version from >>> the link >>> you sent, so it would clear the whole cache. I changed it to this: >>> >>> purge("req.url ~ " req.url "$"); # use regexp to handle all >>> Accept-Encoding's >>> error 200 "Purged in recv."; >>> >>> and now it works perfectly. >>> >>> Chris >>> >>> >>> On 2010/07/09 04:56, Laurence Rowe wrote: >>>> >>>> On 9 July 2010 12:01, Chris Hecker wrote: >>>>> >>>>> I'm trying to put varnish in front of mediawiki, but there are a >>>>> couple >>>>> things going wrong with the purging support built into mediawiki >>>>> (which >>>>> supposedly supports varnish and squid). >>>>> >>>>> 1. Mediawiki was sending back an HTTP/1.0 PURGE command, which had no >>>>> Host: >>>>> header, just the full url after the PURGE, so the hash function wasn't >>>>> matching with the normal HTTP/1.1 requests. I patched mediawiki to fix >>>>> this >>>>> and send an HTTP/1.1 PURGE with a Host: header, not sure what a better >>>>> way >>>>> to do this would be. >>>>> >>>>> 2. The PURGE does not have an Accept-encoding: on it, so it doesn't >>>>> match >>>>> the cached object, which is almost always A-e: gzip. I hacked a gzip >>>>> header >>>>> onto the PURGE, but what I really want is to purge all the records >>>>> associated with an URL. Is there a way to do this? >>>> >>>> See: http://varnish-cache.org/wiki/VCLExamplePurging >>>> >>>> Using purge("req.url == " req.url); should fix both problems. >>>> >>>> Laurence >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>>> >>> >> From checker at d6.com Sun Jul 11 06:22:03 2010 From: checker at d6.com (Chris Hecker) Date: Sat, 10 Jul 2010 23:22:03 -0700 Subject: log pid files on centos and logrotate Message-ID: <4C39630B.5020403@d6.com> On CentOS 5.5, the logrotate.d/varnish has /var/run/varnish/varnishncsa.pid for the pid file, and so does the comment at the top of the rc.d/init.d/varnishncsa script, but then in the body of the script, it actually puts the pidfile in /var/run/varnishncsa.pid, so the SIGHUP never gets to to the log process. Is this a varnish issue, or a CentOS issue? Chris From mike at matejunkie.com Sun Jul 11 10:12:00 2010 From: mike at matejunkie.com (Mike Adolphs) Date: Sun, 11 Jul 2010 12:12:00 +0200 Subject: Problems with Cache-Control: max-age=0 Message-ID: Hi, I've just been playing around with Varnish since yesterday, read the various examples and thought I'll give it a try with caching munin generated png's for 3600s (just as a test). When I now refresh the page I see that the cache is missed. Couple of seconds and a refresh later I see that the cache gets hit. So far so good, but a few minutes later new requests hit the backend again instead of getting its content from the cache. How do I alter the incoming request's header to unset or alter the Cache-Control: max-age=0? Is that possible? My current config looks like the following, but this is obviously for backends that sends "Cache-Control: no-cache, no-store, max-age=0, must-revalidate": sub vcl_recv { if (req.request == "GET" && req.url ~ "\.(html|jpg|jpeg|gif|ico|png)$") { lookup; } } sub vcl_fetch { if (req.request == "GET" && req.url ~ "\.(html|jpg|jpeg|gif|ico|png)$") { set obj.ttl = 3600s; } } Best regards, Mike -- Mike Adolphs Stitenstrasse 24 23554 Luebeck - Germany - Mail. mike at matejunkie.com Web. http://www.matejunkie.com/ XING. http://www.xing.com/profile/Mike_Adolphs From whshub at gmail.com Sun Jul 11 17:54:41 2010 From: whshub at gmail.com (Jacques) Date: Sun, 11 Jul 2010 10:54:41 -0700 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: I tried using a local server and using a vcl rather than command line parameters to define the backend. I still only get 503s. Is there a way to better debug Varnish so that I can determine why it can't make a backend connection? Thanks, Jacques On Sat, Jul 10, 2010 at 12:47 AM, Jorge D?az wrote: > It seems for some reason Varnish cannot connect to backend. > > I have never used parameter " -b 87.238.47.204:80 " > for backend configuration. > > You sould try creating a little VCL configuration file. > Modify default.vcl adding the backend in it, and start varnish with > parameter -f > > -f /opt/extra/etc/varnish/default.vcl > > Inside VCL, configure the backed with: > > backend default { > .host = "87.238.47.204"; > .port = "80"; > } > > If the problem persist, a good idea is to install a Apache webserver in localhost and try It as backend, for example in 127.0.0.1 port 8000 > > in order to see if the problem is varnish connecting to a remote IP or is varnish that not connect to nobody. > > My varnish installation is in same host of Apache webserver and works OK. > > > Regards. > > > 2010/7/9 Jacques > > I'm not currently using a vcl, just configuring via the command line for >> simplicity. Adding the connect_timeout parameter below had no impact on the >> 503s. Nor did adding all the parameters listed at letsdugg.com. >> >> Communication with the external server works perfectly. (Telnet and via >> lynx.) It's actually the varnish-cache.org site just to introduce a >> minimum number of variables. >> >> The only thing that I see in the varnish log is the FetchError, no backend >> connection. >> >> Here is a startup, request, shutdown cycle of varnish log: >> >> # /opt/extra/bin/varnishlog >> 0 WorkThread - fffffd7ff9dc0d20 start >> 0 CLI - Rd vcl.load "boot" ./vcl.ORk8t3RP.so >> 0 CLI - Wr 200 Loaded "./vcl.ORk8t3RP.so" as "boot" >> 0 CLI - Rd vcl.use "boot" >> 0 CLI - Wr 200 >> 0 CLI - Rd start >> 0 Debug - "Acceptor poll space increased to 512" >> 0 Debug - "Acceptor is poll" >> 0 CLI - Wr 200 >> 0 WorkThread - fffffd7ff93c5d20 start >> 0 WorkThread - fffffd7ff91c6d20 start >> 0 WorkThread - fffffd7ff8fc7d20 start >> 0 WorkThread - fffffd7ff8dc8d20 start >> 0 WorkThread - fffffd7ff8bc9d20 start >> 0 WorkThread - fffffd7ff89cad20 start >> 0 WorkThread - fffffd7ff87cbd20 start >> 0 WorkThread - fffffd7ff85ccd20 start >> 0 WorkThread - fffffd7ff83cdd20 start >> 0 CLI - Rd ping >> 0 CLI - Wr 200 PONG 1278687217 1.0 >> 0 CLI - Rd ping >> 0 CLI - Wr 200 PONG 1278687220 1.0 >> 9 SessionOpen c 10.100.200.1 64024 :80 >> 9 ReqStart c 10.100.200.1 64024 1455837036 >> 9 RxRequest c GET >> 9 RxURL c / >> 9 RxProtocol c HTTP/1.0 >> 9 RxHeader c Host: 10.100.200.2 >> 9 RxHeader c Accept: text/html, text/plain, text/css, text/sgml, >> */*;q=0.01 >> 9 RxHeader c Accept-Encoding: gzip, compress, bzip2 >> 9 RxHeader c Accept-Language: en >> 9 RxHeader c User-Agent: Lynx/2.8.6rel.5 libwww-FM/2.14 >> SSL-MM/1.4.1 OpenSSL/0.9.8l >> 9 VCL_call c recv >> 9 VCL_return c lookup >> 9 VCL_call c hash >> 9 VCL_return c hash >> 9 VCL_call c miss >> 9 VCL_return c fetch >> 9 FetchError c no backend connection >> 9 VCL_call c error >> 9 VCL_return c deliver >> 9 Length c 488 >> 9 VCL_call c deliver >> 9 VCL_return c deliver >> 9 TxProtocol c HTTP/1.1 >> 9 TxStatus c 503 >> 9 TxResponse c Service Unavailable >> 9 TxHeader c Server: Varnish >> 9 TxHeader c Retry-After: 0 >> 9 TxHeader c Content-Type: text/html; charset=utf-8 >> 9 TxHeader c Content-Length: 488 >> 9 TxHeader c Date: Fri, 09 Jul 2010 14:53:42 GMT >> 9 TxHeader c X-Varnish: 1455837036 >> 9 TxHeader c Age: 0 >> 9 TxHeader c Via: 1.1 varnish >> 9 TxHeader c Connection: close >> 9 ReqEnd c 1455837036 1278687222.201183081 1278687222.201552391 >> 0.000498056 0.000254393 0.000114918 >> 9 SessionClose c error >> 9 StatSess c 10.100.200.1 64024 0 1 1 0 0 0 235 488 >> 0 CLI - Rd ping >> 0 CLI - Wr 200 PONG 1278687223 1.0 >> 0 CLI - Rd ping >> 0 CLI - Wr 200 PONG 1278687226 1.0 >> 0 CLI - Rd ping >> 0 CLI - Wr 200 PONG 1278687229 1.0 >> >> >> >> >> >> >> On Fri, Jul 9, 2010 at 12:58 AM, Jorge D?az wrote: >> >>> ?There is communication between Varnish host and 87.238.47.204 machine?? >>> >>> Try first making " telnet 87.238.47.204 80 " in order testing there is no >>> firewall or a rule configured in webserver. >>> >>> If comunication is fine, the problem must be in VCL configuration. Try >>> executing varnishlog while you are testing and look for errors in output. >>> >>> In Solaris is also important to run Varnish with proper parameter that >>> are listen in: >>> >>> - http://letsgetdugg.com/2009/12/04/varnish-on-solaris/ >>> >>> >>> I had a lot of comunication problems without parameter: >>> >>> * -p connect_timeout=0s # Important bug work around for Solaris* >>> >>> >>> >>> >>> >>> 2010/7/9 Bedis 9 >>> >>> Hey, >>>> >>>> Can you share your VCL conf? >>>> >>>> cheers >>>> >>>> On Fri, Jul 9, 2010 at 7:08 AM, Jacques wrote: >>>> >>>>> I am trying to use Varnish on OpenSolaris. Due the help of others, I >>>>> have versions compiling and not crashing that are based on three different >>>>> versions of Varnish. >>>>> >>>>> I'm using 4516, 2.1.2 with the 649 patch, and trunk. I've also tried >>>>> compiling with gcc 4.3.3 in addition to the OpenSolaris standard of 3.4.3. >>>>> That changed nothing. >>>>> >>>>> All compiled copies of varnish seem to start just fine. I can use >>>>> varnishlog and see my requests arriving at Varnish. >>>>> >>>>> However, responses are always 503 with " FetchError: no backend >>>>> connection". Using tcpdump, I see no calls to the backend server from >>>>> Varnish, ever. >>>>> >>>>> For simplicity, I'm using the basic start of: >>>>> >>>>> varnishd -a :80 -b 87.238.47.204:80 >>>>> >>>>> What are right steps to determine what is wrong? I see nothing in >>>>> syslogs. Varnish doesn't complain. Using all the same steps on ubuntu, I >>>>> get exactly what I expect, a cached version of the backend server. >>>>> >>>>> Any pointers about how I should figure out what is failing would be >>>>> helpful. >>>>> >>>>> Thanks, >>>>> Jacques >>>>> >>>>> I've also noticed that the cache is starting at 500GB. Changing the >>>>> cache size to 512MB didn't have any impact on the 503s. >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>>>> >>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kacperw at gmail.com Mon Jul 12 03:48:24 2010 From: kacperw at gmail.com (Kacper Wysocki) Date: Mon, 12 Jul 2010 05:48:24 +0200 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: On Sun, Jul 11, 2010 at 7:54 PM, Jacques wrote: > I tried using a local server and using a vcl rather than command line > parameters to define the backend. ?I still only get 503s. ?Is there a way to > better debug Varnish so that I can determine why it can't make a backend > connection? Yup there is a better way to debug. You attach a debugger to the live process and watch your own request come in http://varnish-cache.org/wiki/DebuggingVarnish it's a little hands on but works fine if you have symbols and don't mind getting dirty. A dumb and simple suggestion: can you wget the backend from the varnish host? :-) Still, to me it looks like something quite platform-specific, not so simple. Do the makecheck tests all run fine? Oh, and I have a solaris box here too, my makecheck fails and it looks like there is some IPC problem: #### v1 CLI RX| Unknown request in manager process (child not running).\n ---- v1 CLI debug.xid command failed: 101 Unknown request in manager process (+++ child not running). 0K -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish-2.1.2-opensolaris-2009.fail.log Type: text/x-log Size: 3849 bytes Desc: not available URL: From checker at d6.com Mon Jul 12 06:02:15 2010 From: checker at d6.com (Chris Hecker) Date: Sun, 11 Jul 2010 23:02:15 -0700 Subject: server.ip for "undoing" namevirtualhosts issues Message-ID: <4C3AAFE7.3010306@d6.com> I'm running the configuration #1 I mentioned here: http://lists.varnish-cache.org/pipermail/varnish-misc/2010-June/004342.html Basically, varnishd listening on a bunch of ips, and then httpd listening on localhost:80 and all the sites as NameVirtualHosts. This seems to work well so far, but it has the disadvantage that if somebody goes to the raw ip addresses with their browser, they'll get the same site. Can I use server.ip in vcl_recv to force the host to the right site if there's no host record? What would the code look like to do this? Something like: if(!req.http.host) { if(server.ip == "a.b.c.d") { set req.http.host = "a.com"; } elsif(server.ip == "a.b.c.e") { ...etc Would that work? Has anybody tried that? Thanks, Chris From phk at phk.freebsd.dk Mon Jul 12 07:46:40 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 12 Jul 2010 07:46:40 +0000 Subject: server.ip for "undoing" namevirtualhosts issues In-Reply-To: Your message of "Sun, 11 Jul 2010 23:02:15 MST." <4C3AAFE7.3010306@d6.com> Message-ID: <3556.1278920800@critter.freebsd.dk> In message <4C3AAFE7.3010306 at d6.com>, Chris Hecker writes: >Can I use server.ip in vcl_recv to force the host to the right site if >there's no host record? What would the code look like to do this? >Something like: > >if(!req.http.host) { > if(server.ip == "a.b.c.d") { > set req.http.host = "a.com"; > } elsif(server.ip == "a.b.c.e") { >...etc Yes, that is server.ip was intended to be used. If that does not work, it is a bug. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From checker at d6.com Mon Jul 12 08:28:28 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 12 Jul 2010 01:28:28 -0700 Subject: varnish security Message-ID: <4C3AD22C.6010709@d6.com> It looks like all users can access the log shared memory for varnishd (so they can run varnishlog, varnishstat, varnishncsa, etc.). Is there a way to prevent that? It's not a huge priority for my current setup, but I was just surprised. I noticed there was a thread about the vcl.load interface on securityfocus as well: http://www.securityfocus.com/archive/1/510360 Chris From phk at phk.freebsd.dk Mon Jul 12 08:37:23 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 12 Jul 2010 08:37:23 +0000 Subject: varnish security In-Reply-To: Your message of "Mon, 12 Jul 2010 01:28:28 MST." <4C3AD22C.6010709@d6.com> Message-ID: <3680.1278923843@critter.freebsd.dk> In message <4C3AD22C.6010709 at d6.com>, Chris Hecker writes: >It looks like all users can access the log shared memory for varnishd >(so they can run varnishlog, varnishstat, varnishncsa, etc.). Is there >a way to prevent that? It's not a huge priority for my current setup, >but I was just surprised. Yes: Protect the directory you specify with the -n argument. >I noticed there was a thread about the vcl.load interface on >securityfocus as well: > >http://www.securityfocus.com/archive/1/510360 I pressume you also bothered to read the vendor response ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kristian at varnish-software.com Mon Jul 12 08:44:43 2010 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Mon, 12 Jul 2010 10:44:43 +0200 Subject: About saintmode_threshold behavior In-Reply-To: References: Message-ID: <20100712084443.GB3278@sunrider> On Fri, Jul 09, 2010 at 01:39:10PM -0300, Rodrigo K. Ferreira wrote: > About the error counters what is compared with saintmode_threshold, when it > counter is back to zero ? Just when that backend server are penalized ? Or > always after one backend probe ? > This questions is why is a bit normal dinamic backends servers returns few > 5XX errors, for client reqs bad formed or other reasons. And if isnt back to > zero, backend servers will be labeled sick in some time. Ok, I'm not entirely sure I understand what you're asking, but I'll explain saintmode_threshold anyway. Every time you use the "saintmode" command/directive in VCL, you add an entry to a list of bad objects, hooked up to the backend. So one list for each backend. When Varnish is trying to find a healthy backend, it will check if the objecthead it's looking for is represented on the list. While checking, it will count how many valid entries are present on the list. The only condition required for an entry to be valid is that it has not timed out. If it either finds the objecthead on the list OR finds saintmode_threshold items on the list, the backend is considered sick. This is not affected by health check polling at all. The only way to re-enable a backend that is considered sick because of too many saintmode-items, is time. Do keep in mind, though, that new entries are not added to the list after saintmode_threshold is reached. You might get a couple extra on the account of parallel requests going to the backend, but once the list is large enough, the backend wont be used, and thus cant get new items added to the blacklist. So if you use a 20s timer on saintmode, the maximum time until varnish retries the backend is 20 seconds. Consider saintmode a combination of a buffer until the real health checks detect the problem, and a way to blacklist just one item on one backend. You will need _different_ items on the saintmode blacklist to mark the backend as completely down. Even if a single page returns 500 constantly, that will not bring down the entire backend - it will just make varnish not ask that backend for that specific page. Hope this cleared up some questions, though it might add a few new ones I suppose. - Kristian From checker at d6.com Mon Jul 12 08:45:59 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 12 Jul 2010 01:45:59 -0700 Subject: server.ip for "undoing" namevirtualhosts issues In-Reply-To: <3556.1278920800@critter.freebsd.dk> References: <3556.1278920800@critter.freebsd.dk> Message-ID: <4C3AD647.3070409@d6.com> Okay, I'm trying this and it works. I realized I don't have any NameVirtualHost sites going through varnish right now, so I just stomp the req.http.Host with the correct site name at the top, and that let me remove all the www. stripping from the vcl_recv. I did notice that Host: is set by browsers even on numerical ips (due to HTTP/1.1 I guess), so I think the right if is something like: if(!req.http.host || (req.http.host ~ "^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$")) { blah; } but I haven't tried that yet, since I realized I don't need it yet. Does that look right? Chris On 2010/07/12 00:46, Poul-Henning Kamp wrote: > In message<4C3AAFE7.3010306 at d6.com>, Chris Hecker writes: > >> Can I use server.ip in vcl_recv to force the host to the right site if >> there's no host record? What would the code look like to do this? >> Something like: >> >> if(!req.http.host) { >> if(server.ip == "a.b.c.d") { >> set req.http.host = "a.com"; >> } elsif(server.ip == "a.b.c.e") { >> ...etc > > Yes, that is server.ip was intended to be used. > > If that does not work, it is a bug. > From phk at phk.freebsd.dk Mon Jul 12 08:55:22 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 12 Jul 2010 08:55:22 +0000 Subject: server.ip for "undoing" namevirtualhosts issues In-Reply-To: Your message of "Mon, 12 Jul 2010 01:45:59 MST." <4C3AD647.3070409@d6.com> Message-ID: <3740.1278924922@critter.freebsd.dk> In message <4C3AD647.3070409 at d6.com>, Chris Hecker writes: >if(!req.http.host || > (req.http.host ~ "^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$")) >{ > blah; >} > >but I haven't tried that yet, since I realized I don't need it yet. >Does that look right? yeah, looks pretty good to me. Not sure if you risk seeing a port number in the Host: header also ? Remeber that you can always start a varnish on a different port number to experiement with. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From checker at d6.com Mon Jul 12 09:01:42 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 12 Jul 2010 02:01:42 -0700 Subject: varnish security In-Reply-To: <3680.1278923843@critter.freebsd.dk> References: <3680.1278923843@critter.freebsd.dk> Message-ID: <4C3AD9F6.8020307@d6.com> > Yes: Protect the directory you specify with the -n argument. Ah, okay, thanks. Is that just created with the umask of root or something on startup? Maybe the docs for varnishd should mention this? I tried searching for various terms "permissions", "security", etc. in the docs > I pressume you also bothered to read the vendor response ? Of course. I was just pointing out the related thread. Maybe a wiki page on varnish-cache.org on securing varnish would be useful here. It could contain the thing about the file permissions above, a short discussion of the CLI, etc. That would help, and couldn't hurt. The Husqvarna analogy is slightly flawed since most people can't run yum install husqvarna and have one magically appear at their feet, gassed and ready to go. :) Chris On 2010/07/12 01:37, Poul-Henning Kamp wrote: > In message<4C3AD22C.6010709 at d6.com>, Chris Hecker writes: > >> It looks like all users can access the log shared memory for varnishd >> (so they can run varnishlog, varnishstat, varnishncsa, etc.). Is there >> a way to prevent that? It's not a huge priority for my current setup, >> but I was just surprised. > > Yes: Protect the directory you specify with the -n argument. > >> I noticed there was a thread about the vcl.load interface on >> securityfocus as well: >> >> http://www.securityfocus.com/archive/1/510360 > > I pressume you also bothered to read the vendor response ? > From checker at d6.com Mon Jul 12 09:09:14 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 12 Jul 2010 02:09:14 -0700 Subject: server.ip for "undoing" namevirtualhosts issues In-Reply-To: <3740.1278924922@critter.freebsd.dk> References: <3740.1278924922@critter.freebsd.dk> Message-ID: <4C3ADBBA.3050200@d6.com> > yeah, looks pretty good to me. Not sure if you risk seeing a port > number in the Host: header also ? Hmm, not sure either. > Remeber that you can always start a varnish on a different port > number to experiement with. Yeah, I realized this only after I sort of dove in head first the other day. :) Chris On 2010/07/12 01:55, Poul-Henning Kamp wrote: > In message<4C3AD647.3070409 at d6.com>, Chris Hecker writes: > >> if(!req.http.host || >> (req.http.host ~ "^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$")) >> { >> blah; >> } >> >> but I haven't tried that yet, since I realized I don't need it yet. >> Does that look right? > > yeah, looks pretty good to me. Not sure if you risk seeing a port > number in the Host: header also ? > > Remeber that you can always start a varnish on a different port number > to experiement with. > From phk at phk.freebsd.dk Mon Jul 12 10:20:29 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 12 Jul 2010 10:20:29 +0000 Subject: varnish security In-Reply-To: Your message of "Mon, 12 Jul 2010 02:01:42 MST." <4C3AD9F6.8020307@d6.com> Message-ID: <4127.1278930029@critter.freebsd.dk> In message <4C3AD9F6.8020307 at d6.com>, Chris Hecker writes: >> I pressume you also bothered to read the vendor response ? > >Of course. I was just pointing out the related thread. Uhm, no, you pointed to the message with the bogo-advisory and I do not seem to be able to find any ensuing discussion from there ? >Maybe a wiki page on varnish-cache.org on securing varnish would be >useful here. It could contain the thing about the file permissions >above, a short discussion of the CLI, etc. That would help, and >couldn't hurt. Yeah, our docs need work... >The Husqvarna analogy is slightly flawed since most people can't run >yum install husqvarna >and have one magically appear at their feet, gassed and ready to go. :) That argument would be much more convincing, if sites like this did not exist: http://www.baileysonline.com/search.asp?SKW=HVF%20390XP&catID=11443 Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From zabrane3 at gmail.com Mon Jul 12 10:55:54 2010 From: zabrane3 at gmail.com (zabrane Mikael) Date: Mon, 12 Jul 2010 12:55:54 +0200 Subject: Varnish computes Content-Length and set it to 0 for one of my URL! Message-ID: Hi list, I'm facing a little problem with my actual varnish config. Currently, Varnish have only one backend (it's a caching server). When trying to get a specific URL from the cache through Varnish, it always set the header "Content-Length: 0" for it before sending it back to Firefox. Inside the cache, this URL has no "Content-Length" header as you can see: 1) CACHE BACKEND ---> VARNISH ----------------------------------------------------- HTTP/1.1 200 OK Date: Tue, 15 Jun 2010 09:18:29 GMT Content-Type: text/html; charset=UTF-8 X-Powered-By: ASP.NET Set-Cookie: jsessionid=XXXXXXXXXXXXXXXXXXX; path=/ [payload] 2) VARNISH ----> FIREFOX ----------------------------------------- Content-Type text/html; charset=UTF-8 X-Powered-By ASP.NET Set-Cookie jsessionid=XXXXXXXXXXXXXXXXXXX; path=/ Content-Length 0 Date Mon, 12 Jul 2010 10:11:06 GMT X-Varnish 353742751 Age 0 Via 1.1 varnish [payload] How can I tell "Varnish" to always compute the "Content-Length" if it's not specified (in the headers response from the cache)? Help appreciated guys!!! N.B: getting this URL directly from the cache (i.e without using Varnish) works perfectly. -- Regards Zab -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkferreira at gmail.com Mon Jul 12 11:12:38 2010 From: rkferreira at gmail.com (Rodrigo K. Ferreira) Date: Mon, 12 Jul 2010 08:12:38 -0300 Subject: About saintmode_threshold behavior In-Reply-To: <20100712084443.GB3278@sunrider> References: <20100712084443.GB3278@sunrider> Message-ID: Thanks Kristian. My point is, if at one hour period, some backend returns ten "5XX" errors and that objects are inserted on the list of bad objects. Varnish consider its a sick backend, correct ? But really it isnt a sick backend. Im asking this questions because on my servers architeture used: LoadBalancer ---> Varnish's ---> LoadBalancer ---> Backends So, for instance, one backend called "IMAGES", have just one server (the loadbalancer). And if is marked down, all requests not in cache will return error. Dont you think we could have a "time perspective" related to the thresholds ? For instance: thresholds_items = 10; thresholds_time = 3600 (seconds); And after 1 hour the entire list of bad objects is cleaned. On Mon, Jul 12, 2010 at 5:44 AM, Kristian Lyngstol < kristian at varnish-software.com> wrote: > On Fri, Jul 09, 2010 at 01:39:10PM -0300, Rodrigo K. Ferreira wrote: > > About the error counters what is compared with saintmode_threshold, when > it > > counter is back to zero ? Just when that backend server are penalized ? > Or > > always after one backend probe ? > > This questions is why is a bit normal dinamic backends servers returns > few > > 5XX errors, for client reqs bad formed or other reasons. And if isnt back > to > > zero, backend servers will be labeled sick in some time. > > Ok, I'm not entirely sure I understand what you're asking, but I'll explain > saintmode_threshold anyway. > > Every time you use the "saintmode" command/directive in VCL, you add an > entry to a list of bad objects, hooked up to the backend. So one list for > each backend. > > When Varnish is trying to find a healthy backend, it will check if the > objecthead it's looking for is represented on the list. While checking, it > will count how many valid entries are present on the list. The only > condition required for an entry to be valid is that it has not timed out. > If it either finds the objecthead on the list OR finds saintmode_threshold > items on the list, the backend is considered sick. This is not affected by > health check polling at all. The only way to re-enable a backend that is > considered sick because of too many saintmode-items, is time. > > Do keep in mind, though, that new entries are not added to the list after > saintmode_threshold is reached. You might get a couple extra on the account > of parallel requests going to the backend, but once the list is large > enough, the backend wont be used, and thus cant get new items added to the > blacklist. So if you use a 20s timer on saintmode, the maximum time until > varnish retries the backend is 20 seconds. > > Consider saintmode a combination of a buffer until the real health checks > detect the problem, and a way to blacklist just one item on one backend. > > You will need _different_ items on the saintmode blacklist to mark the > backend as completely down. Even if a single page returns 500 constantly, > that will not bring down the entire backend - it will just make varnish not > ask that backend for that specific page. > > Hope this cleared up some questions, though it might add a few new ones I > suppose. > > - Kristian > -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.vanarragon at lukkien.com Mon Jul 12 11:36:00 2010 From: j.vanarragon at lukkien.com (Jaap van Arragon) Date: Mon, 12 Jul 2010 13:36:00 +0200 Subject: Get Bandwidth statistics Message-ID: Hello, We are using Varnish 2.1.0-2 for about 4 weeks now and we like it! There is only one problem, how can we log the used bandwidth used by the varnish daemon? Normally in Apache we can use the access logs but in varnish there is only memory logging. Can anyone tell me how to get the bandwidth used by a specific daemon? Thank you in advance. Grt, Jaap -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at varnish-software.com Mon Jul 12 11:43:56 2010 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Mon, 12 Jul 2010 13:43:56 +0200 Subject: Get Bandwidth statistics In-Reply-To: References: Message-ID: <20100712114356.GD3278@sunrider> On Mon, Jul 12, 2010 at 01:36:00PM +0200, Jaap van Arragon wrote: > We are using Varnish 2.1.0-2 for about 4 weeks now and we like it! Glad you like it :) > There is only one problem, how can we log the used bandwidth used by the > varnish daemon? Normally in Apache we can use the access logs but in varnish > there is only memory logging. > > Can anyone tell me how to get the bandwidth used by a specific daemon? There are a few different approaches. I'm sure there's some OS-method too, but a common way of doing it is reading it from varnishstat, as that only resets when Varnish restarts. The Munin plugin for Varnish does exactly that (among other things). Essentially executing varnishstat -1 on a regular interval. Could be read directly from the shmlog too, but that's a bit more complicated. - Kristian From jdzstz at gmail.com Mon Jul 12 11:53:22 2010 From: jdzstz at gmail.com (=?ISO-8859-1?Q?Jorge_D=EDaz?=) Date: Mon, 12 Jul 2010 13:53:22 +0200 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: Hello, When I installed Varnish in Solaris 10, I had problems with Ulimit values that might cause problems connecting to backed. I had problem with file descriptors ulimit, that it is 256 by default in Solaris. Executing "ulimit -n 65536" before launching varnish, solves the file descriptors problem. You can also try to configure a probe in backed configuration, that tests if backend is alive. backend default { .host = "backend-host"; .port = "8000"; .probe = { .url = "/heartbeat/heartbeat.html"; .timeout = 10 ms; .interval = 2s; .window = 10; .threshold = 8; } } Later, in varnishadm you can get backend health with: "debug.health" command: varnishadm -T locahost:8800 debug.health (-T is admin port that must be also in varnishd as parameter. varnishd -T locahost:8800 ) 2010/7/12 Kacper Wysocki > On Sun, Jul 11, 2010 at 7:54 PM, Jacques wrote: > > I tried using a local server and using a vcl rather than command line > > parameters to define the backend. I still only get 503s. Is there a way > to > > better debug Varnish so that I can determine why it can't make a backend > > connection? > > Yup there is a better way to debug. You attach a debugger to the live > process and watch your own request come in > http://varnish-cache.org/wiki/DebuggingVarnish > it's a little hands on but works fine if you have symbols and don't > mind getting dirty. > > A dumb and simple suggestion: can you wget the backend from the > varnish host? :-) > > Still, to me it looks like something quite platform-specific, not so > simple. > Do the makecheck tests all run fine? > Oh, and I have a solaris box here too, my makecheck fails and it looks > like there is some IPC problem: > > #### v1 CLI RX| Unknown request in manager process (child not running).\n > ---- v1 CLI debug.xid command failed: 101 Unknown request in manager > process (+++ child not running). > > > > 0K > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastiaan.jansen at kpn.com Mon Jul 12 12:28:01 2010 From: sebastiaan.jansen at kpn.com (sebastiaan.jansen at kpn.com) Date: Mon, 12 Jul 2010 14:28:01 +0200 Subject: Get Bandwidth statistics In-Reply-To: References: Message-ID: Hi Jaap, We're sharing a client.. allerhande.nl Your probably looking for varnishncsa If there's a problem with the logformat (if believe it's still actual), there's an patch in trac that can fix that. With kind regards, Sebastiaan KPN Van: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] Namens Jaap van Arragon Verzonden: maandag 12 juli 2010 13:36 Aan: varnish-misc at varnish-cache.org Onderwerp: Get Bandwidth statistics Hello, We are using Varnish 2.1.0-2 for about 4 weeks now and we like it! There is only one problem, how can we log the used bandwidth used by the varnish daemon? Normally in Apache we can use the access logs but in varnish there is only memory logging. Can anyone tell me how to get the bandwidth used by a specific daemon? Thank you in advance. Grt, Jaap -------------- next part -------------- An HTML attachment was scrubbed... URL: From cosimo at streppone.it Mon Jul 12 12:51:26 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Mon, 12 Jul 2010 14:51:26 +0200 Subject: Get Bandwidth statistics In-Reply-To: <20100712114356.GD3278@sunrider> References: <20100712114356.GD3278@sunrider> Message-ID: On Mon, 12 Jul 2010 13:43:56 +0200, Kristian Lyngstol wrote: > On Mon, Jul 12, 2010 at 01:36:00PM +0200, Jaap van Arragon wrote: > >> There is only one problem, how can we log the used bandwidth used by the >> varnish daemon? Normally in Apache we can use the access logs but in >> varnish there is only memory logging. >> >> Can anyone tell me how to get the bandwidth used by a specific daemon? > > There are a few different approaches. I'm sure there's some OS-method > too, but a common way of doing it is reading it from varnishstat, as > that only > resets when Varnish restarts. The Munin plugin for Varnish does exactly > that (among other things). What about bandwidth limiting? For one bw intensive service, we had bw limiting (Apache + mod_cband). Ofc, putting a varnish in front of it is going to bypass bw limiting mechanisms. I realize it's probably blasphemy, but is there a way to limit bw in Varnish? -- Cosimo From ebowman at boboco.ie Mon Jul 12 13:16:01 2010 From: ebowman at boboco.ie (Eric Bowman) Date: Mon, 12 Jul 2010 14:16:01 +0100 Subject: server.ip for "undoing" namevirtualhosts issues In-Reply-To: <4C3ADBBA.3050200@d6.com> References: <3740.1278924922@critter.freebsd.dk> <4C3ADBBA.3050200@d6.com> Message-ID: <4C3B1591.4010201@boboco.ie> On 07/12/2010 10:09 AM, Chris Hecker wrote: > >> yeah, looks pretty good to me. Not sure if you risk seeing a port >> number in the Host: header also ? > > Hmm, not sure either. I have certainly seen a port come through... -- Eric Bowman Boboco Ltd ebowman at boboco.ie http://www.boboco.ie/ebowman/pubkey.pgp +35318394189/+353872801532 From phk at phk.freebsd.dk Mon Jul 12 13:36:42 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 12 Jul 2010 13:36:42 +0000 Subject: Varnish computes Content-Length and set it to 0 for one of my URL! In-Reply-To: Your message of "Mon, 12 Jul 2010 12:55:54 +0200." Message-ID: <5324.1278941802@critter.freebsd.dk> In message , zabr ane Mikael writes: Can you please open a ticket on this ? I need to read the RFC to find out if this is actually a bug, but it is certainly not conventional of your backend to send HTTP/1.1 and none of "Connection:", "Content-Length:" or "Transfer-Encoding" Poul-Henning >--===============8364418252999650435== >Content-Type: multipart/alternative; boundary=0015174be612c79937048b2e98c0 > >--0015174be612c79937048b2e98c0 >Content-Type: text/plain; charset=ISO-8859-1 > >Hi list, > >I'm facing a little problem with my actual varnish config. >Currently, Varnish have only one backend (it's a caching server). > >When trying to get a specific URL from the cache through Varnish, it always >set the header "Content-Length: 0" for it before sending it back to Firefox. > >Inside the cache, this URL has no "Content-Length" header as you can see: > >1) CACHE BACKEND ---> VARNISH >----------------------------------------------------- >HTTP/1.1 200 OK >Date: Tue, 15 Jun 2010 09:18:29 GMT >Content-Type: text/html; charset=UTF-8 >X-Powered-By: ASP.NET >Set-Cookie: jsessionid=XXXXXXXXXXXXXXXXXXX; path=/ > >[payload] > > >2) VARNISH ----> FIREFOX >----------------------------------------- >Content-Type text/html; charset=UTF-8 >X-Powered-By ASP.NET >Set-Cookie jsessionid=XXXXXXXXXXXXXXXXXXX; path=/ >Content-Length 0 >Date Mon, 12 Jul 2010 10:11:06 GMT >X-Varnish 353742751 >Age 0 >Via 1.1 varnish > >[payload] > > >How can I tell "Varnish" to always compute the "Content-Length" if it's not >specified (in the headers response from the cache)? > >Help appreciated guys!!! > >N.B: getting this URL directly from the cache (i.e without using Varnish) >works perfectly. > >-- >Regards >Zab > >--0015174be612c79937048b2e98c0 >Content-Type: text/html; charset=ISO-8859-1 >Content-Transfer-Encoding: quoted-printable > >Hi list,

I'm facing a little problem with my ac= >tual varnish config.
Currently, Varnish have only one backend (it= >'s a caching server).

When trying to get a spe= >cific URL from the cache through Varnish, it always
>
set the header "Content-Length: 0" for it before sending it = >back to Firefox.

Inside the cache, this URL has no= > "Content-Length" header as you can see:

>
1) CACHE BACKEND ---> VARNISH
---------------------------= >--------------------------
HTTP/1.1 200 OK
Date: Tue, 1= >5 Jun 2010 09:18:29 GMT
Content-Type: text/html; charset=3DUTF-8<= >/div> >
X-Powered-By: ASP.NET
Set-Coo= >kie: jsessionid=3DXXXXXXXXXXXXXXXXXXX; path=3D/

[p= >ayload]


2) VARNISH ----> FIREFOX= >
>
-----------------------------------------
Content-Type<= >span class=3D"Apple-tab-span" style=3D"white-space:pre"> text/html; = >charset=3DUTF-8
X-Powered-By=3D"white-space:pre"> ASP.NET
>
Set-Cookie span>jsessionid=3DXXXXXXXXXXXXXXXXXXX; path=3D/
Content-Lengthan class=3D"Apple-tab-span" style=3D"white-space:pre"> 0
D= >ate Mon, 12= > Jul 2010 10:11:06 GMT
>
X-Varnish pan>353742751
Ageace:pre"> 0
Viate-space:pre"> 1.1 varnish
>

[payload]


>How can I tell "Varnish" to always compute the "Content-Len= >gth" if it's not specified (in the headers response from the cache= >)?
>

Help appreciated guys!!!

N.B: getti= >ng this URL directly from the cache (i.e without using Varnish) works perfe= >ctly.

--
Regards
Zab
>
> >--0015174be612c79937048b2e98c0-- > > >--===============8364418252999650435== >Content-Type: text/plain; charset="us-ascii" >MIME-Version: 1.0 >Content-Transfer-Encoding: 7bit >Content-Disposition: inline > >_______________________________________________ >varnish-misc mailing list >varnish-misc at varnish-cache.org >http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >--===============8364418252999650435==-- > -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From zabrane3 at gmail.com Mon Jul 12 14:12:12 2010 From: zabrane3 at gmail.com (zabrane Mikael) Date: Mon, 12 Jul 2010 16:12:12 +0200 Subject: Varnish computes Content-Length and set it to 0 for one of my URL! In-Reply-To: <5324.1278941802@critter.freebsd.dk> References: <5324.1278941802@critter.freebsd.dk> Message-ID: Hi Poul, Can you please open a ticket on this ? > Done. > I need to read the RFC to find out if this is actually a bug, but it > is certainly not conventional of your backend to send HTTP/1.1 My backend caching server blindly delivers content without inspecting it. This specific page unfortunately comes from a Microsoft IIS ASP server before beign stored in the "cache server". > and none of "Connection:", "Content-Length:" or "Transfer-Encoding" > Yep. But Varnish has to be a bit more tolerant regarding servers deficiencies. Right? Hope this help. Regards Zab 2010/7/12 Poul-Henning Kamp > In message , > zabr > ane Mikael writes: > > Can you please open a ticket on this ? > > I need to read the RFC to find out if this is actually a bug, but it > is certainly not conventional of your backend to send HTTP/1.1 and > none of "Connection:", "Content-Length:" or "Transfer-Encoding" > > Poul-Henning > > >--===============8364418252999650435== > >Content-Type: multipart/alternative; boundary=0015174be612c79937048b2e98c0 > > > >--0015174be612c79937048b2e98c0 > >Content-Type: text/plain; charset=ISO-8859-1 > > > >Hi list, > > > >I'm facing a little problem with my actual varnish config. > >Currently, Varnish have only one backend (it's a caching server). > > > >When trying to get a specific URL from the cache through Varnish, it > always > >set the header "Content-Length: 0" for it before sending it back to > Firefox. > > > >Inside the cache, this URL has no "Content-Length" header as you can see: > > > >1) CACHE BACKEND ---> VARNISH > >----------------------------------------------------- > >HTTP/1.1 200 OK > >Date: Tue, 15 Jun 2010 09:18:29 GMT > >Content-Type: text/html; charset=UTF-8 > >X-Powered-By: ASP.NET > >Set-Cookie: jsessionid=XXXXXXXXXXXXXXXXXXX; path=/ > > > >[payload] > > > > > >2) VARNISH ----> FIREFOX > >----------------------------------------- > >Content-Type text/html; charset=UTF-8 > >X-Powered-By ASP.NET > >Set-Cookie jsessionid=XXXXXXXXXXXXXXXXXXX; path=/ > >Content-Length 0 > >Date Mon, 12 Jul 2010 10:11:06 GMT > >X-Varnish 353742751 > >Age 0 > >Via 1.1 varnish > > > >[payload] > > > > > >How can I tell "Varnish" to always compute the "Content-Length" if it's > not > >specified (in the headers response from the cache)? > > > >Help appreciated guys!!! > > > >N.B: getting this URL directly from the cache (i.e without using Varnish) > >works perfectly. > > > >-- > >Regards > >Zab > > > >--0015174be612c79937048b2e98c0 > >Content-Type: text/html; charset=ISO-8859-1 > >Content-Transfer-Encoding: quoted-printable > > > >Hi list,

I'm facing a little problem with my > ac= > >tual varnish config.
Currently, Varnish have only one backend > (it= > >'s a caching server).

When trying to get a > spe= > >cific URL from the cache through Varnish, it always
> >
set the header "Content-Length: 0" for it before sending it > = > >back to Firefox.

Inside the cache, this URL has > no= > > "Content-Length" header as you can see:

> >
1) CACHE BACKEND ---> > VARNISH
---------------------------= > >--------------------------
HTTP/1.1 200 OK
Date: Tue, > 1= > >5 Jun 2010 09:18:29 GMT
Content-Type: text/html; > charset=3DUTF-8<= > >/div> > >
X-Powered-By: ASP.NET >
Set-Coo= > >kie: jsessionid=3DXXXXXXXXXXXXXXXXXXX; > path=3D/

[p= > >ayload]


2) VARNISH ----> > FIREFOX= > >
> > >
-----------------------------------------
Content-Type<= > >span class=3D"Apple-tab-span" style=3D"white-space:pre"> > text/html; = > >charset=3DUTF-8
X-Powered-By style= > >=3D"white-space:pre"> ASP.NET >
> >
Set-Cookie > >span>jsessionid=3DXXXXXXXXXXXXXXXXXXX; > path=3D/
Content-Length >an class=3D"Apple-tab-span" style=3D"white-space:pre"> > 0
D= > >ate Mon, > 12= > > Jul 2010 10:11:06 GMT
> >
X-Varnish > >pan>353742751
Age style=3D"white-sp= > >ace:pre"> 0
Via style=3D"whi= > >te-space:pre"> 1.1 varnish
> > >

[payload]


>>How can I tell "Varnish" to always compute the > "Content-Len= > >gth" if it's not specified (in the headers response from the > cache= > >)?
> >

Help appreciated guys!!!

N.B: > getti= > >ng this URL directly from the cache (i.e without using Varnish) works > perfe= > >ctly.

--
Regards
Zab
> >
> > > >--0015174be612c79937048b2e98c0-- > > > > > >--===============8364418252999650435== > >Content-Type: text/plain; charset="us-ascii" > >MIME-Version: 1.0 > >Content-Transfer-Encoding: 7bit > >Content-Disposition: inline > > > >_______________________________________________ > >varnish-misc mailing list > >varnish-misc at varnish-cache.org > >http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > >--===============8364418252999650435==-- > > > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -- Regards Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From fla_torres at yahoo.com.br Mon Jul 12 14:35:40 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Mon, 12 Jul 2010 11:35:40 -0300 Subject: server.ip for "undoing" namevirtualhosts issues In-Reply-To: <4C3AAFE7.3010306@d6.com> References: <4C3AAFE7.3010306@d6.com> Message-ID: <4C3B283C.6010608@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/12/2010 03:02 AM, Chris Hecker wrote: > Basically, varnishd listening on a bunch of ips, and then httpd > listening on localhost:80 and all the sites as NameVirtualHosts. > This seems to work well so far, but it has the disadvantage that if > somebody goes to the raw ip addresses with their browser, they'll > get the same site. Hi Chris, How about print a message like "request not allowed" ? example: if (req.http.host ~ "^a.b.c$"){ ... } if (req.http.host ~ "^d.e.f$"){ ... } else { error 403 "Cache: Request not allowed."; } -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkw7KDgACgkQNRQApncg2942LACgilJnzLvaE6BrBbuMeTBgPg/E hAgAoKg5cfZ2EbXQbTreOsv7t8lm0yEB =r0EI -----END PGP SIGNATURE----- From scaunter at topscms.com Mon Jul 12 14:36:42 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Mon, 12 Jul 2010 10:36:42 -0400 Subject: Varnish computes Content-Length and set it to 0 for one of my URL! In-Reply-To: References: <5324.1278941802@critter.freebsd.dk> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net> >My backend caching server blindly delivers content without inspecting it. >This specific page unfortunately comes from a Microsoft IIS ASP server before beign stored in the "cache server". ? >and?none of "Connection:", "Content-Length:" or "Transfer-Encoding" ? >Yep. But Varnish has to be a bit more tolerant regarding servers deficiencies. Right?? Microsft IIS sends these headers, our stack is IIS7 to varnish. Something is stripping headers, see varnishlog output below 316 TxHeader c Content-Type: text/xml; charset=utf-8 316 TxHeader c Content-Encoding: gzip 316 TxHeader c Vary: Accept-Encoding 316 TxHeader c Server: Microsoft-IIS/7.0 316 TxHeader c X-Powered-By: ASP.NET 316 TxHeader c WS: 2-5 316 TxHeader c X-TOPS-CacheReason: Speed 316 TxHeader c cache-control: max-age = 300 316 TxHeader c Content-Length: 3341 316 TxHeader c Date: Mon, 12 Jul 2010 14:33:07 GMT 316 TxHeader c X-Varnish: 788805734 788768049 316 TxHeader c Age: 177 316 TxHeader c Via: 1.1 varnish 316 TxHeader c Connection: keep-alive 316 TxHeader c X-Cache-Svr: topsvarnish2 316 TxHeader c X-Cache: HIT 316 TxHeader c X-Cache-Hits: 4044 Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com 2010/7/12 Poul-Henning Kamp In message , zabr ane Mikael writes: Can you please open a ticket on this ? I need to read the RFC to find out if this is actually a bug, but it is certainly not conventional of your backend to send HTTP/1.1 and none of "Connection:", "Content-Length:" or "Transfer-Encoding" Poul-Henning >--===============8364418252999650435== >Content-Type: multipart/alternative; boundary=0015174be612c79937048b2e98c0 > >--0015174be612c79937048b2e98c0 >Content-Type: text/plain; charset=ISO-8859-1 > >Hi list, > >I'm facing a little problem with my actual varnish config. >Currently, Varnish have only one backend (it's a caching server). > >When trying to get a specific URL from the cache through Varnish, it always >set the header "Content-Length: 0" for it before sending it back to Firefox. > >Inside the cache, this URL has no "Content-Length" header as you can see: > >1) CACHE BACKEND ---> VARNISH >----------------------------------------------------- >HTTP/1.1 200 OK >Date: Tue, 15 Jun 2010 09:18:29 GMT >Content-Type: text/html; charset=UTF-8 >X-Powered-By: ASP.NET >Set-Cookie: jsessionid=XXXXXXXXXXXXXXXXXXX; path=/ > >[payload] > > >2) VARNISH ----> FIREFOX >----------------------------------------- >Content-Type text/html; charset=UTF-8 >X-Powered-By ASP.NET >Set-Cookie jsessionid=XXXXXXXXXXXXXXXXXXX; path=/ >Content-Length 0 >Date Mon, 12 Jul 2010 10:11:06 GMT >X-Varnish 353742751 >Age 0 >Via 1.1 varnish > >[payload] > > >How can I tell "Varnish" to always compute the "Content-Length" if it's not >specified (in the headers response from the cache)? > >Help appreciated guys!!! > >N.B: getting this URL directly from the cache (i.e without using Varnish) >works perfectly. > >-- >Regards >Zab > >--0015174be612c79937048b2e98c0 >Content-Type: text/html; charset=ISO-8859-1 >Content-Transfer-Encoding: quoted-printable > >Hi list,

I'm facing a little problem with my ac= >tual varnish config.
Currently, Varnish have only one backend (it= >'s a caching server).

When trying to get a spe= >cific URL from the cache through Varnish, it always
>
set the header "Content-Length: 0" for it before sending it = >back to Firefox.

Inside the cache, this URL has no= > "Content-Length" header as you can see:

>
1) CACHE BACKEND ---> VARNISH
---------------------------= >--------------------------
HTTP/1.1 200 OK
Date: Tue, 1= >5 Jun 2010 09:18:29 GMT
Content-Type: text/html; charset=3DUTF-8<= >/div> >
X-Powered-By: ASP.NET
Set-Coo= >kie: jsessionid=3DXXXXXXXXXXXXXXXXXXX; path=3D/

[p= >ayload]


2) VARNISH ----> FIREFOX= >
>
-----------------------------------------
Content-Type<= >span class=3D"Apple-tab-span" style=3D"white-space:pre"> ? ? ? text/html; = >charset=3DUTF-8
X-Powered-By=3D"white-space:pre"> ?ASP.NET
>
Set-Cookie ? ? ? span>jsessionid=3DXXXXXXXXXXXXXXXXXXX; path=3D/
Content-Lengthan class=3D"Apple-tab-span" style=3D"white-space:pre"> 0
D= >ate ? Mon, 12= > Jul 2010 10:11:06 GMT
>
X-Varnish ? ? ? ?pan>353742751
Ageace:pre"> ? ? ?0
Viate-space:pre"> 1.1 varnish
>

[payload]


>How can I tell "Varnish" to always compute the "Content-Len= >gth" if it's not specified (in the headers response from the cache= >)?
>

Help appreciated guys!!!

N.B: getti= >ng this URL directly from the cache (i.e without using Varnish) works perfe= >ctly.

--
Regards
Zab
>
> >--0015174be612c79937048b2e98c0-- > > >--===============8364418252999650435== >Content-Type: text/plain; charset="us-ascii" >MIME-Version: 1.0 >Content-Transfer-Encoding: 7bit >Content-Disposition: inline > >_______________________________________________ >varnish-misc mailing list >varnish-misc at varnish-cache.org >http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >--===============8364418252999650435==-- > -- Poul-Henning Kamp ? ? ? | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG ? ? ? ? | TCP/IP since RFC 956 FreeBSD committer ? ? ? | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. -- Regards Zabrane From zabrane3 at gmail.com Mon Jul 12 14:53:33 2010 From: zabrane3 at gmail.com (zabrane Mikael) Date: Mon, 12 Jul 2010 16:53:33 +0200 Subject: Varnish computes Content-Length and set it to 0 for one of my URL! In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net> References: <5324.1278941802@critter.freebsd.dk> <7F0AA702B8A85A4A967C4C8EBAD6902C0C8452@TMG-EVS02.torstar.net> Message-ID: Hi Caunter, Microsft IIS sends these headers, our stack is IIS7 to varnish. > > 316 TxHeader c Content-Type: text/xml; charset=utf-8 > 316 TxHeader c Content-Encoding: gzip > 316 TxHeader c Vary: Accept-Encoding > ... > Yep I can see that, but not in our case. Something is stripping headers, see varnishlog output below Accessing our caching server directly works for us for years. We noticed that problem only today (as we started using Varnish only last week) -- Regards Zabrane -------------- next part -------------- An HTML attachment was scrubbed... URL: From checker at d6.com Mon Jul 12 19:05:43 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 12 Jul 2010 12:05:43 -0700 Subject: varnish security In-Reply-To: <4127.1278930029@critter.freebsd.dk> References: <4127.1278930029@critter.freebsd.dk> Message-ID: <4C3B6787.4000703@d6.com> > Uhm, no, you pointed to the message with the bogo-advisory and I do > not seem to be able to find any ensuing discussion from there ? Scroll down? Search for "Vendor Response", it's got your complete email/rant. :) Chris On 2010/07/12 03:20, Poul-Henning Kamp wrote: > In message<4C3AD9F6.8020307 at d6.com>, Chris Hecker writes: > >>> I pressume you also bothered to read the vendor response ? >> >> Of course. I was just pointing out the related thread. > > Uhm, no, you pointed to the message with the bogo-advisory and > I do not seem to be able to find any ensuing discussion from there ? > >> Maybe a wiki page on varnish-cache.org on securing varnish would be >> useful here. It could contain the thing about the file permissions >> above, a short discussion of the CLI, etc. That would help, and >> couldn't hurt. > > Yeah, our docs need work... > >> The Husqvarna analogy is slightly flawed since most people can't run >> yum install husqvarna >> and have one magically appear at their feet, gassed and ready to go. :) > > That argument would be much more convincing, if sites like this > did not exist: > > http://www.baileysonline.com/search.asp?SKW=HVF%20390XP&catID=11443 > > Poul-Henning > From whshub at gmail.com Tue Jul 13 01:11:11 2010 From: whshub at gmail.com (Jacques) Date: Mon, 12 Jul 2010 18:11:11 -0700 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: Yes, wget works fine. As does lynx, etc. :D I checked ulimit per Jorge's response as well. No impact. I ran make test, (which I should have done before). 123 of 174 tests failed. All the same 503 issue. I'm going to recompile with debugging enabled and see what I can see. FYI, the log of tests is attached if you have any insights. Thanks, Jacques On Sun, Jul 11, 2010 at 8:48 PM, Kacper Wysocki wrote: > On Sun, Jul 11, 2010 at 7:54 PM, Jacques wrote: > > I tried using a local server and using a vcl rather than command line > > parameters to define the backend. I still only get 503s. Is there a way > to > > better debug Varnish so that I can determine why it can't make a backend > > connection? > > Yup there is a better way to debug. You attach a debugger to the live > process and watch your own request come in > http://varnish-cache.org/wiki/DebuggingVarnish > it's a little hands on but works fine if you have symbols and don't > mind getting dirty. > > A dumb and simple suggestion: can you wget the backend from the > varnish host? :-) > > Still, to me it looks like something quite platform-specific, not so > simple. > Do the makecheck tests all run fine? > Oh, and I have a solaris box here too, my makecheck fails and it looks > like there is some IPC problem: > > #### v1 CLI RX| Unknown request in manager process (child not running).\n > ---- v1 CLI debug.xid command failed: 101 Unknown request in manager > process (+++ child not running). > > > > 0K > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: check.log.gz Type: application/x-gzip Size: 43530 bytes Desc: not available URL: From checker at d6.com Tue Jul 13 06:03:12 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 12 Jul 2010 23:03:12 -0700 Subject: using round-robin to redirect some traffic Message-ID: <4C3C01A0.6090600@d6.com> Can I use the director round-robin feature to then check the req.backend with an if statement in vcl_recv, and issue a 302 (using the vcl_error 750 hack) to redirect half the traffic to another server? Is there a better way to do that? Chris From martin.boer at netclever.nl Tue Jul 13 06:32:00 2010 From: martin.boer at netclever.nl (Martin Boer) Date: Tue, 13 Jul 2010 08:32:00 +0200 Subject: Get Bandwidth statistics In-Reply-To: <20100712114356.GD3278@sunrider> References: <20100712114356.GD3278@sunrider> Message-ID: <4C3C0860.4060000@netclever.nl> If you don't use munin it is very easy to write a do the following; write a small script around the following command varnishstat -1 -f s_bodybytes s_hdrbytes (and probably some other counters you like to see) to get the amount of bytes at a given moment and schedule it in cron to run every 5 minutes or so. Regards, Martin Boer Kristian Lyngstol wrote: > On Mon, Jul 12, 2010 at 01:36:00PM +0200, Jaap van Arragon wrote: > >> We are using Varnish 2.1.0-2 for about 4 weeks now and we like it! >> > > Glad you like it :) > > >> There is only one problem, how can we log the used bandwidth used by the >> varnish daemon? Normally in Apache we can use the access logs but in varnish >> there is only memory logging. >> >> Can anyone tell me how to get the bandwidth used by a specific daemon? >> > > There are a few different approaches. I'm sure there's some OS-method too, > but a common way of doing it is reading it from varnishstat, as that only > resets when Varnish restarts. The Munin plugin for Varnish does exactly > that (among other things). > > Essentially executing varnishstat -1 on a regular interval. Could be read > directly from the shmlog too, but that's a bit more complicated. > > - Kristian > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > From michal.taborsky at nrholding.com Tue Jul 13 07:30:08 2010 From: michal.taborsky at nrholding.com (Michal Taborsky - Netretail Holding) Date: Tue, 13 Jul 2010 09:30:08 +0200 Subject: using round-robin to redirect some traffic In-Reply-To: <4C3C01A0.6090600@d6.com> References: <4C3C01A0.6090600@d6.com> Message-ID: <4C3C1600.5040205@nrholding.com> Dne 13.7.2010 8:03, Chris Hecker napsal(a): > > Can I use the director round-robin feature to then check the > req.backend with an if statement in vcl_recv, and issue a 302 (using > the vcl_error 750 hack) to redirect half the traffic to another > server? Is there a better way to do that? > > Chris Hello Chris, I am not sure I understand what exactly are you trying to accomplish, but if you only want to have two identical servers behind your Varnish serving the traffic, that is a very common scenario. You just define a director with your servers in it and then direct your traffic to this director. You can choose either the random director, which allows you to weigh the backends and distribute the load unevenly, or the round-robin type, which is essentially the same as a random with equal weights. Check out http://www.varnish-cache.org/wiki/LoadBalancing If your needs are different (you really need to redirect to a different URL), please clarify. -- Michal T?borsk? chief systems architect Netretail Holding, B.V. http://www.nrholding.com From tfheen at varnish-software.com Tue Jul 13 07:31:46 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 13 Jul 2010 09:31:46 +0200 Subject: Troubleshooting: Setting up varnish 2.1.2 on OpenSolaris In-Reply-To: (Jacques's message of "Wed, 7 Jul 2010 13:47:21 -0700") References: Message-ID: <87tyo37s7x.fsf@qurzaw.linpro.no> ]] Jacques | I always get 503s. When I check the vmlog, I see backend server not | available. To further test, I upped the timeout values in a basic vcl just | in case that was the problem. No change. Solaris doesn't support setting the TCP timeouts, so changing those won't actually make any difference. | My question is, how can I figure out why I'm getting backend server not | available? If I add a probe, I still see no outbound requests and backends | start at healthy and then become sick shortly after (where they stay). Run varnishlog and look at the output (or post it here). -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Tue Jul 13 07:43:18 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 13 Jul 2010 09:43:18 +0200 Subject: Get Bandwidth statistics In-Reply-To: (Cosimo Streppone's message of "Mon, 12 Jul 2010 14:51:26 +0200") References: <20100712114356.GD3278@sunrider> Message-ID: <87pqyr7rop.fsf@qurzaw.linpro.no> ]] "Cosimo Streppone" | I realize it's probably blasphemy, but is there a way to limit bw in | Varnish? No, but you can use tc (on Linux) to limit bandwidth usage. I assume the BSDs have similar tools. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From checker at d6.com Tue Jul 13 07:44:30 2010 From: checker at d6.com (Chris Hecker) Date: Tue, 13 Jul 2010 00:44:30 -0700 Subject: using round-robin to redirect some traffic In-Reply-To: <4C3C1600.5040205@nrholding.com> References: <4C3C01A0.6090600@d6.com> <4C3C1600.5040205@nrholding.com> Message-ID: <4C3C195E.6080800@d6.com> Yeah, I don't want all the network bandwidth going through the varnish server, I want to shove some of it over to another machine. The traffic for the redirect is fine, but the actual file download needs to come from somewhere else. Make sense? Chris On 2010/07/13 00:30, Michal Taborsky - Netretail Holding wrote: > Dne 13.7.2010 8:03, Chris Hecker napsal(a): >> >> Can I use the director round-robin feature to then check the >> req.backend with an if statement in vcl_recv, and issue a 302 (using >> the vcl_error 750 hack) to redirect half the traffic to another >> server? Is there a better way to do that? >> >> Chris > > Hello Chris, > > I am not sure I understand what exactly are you trying to accomplish, > but if you only want to have two identical servers behind your Varnish > serving the traffic, that is a very common scenario. You just define a > director with your servers in it and then direct your traffic to this > director. You can choose either the random director, which allows you to > weigh the backends and distribute the load unevenly, or the round-robin > type, which is essentially the same as a random with equal weights. > > Check out http://www.varnish-cache.org/wiki/LoadBalancing > > If your needs are different (you really need to redirect to a different > URL), please clarify. > From phk at phk.freebsd.dk Tue Jul 13 08:03:01 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 13 Jul 2010 08:03:01 +0000 Subject: varnish security In-Reply-To: Your message of "Mon, 12 Jul 2010 12:05:43 MST." <4C3B6787.4000703@d6.com> Message-ID: <8159.1279008181@critter.freebsd.dk> In message <4C3B6787.4000703 at d6.com>, Chris Hecker writes: > >> Uhm, no, you pointed to the message with the bogo-advisory and I do >> not seem to be able to find any ensuing discussion from there ? > >Scroll down? Search for "Vendor Response", it's got your complete >email/rant. :) Yes, but you indicated a further discussion ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From cosimo at streppone.it Tue Jul 13 08:13:22 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Tue, 13 Jul 2010 10:13:22 +0200 Subject: Get Bandwidth statistics In-Reply-To: <87pqyr7rop.fsf@qurzaw.linpro.no> References: <20100712114356.GD3278@sunrider> <87pqyr7rop.fsf@qurzaw.linpro.no> Message-ID: On Tue, 13 Jul 2010 09:43:18 +0200, Tollef Fog Heen wrote: > ]] "Cosimo Streppone" > > | I realize it's probably blasphemy, but is there a way to limit bw in > | Varnish? > > No, but you can use tc (on Linux) to limit bandwidth usage. I assume > the BSDs have similar tools. Thanks for the tip. I didn't know it. -- Cosimo From mjdavies at glam.ac.uk Tue Jul 13 10:19:10 2010 From: mjdavies at glam.ac.uk (Davies Matt J A (LCSS)) Date: Tue, 13 Jul 2010 10:19:10 +0000 Subject: Set default ttl in seconds VARNISH_TTL=120 Message-ID: <4EC8A4F6-DC05-4463-AC2E-D2DBDA274E97@glam.ac.uk> Morning all I'm just starting to use a config file for my varnish 2.0.4 and an init.d script file, I was just running it from the command line previously. I've got it to work, but I'm a bit confused about one thing. # Set default ttl in seconds VARNISH_TTL=120 Can anyone tell me what that does exactly? Regards Matt From kacperw at gmail.com Tue Jul 13 10:32:43 2010 From: kacperw at gmail.com (Kacper Wysocki) Date: Tue, 13 Jul 2010 12:32:43 +0200 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: On Tue, Jul 13, 2010 at 3:11 AM, Jacques wrote: > Yes, wget works fine. ?As does lynx, etc. ?:D > I checked ulimit per Jorge's response as well. ?No impact. > I ran make test, (which I should have done before). ?123 of 174 tests > failed. ?All the same 503 issue. > I'm going to recompile with debugging enabled and see what I can see. > FYI, the log of tests is attached if you have any insights. I'm getting the same thing on my rig. Varnish _can't_ work with these tests failing. The firstmost basic test checks if anything gets through by setting up a local backend, and even this fails: # top TEST ././tests/b00000.vtc starting # top TEST Does anything get through at all ? Since Jorge is running his rig fine, is there a big difference between Solaris10 and OpenSolaris? Maybe in what kind of libc is employed? Like a libc which doesn't have thread-safe error handling? A little bit of debugging shows me that child creation fails due to basic thread safety checks, the relevant line being: Child (21308) Panic message: Assert error in wrk_herdtimer_thread(), cache_pool.c line 419: root at os# varnishd -a :80 -b 87.238.47.204:80 -d Message from C-compiler: cc: unrecognized option `-Kpic' NB: Storage size limited to 2GB on 32 bit architecture, NB: otherwise we could run out of address space. storage_file: filename: ./varnish.DcaGMP size 2047 MB. Using old SHMFILE Varnish on -sfile,-hcritbit 200 193 ----------------------------- Varnish HTTP accelerator CLI. ----------------------------- Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. start child (21308) Started Pushing vcls failed: CLI communication error Stopping Child 200 0 Child (21308) died signal=6 Child (21308) Panic message: Assert error in wrk_herdtimer_thread(), cache_pool.c line 419: Condition(errno_is_multi_threaded != 0) not true. thread = (wrk_herdtimer) ident = -sfile,-hcritbit,no_waiter Backtrace: 80730cd: /usr/local/sbin/varnishd'pan_ic+0xa9 [0x80730cd] 8074c8e: /usr/local/sbin/varnishd'wrk_herdtimer_thread+0x86 [0x8074c8e] fedacd56: /lib/libc.so.1'_thrp_setup+0x7e [0xfedacd56] fedacfe0: /lib/libc.so.1'_lwp_start+0x0 [0xfedacfe0] Child (-1) said Child (-1) said Child starts Child (-1) said managed to mmap 2147479552 bytes of 2147479552 Child cleanup complete ( yup , for some reason it's a 32-bit install :-() 0K From alex at bengler.no Tue Jul 13 15:18:52 2010 From: alex at bengler.no (Alexander Staubo) Date: Tue, 13 Jul 2010 17:18:52 +0200 Subject: Varnish constantly restarting In-Reply-To: <4C33432E.7040007@yahoo.com.br> References: <4C2F6CF4.3030705@yahoo.com.br> <4C32A5FD.6030202@yahoo.com.br> <4C33432E.7040007@yahoo.com.br> Message-ID: On Tue, Jul 6, 2010 at 4:52 PM, Flavio Torres wrote: > CPU, Memory, disk. 2 x quad-core Intel X5570, 8MB L2 cache, 24GB RAM, SCSI disk. Ubuntu Hardy running Linux 2.6.24-24, 64-bit, CFQ scheduler. I don't think this is a tuning issue. There are no signs that Varnish is being overloaded when it hangs; every metric we are tracking is normal at the time of each incident. > Is your varnish server a DomU ? No virtualization. From whshub at gmail.com Tue Jul 13 16:09:04 2010 From: whshub at gmail.com (Jacques) Date: Tue, 13 Jul 2010 09:09:04 -0700 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: Can you try to install with this as the configure: VCC_CC="/usr/bin/gcc -pthreads -fpic -shared -m64 -o %o %s" \ CC=/usr/bin/gcc \ CFLAGS="-pthreads -m64" \ LDFLAGS="-pthreads" \ ./configure --prefix=/opt/extra --enable-debugging-symbols --enable-diagnostics --enable-dependency-tracking Or change the VCC_CC path if you have your gcc at a different location. I don't get any panics. The child process starts and runs fine. Everything runs just fine (except the 503s). I've connected up to the child process with gdb just fine. My question now is where/what should I breakpoint to see why it is coming back as backend not connected? Thanks, Jacques ---------------------------------------- # /opt/extra/sbin/varnishd -a :80 -b 87.238.47.204:80 -F -p "connect_timeout=0s" -s file,/export/varnish_cache.bin,512M -p waiter=poll storage_file: filename: /export/varnish_cache.bin size 512 MB. Using old SHMFILE child (12769) Started Child (12769) said Child (12769) said Child starts Child (12769) said managed to mmap 536870912 bytes of 536870912 <<<>>> ^CManager got SIGINT Stopping Child -------------------------------------------- On Tue, Jul 13, 2010 at 3:32 AM, Kacper Wysocki wrote: > On Tue, Jul 13, 2010 at 3:11 AM, Jacques wrote: > > Yes, wget works fine. As does lynx, etc. :D > > I checked ulimit per Jorge's response as well. No impact. > > I ran make test, (which I should have done before). 123 of 174 tests > > failed. All the same 503 issue. > > > I'm going to recompile with debugging enabled and see what I can see. > > FYI, the log of tests is attached if you have any insights. > > I'm getting the same thing on my rig. > > Varnish _can't_ work with these tests failing. > The firstmost basic test checks if anything gets through by setting up > a local backend, and even this fails: > # top TEST ././tests/b00000.vtc starting > # top TEST Does anything get through at all ? > > Since Jorge is running his rig fine, is there a big difference between > Solaris10 and OpenSolaris? Maybe in what kind of libc is employed? > Like a libc which doesn't have thread-safe error handling? A little > bit of debugging shows me that child creation fails due to basic > thread safety checks, the relevant line being: > > Child (21308) Panic message: Assert error in wrk_herdtimer_thread(), > cache_pool.c line 419: > > > root at os# varnishd -a :80 -b 87.238.47.204:80 -d > Message from C-compiler: > cc: unrecognized option `-Kpic' > NB: Storage size limited to 2GB on 32 bit architecture, > NB: otherwise we could run out of address space. > storage_file: filename: ./varnish.DcaGMP size 2047 MB. > Using old SHMFILE > Varnish on -sfile,-hcritbit > 200 193 > ----------------------------- > Varnish HTTP accelerator CLI. > ----------------------------- > Type 'help' for command list. > Type 'quit' to close CLI session. > Type 'start' to launch worker process. > > start > child (21308) Started > Pushing vcls failed: CLI communication error > Stopping Child > 200 0 > > Child (21308) died signal=6 > Child (21308) Panic message: Assert error in wrk_herdtimer_thread(), > cache_pool.c line 419: > Condition(errno_is_multi_threaded != 0) not true. > thread = (wrk_herdtimer) > ident = -sfile,-hcritbit,no_waiter > Backtrace: > 80730cd: /usr/local/sbin/varnishd'pan_ic+0xa9 [0x80730cd] > 8074c8e: /usr/local/sbin/varnishd'wrk_herdtimer_thread+0x86 [0x8074c8e] > fedacd56: /lib/libc.so.1'_thrp_setup+0x7e [0xfedacd56] > fedacfe0: /lib/libc.so.1'_lwp_start+0x0 [0xfedacfe0] > > > Child (-1) said > Child (-1) said Child starts > Child (-1) said managed to mmap 2147479552 bytes of 2147479552 > Child cleanup complete > > ( yup , for some reason it's a 32-bit install :-() > > 0K > -------------- next part -------------- An HTML attachment was scrubbed... URL: From checker at d6.com Tue Jul 13 17:03:30 2010 From: checker at d6.com (Chris Hecker) Date: Tue, 13 Jul 2010 10:03:30 -0700 Subject: varnish security In-Reply-To: <8159.1279008181@critter.freebsd.dk> References: <8159.1279008181@critter.freebsd.dk> Message-ID: <4C3C9C62.1060706@d6.com> > Yes, but you indicated a further discussion ? No, I just meant the "thread" on that page, the advisory, followup, your response, etc. Chris On 2010/07/13 01:03, Poul-Henning Kamp wrote: > In message<4C3B6787.4000703 at d6.com>, Chris Hecker writes: >> >>> Uhm, no, you pointed to the message with the bogo-advisory and I do >>> not seem to be able to find any ensuing discussion from there ? >> >> Scroll down? Search for "Vendor Response", it's got your complete >> email/rant. :) > > Yes, but you indicated a further discussion ? > > From lars at oddbit.com Tue Jul 13 17:55:04 2010 From: lars at oddbit.com (Lars Kellogg-Stedman) Date: Tue, 13 Jul 2010 13:55:04 -0400 Subject: Are ESI requests limited to originating backend? Message-ID: Will in Varnish only fetch content from the backend that served the document containing the directive? Or can be used to include arbitrary content? That is, is it possible to do something like this: ...and if not, is this by design? Results so far indicate that ESI requests always go over the backend that sourced the containing document, but I wanted to verify this behavior. Thanks! -- Lars From phk at phk.freebsd.dk Tue Jul 13 18:18:27 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 13 Jul 2010 18:18:27 +0000 Subject: Are ESI requests limited to originating backend? In-Reply-To: Your message of "Tue, 13 Jul 2010 13:55:04 -0400." Message-ID: <40278.1279045107@critter.freebsd.dk> In message , Lars Kellogg-Stedman writes: >Will in Varnish only fetch content from the backend that >served the document containing the directive? No, you can also specify the host part. Your vcl_recv{} is responsible for setting the right backend based on the req.http.host header. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Jul 13 18:20:25 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 13 Jul 2010 18:20:25 +0000 Subject: varnish security In-Reply-To: Your message of "Tue, 13 Jul 2010 10:03:30 MST." <4C3C9C62.1060706@d6.com> Message-ID: <40324.1279045225@critter.freebsd.dk> In message <4C3C9C62.1060706 at d6.com>, Chris Hecker writes: > >> Yes, but you indicated a further discussion ? > >No, I just meant the "thread" on that page, the advisory, followup, your >response, etc. Ahh ok. Sorry for my misunderstanding then. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fla_torres at yahoo.com.br Wed Jul 14 00:50:42 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Tue, 13 Jul 2010 21:50:42 -0300 Subject: Set default ttl in seconds VARNISH_TTL=120 In-Reply-To: <4EC8A4F6-DC05-4463-AC2E-D2DBDA274E97@glam.ac.uk> References: <4EC8A4F6-DC05-4463-AC2E-D2DBDA274E97@glam.ac.uk> Message-ID: <4C3D09E2.7040803@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Em 13/7/2010 07:19, Davies Matt J A (LCSS) escreveu: > I'm just starting to use a config file for my varnish 2.0.4 and an init.d script file, I was just running it from the command line previously. > > I've got it to work, but I'm a bit confused about one thing. > > # Set default ttl in seconds > VARNISH_TTL=120 > > Can anyone tell me what that does exactly? > Hey Matt, - -- - -t ttl Specifies a hard minimum time to live for cached documents. This is a shortcut for specifying the default_ttl run-time parameter. - -- Here you will find more information about daemon options: http://varnish-cache.org/docs/reference/varnishd/ ;) hope this helps []'s - -- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (MingW32) iEYEARECAAYFAkw9CeEACgkQNRQApncg294DYgCgsjav2DYrYENIK5UHDCOHm6c3 Ak8AoKNmLHGctB/Z9YGfsTLiDkMU1xKV =Xk9y -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From lars at oddbit.com Wed Jul 14 01:02:14 2010 From: lars at oddbit.com (Lars Kellogg-Stedman) Date: Tue, 13 Jul 2010 21:02:14 -0400 Subject: Are ESI requests limited to originating backend? In-Reply-To: <40278.1279045107@critter.freebsd.dk> References: <40278.1279045107@critter.freebsd.dk> Message-ID: > Your vcl_recv{} is responsible for setting the right backend based on > the req.http.host header. Is there any way to explicitly distinguish between an ESI request and a request from an outside client? Clients can provide arbitrary Host: headers -- so that's not really an option -- and it looks like for an ESI request that client.ip is still set to the address of the originating client. Or is the answer, "don't do that"? Thanks, -- Lars From phk at phk.freebsd.dk Wed Jul 14 08:22:27 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 14 Jul 2010 08:22:27 +0000 Subject: Are ESI requests limited to originating backend? In-Reply-To: Your message of "Tue, 13 Jul 2010 21:02:14 -0400." Message-ID: <42998.1279095747@critter.freebsd.dk> In message , Lars Kellogg-Stedman writes: > >Is there any way to explicitly distinguish between an ESI request and >a request from an outside client? No easy way. But you can set set a "magic" hostname you can recognize and de-magic in vcl_recv. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slaweuk at gmail.com Wed Jul 14 10:44:47 2010 From: slaweuk at gmail.com (Slawek) Date: Wed, 14 Jul 2010 11:44:47 +0100 Subject: Exposing a real name of the backend in vcl_recv. Message-ID: Hi everyone, Is there way to determine a real backend name when using a director in vcl_recv? When I check req.backend it contains director's name, not the backend's. I'm trying to modify req.url to value dependent on backend selection. Kind Regards, Slawek -------------- next part -------------- An HTML attachment was scrubbed... URL: From kacperw at gmail.com Wed Jul 14 18:41:14 2010 From: kacperw at gmail.com (Kacper Wysocki) Date: Wed, 14 Jul 2010 20:41:14 +0200 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: On Tue, Jul 13, 2010 at 6:09 PM, Jacques wrote: > Can you try to install with this as the configure: > VCC_CC="/usr/bin/gcc?-pthreads -fpic -shared -m64 -o %o %s" \ > CC=/usr/bin/gcc \ > CFLAGS="-pthreads -m64" \ > LDFLAGS="-pthreads" \ > ./configure --prefix=/opt/extra --enable-debugging-symbols > --enable-diagnostics --enable-dependency-tracking > Or change the VCC_CC path if you have your gcc at a different location. > I don't get any panics. ?The child process starts and runs fine. ?Everything > runs just fine (except the 503s). > I've connected up to the child process with gdb just fine. ?My question now > is where/what should I breakpoint to see why it is coming back as backend > not connected? -pthreads seems to fix the issue I was seeing so that's not it and I can't reproduce your problem. Your 503's need debugging as per my 1st post: "make check" should report "All 174 tests passed" 9 FetchError c no backend connection you are hitting cache_fetch.c:357. While debugging, try different directors & debug.health. hth, 0k From j.vanarragon at lukkien.com Thu Jul 15 14:58:36 2010 From: j.vanarragon at lukkien.com (Jaap van Arragon) Date: Thu, 15 Jul 2010 16:58:36 +0200 Subject: Run Multiple Varnish Daemons Message-ID: Hello, I know it is possible to run multiple Varnish daemons on one server with the ?n option but why is it that I only see one daemon In my process list? When I start a second daemon with the command described below I am getting no real error. jvanarragon at VARNISH:/var/log/varnish$ sudo /usr/sbin/varnishd -P /var/run/testjaap.pid -a :80 -T localhost:6082 -f /etc/varnish/testjaap.vcl -s file,/export/varnish/cache/varnish_storage_testlukkien.bin,1G -w 75,2500,120 -n TESTJAAP WARNING: (-sfile) file size reduced to 22238003200 (80% of available disk space) NB: Storage size limited to 2GB on 32 bit architecture, NB: otherwise we could run out of address space. storage_file: filename: /export/varnish/cache/varnish_storage_testjaap.bin size 2047 MB. Using old SHMFILE I already have a daemon started but it seems that I can?t start a second one. WORKING DAEMON: /usr/sbin/varnishd -P /var/run/jaap.pid -a :80 -T localhost:6082 -f /etc/varnish/jaap.vcl -s file,/export/varnish/cache/varnish_storage.bin,2G -w 75,2500,120 -n JAAP Thank you in advance. Jaap -------------- next part -------------- An HTML attachment was scrubbed... URL: From scaunter at topscms.com Thu Jul 15 17:19:57 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Thu, 15 Jul 2010 13:19:57 -0400 Subject: thread_pool_max - is it total of all thread pools? Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C0C89B5@TMG-EVS02.torstar.net> Hi, running varnish on dual processor, dual core system with 4GB ram, we hit thread pool limit (500) this morning, and so are defining 4 thread pools, 200 min, and a 2500 maximum limit. I now see 800 threads created on start. I cannot see in the docs anything that says max limit is for all thread pools. We've set kern.threads.max_threads_per_proc to 4096 and increased the other tuning settings. Can we assume thread_pool_max is the summary of all thread pools? Can we get the thread pool maximum higher with better hardware and a custom kernel set for higher tuning settings? Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From kb+varnish at slide.com Thu Jul 15 20:07:50 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Thu, 15 Jul 2010 13:07:50 -0700 Subject: thread_pool_max - is it total of all thread pools? In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C0C89B5@TMG-EVS02.torstar.net> References: <7F0AA702B8A85A4A967C4C8EBAD6902C0C89B5@TMG-EVS02.torstar.net> Message-ID: The min is per pool, the max is aggregate. That was a little confusing to me too, at first. Memory (stacks) is the primary bottleneck; you'll run out RAM long before modern kernels have trouble with thread counts. Note that increasing the thread count is only useful (vs being a crutch for a bad backend) for long-lived or unreliable backend requests and long-lived client requests (mainly HTTP keepalives). Past around double your CPU core count is all you'd need if Varnish were purely CPU-bound. And that will happen only with many gigabits of traffic. :-) So, assuming you're not obscuring some sort of failure up- or down-stream from Varnish, I'd tune your max to something like 2-3x your steady state worker count, and increase as necessary. Hope it helps, -- Ken On Jul 15, 2010, at 10:19 AM, Caunter, Stefan wrote: > Hi, running varnish on dual processor, dual core system with 4GB ram, we hit thread pool limit (500) this morning, and so are defining 4 thread pools, 200 min, and a 2500 maximum limit. > > I now see 800 threads created on start. I cannot see in the docs anything that says max limit is for all thread pools. We?ve set kern.threads.max_threads_per_proc to 4096 and increased the other tuning settings. > > Can we assume thread_pool_max is the summary of all thread pools? > > Can we get the thread pool maximum higher with better hardware and a custom kernel set for higher tuning settings? > > Stefan Caunter :: Senior Systems Administrator :: TOPS > e: scaunter at topscms.com :: m: (416) 561-4871 > www.thestar.com www.topscms.com > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From kh2008 at kennethhunt.com Thu Jul 15 20:14:50 2010 From: kh2008 at kennethhunt.com (kh2008 at kennethhunt.com) Date: Thu, 15 Jul 2010 16:14:50 -0400 Subject: CDN Application Proxy Front End With Varnish Message-ID: <33167f6a122db63da51590b8f4a064b0.squirrel@sm.webmail.pair.com> I am trying to support COMPRESSION between an IIS web server and a .net SOAP client on the user side. I have configured the server to support compression, but the CLIENT does not send ACCEPT_ENCODING requests. My question , is if using something like the following: sub vcl_recv { if (req.request == "POST") { set req.http.Accept-Encoding = "gzip"; } } I append this request from the PROXY to the server -- but the reply I get back and sent to the client is also compressed. Is it possible to remove the compression from the reply before this is sent to client computer? I know this is probably not an ideal case, but I'm asking for a technical feasibility perspective. In deployment I would have an varnish app proxy at a remote site and this would optimize the WAN traffic for the application. I did a lot of searches before posting this and the closest I found to anyone doing something similar is using VARNISH as a CDN frontend. I am wondering is REMOVE or UNSET the default.vcl actions I need to use and would this go into sub vcl_deliver subsection? Thanks From scaunter at topscms.com Thu Jul 15 20:22:50 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Thu, 15 Jul 2010 16:22:50 -0400 Subject: CDN Application Proxy Front End With Varnish In-Reply-To: <33167f6a122db63da51590b8f4a064b0.squirrel@sm.webmail.pair.com> References: <33167f6a122db63da51590b8f4a064b0.squirrel@sm.webmail.pair.com> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C0C8A4D@TMG-EVS02.torstar.net> > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of kh2008 at kennethhunt.com > Sent: July-15-10 4:15 PM > To: varnish-misc at varnish-cache.org > Subject: CDN Application Proxy Front End With Varnish > > I am trying to support COMPRESSION between an IIS web server and a .net > SOAP client on the user side. > > I have configured the server to support compression, but the CLIENT does > not send ACCEPT_ENCODING requests. > > My question , is if using something like the following: > > sub vcl_recv { > if (req.request == "POST") > { > set req.http.Accept-Encoding = "gzip"; > } > } > I append this request from the PROXY to the server -- but the reply I get > back and sent to the client is also compressed. Is it possible to remove > the compression from the reply before this is sent to client computer? Does it make a difference if you unset compression, either in web.config in the relevant IIS directory, or at the website level? > > Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com > ~ ~ From checker at d6.com Fri Jul 16 06:57:14 2010 From: checker at d6.com (Chris Hecker) Date: Thu, 15 Jul 2010 23:57:14 -0700 Subject: unset req.http.Cookie and TxHeader Message-ID: <4C4002CA.1090707@d6.com> I'm trying to debug some cookie weirdness, and I stuck an unset req.http.Cookie at the top of my vcl_recv, but I'm still getting TxHeader b Cookie: blah lines in my varnishlog for misses. Should those be there? Do I need to kill bereq.http.Cookie as well or something? Chris From checker at d6.com Fri Jul 16 11:00:47 2010 From: checker at d6.com (Chris Hecker) Date: Fri, 16 Jul 2010 04:00:47 -0700 Subject: varnishlog and multiple tags? Message-ID: <4C403BDF.4090004@d6.com> varnishlog -i TxUrl is really useful, but I'd also like it to output the TxHeader Host: entry, since I'm using varnish for multiple sites. Is that possible with the -i, -I, and -o options? I couldn't figure it out, if so. Thanks, Chris From kb+varnish at slide.com Fri Jul 16 19:34:07 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Fri, 16 Jul 2010 12:34:07 -0700 Subject: unset req.http.Cookie and TxHeader In-Reply-To: <4C4002CA.1090707@d6.com> References: <4C4002CA.1090707@d6.com> Message-ID: Yes, I believe the structures are all populated before vcl_recv, so you need to unset the bereq header. It also makes more sense conceptually, IMHO. -- Ken On Jul 15, 2010, at 11:57 PM, Chris Hecker wrote: > > I'm trying to debug some cookie weirdness, and I stuck an unset req.http.Cookie at the top of my vcl_recv, but I'm still getting > > TxHeader b Cookie: blah > > lines in my varnishlog for misses. Should those be there? Do I need to kill bereq.http.Cookie as well or something? > > Chris > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From kb+varnish at slide.com Fri Jul 16 19:37:12 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Fri, 16 Jul 2010 12:37:12 -0700 Subject: varnishlog and multiple tags? In-Reply-To: <4C403BDF.4090004@d6.com> References: <4C403BDF.4090004@d6.com> Message-ID: varnishlog -i TxURL,TxHeader -- Ken On Jul 16, 2010, at 4:00 AM, Chris Hecker wrote: > > varnishlog -i TxUrl is really useful, but I'd also like it to output the TxHeader Host: entry, since I'm using varnish for multiple sites. Is that possible with the -i, -I, and -o options? I couldn't figure it out, if so. > > Thanks, > Chris > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From fla_torres at yahoo.com.br Fri Jul 16 19:46:32 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Fri, 16 Jul 2010 16:46:32 -0300 Subject: varnishlog and multiple tags? In-Reply-To: <4C403BDF.4090004@d6.com> References: <4C403BDF.4090004@d6.com> Message-ID: <4C40B718.90803@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/16/2010 08:00 AM, Chris Hecker wrote: > > varnishlog -i TxUrl is really useful, but I'd also like it to > output the TxHeader Host: entry, since I'm using varnish for > multiple sites. Is that possible with the -i, -I, and -o options? > I couldn't figure it out, if so. Hi Chris, Try with awk or perl regexp. varnishlog -o |awk 'BEGIN { FS="n"; RS=""} {if (/RxURL.*\/imgs\/aba_frente_esq\.jpg/) print }' hope this helps -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkxAtxUACgkQNRQApncg294JdwCgs9p9poidxfcGefjOJ+lW7ETv oi8AoIyTmJ0djy1Dn7QFY58605KubxWb =R8EV -----END PGP SIGNATURE----- -------------- next part -------------- An HTML attachment was scrubbed... URL: From kb+varnish at slide.com Fri Jul 16 20:03:34 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Fri, 16 Jul 2010 13:03:34 -0700 Subject: varnishlog and multiple tags? In-Reply-To: <4C40B718.90803@yahoo.com.br> References: <4C403BDF.4090004@d6.com> <4C40B718.90803@yahoo.com.br> Message-ID: <849AF8E0-D624-4E8A-8490-0E4C9BA47EBF@slide.com> True, but perhaps cleaner: varnishlog -i RxURL -I /imgs/aba_frente_esq.jpg -- Ken On Jul 16, 2010, at 12:46 PM, Flavio Torres wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 07/16/2010 08:00 AM, Chris Hecker wrote: > > > > > varnishlog -i TxUrl is really useful, but I'd also like it to > > > output the TxHeader Host: entry, since I'm using varnish for > > > multiple sites. Is that possible with the -i, -I, and -o > options? > > > I couldn't figure it out, if so. > > > Hi Chris, > > Try with awk or perl regexp. > > varnishlog -o |awk 'BEGIN { FS="n"; RS=""} {if > (/RxURL.*\/imgs\/aba_frente_esq\.jpg/) print }' > > > hope this helps > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkxAtxUACgkQNRQApncg294JdwCgs9p9poidxfcGefjOJ+lW7ETv > oi8AoIyTmJ0djy1Dn7QFY58605KubxWb > =R8EV > -----END PGP SIGNATURE----- > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From checker at d6.com Fri Jul 16 20:04:51 2010 From: checker at d6.com (Chris Hecker) Date: Fri, 16 Jul 2010 13:04:51 -0700 Subject: unset req.http.Cookie and TxHeader In-Reply-To: References: <4C4002CA.1090707@d6.com> Message-ID: <4C40BB63.4090403@d6.com> Digging a bit more, I'm still getting these pages cached, which wouldn't happen if they had cookies, no? Then I found this in my vcl (which I've cobbled together from various ones I've found without really understanding it all, my bad): sub vcl_pipe { # do the right XFF processing set bereq.http.X-Forwarded-For = req.http.X-Forwarded-For; set bereq.http.X-Forwarded-For = regsub(bereq.http.X-Forwarded-For, "$", ", "); set bereq.http.X-Forwarded-For = regsub(bereq.http.X-Forwarded-For, "$", client.ip); set bereq.http.Cookie = req.http.X-Orig-Cookie; set bereq.http.connection = "close"; } The vcl_recv is copying req.http.Cookie to X-Orig-Cookie, and then clearing it, presumably to get caching, but then this restores it just for the backend request. This came from a vcl recommended by the mediawiki folks, I think. I need to sit down and really figure out what I need to be doing here. I've got a single vcl feeding wordpress and mediawiki, and I've got some cookie wonkiness, but things mostly work, so I'm reticent to touch it. Chris On 2010/07/16 12:34, Ken Brownfield wrote: > Yes, I believe the structures are all populated before vcl_recv, so you need to unset the bereq header. It also makes more sense conceptually, IMHO. From checker at d6.com Fri Jul 16 20:06:28 2010 From: checker at d6.com (Chris Hecker) Date: Fri, 16 Jul 2010 13:06:28 -0700 Subject: varnishlog and multiple tags? In-Reply-To: References: <4C403BDF.4090004@d6.com> Message-ID: <4C40BBC4.7060407@d6.com> Ah, cool. But then how do I filter on the Host: but just on the txrequest part? Or I guess I could just pipe to grep at this point. Chris On 2010/07/16 12:37, Ken Brownfield wrote: > varnishlog -i TxURL,TxHeader On Jul 16, 2010, at 4:00 AM, Chris Hecker wrote: > > > > varnishlog -i TxUrl is really useful, but I'd also like it to output the TxHeader Host: entry, since I'm using varnish for multiple sites. Is that possible with the -i, -I, and -o options? I couldn't figure it out, if so. > > > > Thanks, > > Chris > > From checker at d6.com Fri Jul 16 20:10:44 2010 From: checker at d6.com (Chris Hecker) Date: Fri, 16 Jul 2010 13:10:44 -0700 Subject: varnishlog and multiple tags? In-Reply-To: <4C40BBC4.7060407@d6.com> References: <4C403BDF.4090004@d6.com> <4C40BBC4.7060407@d6.com> Message-ID: <4C40BCC4.7080807@d6.com> This appears to be doing the trick: varnishlog -b -i txurl,txheader | grep -i -E "(txurl|host:)" Thanks, Chris On 2010/07/16 13:06, Chris Hecker wrote: > > Ah, cool. But then how do I filter on the Host: but just on the > txrequest part? Or I guess I could just pipe to grep at this point. > > Chris > > On 2010/07/16 12:37, Ken Brownfield wrote: >> varnishlog -i TxURL,TxHeader > > > On Jul 16, 2010, at 4:00 AM, Chris Hecker wrote: > > > > > > varnishlog -i TxUrl is really useful, but I'd also like it to > output the TxHeader Host: entry, since I'm using varnish for multiple > sites. Is that possible with the -i, -I, and -o options? I couldn't > figure it out, if so. > > > > > > Thanks, > > > Chris > > > From kb+varnish at slide.com Fri Jul 16 20:31:39 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Fri, 16 Jul 2010 13:31:39 -0700 Subject: varnishlog and multiple tags? In-Reply-To: <4C40BCC4.7080807@d6.com> References: <4C403BDF.4090004@d6.com> <4C40BBC4.7060407@d6.com> <4C40BCC4.7080807@d6.com> Message-ID: <0920112F-D529-436E-8147-94CFF95058EF@slide.com> varnishlog -i TxURL,TxHeader -I '(TxURL|Host:)' :-) -- Ken On Jul 16, 2010, at 1:10 PM, Chris Hecker wrote: > > This appears to be doing the trick: > > varnishlog -b -i txurl,txheader | grep -i -E "(txurl|host:)" > > Thanks, > Chris > > On 2010/07/16 13:06, Chris Hecker wrote: >> >> Ah, cool. But then how do I filter on the Host: but just on the >> txrequest part? Or I guess I could just pipe to grep at this point. >> >> Chris >> >> On 2010/07/16 12:37, Ken Brownfield wrote: >>> varnishlog -i TxURL,TxHeader >> >> >> On Jul 16, 2010, at 4:00 AM, Chris Hecker wrote: >> > > >> > > varnishlog -i TxUrl is really useful, but I'd also like it to >> output the TxHeader Host: entry, since I'm using varnish for multiple >> sites. Is that possible with the -i, -I, and -o options? I couldn't >> figure it out, if so. >> > > >> > > Thanks, >> > > Chris >> > > From checker at d6.com Fri Jul 16 20:35:58 2010 From: checker at d6.com (Chris Hecker) Date: Fri, 16 Jul 2010 13:35:58 -0700 Subject: varnishlog and multiple tags? In-Reply-To: <0920112F-D529-436E-8147-94CFF95058EF@slide.com> References: <4C403BDF.4090004@d6.com> <4C40BBC4.7060407@d6.com> <4C40BCC4.7080807@d6.com> <0920112F-D529-436E-8147-94CFF95058EF@slide.com> Message-ID: <4C40C2AE.2020804@d6.com> sigh. :) Chris On 2010/07/16 13:31, Ken Brownfield wrote: > varnishlog -i TxURL,TxHeader -I '(TxURL|Host:)' > > :-) From checker at d6.com Fri Jul 16 20:38:49 2010 From: checker at d6.com (Chris Hecker) Date: Fri, 16 Jul 2010 13:38:49 -0700 Subject: varnishlog and multiple tags? In-Reply-To: <4C40C2AE.2020804@d6.com> References: <4C403BDF.4090004@d6.com> <4C40BBC4.7060407@d6.com> <4C40BCC4.7080807@d6.com> <0920112F-D529-436E-8147-94CFF95058EF@slide.com> <4C40C2AE.2020804@d6.com> Message-ID: <4C40C359.9030005@d6.com> Hmm, that appears not to work. Only the TxHeader's are coming through for some reason. Chris On 2010/07/16 13:35, Chris Hecker wrote: > > sigh. > > :) > > Chris > > On 2010/07/16 13:31, Ken Brownfield wrote: >> varnishlog -i TxURL,TxHeader -I '(TxURL|Host:)' >> >> :-) From fla_torres at yahoo.com.br Fri Jul 16 21:04:24 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Fri, 16 Jul 2010 18:04:24 -0300 Subject: varnishlog and multiple tags? In-Reply-To: <849AF8E0-D624-4E8A-8490-0E4C9BA47EBF@slide.com> References: <4C403BDF.4090004@d6.com> <4C40B718.90803@yahoo.com.br> <849AF8E0-D624-4E8A-8490-0E4C9BA47EBF@slide.com> Message-ID: <4C40C958.9030901@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/16/2010 05:03 PM, Ken Brownfield wrote: > True, but perhaps cleaner: > > varnishlog -i RxURL -I /imgs/aba_frente_esq.jpg > > Hi Chris, > > Try with awk or perl regexp. > > varnishlog -o |awk 'BEGIN { FS="n"; RS=""} {if > (/RxURL.*\/imgs\/aba_frente_esq\.jpg/) print }' Yeah, but in 'awk' way I can see entire block if pattern match. It's useful to me, to debug objects with MISS/Cookie (-o -c option - (/RxURL.*enquete_br\.js$/m and /VCL_call.*miss/) ) or something like else... ;) Regards -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkxAyVMACgkQNRQApncg2969FgCgsViG589Rz+ha4GmCQ/YztU+l XMQAn0KV5AObjibUV5FNhc2SHogxUwmU =trVk -----END PGP SIGNATURE----- From yves1 at paran.com Mon Jul 19 01:25:28 2010 From: yves1 at paran.com (=?euc-kr?B?WXZlcyBDaG9p?=) Date: Mon, 19 Jul 2010 10:25:28 +0900 Subject: =?euc-kr?B?RndkOiB2aG9zdCBjb25maWd1cmF0aW9u?= Message-ID: <1279502728.16793.mail71x0@mail71> An HTML attachment was scrubbed... URL: From pprocacci at datapipe.com Mon Jul 19 01:43:14 2010 From: pprocacci at datapipe.com (Paul Procacci) Date: Sun, 18 Jul 2010 21:43:14 -0400 Subject: Your evaluations aren't in any defined functions. Try placing them on vcl_recv. Also you should escape your periods in your regexes. Message-ID: <3iv34tyacsh15rm0wy2k8yy3.1279503806661@email.android.com> What's the error? Yves Choi wrote: Dear all, I am testing the Vanish for the reverse proxy for multilple domains. Like as below diagram, When user access to 10.10.10.10 (request http host name with www.aaa.com ,the Vanish foreard the request to www.aaa.com backend server. user 1 Domain www.aaa.com user 2 Vanish (10.10.10.x) Domain www.bbb.com user 3 Domain www.ccc.com I looked up the main page but can't find a right sample for this configuration. This is a configuration that I add in the default vcl file. When I start the vanishd with below configuration , I got an error. Please let me know what I made a mistake. Regards, Yves ---------------- error messages --------------------- Expected one of 'acl', 'sub', 'backend', or 'director' Found: 'if' at (input Line 16 Pos 1) if (req.http.host ~ "^(www.)?aaa.com$") { ##----------------------------------------- Running VCC-compiler failed, exit 1 VCL compilation failed -------------------------------------------------------- ------------ VCL Configuration ---------------------- backend aaa { .host = "1.1.1.1"; .port = "80"; } backend bbb { .host = "2.2.2.2."; .port = "80"; } if (req.http.host ~ "^(www.)?aaa.com$") { set req.http.host = "www.aaa.com"; set req.backend = aaar; } else if (req.http.host ~ "^(www.)?bbb.com$" { set req.http.host = "www.bbb.cpm"; set req.backend = bbb;} else { req.backend = default.; } } [http://mailimg.kthimg.com/paran/footer/mail_banner_100201.gif] [http://mailimg.kthimg.com/paran/footer/mail_banner_100201.gif] [http://mail71.paran.com/read/put_ack.php?hp=L0Q3MS95L3YvMS95dmVzMUBwYXJhbi5jb20vLkNvbmZpcm0vMTI3OTUwMjcyOC4xNjc5My5tYWlsNzF4MQ==&from=yves1 at paran.com&to=varnish-misc at varnish-cache.org________________________________________________________________________________________________________________________] ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From pprocacci at datapipe.com Mon Jul 19 01:45:36 2010 From: pprocacci at datapipe.com (Paul Procacci) Date: Sun, 18 Jul 2010 21:45:36 -0400 Subject: Your evaluations aren't in any defined functions. Try placing them on vcl_recv. Also you should escape your periods in your regexes. Message-ID: Sorry sent from my droid and totally 's foobar'd the response. Paul Procacci wrote: What's the error? Yves Choi wrote: Dear all, I am testing the Vanish for the reverse proxy for multilple domains. Like as below diagram, When user access to 10.10.10.10 (request http host name with www.aaa.com ,the Vanish foreard the request to www.aaa.com backend server. user 1 Domain www.aaa.com user 2 Vanish (10.10.10.x) Domain www.bbb.com user 3 Domain www.ccc.com I looked up the main page but can't find a right sample for this configuration. This is a configuration that I add in the default vcl file. When I start the vanishd with below configuration , I got an error. Please let me know what I made a mistake. Regards, Yves ---------------- error messages --------------------- Expected one of 'acl', 'sub', 'backend', or 'director' Found: 'if' at (input Line 16 Pos 1) if (req.http.host ~ "^(www.)?aaa.com$") { ##----------------------------------------- Running VCC-compiler failed, exit 1 VCL compilation failed -------------------------------------------------------- ------------ VCL Configuration ---------------------- backend aaa { .host = "1.1.1.1"; .port = "80"; } backend bbb { .host = "2.2.2.2."; .port = "80"; } if (req.http.host ~ "^(www.)?aaa.com$") { set req.http.host = "www.aaa.com"; set req.backend = aaar; } else if (req.http.host ~ "^(www.)?bbb.com$" { set req.http.host = "www.bbb.cpm"; set req.backend = bbb;} else { req.backend = default.; } } [http://mailimg.kthimg.com/paran/footer/mail_banner_100201.gif] [http://mailimg.kthimg.com/paran/footer/mail_banner_100201.gif] [http://mail71.paran.com/read/put_ack.php?hp=L0Q3MS95L3YvMS95dmVzMUBwYXJhbi5jb20vLkNvbmZpcm0vMTI3OTUwMjcyOC4xNjc5My5tYWlsNzF4MQ==&from=yves1 at paran.com&to=varnish-misc at varnish-cache.org________________________________________________________________________________________________________________________] ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From whshub at gmail.com Mon Jul 19 02:00:21 2010 From: whshub at gmail.com (Jacques) Date: Sun, 18 Jul 2010 19:00:21 -0700 Subject: What are the steps to problem solve when all I can get are 503s? In-Reply-To: References: Message-ID: Ok, I tried my best to do a debug: Here is what i have. I'm not a c programmer at all but it appears that I'm Varnish is failing to create a socket as the return from the socket creation call is -1 and errno set at 9 which I believe translates to EBADF/Bad file descriptor. Feel free to tell me what I'm doing wrong or why this might be the case... Thanks Jacques # gdb /opt/extra/sbin/varnishd 14236 << bunch of statements >> (gdb) break vbp_connect (gdb) cont Continuing. [New LWP 8 ] [New Thread 8 (LWP 8)] [Switching to Thread 8 (LWP 8)] Breakpoint 1, vbp_connect (pf=2, sa=0x494290, salen=16, tmo=534) at cache_backend_poll.c:107 107 s = socket(pf, SOCK_STREAM, 0); (gdb) n 108 if (s < 0) (gdb) p s $1 = -1 (gdb) p errno $2 = 9 (gdb) p sa $3 = (const struct sockaddr *) 0x494290 (gdb) p sa.sa_family $4 = 2 (gdb) p sa.sa_data $5 = "\000PW?/?\000\000\000\000\000\000\000" (gdb) c Continuing. On Wed, Jul 14, 2010 at 11:41 AM, Kacper Wysocki wrote: > On Tue, Jul 13, 2010 at 6:09 PM, Jacques wrote: > > Can you try to install with this as the configure: > > VCC_CC="/usr/bin/gcc -pthreads -fpic -shared -m64 -o %o %s" \ > > CC=/usr/bin/gcc \ > > CFLAGS="-pthreads -m64" \ > > LDFLAGS="-pthreads" \ > > ./configure --prefix=/opt/extra --enable-debugging-symbols > > --enable-diagnostics --enable-dependency-tracking > > Or change the VCC_CC path if you have your gcc at a different location. > > I don't get any panics. The child process starts and runs fine. > Everything > > runs just fine (except the 503s). > > I've connected up to the child process with gdb just fine. My question > now > > is where/what should I breakpoint to see why it is coming back as backend > > not connected? > > -pthreads seems to fix the issue I was seeing so that's not it and I > can't reproduce your problem. Your 503's need debugging as per my 1st > post: > "make check" should report "All 174 tests passed" > > 9 FetchError c no backend connection > > you are hitting cache_fetch.c:357. While debugging, try different > directors & debug.health. > > hth, > 0k > -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.gruellich at mapsolute.com Mon Jul 19 16:20:47 2010 From: frank.gruellich at mapsolute.com (Frank Gruellich) Date: Mon, 19 Jul 2010 18:20:47 +0200 Subject: "Child dies" on strace to PID Message-ID: <4C447B5F.5030601@mapsolute.com> Hi, I noticed a constant load of 1 on both of our two varnish servers and was going to look into this. They are running CentOS release 5.4, x86_64 and varnish-2.1.2-1 (recompiled from the spec file in the released tar.gz) with the attached vcl and the following command line: # /usr/sbin/varnishd -P /var/run/varnish.pid -a :6081 -f /etc/varnish/frontend_maptpserver.maptp42.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -s file,/var/lib/varnish/varnish_storage.bin,196G Then I got distracted when I strace'd to the varnish child and it immediately exited with: root at paris [~] $ strace -p 15023 Process 15023 attached - interrupt to quit close(7) = 0 close(8) = 0 write(1, "Child dies", 10) = 10 write(1, "\n", 1) = 1 brk(0) = 0x19bfe000 brk(0x19c1f000) = 0x19c1f000 exit_group(1) = ? Process 15023 detached root at paris [~] $ D'oh, all cache gone! It does that every times, so it seems unrelated to the load I see on the system. In /var/log/messages I found: Jul 19 14:34:09 paris varnishd[5102]: child (15023) Started Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Child starts Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said managed to mmap 210453397504 bytes of 210453397504 Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Probe("GET /exodus.html HTTP/1.1 Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Host: apoll.map24.local. Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Connection: close Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris last message repeated 2 times Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said ", 0.03, 1) Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Probe("GET /exodus.html HTTP/1.1 Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Host: gotthard.map24.local. Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Connection: close Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris last message repeated 2 times Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said ", 0.03, 1) Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Probe("GET /exodus.html HTTP/1.1 Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Host: hybris.map24.local. Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Connection: close Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris last message repeated 2 times Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said ", 0.03, 1) Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Probe("GET /exodus.html HTTP/1.1 Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Host: cure.map24.local. Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Connection: close Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris last message repeated 2 times Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said ", 0.03, 1) Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Probe("GET /exodus.html HTTP/1.1 Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Host: hardy.map24.local. Jul 19 14:34:09 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Connection: close Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris last message repeated 2 times Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said ", 0.03, 1) Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Probe("GET /exodus.html HTTP/1.1 Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Host: goblin.map24.local. Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Connection: close Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris last message repeated 2 times Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said ", 0.03, 1) Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Probe("GET /exodus.html HTTP/1.1 Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Host: leto.map24.local. Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Connection: close Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris last message repeated 2 times Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said ", 0.03, 1) Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Probe("GET /exodus.html HTTP/1.1 Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Host: krantor.map24.local. Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Connection: close Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris last message repeated 2 times Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said ", 0.03, 1) Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Probe("GET /exodus.html HTTP/1.1 Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Host: solons.map24.local. Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Connection: close Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris last message repeated 2 times Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said ", 0.03, 1) Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Probe("GET /exodus.html HTTP/1.1 Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Host: white.map24.local. Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Connection: close Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said Jul 19 14:34:10 paris last message repeated 2 times Jul 19 14:34:10 paris varnishd[5102]: Child (15023) said ", 0.03, 1) Jul 19 14:34:39 paris varnishd[5102]: Child (15023) said Child dies Jul 19 14:34:39 paris varnishd[5102]: Child (15023) died I think 14:34:10 is all from the startup and at 14:34:39 I'm starting strace. That looks like the health check to the backend servers, but other then that I have no idea what it means. I run varnishlog to a file and strace'd the child process again. The file is 3MB, so I have uploaded it FTP: ftp://ftp.lbsp.navteq.com/outgoing/2XyKTXBY/. I cannot read much from it, but I think the point where it restarts is somewhere around: [...] 122 ReqEnd c 887294864 1279552540.103346109 1279552540.504165888 0.000036001 0.000036955 0.400782824 160 Debug c "herding" 0 CLI - EOF on CLI connection, worker stops 0 WorkThread - 0x45559e60 start 0 CLI - Rd vcl.load "boot" ./vcl.1P9zoqAU.so 0 Backend_health - apoll Still sick ------H 1 3 5 0.000000 0.000000 0 Backend_health - apoll Still sick ------H 2 3 5 0.000000 0.000000 [...] I don't know what "EOF on CLI" means. Does CLI refer to the management interface via the -T parameter? That was definitely not in use at this time. Any ideas? Should I be able to strace the PID? (I think so.) If not how can I debug the load issue I have? (Oh, BTW: the load is gone from the system after the child died. No, I don't like that way of solving this problem.) Thanks in advance. Kind regards, -- Navteq (DE) GmbH Frank Gruellich Map24 Systems and Networks Duesseldorfer Strasse 40a 65760 Eschborn Germany Phone: +49 6196 77756-414 Fax: +49 6196 77756-100 HRB 46215, Local Court Frankfurt am Main Managing Directors: Thomas Golob, Hans Pieter Gieszen, Martin Robert Stockman USt-ID-No.: DE 197947163 -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: frontend_maptpserver.maptp42.vcl URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 262 bytes Desc: OpenPGP digital signature URL: From mjdavies at glam.ac.uk Mon Jul 19 16:46:14 2010 From: mjdavies at glam.ac.uk (Davies Matt J A (LCSS)) Date: Mon, 19 Jul 2010 16:46:14 +0000 Subject: forwarding original IP address Message-ID: Hi everyone This is probably very easy but I thought I'd check I've got this right first. On my web server, I want to display the original IP address of the client, not the IP of the varnish server, in the error logs from the web server. I've done some reading and found this example in the docs sub vcl_recv { # Add a unique header containing the client address remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = req.http.rlnclientipaddr; Is that all I need to do? Any help, greatly appreciated. From pprocacci at datapipe.com Mon Jul 19 16:54:09 2010 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Mon, 19 Jul 2010 11:54:09 -0500 Subject: forwarding original IP address In-Reply-To: References: Message-ID: <20100719165409.GA92045@nat.myhome> I have the following which works for me! remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip; ~Paul On Mon, Jul 19, 2010 at 12:46:14PM -0400, Davies Matt J A (LCSS) wrote: > Hi everyone > > This is probably very easy but I thought I'd check I've got this right first. > > On my web server, I want to display the original IP address of the client, not the IP of the varnish server, in the error logs from the web server. > > I've done some reading and found this example in the docs > > sub vcl_recv { > > # Add a unique header containing the client address > > remove req.http.X-Forwarded-For; > set req.http.X-Forwarded-For = req.http.rlnclientipaddr; > > Is that all I need to do? > > Any help, greatly appreciated. > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From scaunter at topscms.com Mon Jul 19 16:56:51 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Mon, 19 Jul 2010 12:56:51 -0400 Subject: FetchError - chunked read_error: 35 Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C13D0CB@TMG-EVS02.torstar.net> I have this in vcl_fetch: if (beresp.status != 200 && beresp.status != 403 && beresp.status != 404 && beres p.status != 301 && beresp.status != 302) { restart; set beresp.grace = 4h; } varnishlog shows it is triggered only with Transfer-encoding: chunked, and that any TE chunked response from backend is triggering this error (and restart). Have I missed something in my configuration with IIS7 backends? /Stef 40 VCL_call c fetch 40 TTL c 1332116192 VCL 600 1279557774 40 VCL_return c deliver 40 ObjProtocol c HTTP/1.1 40 ObjStatus c 200 40 ObjResponse c OK 40 ObjHeader c Content-Type: text/html; charset=utf-8 40 ObjHeader c Expires: Mon, 19 Jul 2010 16:42:37 GMT 40 ObjHeader c Server: Microsoft-IIS/7.0 40 ObjHeader c X-AspNet-Version: 2.0.50727 40 ObjHeader c X-Powered-By: ASP.NET 40 ObjHeader c WS: 2-3 40 ObjHeader c Date: Mon, 19 Jul 2010 16:42:37 GMT 40 ObjHeader c cache-control: max-age = 600 40 FetchError c chunked read_error: 35 151 BackendClose b star3 40 VCL_call c error 40 VCL_return c restart 40 VCL_call c recv 40 VCL_return c lookup 40 VCL_call c hash 40 VCL_return c hash 40 VCL_call c miss 40 VCL_return c fetch Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From mjdavies at glam.ac.uk Mon Jul 19 17:04:43 2010 From: mjdavies at glam.ac.uk (Davies Matt J A (LCSS)) Date: Mon, 19 Jul 2010 17:04:43 +0000 Subject: forwarding original IP address In-Reply-To: <20100719165409.GA92045@nat.myhome> References: <20100719165409.GA92045@nat.myhome> Message-ID: <575A2CE8-50FA-486A-BCF1-C315DAA5F295@glam.ac.uk> Hi Paul, everyone. My bad, I tried your version, was getting the same IP in the logs. Then checked what I was actually telling nginx to output at that point, and I wasn't showing the $http_x_forwarded_for value at all :-). Now I am, I can see that it's showing me the right IP. Thanks again. You live, you learn Matt On 19 Jul 2010, at 17:54, Paul A. Procacci wrote: > I have the following which works for me! > > remove req.http.X-Forwarded-For; > set req.http.X-Forwarded-For = client.ip; > > ~Paul > > On Mon, Jul 19, 2010 at 12:46:14PM -0400, Davies Matt J A (LCSS) wrote: >> Hi everyone >> >> This is probably very easy but I thought I'd check I've got this right first. >> >> On my web server, I want to display the original IP address of the client, not the IP of the varnish server, in the error logs from the web server. >> >> I've done some reading and found this example in the docs >> >> sub vcl_recv { >> >> # Add a unique header containing the client address >> >> remove req.http.X-Forwarded-For; >> set req.http.X-Forwarded-For = req.http.rlnclientipaddr; >> >> Is that all I need to do? >> >> Any help, greatly appreciated. >> >> >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From m.becker at netz98.de Tue Jul 20 13:12:12 2010 From: m.becker at netz98.de (Marc Becker - netz98) Date: Tue, 20 Jul 2010 15:12:12 +0200 Subject: Varnish works fine, but not for typo3-backend Message-ID: Hi there, I got varnish running on a debian lenny machine. Everything works fine. The cache gets filled and request to a url containing media, skin or static in the beginning gets a x-header from varnish. The backend of typo3 is not working, I get a 503 error. I attached my config file. Thanks in advance for every hint. Regards Marc Becker -------------- next part -------------- A non-text attachment was scrubbed... Name: website.vcl Type: application/octet-stream Size: 2933 bytes Desc: website.vcl URL: From tfheen at varnish-software.com Tue Jul 20 05:36:05 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 20 Jul 2010 07:36:05 +0200 Subject: Run Multiple Varnish Daemons In-Reply-To: (Jaap van Arragon's message of "Thu, 15 Jul 2010 16:58:36 +0200") References: Message-ID: <87vd8ar9yy.fsf@qurzaw.linpro.no> ]] Jaap van Arragon | I know it is possible to run multiple Varnish daemons on one server with the | ?n option but why is it that I only see one daemon In my process list? Do you get any errors when you run it with -d -d ? Also, you're aware that you need to make them listen to different ip:port combinations? At least your -T arguments are conflicting. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From mjdavies at glam.ac.uk Wed Jul 21 12:13:13 2010 From: mjdavies at glam.ac.uk (Davies Matt J A (LCSS)) Date: Wed, 21 Jul 2010 12:13:13 +0000 Subject: Redirecting based on http verb and URL Message-ID: <588A907C-5409-4F2E-902D-782AC374EFE0@glam.ac.uk> Hello everyone I'm just checking I'm going down the right path here. I've read the example redirect section on the docs here, http://varnish-cache.org/wiki/VCLExampleRedirectInVCL In my case we've just moved web sites, and some crazy dudes are spamming our site, trying to send a POST form to an one page that may have one time allowed it. My Rails app is picking up these attempts at spam and throwing them out, but I want to handle them, and other normal people who simply click on the old link somewhere, with varnish prior to that. Here's my addition to the vcl_recv if (req.http.host == "^(www.)?webiste/blogs/oldpage$") { error 750 "Moved Temporarily"; if (req.request == "POST"){ error 751 "Please go away" } } and vcl_error sub vcl_error { if (obj.status == 750) { set obj.http.Location = "http://newblogs"; set obj.status = 302; deliver; } if (obj.status == 751) { set obj.status = 500; deliver; } } Can anyone see any glaringly obvious stupid bits in there? Thanks again. From iiminar at gmail.com Wed Jul 21 16:40:14 2010 From: iiminar at gmail.com (Igor Minar) Date: Wed, 21 Jul 2010 09:40:14 -0700 Subject: Accept-Encoding and Vary header issue Message-ID: Hi there, I came across this interesting Varnish 2.1.2 behavior that I think is a bug. vcl: sub vcl_recv { call normalize_accept_encoding; ... } sub vcl_fetch { ... if (beresp.http.Vary ~ ".+") { set beresp.http.Vary = "Accept-Encoding"; #the app doesn't do this } ... } If I make an "uncompressed request" (curl -O -v http://foo.com/bar) I prime the cache with uncompressed variant of bar resource on subsequent uncompressed request, I get cache hit and the resource is returned. if I then (before TTL) make a "compressed request" (curl -O -v -H "Accept-Encoding:gzip" http://foo.com/bar) I'd expect to get cache miss and prime the cache with compressed variant of bar resource, but instead I get cache hit, and the uncompressed variant is returned. once TTL is achieved, if I make a compressed request, varnish correctly fetches the compressed variant of the resources, primes the cache with it and returns it. if I subsequently (before TTL) make an uncompressed request, varnish correctly fetches the uncompressed variant from the backend and caches it. everything is good until the TTL for the compressed variant is reached, from that moment on varnish starts serving uncompressed variant for both compressed and uncompressed requests, until the TTL for the uncompressed variant is reached. My current workaround is to control variants manually via vcl_hash: sub vcl_hash { # consider different Accept-Encoding requests (normalized) as new variants set req.hash += req.http.Accept-Encoding; } but I don't think that this should be necessary. Am I doing something wrong or is this a bug? thanks, Igor From iiminar at gmail.com Thu Jul 22 01:03:21 2010 From: iiminar at gmail.com (Igor Minar) Date: Wed, 21 Jul 2010 18:03:21 -0700 Subject: Varnish is serving an incomplete response In-Reply-To: References: Message-ID: I'm experiencing something similar. Varnish (2.1.2 on OpenSolaris) occasionally passes incomplete requests to the backend or delivers incomplete responses to the client. That said, I don't see any "Write error" in the logs. Even inspecting log for a request that I know was truncated doesn't show anything unusual. I'm not able to reproduce the issue at will, but on two occasions (days apart) when varnish was in front of my app users reported that their POSTs were truncated or that they received an incomplete responses from the server. Once I removed varnish from the stack, the problem never reappeared. I wonder if this has anything to do with the EBADF issue discussed here: http://www.mail-archive.com/varnish-misc at projects.linpro.no/msg03626.html I'm compiling varnish with -mt, but I still see errno=9 (EBADF) in TCP_Assert called from TCP_blocking in tcp.c. This might not be related, but I just wanted to mention it in case it is significant. /i On Wed, Mar 3, 2010 at 5:15 PM, Bayron Guevara wrote: > Hello! > > I'm using Varnish 2.0.5 running on the following server's specification: > > ?2 Quadcore Intel Xeon 2.00Ghz 64bits > ?OS: RHEL 5 (64 bits) > ?8MB RAM > ?1GB Ethernet > > I've configured my network infraestructure with a Load Balancer, a Varnish > dedicated server and five web servers plus database servers. We have the > following network configuration: > external client ---> Load Balancer (public VIP) ---> Varnish Proxy --> Load > Balancer (private VIP) --> Web Servers > > In this configuration, the Load Balancer have the responsability for send > the request to the respective server according to the domain. The Varnish > server have configurated the Load Balancer's private VIP as unique backend. > > Now, let me explain the issue. On a low traffic scenario, the websites are > served correctly, but sometimes the page get blank or partially loaded. In > both cases a 200 OK response code is received and also the response body, > however it is received incomplete. Then I proceed to check the varnishstat > and varnishlog output, and I have some observations: The varnish frecuently > restarted and at execute? varnishlog -i Debug -I I got the following output: > 400 Debug??????? c "Write error, len = 34500/55022, errno = Success" > > I don't know what it means exactly, but some google seach give me a clue: > maybe be caused by an interruption during client communication. So, this > error could show the cause of the problem. Although I don't know why the > cause of this error, I guess a network buffer overflow, so I show you some > OS related values: > > /proc/sys/net/ipv4/ip_local_port_range = 32768?? 61000 > /proc/sys/net/core/rmem_max = 131071 > /proc/sys/net/core/wmem_max = 131071 > /proc/sys/net/ipv4/tcp_mem = 196608? 262144? 393216 > /proc/sys/net/ipv4/tcp_wmem = 4096??? 16384?? 4194304 > /proc/sys/net/ipv4/tcp_fin_timeout = 60 > /proc/sys/net/core/netdev_max_backlog = 1000 > /proc/sys/net/core/somaxconn = 128 > /proc/sys/net/ipv4/tcp_syncookies = 1 > /proc/sys/net/ipv4/tcp_max_orphans = 65536 > /proc/sys/net/ipv4/tcp_max_syn_backlog = 1024 > /proc/sys/net/ipv4/tcp_synack_retries = 5 > /proc/sys/net/ipv4/tcp_syn_retries = 5 > > This same values can be found in this varnish performance article: > http://varnish-cache.org/wiki/Performance. The mine ones seems very low and > maybe it is one of the causes. With the average traffic (around 500 > concurrent users for all sites), the Varnish service not respond and the > server load raise up to 612. Respect to the web site response, a Connection > refused error (Code 503) is returned. In this ocassion I didn't can review > the varnish statistics. > > Here are my varnish params, maybe it can help: > 200 2224 > accept_fd_holdoff????????? 50 [ms] > acceptor?????????????????? default (epoll, poll) > auto_restart?????????????? on [bool] > backend_http11???????????? on [bool] > between_bytes_timeout????? 60.000000 [s] > cache_vbe_conns??????????? off [bool] > cc_command???????????????? "exec cc -fpic -shared -Wl,-x -o %o %s" > cli_buffer???????????????? 8192 [bytes] > cli_timeout??????????????? 5 [seconds] > client_http11????????????? off [bool] > clock_skew???????????????? 10 [s] > connect_timeout??????????? 0.400000 [s] > default_grace????????????? 10 > default_ttl??????????????? 180 [seconds] > diag_bitmap??????????????? 0x0 [bitmap] > err_ttl??????????????????? 0 [seconds] > esi_syntax???????????????? 0 [bitmap] > fetch_chunksize??????????? 128 [kilobytes] > first_byte_timeout???????? 60.000000 [s] > group????????????????????? varnish (103) > listen_address???????????? :80 > listen_depth?????????????? 1024 [connections] > log_hashstring???????????? off [bool] > log_local_address????????? off [bool] > lru_interval?????????????? 360 [seconds] > max_esi_includes?????????? 5 [includes] > max_restarts?????????????? 4 [restarts] > obj_workspace????????????? 8192 [bytes] > overflow_max?????????????? 100 [%] > ping_interval????????????? 3 [seconds] > pipe_timeout?????????????? 60 [seconds] > prefer_ipv6??????????????? off [bool] > purge_dups???????????????? on [bool] > purge_hash???????????????? on [bool] > rush_exponent????????????? 3 [requests per request] > send_timeout?????????????? 600 [seconds] > sess_timeout?????????????? 5 [seconds] > sess_workspace???????????? 65536 [bytes] > session_linger???????????? 100 [ms] > session_max??????????????? 100000 [sessions] > shm_reclen???????????????? 255 [bytes] > shm_workspace????????????? 8192 [bytes] > srcaddr_hash?????????????? 1049 [buckets] > srcaddr_ttl??????????????? 0 [seconds] > thread_pool_add_delay????? 2 [milliseconds] > thread_pool_add_threshold? 2 [requests] > thread_pool_fail_delay???? 200 [milliseconds] > thread_pool_max??????????? 5000 [threads] > thread_pool_min??????????? 150 [threads] > thread_pool_purge_delay??? 1000 [milliseconds] > thread_pool_stack????????? unlimited [bytes] > thread_pool_timeout??????? 120 [seconds] > thread_pools?????????????? 8 [pools] > user?????????????????????? varnish (100) > vcl_trace????????????????? off [bool] > > What are your suggestions? > Is this a Varnish or Operating System configuration problem? > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > > From martin.boer at netclever.nl Thu Jul 22 06:47:58 2010 From: martin.boer at netclever.nl (Martin Boer) Date: Thu, 22 Jul 2010 08:47:58 +0200 Subject: Redirecting based on http verb and URL In-Reply-To: <588A907C-5409-4F2E-902D-782AC374EFE0@glam.ac.uk> References: <588A907C-5409-4F2E-902D-782AC374EFE0@glam.ac.uk> Message-ID: <4C47E99E.3070104@netclever.nl> I'd use ~ instead of == in the first if statement. If doubt 'equal' will fit a regular expression where 'looks something like' will. And perhaps 'website' instead of 'webiste', but that may have come from you disguising the url. Hope either one of them helps. :) Martin On 07/21/2010 02:13 PM, Davies Matt J A (LCSS) wrote: > Hello everyone > > I'm just checking I'm going down the right path here. > > I've read the example redirect section on the docs here, http://varnish-cache.org/wiki/VCLExampleRedirectInVCL > > In my case we've just moved web sites, and some crazy dudes are spamming our site, trying to send a POST form to an one page that may have one time allowed it. > > My Rails app is picking up these attempts at spam and throwing them out, but I want to handle them, and other normal people who simply click on the old link somewhere, with varnish prior to that. > > Here's my addition to the vcl_recv > > if (req.http.host == "^(www.)?webiste/blogs/oldpage$") { > error 750 "Moved Temporarily"; > if (req.request == "POST"){ > error 751 "Please go away" > } > } > > and vcl_error > > sub vcl_error { > if (obj.status == 750) { > set obj.http.Location = "http://newblogs"; > set obj.status = 302; > deliver; > } > if (obj.status == 751) { > set obj.status = 500; > deliver; > } > } > > Can anyone see any glaringly obvious stupid bits in there? > > Thanks again. > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > From fancy3g at gmail.com Thu Jul 22 09:05:13 2010 From: fancy3g at gmail.com (3g fancy) Date: Thu, 22 Jul 2010 17:05:13 +0800 Subject: does anyone successfully used persistent storage in varnish? Message-ID: Hi All, Sorry for such a general question. But seems it does not work just simply use the "-s persistent , " when start. Is there something i am missing? Does anyone successfully used persistent storage in varnish? Thanks, Deric -------------- next part -------------- An HTML attachment was scrubbed... URL: From marc.fournier at camptocamp.com Thu Jul 22 09:30:08 2010 From: marc.fournier at camptocamp.com (Marc Fournier) Date: Thu, 22 Jul 2010 11:30:08 +0200 Subject: does anyone successfully used persistent storage in varnish? References: Message-ID: <20100722113008.2a08f65d@lonquimay.wrk.lsn.camptocamp.com> Hello, > Sorry for such a general question. But seems it does not work just > simply use the "-s persistent , " when start. > Is there something i am missing? > Does anyone successfully used persistent storage in varnish? Glad to see I'm not the only one having trouble using this feature :-) There was this thread on the list a couple of weeks ago: http://lists.varnish-cache.org/pipermail/varnish-misc/2010-June/004307.html As I understand, it might work, depending on how and when the shutdown of varnishd happens. But I have no clue how to stop it in a way which would allow a successful reload of the existing cache at the next start. Marc From piotr.teodorowski at contium.pl Thu Jul 22 10:20:03 2010 From: piotr.teodorowski at contium.pl (Piotr Teodorowski) Date: Thu, 22 Jul 2010 12:20:03 +0200 Subject: does anyone successfully used persistent storage in varnish? In-Reply-To: <20100722113008.2a08f65d@lonquimay.wrk.lsn.camptocamp.com> References: <20100722113008.2a08f65d@lonquimay.wrk.lsn.camptocamp.com> Message-ID: <201007221220.03787.piotr.teodorowski@contium.pl> On Thursday 22 of July 2010 11:30:08 Marc Fournier wrote: > Hello, > > > Sorry for such a general question. But seems it does not work just > > simply use the "-s persistent , " when start. > > Is there something i am missing? > > Does anyone successfully used persistent storage in varnish? > > Glad to see I'm not the only one having trouble using this feature :-) > > There was this thread on the list a couple of weeks ago: > http://lists.varnish-cache.org/pipermail/varnish-misc/2010-June/004307.html > > As I understand, it might work, depending on how and when the shutdown > of varnishd happens. But I have no clue how to stop it in a way which > would allow a successful reload of the existing cache at the next > start. > > Marc Hi, varnishd -V varnishd (varnish-2.1.2 SVN ) (debian squeeze) DAEMON_OPTS="-a :8081 \ -T 127.0.0.1:6082 \ -f /etc/varnish/default.vcl \ -w 100,2000 \ -s persistent,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G" stop & start: varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 stop varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 start and it seems that it works. If I use /etc/init.d/varnish stop it doesn't work (it uses start-stop-daemon) Piotr From mjdavies at glam.ac.uk Thu Jul 22 13:31:47 2010 From: mjdavies at glam.ac.uk (Davies Matt J A (LCSS)) Date: Thu, 22 Jul 2010 13:31:47 +0000 Subject: Redirecting based on http verb and URL In-Reply-To: <4C47E99E.3070104@netclever.nl> References: <588A907C-5409-4F2E-902D-782AC374EFE0@glam.ac.uk> <4C47E99E.3070104@netclever.nl> Message-ID: Thanks Martin Jobs a good un :-) Matt On 22 Jul 2010, at 07:47, Martin Boer wrote: > I'd use ~ instead of == in the first if statement. If doubt 'equal' will fit a regular expression where 'looks something like' will. > > And perhaps 'website' instead of 'webiste', but that may have come from you disguising the url. > > Hope either one of them helps. :) > Martin > > > On 07/21/2010 02:13 PM, Davies Matt J A (LCSS) wrote: >> Hello everyone >> >> I'm just checking I'm going down the right path here. >> >> I've read the example redirect section on the docs here, http://varnish-cache.org/wiki/VCLExampleRedirectInVCL >> >> In my case we've just moved web sites, and some crazy dudes are spamming our site, trying to send a POST form to an one page that may have one time allowed it. >> >> My Rails app is picking up these attempts at spam and throwing them out, but I want to handle them, and other normal people who simply click on the old link somewhere, with varnish prior to that. >> >> Here's my addition to the vcl_recv >> >> if (req.http.host == "^(www.)?webiste/blogs/oldpage$") { >> error 750 "Moved Temporarily"; >> if (req.request == "POST"){ >> error 751 "Please go away" >> } >> } >> >> and vcl_error >> >> sub vcl_error { >> if (obj.status == 750) { >> set obj.http.Location = "http://newblogs"; >> set obj.status = 302; >> deliver; >> } >> if (obj.status == 751) { >> set obj.status = 500; >> deliver; >> } >> } >> >> Can anyone see any glaringly obvious stupid bits in there? >> >> Thanks again. >> >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From fancy3g at gmail.com Fri Jul 23 02:42:52 2010 From: fancy3g at gmail.com (3g fancy) Date: Fri, 23 Jul 2010 10:42:52 +0800 Subject: does anyone successfully used persistent storage in varnish? In-Reply-To: <201007221220.03787.piotr.teodorowski@contium.pl> References: <20100722113008.2a08f65d@lonquimay.wrk.lsn.camptocamp.com> <201007221220.03787.piotr.teodorowski@contium.pl> Message-ID: thanks Marc, thanks Piotr. so the persistent only works when restart gentally use varnishadm, right? i will check that. but we want to use varnish in our production environment, and we cache lots of objects. so what we need is if there's power outage or something happened and varnish need to restart, we don't want varnish to recache again and that will be big impact to backend. we are now using 100+ squid nodes and want to move to varnish since it might have better performance and we can reduce the server number. but if persistent storage does not work, we can not use it. 2010/7/22 Piotr Teodorowski > On Thursday 22 of July 2010 11:30:08 Marc Fournier wrote: > > Hello, > > > > > Sorry for such a general question. But seems it does not work just > > > simply use the "-s persistent , " when start. > > > Is there something i am missing? > > > Does anyone successfully used persistent storage in varnish? > > > > Glad to see I'm not the only one having trouble using this feature :-) > > > > There was this thread on the list a couple of weeks ago: > > > http://lists.varnish-cache.org/pipermail/varnish-misc/2010-June/004307.html > > > > As I understand, it might work, depending on how and when the shutdown > > of varnishd happens. But I have no clue how to stop it in a way which > > would allow a successful reload of the existing cache at the next > > start. > > > > Marc > Hi, > > varnishd -V > varnishd (varnish-2.1.2 SVN ) (debian squeeze) > > DAEMON_OPTS="-a :8081 \ > -T 127.0.0.1:6082 \ > -f /etc/varnish/default.vcl \ > -w 100,2000 \ > -s > persistent,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G" > > stop & start: > varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 stop > varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 start > > and it seems that it works. > > If I use /etc/init.d/varnish stop it doesn't work (it uses > start-stop-daemon) > > Piotr > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From l at lrowe.co.uk Sat Jul 24 20:46:21 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Sat, 24 Jul 2010 21:46:21 +0100 Subject: Accept-Encoding and Vary header issue In-Reply-To: References: Message-ID: Use varnishlog to see what is going on here. My bet would be on the backend not setting any Vary at all, so ``set beresp.http.Vary = "Accept-Encoding";`` never gets called. What happens when you make that call unconditionally? Laurence On 21 July 2010 17:40, Igor Minar wrote: > Hi there, > > I came across this interesting Varnish 2.1.2 behavior that I think is a bug. > > vcl: > > sub vcl_recv { > ?call normalize_accept_encoding; > ?... > } > > > sub vcl_fetch { > ?... > ?if (beresp.http.Vary ~ ".+") { > ? ?set beresp.http.Vary = "Accept-Encoding"; #the app doesn't do this > ?} > ?... > } > > > If I make an "uncompressed request" (curl -O -v http://foo.com/bar) > I prime the cache with uncompressed variant of bar resource > > on subsequent uncompressed request, I get cache hit and the resource > is returned. > > if I then (before TTL) make a "compressed request" (curl -O -v -H > "Accept-Encoding:gzip" ?http://foo.com/bar) > I'd expect to get cache miss and prime the cache with compressed > variant of bar resource, > but instead I get cache hit, and the uncompressed variant is returned. > > once TTL is achieved, if I make a compressed request, varnish > correctly fetches the compressed variant of the resources, primes the > cache with it and returns it. > > if I subsequently (before TTL) make an uncompressed request, varnish > correctly fetches the uncompressed variant from the backend and caches > it. > > everything is good until the TTL for the compressed variant is > reached, from that moment on varnish starts serving uncompressed > variant for both compressed and uncompressed requests, until the TTL > for the uncompressed variant is reached. > > My current workaround is to control variants manually via vcl_hash: > > sub vcl_hash { > ?# consider different Accept-Encoding requests (normalized) as new variants > ?set req.hash += req.http.Accept-Encoding; > } > > but I don't think that this should be necessary. Am I doing something > wrong or is this a bug? > > thanks, > Igor > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From l at lrowe.co.uk Sat Jul 24 20:51:25 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Sat, 24 Jul 2010 21:51:25 +0100 Subject: vhost configuration In-Reply-To: <1279502728.16793.mail71x0@mail71> References: <1279502728.16793.mail71x0@mail71> Message-ID: > ---------------- error messages --------------------- > > Expected one of > ?'acl', 'sub', 'backend',? or 'director' > Found: 'if' at > (input Line 16 Pos 1) > if (req.http.host ~ "^(www.)?aaa.com$") { > ##----------------------------------------- > Running VCC-compiler failed, exit 1 > VCL compilation failed > > -------------------------------------------------------- You need to put this in vcl_recv: > ------------ VCL Configuration ---------------------- > > backend?aaa { > ???? .host = "1.1.1.1"; > ???? .port = "80"; > ?} > backend bbb { > ???? .host = "2.2.2.2."; > ???? .port = "80"; > ?} sub vcl_recv { > if (req.http.host ~ "^(www.)?aaa.com$") { > ?? set req.http.host = "www.aaa.com"; > ?? set req.backend = aaar; > > } else if (req.http.host ~ "^(www.)?bbb.com$" { > ?? set req.http.host = "www.bbb.cpm"; > ?? set? req.backend = bbb;} > ? else { > ???????? req.backend = default.; > ?????? } > } } Laurence From iiminar at gmail.com Sun Jul 25 04:35:31 2010 From: iiminar at gmail.com (Igor Minar) Date: Sat, 24 Jul 2010 21:35:31 -0700 Subject: Accept-Encoding and Vary header issue In-Reply-To: References: Message-ID: In an attempt to remove all the noise, I left the "if" in my snippet and made things more confusing. My app _always_ sends: "Vary: User-Agent", when the correct header should be "Vary: Accept-Encoding,User-Agent". However, since setting User-Agent as Vary is a terrible terrible thing, I normalize the user agent and then handle the variations via vcl_hash. I rewrite the Vary header with "Accept-Encoding" and would like to let Varnish do the right thing, but that doesn't work. For now I just moved to using vcl_hash for handling all the variations and I remove Vary header in vcl_fetch. That works well. /i On Sat, Jul 24, 2010 at 1:46 PM, Laurence Rowe wrote: > Use varnishlog to see what is going on here. My bet would be on the > backend not setting any Vary at all, so ``set beresp.http.Vary = > "Accept-Encoding";`` never gets called. What happens when you make > that call unconditionally? > > Laurence > > On 21 July 2010 17:40, Igor Minar wrote: >> Hi there, >> >> I came across this interesting Varnish 2.1.2 behavior that I think is a bug. >> >> vcl: >> >> sub vcl_recv { >> ?call normalize_accept_encoding; >> ?... >> } >> >> >> sub vcl_fetch { >> ?... >> ?if (beresp.http.Vary ~ ".+") { >> ? ?set beresp.http.Vary = "Accept-Encoding"; #the app doesn't do this >> ?} >> ?... >> } >> >> >> If I make an "uncompressed request" (curl -O -v http://foo.com/bar) >> I prime the cache with uncompressed variant of bar resource >> >> on subsequent uncompressed request, I get cache hit and the resource >> is returned. >> >> if I then (before TTL) make a "compressed request" (curl -O -v -H >> "Accept-Encoding:gzip" ?http://foo.com/bar) >> I'd expect to get cache miss and prime the cache with compressed >> variant of bar resource, >> but instead I get cache hit, and the uncompressed variant is returned. >> >> once TTL is achieved, if I make a compressed request, varnish >> correctly fetches the compressed variant of the resources, primes the >> cache with it and returns it. >> >> if I subsequently (before TTL) make an uncompressed request, varnish >> correctly fetches the uncompressed variant from the backend and caches >> it. >> >> everything is good until the TTL for the compressed variant is >> reached, from that moment on varnish starts serving uncompressed >> variant for both compressed and uncompressed requests, until the TTL >> for the uncompressed variant is reached. >> >> My current workaround is to control variants manually via vcl_hash: >> >> sub vcl_hash { >> ?# consider different Accept-Encoding requests (normalized) as new variants >> ?set req.hash += req.http.Accept-Encoding; >> } >> >> but I don't think that this should be necessary. Am I doing something >> wrong or is this a bug? >> >> thanks, >> Igor >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > From sanelson at gmail.com Mon Jul 26 06:40:26 2010 From: sanelson at gmail.com (Stephen Nelson-Smith) Date: Mon, 26 Jul 2010 07:40:26 +0100 Subject: Clever Redirect Message-ID: Hello, I might be trying to do this entirely the wrong way, but here's the plan: I have a large number of machines on a private VLAN, behind a firewall, and without publicly routeable IP addresses. I want to be able to hit a page (the same page) on each of them, from a known, remote location. I don't wish to set up 90 different IPs on the external firewall. What I thought I could do was something like this: * Have varnish listening on a machine with an external IP address. * Hit http://varnish-server.com/$servername * Have varnish go to http://$servername/mypage.xml and return it to the initial requestor I have this working with a nasty cgi script, but it seems to me as if this could be done via Varnish. Am I right? If so, how would I go about it? Thanks, S. -- Stephen Nelson-Smith Technical Director Atalanta Systems Ltd www.atalanta-systems.com From pprocacci at datapipe.com Mon Jul 26 07:38:37 2010 From: pprocacci at datapipe.com (Paul A. Procacci) Date: Mon, 26 Jul 2010 02:38:37 -0500 Subject: Clever Redirect In-Reply-To: References: Message-ID: <20100726073837.GC20644@nat.myhome> On Mon, Jul 26, 2010 at 02:40:26AM -0400, Stephen Nelson-Smith wrote: > Hello, > > I might be trying to do this entirely the wrong way, but here's the plan: > > I have a large number of machines on a private VLAN, behind a > firewall, and without publicly routeable IP addresses. I want to be > able to hit a page (the same page) on each of them, from a known, > remote location. I don't wish to set up 90 different IPs on the > external firewall. What I thought I could do was something like this: > > * Have varnish listening on a machine with an external IP address. > * Hit http://varnish-server.com/$servername > * Have varnish go to http://$servername/mypage.xml and return it to > the initial requestor This is entirely feasable. First you define your backends: backend virdomain1 { .host = "10.5.21.1";; .port = "80"; } backend virdomain1 { .host = "10.5.21.2"; .port = "80"; } ..... Then in vcl_recv, you look for what it is you exactly want to match against: if (req.url ~ "^/(www\.)?virtdomain1\.com/?$" ){ set req.http.host = "www.virtdomain1.com"; set req.backend = host1; set req.url = "/mypage.xml"; } elseif { #ETC } else { # Use default } ....... This is a lot of typing for a lot of hosts, but to my knowledge, there is no way to to do tihs dynamically. A potential example that would have been nice: if (req.url ~ "^/(www\.)?([^\/])\/?$") { set req.http.host = "www.$2"; set req.backend = $2; set req.url = "/mypage.xml"; } if(!defined(req.backend)){ # Perl'ism backend = default; } ....... With that said, maybe the "quick and dirty" way is possible and I'm not up to speed. > I have this working with a nasty cgi script, but it seems to me as if > this could be done via Varnish. Am I right? If so, how would I go > about it? To answer the question, yes it's possible with varnish. Cleanly however? I guess that depends on what you consider clean. > Thanks, > > S. ~Paul This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. From stewsnooze at gmail.com Mon Jul 26 08:32:35 2010 From: stewsnooze at gmail.com (Stewart Robinson) Date: Mon, 26 Jul 2010 09:32:35 +0100 Subject: Clever Redirect In-Reply-To: References: Message-ID: <2DB1AF2B-355A-4866-9048-467D08EDEFB3@gmail.com> Hi, You would probably have each backend defined backend a_particular_backend { .host = "74.200.73.185"; .port = "81"; } backend a_spectacular_backend { .host = "74.200.73.186"; .port = "81"; } Then you could probably do something in the vcl_recv function for instance sub vcl_recv { if ( req.url == "a_particular_backend") { set req.backend = a_particular_backend; set req.url = "/somepage.xml"; } else if (req.url == "a_spectacular_backend") { set req.backend = a_spectacular_backend; set req.url = "/somepage.xml"; } e.t.c. Will this work? Stew On 26 Jul 2010, at 07:40, Stephen Nelson-Smith wrote: > Hello, > > I might be trying to do this entirely the wrong way, but here's the plan: > > I have a large number of machines on a private VLAN, behind a > firewall, and without publicly routeable IP addresses. I want to be > able to hit a page (the same page) on each of them, from a known, > remote location. I don't wish to set up 90 different IPs on the > external firewall. What I thought I could do was something like this: > > * Have varnish listening on a machine with an external IP address. > * Hit http://varnish-server.com/$servername > * Have varnish go to http://$servername/mypage.xml and return it to > the initial requestor > > I have this working with a nasty cgi script, but it seems to me as if > this could be done via Varnish. Am I right? If so, how would I go > about it? > > Thanks, > > S. > > -- > Stephen Nelson-Smith > Technical Director > Atalanta Systems Ltd > www.atalanta-systems.com > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From checker at d6.com Mon Jul 26 21:02:37 2010 From: checker at d6.com (Chris Hecker) Date: Mon, 26 Jul 2010 14:02:37 -0700 Subject: wp-varnish and blah/comment-page-1/ Message-ID: <4C4DF7ED.1020507@d6.com> I'm using wp-varnish, and it seems to work well, but I think it needs to be modified to also purge page/comment-page-1/ and friends, since wp seems to redirect you to those pages when you comment on the page itself. Anybody make these mods yet? Chris From jerais at kernel.com.sa Tue Jul 27 09:54:19 2010 From: jerais at kernel.com.sa (Jerais Al-Jeraisy) Date: Tue, 27 Jul 2010 12:54:19 +0300 Subject: Mapping a domain to URI of drupal Message-ID: Hello, We have varnish-2.0.6 on CentOS 5.5 in front of Apache & drupal, and we tried to map the domain ww.xx.yy.zz to a drupal URI, like /websites/a/b/c, so any one can hit ww.xx.yy.zz directly insted of wiritng ww.xx.yy.zz/websites/a/b/c. it works fine for the first time, but the next time i hit the same URL, it directs me to the main page of drupal, not to the URI, and i should delete all my temp files to get it to work again. here's the config file: backend default { .host = "127.0.0.1"; .port = "8080"; } sub vcl_recv { if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { return (pipe); } if (req.request != "GET" && req.request != "HEAD") { return (pass); } if (req.http.Authorization || req.http.Cookie) { return (pass); } if (req.http.host ~ "^ww\.xx\.yy\.zz$") { set req.url = "/websites/a/b/c/"; } return (lookup); } sub vcl_pipe { return (pipe); } sub vcl_pass { return (pass); } sub vcl_hash { set req.hash += req.url; if (req.http.host) { set req.hash += req.http.host; } else { set req.hash += server.ip; } return (hash); } sub vcl_hit { if (!obj.cacheable) { return (pass); } return (deliver); } sub vcl_miss { return (fetch); } sub vcl_fetch { if (!obj.cacheable) { return (pass); } if (obj.http.Set-Cookie) { return (pass); } set obj.prefetch = -30s; return (deliver); } sub vcl_deliver { return (deliver); } sub vcl_discard { return (discard); } sub vcl_prefetch { return (fetch); } sub vcl_timeout { return (discard); } sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; synthetic {" "} obj.status " " obj.response {"

Error "} obj.status " " obj.response {"

"} obj.response {"

Guru Meditation:

XID: "} req.xid {"

Varnish cache server
"}; return (deliver); } //JJ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fla_torres at yahoo.com.br Tue Jul 27 10:53:58 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Tue, 27 Jul 2010 07:53:58 -0300 Subject: Mapping a domain to URI of drupal In-Reply-To: References: Message-ID: <4C4EBAC6.3040809@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Em 27/7/2010 06:54, Jerais Al-Jeraisy escreveu: > We have varnish-2.0.6 on CentOS 5.5 in front of Apache & drupal, and > we tried to map the domain ww.xx.yy.zz to a drupal URI, like > /websites/a/b/c, so any one can hit ww.xx.yy.zz directly insted of > wiritng ww.xx.yy.zz/websites/a/b/c. > it works fine for the first time, but the next time i hit the same > URL, it directs me to the main page of drupal, not to the URI, and i > should delete all my temp files to get it to work again. Jerais, I think rewrite should be useful, try this: if (req.http.host ~ "^(www\.)?your\.domain\.com$") { set req.url = regsub(req.url, "^", "/websites/a/b/c"); } regards, -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (MingW32) iEYEARECAAYFAkxOusYACgkQNRQApncg295jDwCghEoqNx5OK4wjt4g8axxini++ JGYAoMgax11t56NJFHY/frolqg97ZQUr =OwwC -----END PGP SIGNATURE----- From jerais at kernel.com.sa Tue Jul 27 11:17:56 2010 From: jerais at kernel.com.sa (Jerais Al-Jeraisy) Date: Tue, 27 Jul 2010 14:17:56 +0300 Subject: Mapping a domain to URI of drupal In-Reply-To: <4C4EBAC6.3040809@yahoo.com.br> References: <4C4EBAC6.3040809@yahoo.com.br> Message-ID: Thank you for reply. still the same issue. one thing, is there any difference of the place of the rule? because if i placed it directly after the declaration "sub vcl_recv {" it opens for me BUT after long loading and it doesn't open all the content of the page, without the CSS and images for example. and if i placed it just before the " return (lookup); }", the end of sub vcl_recv, it opens for once and next time it opens the default page without the URI, the same issue as before. appropriated your help. //JJ On Tue, Jul 27, 2010 at 13:53, Flavio Torres wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Em 27/7/2010 06:54, Jerais Al-Jeraisy escreveu: > > We have varnish-2.0.6 on CentOS 5.5 in front of Apache & drupal, and > > we tried to map the domain ww.xx.yy.zz to a drupal URI, like > > /websites/a/b/c, so any one can hit ww.xx.yy.zz directly insted of > > wiritng ww.xx.yy.zz/websites/a/b/c. > > it works fine for the first time, but the next time i hit the same > > URL, it directs me to the main page of drupal, not to the URI, and i > > should delete all my temp files to get it to work again. > > Jerais, > > I think rewrite should be useful, try this: > > if (req.http.host ~ "^(www\.)?your\.domain\.com$") { > set req.url = regsub(req.url, "^", "/websites/a/b/c"); > } > > > regards, > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.8 (MingW32) > > iEYEARECAAYFAkxOusYACgkQNRQApncg295jDwCghEoqNx5OK4wjt4g8axxini++ > JGYAoMgax11t56NJFHY/frolqg97ZQUr > =OwwC > -----END PGP SIGNATURE----- > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fla_torres at yahoo.com.br Tue Jul 27 15:13:04 2010 From: fla_torres at yahoo.com.br (Flavio Torres) Date: Tue, 27 Jul 2010 12:13:04 -0300 Subject: Mapping a domain to URI of drupal In-Reply-To: References: <4C4EBAC6.3040809@yahoo.com.br> Message-ID: <4C4EF780.7070002@yahoo.com.br> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/27/2010 08:17 AM, Jerais Al-Jeraisy wrote: > one thing, is there any difference of the place of the rule? > because if i placed it directly after the declaration "sub vcl_recv > {" it opens for me BUT after long loading and it doesn't open all > the content of the page, without the CSS and images for example. Yeah, in recv. Can you see, for these contents - with firebug for example - what is the URL/Message response (ex: 404) ? > and if i placed it just before the " return (lookup); }", the end of > sub vcl_recv, it opens for once and next time it opens the default > page without the URI, the same issue as before. > > appropriated your help. I don't have Drupal here, but there are some issues with cookie session, check this out: http://highervisibilitywebsites.com/example-varnish-vcl-drupal-pressflow-site hope this helps -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkxO930ACgkQNRQApncg297QFgCguAWCp/Nj+BTZ43OsSxfNhTCG JzIAoIpcT2ht4zBQQgfulP49LoPienXg =7Zty -----END PGP SIGNATURE----- From szlists at szarka.org Tue Jul 27 18:00:57 2010 From: szlists at szarka.org (Rob Szarka) Date: Tue, 27 Jul 2010 14:00:57 -0400 Subject: Varnish Newbie Logging Question Message-ID: <201007271801.o6RI13sq089604@hustle.szarka.net> Hi, folks! I'm new to Varnish, but already loving it. :) Having decided to deploy it for real, I need to get logging set up properly. My aim is to get the data from my Varnish servers back to my stats server in Apache common log format so I can crunch the data there. 1. Does anyone have example scripts to share? 2. I am assuming that taking periodic dumps of new entries from the memory log just isn't feasible, and thus running either varnishlog or varnishncsa as a daemon is the way to go, right? 3. If I'm going to write logs to disk on the varnish server before transferring them off to my stats server, will it make a significant difference whether I use varnishncsa directly vs. using varnishlog and then translating the logs with varnishncsa once they get to my stats server? 4. I assume that I need to start the varnishlog or varnishncsa daemon after I start varnishd. Is that why the -d switch is there? To catch any entries that sneak in between startup of the two processes? Thanks for your help, Rob From jdzstz at gmail.com Tue Jul 27 21:00:30 2010 From: jdzstz at gmail.com (=?ISO-8859-1?Q?Jorge_D=EDaz?=) Date: Tue, 27 Jul 2010 23:00:30 +0200 Subject: Varnish Newbie Logging Question In-Reply-To: <201007271801.o6RI13sq089604@hustle.szarka.net> References: <201007271801.o6RI13sq089604@hustle.szarka.net> Message-ID: Hello, We are using in production Varnish 2.1.2 in Solaris 10 machine. To dump logs we are using "varnishncsa" and "rotatelogs" program from our Apache Web server instalation: *varnishncsa -P ${HOME}/varnish/conf/varnishncsa.pid |${HOME}/srv_httpd/bin/rotatelogs -l ${HOME}/varnish/logs/acc_%Y%m%d_%H%M%S.log 3600* varnishncsa is started at same time that varnishd as Solaris service. We rotate logs once every hour (we have a lot of requests) outside we have a machine with Apache+Awstat with a crontab script that retrieves logs with SFTP. 2010/7/27 Rob Szarka > > Hi, folks! I'm new to Varnish, but already loving it. :) Having decided to > deploy it for real, I need to get logging set up properly. My aim is to get > the data from my Varnish servers back to my stats server in Apache common > log format so I can crunch the data there. > > 1. Does anyone have example scripts to share? > > 2. I am assuming that taking periodic dumps of new entries from the memory > log just isn't feasible, and thus running either varnishlog or varnishncsa > as a daemon is the way to go, right? > > 3. If I'm going to write logs to disk on the varnish server before > transferring them off to my stats server, will it make a significant > difference whether I use varnishncsa directly vs. using varnishlog and then > translating the logs with varnishncsa once they get to my stats server? > > 4. I assume that I need to start the varnishlog or varnishncsa daemon after > I start varnishd. Is that why the -d switch is there? To catch any entries > that sneak in between startup of the two processes? > > Thanks for your help, > Rob > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From checker at d6.com Wed Jul 28 06:02:50 2010 From: checker at d6.com (Chris Hecker) Date: Tue, 27 Jul 2010 23:02:50 -0700 Subject: how small can I make the thread stack size? Message-ID: <4C4FC80A.6020503@d6.com> I'm on CentOS 5.5 32-bit (I know, I know), and I'm wondering how small I can make my thread stacks and still be safe. The defaul was 10mb, but I've been running at 1mb for a few weeks and it's been fine, but it would be nice to get it down even farther. I haven't been hit really hard yet, though. I've seen people talking about 256kb, but then the docs are pretty sketchy on whether it's a good idea to change it at all, etc. Any advice? Thanks, Chris From cosimo at streppone.it Wed Jul 28 08:29:42 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Wed, 28 Jul 2010 10:29:42 +0200 Subject: how small can I make the thread stack size? In-Reply-To: <4C4FC80A.6020503@d6.com> References: <4C4FC80A.6020503@d6.com> Message-ID: On Wed, 28 Jul 2010 08:02:50 +0200, Chris Hecker wrote: > [stack size] I've seen people talking about 256kb, but then the docs are > pretty sketchy on whether it's a good idea to change it at all, > Any advice? We're running several varnish instances. All of them x86-64 though, so YMMV. For the busiest machines, stack size has been set to 256kb, by putting 'ulimit -s 256' in the init script. IIRC, we started doing this because we saw increases in the 'thread_limited' counter. We saw that varnishd couldn't create new threads, even if the hard limit (thread_pool_max) hadn't been reached yet. Check: http://lists.varnish-cache.org/pipermail/varnish-misc/2009-August/002977.html These particular instances are constantly running 7k threads, occasionally going to 8-9k. With 24 Gb of RAM. -- Cosimo From checker at d6.com Wed Jul 28 09:24:53 2010 From: checker at d6.com (Chris Hecker) Date: Wed, 28 Jul 2010 02:24:53 -0700 Subject: how small can I make the thread stack size? In-Reply-To: References: <4C4FC80A.6020503@d6.com> Message-ID: <4C4FF765.3070904@d6.com> Alright, I'll try 256kb and see how it goes. I'm doing it with a config param to varnishd, which unfortunately makes it still use 10mb stacks for the other 6 threads it's running. I wonder if I should use ulimit in the init script to avoid this and save 60mb... Thanks, Chris On 2010/07/28 01:29, Cosimo Streppone wrote: > On Wed, 28 Jul 2010 08:02:50 +0200, Chris Hecker wrote: > >> [stack size] I've seen people talking about 256kb, but then the docs >> are pretty sketchy on whether it's a good idea to change it at all, >> Any advice? > > We're running several varnish instances. > All of them x86-64 though, so YMMV. > > For the busiest machines, stack size has been set to 256kb, by putting > 'ulimit -s 256' in the init script. > > IIRC, we started doing this because we saw increases in the > 'thread_limited' > counter. > We saw that varnishd couldn't create new threads, > even if the hard limit (thread_pool_max) hadn't been reached yet. > Check: > > http://lists.varnish-cache.org/pipermail/varnish-misc/2009-August/002977.html > > > These particular instances are constantly running 7k threads, > occasionally going to 8-9k. With 24 Gb of RAM. > From kb+varnish at slide.com Wed Jul 28 13:39:54 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Wed, 28 Jul 2010 06:39:54 -0700 Subject: how small can I make the thread stack size? In-Reply-To: <4C4FF765.3070904@d6.com> References: <4C4FC80A.6020503@d6.com> <4C4FF765.3070904@d6.com> Message-ID: +1. 256k was as low as I could safely go on x86_64, but it's been stable. I used the ulimit hack for a bit, and it worked fine. The official patch only controls workers, but the patch below will extend the control to the backend threads, which also scale with load. If you aren't already building varnish from source, the ulimit trick is probably just fine. -- Ken --- bin/varnishd/cache_backend_poll.c.orig 2009-04-01 06:35:15.000000000 -0700 +++ bin/varnishd/cache_backend_poll.c 2009-07-27 13:23:55.000000000 -0700 @@ -406,6 +414,7 @@ { struct vbp_target *vt; struct vsb *vsb; + pthread_attr_t be_attr; ASSERT_CLI(); @@ -444,7 +453,15 @@ VTAILQ_INSERT_TAIL(&vbp_list, vt, list); - AZ(pthread_create(&vt->thread, NULL, vbp_wrk_poll_backend, vt)); + /* Set the stacksize for backend threads */ + pthread_attr_init(&be_attr); + if (params->wthread_stacksize) { + int pt_res = pthread_attr_setstacksize(&be_attr, params->wthread_stacksize); + VSL(SLT_Debug, 0, "Attempt to set backend thread stacksize to %u %s", + params->wthread_stacksize, pt_res ? "FAILED" : "succeeded"); + } + + AZ(pthread_create(&vt->thread, &be_attr, vbp_wrk_poll_backend, vt)); } void On Jul 28, 2010, at 2:24 AM, Chris Hecker wrote: > > Alright, I'll try 256kb and see how it goes. I'm doing it with a config param to varnishd, which unfortunately makes it still use 10mb stacks for the other 6 threads it's running. I wonder if I should use ulimit in the init script to avoid this and save 60mb... > > Thanks, > Chris > > > On 2010/07/28 01:29, Cosimo Streppone wrote: >> On Wed, 28 Jul 2010 08:02:50 +0200, Chris Hecker wrote: >> >>> [stack size] I've seen people talking about 256kb, but then the docs >>> are pretty sketchy on whether it's a good idea to change it at all, >>> Any advice? >> >> We're running several varnish instances. >> All of them x86-64 though, so YMMV. >> >> For the busiest machines, stack size has been set to 256kb, by putting >> 'ulimit -s 256' in the init script. >> >> IIRC, we started doing this because we saw increases in the >> 'thread_limited' >> counter. >> We saw that varnishd couldn't create new threads, >> even if the hard limit (thread_pool_max) hadn't been reached yet. >> Check: >> >> http://lists.varnish-cache.org/pipermail/varnish-misc/2009-August/002977.html >> >> >> These particular instances are constantly running 7k threads, >> occasionally going to 8-9k. With 24 Gb of RAM. >> From kb+varnish at slide.com Wed Jul 28 13:48:23 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Wed, 28 Jul 2010 06:48:23 -0700 Subject: health check path doesn't change after VCL reload (2.0.6) In-Reply-To: References: <15628.1267046287@critter.freebsd.dk> Message-ID: I also recently confirmed that the backend specs aren't reset with a VCL reload. We had a backend timeout that was too short, but we had to do a hard restart of all instances to pick up the change. Not the end of the world; worst case this might belong on the wiki. The code to modify or respawn backend threads might be somewhat complicated, but it's probably worth it, IMHO. -- Ken On Feb 24, 2010, at 1:25 PM, John Norman wrote: > No, only the former / old path. > > I'm not super-troubled right now because a Varnish restart did pick up the new path (but at the cost of my cache) -- but I'm a bit worried about the next time I have to change it. > > I will be changing the probe interval soon, so that will give me a chance to reproduce the problem, if it even exists. > > As a bit of background: > > I automate the VCL update to multiple servers, when/if the VCL file has changed. > > Before the update, I also remove all of the inactive/old VCL's that are sitting there. > > Then I add the new one and "use" it. > > When I observed in my backend logs the probes going to the old URLs, I did check the "active" VCL on all systems, and they all showed the new path. > > In any case, I will try to reproduce and will send the results. > > One last thing: During the restart on one system, I observed the issue reported here: http://zarathustrashallspeak.com/2009/11/28/varnish-startup-issue/ > > John > > On Wed, Feb 24, 2010 at 4:18 PM, Poul-Henning Kamp wrote: > In message , John N > orman writes: > > >Still, the VCL indicated as "active" had a different path for the health > >check. > > Hopefully both got probed ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From scaunter at topscms.com Wed Jul 28 19:28:15 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Wed, 28 Jul 2010 15:28:15 -0400 Subject: health check path doesn't change after VCL reload (2.0.6) In-Reply-To: References: <15628.1267046287@critter.freebsd.dk> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C13DAED@TMG-EVS02.torstar.net> Confirmed. Changing director load balance logic, backend membership, and backend health checks requires a restart. Makes for exciting times if you need to change .probe parameters on the fly when varnish thinks all your backends are sick. Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ken Brownfield Sent: July-28-10 9:48 AM To: John Norman Cc: Varnish misc Subject: Re: health check path doesn't change after VCL reload (2.0.6) I also recently confirmed that the backend specs aren't reset with a VCL reload. We had a backend timeout that was too short, but we had to do a hard restart of all instances to pick up the change. Not the end of the world; worst case this might belong on the wiki. The code to modify or respawn backend threads might be somewhat complicated, but it's probably worth it, IMHO. -- Ken On Feb 24, 2010, at 1:25 PM, John Norman wrote: No, only the former / old path. I'm not super-troubled right now because a Varnish restart did pick up the new path (but at the cost of my cache) -- but I'm a bit worried about the next time I have to change it. I will be changing the probe interval soon, so that will give me a chance to reproduce the problem, if it even exists. As a bit of background: I automate the VCL update to multiple servers, when/if the VCL file has changed. Before the update, I also remove all of the inactive/old VCL's that are sitting there. Then I add the new one and "use" it. When I observed in my backend logs the probes going to the old URLs, I did check the "active" VCL on all systems, and they all showed the new path. In any case, I will try to reproduce and will send the results. One last thing: During the restart on one system, I observed the issue reported here: http://zarathustrashallspeak.com/2009/11/28/varnish-startup-issue/ John On Wed, Feb 24, 2010 at 4:18 PM, Poul-Henning Kamp wrote: In message , John N orman writes: >Still, the VCL indicated as "active" had a different path for the health >check. Hopefully both got probed ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. -------------- next part -------------- An HTML attachment was scrubbed... URL: From checker at d6.com Thu Jul 29 10:59:01 2010 From: checker at d6.com (Chris Hecker) Date: Thu, 29 Jul 2010 03:59:01 -0700 Subject: "Total body bytes" not actually bytes delivered? Message-ID: <4C515EF5.70904@d6.com> Today I had somebody repeatedly requesting a big file, but always failing pretty fast, I'm assuming due to my version of varnish not supporting the Range: header. But, what surprised me was how fast the Total body bytes was building up. It was a 14mb file, and it was being requested a couple times a second, but bailing at < 200kb, but by the end of the day TBB was 300gb. Is that by design, or should it be the number of bytes successfully delivered, which seems more useful? Lots of these in the logs: - 13 Debug c "Write error, len = 61920/14628065, errno = Success" An example of a varnishstat moment when the file was being requested: - 166499573986 14617409.58 2954687.12 Total body bytes Chris From scaunter at topscms.com Thu Jul 29 17:40:20 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Thu, 29 Jul 2010 13:40:20 -0400 Subject: Use varnish to set one week expires header Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C13DBB2@TMG-EVS02.torstar.net> I have varnish sending cache-control and Expires headers in vcl_fetch. Does anyone know of inline C to take the Last-Modified header and add a week to it? The Expires date is currently hard-coded in the config which is not ideal. Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From hc at codecompany.dk Fri Jul 30 14:43:10 2010 From: hc at codecompany.dk (HC Saustrup) Date: Fri, 30 Jul 2010 16:43:10 +0200 Subject: Sporadic "backend write error: 11" errors Message-ID: <4C52E4FE.3090001@codecompany.dk> Hi everyone! I've recently deployed Varnish 2.1.3 on a 64-bit CentOS 5.5, but it seems to fail on certain requests under stress. It's a POST request that's PASSed through to the backend. The backend is rather bulky Windows CMS, but it usually answers these requests quite fast. According to the logs, it fails while talking to the backend and returns a 503 error to the client after hanging for approximately 5000ms. The Windows backend has a connection timeout of 120s, and Varnish is configured with a timeout of 60s (connect_timeout, first_byte_timeout and between_bytes_timeout) so that's a bit weird. This happens in approximately 5-10% of these requests, which doesn't point to bad mojo in my VCL files - but I could be wrong. Let me know if these would be helpful. Here's a typical failed request: 46 Debug c "herding" 46 SessionClose c timeout 46 StatSess c x.x.x.x 16838 7 1 3 0 2 2 833 52886 46 SessionOpen c x.x.x.x 20290 :80 46 Debug c "herding" 46 ReqStart c x.x.x.x 20290 2127593665 46 RxRequest c POST 46 RxURL c /webservices/RelatedProducts.asmx/GetRelatedProducts 46 RxProtocol c HTTP/1.1 46 RxHeader c Accept: */* 46 RxHeader c Accept-Language: da 46 RxHeader c Referer: http://xxx 46 RxHeader c Content-Type: application/json; charset=utf-8 46 RxHeader c UA-CPU: x86 46 RxHeader c Accept-Encoding: gzip, deflate 46 RxHeader c User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618) 46 RxHeader c Host: xxx 46 RxHeader c Content-Length: 106 46 RxHeader c Connection: Keep-Alive 46 RxHeader c Cache-Control: no-cache 46 RxHeader c Cookie: xxx 46 VCL_call c recv 46 VCL_acl c MATCH allowedClients 0.0.0.0/0 46 VCL_return c pass 46 VCL_call c hash 46 VCL_return c hash 46 VCL_call c pass 46 VCL_return c pass 46 Backend c 114 default default 46 FetchError c backend write error: 11 <-- oops! 46 VCL_call c error 46 VCL_return c deliver 46 VCL_call c deliver 46 VCL_return c deliver 46 TxProtocol c HTTP/1.1 46 TxStatus c 503 46 TxResponse c Service Unavailable 46 TxHeader c Server: Varnish 46 TxHeader c Retry-After: 0 46 TxHeader c Content-Type: text/html; charset=utf-8 46 TxHeader c Content-Length: 1012 46 TxHeader c Date: Fri, 30 Jul 2010 16:22:01 GMT 46 TxHeader c Age: 5 46 TxHeader c Connection: close 46 TxHeader c X-Cache: MISS 46 Length c 1012 46 ReqEnd c 2127593665 1280506916.547359943 1280506921.548347950 0.000546932 5.000926018 0.000061989 Let me know if there's anything I can do to help resolve this. Cheers, HC -- Hans Christian Saustrup Partner, CodeCompany.dk, Skolebakken 7, 8000 ?rhus C Mobil +45-30261024 From hc at codecompany.dk Fri Jul 30 19:09:26 2010 From: hc at codecompany.dk (HC Saustrup) Date: Fri, 30 Jul 2010 21:09:26 +0200 Subject: Sporadic "backend write error: 11" errors In-Reply-To: <4C52E4FE.3090001@codecompany.dk> References: <4C52E4FE.3090001@codecompany.dk> Message-ID: <4C532366.6020308@codecompany.dk> On 7/30/10 4:43 PM, HC Saustrup wrote: > I've recently deployed Varnish 2.1.3 on a 64-bit CentOS 5.5, but it > seems to fail on certain requests under stress. It's a POST request > that's PASSed through to the backend. > > According to the logs, it fails while talking to the backend and > returns a 503 error to the client after hanging for approximately 5000ms. FYI - this also happens with SVN revision 5060 under CentOS 5.5. /HC From jerais at kernel.com.sa Sat Jul 31 09:23:22 2010 From: jerais at kernel.com.sa (Jerais Al-Jeraisy) Date: Sat, 31 Jul 2010 12:23:22 +0300 Subject: Mapping a domain to URI of drupal In-Reply-To: <4C4EF780.7070002@yahoo.com.br> References: <4C4EBAC6.3040809@yahoo.com.br> <4C4EF780.7070002@yahoo.com.br> Message-ID: Hello, just to update, we did a workaround out of Varnish and it works. i'm gonna test the 404 files with Firbug to see what's going on. Thank You, On Tue, Jul 27, 2010 at 18:13, Flavio Torres wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 07/27/2010 08:17 AM, Jerais Al-Jeraisy wrote: > > one thing, is there any difference of the place of the rule? > > because if i placed it directly after the declaration "sub vcl_recv > > {" it opens for me BUT after long loading and it doesn't open all > > the content of the page, without the CSS and images for example. > > Yeah, in recv. > > Can you see, for these contents - with firebug for example - what is > the URL/Message response (ex: 404) ? > > > > and if i placed it just before the " return (lookup); }", the end of > > sub vcl_recv, it opens for once and next time it opens the default > > page without the URI, the same issue as before. > > > > appropriated your help. > > I don't have Drupal here, but there are some issues with cookie > session, check this out: > > http://highervisibilitywebsites.com/example-varnish-vcl-drupal-pressflow-site > > > hope this helps > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkxO930ACgkQNRQApncg297QFgCguAWCp/Nj+BTZ43OsSxfNhTCG > JzIAoIpcT2ht4zBQQgfulP49LoPienXg > =7Zty > -----END PGP SIGNATURE----- > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bai.xiaoyu at gmail.com Thu Jul 1 08:06:01 2010 From: bai.xiaoyu at gmail.com (bai.xiaoyu at gmail.com) Date: Thu, 01 Jul 2010 08:06:01 -0000 Subject: varnish serves truncated files Message-ID: I have some files served through varnish. Users can only get part of the file. For example, a mp3 is 26mb, but the browser can only get the 600+kb (by streaming or direct downloading), except chrome, which get the file of correct size. When user get the file directly from apache, all browsers can get the file of correct size. In the http header, I see varnish reponse with the correct content-length, but i don't why the browsers only get part of the file. http://example.com/disney/mp3/FindingNemo.mp3 GET /disney/mp3/FindingNemo.mp3 HTTP/1.1 Host: cdn1.2u4u.com.cn User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 115 Connection: keep-alive HTTP/1.1 200 OK Server: Apache Cache-Control: public, no-transform Expires: Thu, 15 Apr 2012 20:00:00 GMT Content-Type: audio/mpeg Content-Length: 26886896 Date: Thu, 01 Jul 2010 07:16:13 GMT X-Varnish: 1641609202 Age: 0 Via: 1.1 varnish Connection: keep-alive ## This is my varnish config file in whole backend default { .host = "127.0.0.1"; .port = "8080"; .connect_timeout = 600s; .first_byte_timeout = 600s; .between_bytes_timeout = 600s; } sub vcl_recv { if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { lookup; } } sub vcl_fetch { if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") { remove obj.http.Set-Cookie; } } From slaweuk at googlemail.com Tue Jul 13 14:06:10 2010 From: slaweuk at googlemail.com (s u) Date: Tue, 13 Jul 2010 14:06:10 -0000 Subject: Exposing real backend name in vcl_recv Message-ID: Hi everyone, I'm trying to alter req.url to custom value based on backend that Varnish will use. Following config snippet worked fine before I've switched to director: --------------------- 8-< --------------------- #set req.backend = index3; # direct connection to backend set req.backend = search; # using rr director (for index3 and 4) if ( req.url ~ "^/solr/round-robin" ) { if ( req.backend == index3 ) { set req.url = regsub(req.url, "round-robin", "index3"); } if ( req.backend == index4 ) { set req.url = regsub(req.url, "round-robin", "index4"); } } --------------------- >-8 --------------------- Is there way to access the real backend's name - or something else - which would tell me to which backend a request will be sent during the recv phase? Kind Regards, Slawek -------------- next part -------------- An HTML attachment was scrubbed... URL: From flo at andersground.net Thu Jul 15 10:39:50 2010 From: flo at andersground.net (Florian Gilcher) Date: Thu, 15 Jul 2010 10:39:50 -0000 Subject: Perform Backend Request with a Body Message-ID: <9695D7DC-8BD9-4A79-83FF-3355DA92891E@andersground.net> Hi, This is kind of similar to the question "can I cache POSTs"? My situation is the following: We use ElasticSearch[1] for search indexing and would like to put a varnish in front of it, to cache recurring queries for some time (we do a lot of automated queries). Now, ElasticSearch has a rather 'creative' HTTP Api, mostly due to their complex Query system, wrapped into a JSON format[2]. The ElasticSearch backend expects this JSON as body of either a GET or a POST request. The first is non-compliant, because servers should ignore bodies on GET-requests, the second is non-compliant because POSTs are not intended for reading data *sigh*. Nevertheless, thats what I am stuck with. Now, back to varnish: it seem like varnish always strips the body before emitting a backend request if it is not in pass mode. Is there any way to circumvent this? I already send a special header containing a hash of the body to make it easier for varnish to lookup the object to this request. But without passing on the body, this is futile. Is there a solution using VCL for this or should I look for other solutions? Regards, Florian [1]: [2]: -- Florian Gilcher smtp: flo at andersground.net jabber: Skade at jabber.ccc.de gpg: 533148E2 From jerais at kernel.com.sa Sat Jul 24 10:12:19 2010 From: jerais at kernel.com.sa (Jerais Al-Jeraisy) Date: Sat, 24 Jul 2010 10:12:19 -0000 Subject: 503 error after idle session Message-ID: Hello, We started to implement varnish @ our infrastructure as a reverse proxy in front of Sharepoint, and it's doing good. but there's a strange problem, if i requested varnish for the first time it works fine, but after, say 2 mins, if i hit the page again or refresh it it takes too long to respond and throw 503 error. it works the the default configurations, changes only on backend host. server details: host: CentOS release 5.5 (Final) x64. varnish: varnish-2.0.6 here's varnishlog: 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963716 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963719 1.0 13 SessionOpen c 10.131.36.170 52969 :80 13 ReqStart c 10.131.36.170 52969 374027480 13 RxRequest c GET 13 RxURL c /default.aspx 13 RxProtocol c HTTP/1.1 13 RxHeader c Host: xx.xx.xx.xx 13 RxHeader c User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.15) Gecko/2009102704 Fedora/3.0.15-1.fc10 Firefox/3.0.15 13 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 13 RxHeader c Accept-Language: en-us,en;q=0.5 13 RxHeader c Accept-Encoding: gzip,deflate 13 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 13 RxHeader c Keep-Alive: 300 13 RxHeader c Connection: keep-alive 13 RxHeader c Cookie: SESS253f3165f282ca21e0aff96094b2f186=4om57skb5bthlv12hno77nefg3; SESS3c5368fdadd84553468b7aecbb6b8c56=m9okd8i89jeq777sn3c363cge7; ISAWPLB{30E7D663-ACFC-44A1-A8C4-C17514BA8234}={20C55D85-138B-4814-9748-CEA17E9D6759} 13 VCL_call c recv 13 VCL_return c pass 13 VCL_call c pass 13 VCL_return c pass 13 Backend c 22 default default 22 TxRequest - GET 22 TxURL - /default.aspx 22 TxProtocol - HTTP/1.1 22 TxHeader - Host: xx.xx.xx.xx 22 TxHeader - User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.15) Gecko/2009102704 Fedora/3.0.15-1.fc10 Firefox/3.0.15 22 TxHeader - Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 22 TxHeader - Accept-Language: en-us,en;q=0.5 22 TxHeader - Accept-Encoding: gzip,deflate 22 TxHeader - Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 22 TxHeader - Cookie: SESS253f3165f282ca21e0aff96094b2f186=4om57skb5bthlv12hno77nefg3; SESS3c5368fdadd84553468b7aecbb6b8c56=m9okd8i89jeq777sn3c363cge7; ISAWPLB{30E7D663-ACFC-44A1-A8C4-C17514BA8234}={20C55D85-138B-4814-9748-CEA17E9D6759} 22 TxHeader - X-Varnish: 374027480 22 TxHeader - X-Forwarded-For: 10.131.36.170 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963722 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963725 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963728 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963731 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963734 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963737 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963740 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963743 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963746 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963749 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963752 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963755 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963758 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963761 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963764 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963767 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963770 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963773 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963776 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963779 1.0 13 FetchError c http read error: 11 22 BackendClose - default 13 VCL_call c error 13 VCL_return c deliver 13 Length c 471 13 VCL_call c deliver 13 VCL_return c deliver 13 TxProtocol c HTTP/1.1 13 TxStatus c 503 13 TxResponse c Service Unavailable 13 TxHeader c Server: Varnish 13 TxHeader c Retry-After: 0 13 TxHeader c Content-Type: text/html; charset=utf-8 13 TxHeader c Content-Length: 471 13 TxHeader c Date: Sat, 24 Jul 2010 09:29:40 GMT 13 TxHeader c X-Varnish: 374027480 13 TxHeader c Age: 60 13 TxHeader c Via: 1.1 varnish 13 TxHeader c Connection: close 13 ReqEnd c 374027480 1279963720.108640909 1279963780.108922005 0.000058889 60.000254154 0.000026941 13 SessionClose c error 13 StatSess c 10.131.36.170 52969 60 1 1 0 1 0 235 471 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963782 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963785 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963788 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963791 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963794 1.0 varnishstats: 69 0.00 0.01 Client connections accepted 605 0.00 0.10 Client requests received 40 0.00 0.01 Backend conn. success 565 0.00 0.09 Backend conn. reuses 16 0.00 0.00 Backend conn. was closed 582 0.00 0.10 Backend conn. recycles 550 0.00 0.09 Fetch with Length 28 0.00 0.00 Fetch chunked 7 . . N struct sess_mem 1 . . N struct sess 0 . . N struct object 8 . . N struct objecthead 1 . . N struct smf 1 . . N large free smf 1 . . N struct vbe_conn 6 . . N struct bereq 10 . . N worker threads 10 0.00 0.00 N worker threads created 1 . . N backends 125 0.00 0.02 Objects sent with write 69 0.00 0.01 Total Sessions 605 0.00 0.10 Total Requests 605 0.00 0.10 Total pass 594 0.00 0.10 Total fetch -------------- next part -------------- An HTML attachment was scrubbed... URL: