From alessandro.ronchi at soasi.com Mon Mar 1 08:45:45 2010 From: alessandro.ronchi at soasi.com (Alessandro Ronchi) Date: Mon, 1 Mar 2010 09:45:45 +0100 Subject: Varnish and language cookie In-Reply-To: References: <744355347-1267175044-cardhu_decombobulator_blackberry.rim.net-272404610-@bda196.bisx.produk.on.blackberry> Message-ID: 2010/2/26 Yann Malet : >> The interesting parts are in vcl_recv : >> # This is the only important part of the cookie because The problem with your configuration is that I need to pass pages with some cookies (like sessionid) because it's mean the user can have a cart (and sure I want to let them go to the checkout). I need the same behaviour of your conf but If I have only the language cookie I need not to pass. I didn't solved this problem. This is my actual conf: http://dpaste.com/166300/ With that I'm able to cache pages without cookies, eliminate analytics cookies, cache pages without language cookie but NOT return cached pages if the users have the language cookie. If I unset the language cookie, the backend doesn't know what language it has to return. If I leave it, it's not cached. -- Alessandro Ronchi http://www.soasi.com SOASI - Sviluppo Software e Sistemi Open Source http://hobbygiochi.com Hobby & Giochi, l'e-commerce del divertimento From tfheen at varnish-software.com Mon Mar 1 09:32:34 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 01 Mar 2010 10:32:34 +0100 Subject: Weird situation with unexpected 206 from the backend, hit-for-pass and default_ttl In-Reply-To: <20100227155113.GA27913@pern.ankh.fr.EU.org> (Thomas Parmelan's message of "Sat, 27 Feb 2010 16:51:13 +0100") References: <20100227155113.GA27913@pern.ankh.fr.EU.org> Message-ID: <87ocj8xu8d.fsf@qurzaw.linpro.no> ]] Thomas Parmelan | - seeing as wrong as this backend answer is, I think it shouldn't be | dealt with "normally" by varnish, it would me more interesting to | let the next request have its chance to populate the cache with a | valid (200) answer, in order to keep the following requests being | able to be served from the cache. Is this something that should be | changed in varnish, or in the vcl (and how) ? We generally try to avoid guessing in the C code, so I think this should be left for VCL. | - the TTL used for the HitPass object seems to be the default_ttl (-t | 86400 in my setup); I tried to keep it low (0s or 10s) from the vcl | config, like this : | | sub vcl_fetch { | if (!obj.cacheable) { | set obj.ttl = 0s; | return (pass); | } | /* ... */ | } What happens if you set obj.cacheable to true and ttl to 1s if the response is a 206? Does that work? | - I also tried to purge the hit-for-pass culprit with the CLI (purge | req.host == the.host && req.url == /the/url), but it didn't work | (the object was still there after). Any idea how to purge such | objects ? Is this a bug ? According to phk on IRC, this should work and it not working is a bug. If you have a chance to reproduce this in a test environment with both 2.0.6 and trunk, that'd be appreciated. -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From tom+varnish at ankh.fr.eu.org Mon Mar 1 11:58:52 2010 From: tom+varnish at ankh.fr.eu.org (Thomas Parmelan) Date: Mon, 1 Mar 2010 12:58:52 +0100 Subject: Weird situation with unexpected 206 from the backend, hit-for-pass and default_ttl In-Reply-To: <87ocj8xu8d.fsf@qurzaw.linpro.no> References: <20100227155113.GA27913@pern.ankh.fr.EU.org> <87ocj8xu8d.fsf@qurzaw.linpro.no> Message-ID: <20100301115852.GA22311@pern.ankh.fr.EU.org> Le lundi 01 mars 2010 ? 10:32, d'apr?s Tollef Fog Heen : > We generally try to avoid guessing in the C code, so I think this should > be left for VCL. I tried the "unexpected 206" with -trunk (r4596), and it acts just as I wanted! So I guess this was already fixed in the C code. Or maybe just a side effect of some other fixes? > What happens if you set obj.cacheable to true and ttl to 1s if the > response is a 206? Does that work? I will give it a try with 2.0.6 and report the result. However, the VCL trick that didn't work for me is supposed to work according to the FAQ: http://varnish-cache.org/wiki/FAQ#WhyamIgettingacachehitbutarequestisstillgoingtomybackend Also, I found some more interestings things about the hit-for-pass TTL, be it in 2.0.6 or in -trunk, it appears from my various tests that it is computed as the maximum between default_ttl and the "RFC" TTL from the obj/beresp headers. My default_ttl was 86400... Is this a known feature? I didn't see any explanation about this on the Wiki. > | - I also tried to purge the hit-for-pass culprit with the CLI (purge > | req.host == the.host && req.url == /the/url), but it didn't work > | (the object was still there after). Any idea how to purge such > | objects ? Is this a bug ? > > According to phk on IRC, this should work and it not working is a bug. I'm confused about this one, as it definitely worked for me on a new test yesterday. I think we can put this one on human error on my part for the moment (and I will probably do some more tests just to be sure). Or maybe it as something to do with default_ttl (since I put it back to its 120s default value before doing the test that worked)? > If you have a chance to reproduce this in a test environment with both > 2.0.6 and trunk, that'd be appreciated. Yes, I now have a test environment where I can reproduce all this (that's how I noticed that -trunk behaves better). Will do the purge/default_ttl test again. If you want another specific test, just ask. -- Thomas Parmelan -+- Thomas.Parmelan at free.fr -+- tom at ankh.fr.EU.org From alessandro.ronchi at soasi.com Mon Mar 1 16:53:55 2010 From: alessandro.ronchi at soasi.com (Alessandro Ronchi) Date: Mon, 1 Mar 2010 17:53:55 +0100 Subject: purge cache for a domain Message-ID: How can I purge the varnish cache for a domain ? Is it possible to avoid varnish cache when debugging a website? -- Alessandro Ronchi http://www.soasi.com SOASI - Sviluppo Software e Sistemi Open Source http://hobbygiochi.com Hobby & Giochi, l'e-commerce del divertimento From perbu at varnish-software.com Mon Mar 1 18:11:20 2010 From: perbu at varnish-software.com (Per Buer) Date: Mon, 1 Mar 2010 19:11:20 +0100 Subject: purge cache for a domain In-Reply-To: References: Message-ID: <52220de01003011011q605352baqdf4f5e9741c7e39c@mail.gmail.com> On Mon, Mar 1, 2010 at 5:53 PM, Alessandro Ronchi < alessandro.ronchi at soasi.com> wrote: > How can I purge the varnish cache for a domain ? > in the cli: purge req.http.host ~ www.foo.com > Is it possible to avoid varnish cache when debugging a website? > Yes. You could either set an ACL and do "pass" for this ACL in vcl_recv or disable caching by passing all requests. -- Per Andreas Buer, CEO, Varnish Software AS Phone: +47 21 54 41 21 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From tom+varnish at ankh.fr.eu.org Mon Mar 1 19:13:05 2010 From: tom+varnish at ankh.fr.eu.org (Thomas Parmelan) Date: Mon, 1 Mar 2010 20:13:05 +0100 Subject: Weird situation with unexpected 206 from the backend, hit-for-pass and default_ttl In-Reply-To: <20100301115852.GA22311@pern.ankh.fr.EU.org> References: <20100227155113.GA27913@pern.ankh.fr.EU.org> <87ocj8xu8d.fsf@qurzaw.linpro.no> <20100301115852.GA22311@pern.ankh.fr.EU.org> Message-ID: <20100301191305.GA19176@pern.ankh.fr.EU.org> Le lundi 01 mars 2010 ? 12:58, d'apr?s Thomas Parmelan : > > What happens if you set obj.cacheable to true and ttl to 1s if the > > response is a 206? Does that work? > > I will give it a try with 2.0.6 and report the result. Nope, doesn't change a thing. > Will do the purge/default_ttl test again. Ok I found the explanation for this one: my purge command was "purge req.http.host == img.s-sfr.fr && req.url == /foo/bar", but in vcl_recv I change req.http.host to rewrite Host so that the backend recognize it. If I do the purge with the "corrected" req.http.host, the hit-for-pass is purged as expected. (this test was on 2.0.6, if it matters). -- Thomas Parmelan -+- Thomas.Parmelan at free.fr -+- tom at ankh.fr.EU.org From bayron.guevara at gmail.com Wed Mar 3 13:31:33 2010 From: bayron.guevara at gmail.com (Bayron Guevara) Date: Wed, 3 Mar 2010 07:31:33 -0600 Subject: Incomplete response body Message-ID: Hello! Recently, I've implemented Varnish in our websites http://www.laprensa.hn, http://www.elheraldo.hn and http://www.diez.hn. After a few navigation, I discovered that frecuently the pages get blank or their content is partially loaded. When I examined the Response Body, I found that effectively still when the pages display blank, only a few HTML content is received. Looking for possible problems, I proceed to examine the varnishlog -i Debug -I error output, I found this error message: 400 Debug c "Write error, len = 34500/55022, errno = Success" Which appear aprox. twice by second, google give me a clue that confirm the browser behaviour, someone said that this error is showed when a client have canceled a request but he haven't received all content, and this is what has been happening, the connection is closed, before the reception of all content. Next, here are my Varnish params: param.show 200 2224 accept_fd_holdoff 50 [ms] acceptor default (epoll, poll) auto_restart on [bool] backend_http11 on [bool] between_bytes_timeout 60.000000 [s] cache_vbe_conns off [bool] cc_command "exec cc -fpic -shared -Wl,-x -o %o %s" cli_buffer 8192 [bytes] cli_timeout 5 [seconds] client_http11 off [bool] clock_skew 10 [s] connect_timeout 0.400000 [s] default_grace 10 default_ttl 180 [seconds] diag_bitmap 0x0 [bitmap] err_ttl 0 [seconds] esi_syntax 0 [bitmap] fetch_chunksize 128 [kilobytes] first_byte_timeout 60.000000 [s] group varnish (103) listen_address :80 listen_depth 1024 [connections] log_hashstring off [bool] log_local_address off [bool] lru_interval 360 [seconds] max_esi_includes 5 [includes] max_restarts 4 [restarts] obj_workspace 8192 [bytes] overflow_max 100 [%] ping_interval 3 [seconds] pipe_timeout 60 [seconds] prefer_ipv6 off [bool] purge_dups on [bool] purge_hash on [bool] rush_exponent 3 [requests per request] send_timeout 600 [seconds] sess_timeout 5 [seconds] sess_workspace 65536 [bytes] session_linger 100 [ms] session_max 100000 [sessions] shm_reclen 255 [bytes] shm_workspace 8192 [bytes] srcaddr_hash 1049 [buckets] srcaddr_ttl 0 [seconds] thread_pool_add_delay 2 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_fail_delay 200 [milliseconds] thread_pool_max 5000 [threads] thread_pool_min 150 [threads] thread_pool_purge_delay 1000 [milliseconds] thread_pool_stack unlimited [bytes] thread_pool_timeout 120 [seconds] thread_pools 8 [pools] user varnish (100) vcl_trace off [bool] What could be the problem, maybe an incorrect configuration? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bayron.guevara at gmail.com Thu Mar 4 00:15:28 2010 From: bayron.guevara at gmail.com (Bayron Guevara) Date: Wed, 3 Mar 2010 18:15:28 -0600 Subject: Varnish is serving an incomplete response Message-ID: Hello! I'm using Varnish 2.0.5 running on the following server's specification: - 2 Quadcore Intel Xeon 2.00Ghz 64bits - OS: RHEL 5 (64 bits) - 8MB RAM - 1GB Ethernet I've configured my network infraestructure with a Load Balancer, a Varnish dedicated server and five web servers plus database servers. We have the following network configuration: external client ---> Load Balancer (public VIP) ---> Varnish Proxy --> Load Balancer (private VIP) --> Web Servers In this configuration, the Load Balancer have the responsability for send the request to the respective server according to the domain. The Varnish server have configurated the Load Balancer's private VIP as unique backend. Now, let me explain the issue. On a low traffic scenario, the websites are served correctly, but sometimes the page get blank or partially loaded. In both cases a 200 OK response code is received and also the response body, however it is received incomplete. Then I proceed to check the varnishstat and varnishlog output, and I have some observations: The varnish frecuently restarted and at execute *varnishlog -i Debug -I* I got the following output: 400 Debug c "Write error, len = 34500/55022, errno = Success" I don't know what it means exactly, but some google seach give me a clue: maybe be caused by an interruption during client communication. So, this error could show the cause of the problem. Although I don't know why the cause of this error, I guess a network buffer overflow, so I show you some OS related values: /proc/sys/net/ipv4/ip_local_port_range = 32768 61000 /proc/sys/net/core/rmem_max = 131071 /proc/sys/net/core/wmem_max = 131071 /proc/sys/net/ipv4/tcp_mem = 196608 262144 393216 /proc/sys/net/ipv4/tcp_wmem = 4096 16384 4194304 /proc/sys/net/ipv4/tcp_fin_timeout = 60 /proc/sys/net/core/netdev_max_backlog = 1000 /proc/sys/net/core/somaxconn = 128 /proc/sys/net/ipv4/tcp_syncookies = 1 /proc/sys/net/ipv4/tcp_max_orphans = 65536 /proc/sys/net/ipv4/tcp_max_syn_backlog = 1024 /proc/sys/net/ipv4/tcp_synack_retries = 5 /proc/sys/net/ipv4/tcp_syn_retries = 5 This same values can be found in this varnish performance article: http://varnish-cache.org/wiki/Performance. The mine ones seems very low and maybe it is one of the causes. With the average traffic (around 500 concurrent users for all sites), the Varnish service not respond and the server load raise up to 612. Respect to the web site response, a Connection refused error (Code 503) is returned. In this ocassion I didn't can review the varnish statistics. Here are my varnish params, maybe it can help: 200 2224 accept_fd_holdoff 50 [ms] acceptor default (epoll, poll) auto_restart on [bool] backend_http11 on [bool] between_bytes_timeout 60.000000 [s] cache_vbe_conns off [bool] cc_command "exec cc -fpic -shared -Wl,-x -o %o %s" cli_buffer 8192 [bytes] cli_timeout 5 [seconds] client_http11 off [bool] clock_skew 10 [s] connect_timeout 0.400000 [s] default_grace 10 default_ttl 180 [seconds] diag_bitmap 0x0 [bitmap] err_ttl 0 [seconds] esi_syntax 0 [bitmap] fetch_chunksize 128 [kilobytes] first_byte_timeout 60.000000 [s] group varnish (103) listen_address :80 listen_depth 1024 [connections] log_hashstring off [bool] log_local_address off [bool] lru_interval 360 [seconds] max_esi_includes 5 [includes] max_restarts 4 [restarts] obj_workspace 8192 [bytes] overflow_max 100 [%] ping_interval 3 [seconds] pipe_timeout 60 [seconds] prefer_ipv6 off [bool] purge_dups on [bool] purge_hash on [bool] rush_exponent 3 [requests per request] send_timeout 600 [seconds] sess_timeout 5 [seconds] sess_workspace 65536 [bytes] session_linger 100 [ms] session_max 100000 [sessions] shm_reclen 255 [bytes] shm_workspace 8192 [bytes] srcaddr_hash 1049 [buckets] srcaddr_ttl 0 [seconds] thread_pool_add_delay 2 [milliseconds] thread_pool_add_threshold 2 [requests] thread_pool_fail_delay 200 [milliseconds] thread_pool_max 5000 [threads] thread_pool_min 150 [threads] thread_pool_purge_delay 1000 [milliseconds] thread_pool_stack unlimited [bytes] thread_pool_timeout 120 [seconds] thread_pools 8 [pools] user varnish (100) vcl_trace off [bool] What are your suggestions? Is this a Varnish or Operating System configuration problem? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sime at sime.net.au Thu Mar 4 02:57:50 2010 From: sime at sime.net.au (Simon Males) Date: Thu, 4 Mar 2010 13:57:50 +1100 Subject: Still running out of workspaces ? (sess_workspace=551077) Message-ID: Hello all, Whenever my Varnish (2.0.6) servers experience a traffic spike after a couple days of uptime I see the following message in syslog: Mar 4 13:26:09 cache02 varnishd[28881]: Child (28882) Panic message: Missing errorhandling code in HSH_Prepare(), cache_hash.c line 199: Condition((p) != 0) not true.thread = (cache-worker) Backtrace: 0x41d63a: /usr/sbin/varnishd [0x41d63a] 0x417ed8: /usr/sbin/varnishd(HSH_Prepare+0xb8) [0x417ed8] 0x410eaa: /usr/sbin/varnishd [0x410eaa] 0x41286d: /usr/sbin/varnishd(CNT_Session+0x41d) [0x41286d] 0x427852: /usr/sbin/varnishd(ESI_Deliver+0x282) [0x427852] 0x41fe3f: /usr/sbin/varnishd(RES_WriteObj+0x25f) [0x41fe3f] 0x411a72: /usr/sbin/varnishd [0x411a72] 0x4127cd: /usr/sbin/varnishd(CNT_Session+0x37d) [0x4127cd] 0x41ec93: /usr/sbin/varnishd [0x41ec93] 0x41f402: /usr/sbin/varnishd [0x41f402] sp = 0x7fb225202008 { fd = 63, id = 63, xid = 0, client = 124.168.62.14:17605, step = STP_LOOKUP, handling = hash, restarts = 0, esis = 1 ws = 0x7fb225202080 { overflow id = "sess", {s,f,r,e} = {0x7fb225202820,+551046,(nil),+551077}, }, http[req] = { ws = 0x7fb225202080[sess] I don't really know how to interpret this, but I see references to "esi" and "session". ESI is used heavily and sess_workspace is 551077. Below I have the Varnish startup parameters. Should I just continue to increase the sess_workspace until it stops happening? I have also seen mentioned online that a backend could be potentially dying though I can't see to gather that from the above output. DAEMON_OPTS="-a :80 \ -T :6082 \ -f /etc/varnish/default.vcl \ -w 40,100,60 \ -p obj_workspace=4096 \ -p sess_workspace=551077 \ -p listen_depth=8192 \ -p log_hashstring=off \ -p lru_interval=60 \ -p sess_timeout=10 \ -p shm_workspace=32768 \ -p ping_interval=1 \ -p thread_pools=4 \ -p thread_pool_min=50 \ -p srcaddr_ttl=0 \ -p esi_syntax=0x00000003 \ -s malloc,2G" Thanks. -- Simon Males From perbu at varnish-software.com Thu Mar 4 08:44:59 2010 From: perbu at varnish-software.com (Per Buer) Date: Thu, 4 Mar 2010 09:44:59 +0100 Subject: Still running out of workspaces ? (sess_workspace=551077) In-Reply-To: References: Message-ID: <52220de01003040044m9db8b7dj717dcb30f5c2860@mail.gmail.com> On Thu, Mar 4, 2010 at 3:57 AM, Simon Males wrote: > I don't really know how to interpret this, but I see references to > "esi" and "session". ESI is used heavily and sess_workspace is 551077. > I know of people running with a session workspace of ~5M. And yes, this really needs to be an autotuning setting. > Below I have the Varnish startup parameters. Should I just continue to > increase the sess_workspace until it stops happening? Yes. -- Per Andreas Buer, CEO, Varnish Software AS Phone: +47 21 54 41 21 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From kontakt at web30.ch Thu Mar 4 10:27:27 2010 From: kontakt at web30.ch (Simon Kammerer) Date: Thu, 04 Mar 2010 11:27:27 +0100 Subject: Restart needed after new backend declared? Message-ID: <4B8F8B0F.6090202@web30.ch> Hi list, I declared a new backend and added a simple rule in vcl_recv like if(req.http.host ~ "xx.yyy.com"){ set req.backend = newbackend; } and then used Kristians script to reload VCL: ------------------------- #!/bin/bash # Reload a varnish config # Author: Kristian Lyngstol FILE="/etc/varnish/default.vcl" # Hostname and management port # (defined in /etc/default/varnish or on startup) HOSTPORT="localhost:6082" NOW=`date +%s` error() { echo 1>&2 "Failed to reload $FILE." exit 1 } varnishadm -T $HOSTPORT vcl.load reload$NOW $FILE || error varnishadm -T $HOSTPORT vcl.use reload$NOW || error echo Current configs: varnishadm -T $HOSTPORT vcl.list ------------------------- But the new backend is not used, I had to restart varnish to use the new configuration. Am I missing something? Mit freundlichem Gru? Simon Kammerer -- Web 3.0 AG Moskau 314B CH-8262 Ramsen Tel: +41 (0) 52 533 4725 Fax: +41 (0) 52 533 4625 simon.kammerer at web30.ch From wrighty+varnishmisc at gmail.com Thu Mar 4 11:53:10 2010 From: wrighty+varnishmisc at gmail.com (Paul Wright) Date: Thu, 4 Mar 2010 11:53:10 +0000 Subject: Child panics on OpenSolaris In-Reply-To: <282e72051002221002t1ac7be82q6ab8f21c60a21365@mail.gmail.com> References: <282e72051002170852g532cd8acqb570e6db4b82974c@mail.gmail.com> <6331.1266432386@critter.freebsd.dk> <282e72051002190945i63a94283s1718f54d79f0f2a7@mail.gmail.com> <282e72051002221002t1ac7be82q6ab8f21c60a21365@mail.gmail.com> Message-ID: <282e72051003040353t28bff803y42d3d5b824516c31@mail.gmail.com> On 22 February 2010 18:02, Paul Wright wrote: ... > For anyone else following along I've now had varnish running for over > 5 hours without issue, here are the things I found out: > > * add the Range unsetting code to ensure that such requests don't make > it through to the back end > * remove the TCP_Assert() that wraps the setsockopt() call on line 184 > of bin/varnishd/cache_acceptor.c > * compile with gcc, not Sun Studio (there's still some sort of > funniness with TCP_(non)blocking() ) > > CC=/usr/bin/gcc CFLAGS="-O3 -L/lib/amd64 -pthreads -m64 > -fomit-frame-pointer" LDFLAGS="-lumem -pthreads" ./configure > --prefix=/opt > > * pass the right flags through to gcc when launching vanishd > > newtask -p highfile /opt/sbin/varnishd -f /opt/etc/varnish/firebox.vcl -F \ > -p 'cc_command=/usr/bin/gcc -fpic -shared -m64 -o %o %s' \ > -T 127.0.0.1:9001 \ > -s malloc,2G \ > -p sess_timeout=5s \ > -p max_restarts=12 \ > -p waiter=poll \ > -p connect_timeout=0s \ > -p sess_workspace=65536 > > * keep checking http://letsgetdugg.com/2009/12/04/varnish-on-solaris/ > for hints and suggestions Latest update, we're seeing "Connection refused" panics like the following: Child (1955) died signal=6 Child (1955) Panic message: Assert error in TCP_blocking(), tcp.c line 164: Condition(TCP_Check(j)) not true. errno = 146 (Connection refused) thread = (cache-worker) ident = -smalloc,-hcritbit,poll Backtrace: 42fb51: /opt/sbin/varnishd'pan_ic+0xb1 [0x42fb51] 2f: [0x2f] sp = 13570018 { fd = 47, id = 47, xid = 0, client = ?.?.?.?:?, step = STP_FIRST, handling = deliver, restarts = 0, esis = 0 ws = 13570088 { id = "sess", {s,f,r,e} = {13570d90,13570d90,0,+65536}, }, http[req] = { ws = 13570088[sess] "", "/i/nl/all/0.gif", "HTTP/1.1", "Host: media.firebox.com", "User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko)", "Accept: */*", "Accept-Language: en-us", "If-Modified-Since: Tue, 22 Sep 2009 15:34:33 GMT", "If-None-Match: "16a8070-2b-4742c55c1f440"", "Connection: keep-alive", "X-Forwarded-For: 85.189.102.193", }, worker = fffffd7ff7bf1d80 { ws = fffffd7ff7bf1ec8 { id = "wrk", {s,f,r,e} = {fffffd7ff7bdfcb0,fffffd7ff7bdfcb0,0,+65536}, }, }, }, Interesting things to note, we're confident that this request is a cache hit which rules out the backend (handling = deliver). Also the client address appears to have been mangled: client = ?.?.?.?:?, Would this cause varnish to attempt opening a connection which is then refused? As a workaround would it be advisable to add a clause to TCP_Check (in include/libvarnish.h) to skip over errno 146 (Connection refused) along with the existing ECONNRESET and ENOTCONN clauses? Cheers, Paul. From rodrigo at mercadolibre.com Mon Mar 8 20:56:41 2010 From: rodrigo at mercadolibre.com (Rodrigo Benzaquen) Date: Mon, 8 Mar 2010 17:56:41 -0300 Subject: Survey; how do you use Varnish? In-Reply-To: <630234885.83667.1264776507774.JavaMail.root@claudius.linpro.no> References: <1972754207.83637.1264776184796.JavaMail.root@claudius.linpro.no> <630234885.83667.1264776507774.JavaMail.root@claudius.linpro.no> Message-ID: 1) How many servers do you have running Varnish? 30 2) What sort of total load are you having? Mbit/s or hits per second are preferred metrics. 2gbt per second 3) What sort of site is it? ecommerce *) Online media *) Cooperate website (ibm.com or similar) *) Retail *) Educational *) Social website 4) Do you use ESI? no 5) What features are you missing from Varnish. Max three features, prioritized. Please refer to http://varnish-cache.org/wiki/PostTwoShoppingList for features. On Fri, Jan 29, 2010 at 11:48 AM, Per Andreas Buer < per.buer at redpill-linpro.com> wrote: > Hi list. > > I'm working for Redpill Linpro, you might have heard of us - we're the main > sponsor of Varnish development. We're a bit curious about how Varnish is > used, what features are used and what is missing. What does a typical > installation look like? The information you would choose to reveal to me > would be aggregated and deleted and I promise you I won't use it for any > sales activities or harass you in any way. We will pubish the result on this > list if the feedback is significant. If you have the time and would like to > help us please take some time and answer the questions in a direct mail to > me. Thanks. > > 1) How many servers do you have running Varnish? > > 2) What sort of total load are you having? Mbit/s or hits per second are > preferred metrics. > > 3) What sort of site is it? > *) Online media > *) Cooperate website (ibm.com or similar) > *) Retail > *) Educational > *) Social website > > 4) Do you use ESI? > > 5) What features are you missing from Varnish. Max three features, > prioritized. Please refer to > http://varnish-cache.org/wiki/PostTwoShoppingList for features. > > > > -- > Per Andreas Buer > Redpill Linpro Group - Changing the Game > > Mobile +47 958 39 117 / Phone: +47 21 54 41 21 > _______________________________________________ > varnish-misc mailing list > varnish-misc at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From schickb at gmail.com Tue Mar 9 01:48:32 2010 From: schickb at gmail.com (Brad Schick) Date: Mon, 8 Mar 2010 17:48:32 -0800 Subject: Log slow responses Message-ID: Is there a way to get varnishlog to print requests that take over a certain amount of time to complete? My understanding is that ReqEnd contains: XID req_timestamp comp_timestamp idle_time proc_time xmit_time I'd like to print entries that have xmit_time over a certain amount. -Brad From rtshilston at gmail.com Tue Mar 9 09:00:04 2010 From: rtshilston at gmail.com (Rob S) Date: Tue, 09 Mar 2010 09:00:04 +0000 Subject: varnishncsa of backend requests (changeset 4480) Message-ID: <4B960E14.7060500@gmail.com> Hi, Since http://varnish-cache.org/changeset/4480, we've lost the ability to view backend requests in varnishncsa. Is there a recommended work around? Can varnishlog be coaxed into spitting out similar information, or is there another solution? Thanks, Rob From alessandro.ronchi at soasi.com Tue Mar 9 09:42:16 2010 From: alessandro.ronchi at soasi.com (Alessandro Ronchi) Date: Tue, 9 Mar 2010 10:42:16 +0100 Subject: UnicodeDecodeError: 'utf8' codec can't decode bytes in position 2-4: invalid data Message-ID: I am using django and I have a lot of this unicode error I didn't get before use varnish: http://dpaste.com/167139/ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ handler.addQuickElement(u"pubDate", rfc2822_date(item['pubdate']).decode('utf-8')) File "/usr/lib/python2.4/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode bytes in position 2-4: invalid data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is my varnish configuration: http://dpaste.com/169807/ It's strange because I made HTTP_ACCEPT_CHARSET header to only utf-8, and I have only unicode. as you can see here: http://hobbygiochi.com/xml/cat/da-vinci-games the page works, but not always. Any hint? Thanks in advance. -- Alessandro Ronchi http://www.soasi.com SOASI - Sviluppo Software e Sistemi Open Source http://hobbygiochi.com Hobby & Giochi, l'e-commerce del divertimento From kristian at varnish-software.com Tue Mar 9 11:05:25 2010 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Tue, 9 Mar 2010 12:05:25 +0100 Subject: Log slow responses In-Reply-To: References: Message-ID: <20100309110524.GA5353@kjeks.varnish-software.com> On Mon, Mar 08, 2010 at 05:48:32PM -0800, Brad Schick wrote: > Is there a way to get varnishlog to print requests that take over a > certain amount of time to complete? My understanding is that ReqEnd > contains: > XID req_timestamp comp_timestamp idle_time proc_time xmit_time > > I'd like to print entries that have xmit_time over a certain amount. You can do it with a regex: $ varnishlog -c -o ReqEnd '1\.[0-9]+$' This will catch requests that take 1.x seconds. varnishlog -c -o ReqEnd '[1-9]+\.[0-9]+$' Will catch requests taking 1 or more seconds, 11 seconds, 12 seconds etc. Finding one that takes 1+ (including 10) seconds is left as an exercise to the reader. - Kristian -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From harri.paivaniemi at tieto.com Tue Mar 9 11:53:07 2010 From: harri.paivaniemi at tieto.com (=?iso-8859-1?q?H=2EP=E4iv=E4niemi?=) Date: Tue, 9 Mar 2010 13:53:07 +0200 Subject: Change from Oracle WC to Varnish Message-ID: <201003091353.07106.harri.paivaniemi@tieto.com> Hi all, We are currently using Oracle Web cache on quite a big site. We have 18 cache servers and caching is done by reverse-proxy method. There is appr. 50000 http req/s on our cache servers together. System is running Linux. Because Oracle licenses are quite expensive, we would like to know if Varnish could replace OWC on our site. I would be _very_ pleased if any of you could gimme comments and thougts about this. My main consern is if Varnish is or is not able to work in this kind of situation where we are using: - multiple ip-based virtual hosts (about 50) - multiple origin server pools of course - origin server polling to see if they are ok - both common and site-specific cache-rules - different kind of expiration policies - cookie-based and header-based personalization to cache multiple versions of some sites - origin server capacity throttling / limiting etc - caching both GET, Get with query strings and POST - have to have some kind of method to order caches to expire (clear) specific content immediately when asked So how is it - can Varnish be used or are we gonna have major problems or limitations? Thank you, Harri from Finland From phk at phk.freebsd.dk Tue Mar 9 12:25:11 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 09 Mar 2010 12:25:11 +0000 Subject: Change from Oracle WC to Varnish In-Reply-To: Your message of "Tue, 09 Mar 2010 13:53:07 +0200." <201003091353.07106.harri.paivaniemi@tieto.com> Message-ID: <62840.1268137511@critter.freebsd.dk> In message <201003091353.07106.harri.paivaniemi at tieto.com>, =?iso-8859-1?q?H=2EP =E4iv=E4niemi?= writes: >- multiple ip-based virtual hosts (about 50) >- multiple origin server pools of course >- origin server polling to see if they are ok >- both common and site-specific cache-rules >- different kind of expiration policies So far no problems. >- cookie-based and header-based personalization to cache multiple versions of some sites Varnish allows you to define what goes into the hash-key, and supports "Vary:" headers, so I think is covered. Our VCL syntax for dealing with cookies needs to be improved, which will happen Real Soon Now. >- origin server capacity throttling / limiting etc Right now we only have a limit for maximum number of connections. >- caching both GET, Get with query strings and POST GET with or without query strings is supported. POST can be done, provided the message-body can be ignored. >- have to have some kind of method to order caches to expire (clear) > specific content immediately when asked Got that. >So how is it - can Varnish be used or are we gonna have major >problems or limitations? I certainly think Varnish can be used. Seriously consider doing a gradual phaseover, where you let Varnish take more and more load from the Oracle WebCache machines. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From janl at redpill-linpro.com Tue Mar 9 13:33:06 2010 From: janl at redpill-linpro.com (Nicolai Langfeldt) Date: Tue, 09 Mar 2010 14:33:06 +0100 Subject: Moving varnish-cache.org tomorrow Message-ID: <4B964E12.1010200@redpill-linpro.com> Hi, We'll be moving varnish-cache.org tomorrow, Wednesday 2010-03-10 at about 11:00 CET. The site will be unavailable for about 5 (or 30) minutes while we move it. We'll be getting a newer trac version but other than that there should be no user-noticeable changes. At the same time the mailing lists will change address to * varnish-announce at varnish-cache.org * varnish-bugs at varnish-cache.org * varnish-commit at varnish-cache.org * varnish-dev at varnish-cache.org * varnish-dist at varnish-cache.org * varnish-misc at varnish-cache.org * varnish-test at varnish-cache.org All subscriptions will be moved as is, and all user settings will be preserved. The old addresses will work for "quite a long time". The nntp gating to nntp.linpro.no will end at that time. If you have been reading the lists there and want to continue reading them please subscribe by email. The SVN repository will be off-line for the duration of the move to ensure that we copy a consistent repository. Regards, Nicolai Langfeldt on behalf of Varnish Software AS From schickb at gmail.com Wed Mar 10 03:10:51 2010 From: schickb at gmail.com (Brad Schick) Date: Tue, 9 Mar 2010 19:10:51 -0800 Subject: Number of cached objects Message-ID: Its not clear to which field is the number of cache entries. Perhaps shm_records? Thanks, -brad From phk at phk.freebsd.dk Wed Mar 10 05:30:05 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 10 Mar 2010 05:30:05 +0000 Subject: Number of cached objects In-Reply-To: Your message of "Tue, 09 Mar 2010 19:10:51 PST." Message-ID: <2826.1268199005@critter.freebsd.dk> In message , Brad Schick writes: >Its not clear to which field is the number of cache entries. Perhaps >shm_records? You want n_object -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From schickb at gmail.com Wed Mar 10 05:46:19 2010 From: schickb at gmail.com (Brad Schick) Date: Tue, 9 Mar 2010 21:46:19 -0800 Subject: Number of cached objects In-Reply-To: <2826.1268199005@critter.freebsd.dk> References: <2826.1268199005@critter.freebsd.dk> Message-ID: <60338EEF-CA31-4F1E-939A-594128E6409F@gmail.com> On Mar 9, 2010, at 9:30 PM, Poul-Henning Kamp wrote: > In message , Brad > Schick writes: >> Its not clear to which field is the number of cache entries. Perhaps >> shm_records? > > You want n_object Thanks. It would be nice if the output of "varnishstat -l" gave a bit more info. Currently the descriptions aren't really any more descriptive than the field names. I would not likely have guessed the "N struct object" is the number of cached objects. Also unclear what the difference is between sm, sma, sms, etc. -Brad From tfheen at varnish-software.com Wed Mar 10 07:30:04 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 10 Mar 2010 08:30:04 +0100 Subject: varnishncsa of backend requests (changeset 4480) In-Reply-To: <4B960E14.7060500@gmail.com> (Rob S.'s message of "Tue, 09 Mar 2010 09:00:04 +0000") References: <4B960E14.7060500@gmail.com> Message-ID: <87eijsk51f.fsf@qurzaw.linpro.no> ]] Rob S | Since http://varnish-cache.org/changeset/4480, we've lost the ability to | view backend requests in varnishncsa. Yes, it didn't work properly, so we disabled it until we can provide the information properly. | Is there a recommended work around? Can varnishlog be coaxed into | spitting out similar information, or is there another solution? If you need it and are ok with the limitations, just revert the patch locally? -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 From bschick at skytap.com Wed Mar 10 07:07:39 2010 From: bschick at skytap.com (Brad Schick) Date: Tue, 9 Mar 2010 23:07:39 -0800 Subject: varnishtop stdin Message-ID: It would be a nice feature if varnishtop accepted input from stdin, then I could do something like the following to easily get a view on the most missed urls: varnishlog -c -o VCL_call "^miss" | varnishtop -i RxUrl I assume I could do this by using varnishlog to write to a file and varnishtop to read it, but then I'd end up with huge files that need cleaning. -Brad From tfheen at varnish-software.com Wed Mar 10 11:10:05 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 10 Mar 2010 12:10:05 +0100 Subject: Moving varnish-cache.org tomorrow In-Reply-To: <4B964E12.1010200@redpill-linpro.com> (Nicolai Langfeldt's message of "Tue, 09 Mar 2010 14:33:06 +0100") References: <4B964E12.1010200@redpill-linpro.com> Message-ID: <87pr3c77qq.fsf@qurzaw.linpro.no> ]] Nicolai Langfeldt | We'll be moving varnish-cache.org tomorrow, Wednesday 2010-03-10 at | about 11:00 CET. The site will be unavailable for about 5 (or 30) | minutes while we move it. After a long 5 minutes, we have now moved. [...] | At the same time the mailing lists will change address to | | * varnish-announce at varnish-cache.org | * varnish-bugs at varnish-cache.org | * varnish-commit at varnish-cache.org | * varnish-dev at varnish-cache.org | * varnish-dist at varnish-cache.org | * varnish-misc at varnish-cache.org | * varnish-test at varnish-cache.org This is in effect now. Please note that you might need to update any filters you have to ensure mails go into the right folder. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From david.birdsong at gmail.com Wed Mar 10 16:54:23 2010 From: david.birdsong at gmail.com (David Birdsong) Date: Wed, 10 Mar 2010 08:54:23 -0800 Subject: varnishtop stdin In-Reply-To: References: Message-ID: Pretty sure that: varnishtop -i txurl Provides what you're after. On Mar 9, 2010 11:33 PM, "Brad Schick" wrote: It would be a nice feature if varnishtop accepted input from stdin, then I could do something like the following to easily get a view on the most missed urls: varnishlog -c -o VCL_call "^miss" | varnishtop -i RxUrl I assume I could do this by using varnishlog to write to a file and varnishtop to read it, but then I'd end up with huge files that need cleaning. -Brad _______________________________________________ varnish-misc mailing list varnish-misc at projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Thu Mar 11 09:20:37 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 11 Mar 2010 10:20:37 +0100 Subject: varnishtop stdin In-Reply-To: (David Birdsong's message of "Wed, 10 Mar 2010 08:54:23 -0800") References: Message-ID: <87bpev5i56.fsf@qurzaw.linpro.no> ]] David Birdsong | Pretty sure that: | | varnishtop -i txurl | | Provides what you're after. In this particular case, sure. However, it would allow us to do something like: varnishlog -o RxUrl "/foo" | varnishtop -i RxURL It'd certainly be useful to be able to, I don't know if you can trick it with getting varnishlog to write its binary format to /dev/stdin and varnishtop to /dev/stdout, but if not, we could add that fairly easily. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From miams at reisys.com Thu Mar 11 09:52:30 2010 From: miams at reisys.com (Michael Iams) Date: Thu, 11 Mar 2010 04:52:30 -0500 Subject: varnishtop stdin In-Reply-To: <87bpev5i56.fsf@qurzaw.linpro.no> References: <87bpev5i56.fsf@qurzaw.linpro.no> Message-ID: <830984239864EA43AD45501FC8A379B70495F56DD8@mail.reisys.com> We are seeing an issue where one-for-one increments in "sms_nreq/SMS allocator requests" are resulting in 503 errors. We are not getting any backend related errors. This is happening on a very small percentage of the page requests, but it is repeatable in our load-tests. A couple questions: 1) Is this a problem that has been seen before? Is there a known configuration adjustment, work-around? 2) What specific function is performed by the SMS metrics measured in varnishstat? The most I've found in my digging is that it is used for internal memory house-keeping. There was a similar inquiry a couple of days ago on this message-list regarding SMS, SMA, SM, etc. It would be good if there was something published somewhere that elaborates on what these measure. From tami.lee at gmail.com Fri Mar 12 04:05:24 2010 From: tami.lee at gmail.com (Tami Lee) Date: Thu, 11 Mar 2010 20:05:24 -0800 Subject: No output from varnishncsa Message-ID: <3e3a0c981003112005s1c45ffandc7a5c104c78ca23@mail.gmail.com> Hi All, I am new to Varnish. I started Varnish with the command below and hit it with a few requests. It worked fine. However, I am unable to get output from varnishncsa (varnishncsa -r /tmp/varnish/working/_.vsl or varnishncsa -n /tmp/varnish/working). Am I missing something? $ /usr/sbin/varnishd -a :4080 -T 127.0.0.1:4082 -f /tmp/varnish/test.vcl -s file,/tmp/varnish/cache/varnish.cache,128M -n /tmp/varnish/working -P /tmp/varnish/working/pid Thanks in advance for your help! Tami -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Fri Mar 12 12:01:42 2010 From: slink at schokola.de (Nils Goroll) Date: Fri, 12 Mar 2010 13:01:42 +0100 Subject: autoconf macro so generate pthread compiler flags // Re: Child panics on OpenSolaris In-Reply-To: <4B7BBDFA.7080502@schokola.de> References: <81081.1266353325@critter.freebsd.dk> <4B7BBDFA.7080502@schokola.de> Message-ID: <4B9A2D26.1030103@schokola.de> > Fantastic. Thank you for putting so much effort into this. > > But also: Stupid I didn't think about this. See how I'm compiling varnish since > I started working with it: > > ## 64bit > LDFLAGS="-lpthread" > CFLAGS="-D_REENTRANT -m64" \ > VCC_CC="exec gcc -fpic -D_REENTRANT -m64 -shared -o %o %s" \ > ./configure \ > '--enable-debugging-symbols' \ > '--enable-developer-warnings' \ > '--enable-dependency-tracking' \ > .... I've tried to automate this configuration in configure.ac. See: http://varnish-cache.org/ticket/663 Nils From slink at schokola.de Fri Mar 12 12:02:50 2010 From: slink at schokola.de (Nils Goroll) Date: Fri, 12 Mar 2010 13:02:50 +0100 Subject: autoconf macro so generate pthread compiler flags // Re: Child panics on OpenSolaris In-Reply-To: <4B7BBDFA.7080502@schokola.de> References: <81081.1266353325@critter.freebsd.dk> <4B7BBDFA.7080502@schokola.de> Message-ID: <4B9A2D6A.2060301@schokola.de> (repost to varnish-cache.org. Sorry, haven't adapted the new domain yet) > Fantastic. Thank you for putting so much effort into this. > > But also: Stupid I didn't think about this. See how I'm compiling varnish since > I started working with it: > > ## 64bit > LDFLAGS="-lpthread" > CFLAGS="-D_REENTRANT -m64" \ > VCC_CC="exec gcc -fpic -D_REENTRANT -m64 -shared -o %o %s" \ > ./configure \ > '--enable-debugging-symbols' \ > '--enable-developer-warnings' \ > '--enable-dependency-tracking' \ > .... I've tried to automate this configuration in configure.ac. See: http://varnish-cache.org/ticket/663 Nils From David at firechaser.com Sat Mar 13 09:58:53 2010 From: David at firechaser.com (David Murphy) Date: Sat, 13 Mar 2010 09:58:53 +0000 Subject: Handling cookies Message-ID: <345BD8B3F8775748A4676A625EADA22417D87508@DURAN.firechaser.local> Hello I am trying to configure Varnish to cache everything except a few specific URLs that need to be passed back to Apache/PHP. This is what I have so far sub vcl_recv { if (req.request == "POST" || req.url ~ "^/my-details" || req.url ~ "^/cart" || req.url ~ "^/billing-details" || req.url ~ "^/delivery" ) { pass; } if (req.request == "GET") { unset req.http.cookie; unset req.http.Authorization; lookup; } //default action, just in case lookup; } sub vcl_fetch { if (req.request == "GET") { unset obj.http.Set-Cookie; set obj.ttl = 30m; } } sub vcl_hash { if (req.http.Cookie) { set req.hash += req.http.Cookie; } What I'm finding are two issues: 1. javascript and css files are not being cached (I am monitoring using varnishtop -b -i TxURL ) even though my vcl_recv would appear to be caching everything that is pulled via GET 2. The non-cached pages ( e.g. /my-details accessed via GET) do not seem to pick up all of the cookies generated in PHP, yet if I call the same page via Apache directly ( by passing Varnish) the cookies are found fine. If someone could point me in the right direction that'd be much appreciated. Thanks, David From David at firechaser.com Sat Mar 13 11:39:41 2010 From: David at firechaser.com (David Murphy) Date: Sat, 13 Mar 2010 11:39:41 +0000 Subject: Cache miss for same page from different browsers Message-ID: <345BD8B3F8775748A4676A625EADA22417D87509@DURAN.firechaser.local> Hello I've been testing a simple txt page to confirm that when viewed by one browser (and cached in Varnish) that it is delivered as a Varnish cached file to a second browser. My test is very simple: I have both Chrome and Firefox browsers open, and am trying access http:127.0.0.1/test.txt 1. Restart varnishd + apache2 2. Clear cookies/browser cache in obth browsers 3. Visit http:127.0.0.1/test.txt in Chrome and confirm via varnishtop -b -i TxURL that /text.txt has been cached === list length 2 1.00 TxURL /robots.txt 1.00 TxURL /favicon.ico === 4.Then visit same URL in Firefox Now, the header results from (4) when I reach /test.txt (but already cached in Varnish) is X-Cacheable YES Date Sat, 13 Mar 2010 11:33:22 GMT X-Varnish 1385957804 Age 43 Via 1.1 varnish X-Cache MISS Then if I hit refresh on this page (not force reload) I get X-Cacheable YES Date Sat, 13 Mar 2010 11:34:15 GMT X-Varnish 1385957805 Age 43 Via 1.1 varnish X-Cache HIT Note that there is only a single X-Varnish request number (1385957805) even though we get X-Cache: HIT If I refresh again I get the same results, but if I force reload (shift + reload) I get: X-Cacheable YES Date Sat, 13 Mar 2010 11:36:28 GMT X-Varnish 1385957807 1385957804 Age 187 Via 1.1 varnish Connection keep-alive X-Cache HIT My VCL file is below: =============== GNU nano 2.0.9 File: /etc/varnish/default.vcl backend default { .host = "127.0.0.1"; .port = "8080"; .connect_timeout = 600s; .first_byte_timeout = 600s; .between_bytes_timeout = 600s; } sub vcl_recv { unset req.http.cookie; lookup; } sub vcl_fetch { unset obj.http.Set-Cookie; # Varnish determined the object was not cacheable if (!obj.cacheable) { set obj.http.X-Cacheable = "NO:Not Cacheable"; # You don't wish to cache content for logged in users } elsif(req.http.Cookie ~"(UserID|_session)") { set obj.http.X-Cacheable = "NO:Got Session"; pass; # You are respecting the Cache-Control=private header from the backend } elsif ( obj.http.Cache-Control ~ "private") { set obj.http.X-Cacheable = "NO:Cache-Control=private"; pass; # You are extending the lifetime of the object artificially } elsif ( obj.ttl < 1s ) { set obj.ttl = 5s; set obj.grace = 5s; set obj.http.X-Cacheable = "YES:FORCED"; # Varnish determined the object was cacheable } else { set obj.http.X-Cacheable = "YES"; } deliver; } =============== Thanks for any assistance. David From l at lrowe.co.uk Sat Mar 13 16:49:07 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Sat, 13 Mar 2010 16:49:07 +0000 Subject: Cache miss for same page from different browsers In-Reply-To: <345BD8B3F8775748A4676A625EADA22417D87509@DURAN.firechaser.local> References: <345BD8B3F8775748A4676A625EADA22417D87509@DURAN.firechaser.local> Message-ID: You probably have a Vary: Accept-Encoding header on the response. See http://varnish-cache.org/wiki/FAQ/Compression Laurence On 13 March 2010 11:39, David Murphy wrote: > Hello > > I've been testing a simple txt page to confirm that when viewed by one browser (and cached in Varnish) that it is delivered as a Varnish cached file to a second browser. > > My test is very simple: I have both Chrome and Firefox browsers open, and am trying access http:127.0.0.1/test.txt > > 1. Restart varnishd + apache2 > 2. Clear cookies/browser cache in obth browsers > 3. Visit http:127.0.0.1/test.txt in Chrome and confirm via varnishtop -b -i TxURL that /text.txt has been cached > > === > list length 2 > ? ? 1.00 TxURL ? ? ? ? ?/robots.txt > ? ? 1.00 TxURL ? ? ? ? ?/favicon.ico > === > > 4.Then visit same URL in Firefox > > Now, the header results from (4) when I reach /test.txt (but already cached in Varnish) is > > X-Cacheable ? ? YES > Date ? ?Sat, 13 Mar 2010 11:33:22 GMT > X-Varnish ? ? ? 1385957804 > Age ? ? 43 > Via ? ? 1.1 varnish > X-Cache MISS > > Then if I hit refresh on this page (not force reload) I get > > X-Cacheable ? ? YES > Date ? ?Sat, 13 Mar 2010 11:34:15 GMT > X-Varnish ? ? ? 1385957805 > Age ? ? 43 > Via ? ? 1.1 varnish > X-Cache HIT > > Note that there is only a single X-Varnish request number (1385957805) even though we get X-Cache: HIT > > If I refresh again I get ?the same results, but if I force reload (shift + reload) I get: > > X-Cacheable ? ? YES > Date ? ?Sat, 13 Mar 2010 11:36:28 GMT > X-Varnish ? ? ? 1385957807 1385957804 > Age ? ? 187 > Via ? ? 1.1 varnish > Connection ? ? ?keep-alive > X-Cache HIT > > My VCL file is below: > > =============== > > ?GNU nano 2.0.9 ? ? ? ? ? ? ? ? ? ? ? ?File: /etc/varnish/default.vcl > > backend default { > .host = "127.0.0.1"; > .port = "8080"; > .connect_timeout = 600s; > .first_byte_timeout = 600s; > .between_bytes_timeout = 600s; > } > > sub vcl_recv { > unset req.http.cookie; > lookup; > } > > sub vcl_fetch { > unset obj.http.Set-Cookie; > > ? ?# Varnish determined the object was not cacheable > ? ?if (!obj.cacheable) { > ? ? ? ?set obj.http.X-Cacheable = "NO:Not Cacheable"; > > ? ?# You don't wish to cache content for logged in users > ? ?} elsif(req.http.Cookie ~"(UserID|_session)") { > ? ? ? ?set obj.http.X-Cacheable = "NO:Got Session"; > ? ? ? ?pass; > > ? ?# You are respecting the Cache-Control=private header from the backend > ? ?} elsif ( obj.http.Cache-Control ~ "private") { > ? ? ? ?set obj.http.X-Cacheable = "NO:Cache-Control=private"; > ? ? ? ?pass; > > ? ?# You are extending the lifetime of the object artificially > ? ?} elsif ( obj.ttl < 1s ) { > ? ? ? ?set obj.ttl ? = 5s; > ? ? ? ?set obj.grace = 5s; > ? ? ? ?set obj.http.X-Cacheable = "YES:FORCED"; > > ? ?# Varnish determined the object was cacheable > ? ?} else { > ? ? ? ?set obj.http.X-Cacheable = "YES"; > ? ?} > > ? ?deliver; > } > > =============== > > > Thanks for any assistance. > > David > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From David at firechaser.com Sat Mar 13 17:09:47 2010 From: David at firechaser.com (David Murphy) Date: Sat, 13 Mar 2010 17:09:47 +0000 Subject: Cache miss for same page from different browsers In-Reply-To: References: <345BD8B3F8775748A4676A625EADA22417D87509@DURAN.firechaser.local>, Message-ID: <345BD8B3F8775748A4676A625EADA22417D87516@DURAN.firechaser.local> That did the trick, thank you Laurence. ________________________________________ From: laurencerowe at gmail.com [laurencerowe at gmail.com] On Behalf Of Laurence Rowe [l at lrowe.co.uk] Sent: 13 March 2010 16:49 To: David Murphy Cc: varnish-misc at varnish-cache.org Subject: Re: Cache miss for same page from different browsers You probably have a Vary: Accept-Encoding header on the response. See http://varnish-cache.org/wiki/FAQ/Compression Laurence On 13 March 2010 11:39, David Murphy wrote: > Hello > > I've been testing a simple txt page to confirm that when viewed by one browser (and cached in Varnish) that it is delivered as a Varnish cached file to a second browser. > > My test is very simple: I have both Chrome and Firefox browsers open, and am trying access http:127.0.0.1/test.txt > > 1. Restart varnishd + apache2 > 2. Clear cookies/browser cache in obth browsers > 3. Visit http:127.0.0.1/test.txt in Chrome and confirm via varnishtop -b -i TxURL that /text.txt has been cached > > === > list length 2 > 1.00 TxURL /robots.txt > 1.00 TxURL /favicon.ico > === > > 4.Then visit same URL in Firefox > > Now, the header results from (4) when I reach /test.txt (but already cached in Varnish) is > > X-Cacheable YES > Date Sat, 13 Mar 2010 11:33:22 GMT > X-Varnish 1385957804 > Age 43 > Via 1.1 varnish > X-Cache MISS > > Then if I hit refresh on this page (not force reload) I get > > X-Cacheable YES > Date Sat, 13 Mar 2010 11:34:15 GMT > X-Varnish 1385957805 > Age 43 > Via 1.1 varnish > X-Cache HIT > > Note that there is only a single X-Varnish request number (1385957805) even though we get X-Cache: HIT > > If I refresh again I get the same results, but if I force reload (shift + reload) I get: > > X-Cacheable YES > Date Sat, 13 Mar 2010 11:36:28 GMT > X-Varnish 1385957807 1385957804 > Age 187 > Via 1.1 varnish > Connection keep-alive > X-Cache HIT > > My VCL file is below: > > =============== > > GNU nano 2.0.9 File: /etc/varnish/default.vcl > > backend default { > .host = "127.0.0.1"; > .port = "8080"; > .connect_timeout = 600s; > .first_byte_timeout = 600s; > .between_bytes_timeout = 600s; > } > > sub vcl_recv { > unset req.http.cookie; > lookup; > } > > sub vcl_fetch { > unset obj.http.Set-Cookie; > > # Varnish determined the object was not cacheable > if (!obj.cacheable) { > set obj.http.X-Cacheable = "NO:Not Cacheable"; > > # You don't wish to cache content for logged in users > } elsif(req.http.Cookie ~"(UserID|_session)") { > set obj.http.X-Cacheable = "NO:Got Session"; > pass; > > # You are respecting the Cache-Control=private header from the backend > } elsif ( obj.http.Cache-Control ~ "private") { > set obj.http.X-Cacheable = "NO:Cache-Control=private"; > pass; > > # You are extending the lifetime of the object artificially > } elsif ( obj.ttl < 1s ) { > set obj.ttl = 5s; > set obj.grace = 5s; > set obj.http.X-Cacheable = "YES:FORCED"; > > # Varnish determined the object was cacheable > } else { > set obj.http.X-Cacheable = "YES"; > } > > deliver; > } > > =============== > > > Thanks for any assistance. > > David > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From abc at digithi.de Sun Mar 14 11:30:34 2010 From: abc at digithi.de (Thimo E.) Date: Sun, 14 Mar 2010 12:30:34 +0100 Subject: Connections to backend not closing In-Reply-To: References: <1710.1265925572@critter.freebsd.dk> <4B748EF8.3050606@digithi.de> Message-ID: <4B9CC8DA.10607@digithi.de> Hi folks, just wanted to inform you that my problem is solved. It turned out that the iptables conntection tracking module with the following iptables rules iptables -A INPUT -m state --state INVALID -j DROP iptables -A OUTPUT -m state --state INVALID -j DROP have eaten the last FIN and ACK packets. It seems that iptables declares FIN, ACK packets as invalid if the TCP teardown takes too lang (> 180 seconds). Paul, Michael, Laurence...Thanks for your help! Best regards Thimo E. Am 12.02.2010 11:53, schrieb Laurence Rowe: > On 12 February 2010 00:12, Thimo E. wrote: > >> Hello Poul, hello Michael, >> >> >The impact [of sockets in FIN_WAIT2] should be no more than a bit of RAM. >> I disagree slightly :) The application which is waiting in FIN_WAIT2 has >> allocated structures, threads which (may or may not) consume CPU time, >> ... and last but not least the value of max opened sockets will be >> reduced by those dead sockets. >> And..as I wrote already..due to that many opened sockets my backend >> stops responding because of "Too many open connections". >> >> >> Situation after 2 days running varnish: >> >> netstat -p: >> 520 connections in FIN_WAIT2 state >> >> varnishstat: >> ... >> 438 0.00 0.01 Backend conn. reuses >> 547 0.00 0.01 Backend conn. was closed >> 988 0.00 0.02 Backend conn. recycles >> ... >> >> >>> If you look in varnishstat, does the number correlate to the >>> "Backend Conn." activity counters in any way ? >>> >> Poul, the 547 closed backend connections are quite near to 520 FIN_WAIT2 >> connections. >> >> Any suggestions ? >> > From erik at erikwickstrom.com Mon Mar 15 05:38:12 2010 From: erik at erikwickstrom.com (Erik Wickstrom) Date: Sun, 14 Mar 2010 22:38:12 -0700 Subject: Varnish won't start Message-ID: <3d381e171003142238p6e0384ebhe57be8042fe3377e@mail.gmail.com> Hi, I just installed varnish via apt on Ubuntu 9.10. When I try to start it, nothing happens. If I turn debugging on, I get this error: root at varnish-1:~# /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -s file,/var/lib/varnish/server/varnish_storage.bin,1G -d -d storage_file: filename: /var/lib/varnish/server/varnish_storage.bin size 1024 MB. Using old SHMFILE Debugging mode, enter "start" to start child start Assert error in start_child(), mgt_child.c line 300: Condition(open("/dev/null", O_RDONLY) == STDIN_FILENO) not true. errno = 13 (Permission denied) child (19962) Started Pushing vcls failed: CLI communication error root at varnish-1:~# uname -a Linux varnish-1 2.6.18-128.2.1.el5.028stab064.8 #1 SMP Fri Nov 6 12:26:59 MSK 2009 i686 GNU/Linux root at varnish-1:~# I can't even start it as root. Same error. Thanks for your help! Erik -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.boer at netclever.nl Mon Mar 15 07:17:00 2010 From: martin.boer at netclever.nl (Martin Boer) Date: Mon, 15 Mar 2010 08:17:00 +0100 Subject: Cache miss for same page from different browsers In-Reply-To: <345BD8B3F8775748A4676A625EADA22417D87509@DURAN.firechaser.local> References: <345BD8B3F8775748A4676A625EADA22417D87509@DURAN.firechaser.local> Message-ID: <4B9DDEEC.8090802@netclever.nl> Hello David, I'm afraid you need some coffee; David Murphy wrote: > Hello > > I've been testing a simple txt page to confirm that when viewed by one browser (and cached in Varnish) that it is delivered as a Varnish cached file to a second browser. > > My test is very simple: I have both Chrome and Firefox browsers open, and am trying access http:127.0.0.1/test.txt > > 1. Restart varnishd + apache2 > 2. Clear cookies/browser cache in obth browsers > 3. Visit http:127.0.0.1/test.txt in Chrome and confirm via varnishtop -b -i TxURL that /text.txt has been cached > > === > list length 2 > 1.00 TxURL /robots.txt > 1.00 TxURL /favicon.ico > === > > test.txt hasn't been cached here but robots.txt is. I'm afraid you misread. > 4.Then visit same URL in Firefox > > Now, the header results from (4) when I reach /test.txt (but already cached in Varnish) is > > X-Cacheable YES > Date Sat, 13 Mar 2010 11:33:22 GMT > X-Varnish 1385957804 > Age 43 > Via 1.1 varnish > X-Cache MISS > > Then if I hit refresh on this page (not force reload) I get > > X-Cacheable YES > Date Sat, 13 Mar 2010 11:34:15 GMT > X-Varnish 1385957805 > Age 43 > Via 1.1 varnish > X-Cache HIT > > Note that there is only a single X-Varnish request number (1385957805) even though we get X-Cache: HIT > > If I refresh again I get the same results, but if I force reload (shift + reload) I get: > > X-Cacheable YES > Date Sat, 13 Mar 2010 11:36:28 GMT > X-Varnish 1385957807 1385957804 > Age 187 > Via 1.1 varnish > Connection keep-alive > X-Cache HIT > > My VCL file is below: > > =============== > > GNU nano 2.0.9 File: /etc/varnish/default.vcl > > backend default { > .host = "127.0.0.1"; > .port = "8080"; > .connect_timeout = 600s; > .first_byte_timeout = 600s; > .between_bytes_timeout = 600s; > } > > sub vcl_recv { > unset req.http.cookie; > lookup; > } > > sub vcl_fetch { > unset obj.http.Set-Cookie; > > # Varnish determined the object was not cacheable > if (!obj.cacheable) { > set obj.http.X-Cacheable = "NO:Not Cacheable"; > > # You don't wish to cache content for logged in users > } elsif(req.http.Cookie ~"(UserID|_session)") { > set obj.http.X-Cacheable = "NO:Got Session"; > pass; > > # You are respecting the Cache-Control=private header from the backend > } elsif ( obj.http.Cache-Control ~ "private") { > set obj.http.X-Cacheable = "NO:Cache-Control=private"; > pass; > > # You are extending the lifetime of the object artificially > } elsif ( obj.ttl < 1s ) { > set obj.ttl = 5s; > set obj.grace = 5s; > set obj.http.X-Cacheable = "YES:FORCED"; > > # Varnish determined the object was cacheable > } else { > set obj.http.X-Cacheable = "YES"; > } > > deliver; > } > > =============== > > > Thanks for any assistance. > > David > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > From phk at phk.freebsd.dk Mon Mar 15 08:07:21 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 15 Mar 2010 08:07:21 +0000 Subject: Varnish won't start In-Reply-To: Your message of "Sun, 14 Mar 2010 22:38:12 MST." <3d381e171003142238p6e0384ebhe57be8042fe3377e@mail.gmail.com> Message-ID: <42439.1268640441@critter.freebsd.dk> In message <3d381e171003142238p6e0384ebhe57be8042fe3377e at mail.gmail.com>, Erik Wickstrom writes: >Assert error in start_child(), mgt_child.c line 300: > Condition(open("/dev/null", O_RDONLY) == STDIN_FILENO) not true. Do you have a /dev/null ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From David at firechaser.com Mon Mar 15 08:49:37 2010 From: David at firechaser.com (David Murphy) Date: Mon, 15 Mar 2010 08:49:37 +0000 Subject: Cache miss for same page from different browsers In-Reply-To: <4B9DDEEC.8090802@netclever.nl> References: <345BD8B3F8775748A4676A625EADA22417D87509@DURAN.firechaser.local>, <4B9DDEEC.8090802@netclever.nl> Message-ID: <345BD8B3F8775748A4676A625EADA22417D8751F@DURAN.firechaser.local> Thanks Martin Yes, sorry that was just a typo (should have been 'tobots.txt' not 'text.txt'), but the details of the issue are correct. Laurence kindly suggested it was a Vary: Accept-Encoding header issue and after following http://varnish-cache.org/wiki/FAQ/Compression the issue is now sorted. Best, David ________________________________________ From: Martin Boer [martin.boer at netclever.nl] Sent: 15 March 2010 07:17 To: David Murphy Cc: varnish-misc at varnish-cache.org Subject: Re: Cache miss for same page from different browsers Hello David, I'm afraid you need some coffee; David Murphy wrote: > Hello > > I've been testing a simple txt page to confirm that when viewed by one browser (and cached in Varnish) that it is delivered as a Varnish cached file to a second browser. > > My test is very simple: I have both Chrome and Firefox browsers open, and am trying access http:127.0.0.1/test.txt > > 1. Restart varnishd + apache2 > 2. Clear cookies/browser cache in obth browsers > 3. Visit http:127.0.0.1/test.txt in Chrome and confirm via varnishtop -b -i TxURL that /text.txt has been cached > > === > list length 2 > 1.00 TxURL /robots.txt > 1.00 TxURL /favicon.ico > === > > test.txt hasn't been cached here but robots.txt is. I'm afraid you misread. > 4.Then visit same URL in Firefox > > Now, the header results from (4) when I reach /test.txt (but already cached in Varnish) is > > X-Cacheable YES > Date Sat, 13 Mar 2010 11:33:22 GMT > X-Varnish 1385957804 > Age 43 > Via 1.1 varnish > X-Cache MISS > > Then if I hit refresh on this page (not force reload) I get > > X-Cacheable YES > Date Sat, 13 Mar 2010 11:34:15 GMT > X-Varnish 1385957805 > Age 43 > Via 1.1 varnish > X-Cache HIT > > Note that there is only a single X-Varnish request number (1385957805) even though we get X-Cache: HIT > > If I refresh again I get the same results, but if I force reload (shift + reload) I get: > > X-Cacheable YES > Date Sat, 13 Mar 2010 11:36:28 GMT > X-Varnish 1385957807 1385957804 > Age 187 > Via 1.1 varnish > Connection keep-alive > X-Cache HIT > > My VCL file is below: > > =============== > > GNU nano 2.0.9 File: /etc/varnish/default.vcl > > backend default { > .host = "127.0.0.1"; > .port = "8080"; > .connect_timeout = 600s; > .first_byte_timeout = 600s; > .between_bytes_timeout = 600s; > } > > sub vcl_recv { > unset req.http.cookie; > lookup; > } > > sub vcl_fetch { > unset obj.http.Set-Cookie; > > # Varnish determined the object was not cacheable > if (!obj.cacheable) { > set obj.http.X-Cacheable = "NO:Not Cacheable"; > > # You don't wish to cache content for logged in users > } elsif(req.http.Cookie ~"(UserID|_session)") { > set obj.http.X-Cacheable = "NO:Got Session"; > pass; > > # You are respecting the Cache-Control=private header from the backend > } elsif ( obj.http.Cache-Control ~ "private") { > set obj.http.X-Cacheable = "NO:Cache-Control=private"; > pass; > > # You are extending the lifetime of the object artificially > } elsif ( obj.ttl < 1s ) { > set obj.ttl = 5s; > set obj.grace = 5s; > set obj.http.X-Cacheable = "YES:FORCED"; > > # Varnish determined the object was cacheable > } else { > set obj.http.X-Cacheable = "YES"; > } > > deliver; > } > > =============== > > > Thanks for any assistance. > > David > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > From vlists at veus.hr Mon Mar 15 12:57:49 2010 From: vlists at veus.hr (Vladimir Vuksan) Date: Mon, 15 Mar 2010 08:57:49 -0400 (EDT) Subject: Varnish won't start In-Reply-To: <3d381e171003142238p6e0384ebhe57be8042fe3377e@mail.gmail.com> References: <3d381e171003142238p6e0384ebhe57be8042fe3377e@mail.gmail.com> Message-ID: You are running Ubuntu however your kernel is?2.6.18-128.2.1.el5.028stab064.8 (presumably a hand rolled kernel distributed with RH). I don't know if that is the cause of the problem. Perhaps not. The only other thing I can think of is AppArmor although presumably if it was packaged by Ubuntu they would have included an AppArmor profile. Vladimir On Sun, 14 Mar 2010, Erik Wickstrom wrote: > I just installed varnish via apt on Ubuntu 9.10. ?When I try to start it, nothing happens. ?If I > turn debugging on, I get this error: > > root at varnish-1:~# /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082 -f > /etc/varnish/default.vcl -s file,/var/lib/varnish/server/varnish_storage.bin,1G -d -d > storage_file: filename: /var/lib/varnish/server/varnish_storage.bin size 1024 MB. > Using old SHMFILE > Debugging mode, enter "start" to start child > start > Assert error in start_child(), mgt_child.c line 300: > ??Condition(open("/dev/null", O_RDONLY) == STDIN_FILENO) not true. > ??errno = 13 (Permission denied) > child (19962) Started > Pushing vcls failed: CLI communication error > root at varnish-1:~# uname -a > Linux?varnish-1?2.6.18-128.2.1.el5.028stab064.8 #1 SMP Fri Nov 6 12:26:59 MSK 2009 i686 GNU/Linux > root at varnish-1:~#? > > I can't even start it as root. ?Same error. From ketil at froyn.name Mon Mar 15 23:56:01 2010 From: ketil at froyn.name (Ketil Froyn) Date: Tue, 16 Mar 2010 00:56:01 +0100 Subject: Cache size Message-ID: <8585e2641003151656w6c898c4by4ef3758a526ab980@mail.gmail.com> Hi, The Varnish FAQ says: Q: How large a cache do I need? A: You should have a cache at least the size of your working data set. ... Is there a good way to determine the size of the working data set, so that I can be pretty sure I'm allocating the right amount of resources for this? Regards, Ketil From robertbrogers at gmail.com Tue Mar 16 03:28:45 2010 From: robertbrogers at gmail.com (Rob Rogers) Date: Mon, 15 Mar 2010 20:28:45 -0700 Subject: varnish && zope/plone: varnish does not find cachefu purges: returns 404 instead. Message-ID: The Issue: cachefu fails to purge cache on object changes. This means user sees stale, incorrect content. Product Versions: Plone-2.5.3, plone2.5.3, CacheFu-1.0.2, varnish-2.0.6 To reproduce: * In browser1: login and go to any page * In browser 2: view same page. * In browser 1: edit page (add some dots to title). save * In browser 2: refresh page THE BUG: browser 2 shows old version of page Another way to reproduce this is to use portal_squid tool to purge article. Same problem occurs. ./bin/varnishlog shows the error: """ 16 RxURL c http://dude.com.moneyXXX/archive/2010/february/the-thirteen-races-to-watch 16 RxProtocol c HTTP/1.0 16 VCL_call c recv 16 VCL_acl c MATCH purge localhost 16 VCL_return c lookup 16 VCL_call c hash 16 VCL_return c hash 16 VCL_call c miss 0 Debug - "VCL_error(404, Not in cache)" """ The odd thing is that normal requests from the browser produce a RxURL like: >> 13 RxURL c /archive/2010/february/the-thirteen-races-to- watch I find this odd, because the requests from the browser to varnish don't include the http host. I wonder if this version of cachefu doesn't work with varnish. A couple notes: * I tried messing with the varnish.vcl to normalize the http.host /* if (req.http.host ~ "(www.)?xxx.org") { */ /* set req.http.host = "xxx.org"; */ /* } else { */ /* error 404 req.http.host; /\* "Unknown virtual host." *\/ */ /* } */ but, it failed to produce desired results * i started messing with it because of this post: http://davidjb.com/blog/2009/01/plone-varnish-configuration-cache-hits-purge-fails and this: http://varnish-cache.org/wiki/VCLExampleNormalizingReq Thanks for any help or guidance. -Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin.boer at netclever.nl Tue Mar 16 07:09:27 2010 From: martin.boer at netclever.nl (Martin Boer) Date: Tue, 16 Mar 2010 08:09:27 +0100 Subject: Cache size In-Reply-To: <8585e2641003151656w6c898c4by4ef3758a526ab980@mail.gmail.com> References: <8585e2641003151656w6c898c4by4ef3758a526ab980@mail.gmail.com> Message-ID: <4B9F2EA7.8080902@netclever.nl> Ketil, If the server has enough diskspace it doesn't really matter. I use 15GB as that is the free space of the largest disk partition on my smallish server. I'd suggest you just start using and finetuning varnish. And if you can spare the diskspace, don't be shy about it. Just throw a lot at varnish. You might or might not want to cache everything, might want to cache the same content multiple times for several browsertypes, might want to cache some stuff only for an hour while other stuff for ages and ages. All these things will affect the amount of cache you use. Varnish will increase performance from your current webserver(s), so you've won already. :) Regards, Martin Ketil Froyn wrote: > Hi, > > The Varnish FAQ says: > > Q: How large a cache do I need? > A: You should have a cache at least the size of your working data set. ... > > Is there a good way to determine the size of the working data set, so > that I can be pretty sure I'm allocating the right amount of resources > for this? > > Regards, Ketil > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > From David at firechaser.com Tue Mar 16 08:04:13 2010 From: David at firechaser.com (David Murphy) Date: Tue, 16 Mar 2010 08:04:13 +0000 Subject: varnish && zope/plone: varnish does not find cachefu purges: returns 404 instead. In-Reply-To: References: Message-ID: <345BD8B3F8775748A4676A625EADA22417D8753A@DURAN.firechaser.local> Hi Rob Not sure if this is the same issue (I'm new to Varnish) but http://varnish-cache.org/wiki/FAQ/Compression helped me sort out an issue I had where one browser was caching the page, but a different browser was not grabbing the cached data due to Vary: Accept-Encoding header issue. Best, David ________________________________________ From: varnish-misc-bounces at varnish-cache.org [varnish-misc-bounces at varnish-cache.org] On Behalf Of Rob Rogers [robertbrogers at gmail.com] Sent: 16 March 2010 03:28 To: varnish-misc at varnish-cache.org Subject: varnish && zope/plone: varnish does not find cachefu purges: returns 404 instead. The Issue: cachefu fails to purge cache on object changes. This means user sees stale, incorrect content. Product Versions: Plone-2.5.3, plone2.5.3, CacheFu-1.0.2, varnish-2.0.6 To reproduce: * In browser1: login and go to any page * In browser 2: view same page. * In browser 1: edit page (add some dots to title). save * In browser 2: refresh page THE BUG: browser 2 shows old version of page Another way to reproduce this is to use portal_squid tool to purge article. Same problem occurs. ./bin/varnishlog shows the error: """ 16 RxURL c http://dude.com.moneyXXX/archive/2010/february/the-thirteen-races-to-watch 16 RxProtocol c HTTP/1.0 16 VCL_call c recv 16 VCL_acl c MATCH purge localhost 16 VCL_return c lookup 16 VCL_call c hash 16 VCL_return c hash 16 VCL_call c miss 0 Debug - "VCL_error(404, Not in cache)" """ The odd thing is that normal requests from the browser produce a RxURL like: >> 13 RxURL c /archive/2010/february/the-thirteen-races-to-watch I find this odd, because the requests from the browser to varnish don't include the http host. I wonder if this version of cachefu doesn't work with varnish. A couple notes: * I tried messing with the varnish.vcl to normalize the http.host /* if (req.http.host ~ "(www.)?xxx.org") { */ /* set req.http.host = "xxx.org"; */ /* } else { */ /* error 404 req.http.host; /\* "Unknown virtual host." *\/ */ /* } */ but, it failed to produce desired results * i started messing with it because of this post: http://davidjb.com/blog/2009/01/plone-varnish-configuration-cache-hits-purge-fails and this: http://varnish-cache.org/wiki/VCLExampleNormalizingReq Thanks for any help or guidance. -Rob From ketil at froyn.name Tue Mar 16 10:47:46 2010 From: ketil at froyn.name (Ketil Froyn) Date: Tue, 16 Mar 2010 11:47:46 +0100 Subject: Cache size In-Reply-To: <4B9F2EA7.8080902@netclever.nl> References: <8585e2641003151656w6c898c4by4ef3758a526ab980@mail.gmail.com> <4B9F2EA7.8080902@netclever.nl> Message-ID: <8585e2641003160347m76d81da8w94c519364ad2508@mail.gmail.com> Hi Martin, Thanks! :) My main concern has to do with adding one more level of complexity into the service I'm running, as a problem with my Varnish configuration could result in downtime of the service. What I'm considering is to put Varnish in front of a static webserver which has about 500GB of images and preview files, distributed across a few hundred thousand files of anything from a couple of Kb to a few Mb. It's all pretty static content, but more is added all the time, some files are fetched a lot more often than others, and new files tend to be fetched a fair bit. Since there's that much data, I'm concerned about running into some thrashing issues during traffic peaks, for instance. Assuming the definition of "working data set" is approximately the size of all distinct files loaded in TTL time (eg one hour), plus a bit extra per file for headers etc, let's say that the working data set size is about 10GB. Would I get away with a varnish server with 1GB RAM and 15-20GB allocated disk space, just to have a bit of headroom? Cheers, Ketil On Tue, Mar 16, 2010 at 8:09 AM, Martin Boer wrote: > Ketil, > > If the server has enough diskspace it doesn't really matter. > I use 15GB as that is the free space of the largest disk partition on my > smallish server. > I'd suggest you just start using and finetuning varnish. And if you can > spare the diskspace, > don't be shy about it. Just throw a lot at varnish. > > You might or might not want to cache everything, might want to cache the > same content multiple times for several browsertypes, might want to cache > some stuff only for an hour while other stuff for ages and ages. > All these things will affect the amount of cache you use. > > Varnish will increase performance from your current webserver(s), so you've > won already. :) > > Regards, > Martin > > Ketil Froyn wrote: >> >> Hi, >> >> The Varnish FAQ says: >> >> Q: How large a cache do I need? >> A: You should have a cache at least the size of your working data set. ... >> >> Is there a good way to determine the size of the working data set, so >> that I can be pretty sure I'm allocating the right amount of resources >> for this? >> >> Regards, Ketil >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> >> >> > > From scabrera-varnish at portachtzig.de Tue Mar 16 16:22:53 2010 From: scabrera-varnish at portachtzig.de (Sebastian Cabrera) Date: Tue, 16 Mar 2010 17:22:53 +0100 Subject: Using GeoIP Functionality in Varnish Message-ID: <4B9FB05D.2050500@portachtzig.de> Hi everybody, On a Website i want to have the posibility to do some rewriting due to the Country he's comming from. So if someone visits the Mainpage (domain.com) from US i want to have him visiting domain.com/us/. Does anyone have any experiences with that? How can i do this? I just downloaded and installed the GeoIP Plugin from the varnish wiki (http://varnish-cache.org/wiki/GeoipUsingInlineC), but how can i use it? Is there any possibility to do something like if (get_country_code(client.ip) == "US") { set req.url = "/us/" } ? thanks for your help, regards, Sebastian From michael at dynamine.net Tue Mar 16 16:50:58 2010 From: michael at dynamine.net (Michael Fischer) Date: Tue, 16 Mar 2010 09:50:58 -0700 Subject: Cache size In-Reply-To: <8585e2641003160347m76d81da8w94c519364ad2508@mail.gmail.com> References: <8585e2641003151656w6c898c4by4ef3758a526ab980@mail.gmail.com> <4B9F2EA7.8080902@netclever.nl> <8585e2641003160347m76d81da8w94c519364ad2508@mail.gmail.com> Message-ID: Unless you're using Varnish to implement your own edge-caching service, I am not sure you're going to see significant improvement in overall image-fetch latency. As has been discussed in many threads in the past, Apache and other general-purpose webservers can serve responses for static resource requests as fast as Varnish can. If there is truly a hot set of images, then your image server's kernel will automatically place the popular images in its buffer cache and Apache/whatever will serve directly from RAM anyway. And if the working set is larger than your total image server's RAM, then adding a Varnish layer will only complicate matters without significantly boosting performance. Varnish does not make slow disks fast. You may well be better off just adding more RAM to your image servers instead of wasting it on an unnecessary Varnish tier. And if you're not using a general-purpose webserver on your image servers, fix that first. There's no benefit in running application-specific webservers for the purpose of serving static content. --Michael On Tue, Mar 16, 2010 at 3:47 AM, Ketil Froyn wrote: > Hi Martin, > > Thanks! :) My main concern has to do with adding one more level of > complexity into the service I'm running, as a problem with my Varnish > configuration could result in downtime of the service. > > What I'm considering is to put Varnish in front of a static webserver > which has about 500GB of images and preview files, distributed across > a few hundred thousand files of anything from a couple of Kb to a few > Mb. It's all pretty static content, but more is added all the time, > some files are fetched a lot more often than others, and new files > tend to be fetched a fair bit. Since there's that much data, I'm > concerned about running into some thrashing issues during traffic > peaks, for instance. > > Assuming the definition of "working data set" is approximately the > size of all distinct files loaded in TTL time (eg one hour), plus a > bit extra per file for headers etc, let's say that the working data > set size is about 10GB. Would I get away with a varnish server with > 1GB RAM and 15-20GB allocated disk space, just to have a bit of > headroom? > > Cheers, Ketil > > On Tue, Mar 16, 2010 at 8:09 AM, Martin Boer wrote: >> Ketil, >> >> If the server has enough diskspace it doesn't really matter. >> I use 15GB as that is the free space of the largest disk partition on my >> smallish server. >> I'd suggest you just start using and finetuning varnish. And if you can >> spare the diskspace, >> don't be shy about it. Just throw a lot at varnish. >> >> You might or might not want to cache everything, might want to cache the >> same content multiple times for several browsertypes, might want to cache >> some stuff only for an hour while other stuff for ages and ages. >> All these things will affect the amount of cache you use. >> >> Varnish will increase performance from your current webserver(s), so you've >> won already. :) >> >> Regards, >> Martin >> >> Ketil Froyn wrote: >>> >>> Hi, >>> >>> The Varnish FAQ says: >>> >>> Q: How large a cache do I need? >>> A: You should have a cache at least the size of your working data set. ... >>> >>> Is there a good way to determine the size of the working data set, so >>> that I can be pretty sure I'm allocating the right amount of resources >>> for this? >>> >>> Regards, Ketil >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> >>> >>> >> >> > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From richard.chiswell at mangahigh.com Tue Mar 16 16:56:56 2010 From: richard.chiswell at mangahigh.com (Richard Chiswell) Date: Tue, 16 Mar 2010 16:56:56 +0000 Subject: Using GeoIP Functionality in Varnish In-Reply-To: <4B9FB05D.2050500@portachtzig.de> References: <4B9FB05D.2050500@portachtzig.de> Message-ID: <4B9FB858.1060608@mangahigh.com> Hi Sebastian, As my C is a bit rusty and as I needed this working before the GeoIP plugin worked, I created something on the basis: sub vcl_recv { ... if (req.url !~ "^/(en_gb|en_us)/*") { set req.url = "/geoip.php?ip=" client.ip "&originalurl=" req.http.x-originalurl; lookup; } .. } This looks for urls which do not start en_gb or en_us and bounces them through geoip.php which is a simple script (basically Maxmind's default PHP script with a few minor tweaks) which then just returns a 302 redirect to example.com/(countrycode)/(original url). Hope it helps. Richard Chiswell Sebastian Cabrera wrote: > Hi everybody, > > On a Website i want to have the posibility to do some rewriting due to > the Country he's comming from. So if someone visits the Mainpage > (domain.com) from US i want to have him visiting domain.com/us/. > Does anyone have any experiences with that? How can i do this? > I just downloaded and installed the GeoIP Plugin from the varnish wiki > (http://varnish-cache.org/wiki/GeoipUsingInlineC), but how can i use it? > Is there any possibility to do something like > > if (get_country_code(client.ip) == "US") { > set req.url = "/us/" > } > > ? > > thanks for your help, > regards, > > Sebastian > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From kb+varnish at slide.com Tue Mar 16 19:21:13 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Tue, 16 Mar 2010 12:21:13 -0700 Subject: Using GeoIP Functionality in Varnish In-Reply-To: <4B9FB05D.2050500@portachtzig.de> References: <4B9FB05D.2050500@portachtzig.de> Message-ID: <5489B7D9-2350-4D85-96DF-E88E18AD2BAA@slide.com> I think what would work for you is to add only the C{} block from the top of the plugin.vcl to the top of your VCL, and do something like the following where you need to (vcl_recv() presumably): C{ if (get_country_code) { char *country = (*get_country_code)(VRT_IP_string(sp, VRT_r_server_ip(sp))); if (!strncmp(country, "US", 2)) VRT_l_req_url(sp, "/us/", vrt_magic_string_end); } }C Note that GeoIP is only accessible via C{} blocks, not VCL directly. Caveats: * No guarantees ;) * You can remove the extra "char *country" step if you want; I did it for readability * Assumes replacing the URL with "/us/" instead of prepending to the existing URL * The GeoIP plugin on the twiki could use some love; it seems overly complicated and limited. -- kb On Mar 16, 2010, at 9:22 AM, Sebastian Cabrera wrote: > Hi everybody, > > On a Website i want to have the posibility to do some rewriting due to > the Country he's comming from. So if someone visits the Mainpage > (domain.com) from US i want to have him visiting domain.com/us/. > Does anyone have any experiences with that? How can i do this? > I just downloaded and installed the GeoIP Plugin from the varnish wiki > (http://varnish-cache.org/wiki/GeoipUsingInlineC), but how can i use it? > Is there any possibility to do something like > > if (get_country_code(client.ip) == "US") { > set req.url = "/us/" > } > > ? > > thanks for your help, > regards, > > Sebastian From cosimo at streppone.it Tue Mar 16 19:54:01 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Tue, 16 Mar 2010 20:54:01 +0100 Subject: Using GeoIP Functionality in Varnish In-Reply-To: <4B9FB05D.2050500@portachtzig.de> References: <4B9FB05D.2050500@portachtzig.de> Message-ID: In data 16 marzo 2010 alle ore 17:22:53, Sebastian Cabrera ha scritto: > On a Website i want to have the posibility to do some rewriting due to > the Country he's comming from. So if someone visits the Mainpage > (domain.com) from US i want to have him visiting domain.com/us/. > Does anyone have any experiences with that? How can i do this? Maybe you could be interested at this code, http://github.com/cosimo/varnish-geoip/ We're *not* using it in production yet. However, we're using a very similar one, http://github.com/cosimo/varnish-accept-language/ And it proved very useful and stable since we increased the default sess_workspace to 128k. > I just downloaded and installed the GeoIP Plugin from the varnish wiki > (http://varnish-cache.org/wiki/GeoipUsingInlineC), but how can i use it? > Is there any possibility to do something like > > if (get_country_code(client.ip) == "US") { > set req.url = "/us/" > } No, but you can use "C{ }C" blocks. AFAIK the most common approach is to use HTTP headers as a sort of "temporary storage", hence the need of upping sess_workspace. The inline C code writes a new header, say "X-GeoIP-Country", and then you can say: sub vcl_recv { ... C{ /* "sp" is the varnish session data structure */ my_c_function(sp); }C ... if (req.http.X-GeoIP-Country ~ "us") { set req.url = "/us/"; } } Have fun :) -- Cosimo From scabrera-varnish at portachtzig.de Tue Mar 16 20:07:44 2010 From: scabrera-varnish at portachtzig.de (Sebastian Cabrera) Date: Tue, 16 Mar 2010 21:07:44 +0100 Subject: Using GeoIP Functionality in Varnish In-Reply-To: <4B9FB05D.2050500@portachtzig.de> References: <4B9FB05D.2050500@portachtzig.de> Message-ID: <4B9FE510.8040909@portachtzig.de> Hi everybody, i just replied to my own request, because i could not make a decision whom to write back :) You all gave me some very good ideas, i have to discuss them with my collegues. I also had a look at http://github.com/cosimo/varnish-geoip/, but also read something about the need to have the "city edition of GeoIP" and was not sure what this could mean. thank you very much, regards, Sebastian On 03/16/2010 05:22 PM, Sebastian Cabrera wrote: > Hi everybody, > > On a Website i want to have the posibility to do some rewriting due to > the Country he's comming from. So if someone visits the Mainpage > (domain.com) from US i want to have him visiting domain.com/us/. > Does anyone have any experiences with that? How can i do this? > I just downloaded and installed the GeoIP Plugin from the varnish wiki > (http://varnish-cache.org/wiki/GeoipUsingInlineC), but how can i use it? > Is there any possibility to do something like > > if (get_country_code(client.ip) == "US") { > set req.url = "/us/" > } > > ? > > thanks for your help, > regards, > > Sebastian > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -- A: Yes. >Q: Are you sure? >>A: Because it reverses the logical flow of conversation. >>>Q: Why is top posting frowned upon? From kb+varnish at slide.com Tue Mar 16 21:00:30 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Tue, 16 Mar 2010 14:00:30 -0700 Subject: Using GeoIP Functionality in Varnish In-Reply-To: <4B9FE510.8040909@portachtzig.de> References: <4B9FB05D.2050500@portachtzig.de> <4B9FE510.8040909@portachtzig.de> Message-ID: MaxMind has several different databases; the database that is the only one supported by most GeoIP "plugins" is Country, which can only map an IP to a country. The City edition can map IPs to cities, zip codes, latitude/longitude, etc.. The plugin you mention does require the City edition, but MaxMind has a free, less accurate GeoLite City DB. The City DB will provide country as well as the finer granularity, so for your purposes (country only) it's fine. I do like the custom header idea, mainly because you can potentially pass the header to a backend for possible processing, but also because the conditionals are easier in VCL. There are a lot of cool things you can do with the City DB beyond static mappings (see bind-geoip on Google Code) that would be cool to see in cosimo/varnish-geoip. -- kb On Mar 16, 2010, at 1:07 PM, Sebastian Cabrera wrote: > Hi everybody, > > i just replied to my own request, because i could not make a decision > whom to write back :) > > You all gave me some very good ideas, i have to discuss them with my > collegues. I also had a look at http://github.com/cosimo/varnish-geoip/, > but also read something about the need to have the "city edition of > GeoIP" and was not sure what this could mean. > > thank you very much, > regards, > > Sebastian > > On 03/16/2010 05:22 PM, Sebastian Cabrera wrote: >> Hi everybody, >> >> On a Website i want to have the posibility to do some rewriting due to >> the Country he's comming from. So if someone visits the Mainpage >> (domain.com) from US i want to have him visiting domain.com/us/. >> Does anyone have any experiences with that? How can i do this? >> I just downloaded and installed the GeoIP Plugin from the varnish wiki >> (http://varnish-cache.org/wiki/GeoipUsingInlineC), but how can i use it? >> Is there any possibility to do something like >> >> if (get_country_code(client.ip) == "US") { >> set req.url = "/us/" >> } >> >> ? >> >> thanks for your help, >> regards, >> >> Sebastian >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > -- > A: Yes. >> Q: Are you sure? >>> A: Because it reverses the logical flow of conversation. >>>> Q: Why is top posting frowned upon? > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -- kb From mrbits.dcf at gmail.com Tue Mar 16 21:08:57 2010 From: mrbits.dcf at gmail.com (MrBiTs) Date: Tue, 16 Mar 2010 18:08:57 -0300 Subject: Using GeoIP Functionality in Varnish In-Reply-To: References: <4B9FB05D.2050500@portachtzig.de> <4B9FE510.8040909@portachtzig.de> Message-ID: <4B9FF369.3010906@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 03/16/2010 06:00 PM, Ken Brownfield wrote: > MaxMind has several different databases; the database that is the only one supported by most GeoIP "plugins" is Country, which can only map an IP to a country. The City edition can map IPs to cities, zip codes, latitude/longitude, etc.. > > The plugin you mention does require the City edition, but MaxMind has a free, less accurate GeoLite City DB. The City DB will provide country as well as the finer granularity, so for your purposes (country only) it's fine. > > I do like the custom header idea, mainly because you can potentially pass the header to a backend for possible processing, but also because the conditionals are easier in VCL. > > There are a lot of cool things you can do with the City DB beyond static mappings (see bind-geoip on Google Code) that would be cool to see in cosimo/varnish-geoip. Hi, all I know other great Geo database player called Quova (http://www.quova.com) with lots of geo databases focused not only in Country, City, State or Province but Bandwidth, Zip, Area Code. We used it with great success at 24/7 RealMedia's Open AdStream and its database is highly customizable and I easy to access. Maybe could be another solution, and Varnish Software could keep contact with Quova people to think about a great module. - -- CheerS .0. MrBiTs - mrbits.dcf at gmail.com ..0 GnuPG - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6EC818FC2B3CA5AB 000 http://www.mrbits.com.br -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iQEcBAEBCAAGBQJLn/NpAAoJEG7IGPwrPKWr2RQIAIVvGsTPEWQhf0yVo9JHElH1 54+lt8y5q7RDCWS0WKqlWcXQIRJ4gpTp39qnUk0FwL+6qc//MEyYvQK1bvMwe42f m4SRItE07kPV5+SoNfR4LZPiK2/EI/SlWaBKGcTeYYPP4ysxZ3Gzf9SR8PDRPGYP hMfljzX8A/bi4GeRH3Qzc3eALV5Xsvr7IPdOgStWkOQxG3+Ydp+KTs+nVmUpBhg+ fx5IndKOmbIUTHDTS+43Uq9Jyn12valSx8F3XBV3xb1/qsmeTI/9k4yXnrH583ZO 3SXXyGwpTxliT789+2ecKoYYExUSe69KkPQhlw75JRvujQRHQoEl5uxxynicTus= =IYkp -----END PGP SIGNATURE----- From kb+varnish at slide.com Tue Mar 16 22:31:44 2010 From: kb+varnish at slide.com (Ken Brownfield) Date: Tue, 16 Mar 2010 15:31:44 -0700 Subject: Using GeoIP Functionality in Varnish In-Reply-To: <4B9FF369.3010906@gmail.com> References: <4B9FB05D.2050500@portachtzig.de> <4B9FE510.8040909@portachtzig.de> <4B9FF369.3010906@gmail.com> Message-ID: Yeah, there are several commercial ones out there (add NetAcuity to the list) but I'm not personally aware of any others that provide a free, very usable database and API, and MaxMind covers the same ground. I think the commercial (read: non-OSS) houses will have a hard time convincing anyone to support their products for them, especially with no documented APIs. :-/ -- kb On Mar 16, 2010, at 2:08 PM, MrBiTs wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 03/16/2010 06:00 PM, Ken Brownfield wrote: >> MaxMind has several different databases; the database that is the only one supported by most GeoIP "plugins" is Country, which can only map an IP to a country. The City edition can map IPs to cities, zip codes, latitude/longitude, etc.. >> >> The plugin you mention does require the City edition, but MaxMind has a free, less accurate GeoLite City DB. The City DB will provide country as well as the finer granularity, so for your purposes (country only) it's fine. >> >> I do like the custom header idea, mainly because you can potentially pass the header to a backend for possible processing, but also because the conditionals are easier in VCL. >> >> There are a lot of cool things you can do with the City DB beyond static mappings (see bind-geoip on Google Code) that would be cool to see in cosimo/varnish-geoip. > > Hi, all > > I know other great Geo database player called Quova (http://www.quova.com) with lots of geo databases focused not only in Country, > City, State or Province but Bandwidth, Zip, Area Code. We used it with great success at 24/7 RealMedia's Open AdStream and its > database is highly customizable and I easy to access. > > Maybe could be another solution, and Varnish Software could keep contact with Quova people to think about a great module. > > - -- > > CheerS > > .0. MrBiTs - mrbits.dcf at gmail.com > ..0 GnuPG - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6EC818FC2B3CA5AB > 000 http://www.mrbits.com.br > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (GNU/Linux) > > iQEcBAEBCAAGBQJLn/NpAAoJEG7IGPwrPKWr2RQIAIVvGsTPEWQhf0yVo9JHElH1 > 54+lt8y5q7RDCWS0WKqlWcXQIRJ4gpTp39qnUk0FwL+6qc//MEyYvQK1bvMwe42f > m4SRItE07kPV5+SoNfR4LZPiK2/EI/SlWaBKGcTeYYPP4ysxZ3Gzf9SR8PDRPGYP > hMfljzX8A/bi4GeRH3Qzc3eALV5Xsvr7IPdOgStWkOQxG3+Ydp+KTs+nVmUpBhg+ > fx5IndKOmbIUTHDTS+43Uq9Jyn12valSx8F3XBV3xb1/qsmeTI/9k4yXnrH583ZO > 3SXXyGwpTxliT789+2ecKoYYExUSe69KkPQhlw75JRvujQRHQoEl5uxxynicTus= > =IYkp > -----END PGP SIGNATURE----- From robertbrogers at gmail.com Wed Mar 17 10:21:00 2010 From: robertbrogers at gmail.com (Rob Rogers) Date: Wed, 17 Mar 2010 03:21:00 -0700 Subject: Making cachefu page purging work with vary headers : one solution nuke the Accept-Language <-- is this wise? Message-ID: The issue: html type pages (think objects in zope) cached in varnish, were not purged successfully with zope/plone/cachefu purging methods. Images were purged. but html content was not. Given we depend on caching pages in accelerator/zope/plone this is a showstopper. (e.g. we use this: cache-in-proxy-1-hour for (most) html content) In order to make purging work, I had to normalize or unset a couple http.req attributes. This is what I came up with. The 'working' solution: sub vcl_recv { ... if (req.http.Accept-Encoding) { /* lifted from varnish site */ if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { # No point in compressing these remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { # unkown algorithm remove req.http.Accept-Encoding; } } unset req.http.Accept-Language; /* is this DANGEROUS? */ unset req.http.user-agent; set req.http.host = "my.cashmoney.com"; ... } Based on what I have read, unsetting these attributes help keep multiple cache objects to a minimum. Which is the reasoning behind normalizing the Accept-Encoding per http://varnish-cache.org/wiki/FAQ/Compression So, I guess I have 2 questions: 1) Given the site is only served in english, is there a danger of serving up the wrong hit for a browser with a different Accept-Language? 2) Will I still get purge misses for objects cached by some browsers, like ones presenting deflate instead of gzip in the header? Thanks, Rob From l at lrowe.co.uk Wed Mar 17 11:57:51 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Wed, 17 Mar 2010 11:57:51 +0000 Subject: Making cachefu page purging work with vary headers : one solution nuke the Accept-Language <-- is this wise? In-Reply-To: References: Message-ID: Take a look at my improved vcl for plone.recipe.varnish here: http://dev.plone.org/collective/browser/buildout/plone.recipe.varnish/branches/elro-better-vcl/plone/recipe/varnish/template.vcl While you do want to normalize Accept-Encoding (remember Zope only does gzip) you still have to account for browsers which do not handle gzip. As there is currently no way to invalidate varies, I use purge to add it to the ban list. You should just set English as the only allowed language in portal_languages to prevent internationalization of message strings. Laurence On 17 March 2010 10:21, Rob Rogers wrote: > The issue: > html type pages (think objects in zope) cached in varnish, were not purged > successfully with zope/plone/cachefu purging methods. > Images were purged. but html content was not. > > Given we depend on caching pages in accelerator/zope/plone this is ?a > showstopper. (e.g. we use this: ?cache-in-proxy-1-hour for (most) html > content) > > In order to make purging work, I had to normalize or unset a couple http.req > attributes. This is what I came up with. > > The 'working' solution: > sub vcl_recv { > ? ? ? ?... > ? ? ? ?if (req.http.Accept-Encoding) { /* lifted from varnish site */ > ? ? ? ? ?if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { > ? ? ? ? ? ? ? ? ? ? ? ?# No point in compressing these > ? ? ? ? ? ?remove req.http.Accept-Encoding; > ? ? ? ? ?} elsif (req.http.Accept-Encoding ~ "gzip") { > ? ? ? ? ? ?set req.http.Accept-Encoding = "gzip"; > ? ? ? ? ?} elsif (req.http.Accept-Encoding ~ "deflate") { > ? ? ? ? ? ?set req.http.Accept-Encoding = "deflate"; > ? ? ? ? ?} else { > ? ? ? ? ? ? ? ? ? ? ? ?# unkown algorithm > ? ? ? ? ? ?remove req.http.Accept-Encoding; > ? ? ? ? ?} > ? ? ? ?} > ? ? ? ?unset req.http.Accept-Language; /* is this DANGEROUS? */ > ? ? ? ?unset req.http.user-agent; > ? ? ? ?set req.http.host = "my.cashmoney.com"; > ? ? ? ?... > } > > Based on what I have read, unsetting these attributes help keep multiple > cache objects to a minimum. > Which is the reasoning behind normalizing the Accept-Encoding per > http://varnish-cache.org/wiki/FAQ/Compression > > So, I guess I have 2 questions: > 1) Given the site is only served in english, is there a danger of serving up > the wrong hit for a browser with a different Accept-Language? > 2) Will I still get purge misses for objects ?cached by some browsers, like > ones presenting deflate instead of gzip in the header? > > > Thanks, > > Rob > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From robertbrogers at gmail.com Thu Mar 18 06:13:10 2010 From: robertbrogers at gmail.com (Rob Rogers) Date: Wed, 17 Mar 2010 23:13:10 -0700 Subject: Making cachefu page purging work with vary headers : one solution nuke the Accept-Language <-- is this wise? In-Reply-To: References: Message-ID: Laurence. First thank you so much. Second, you say: > I use purge > to add it to the ban list. What is the ban list? And, how can I replicate this? (I don't see it referenced in your recipe template.) Finally, at the risk of sounding a total fool, how do a swap your recipe in place of the current one for the buildout? Just clobber this one: ./eggs/plone.recipe.varnish-1.0.2-py2.4.egg/ plone/recipe/varnish/template.vcl ? Cheers, -Rob On Mar 17, 2010, at 4:57 AM, Laurence Rowe wrote: > Take a look at my improved vcl for plone.recipe.varnish here: > > http://dev.plone.org/collective/browser/buildout/plone.recipe.varnish/branches/elro-better-vcl/plone/recipe/varnish/template.vcl > > While you do want to normalize Accept-Encoding (remember Zope only > does gzip) you still have to account for browsers which do not handle > gzip. As there is currently no way to invalidate varies, I use purge > to add it to the ban list. > > You should just set English as the only allowed language in > portal_languages to prevent internationalization of message strings. > > Laurence > > On 17 March 2010 10:21, Rob Rogers wrote: >> The issue: >> html type pages (think objects in zope) cached in varnish, were not >> purged >> successfully with zope/plone/cachefu purging methods. >> Images were purged. but html content was not. >> >> Given we depend on caching pages in accelerator/zope/plone this is a >> showstopper. (e.g. we use this: cache-in-proxy-1-hour for (most) >> html >> content) >> >> In order to make purging work, I had to normalize or unset a couple >> http.req >> attributes. This is what I came up with. >> >> The 'working' solution: >> sub vcl_recv { >> ... >> if (req.http.Accept-Encoding) { /* lifted from varnish site */ >> if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { >> # No point in compressing these >> remove req.http.Accept-Encoding; >> } elsif (req.http.Accept-Encoding ~ "gzip") { >> set req.http.Accept-Encoding = "gzip"; >> } elsif (req.http.Accept-Encoding ~ "deflate") { >> set req.http.Accept-Encoding = "deflate"; >> } else { >> # unkown algorithm >> remove req.http.Accept-Encoding; >> } >> } >> unset req.http.Accept-Language; /* is this DANGEROUS? */ >> unset req.http.user-agent; >> set req.http.host = "my.cashmoney.com"; >> ... >> } >> >> Based on what I have read, unsetting these attributes help keep >> multiple >> cache objects to a minimum. >> Which is the reasoning behind normalizing the Accept-Encoding per >> http://varnish-cache.org/wiki/FAQ/Compression >> >> So, I guess I have 2 questions: >> 1) Given the site is only served in english, is there a danger of >> serving up >> the wrong hit for a browser with a different Accept-Language? >> 2) Will I still get purge misses for objects cached by some >> browsers, like >> ones presenting deflate instead of gzip in the header? >> >> >> Thanks, >> >> Rob >> >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> From audun at ytterdal.net Thu Mar 18 10:36:18 2010 From: audun at ytterdal.net (Audun Ytterdal) Date: Thu, 18 Mar 2010 11:36:18 +0100 Subject: syntetic images Message-ID: <8318f61f1003180336r3f4a4d49r49d337a7248ce162@mail.gmail.com> Hi People. Nowdays I have a nice errorpage for varnish using syntetic This can be triggered by adding ?vcheck=1 to any url http://www.vg.no/?vheck=1 for example But we want to jazz it up a bit. Adding innline CCS is a start ofcourse, but it would be really cool if we in some way could add a few pictures as well Is this is some way doable: if (req.url ~ "error/img.jpg") { include "/some_jpg_file_maybeprefixed_with_syntetic.jpg"; } (See you all on VUG2) -- Audun Ytterdal http://audun.ytterdal.net From l at lrowe.co.uk Thu Mar 18 12:58:20 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Thu, 18 Mar 2010 12:58:20 +0000 Subject: Making cachefu page purging work with vary headers : one solution nuke the Accept-Language <-- is this wise? In-Reply-To: References: Message-ID: See the two ways of purging: http://varnish-cache.org/wiki/VCLExamplePurging purge() adds regular expressions to a ban list which is checked before a cached copy is served. The easiest way to use that recipe is probably to check out the branch (pinned at a version, as I will eventually get around to working on it again) into your buildout and develop it. The mr.developer extension is a good way to do this. Laurence On 18 March 2010 06:13, Rob Rogers wrote: > Laurence. > > First thank you so much. > > Second, you say: >> >> I use purge >> to add it to the ban list. > > What is the ban list? And, how can I replicate this? (I don't see it > referenced in your recipe template.) > > Finally, at the risk of sounding a total fool, how do a swap your recipe in > place of the current one for the buildout? > Just clobber this one: > ./eggs/plone.recipe.varnish-1.0.2-py2.4.egg/plone/recipe/varnish/template.vcl > ? > > Cheers, > > -Rob > > > On Mar 17, 2010, at 4:57 AM, Laurence Rowe wrote: > >> Take a look at my improved vcl for plone.recipe.varnish here: >> >> >> http://dev.plone.org/collective/browser/buildout/plone.recipe.varnish/branches/elro-better-vcl/plone/recipe/varnish/template.vcl >> >> While you do want to normalize Accept-Encoding (remember Zope only >> does gzip) you still have to account for browsers which do not handle >> gzip. As there is currently no way to invalidate varies, I use purge >> to add it to the ban list. >> >> You should just set English as the only allowed language in >> portal_languages to prevent internationalization of message strings. >> >> Laurence >> >> On 17 March 2010 10:21, Rob Rogers wrote: >>> >>> The issue: >>> html type pages (think objects in zope) cached in varnish, were not >>> purged >>> successfully with zope/plone/cachefu purging methods. >>> Images were purged. but html content was not. >>> >>> Given we depend on caching pages in accelerator/zope/plone this is ?a >>> showstopper. (e.g. we use this: ?cache-in-proxy-1-hour for (most) html >>> content) >>> >>> In order to make purging work, I had to normalize or unset a couple >>> http.req >>> attributes. This is what I came up with. >>> >>> The 'working' solution: >>> sub vcl_recv { >>> ? ? ? ... >>> ? ? ? if (req.http.Accept-Encoding) { /* lifted from varnish site */ >>> ? ? ? ? if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { >>> ? ? ? ? ? ? ? ? ? ? ? # No point in compressing these >>> ? ? ? ? ? remove req.http.Accept-Encoding; >>> ? ? ? ? } elsif (req.http.Accept-Encoding ~ "gzip") { >>> ? ? ? ? ? set req.http.Accept-Encoding = "gzip"; >>> ? ? ? ? } elsif (req.http.Accept-Encoding ~ "deflate") { >>> ? ? ? ? ? set req.http.Accept-Encoding = "deflate"; >>> ? ? ? ? } else { >>> ? ? ? ? ? ? ? ? ? ? ? # unkown algorithm >>> ? ? ? ? ? remove req.http.Accept-Encoding; >>> ? ? ? ? } >>> ? ? ? } >>> ? ? ? unset req.http.Accept-Language; /* is this DANGEROUS? */ >>> ? ? ? unset req.http.user-agent; >>> ? ? ? set req.http.host = "my.cashmoney.com"; >>> ? ? ? ... >>> } >>> >>> Based on what I have read, unsetting these attributes help keep multiple >>> cache objects to a minimum. >>> Which is the reasoning behind normalizing the Accept-Encoding per >>> http://varnish-cache.org/wiki/FAQ/Compression >>> >>> So, I guess I have 2 questions: >>> 1) Given the site is only served in english, is there a danger of serving >>> up >>> the wrong hit for a browser with a different Accept-Language? >>> 2) Will I still get purge misses for objects ?cached by some browsers, >>> like >>> ones presenting deflate instead of gzip in the header? >>> >>> >>> Thanks, >>> >>> Rob >>> >>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> > > From l at lrowe.co.uk Thu Mar 18 13:00:43 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Thu, 18 Mar 2010 13:00:43 +0000 Subject: syntetic images In-Reply-To: <8318f61f1003180336r3f4a4d49r49d337a7248ce162@mail.gmail.com> References: <8318f61f1003180336r3f4a4d49r49d337a7248ce162@mail.gmail.com> Message-ID: It's possible to include an encoded image using a data: url, see http://en.wikipedia.org/wiki/Data_URI_scheme (works with firefox, chrome/safari and IE8+) Laurence On 18 March 2010 10:36, Audun Ytterdal wrote: > Hi People. > > Nowdays I have a nice errorpage for varnish using syntetic > > This can be triggered by adding ?vcheck=1 to any url > > http://www.vg.no/?vheck=1 for example > > But we want to jazz it up a bit. Adding innline CCS is a start > ofcourse, but it would be really cool if we in some way could add a > few pictures as well > > Is this is some way doable: > > if (req.url ~ "error/img.jpg") { > ?include "/some_jpg_file_maybeprefixed_with_syntetic.jpg"; > } > > (See you all on VUG2) > > -- > Audun Ytterdal > http://audun.ytterdal.net > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From audun at ytterdal.net Thu Mar 18 13:20:39 2010 From: audun at ytterdal.net (Audun Ytterdal) Date: Thu, 18 Mar 2010 14:20:39 +0100 Subject: syntetic images In-Reply-To: References: <8318f61f1003180336r3f4a4d49r49d337a7248ce162@mail.gmail.com> Message-ID: Cool. I'll look into it. By the way. The parameter for the errorpage is vcheck, not vheck Den 18. mars 2010 kl. 14.00 skrev Laurence Rowe : > It's possible to include an encoded image using a data: url, see > http://en.wikipedia.org/wiki/Data_URI_scheme (works with firefox, > chrome/safari and IE8+) > > Laurence > > On 18 March 2010 10:36, Audun Ytterdal wrote: >> Hi People. >> >> Nowdays I have a nice errorpage for varnish using syntetic >> >> This can be triggered by adding ?vcheck=1 to any url >> >> http://www.vg.no/?vheck=1 for example >> >> But we want to jazz it up a bit. Adding innline CCS is a start >> ofcourse, but it would be really cool if we in some way could add a >> few pictures as well >> >> Is this is some way doable: >> >> if (req.url ~ "error/img.jpg") { >> include "/some_jpg_file_maybeprefixed_with_syntetic.jpg"; >> } >> >> (See you all on VUG2) >> >> -- >> Audun Ytterdal >> http://audun.ytterdal.net >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> From hijinks at gmail.com Thu Mar 18 16:25:16 2010 From: hijinks at gmail.com (Mike) Date: Thu, 18 Mar 2010 12:25:16 -0400 Subject: cache differences from ff to opera/ie Message-ID: <7227c6c71003180925i7edc539co2262d02e17913f02@mail.gmail.com> When I hit a basic .html page with firefox, I get another hit if I use chrome. If I use IE and Opera the first hit is always a miss then if i reload I get a hit. I'm using the following config. backend default { .host = "192.168.100.53"; .port = "80"; } sub vcl_recv { # Normalize Content-Encoding if (req.http.Accept-Encoding) { if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|lzma|tbz)(\?.*|)$") { remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { remove req.http.Accept-Encoding; } } # Remove cookies and query string for real static files if (req.url ~ "^/[^?]+\.(jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.*|)$") { unset req.http.cookie; set req.url = regsub(req.url, "\?.*$", ""); } # Remove cookies from front page if (req.url ~ "^/$") { unset req.http.cookie; } } sub vcl_fetch { if (req.url ~ "^/$") { unset obj.http.set-cookie; } } sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Cache = "Varnish: HIT " obj.hits; } else { set resp.http.X-Cache = "Varnish: MISS"; } } sub vcl_error { if (obj.status == 503 && req.restarts <= 5) { restart; } } Here is a varinishlog for opera 12 SessionOpen c 192.168.71.157 57733 :80 12 ReqStart c 192.168.71.157 57733 1009797700 12 RxRequest c GET 12 RxURL c /test.html 12 RxProtocol c HTTP/1.1 12 RxHeader c User-Agent: Opera/9.80 (X11; Linux i686; U; en) Presto/2.2.15 Version/10.10 12 RxHeader c Host: www.theopenskyproject.stg 12 RxHeader c Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1 12 RxHeader c Accept-Language: en-US,en;q=0.9 12 RxHeader c Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1 12 RxHeader c Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0 12 RxHeader c Cache-Control: no-cache 12 RxHeader c Connection: Keep-Alive, TE 12 RxHeader c TE: deflate, gzip, chunked, identity, trailers 12 VCL_call c recv 12 VCL_return c lookup 12 VCL_call c hash 12 VCL_return c hash 12 VCL_call c miss 12 VCL_return c fetch 13 BackendOpen b default 192.168.100.54 46963 192.168.100.53 80 12 Backend c 13 default default 13 TxRequest b GET 13 TxURL b /test.html 13 TxProtocol b HTTP/1.1 13 TxHeader b User-Agent: Opera/9.80 (X11; Linux i686; U; en) Presto/2.2.15 Version/10.10 13 TxHeader b Host: www.theopenskyproject.stg 13 TxHeader b Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1 13 TxHeader b Accept-Language: en-US,en;q=0.9 13 TxHeader b Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1 13 TxHeader b Accept-Encoding: gzip 13 TxHeader b X-Varnish: 1009797700 13 TxHeader b X-Forwarded-For: 192.168.71.157 13 RxProtocol b HTTP/1.1 13 RxStatus b 200 13 RxResponse b OK 13 RxHeader b Date: Thu, 18 Mar 2010 16:23:34 GMT 13 RxHeader b Server: Apache 13 RxHeader b Accept-Ranges: bytes 13 RxHeader b Cache-Control: max-age=7200 13 RxHeader b Expires: Thu, 18 Mar 2010 18:23:34 GMT 13 RxHeader b Content-Length: 14 13 RxHeader b Connection: close 13 RxHeader b Content-Type: text/html; charset=UTF-8 12 ObjProtocol c HTTP/1.1 12 ObjStatus c 200 12 ObjResponse c OK 12 ObjHeader c Date: Thu, 18 Mar 2010 16:23:34 GMT 12 ObjHeader c Server: Apache 12 ObjHeader c Cache-Control: max-age=7200 12 ObjHeader c Expires: Thu, 18 Mar 2010 18:23:34 GMT 12 ObjHeader c Content-Type: text/html; charset=UTF-8 13 BackendClose b default 12 TTL c 1009797700 RFC 7200 1268929414 0 0 7200 0 12 VCL_call c fetch 12 VCL_return c deliver 12 Length c 14 12 VCL_call c deliver 12 VCL_return c deliver 12 TxProtocol c HTTP/1.1 12 TxStatus c 200 12 TxResponse c OK 12 TxHeader c Server: Apache 12 TxHeader c Cache-Control: max-age=7200 12 TxHeader c Expires: Thu, 18 Mar 2010 18:23:34 GMT 12 TxHeader c Content-Type: text/html; charset=UTF-8 12 TxHeader c Content-Length: 14 12 TxHeader c Date: Thu, 18 Mar 2010 16:23:34 GMT 12 TxHeader c X-Varnish: 1009797700 12 TxHeader c Age: 0 12 TxHeader c Via: 1.1 varnish 12 TxHeader c Connection: keep-alive 12 TxHeader c X-Cache: Varnish: MISS 12 ReqEnd c 1009797700 1268929414.785295010 1268929414.787596941 0.000219107 0.002166986 0.000134945 12 ReqStart c 192.168.71.157 57733 1009797701 Here is a log from chrome 12 SessionOpen c 192.168.71.157 57742 :80 12 ReqStart c 192.168.71.157 57742 1009797702 12 RxRequest c GET 12 RxURL c /test.html 12 RxProtocol c HTTP/1.1 12 RxHeader c Host: www.theopenskyproject.stg 12 RxHeader c Connection: keep-alive 12 RxHeader c User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/532.9 (KHTML, like Gecko) Chrome/5.0.307.11 Safari/532.9 12 RxHeader c Cache-Control: max-age=0 12 RxHeader c Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 12 RxHeader c Accept-Encoding: gzip,deflate,sdch 12 RxHeader c Cookie: __utmz=193312050.1268919813.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); frontend=tmj6eo0u987s093h00gdd5t287; __utma=193312050.1928613275.1268919813.1268919813.1268919813.1; __utmc=193312050 12 RxHeader c Accept-Language: en-US,en;q=0.8 12 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 12 VCL_call c recv 12 VCL_return c lookup 12 VCL_call c hash 12 VCL_return c hash 12 Hit c 1009797700 12 VCL_call c hit 12 VCL_return c deliver 12 Length c 14 12 VCL_call c deliver 12 VCL_return c deliver 12 TxProtocol c HTTP/1.1 12 TxStatus c 200 12 TxResponse c OK 12 TxHeader c Server: Apache 12 TxHeader c Cache-Control: max-age=7200 12 TxHeader c Expires: Thu, 18 Mar 2010 18:23:34 GMT 12 TxHeader c Content-Type: text/html; charset=UTF-8 12 TxHeader c Content-Length: 14 12 TxHeader c Date: Thu, 18 Mar 2010 16:25:01 GMT 12 TxHeader c X-Varnish: 1009797702 1009797700 12 TxHeader c Age: 87 12 TxHeader c Via: 1.1 varnish 12 TxHeader c Connection: keep-alive 12 TxHeader c X-Cache: Varnish: HIT 1 12 ReqEnd c 1009797702 1268929501.792177916 1268929501.792538881 0.000161886 0.000236988 0.000123978 12 Debug c "herding" 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1268929502 1.0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From plfgoa at gmail.com Fri Mar 19 05:39:47 2010 From: plfgoa at gmail.com (Paras Fadte) Date: Fri, 19 Mar 2010 11:09:47 +0530 Subject: Future of ESI in Varnish Message-ID: <75cf5801003182239q7e6ba404jdbe9641168e5df39@mail.gmail.com> Hi , Wanted to know as to what is the future of development of ESI in varnish ? Is it something which will be continued and would include new features as time goes by ? Or is it something which will always have a very low priority in varnish project? Thank you. -Paras -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Mar 19 08:39:39 2010 From: perbu at varnish-software.com (Per Buer) Date: Fri, 19 Mar 2010 09:39:39 +0100 Subject: Future of ESI in Varnish In-Reply-To: <75cf5801003182239q7e6ba404jdbe9641168e5df39@mail.gmail.com> References: <75cf5801003182239q7e6ba404jdbe9641168e5df39@mail.gmail.com> Message-ID: <52220de01003190139n29990c92v3fcd75fe1b6ae0a@mail.gmail.com> Hi. On Fri, Mar 19, 2010 at 6:39 AM, Paras Fadte wrote: > > Wanted to know as to what is the future of development of ESI in varnish ? > Is it something which will be continued and would include new features as > time goes by ? Or is it something which will always have a very low priority > in varnish project? > I think expansion of ESI will be crucial in the years to come. Varnish already has an important role as a pure web accelerator. If the ESI implementation can be expanded to include some conditionals and cookie handling it will be a very powerfull tool for web development. So, ESI will be high on my agenda when persistence and gzip are more or less under control. We'll know more about this after the user group meeting. BTW: If someone feels up to the task of doing professional development on Varnish on a project-by-project basis, undertaking tasks as expanding the ESI support, please get in touch with me. -- Per Andreas Buer, CEO, Varnish Software AS Phone: +47 21 54 41 21 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertbrogers at gmail.com Fri Mar 19 09:35:19 2010 From: robertbrogers at gmail.com (Rob Rogers) Date: Fri, 19 Mar 2010 02:35:19 -0700 Subject: Making cachefu page purging work with vary headers : one solution nuke the Accept-Language <-- is this wise? In-Reply-To: References: Message-ID: <1BFF9E29-E299-481B-915E-5107D2DD8EAD@gmail.com> Thanks Laurence, It looks like purge() instead of purge_url() and bans are the trick. FYI, I did a purge("req.url == " req.url); wild card purges don't look necessary. i.e., == instead of ~ -rob On Mar 17, 2010, at 4:57 AM, Laurence Rowe wrote: > Take a look at my improved vcl for plone.recipe.varnish here: > > http://dev.plone.org/collective/browser/buildout/plone.recipe.varnish/branches/elro-better-vcl/plone/recipe/varnish/template.vcl > > While you do want to normalize Accept-Encoding (remember Zope only > does gzip) you still have to account for browsers which do not handle > gzip. As there is currently no way to invalidate varies, I use purge > to add it to the ban list. > > You should just set English as the only allowed language in > portal_languages to prevent internationalization of message strings. > > Laurence > > On 17 March 2010 10:21, Rob Rogers wrote: >> The issue: >> html type pages (think objects in zope) cached in varnish, were not >> purged >> successfully with zope/plone/cachefu purging methods. >> Images were purged. but html content was not. >> >> Given we depend on caching pages in accelerator/zope/plone this is a >> showstopper. (e.g. we use this: cache-in-proxy-1-hour for (most) >> html >> content) >> >> In order to make purging work, I had to normalize or unset a couple >> http.req >> attributes. This is what I came up with. >> >> The 'working' solution: >> sub vcl_recv { >> ... >> if (req.http.Accept-Encoding) { /* lifted from varnish site */ >> if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { >> # No point in compressing these >> remove req.http.Accept-Encoding; >> } elsif (req.http.Accept-Encoding ~ "gzip") { >> set req.http.Accept-Encoding = "gzip"; >> } elsif (req.http.Accept-Encoding ~ "deflate") { >> set req.http.Accept-Encoding = "deflate"; >> } else { >> # unkown algorithm >> remove req.http.Accept-Encoding; >> } >> } >> unset req.http.Accept-Language; /* is this DANGEROUS? */ >> unset req.http.user-agent; >> set req.http.host = "my.cashmoney.com"; >> ... >> } >> >> Based on what I have read, unsetting these attributes help keep >> multiple >> cache objects to a minimum. >> Which is the reasoning behind normalizing the Accept-Encoding per >> http://varnish-cache.org/wiki/FAQ/Compression >> >> So, I guess I have 2 questions: >> 1) Given the site is only served in english, is there a danger of >> serving up >> the wrong hit for a browser with a different Accept-Language? >> 2) Will I still get purge misses for objects cached by some >> browsers, like >> ones presenting deflate instead of gzip in the header? >> >> >> Thanks, >> >> Rob >> >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> From l at lrowe.co.uk Fri Mar 19 11:54:00 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Fri, 19 Mar 2010 11:54:00 +0000 Subject: Making cachefu page purging work with vary headers : one solution nuke the Accept-Language <-- is this wise? In-Reply-To: <1BFF9E29-E299-481B-915E-5107D2DD8EAD@gmail.com> References: <1BFF9E29-E299-481B-915E-5107D2DD8EAD@gmail.com> Message-ID: The use of purge_url wasn't the problem, more that it tried to do this in vcl_hit rather than vcl_recv. Laurence On 19 March 2010 09:35, Rob Rogers wrote: > Thanks Laurence, > > It looks like purge() instead of purge_url() and bans are the trick. > > FYI, I did a purge("req.url == " req.url); wild card purges don't look > necessary. i.e., == instead of ~ > > -rob > On Mar 17, 2010, at 4:57 AM, Laurence Rowe wrote: > >> Take a look at my improved vcl for plone.recipe.varnish here: >> >> >> http://dev.plone.org/collective/browser/buildout/plone.recipe.varnish/branches/elro-better-vcl/plone/recipe/varnish/template.vcl >> >> While you do want to normalize Accept-Encoding (remember Zope only >> does gzip) you still have to account for browsers which do not handle >> gzip. As there is currently no way to invalidate varies, I use purge >> to add it to the ban list. >> >> You should just set English as the only allowed language in >> portal_languages to prevent internationalization of message strings. >> >> Laurence >> >> On 17 March 2010 10:21, Rob Rogers wrote: >>> >>> The issue: >>> html type pages (think objects in zope) cached in varnish, were not >>> purged >>> successfully with zope/plone/cachefu purging methods. >>> Images were purged. but html content was not. >>> >>> Given we depend on caching pages in accelerator/zope/plone this is ?a >>> showstopper. (e.g. we use this: ?cache-in-proxy-1-hour for (most) html >>> content) >>> >>> In order to make purging work, I had to normalize or unset a couple >>> http.req >>> attributes. This is what I came up with. >>> >>> The 'working' solution: >>> sub vcl_recv { >>> ? ? ? ... >>> ? ? ? if (req.http.Accept-Encoding) { /* lifted from varnish site */ >>> ? ? ? ? if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { >>> ? ? ? ? ? ? ? ? ? ? ? # No point in compressing these >>> ? ? ? ? ? remove req.http.Accept-Encoding; >>> ? ? ? ? } elsif (req.http.Accept-Encoding ~ "gzip") { >>> ? ? ? ? ? set req.http.Accept-Encoding = "gzip"; >>> ? ? ? ? } elsif (req.http.Accept-Encoding ~ "deflate") { >>> ? ? ? ? ? set req.http.Accept-Encoding = "deflate"; >>> ? ? ? ? } else { >>> ? ? ? ? ? ? ? ? ? ? ? # unkown algorithm >>> ? ? ? ? ? remove req.http.Accept-Encoding; >>> ? ? ? ? } >>> ? ? ? } >>> ? ? ? unset req.http.Accept-Language; /* is this DANGEROUS? */ >>> ? ? ? unset req.http.user-agent; >>> ? ? ? set req.http.host = "my.cashmoney.com"; >>> ? ? ? ... >>> } >>> >>> Based on what I have read, unsetting these attributes help keep multiple >>> cache objects to a minimum. >>> Which is the reasoning behind normalizing the Accept-Encoding per >>> http://varnish-cache.org/wiki/FAQ/Compression >>> >>> So, I guess I have 2 questions: >>> 1) Given the site is only served in english, is there a danger of serving >>> up >>> the wrong hit for a browser with a different Accept-Language? >>> 2) Will I still get purge misses for objects ?cached by some browsers, >>> like >>> ones presenting deflate instead of gzip in the header? >>> >>> >>> Thanks, >>> >>> Rob >>> >>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >>> > > From justinp at newmediagateway.com Fri Mar 19 22:16:25 2010 From: justinp at newmediagateway.com (Justin Pasher) Date: Fri, 19 Mar 2010 17:16:25 -0500 Subject: Varnish child crashing on assert() starting with r4631 Message-ID: <4BA3F7B9.4080503@newmediagateway.com> Hello, I recently upgrade from trunk r4602 to trunk r4632 and now I'm starting to see a lot of panics in the syslog for varnish. I looked through the changelog and it appears in r4631, an assert() was added to bin/varnishd/cache_hash.c. Now the code seems to be hitting that assert and failing, thus crashing the child process. Varnish seems to keep running without problems, although the cache gets completely cleared out according to varnishstats. I'm not seeing a definitive pattern in when the assertion fails, but it's roughly every five minutes. The overall traffic to the site is not huge (probably only about 3000 page requests during that time right before it crashes). Some excerpts from syslog are below: Mar 19 16:51:19 wc2 tvan[13699]: Child (11787) died signal=6 Mar 19 16:51:19 wc2 tvan[13699]: Child (11787) Panic message: Assert error in HSH_DeleteObjHead(), cache_hash.c line 157:#012 Condition(VTAILQ_EMPTY(&oh->waitinglist)) not true.#012thread = (hcb_cleaner)#012ident = Linux,2.6.22.10-NMG,i686,-sfile,-hcritbit,epoll#012Backtrace:#012 0x806e382: pan_ic+d2#012 0x80663b7: HSH_DeleteObjHead+97#012 0x807c907: hcb_cleaner+c7#012 0xb7ef34c0: _end+afe36c94#012 0xb7e336de: _end+afd76eb2#012 Mar 19 16:51:19 wc2 tvan[13699]: Child cleanup complete Mar 19 16:51:19 wc2 tvan[13699]: child (12099) Started Mar 19 16:51:19 wc2 tvan[13699]: Child (12099) said Closed fds: 4 5 6 9 10 12 13 Mar 19 16:51:19 wc2 tvan[13699]: Child (12099) said Child starts Mar 19 16:51:19 wc2 tvan[13699]: Child (12099) said managed to mmap 1073741824 bytes of 1073741824 Mar 19 16:57:30 wc2 tvan[13699]: Child (12099) died signal=6 Mar 19 16:57:30 wc2 tvan[13699]: Child (12099) Panic message: Assert error in HSH_DeleteObjHead(), cache_hash.c line 157:#012 Condition(VTAILQ_EMPTY(&oh->waitinglist)) not true.#012thread = (hcb_cleaner)#012ident = Linux,2.6.22.10-NMG,i686,-sfile,-hcritbit,epoll#012Backtrace:#012 0x806e382: pan_ic+d2#012 0x80663b7: HSH_DeleteObjHead+97#012 0x807c907: hcb_cleaner+c7#012 0xb7ef34c0: _end+afe36c94#012 0xb7e336de: _end+afd76eb2#012 Mar 19 16:57:30 wc2 tvan[13699]: Child cleanup complete Mar 19 16:57:30 wc2 tvan[13699]: child (12372) Started Mar 19 16:57:30 wc2 tvan[13699]: Child (12372) said Closed fds: 4 5 6 9 10 12 13 Mar 19 16:57:30 wc2 tvan[13699]: Child (12372) said Child starts Mar 19 16:57:30 wc2 tvan[13699]: Child (12372) said managed to mmap 1073741824 bytes of 1073741824 Mar 19 17:03:43 wc2 tvan[13699]: Child (12372) died signal=6 Mar 19 17:03:43 wc2 tvan[13699]: Child (12372) Panic message: Assert error in HSH_DeleteObjHead(), cache_hash.c line 157:#012 Condition(VTAILQ_EMPTY(&oh->waitinglist)) not true.#012thread = (hcb_cleaner)#012ident = Linux,2.6.22.10-NMG,i686,-sfile,-hcritbit,epoll#012Backtrace:#012 0x806e382: pan_ic+d2#012 0x80663b7: HSH_DeleteObjHead+97#012 0x807c907: hcb_cleaner+c7#012 0xb7ef34c0: _end+afe36c94#012 0xb7e336de: _end+afd76eb2#012 Mar 19 17:03:43 wc2 tvan[13699]: Child cleanup complete Mar 19 17:03:43 wc2 tvan[13699]: child (12750) Started Mar 19 17:03:43 wc2 tvan[13699]: Child (12750) said Closed fds: 4 5 6 9 10 12 13 Mar 19 17:03:43 wc2 tvan[13699]: Child (12750) said Child starts Mar 19 17:03:43 wc2 tvan[13699]: Child (12750) said managed to mmap 1073741824 bytes of 1073741824 I actually have two separate varnish instances running off of the same binaries so that I can easily control them independently (and use different IP addresses). Both of them are acting the same. Any ideas on what I can do to troubleshoot this further? Justin Pasher From phk at phk.freebsd.dk Fri Mar 19 22:46:48 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 19 Mar 2010 22:46:48 +0000 Subject: Varnish child crashing on assert() starting with r4631 In-Reply-To: Your message of "Fri, 19 Mar 2010 17:16:25 EST." <4BA3F7B9.4080503@newmediagateway.com> Message-ID: <31643.1269038808@critter.freebsd.dk> In message <4BA3F7B9.4080503 at newmediagateway.com>, Justin Pasher writes: >I recently upgrade from trunk r4602 to trunk r4632 and now I'm starting >to see a lot of panics in the syslog for varnish. My fault, that assert is invalid, I have removed it in r4633. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From justinp at newmediagateway.com Fri Mar 19 22:47:23 2010 From: justinp at newmediagateway.com (Justin Pasher) Date: Fri, 19 Mar 2010 17:47:23 -0500 Subject: Moving varnish-cache.org tomorrow In-Reply-To: <87pr3c77qq.fsf@qurzaw.linpro.no> References: <4B964E12.1010200@redpill-linpro.com> <87pr3c77qq.fsf@qurzaw.linpro.no> Message-ID: <4BA3FEFB.8070101@newmediagateway.com> Tollef Fog Heen wrote: > | At the same time the mailing lists will change address to > | > | * varnish-announce at varnish-cache.org > | * varnish-bugs at varnish-cache.org > | * varnish-commit at varnish-cache.org > | * varnish-dev at varnish-cache.org > | * varnish-dist at varnish-cache.org > | * varnish-misc at varnish-cache.org > | * varnish-test at varnish-cache.org > > This is in effect now. Please note that you might need to update any > filters you have to ensure mails go into the right folder FYI, the new outgoing SMTP server (87.238.47.204) doesn't have proper reverse DNS setup. This was causing my mail server (and I'm sure many others) to tempfail any deliveries (a very common practice). I had to temporarily whitelist the IP to get new emails from the list. -- Justin Pasher From mahesho at rediff.co.in Mon Mar 22 11:56:01 2010 From: mahesho at rediff.co.in (Mahesh Ollalwar) Date: Mon, 22 Mar 2010 17:26:01 +0530 Subject: set obj.ttl to response header Message-ID: <4BA75AD1.2000108@rediff.co.in> An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Mon Mar 22 12:59:17 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 22 Mar 2010 13:59:17 +0100 Subject: set obj.ttl to response header In-Reply-To: <4BA75AD1.2000108@rediff.co.in> (Mahesh Ollalwar's message of "Mon, 22 Mar 2010 17:26:01 +0530") References: <4BA75AD1.2000108@rediff.co.in> Message-ID: <87bpeg5x7e.fsf@qurzaw.linpro.no> ]] Mahesh Ollalwar | Is it possible to set obj.ttl to the value getting from a backend | response header. My application is calculating the TTL value and | setting a header "upstream-ttl". I'm using below syntax in vcl_fetch | but it's not working. No, you need to set it using something which is a number. This will be possible in a later version. If you can make the backend just set the Cache control header properly and then override the header in vcl_fetch, that should do the right thing. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Mon Mar 22 13:00:29 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 22 Mar 2010 14:00:29 +0100 Subject: Moving varnish-cache.org tomorrow In-Reply-To: <4BA3FEFB.8070101@newmediagateway.com> (Justin Pasher's message of "Fri, 19 Mar 2010 17:47:23 -0500") References: <4B964E12.1010200@redpill-linpro.com> <87pr3c77qq.fsf@qurzaw.linpro.no> <4BA3FEFB.8070101@newmediagateway.com> Message-ID: <877hp45x5e.fsf@qurzaw.linpro.no> ]] Justin Pasher | FYI, the new outgoing SMTP server (87.238.47.204) doesn't have proper | reverse DNS setup. This was causing my mail server (and I'm sure many | others) to tempfail any deliveries (a very common practice). I had to | temporarily whitelist the IP to get new emails from the list. I am working on fixing this now, so mails should flow again soonish. Sorry about this. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Mon Mar 22 13:17:29 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 22 Mar 2010 14:17:29 +0100 Subject: cache differences from ff to opera/ie In-Reply-To: <7227c6c71003180925i7edc539co2262d02e17913f02@mail.gmail.com> (Mike's message of "Thu, 18 Mar 2010 12:25:16 -0400") References: <7227c6c71003180925i7edc539co2262d02e17913f02@mail.gmail.com> Message-ID: <8739zs5wd2.fsf@qurzaw.linpro.no> ]] Mike Hi, | When I hit a basic .html page with firefox, I get another hit if I use | chrome. If I use IE and Opera the first hit is always a miss then if i | reload I get a hit. [...] | 13 RxHeader b Date: Thu, 18 Mar 2010 16:23:34 GMT | 13 RxHeader b Expires: Thu, 18 Mar 2010 18:23:34 GMT So it expires the second you deliver it, making it uncacheable, if you look at the chrome log: | 12 TxHeader c Expires: Thu, 18 Mar 2010 18:23:34 GMT | 12 TxHeader c Date: Thu, 18 Mar 2010 16:25:01 GMT Is your backend doing something weird here? -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From mahesho at rediff.co.in Mon Mar 22 11:32:41 2010 From: mahesho at rediff.co.in (Mahesh Ollalwar) Date: Mon, 22 Mar 2010 17:02:41 +0530 Subject: set obj.ttl to response header Message-ID: <4BA75559.6080603@rediff.co.in> An HTML attachment was scrubbed... URL: From mahesho at rediff.co.in Tue Mar 23 10:07:26 2010 From: mahesho at rediff.co.in (Mahesh Ollalwar) Date: Tue, 23 Mar 2010 15:37:26 +0530 Subject: set obj.ttl to response header In-Reply-To: <87bpeg5x7e.fsf@qurzaw.linpro.no> References: <4BA75AD1.2000108@rediff.co.in> <87bpeg5x7e.fsf@qurzaw.linpro.no> Message-ID: <4BA892DE.1040404@rediff.co.in> An HTML attachment was scrubbed... URL: From plfgoa at gmail.com Wed Mar 24 07:09:01 2010 From: plfgoa at gmail.com (Paras Fadte) Date: Wed, 24 Mar 2010 12:39:01 +0530 Subject: Cache size Message-ID: <75cf5801003240009t314accefkcae4c22748b94e05@mail.gmail.com> Hi, Is it possible to determine in varnish in real time the cache size that is being used from the total allocated size ? Thank you. -Paras -------------- next part -------------- An HTML attachment was scrubbed... URL: From dfaulkner at pobox.com Wed Mar 24 20:46:24 2010 From: dfaulkner at pobox.com (Don Faulkner) Date: Wed, 24 Mar 2010 15:46:24 -0500 Subject: WebDAV through Varnish? Message-ID: <89238094-CE52-46CE-824F-B784616585E4@pobox.com> Hi all, I'm working on a caching and acceleration architecture for a site that implements WebDAV as well as the usual HTTP methods. I'm curious about how Varnish might act if it were used as the caching layer in front of those servers. I asked about this yesterday on the IRC channel, and was told that varnish passes thru all HTTP request methods except for GET, so it shouldn't have any effect, but I was also advised to ask the list, so here I am. :) If it matters, I'm looking at an architecture that looks something like this: pound (to terminate ssl) haproxy (maybe not necessary for what I'm doing?) varnish servers At least 2 tier (pound/haproxy/varnish -> servers), and maybe 3 tier (varnish on its own box). -- Don Faulkner, KB5WPM All that is gold does not glitter. Not all those who wander are lost. From slink at schokola.de Thu Mar 25 13:36:06 2010 From: slink at schokola.de (Nils Goroll) Date: Thu, 25 Mar 2010 14:36:06 +0100 Subject: Is it possible to simulate this rewrite in vcl? In-Reply-To: <11187.1254138707@critter.freebsd.dk> References: <11187.1254138707@critter.freebsd.dk> Message-ID: <4BAB66C6.4060709@schokola.de> Regarding this old thread: >> I've had to put an instance of apache behind varnish as a redirector as I >> can't think of a way of making varnish do it. Can this apache rewrite be >> done in vcl? >> >> RewriteRule ^/(.*)/home/(.*)$ http://$1_host.example.com/$2 [P] > > Not right of the bat. > > Provided you add backend instances for all the servers serving > these domains it is possible, but you need to do the $1 part as > a if/elseif/elseif/elseif/.../else to set the req.backend. Something I have just published recently might help: http://varnish-cache.org/ticket/666 I haven't tested Rob's specific case, but something like the following *SHOULD* work (don't pick on syntax errors please): if (req.url ~ "(.*)/home/(.*)") { set req.url = pmatch.2; if (pmatch.1 == "server1") { set req.backend = "server1"; } elseif (pmatch.1 == "server2") { ... } else { error 543 "wrong server" } } Please report bugs of the pmatch feature if you find any. Cheers, Nils From Jens.Schleusener at t-systems-sfr.com Mon Mar 29 10:42:41 2010 From: Jens.Schleusener at t-systems-sfr.com (Jens Schleusener) Date: Mon, 29 Mar 2010 12:42:41 +0200 (CEST) Subject: varnish v2.1: Where is "set obj.ttl" now allowed? Message-ID: Hi, as a varnish newbie I tried a very simple accelerator configuration: varnish v2.1 in front of a CMS backend that issues a "unnecessary" restrictive non-caching policy via the HTTP header like Cache-Control: private, must-revalidate, max-age=0 Vary: User-Agent To hold data nevertheless one minute in the varnish cache I tried to use the following kind of code (found similar in some examples and tutorials) sub vcl_fetch { if (obj.ttl < 60s) { set obj.ttl = 60s; } } But starting varnishd I always get the error Message from VCC-compiler: Variable 'obj.ttl' not accessible in method 'vcl_fetch'. At: (input Line 7 Pos 10) if (obj.ttl < 60s) { ---------#######--------- Running VCC-compiler failed, exit 1 VCL compilation failed http://varnish-cache.org/wiki/WikiStart#a2010-03-24Varnish2.1.0released shows a short summary of changes and the hint "see the changelog for a fuller list". But I couldn't find an according changelog (the file in the sources is "empty"). But I detected the fact that within the corresponding (?) testfile "bin/varnishtest/tests/v00016.vtc" the "obj.ttl" was changed to "beresp.ttl" (from v2.0.6 to v2.1). Is that possibly relevant to my problem? So my question: In which method the mentioned functionality (forcing caching of an not-cachable object for a short meaningful time) must be configured in varnish v2.1? And another currently not important question: Principally much of the backend pages are delivered in two versions (for user-agent IE and for user-agent Firefox incl. others). So an idea may be to have two specific varnish caches like /- varnish IE -\ Apache Backend CMS \- varnish FF -/ The Apache as "user-agent"-scheduler is just choosen since it could/should also do other tasks. Would that be meaningful or can varnish control that already itself? And a small request: Some basic complete VCL examples would make the start in the varnish world much easier ;-) Sorry, if my question are FAQs. Regards Jens From perbu at varnish-software.com Mon Mar 29 11:58:13 2010 From: perbu at varnish-software.com (Per Buer) Date: Mon, 29 Mar 2010 13:58:13 +0200 Subject: varnish v2.1: Where is "set obj.ttl" now allowed? In-Reply-To: References: Message-ID: <52220de01003290458x1db57a18ue580a7f853f03856@mail.gmail.com> On Mon, Mar 29, 2010 at 12:42 PM, Jens Schleusener: > ?sub vcl_fetch { > ? if (obj.ttl < 60s) { > ? ?set obj.ttl = 60s; > ? } > ?} sub vcl_fetch { set beresp.ttl = 60s; esi; } (forgot to Cc varnish-misc on the first post) -- Per Buer, Varnish Software Phone: +47 21 54 41 21 / Mobile: +47 958 39 117 / skype: per.buer From l at lrowe.co.uk Mon Mar 29 15:57:47 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Mon, 29 Mar 2010 16:57:47 +0100 Subject: varnish v2.1: Where is "set obj.ttl" now allowed? In-Reply-To: References: Message-ID: On 29 March 2010 11:42, Jens Schleusener wrote: > And another currently not important question: Principally much of the > backend pages are delivered in two versions (for user-agent IE and for > user-agent Firefox incl. others). So an idea may be to have two specific > varnish caches like > > ? ? ? /- varnish IE -\ > ?Apache ? ? ? ? ? ? ? ?Backend CMS > ? ? ? \- varnish FF -/ > > The Apache as "user-agent"-scheduler is just choosen since it could/should > also do other tasks. Would that be meaningful or can varnish control that > already itself? Your backend probably compresses responses and adds a "Vary: Accept-Encoding" header. As IE and FF have slightly different versions of this header, you need should normalise it in vcl_recv. See http://varnish-cache.org/wiki/FAQ/Compression Laurence From justinp at newmediagateway.com Tue Mar 30 15:02:19 2010 From: justinp at newmediagateway.com (Justin Pasher) Date: Tue, 30 Mar 2010 10:02:19 -0500 Subject: Tracking down sporadic 503 errors Message-ID: <4BB2127B.5050809@newmediagateway.com> Hello, I'm currently running Varnish r4633 (I can upgrade if absolutely necessary), and we've been receiving very sporadic 503 errors listed in the log files generated by varnishncsa. It's a very small percentage, but nonetheless, when it happens on an important page, it's noticeable. Stats from yesterday's log file show 116 "503" errors out of about 4.1 million hits. About 80% of the failed requests are POST requests, which it setup in my VCL as a "pass through". If I look in the apache logs (the backend server), I only see one 503 error returned by apache itself, so maybe there's a timeout issue somewhere. I'm trying to figure out the best way to troubleshoot this, since it's too inconsistent to just sit watching the output of varnishlog. Perhaps it's hitting the default value for between_bytes_timeout (60 seconds)? If processing the data on the backend takes too long, then varnish would time out after 60 seconds of no data, even if the backend is still churning, right? I guess the question is what situations cause Varnish to return a 503 aside from when the backend itself returns a 503. I can post details of my VCL if needed, but it's pretty simple (mostly taken from examples on the site). -- Justin Pasher From vlists at veus.hr Tue Mar 30 18:32:59 2010 From: vlists at veus.hr (Vladimir) Date: Tue, 30 Mar 2010 14:32:59 -0400 (EDT) Subject: Tracking down sporadic 503 errors In-Reply-To: <4BB2127B.5050809@newmediagateway.com> References: <4BB2127B.5050809@newmediagateway.com> Message-ID: This is what I have seen as well. I could not pin it down. What made it somewhat better is adding if (obj.status == 503 && req.restarts < 4) { restart; } under vcl_error subroutine. It will rerequest the document however even with that behavior is still happening though much less :-(. I have attached some graphs of 500 responses from varnish and corresponding apache responses. Units are hits per second. I even looked at corresponding responses from Apache and Apache would claim that the request succeeded yet varnish would throw a 500. Vladimir On Tue, 30 Mar 2010, Justin Pasher wrote: > I'm currently running Varnish r4633 (I can upgrade if absolutely necessary), > and we've been receiving very sporadic 503 errors listed in the log files > generated by varnishncsa. It's a very small percentage, but nonetheless, when > it happens on an important page, it's noticeable. > > Stats from yesterday's log file show 116 "503" errors out of about 4.1 > million hits. About 80% of the failed requests are POST requests, which it > setup in my VCL as a "pass through". If I look in the apache logs (the > backend server), I only see one 503 error returned by apache itself, so maybe > there's a timeout issue somewhere. I'm trying to figure out the best way to > troubleshoot this, since it's too inconsistent to just sit watching the > output of varnishlog. > > Perhaps it's hitting the default value for between_bytes_timeout (60 > seconds)? If processing the data on the backend takes too long, then varnish > would time out after 60 seconds of no data, even if the backend is still > churning, right? I guess the question is what situations cause Varnish to > return a 503 aside from when the backend itself returns a 503. > > I can post details of my VCL if needed, but it's pretty simple (mostly taken > from examples on the site). -------------- next part -------------- A non-text attachment was scrubbed... Name: apache_500.png Type: image/png Size: 9783 bytes Desc: URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish_500.png Type: image/png Size: 11736 bytes Desc: URL: From jfrias at gmail.com Tue Mar 30 19:13:02 2010 From: jfrias at gmail.com (Javier Frias) Date: Tue, 30 Mar 2010 15:13:02 -0400 Subject: Tracking down sporadic 503 errors In-Reply-To: References: <4BB2127B.5050809@newmediagateway.com> Message-ID: <22964b961003301213x329b94e8q43f6c3bf339dfcdb@mail.gmail.com> Do you have a back-end test, if so, have checked the varnish logs for back-end failed messages? I'm not sure what the varnish timeout may be, but you could try reading the times of the 503's in the varnishlog, if they all look to be about the same, you may be hitting a time out issue. ( if you log times in apache, I would try to check those as well ). -Javier On Tue, Mar 30, 2010 at 2:32 PM, Vladimir wrote: > This is what I have seen as well. I could not pin it down. What made it > somewhat better is adding > > ? ?if (obj.status == 503 && req.restarts < 4) { > ? ? ? ? ? ? ? ?restart; > ? ?} > > under vcl_error subroutine. It will rerequest the document however even with > that behavior is still happening though much less :-(. I have attached some > graphs of 500 responses from varnish and corresponding apache responses. > Units are hits per second. > > I even looked at corresponding responses from Apache and Apache would claim > that the request succeeded yet varnish would throw a 500. > > Vladimir > > On Tue, 30 Mar 2010, Justin Pasher wrote: > >> I'm currently running Varnish r4633 (I can upgrade if absolutely >> necessary), and we've been receiving very sporadic 503 errors listed in the >> log files generated by varnishncsa. It's a very small percentage, but >> nonetheless, when it happens on an important page, it's noticeable. >> >> Stats from yesterday's log file show 116 "503" errors out of about 4.1 >> million hits. About 80% of the failed requests are POST requests, which it >> setup in my VCL as a "pass through". If I look in the apache logs (the >> backend server), I only see one 503 error returned by apache itself, so >> maybe there's a timeout issue somewhere. I'm trying to figure out the best >> way to troubleshoot this, since it's too inconsistent to just sit watching >> the output of varnishlog. >> >> Perhaps it's hitting the default value for between_bytes_timeout (60 >> seconds)? If processing the data on the backend takes too long, then varnish >> would time out after 60 seconds of no data, even if the backend is still >> churning, right? I guess the question is what situations cause Varnish to >> return a 503 aside from when the backend itself returns a 503. >> >> I can post details of my VCL if needed, but it's pretty simple (mostly >> taken from examples on the site). > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From torrance123 at gmail.com Tue Mar 30 19:20:25 2010 From: torrance123 at gmail.com (Torrance) Date: Wed, 31 Mar 2010 08:20:25 +1300 Subject: Tracking down sporadic 503 errors In-Reply-To: <4BB2127B.5050809@newmediagateway.com> References: <4BB2127B.5050809@newmediagateway.com> Message-ID: <4BB24EF9.40706@gmail.com> Hi Justin, I had a similar problem with receiving fairly consistent 503 errors on POST requests. In my case, the 503 errors were returning generally in less than a second, despite my having set the various timeout limits to be substantially higher. If it's the same problem, then it's possibly not a timeout problem. I wrote to the list earlier this year and posted some logs [1], but was asked to post a tcpdump of all communications for further investigation which was (is) beyond my skills. I hope you have more success resolving this, Torrace [1] http://lists.varnish-cache.org/pipermail/varnish-misc/2010-February/003799.html On 31/03/10 4:02 AM, Justin Pasher wrote: > Hello, > > I'm currently running Varnish r4633 (I can upgrade if absolutely > necessary), and we've been receiving very sporadic 503 errors listed > in the log files generated by varnishncsa. It's a very small > percentage, but nonetheless, when it happens on an important page, > it's noticeable. > > Stats from yesterday's log file show 116 "503" errors out of about 4.1 > million hits. About 80% of the failed requests are POST requests, > which it setup in my VCL as a "pass through". If I look in the apache > logs (the backend server), I only see one 503 error returned by apache > itself, so maybe there's a timeout issue somewhere. I'm trying to > figure out the best way to troubleshoot this, since it's too > inconsistent to just sit watching the output of varnishlog. > > Perhaps it's hitting the default value for between_bytes_timeout (60 > seconds)? If processing the data on the backend takes too long, then > varnish would time out after 60 seconds of no data, even if the > backend is still churning, right? I guess the question is what > situations cause Varnish to return a 503 aside from when the backend > itself returns a 503. > > I can post details of my VCL if needed, but it's pretty simple (mostly > taken from examples on the site). > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 905 bytes Desc: OpenPGP digital signature URL: From michal.taborsky at nrholding.com Tue Mar 30 19:55:46 2010 From: michal.taborsky at nrholding.com (Michal Taborsky - Netretail Holding) Date: Tue, 30 Mar 2010 21:55:46 +0200 Subject: Tracking down sporadic 503 errors In-Reply-To: <4BB2127B.5050809@newmediagateway.com> References: <4BB2127B.5050809@newmediagateway.com> Message-ID: <4BB25742.9040509@nrholding.com> Hi Justin, not that I know anything about your problem, but we've seen this kind of behavior, when our Apaches were dying while processing the request. The apache log would not show any 503 (because there is no-one to log it), but Varnish was returning 503. In this case, you should see something like Segmentation fault in your error log on the backend (but I guess that depends on your platform). Hope this helps. Michal Dne 30.3.2010 17:02, Justin Pasher napsal(a): > Hello, > > I'm currently running Varnish r4633 (I can upgrade if absolutely > necessary), and we've been receiving very sporadic 503 errors listed > in the log files generated by varnishncsa. It's a very small > percentage, but nonetheless, when it happens on an important page, > it's noticeable. > > Stats from yesterday's log file show 116 "503" errors out of about 4.1 > million hits. About 80% of the failed requests are POST requests, > which it setup in my VCL as a "pass through". If I look in the apache > logs (the backend server), I only see one 503 error returned by apache > itself, so maybe there's a timeout issue somewhere. I'm trying to > figure out the best way to troubleshoot this, since it's too > inconsistent to just sit watching the output of varnishlog. > > Perhaps it's hitting the default value for between_bytes_timeout (60 > seconds)? If processing the data on the backend takes too long, then > varnish would time out after 60 seconds of no data, even if the > backend is still churning, right? I guess the question is what > situations cause Varnish to return a 503 aside from when the backend > itself returns a 503. > > I can post details of my VCL if needed, but it's pretty simple (mostly > taken from examples on the site). > -- Michal T?borsk? chief systems architect Netretail Holding, B.V. http://www.nrholding.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From David at firechaser.com Wed Mar 31 11:48:09 2010 From: David at firechaser.com (David Murphy) Date: Wed, 31 Mar 2010 12:48:09 +0100 Subject: Varnish 2.1 RPM form Redhat Message-ID: <345BD8B3F8775748A4676A625EADA22417D87665@DURAN.firechaser.local> Hello Could someone kindly point me to the Varnish 2.1 rpm for RHEL5, 64bit please? Thanks, David From vlists at veus.hr Wed Mar 31 12:06:08 2010 From: vlists at veus.hr (Vladimir) Date: Wed, 31 Mar 2010 08:06:08 -0400 (EDT) Subject: Varnish 2.1 RPM form Redhat In-Reply-To: <345BD8B3F8775748A4676A625EADA22417D87665@DURAN.firechaser.local> References: <345BD8B3F8775748A4676A625EADA22417D87665@DURAN.firechaser.local> Message-ID: I just built those myself. All you need to do is put varnish-2.1.tar.gz is install rpm-build package then put varnish-2.1.tar.gz in /usr/src/redhat/SOURCES/ then tar -xvf varnish-2.1.tar.gz varnish-2.1/redhat/varnish.spec edit the varnish.spec file to say 2.1 instead of 2.0.7. Then rpmbuild -bb varnish-2.1/redhat/varnish.spec That should build the packages for you. One note is that for whatever reason make check fails on any machine I have tried this on so only way I could get it going was to take out everything from %check to %install and rebuild. Anyone else encounter this issue ? On Wed, 31 Mar 2010, David Murphy wrote: > Could someone kindly point me to the Varnish 2.1 rpm for RHEL5, 64bit please? From David at firechaser.com Wed Mar 31 12:15:14 2010 From: David at firechaser.com (David Murphy) Date: Wed, 31 Mar 2010 13:15:14 +0100 Subject: Varnish 2.1 RPM form Redhat In-Reply-To: References: <345BD8B3F8775748A4676A625EADA22417D87665@DURAN.firechaser.local>, Message-ID: <345BD8B3F8775748A4676A625EADA22417D87666@DURAN.firechaser.local> Thanks Vladimir Have you had any performance issues/errors with your new RPMs when tested under load? Just curious, seeing that you had a build issue when making the packages. Best, David ________________________________________ From: Vladimir [vlists at veus.hr] Sent: 31 March 2010 13:06 To: David Murphy Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish 2.1 RPM form Redhat I just built those myself. All you need to do is put varnish-2.1.tar.gz is install rpm-build package then put varnish-2.1.tar.gz in /usr/src/redhat/SOURCES/ then tar -xvf varnish-2.1.tar.gz varnish-2.1/redhat/varnish.spec edit the varnish.spec file to say 2.1 instead of 2.0.7. Then rpmbuild -bb varnish-2.1/redhat/varnish.spec That should build the packages for you. One note is that for whatever reason make check fails on any machine I have tried this on so only way I could get it going was to take out everything from %check to %install and rebuild. Anyone else encounter this issue ? On Wed, 31 Mar 2010, David Murphy wrote: > Could someone kindly point me to the Varnish 2.1 rpm for RHEL5, 64bit please? From cosimo at streppone.it Wed Mar 31 16:20:38 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Wed, 31 Mar 2010 18:20:38 +0200 Subject: When does a session close with a "blast"? Message-ID: We're running varnish 2.0.4 on some moderately busy servers, ~1200 req/s peak. Today I was looking at the output of: varnishtop -i SessionClose which is very interesting BTW, when I noticed that a new value was suddenly added to the list, that looked like this, more or less: 7557.34 SessionClose timeout 901.28 SessionClose no request 753.54 SessionClose remote closed 317.48 SessionClose error 73.18 SessionClose Connection: close 60.32 SessionClose overflow 20.81 SessionClose not HTTP/1.1 0.05 SessionClose blast <--- HERE! So, questions: 1) Does this distribution look "normal"? These servers have a sess_timeout of 2s, and they are serving a large (tens of millions) set of medium to large files. 2) What does "blast" mean for a SessionClose? Reading the source code for 2.0.4, it would seem to me that "blast" originates from overflowing sess_workspace. If this is correct, would a simple increase of sess_workspace prevent blasts? And yes, I'm assuming a blast is a bad thing. :) Thanks, -- Cosimo From mrbits.dcf at gmail.com Wed Mar 31 16:38:28 2010 From: mrbits.dcf at gmail.com (MrBiTs) Date: Wed, 31 Mar 2010 13:38:28 -0300 Subject: Varnish over Apache mod_vhost_alias Message-ID: <4BB37A84.3010805@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi, all We're running varnish 2.0.6 with an Apache 2.2 with mod_vhost_alias configured to read FQDN as root directory. If I point my browser to test.dom.com (my valid vhost, so Apache will check /var/www/test.dom.com directory, that exists), Varnish delivers my page pretty well. If I pointo to anotherurl.dom.com (a vhost that doesn't have associated directory (/var/www/anotherurl.dom.com) created) Varnish is delivering the same page I have at test.dom.com. Does anybody is working with Apache and mod_vhost_alias under Varnish that can help me with this issue ? Thanks a lot .0. MrBiTs - mrbits.dcf at gmail.com ..0 GnuPG - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x6EC818FC2B3CA5AB 000 http://www.mrbits.com.br -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iQEcBAEBCAAGBQJLs3qEAAoJEG7IGPwrPKWrs54H/24Gv+zgGDshYrpwH4MOwEoS FlnaeOzeAyq4O/Pfb4x6i1Z8T8e4z9aVMYVrTdR31ATjGHh7mvmdDlsw8Tf7JypW oelPHNG3BuBbR9qm+y5m0x44cI50By+UZXAiuuNc29PN5bmzTpPnM6EjvLsk8IC/ 45Z1d3FwoGadmJX9N8vajDqWcwSPSttNJVy6YxYtCdAl9+UABE3DVlDjLzb5pCrN niytHwNr1uXsYAXFdeGrqw8RJrUKnBombzo+X9wMvbMkuUkJ3YQToXzLQqRTYHMH 36d8pX/3eiIsXkjZDkRC7Q06uSRhTBYm3DoPm0dpAiR6AZdvYiOTVPNXQcnaT5E= =QRZs -----END PGP SIGNATURE----- From jay.janssen at gmail.com Wed Mar 24 02:06:15 2010 From: jay.janssen at gmail.com (Jay Janssen) Date: Wed, 24 Mar 2010 02:06:15 -0000 Subject: varnish linux memory usage Message-ID: <565896c51003231905i10656c52td698096fd7d5afa3@mail.gmail.com> Hi all, I'm using varnish as a reverse proxy frontend on my VPS webservers in front of apache. I have a relatively small cache size of 100M on disk which seems fine for my needs. However, I noticed in 'top' that Linux thinks my main varnish process is using 332M of swap. If I increase the thread_pool_min from 5 (where it is now) up to 100, I see that swap number go up to 1.8G! Is that just the kernel thinking that each thread has its own copy of the cache file open? Is it anything to be concerned about? Jay Janssen -------------- next part -------------- An HTML attachment was scrubbed... URL: From hatesbananas at googlemail.com Mon Mar 29 13:34:24 2010 From: hatesbananas at googlemail.com (Sam Bnn) Date: Mon, 29 Mar 2010 13:34:24 -0000 Subject: Sane values for ban_lurker_sleep Message-ID: <46205a9e1003290626g2218cdm9bf6f80446a723bd@mail.gmail.com> Hi, I'd like to test-drive the new ban_lurker_sleep param, but the man page is rather thin on details here. Even after reading the code, I'm not quite sure what the lurker thread does exactly between the sleeps. Is it (a) fetching the last ban and testing all objects which haven't yet been tested against it? Looking at the while-loop in http://varnish-cache.org/browser/trunk/varnish-cache/bin/varnishd/cache_ban.c#L521 it seems to me that it (b) always just checks one object at a time, which would be rather pointless for a cache with a million objects and the sleep-param set to >1 second. So, what does it really do, and can I expect this new feature to help me given a cache with >1M objects and a purge list which regularly grows over 250 entries. Cheers, Sam -------------- next part -------------- An HTML attachment was scrubbed... URL: From schoefmann at googlemail.com Tue Mar 30 15:52:20 2010 From: schoefmann at googlemail.com (=?ISO-8859-1?Q?Maximilian_Sch=F6fmann?=) Date: Tue, 30 Mar 2010 15:52:20 -0000 Subject: Testing -s persistent, getting "Out of space in persistent silo" Message-ID: Hi all, I'm currently testing 2.1 with -s persistent,/var/lib/varnish/$INSTANCE/cache.bin,14G (on Ubuntu) After a while, I start seeing these every few minutes varnishd[16181]: Child (7977) said Out of space in persistent silo varnishd[16181]: Child (7977) said Committing suicide, restart will make space with the effect of the cache being empty after the child restarts (it was far from filled before these crashes...). Best, Max