From l at lrowe.co.uk Thu May 7 12:21:26 2009 From: l at lrowe.co.uk (Laurence Rowe) Date: Thu, 7 May 2009 14:21:26 +0200 Subject: PostTwoShoppingList / Feature requests Message-ID: Hi, Just a couple of comments on the feature requests Instead of directly supporting X-Sendfile, I wander if this could be supported by passing an argument string from restart, e.g. << restart "/some/new/url"; >> The argument would then be accessible from the restarted request object, e.g: << set req.url = req.restart.data; >> This would make it possible to support other similar schemes such as Nginx's X-Accel-Redirect (http://wiki.nginx.org/NginxXSendfile). File upload buffering. Great! It's a vital feature for threaded app servers. The ability to parse a request body and save files to a temporary directory is also handy, e.g. http://www.grid.net.ru/nginx/upload.en.html VCL cookie handling. Great! I'd also like to see an option to use PCRE for regular expressions, in certain situations (parsing cookies or other complex headers) the ability to specify conditional subpatterns is handy, and they are familiar for Python and PHP programmers. It may also offer some performance benefits according to this analysis: http://www.boost.org/doc/libs/1_38_0/libs/regex/doc/gcc-performance.html. Perhaps more for some time in the future, I could imagine that it would be useful to support dependencies between cached objects. If more complex transforms such as xslt were ever to be supported, it could be nice to also cache the result of the transform, but you'd want to be able to invalidate this document when any of the input documents changed. Maybe it could also be useful for caching gzip variants if gzipping was moved into varnish. mod_depends (http://svn.outoforder.cc/chora/?rt=mod_depends) was written for this use with mod_transform and may have some interesting ideas. Laurence From aotto at mosso.com Fri May 8 17:35:54 2009 From: aotto at mosso.com (Adrian Otto) Date: Fri, 8 May 2009 10:35:54 -0700 Subject: Regression test failure on v2.0.4 Message-ID: Varnish Devs, I'm getting intermittent failures with regression test 387 when running "make check" against 2.0.4. The excerpt-1 below is from the "make check" output, and the excerpt-2 and excerpt-3 are the verbose output of the same test with failed, and passed results respectively (one run right after the other against the same source code and binary). You'll notice that the ordering of this event: #### s1 send| 00000000\r\n ...is different between the two. It seems that in the failed result, the client begins to act upon the response before the server finishes sending it. Regards, Adrian excerpt-1 (from make test): # top TEST ././tests/r00387.vtc starting # TEST Regression test for #387: too long chunk header ## s1 Starting server ### s1 listen on 127.0.0.1:9080 (fd 3) ## s1 Started on 127.0.0.1:9080 ## v1 Launch ### v1 CMD: cd ../varnishd && ./varnishd -d -d -n /tmp/__v1 -a '127.0.0.1:9081' -T 127.0.0.1:9001 -P /tmp/__v1/varnishd.pid ### v1 opening CLI connection ### v1 CLI connection fd = 5 ### v1 CLI STATUS 200 ### v1 CLI STATUS 200 ## v1 Start ### v1 CLI STATUS 200 ### v1 CLI STATUS 200 ## c1 Starting client ## c1 Started ### c1 Connect to 127.0.0.1:9081 ### c1 Connected to 127.0.0.1:9081 fd is 9 ### c1 rxresp ## c1 Waiting for client ### s1 Accepted socket fd is 4 ### s1 rxreq ### c1 Closing fd 9 ## c1 Ending FAIL: ./tests/r00387.vtc excerpt-2 (failed varnishtest -v): $ ./varnishtest -v tests/r00387.vtc # top TEST tests/r00387.vtc starting # TEST Regression test for #387: too long chunk header ## s1 Starting server ### s1 listen on 127.0.0.1:9080 (fd 3) ## s1 Started on 127.0.0.1:9080 ## v1 Launch ### v1 CMD: cd ../varnishd && ./varnishd -d -d -n /tmp/__v1 -a '127.0.0.1:9081' -T 127.0.0.1:9001 -P /tmp/__v1/varnishd.pid ### v1 opening CLI connection #### v1 debug| storage_file: filename: ./varnish.JfeL0S (unlinked) size 17681 MB.\n #### v1 debug| Using old SHMFILE\n #### v1 debug| Debugging mode, enter "start" to start child\n ### v1 CLI connection fd = 5 #### v1 CLI TX| vcl.inline vcl1 "backend s1 { .host = \"127.0.0.1 \"; .port = \"9080\"; }\n" #### v1 CLI RX| VCL compiled. ### v1 CLI STATUS 200 #### v1 CLI TX| vcl.use vcl1 ### v1 CLI STATUS 200 ## v1 Start #### v1 CLI TX| start #### v1 debug| child (16909) Started\n #### v1 debug| Child (16909) said Closed fds: 3 5 7 8 11 12 14 15\n ### v1 CLI STATUS 200 #### v1 CLI TX| debug.xid 1000 #### v1 debug| Child (16909) said Child starts\n #### v1 debug| Child (16909) said managed to mmap 18540429312 bytes of 18540429312\n #### v1 debug| Child (16909) said Ready\n #### v1 CLI RX| XID is 1000 ### v1 CLI STATUS 200 ## c1 Starting client ## c1 Started ### c1 Connect to 127.0.0.1:9081 ### c1 Connected to 127.0.0.1:9081 fd is 9 #### c1 txreq| GET / HTTP/1.1\r\n #### c1 txreq| \r\n ### c1 rxresp ## c1 Waiting for client ### s1 Accepted socket fd is 4 ### s1 rxreq #### s1 rxhdr| GET / HTTP/1.1\r\n #### s1 rxhdr| X-Varnish: 1001\r\n #### s1 rxhdr| X-Forwarded-For: 127.0.0.1\r\n #### s1 rxhdr| Host: 127.0.0.1\r\n #### s1 rxhdr| \r\n #### s1 http[ 0] | GET #### s1 http[ 1] | / #### s1 http[ 2] | HTTP/1.1 #### s1 http[ 3] | X-Varnish: 1001 #### s1 http[ 4] | X-Forwarded-For: 127.0.0.1 #### s1 http[ 5] | Host: 127.0.0.1 #### s1 bodylen = 0 #### s1 send| HTTP/1.1 200 Ok\r\n #### s1 send| Transfer-encoding: chunked\r\n #### s1 send| \r\n #### s1 send| 004\r\n #### s1 send| 1234\r\n #### s1 send| 000000000000000000001\r\n #### s1 send| @\r\n #### c1 rxhdr| HTTP/1.1 503 Service Unavailable\r\n #### c1 rxhdr| Server: Varnish\r\n #### c1 rxhdr| Retry-After: 0\r\n #### c1 rxhdr| Content-Type: text/html; charset=utf-8\r\n #### c1 rxhdr| Content-Length: 460\r\n #### c1 rxhdr| Date: Fri, 08 May 2009 16:52:19 GMT\r\n #### c1 rxhdr| X-Varnish: 1001\r\n #### c1 rxhdr| Age: 0\r\n #### c1 rxhdr| Via: 1.1 varnish\r\n #### c1 rxhdr| Connection: close\r\n #### c1 rxhdr| \r\n #### c1 http[ 0] | HTTP/1.1 #### c1 http[ 1] | 503 #### c1 http[ 2] | Service Unavailable #### c1 http[ 3] | Server: Varnish #### c1 http[ 4] | Retry-After: 0 #### c1 http[ 5] | Content-Type: text/html; charset=utf-8 #### c1 http[ 6] | Content-Length: 460 #### c1 http[ 7] | Date: Fri, 08 May 2009 16:52:19 GMT #### c1 http[ 8] | X-Varnish: 1001 #### c1 http[ 9] | Age: 0 #### c1 http[10] | Via: 1.1 varnish #### c1 http[11] | Connection: close #### c1 body| \n #### c1 body| \n #### c1 body| \n #### c1 body| \n #### c1 body| \n #### c1 body| 503 Service Unavailable\n #### c1 body| \n #### c1 body| \n #### c1 body|

Error 503 Service Unavailable

\n #### c1 body|

Service Unavailable

\n #### c1 body|

Guru Meditation:

\n #### c1 body|

XID: 1001

\n #### c1 body|
\n #### c1 body| Varnish\n #### c1 body|
\n #### c1 body| \n #### c1 body| \n #### c1 bodylen = 460 #### c1 EXPECT resp.status (503) == 503 (503) match ### c1 Closing fd 9 ## c1 Ending #### s1 send| 00000000\r\n Assert error in cmd_http_send(), vtc_http.c line 710: Condition(i == strlen(av[1])) not true. errno = 104 (Connection reset by peer) Aborted excerpt-3 (passed varnishtest -v): $ ./varnishtest -v tests/r00387.vtc # top TEST tests/r00387.vtc starting # TEST Regression test for #387: too long chunk header ## s1 Starting server ### s1 listen on 127.0.0.1:9080 (fd 3) ## s1 Started on 127.0.0.1:9080 ## v1 Launch ### v1 CMD: cd ../varnishd && ./varnishd -d -d -n /tmp/__v1 -a '127.0.0.1:9081' -T 127.0.0.1:9001 -P /tmp/__v1/varnishd.pid ### v1 opening CLI connection #### v1 debug| storage_file: filename: ./varnish.H4chHT (unlinked) size 17681 MB.\n #### v1 debug| Using old SHMFILE\n #### v1 debug| Debugging mode, enter "start" to start child\n ### v1 CLI connection fd = 5 #### v1 CLI TX| vcl.inline vcl1 "backend s1 { .host = \"127.0.0.1 \"; .port = \"9080\"; }\n" #### v1 CLI RX| VCL compiled. ### v1 CLI STATUS 200 #### v1 CLI TX| vcl.use vcl1 ### v1 CLI STATUS 200 ## v1 Start #### v1 CLI TX| start #### v1 debug| child (16970) Started\n ### v1 CLI STATUS 200 #### v1 CLI TX| debug.xid 1000 #### v1 debug| Child (16970) said Closed fds: 3 5 7 8 11 12 14 15\n #### v1 debug| Child (16970) said Child starts\n #### v1 debug| Child (16970) said managed to mmap 18540429312 bytes of 18540429312\n #### v1 debug| Child (16970) said Ready\n #### v1 CLI RX| XID is 1000 ### v1 CLI STATUS 200 ## c1 Starting client ## c1 Started ### c1 Connect to 127.0.0.1:9081 ### c1 Connected to 127.0.0.1:9081 fd is 9 #### c1 txreq| GET / HTTP/1.1\r\n #### c1 txreq| \r\n ### c1 rxresp ## c1 Waiting for client ### s1 Accepted socket fd is 4 ### s1 rxreq #### s1 rxhdr| GET / HTTP/1.1\r\n #### s1 rxhdr| X-Varnish: 1001\r\n #### s1 rxhdr| X-Forwarded-For: 127.0.0.1\r\n #### s1 rxhdr| Host: 127.0.0.1\r\n #### s1 rxhdr| \r\n #### s1 http[ 0] | GET #### s1 http[ 1] | / #### s1 http[ 2] | HTTP/1.1 #### s1 http[ 3] | X-Varnish: 1001 #### s1 http[ 4] | X-Forwarded-For: 127.0.0.1 #### s1 http[ 5] | Host: 127.0.0.1 #### s1 bodylen = 0 #### s1 send| HTTP/1.1 200 Ok\r\n #### s1 send| Transfer-encoding: chunked\r\n #### s1 send| \r\n #### s1 send| 004\r\n #### s1 send| 1234\r\n #### s1 send| 000000000000000000001\r\n #### s1 send| @\r\n #### s1 send| 00000000\r\n ### s1 shutting fd 4 ## s1 Ending #### c1 rxhdr| HTTP/1.1 503 Service Unavailable\r\n #### c1 rxhdr| Server: Varnish\r\n #### c1 rxhdr| Retry-After: 0\r\n #### c1 rxhdr| Content-Type: text/html; charset=utf-8\r\n #### c1 rxhdr| Content-Length: 460\r\n #### c1 rxhdr| Date: Fri, 08 May 2009 16:53:31 GMT\r\n #### c1 rxhdr| X-Varnish: 1001\r\n #### c1 rxhdr| Age: 0\r\n #### c1 rxhdr| Via: 1.1 varnish\r\n #### c1 rxhdr| Connection: close\r\n #### c1 rxhdr| \r\n #### c1 http[ 0] | HTTP/1.1 #### c1 http[ 1] | 503 #### c1 http[ 2] | Service Unavailable #### c1 http[ 3] | Server: Varnish #### c1 http[ 4] | Retry-After: 0 #### c1 http[ 5] | Content-Type: text/html; charset=utf-8 #### c1 http[ 6] | Content-Length: 460 #### c1 http[ 7] | Date: Fri, 08 May 2009 16:53:31 GMT #### c1 http[ 8] | X-Varnish: 1001 #### c1 http[ 9] | Age: 0 #### c1 http[10] | Via: 1.1 varnish #### c1 http[11] | Connection: close #### c1 body| \n #### c1 body| \n #### c1 body| \n #### c1 body| \n #### c1 body| \n #### c1 body| 503 Service Unavailable\n #### c1 body| \n #### c1 body| \n #### c1 body|

Error 503 Service Unavailable

\n #### c1 body|

Service Unavailable

\n #### c1 body|

Guru Meditation:

\n #### c1 body|

XID: 1001

\n #### c1 body|
\n #### c1 body| Varnish\n #### c1 body|
\n #### c1 body| \n #### c1 body| \n #### c1 bodylen = 460 #### c1 EXPECT resp.status (503) == 503 (503) match ### c1 Closing fd 9 ## c1 Ending # top RESETTING after tests/r00387.vtc ## s1 Waiting for server ## v1 Stop #### v1 CLI TX| stop #### v1 debug| Stopping Child\n #### v1 debug| Child (16970) ended\n #### v1 debug| Child (-1) said EOF on CLI connection, exiting\n #### v1 debug| Child cleanup complete\n ### v1 CLI STATUS 200 #### v1 CLI TX| status #### v1 CLI RX| Child in state stopped ### v1 CLI STATUS 200 ## v1 Wait ## v1 R 16947 Status: 0200 # top TEST tests/r00387.vtc completed From cloude at instructables.com Thu May 14 00:17:16 2009 From: cloude at instructables.com (Cloude Porteus) Date: Wed, 13 May 2009 17:17:16 -0700 Subject: Debugging restarts/crashes? Message-ID: <4a05e1020905131717h1de721fci52408f02a996fbd@mail.gmail.com> Varnish seems to restart itself every so often, usually after a few days. We finally installed some SSD's, which made all the difference and our system load came down to the .02 - .2 range. I don't currently have any logging turned on. If I did, would it have any useful information for me? We typically don't have more that 700k objects cached. I also never see the worker threads go above 400. We are running on CentOS 5. Our config: NFILES=131072 MEMLOCK=90000 DAEMON_COREFILE_LIMIT="unlimited" VARNISH_VCL_CONF=/etc/varnish/instructables.vcl VARNISH_LISTEN_ADDRESS= VARNISH_LISTEN_PORT=80 VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 VARNISH_ADMIN_LISTEN_PORT=82 VARNISH_MIN_THREADS=400 VARNISH_MAX_THREADS=4000 VARNISH_THREAD_TIMEOUT=60 VARNISH_STORAGE_FILE=/var/lib/varnish/mmap VARNISH_STORAGE_SIZE=50G VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}" VARNISH_TTL=1800 DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ -f ${VARNISH_VCL_CONF} \ -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ -t ${VARNISH_TTL} \ -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \ -u varnish -g varnish \ -s ${VARNISH_STORAGE} \ -p obj_workspace=4096 \ -p sess_workspace=262144 \ -p lru_interval=3600 \ -p listen_depth=8192 \ -p log_hashstring=off \ -p sess_timeout=10 \ -p shm_workspace=32768 \ -p ping_interval=1 \ -p thread_pools=4 \ -p thread_pool_min=100 \ -p thread_pool_max=4000 \ -p srcaddr_ttl=0 \ -p esi_syntax=1 " I don't think out load is all that heavy and we have a high hitrate ratio. Our varnishstat output. 0+11:02:14 Hitrate ratio: 10 100 133 Hitrate avg: 0.9875 0.9833 0.9832 15370527 325.94 386.84 Client connections accepted 15370517 325.94 386.84 Client requests received 14604988 320.94 367.57 Cache hits 182795 4.00 4.60 Cache hits for pass 581426 1.00 14.63 Cache misses 765513 5.00 19.27 Backend connections success 1 0.00 0.00 Backend connections failures 119 . . N struct sess_mem 10 . . N struct sess 447866 . . N struct object 440545 . . N struct objecthead 913238 . . N struct smf 4490 . . N small free smf 2 . . N large free smf 0 . . N struct vbe_conn 112 . . N struct bereq 400 . . N worker threads 400 0.00 0.01 N worker threads created 313048 0.00 7.88 N worker threads limited 1 . . N backends 126326 . . N expired objects 1250255 . . N LRU moved objects 18 0.00 0.00 HTTP header overflows 13645614 292.95 343.42 Objects sent with write 15370518 327.94 386.84 Total Sessions 15370510 327.94 386.84 Total Requests 7 0.00 0.00 Total pipe 191394 5.00 4.82 Total pass 765481 9.00 19.27 Total fetch 7252376430 156332.93 182523.19 Total header bytes 74103505138 772233.39 1864989.81 Total body bytes 15370160 327.94 386.83 Session Closed 365 0.00 0.01 Session herd 758091326 15625.09 19079.16 SHM records 62365137 1317.75 1569.57 SHM writes 468 0.00 0.01 SHM flushes due to overflow 589 0.00 0.01 SHM MTX contention 307 0.00 0.01 SHM cycles through buffer 1573886 15.00 39.61 allocator requests 908746 . . outstanding allocations 7766196224 . . bytes allocated 45920894976 . . bytes free -- VP of Product Development Instructables.com http://www.instructables.com/member/lebowski -------------- next part -------------- An HTML attachment was scrubbed... URL: From jna at twitter.com Thu May 14 00:57:51 2009 From: jna at twitter.com (John Adams) Date: Wed, 13 May 2009 17:57:51 -0700 Subject: Debugging restarts/crashes? In-Reply-To: <4a05e1020905131717h1de721fci52408f02a996fbd@mail.gmail.com> References: <4a05e1020905131717h1de721fci52408f02a996fbd@mail.gmail.com> Message-ID: <75C50182-9258-4282-A6CE-DC46CFB3E7DE@twitter.com> Go look in /var/log/daemon.log, and find out why the children are dying. If you're getting SEGVs, that's going to be listed in your syslog. A big reason for crashing is running out of session workspace (sess_workspace). Find out if that's happening, and if it is, increase it. (say, to 512k) -j On May 13, 2009, at 5:17 PM, Cloude Porteus wrote: > Varnish seems to restart itself every so often, usually after a few > days. We finally installed some SSD's, which made all the difference > and our system load came down to the .02 - .2 range. > > I don't currently have any logging turned on. If I did, would it > have any useful information for me? > > We typically don't have more that 700k objects cached. I also never > see the worker threads go above 400. We are running on CentOS 5. > > Our config: > > NFILES=131072 > MEMLOCK=90000 > DAEMON_COREFILE_LIMIT="unlimited" > VARNISH_VCL_CONF=/etc/varnish/instructables.vcl > VARNISH_LISTEN_ADDRESS= > VARNISH_LISTEN_PORT=80 > VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 > VARNISH_ADMIN_LISTEN_PORT=82 > VARNISH_MIN_THREADS=400 > VARNISH_MAX_THREADS=4000 > VARNISH_THREAD_TIMEOUT=60 > VARNISH_STORAGE_FILE=/var/lib/varnish/mmap > VARNISH_STORAGE_SIZE=50G > VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}" > VARNISH_TTL=1800 > > DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ > -f ${VARNISH_VCL_CONF} \ > -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:$ > {VARNISH_ADMIN_LISTEN_PORT} \ > -t ${VARNISH_TTL} \ > -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},$ > {VARNISH_THREAD_TIMEOUT} \ > -u varnish -g varnish \ > -s ${VARNISH_STORAGE} \ > -p obj_workspace=4096 \ > -p sess_workspace=262144 \ > -p lru_interval=3600 \ > -p listen_depth=8192 \ > -p log_hashstring=off \ > -p sess_timeout=10 \ > -p shm_workspace=32768 \ > -p ping_interval=1 \ > -p thread_pools=4 \ > -p thread_pool_min=100 \ > -p thread_pool_max=4000 \ > -p srcaddr_ttl=0 \ > -p esi_syntax=1 " > > I don't think out load is all that heavy and we have a high hitrate > ratio. Our varnishstat output. > > 0+11:02:14 > Hitrate ratio: 10 100 133 > Hitrate avg: 0.9875 0.9833 0.9832 > > 15370527 325.94 386.84 Client connections accepted > 15370517 325.94 386.84 Client requests received > 14604988 320.94 367.57 Cache hits > 182795 4.00 4.60 Cache hits for pass > 581426 1.00 14.63 Cache misses > 765513 5.00 19.27 Backend connections success > 1 0.00 0.00 Backend connections failures > 119 . . N struct sess_mem > 10 . . N struct sess > 447866 . . N struct object > 440545 . . N struct objecthead > 913238 . . N struct smf > 4490 . . N small free smf > 2 . . N large free smf > 0 . . N struct vbe_conn > 112 . . N struct bereq > 400 . . N worker threads > 400 0.00 0.01 N worker threads created > 313048 0.00 7.88 N worker threads limited > 1 . . N backends > 126326 . . N expired objects > 1250255 . . N LRU moved objects > 18 0.00 0.00 HTTP header overflows > 13645614 292.95 343.42 Objects sent with write > 15370518 327.94 386.84 Total Sessions > 15370510 327.94 386.84 Total Requests > 7 0.00 0.00 Total pipe > 191394 5.00 4.82 Total pass > 765481 9.00 19.27 Total fetch > 7252376430 156332.93 182523.19 Total header bytes > 74103505138 772233.39 1864989.81 Total body bytes > 15370160 327.94 386.83 Session Closed > 365 0.00 0.01 Session herd > 758091326 15625.09 19079.16 SHM records > 62365137 1317.75 1569.57 SHM writes > 468 0.00 0.01 SHM flushes due to overflow > 589 0.00 0.01 SHM MTX contention > 307 0.00 0.01 SHM cycles through buffer > 1573886 15.00 39.61 allocator requests > 908746 . . outstanding allocations > 7766196224 . . bytes allocated > 45920894976 . . bytes free > > > -- > VP of Product Development > Instructables.com > > http://www.instructables.com/member/lebowski > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev --- John Adams Twitter Operations jna at twitter.com http://twitter.com/netik -------------- next part -------------- An HTML attachment was scrubbed... URL: From cloude at instructables.com Thu May 14 16:26:27 2009 From: cloude at instructables.com (Cloude Porteus) Date: Thu, 14 May 2009 09:26:27 -0700 Subject: Debugging restarts/crashes? In-Reply-To: <75C50182-9258-4282-A6CE-DC46CFB3E7DE@twitter.com> References: <4a05e1020905131717h1de721fci52408f02a996fbd@mail.gmail.com> <75C50182-9258-4282-A6CE-DC46CFB3E7DE@twitter.com> Message-ID: <4a05e1020905140926v1c78c468wa5084ed64ee92771@mail.gmail.com> I was able to find this entry in /var/log/messages, but it's hard to tell if it's going to be helpful or not for debugging. I can't seem to find a daemon.log in /var/log. May 13 13:06:48 squid03 varnishd[14993]: Child (14994) not responding to ping, killing it. May 13 13:06:48 squid03 varnishd[14993]: Child (14994) died signal=6 May 13 13:06:48 squid03 varnishd[14993]: Child (14994) Panic message: Assert error in Tcheck(), cache.h line 648: Condition((t.e) != 0) not true. thread = (cache-worker)sp = 0x2ab733984008 { fd = 19, id = 19, xid = 2173533104, client = 10.0.0.1:58322, step = STP_FETCH, handling = fetch, ws = 0x2ab733984078 { id = "sess", {s,f,r,e} = {0x2ab733984808,,+1162,(nil),+262144}, }, worker = 0x2ab7b4a46bd0 { }, vcl = { srcname = { "input", "Default", }, }, obj = 0x2aaddddaf000 { refcnt = 1, xid = 2173533104, ws = 0x2aaddddaf028 { id = "obj", {s,f,r,e} = {0x2aaddddaf358,,0x2aaddddaf358,(nil),+3240}, }, http = { ws = 0x2aaddddaf028 { id = "obj", {s,f,r,e} = {0x2aaddddaf358,,0x2aaddddaf358,(nil),+3240}, }, }, len = 0, store = { }, }, }, May 13 13:06:48 squid03 varnishd[14993]: child (32738) Started May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said Closed fds: 4 5 6 10 11 13 14 May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said Child starts May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said managed to mmap 53687091200 bytes of 53687091200 May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said Ready May 13 13:06:50 squid03 varnishd[14993]: Child (32738) died signal=6 May 13 13:06:50 squid03 varnishd[14993]: Child (32738) Panic message: Assert error in Tcheck(), cache.h line 648: Condition((t.e) != 0) not true. thread = (cache-worker)sp = 0x2ab736402008 { fd = 267, id = 267, xid = 1961344622, client = 10.0.0.1:59424, step = STP_FETCH, handling = fetch, ws = 0x2ab736402078 { id = "sess", {s,f,r,e} = {0x2ab736402808,,+1162,(nil),+262144}, }, worker = 0x2ab7432ffbd0 { }, vcl = { srcname = { "input", "Default", }, }, obj = 0x2aaab0ce0000 { refcnt = 1, xid = 1961344622, ws = 0x2aaab0ce0028 { id = "obj", {s,f,r,e} = {0x2aaab0ce0358,,0x2aaab0ce0358,(nil),+3240}, }, http = { ws = 0x2aaab0ce0028 { id = "obj", {s,f,r,e} = {0x2aaab0ce0358,,0x2aaab0ce0358,(nil),+3240}, }, }, len = 0, store = { }, }, }, May 13 13:06:50 squid03 varnishd[14993]: child (390) Started May 13 13:06:50 squid03 varnishd[14993]: Child (390) said Closed fds: 4 5 6 10 11 13 14 May 13 13:06:50 squid03 varnishd[14993]: Child (390) said Child starts May 13 13:06:50 squid03 varnishd[14993]: Child (390) said managed to mmap 53687091200 bytes of 53687091200 May 13 13:06:50 squid03 varnishd[14993]: Child (390) said Ready On Wed, May 13, 2009 at 5:57 PM, John Adams wrote: > Go look in /var/log/daemon.log, and find out why the children are dying. If > you're getting SEGVs, that's going to be listed in your syslog. > A big reason for crashing is running out of session workspace > (sess_workspace). Find out if that's happening, and if it is, increase it. > (say, to 512k) > > > -j > > On May 13, 2009, at 5:17 PM, Cloude Porteus wrote: > > Varnish seems to restart itself every so often, usually after a few days. > We finally installed some SSD's, which made all the difference and our > system load came down to the .02 - .2 range. > > I don't currently have any logging turned on. If I did, would it have any > useful information for me? > > We typically don't have more that 700k objects cached. I also never see the > worker threads go above 400. We are running on CentOS 5. > > Our config: > > NFILES=131072 > MEMLOCK=90000 > DAEMON_COREFILE_LIMIT="unlimited" > VARNISH_VCL_CONF=/etc/varnish/instructables.vcl > VARNISH_LISTEN_ADDRESS= > VARNISH_LISTEN_PORT=80 > VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 > VARNISH_ADMIN_LISTEN_PORT=82 > VARNISH_MIN_THREADS=400 > VARNISH_MAX_THREADS=4000 > VARNISH_THREAD_TIMEOUT=60 > VARNISH_STORAGE_FILE=/var/lib/varnish/mmap > VARNISH_STORAGE_SIZE=50G > VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}" > VARNISH_TTL=1800 > > DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ > -f ${VARNISH_VCL_CONF} \ > -T > ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ > -t ${VARNISH_TTL} \ > -w > ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \ > -u varnish -g varnish \ > -s ${VARNISH_STORAGE} \ > -p obj_workspace=4096 \ > -p sess_workspace=262144 \ > -p lru_interval=3600 \ > -p listen_depth=8192 \ > -p log_hashstring=off \ > -p sess_timeout=10 \ > -p shm_workspace=32768 \ > -p ping_interval=1 \ > -p thread_pools=4 \ > -p thread_pool_min=100 \ > -p thread_pool_max=4000 \ > -p srcaddr_ttl=0 \ > -p esi_syntax=1 " > > I don't think out load is all that heavy and we have a high hitrate ratio. > Our varnishstat output. > > 0+11:02:14 > > Hitrate ratio: 10 100 133 > Hitrate avg: 0.9875 0.9833 0.9832 > > 15370527 325.94 386.84 Client connections accepted > 15370517 325.94 386.84 Client requests received > 14604988 320.94 367.57 Cache hits > 182795 4.00 4.60 Cache hits for pass > 581426 1.00 14.63 Cache misses > 765513 5.00 19.27 Backend connections success > 1 0.00 0.00 Backend connections failures > 119 . . N struct sess_mem > 10 . . N struct sess > 447866 . . N struct object > 440545 . . N struct objecthead > 913238 . . N struct smf > 4490 . . N small free smf > 2 . . N large free smf > 0 . . N struct vbe_conn > 112 . . N struct bereq > 400 . . N worker threads > 400 0.00 0.01 N worker threads created > 313048 0.00 7.88 N worker threads limited > 1 . . N backends > 126326 . . N expired objects > 1250255 . . N LRU moved objects > 18 0.00 0.00 HTTP header overflows > 13645614 292.95 343.42 Objects sent with write > 15370518 327.94 386.84 Total Sessions > 15370510 327.94 386.84 Total Requests > 7 0.00 0.00 Total pipe > 191394 5.00 4.82 Total pass > 765481 9.00 19.27 Total fetch > 7252376430 156332.93 182523.19 Total header bytes > 74103505138 772233.39 1864989.81 Total body bytes > 15370160 327.94 386.83 Session Closed > 365 0.00 0.01 Session herd > 758091326 15625.09 19079.16 SHM records > 62365137 1317.75 1569.57 SHM writes > 468 0.00 0.01 SHM flushes due to overflow > 589 0.00 0.01 SHM MTX contention > 307 0.00 0.01 SHM cycles through buffer > 1573886 15.00 39.61 allocator requests > 908746 . . outstanding allocations > 7766196224 . . bytes allocated > 45920894976 . . bytes free > > > -- > VP of Product Development > Instructables.com > > http://www.instructables.com/member/lebowski > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > > --- > John Adams > Twitter Operations > jna at twitter.com > http://twitter.com/netik > > > > > -- VP of Product Development Instructables.com http://www.instructables.com/member/lebowski -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at crucially.net Thu May 14 16:28:22 2009 From: sky at crucially.net (Artur Bergman) Date: Thu, 14 May 2009 09:28:22 -0700 Subject: Debugging restarts/crashes? In-Reply-To: <4a05e1020905140926v1c78c468wa5084ed64ee92771@mail.gmail.com> References: <4a05e1020905131717h1de721fci52408f02a996fbd@mail.gmail.com> <75C50182-9258-4282-A6CE-DC46CFB3E7DE@twitter.com> <4a05e1020905140926v1c78c468wa5084ed64ee92771@mail.gmail.com> Message-ID: <447CB7A5-B73B-4098-AEC4-495B685A617F@crucially.net> Can you share your VCL? Artur On May 14, 2009, at 9:26 AM, Cloude Porteus wrote: > I was able to find this entry in /var/log/messages, but it's hard to > tell if it's going to be helpful or not for debugging. I can't seem > to find a daemon.log in /var/log. > > May 13 13:06:48 squid03 varnishd[14993]: Child (14994) not > responding to ping, killing it. > May 13 13:06:48 squid03 varnishd[14993]: Child (14994) died signal=6 > May 13 13:06:48 squid03 varnishd[14993]: Child (14994) Panic > message: Assert error in Tcheck(), cache.h line 648: > Condition((t.e) != 0) not true. thread = (cache-worker)sp = > 0x2ab733984008 { fd = 19, id = 19, xid = 2173533104, client = > 10.0.0.1:58322, step = STP_FETCH, handling = fetch, ws = > 0x2ab733984078 { id = "sess", {s,f,r,e} = > {0x2ab733984808,,+1162,(nil),+262144}, }, worker = > 0x2ab7b4a46bd0 { }, vcl = { srcname = > { "input", "Default", }, }, obj = > 0x2aaddddaf000 { refcnt = 1, xid = 2173533104, ws = > 0x2aaddddaf028 { id = "obj", {s,f,r,e} = > {0x2aaddddaf358,,0x2aaddddaf358,(nil),+3240}, }, http = > { ws = 0x2aaddddaf028 { id = "obj", {s,f,r,e} > = {0x2aaddddaf358,,0x2aaddddaf358,(nil),+3240}, }, }, > len = 0, store = { }, }, }, > May 13 13:06:48 squid03 varnishd[14993]: child (32738) Started > May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said Closed > fds: 4 5 6 10 11 13 14 > May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said Child > starts > May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said managed > to mmap 53687091200 bytes of 53687091200 > May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said Ready > May 13 13:06:50 squid03 varnishd[14993]: Child (32738) died signal=6 > May 13 13:06:50 squid03 varnishd[14993]: Child (32738) Panic > message: Assert error in Tcheck(), cache.h line 648: > Condition((t.e) != 0) not true. thread = (cache-worker)sp = > 0x2ab736402008 { fd = 267, id = 267, xid = 1961344622, client = > 10.0.0.1:59424, step = STP_FETCH, handling = fetch, ws = > 0x2ab736402078 { id = "sess", {s,f,r,e} = > {0x2ab736402808,,+1162,(nil),+262144}, }, worker = > 0x2ab7432ffbd0 { }, vcl = { srcname = > { "input", "Default", }, }, obj = > 0x2aaab0ce0000 { refcnt = 1, xid = 1961344622, ws = > 0x2aaab0ce0028 { id = "obj", {s,f,r,e} = > {0x2aaab0ce0358,,0x2aaab0ce0358,(nil),+3240}, }, http = > { ws = 0x2aaab0ce0028 { id = "obj", {s,f,r,e} > = {0x2aaab0ce0358,,0x2aaab0ce0358,(nil),+3240}, }, }, > len = 0, store = { }, }, }, > May 13 13:06:50 squid03 varnishd[14993]: child (390) Started > May 13 13:06:50 squid03 varnishd[14993]: Child (390) said Closed > fds: 4 5 6 10 11 13 14 > May 13 13:06:50 squid03 varnishd[14993]: Child (390) said Child starts > May 13 13:06:50 squid03 varnishd[14993]: Child (390) said managed to > mmap 53687091200 bytes of 53687091200 > May 13 13:06:50 squid03 varnishd[14993]: Child (390) said Ready > > > On Wed, May 13, 2009 at 5:57 PM, John Adams wrote: > Go look in /var/log/daemon.log, and find out why the children are > dying. If you're getting SEGVs, that's going to be listed in your > syslog. > > A big reason for crashing is running out of session workspace > (sess_workspace). Find out if that's happening, and if it is, > increase it. (say, to 512k) > > > -j > > On May 13, 2009, at 5:17 PM, Cloude Porteus wrote: > >> Varnish seems to restart itself every so often, usually after a few >> days. We finally installed some SSD's, which made all the >> difference and our system load came down to the .02 - .2 range. >> >> I don't currently have any logging turned on. If I did, would it >> have any useful information for me? >> >> We typically don't have more that 700k objects cached. I also never >> see the worker threads go above 400. We are running on CentOS 5. >> >> Our config: >> >> NFILES=131072 >> MEMLOCK=90000 >> DAEMON_COREFILE_LIMIT="unlimited" >> VARNISH_VCL_CONF=/etc/varnish/instructables.vcl >> VARNISH_LISTEN_ADDRESS= >> VARNISH_LISTEN_PORT=80 >> VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 >> VARNISH_ADMIN_LISTEN_PORT=82 >> VARNISH_MIN_THREADS=400 >> VARNISH_MAX_THREADS=4000 >> VARNISH_THREAD_TIMEOUT=60 >> VARNISH_STORAGE_FILE=/var/lib/varnish/mmap >> VARNISH_STORAGE_SIZE=50G >> VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},$ >> {VARNISH_STORAGE_SIZE}" >> VARNISH_TTL=1800 >> >> DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ >> -f ${VARNISH_VCL_CONF} \ >> -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:$ >> {VARNISH_ADMIN_LISTEN_PORT} \ >> -t ${VARNISH_TTL} \ >> -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},$ >> {VARNISH_THREAD_TIMEOUT} \ >> -u varnish -g varnish \ >> -s ${VARNISH_STORAGE} \ >> -p obj_workspace=4096 \ >> -p sess_workspace=262144 \ >> -p lru_interval=3600 \ >> -p listen_depth=8192 \ >> -p log_hashstring=off \ >> -p sess_timeout=10 \ >> -p shm_workspace=32768 \ >> -p ping_interval=1 \ >> -p thread_pools=4 \ >> -p thread_pool_min=100 \ >> -p thread_pool_max=4000 \ >> -p srcaddr_ttl=0 \ >> -p esi_syntax=1 " >> >> I don't think out load is all that heavy and we have a high hitrate >> ratio. Our varnishstat output. >> >> 0+11:02:14 >> Hitrate ratio: 10 100 133 >> Hitrate avg: 0.9875 0.9833 0.9832 >> >> 15370527 325.94 386.84 Client connections accepted >> 15370517 325.94 386.84 Client requests received >> 14604988 320.94 367.57 Cache hits >> 182795 4.00 4.60 Cache hits for pass >> 581426 1.00 14.63 Cache misses >> 765513 5.00 19.27 Backend connections success >> 1 0.00 0.00 Backend connections failures >> 119 . . N struct sess_mem >> 10 . . N struct sess >> 447866 . . N struct object >> 440545 . . N struct objecthead >> 913238 . . N struct smf >> 4490 . . N small free smf >> 2 . . N large free smf >> 0 . . N struct vbe_conn >> 112 . . N struct bereq >> 400 . . N worker threads >> 400 0.00 0.01 N worker threads created >> 313048 0.00 7.88 N worker threads limited >> 1 . . N backends >> 126326 . . N expired objects >> 1250255 . . N LRU moved objects >> 18 0.00 0.00 HTTP header overflows >> 13645614 292.95 343.42 Objects sent with write >> 15370518 327.94 386.84 Total Sessions >> 15370510 327.94 386.84 Total Requests >> 7 0.00 0.00 Total pipe >> 191394 5.00 4.82 Total pass >> 765481 9.00 19.27 Total fetch >> 7252376430 156332.93 182523.19 Total header bytes >> 74103505138 772233.39 1864989.81 Total body bytes >> 15370160 327.94 386.83 Session Closed >> 365 0.00 0.01 Session herd >> 758091326 15625.09 19079.16 SHM records >> 62365137 1317.75 1569.57 SHM writes >> 468 0.00 0.01 SHM flushes due to overflow >> 589 0.00 0.01 SHM MTX contention >> 307 0.00 0.01 SHM cycles through buffer >> 1573886 15.00 39.61 allocator requests >> 908746 . . outstanding allocations >> 7766196224 . . bytes allocated >> 45920894976 . . bytes free >> >> >> -- >> VP of Product Development >> Instructables.com >> >> http://www.instructables.com/member/lebowski >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-dev > > --- > John Adams > Twitter Operations > jna at twitter.com > http://twitter.com/netik > > > > > > > > -- > VP of Product Development > Instructables.com > > http://www.instructables.com/member/lebowski > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From cloude at instructables.com Thu May 14 16:31:45 2009 From: cloude at instructables.com (Cloude Porteus) Date: Thu, 14 May 2009 09:31:45 -0700 Subject: Debugging restarts/crashes? In-Reply-To: <447CB7A5-B73B-4098-AEC4-495B685A617F@crucially.net> References: <4a05e1020905131717h1de721fci52408f02a996fbd@mail.gmail.com> <75C50182-9258-4282-A6CE-DC46CFB3E7DE@twitter.com> <4a05e1020905140926v1c78c468wa5084ed64ee92771@mail.gmail.com> <447CB7A5-B73B-4098-AEC4-495B685A617F@crucially.net> Message-ID: <4a05e1020905140931g73f3b423re4bb58ef8a21905b@mail.gmail.com> Sure. ~~~default.vcl~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NFILES=131072 MEMLOCK=90000 DAEMON_COREFILE_LIMIT="unlimited" VARNISH_VCL_CONF=/etc/varnish/instructables.vcl VARNISH_LISTEN_ADDRESS= VARNISH_LISTEN_PORT=80 VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 VARNISH_ADMIN_LISTEN_PORT=82 VARNISH_MIN_THREADS=400 VARNISH_MAX_THREADS=4000 VARNISH_THREAD_TIMEOUT=60 VARNISH_STORAGE_FILE=/var/lib/varnish/mmap VARNISH_STORAGE_SIZE=50G VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}" VARNISH_TTL=1800 DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ -f ${VARNISH_VCL_CONF} \ -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ -t ${VARNISH_TTL} \ -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \ -u varnish -g varnish \ -s ${VARNISH_STORAGE} \ -p obj_workspace=4096 \ -p sess_workspace=262144 \ -p lru_interval=3600 \ -p listen_depth=8192 \ -p log_hashstring=off \ -p sess_timeout=10 \ -p shm_workspace=32768 \ -p ping_interval=1 \ -p thread_pools=4 \ -p thread_pool_min=100 \ -p thread_pool_max=4000 \ -p srcaddr_ttl=0 \ -p esi_syntax=1 " ~~~instructables.vcl~~~~~~~~~~~~~~~~~~~ C{ #include #include char myhostname[255] = ""; }C acl purge { "localhost"; /* myself */ "backup"; /* purge requests will always come from backup */ } backend default { /* * Default backend definition. Set this to point to your content * server. */ .host = "10.0.0.1"; /* put your back end server here! */ .port = "8080"; } sub vcl_recv { C{ if (myhostname[0] == '\0') { /* only get hostname once - restart required if hostname changes */ gethostname(myhostname, 255); } }C /* Do not cache if request is not GET or HEAD */ if (req.request != "GET" && req.request != "HEAD") { /* Forward to 'lookup' if request is an authorized PURGE request */ if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } lookup; } set req.http.connection = "close"; pipe; } /* Do not cache if request contains an Expect header */ if (req.http.Expect) { set req.http.connection = "close"; pipe; } # normalize Accept-Encoding to reduce vary if (req.http.Accept-Encoding) { if (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } } if (req.url ~ "^/(you|edit|account)") { pass; } /* Enable grace period (varnish serves stale (but cacheable) objects while retriving * object from backend) */ set req.grace = 30s; if (req.request == "GET" && req.http.cookie) { lookup; } } sub vcl_fetch { set obj.http.X-Cache = "MISS"; /*if (!obj.valid) { error; }*/ if (!obj.cacheable) { pass; } if (obj.http.Set-Cookie) { pass; } if(obj.status >= 500 && obj.status < 600) { pass; } /* Do not cache if response contains any 'no cache' tokens */ if (obj.http.Cache-Control ~ "(private|no-cache|no-store)") { pass; } /* * object from backend) */ set obj.grace = 30s; /* you might want to change this, I have a default TTL of 10 min here. */ if (obj.ttl < 1s) { set obj.ttl = 1711s; } deliver; } sub vcl_pipe { pipe; } sub vcl_pass { pass; } sub vcl_hash { set req.hash += req.url; set req.hash += req.http.host; hash; } sub vcl_hit { if (req.request == "PURGE") { purge_url(req.url); error 200 "Purged"; } if (!obj.cacheable) { pass; } deliver; } sub vcl_miss { /* Varnish doesn't do IMS to backend, so if not in cache just pass it through */ if (req.http.If-Modified-Since) { pass; } if (req.request == "PURGE") { error 404 "Not in cache"; } fetch; } sub vcl_hit { if (req.request == "PURGE") { purge_url(req.url); error 200 "Purged"; } if (!obj.cacheable) { pass; } deliver; } sub vcl_timeout { discard; } sub vcl_discard { discard; } sub vcl_deliver { C{ VRT_SetHdr(sp, HDR_RESP, "\014X-Cache-Svr:", myhostname, vrt_magic_string_end); }C /* mark hit/miss on the request */ if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; set resp.http.X-Cache-Hits = obj.hits; } else { set resp.http.X-Cache = "MISS"; } deliver; } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On Thu, May 14, 2009 at 9:28 AM, Artur Bergman wrote: > Can you share your VCL? > Artur > > On May 14, 2009, at 9:26 AM, Cloude Porteus wrote: > > I was able to find this entry in /var/log/messages, but it's hard to tell > if it's going to be helpful or not for debugging. I can't seem to find a > daemon.log in /var/log. > May 13 13:06:48 squid03 varnishd[14993]: Child (14994) not responding to > ping, killing it. > May 13 13:06:48 squid03 varnishd[14993]: Child (14994) died signal=6 > May 13 13:06:48 squid03 varnishd[14993]: Child (14994) Panic message: > Assert error in Tcheck(), cache.h line 648: Condition((t.e) != 0) not > true. thread = (cache-worker)sp = 0x2ab733984008 { fd = 19, id = 19, xid > = 2173533104, client = 10.0.0.1:58322, step = STP_FETCH, handling = > fetch, ws = 0x2ab733984078 { id = "sess", {s,f,r,e} = > {0x2ab733984808,,+1162,(nil),+262144}, }, worker = 0x2ab7b4a46bd0 { > }, vcl = { srcname = { "input", "Default", > }, }, obj = 0x2aaddddaf000 { refcnt = 1, xid = 2173533104, ws > = 0x2aaddddaf028 { id = "obj", {s,f,r,e} = > {0x2aaddddaf358,,0x2aaddddaf358,(nil),+3240}, }, http = { ws = > 0x2aaddddaf028 { id = "obj", {s,f,r,e} = > {0x2aaddddaf358,,0x2aaddddaf358,(nil),+3240}, }, }, len = 0, > store = { }, }, }, > May 13 13:06:48 squid03 varnishd[14993]: child (32738) Started > May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said Closed fds: 4 5 > 6 10 11 13 14 > May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said Child starts > May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said managed to mmap > 53687091200 bytes of 53687091200 > May 13 13:06:48 squid03 varnishd[14993]: Child (32738) said Ready > May 13 13:06:50 squid03 varnishd[14993]: Child (32738) died signal=6 > May 13 13:06:50 squid03 varnishd[14993]: Child (32738) Panic message: > Assert error in Tcheck(), cache.h line 648: Condition((t.e) != 0) not > true. thread = (cache-worker)sp = 0x2ab736402008 { fd = 267, id = 267, > xid = 1961344622, client = 10.0.0.1:59424, step = STP_FETCH, > handling = fetch, ws = 0x2ab736402078 { id = "sess", {s,f,r,e} = > {0x2ab736402808,,+1162,(nil),+262144}, }, worker = 0x2ab7432ffbd0 { > }, vcl = { srcname = { "input", "Default", > }, }, obj = 0x2aaab0ce0000 { refcnt = 1, xid = 1961344622, ws > = 0x2aaab0ce0028 { id = "obj", {s,f,r,e} = > {0x2aaab0ce0358,,0x2aaab0ce0358,(nil),+3240}, }, http = { ws = > 0x2aaab0ce0028 { id = "obj", {s,f,r,e} = > {0x2aaab0ce0358,,0x2aaab0ce0358,(nil),+3240}, }, }, len = 0, > store = { }, }, }, > May 13 13:06:50 squid03 varnishd[14993]: child (390) Started > May 13 13:06:50 squid03 varnishd[14993]: Child (390) said Closed fds: 4 5 6 > 10 11 13 14 > May 13 13:06:50 squid03 varnishd[14993]: Child (390) said Child starts > May 13 13:06:50 squid03 varnishd[14993]: Child (390) said managed to mmap > 53687091200 bytes of 53687091200 > May 13 13:06:50 squid03 varnishd[14993]: Child (390) said Ready > > > On Wed, May 13, 2009 at 5:57 PM, John Adams wrote: > >> Go look in /var/log/daemon.log, and find out why the children are dying. >> If you're getting SEGVs, that's going to be listed in your syslog. >> A big reason for crashing is running out of session workspace >> (sess_workspace). Find out if that's happening, and if it is, increase it. >> (say, to 512k) >> >> >> -j >> >> On May 13, 2009, at 5:17 PM, Cloude Porteus wrote: >> >> Varnish seems to restart itself every so often, usually after a few days. >> We finally installed some SSD's, which made all the difference and our >> system load came down to the .02 - .2 range. >> >> I don't currently have any logging turned on. If I did, would it have any >> useful information for me? >> >> We typically don't have more that 700k objects cached. I also never see >> the worker threads go above 400. We are running on CentOS 5. >> >> Our config: >> >> NFILES=131072 >> MEMLOCK=90000 >> DAEMON_COREFILE_LIMIT="unlimited" >> VARNISH_VCL_CONF=/etc/varnish/instructables.vcl >> VARNISH_LISTEN_ADDRESS= >> VARNISH_LISTEN_PORT=80 >> VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 >> VARNISH_ADMIN_LISTEN_PORT=82 >> VARNISH_MIN_THREADS=400 >> VARNISH_MAX_THREADS=4000 >> VARNISH_THREAD_TIMEOUT=60 >> VARNISH_STORAGE_FILE=/var/lib/varnish/mmap >> VARNISH_STORAGE_SIZE=50G >> VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}" >> VARNISH_TTL=1800 >> >> DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \ >> -f ${VARNISH_VCL_CONF} \ >> -T >> ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \ >> -t ${VARNISH_TTL} \ >> -w >> ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \ >> -u varnish -g varnish \ >> -s ${VARNISH_STORAGE} \ >> -p obj_workspace=4096 \ >> -p sess_workspace=262144 \ >> -p lru_interval=3600 \ >> -p listen_depth=8192 \ >> -p log_hashstring=off \ >> -p sess_timeout=10 \ >> -p shm_workspace=32768 \ >> -p ping_interval=1 \ >> -p thread_pools=4 \ >> -p thread_pool_min=100 \ >> -p thread_pool_max=4000 \ >> -p srcaddr_ttl=0 \ >> -p esi_syntax=1 " >> >> I don't think out load is all that heavy and we have a high hitrate ratio. >> Our varnishstat output. >> >> 0+11:02:14 >> >> Hitrate ratio: 10 100 133 >> Hitrate avg: 0.9875 0.9833 0.9832 >> >> 15370527 325.94 386.84 Client connections accepted >> 15370517 325.94 386.84 Client requests received >> 14604988 320.94 367.57 Cache hits >> 182795 4.00 4.60 Cache hits for pass >> 581426 1.00 14.63 Cache misses >> 765513 5.00 19.27 Backend connections success >> 1 0.00 0.00 Backend connections failures >> 119 . . N struct sess_mem >> 10 . . N struct sess >> 447866 . . N struct object >> 440545 . . N struct objecthead >> 913238 . . N struct smf >> 4490 . . N small free smf >> 2 . . N large free smf >> 0 . . N struct vbe_conn >> 112 . . N struct bereq >> 400 . . N worker threads >> 400 0.00 0.01 N worker threads created >> 313048 0.00 7.88 N worker threads limited >> 1 . . N backends >> 126326 . . N expired objects >> 1250255 . . N LRU moved objects >> 18 0.00 0.00 HTTP header overflows >> 13645614 292.95 343.42 Objects sent with write >> 15370518 327.94 386.84 Total Sessions >> 15370510 327.94 386.84 Total Requests >> 7 0.00 0.00 Total pipe >> 191394 5.00 4.82 Total pass >> 765481 9.00 19.27 Total fetch >> 7252376430 156332.93 182523.19 Total header bytes >> 74103505138 772233.39 1864989.81 Total body bytes >> 15370160 327.94 386.83 Session Closed >> 365 0.00 0.01 Session herd >> 758091326 15625.09 19079.16 SHM records >> 62365137 1317.75 1569.57 SHM writes >> 468 0.00 0.01 SHM flushes due to overflow >> 589 0.00 0.01 SHM MTX contention >> 307 0.00 0.01 SHM cycles through buffer >> 1573886 15.00 39.61 allocator requests >> 908746 . . outstanding allocations >> 7766196224 . . bytes allocated >> 45920894976 . . bytes free >> >> >> -- >> VP of Product Development >> Instructables.com >> >> http://www.instructables.com/member/lebowski >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at projects.linpro.no >> http://projects.linpro.no/mailman/listinfo/varnish-dev >> >> >> --- >> John Adams >> Twitter Operations >> jna at twitter.com >> http://twitter.com/netik >> >> >> >> >> > > > -- > VP of Product Development > Instructables.com > > http://www.instructables.com/member/lebowski > _______________________________________________ > varnish-dev mailing list > varnish-dev at projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-dev > > > -- VP of Product Development Instructables.com http://www.instructables.com/member/lebowski -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish-dev at projects.linpro.no Tue May 19 14:56:12 2009 From: varnish-dev at projects.linpro.no (Caslin Sheryl) Date: Tue, 19 May 2009 16:56:12 +0200 (CEST) Subject: Dude, join us Message-ID: <200905192530.236861BC607C2D@bxi112.neoplus.adsl.tpnet.pl> An HTML attachment was scrubbed... URL: From cloude at instructables.com Wed May 20 15:49:34 2009 From: cloude at instructables.com (Cloude Porteus) Date: Wed, 20 May 2009 08:49:34 -0700 Subject: Huge number for N struct vbe_conn Message-ID: <4a05e1020905200849s3c79f8e3n498b659341e2f328@mail.gmail.com> The vbe_conn is flickering between 0-2 and 18446744073709551615. See varnishstat below. Any ideas what might cause this? 7+02:38:17 squid02.instructables.com Hitrate ratio: 10 53 53 Hitrate avg: 0.9901 0.9921 0.9921 216720678 264.96 352.79 Client connections accepted 216720561 263.96 352.79 Client requests received 210371940 262.96 342.46 Cache hits 3969778 1.00 6.46 Cache hits for pass 2353478 0.00 3.83 Cache misses 6348344 1.00 10.33 Backend connections success 21 0.00 0.00 Backend connections failures 505 . . N struct sess_mem 9 . . N struct sess 882772 . . N struct object 877128 . . N struct objecthead 1821113 . . N struct smf 47639 . . N small free smf 53 . . N large free smf 18446744073709551615 . . N struct vbe_conn 236 . . N struct bereq 400 . . N worker threads 405 0.00 0.00 N worker threads created 181 0.00 0.00 N overflowed work requests 1 . . N backends 1392097 . . N expired objects 15124429 . . N LRU moved objects 269 0.00 0.00 HTTP header overflows 178241867 249.96 290.16 Objects sent with write 216720668 270.96 352.79 Total Sessions 216720576 270.96 352.79 Total Requests 18741 0.00 0.03 Total pipe 4054622 2.00 6.60 Total pass 6328784 7.00 10.30 Total fetch 99334274932 126775.84 161703.99 Total header bytes 1050072439149 1016298.38 1709388.84 Total body bytes 216720649 270.96 352.79 Session Closed 106 0.00 0.00 Session herd 10383942441 12795.97 16903.78 SHM records 874958544 1081.83 1424.32 SHM writes 4880 0.00 0.01 SHM flushes due to overflow 18440 0.00 0.03 SHM MTX contention 4265 0.00 0.01 SHM cycles through buffer 12403028 13.00 20.19 allocator requests 1773421 . . outstanding allocations 16170405888 . . bytes allocated 37516685312 . . bytes free -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed May 20 16:23:36 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 May 2009 16:23:36 +0000 Subject: Huge number for N struct vbe_conn In-Reply-To: Your message of "Wed, 20 May 2009 08:49:34 MST." <4a05e1020905200849s3c79f8e3n498b659341e2f328@mail.gmail.com> Message-ID: <7598.1242836616@critter.freebsd.dk> In message <4a05e1020905200849s3c79f8e3n498b659341e2f328 at mail.gmail.com>, Cloud e Porteus writes: >The vbe_conn is flickering between 0-2 and 18446744073709551615. See >varnishstat below. > >Any ideas what might cause this? Yes, it is an unlocked counter, and you had a race updating it: 18446744073709551615 - 2^64 = -1 -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From cloude at instructables.com Wed May 20 17:57:51 2009 From: cloude at instructables.com (Cloude Porteus) Date: Wed, 20 May 2009 10:57:51 -0700 Subject: Huge number for N struct vbe_conn In-Reply-To: <7598.1242836616@critter.freebsd.dk> References: <4a05e1020905200849s3c79f8e3n498b659341e2f328@mail.gmail.com> <7598.1242836616@critter.freebsd.dk> Message-ID: <4a05e1020905201057v56d2ff47j7b345e19b5602c75@mail.gmail.com> Is this something I should be concerned with? If so, is there something I need to change in our vcl? thanks, cloude On Wed, May 20, 2009 at 9:23 AM, Poul-Henning Kamp wrote: > In message <4a05e1020905200849s3c79f8e3n498b659341e2f328 at mail.gmail.com>, > Cloud > e Porteus writes: > > >The vbe_conn is flickering between 0-2 and 18446744073709551615. See > >varnishstat below. > > > >Any ideas what might cause this? > > Yes, it is an unlocked counter, and you had a race updating it: > > 18446744073709551615 - 2^64 = -1 > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -- VP of Product Development Instructables.com http://www.instructables.com/member/lebowski -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed May 20 17:58:24 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 20 May 2009 17:58:24 +0000 Subject: Huge number for N struct vbe_conn In-Reply-To: Your message of "Wed, 20 May 2009 10:57:51 MST." <4a05e1020905201057v56d2ff47j7b345e19b5602c75@mail.gmail.com> Message-ID: <8004.1242842304@critter.freebsd.dk> In message <4a05e1020905201057v56d2ff47j7b345e19b5602c75 at mail.gmail.com>, Cloud e Porteus writes: >Is this something I should be concerned with? No, just ignore it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From yu at irx.jp Fri May 29 12:13:14 2009 From: yu at irx.jp (Yutaro Shimamura) Date: Fri, 29 May 2009 21:13:14 +0900 Subject: Expired live object Message-ID: <218DF084-8A53-4EEE-AD99-633FD0F647B3@irx.jp> Hi, I'm trying varnish with small-objects. System had enough free-memory,(varnishstat: N bytes free) and objects ttl was live. but "N expired objectse" increased. Envelopment is this. OS: FreeBSD 6.2-RELEASE-p11 Varnish: rev.4088 start up with /usr/local/sbin/varnishd \ -T 127.0.0.1:10401 \ -f /var/home/yu/etc/test.vcl \ -s file,/tmp/cache,20g \ -p thread_pool_min=5 \ -p thread_pool_max=30 \ -p thread_pools=1 \ -p client_http11=on \ -P /var/run/varnishd.pid \ -w 5,300,300 \ -u www \ -a 0.0.0.0:80 vcl: backend store_1 { .host = "rs1.file.ryne.jp"; .port = "80"; } sub vcl_recv { set req.backend = store_1; set req.url = regsub(req.url, "^/(.)(.)(.+)", "/\1/\2/\1\2\3"); lookup; } sub vcl_hit { set obj.ttl = 14d; } "obj.ttl = 14d" defined here, It will fix all objects ttl to 14days, force. "N expired objectse" increases when the object TTL was over, or cache-out by LRU based algorithm I understood. Why did varnish expire under alive objects? thanks for your help, regards. ================= ?? ??? / Yutaro Shimamura yu at irx.jp From varnish-dev at projects.linpro.no Sun May 31 14:24:47 2009 From: varnish-dev at projects.linpro.no (Sharolyn) Date: Sun, 31 May 2009 16:24:47 +0200 (CEST) Subject: Our new one's blog Message-ID: <701145055315370.KXMUOTXXOONXHTO@h24006.upc-h.chello.nl> An HTML attachment was scrubbed... URL: From varnish-dev at projects.linpro.no Sun May 31 21:58:06 2009 From: varnish-dev at projects.linpro.no (Roselia) Date: Sun, 31 May 2009 23:58:06 +0200 (CEST) Subject: Check it now Message-ID: <493492834988551.WZBGSORCUDAKLFD@edunet-static-233.87-197-8.telecom.sk> An HTML attachment was scrubbed... URL: