From varnish-bugs at projects.linpro.no Mon Oct 1 07:08:44 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 01 Oct 2007 07:08:44 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation Message-ID: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation ----------------------+----------------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Keywords: varnishd core dump ----------------------+----------------------------------------------------- I am running Varnish trunk (up to date to commit 2030) in FreeBSD 6.2-RELEASE-p7/amd64, on Intel hardware (OS and Varnish built with -g and no stripping). After running for five hours, with high load and 1 million objects in the cache, child died: Sep 30 19:27:26 cache12 kernel: pid 925 (varnishd), uid 0: exited on signal 11 (core dumped) On the console I got: {{{ Cache child died pid=925 status=0x8b Clean child Child cleaned }}} My VCL: {{{ backend default { set backend.host = "192.168.0.3"; set backend.port = "80"; } acl purge { "192.168.0.4"/32; } sub vcl_recv { if ((req.http.host ~ "^(cache.finn.no|finn.no|www.finn.no)$") || (req.http.host == "banner.finn.no" && req.url ~ "^/(jsp2|finn/gojsp|daily|board|auximg/papirfly|finn/cacheable|crossdomain.xml)")) { if (req.request == "GET" || req.request == "HEAD") { lookup; } elsif (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } lookup; } else { pipe; } } else { error 403 "Access denied. Contact cacheadmin at finn.no if you have problems."; } } sub vcl_miss { if (req.request == "PURGE") { error 404 "Not in cache."; } else { fetch; } } sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } else { deliver; } } sub vcl_fetch { if (obj.ttl < 86400s) { set obj.ttl = 604800s; } if (obj.http.Cookie) { remove obj.http.Cookie; } if (obj.http.Set-Cookie) { remove obj.http.Set-Cookie; } insert; } }}} Backtracing the core dump, I get: {{{ root at cache12:/usr/local/var/varnish/cache12.finn.no# gdb -c varnishd.core -s /usr/local/sbin/varnishd GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `varnishd'. Program terminated with signal 11, Segmentation fault. #0 CNT_Session (sp=0xa2e03008) at cache_center.c:388 388 sp->wrk->acct.sess++; (gdb) bt #0 CNT_Session (sp=0xa2e03008) at cache_center.c:388 #1 0x00000000004153f9 in wrk_thread (priv=0xa2e03008) at cache_pool.c:192 #2 0x0000000800991c71 in ?? () #3 0x0000000000000000 in ?? () #4 0x00007fffd84c3fd8 in ?? () #5 0x0000000800aa0b40 in ?? () #6 0x00000000915f7200 in ?? () #7 0x0000000000000000 in ?? () #8 0x00007ffffb5dd000 in ?? () Cannot access memory at address 0x7ffffb7dd000 (gdb) }}} This ticket may be the same problem as #142. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 1 07:11:16 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 01 Oct 2007 07:11:16 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.5ece8c3b2f0bfbf6838b2b68f0a2c826@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Comment (by anders): PHK wanted this additional information: {{{ (gdb) print *sp $1 = {magic = 741317722, fd = 5310, id = 5310, xid = 0, wrk = 0x0, sockaddrlen = 16, mysockaddrlen = 128, sockaddr = 0xa2e03410, mysockaddr = 0xa2e03490, addr = '\0' , port = '\0' , srcaddr = 0x0, doclose = 0x0, http = 0xa2e03180, t_open = 1191173037.247371, t_req = nan(0x8000000000000), t_resp = nan(0x8000000000000), t_end = nan(0x8000000000000), step = STP_FIRST, handling = 0, wantbody = 0 '\0', err_code = 0, err_reason = 0x0, list = {tqe_next = 0x0, tqe_prev = 0x0}, backend = 0x0, bereq = 0x0, obj = 0x0, vcl = 0x0, mem = 0xa2e03000, workreq = {list = { tqe_next = 0xa2e03110, tqe_prev = 0xa2e03110}, sess = 0xa2e03008}, acct = {first = nan(0x8000000000000), sess = 0, req = 0, pipe = 0, pass = 0, fetch = 0, hdrbytes = 0, bodybytes = 0}, nhashptr = 0, ihashptr = 0, lhashptr = 0, hashptr = 0x0} }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 1 07:16:55 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 01 Oct 2007 07:16:55 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.5c3ba9f1f3a3e9268e9b8d4bbfc390e1@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Comment (by phk): Once again, a "this cannot happen" panic. We accessed sp->wrk in the line above and now it's gone. Smells like pointer-tango. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 1 09:54:28 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 01 Oct 2007 09:54:28 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.e15c57aa606b8fd84d8eaa3e89e5c9c9@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Comment (by anders): For the record, the server being used here has two physical processores with four cores each, 8 cores in total. Also I use three storage files. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 1 12:27:36 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 01 Oct 2007 12:27:36 -0000 Subject: [Varnish] #165: Varnishd crashes due to assert error in cache_pipe.c Message-ID: <052.03b4543d19e45e716fa277c4a64aba43@projects.linpro.no> #165: Varnishd crashes due to assert error in cache_pipe.c ----------------------+----------------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Keywords: varnishd core dump rdf cache_pipe ----------------------+----------------------------------------------------- I am running Varnish trunk (up to date to commit 2030) in FreeBSD 6.2-RELEASE-p7/amd64, on Intel hardware (OS and Varnish built with -g and no stripping). Oct 1 14:17:25 cache12 kernel: pid 21232 (varnishd), uid 0: exited on signal 6 (core dumped) On the console, I got: {{{ Child said (2, 21232): <> Cache child died pid=21232 status=0x86 }}} Unfortunately, backtrace gives me only ??. No symbols matched. A new bug? Didn't see this one before, with trunk 2030 or earlier. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 1 12:28:23 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 01 Oct 2007 12:28:23 -0000 Subject: [Varnish] #165: Varnishd crashes due to assert error in cache_pipe.c In-Reply-To: <052.03b4543d19e45e716fa277c4a64aba43@projects.linpro.no> References: <052.03b4543d19e45e716fa277c4a64aba43@projects.linpro.no> Message-ID: <061.f96e22453aa840ef5788b60a314f1910@projects.linpro.no> #165: Varnishd crashes due to assert error in cache_pipe.c -----------------------------------------------+---------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump rdf cache_pipe | -----------------------------------------------+---------------------------- Comment (by anders): Sorry. This was with trunk 2074, not 2030. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 1 12:52:58 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 01 Oct 2007 12:52:58 -0000 Subject: [Varnish] #165: Varnishd crashes due to assert error in cache_pipe.c In-Reply-To: <052.03b4543d19e45e716fa277c4a64aba43@projects.linpro.no> References: <052.03b4543d19e45e716fa277c4a64aba43@projects.linpro.no> Message-ID: <061.cc156f6a2ba18abfedd29fc944b05b66@projects.linpro.no> #165: Varnishd crashes due to assert error in cache_pipe.c -----------------------------------------------+---------------------------- Reporter: anders | Owner: phk Type: defect | Status: closed Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: fixed Keywords: varnishd core dump rdf cache_pipe | -----------------------------------------------+---------------------------- Changes (by phk): * status: new => closed * resolution: => fixed Comment: Hopefully fixed in #2077. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 1 13:01:30 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 01 Oct 2007 13:01:30 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.021f96ffb36aa2a1c7723d5fabe3a8a7@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): This still happens with trunk/2077. Backtrace mostly empty: {{{ # gdb -c varnishd.core -s /usr/local/sbin/varnishd GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `varnishd'. Program terminated with signal 6, Aborted. #0 0x0000000800c3373c in ?? () (gdb) bt #0 0x0000000800c3373c in ?? () #1 0x00000008009a569f in ?? () #2 0x0000000600541e00 in ?? () #3 0x0000000000541e00 in ?? () #4 0x00007fffff5faea0 in ?? () #5 0x000000080099aff8 in ?? () #6 0x00007fffff5faebc in ?? () #7 0x000000060056cc00 in ?? () #8 0x00007fffff5faed0 in ?? () #9 0x0000000800ce5216 in ?? () #10 0x0000000000000080 in ?? () #11 0xffffffdf00000206 in ?? () #12 0xffffffffffffffff in ?? () #13 0x00000008ffffffff in ?? () #14 0x00007fffff5faf20 in ?? () #15 0x0000000800678ba9 in ?? () #16 0x0000000000000009 in ?? () #17 0x0000000800e23c80 in ?? () #18 0x000000000043136a in __func__.6 () Cannot access memory at address 0x431338 (gdb) info threads 188 process 100770 0x0000000800c806fc in ?? () 187 process 100116 0x0000000800cc402c in ?? () 186 process 100209 0x0000000800cc402c in ?? () 185 process 100230 0x0000000800cc402c in ?? () 184 process 100261 0x0000000800c806fc in ?? () 183 process 100267 0x0000000800ce7bcc in ?? () 182 process 100272 0x0000000800ce7bcc in ?? () 181 process 100273 0x0000000800ce7bcc in ?? () 180 process 100275 0x0000000800ce7bcc in ?? () 179 process 100276 0x0000000800ce7bcc in ?? () 178 process 100279 0x0000000800ce7bcc in ?? () 177 process 100280 0x0000000800ce7bcc in ?? () 176 process 100281 0x0000000800ce7bcc in ?? () 175 process 100282 0x0000000800ce7bcc in ?? () 174 process 100289 0x0000000800cbce6c in ?? () 173 process 100291 0x0000000800ce7bcc in ?? () 172 process 100294 0x0000000800ce7bcc in ?? () 171 process 100300 0x0000000800ce7bcc in ?? () 170 process 100304 0x0000000800ce7bcc in ?? () 169 process 100314 0x0000000800ce7bcc in ?? () 168 process 100317 0x0000000800ce7bcc in ?? () 167 process 100318 0x0000000800ce7bcc in ?? () 166 process 100319 0x0000000800ce7bcc in ?? () 165 process 100320 0x0000000800cbce6c in ?? () 164 process 100321 0x0000000800ce7bcc in ?? () 163 process 100322 0x0000000800ce7bcc in ?? () 162 process 100323 0x0000000800ce7bcc in ?? () 161 process 100324 0x0000000800ce7bcc in ?? () 160 process 100326 0x0000000800ce7bcc in ?? () 159 process 100327 0x0000000800ce7bcc in ?? () 158 process 100328 0x0000000800ce7bcc in ?? () 157 process 100329 0x0000000800ce7bcc in ?? () 156 process 100331 0x0000000800ce7bcc in ?? () 155 process 100332 0x0000000800ce7bcc in ?? () 154 process 100333 0x0000000800ce7bcc in ?? () 153 process 100335 0x0000000800ce7bcc in ?? () 152 process 100336 0x0000000800ce7bcc in ?? () 151 process 100338 0x0000000800ce7bcc in ?? () 150 process 100339 0x0000000800ce7bcc in ?? () 149 process 100340 0x0000000800ce7bcc in ?? () 148 process 100342 0x0000000800ce7bcc in ?? () 147 process 100343 0x0000000800ce7bcc in ?? () 146 process 100344 0x0000000800ce7bcc in ?? () 145 process 100345 0x0000000800ce7bcc in ?? () 144 process 100346 0x0000000800ce7bcc in ?? () 143 process 100347 0x0000000800ce7bcc in ?? () 142 process 100348 0x0000000800ce7bcc in ?? () 141 process 100349 0x0000000800ce7bcc in ?? () 140 process 100350 0x0000000800cca44c in ?? () ---Type to continue, or q to quit--- 139 process 100351 0x0000000800ce7bcc in ?? () 138 process 100352 0x0000000800ce7bcc in ?? () 137 process 100353 0x0000000800ce7bcc in ?? () 136 process 100354 0x0000000800ce7bcc in ?? () 135 process 100355 0x0000000800ce7bcc in ?? () 134 process 100356 0x0000000800ce7bcc in ?? () 133 process 100357 0x0000000800ce7bcc in ?? () 132 process 100358 0x0000000800ce7bcc in ?? () 131 process 100359 0x0000000800ce7bcc in ?? () 130 process 100361 0x0000000800ce7bcc in ?? () 129 process 100364 0x0000000800cca44c in ?? () 128 process 100365 0x0000000800ce7bcc in ?? () 127 process 100366 0x0000000800cbce6c in ?? () 126 process 100367 0x0000000800ce7bcc in ?? () 125 process 100368 0x0000000800cbce6c in ?? () 124 process 100369 0x0000000800ce7bcc in ?? () 123 process 100370 0x0000000800ce7bcc in ?? () 122 process 100371 0x0000000800cca44c in ?? () 121 process 100374 0x0000000800ce7bcc in ?? () 120 process 100375 0x0000000800ce7bcc in ?? () 119 process 100376 0x0000000800ce7bcc in ?? () 118 process 100377 0x0000000800ce7bcc in ?? () 117 process 100378 0x0000000800ce7bcc in ?? () 116 process 100379 0x0000000800ce7bcc in ?? () 115 process 100380 0x0000000800cca44c in ?? () 114 process 100381 0x0000000800ce7bcc in ?? () 113 process 100383 0x0000000800cca44c in ?? () 112 process 100384 0x0000000800ce7bcc in ?? () 111 process 100385 0x0000000800ce7bcc in ?? () 110 process 100386 0x0000000800ce7bcc in ?? () 109 process 100387 0x0000000800ce7bcc in ?? () 108 process 100388 0x0000000800ce7bcc in ?? () 107 process 100389 0x0000000800cca44c in ?? () 106 process 100390 0x0000000800ce7bcc in ?? () 105 process 100391 0x0000000800ce7bcc in ?? () 104 process 100392 0x0000000800ce7bcc in ?? () 103 process 100393 0x0000000800cbce6c in ?? () 102 process 100394 0x0000000800ce7bcc in ?? () 101 process 100395 0x0000000800ce7bcc in ?? () 100 process 100396 0x0000000800ce7bcc in ?? () 99 process 100397 0x0000000800ce7bcc in ?? () 98 process 100398 0x0000000800ce7bcc in ?? () 97 process 100400 0x0000000800cca44c in ?? () 96 process 100401 0x0000000800cbce6c in ?? () 95 process 100402 0x0000000800ce7bcc in ?? () 94 process 100404 0x0000000800ce7bcc in ?? () 93 process 100405 0x0000000800ce7bcc in ?? () 92 process 100406 0x0000000800ce7bcc in ?? () 91 process 100407 0x0000000800ce7bcc in ?? () ---Type to continue, or q to quit--- 90 process 100409 0x0000000800ce7bcc in ?? () 89 process 100410 0x0000000800ce7bcc in ?? () 88 process 100412 0x0000000800ce7bcc in ?? () 87 process 100413 0x0000000800ce7bcc in ?? () 86 process 100414 0x0000000800ce7bcc in ?? () 85 process 100415 0x0000000800ce7bcc in ?? () 84 process 100416 0x0000000800ce7bcc in ?? () 83 process 100417 0x0000000800ce7bcc in ?? () 82 process 100419 0x0000000800ce7bcc in ?? () 81 process 100421 0x0000000800ce7bcc in ?? () 80 process 100422 0x0000000800ce7bcc in ?? () 79 process 100423 0x0000000800ce7bcc in ?? () 78 process 100425 0x0000000800ce7bcc in ?? () 77 process 100426 0x0000000800ce7bcc in ?? () 76 process 100429 0x0000000800ce7bcc in ?? () 75 process 100430 0x0000000800ce7bcc in ?? () 74 process 100431 0x0000000800ce7bcc in ?? () 73 process 100432 0x0000000800ce7bcc in ?? () 72 process 100434 0x0000000800cbce6c in ?? () 71 process 100435 0x0000000800ce7bcc in ?? () 70 process 100436 0x0000000800ce7bcc in ?? () 69 process 100437 0x0000000800ce7bcc in ?? () 68 process 100438 0x0000000800ce7bcc in ?? () 67 process 100440 0x0000000800ce7bcc in ?? () 66 process 100441 0x0000000800ce7bcc in ?? () 65 process 100442 0x0000000800ce7bcc in ?? () 64 process 100443 0x0000000800ce7bcc in ?? () 63 process 100445 0x0000000800cbce6c in ?? () 62 process 100446 0x0000000800ce7bcc in ?? () 61 process 100447 0x0000000800ce7bcc in ?? () 60 process 100448 0x0000000800cca44c in ?? () 59 process 100452 0x0000000800ce7bcc in ?? () 58 process 100454 0x0000000800ce7bcc in ?? () 57 process 100455 0x0000000800ce7bcc in ?? () 56 process 100456 0x0000000800ce7bcc in ?? () 55 process 100460 0x0000000800ce7bcc in ?? () 54 process 100461 0x0000000800ce7bcc in ?? () 53 process 100462 0x0000000800ce7bcc in ?? () 52 process 100463 0x0000000800ce7bcc in ?? () 51 process 100464 0x0000000800ce7bcc in ?? () 50 process 100465 0x0000000800ce7bcc in ?? () 49 process 100466 0x0000000800ce7bcc in ?? () 48 process 100467 0x0000000800ce7bcc in ?? () 47 process 100468 0x0000000800cbce6c in ?? () 46 process 100469 0x0000000800ce7bcc in ?? () 45 process 100470 0x0000000800ce7bcc in ?? () 44 process 100472 0x0000000800ce7bcc in ?? () 43 process 100475 0x0000000800ce7bcc in ?? () 42 process 100476 0x0000000800ce7bcc in ?? () ---Type to continue, or q to quit--- 41 process 100477 0x0000000800ce7bcc in ?? () 40 process 100479 0x0000000800ce7bcc in ?? () 39 process 100480 0x0000000800ce7bcc in ?? () 38 process 100482 0x0000000800ce7bcc in ?? () 37 process 100484 0x0000000800ce7bcc in ?? () 36 process 100485 0x0000000800ce7bcc in ?? () 35 process 100486 0x0000000800ce7bcc in ?? () 34 process 100488 0x0000000800ce7bcc in ?? () 33 process 100490 0x0000000800ce7bcc in ?? () 32 process 100491 0x0000000800ce7bcc in ?? () 31 process 100493 0x0000000800ce7bcc in ?? () 30 process 100494 0x0000000800ce7bcc in ?? () 29 process 100495 0x0000000800ce7bcc in ?? () 28 process 100496 0x0000000800ce7b8c in ?? () 27 process 100497 0x0000000800ce7bcc in ?? () 26 process 100499 0x0000000800ce7bcc in ?? () 25 process 100500 0x0000000800ce7bcc in ?? () 24 process 100502 0x0000000800ce7bcc in ?? () 23 process 100503 0x0000000800ce7bcc in ?? () 22 process 100504 0x0000000800ce7bcc in ?? () 21 process 100507 0x0000000800ce7bcc in ?? () 20 process 100512 0x0000000800ce7bcc in ?? () 19 process 100516 0x0000000800ce7bcc in ?? () 18 process 100517 0x0000000800cca44c in ?? () 17 process 100564 0x0000000800ce7bcc in ?? () 16 process 100565 0x0000000800ce7bcc in ?? () 15 process 100569 0x0000000800ce7bcc in ?? () 14 process 100573 0x0000000800ce7bcc in ?? () 13 process 100574 0x0000000800ce7bcc in ?? () 12 process 100575 0x0000000800ce7bcc in ?? () 11 process 100577 0x0000000800ce7bcc in ?? () 10 process 100580 0x0000000800ce7bcc in ?? () 9 process 100581 0x0000000800ce7bcc in ?? () 8 process 100583 0x0000000800ce7bcc in ?? () 7 process 100589 0x0000000800ce7bcc in ?? () 6 process 100594 0x0000000800ce7bcc in ?? () 5 process 100595 0x0000000800ce7bcc in ?? () 4 process 100596 0x0000000800ce7bcc in ?? () 3 process 100597 0x0000000800ce7bcc in ?? () 2 process 100600 0x0000000800ce7bcc in ?? () * 1 process 100258 0x0000000800c3373c in ?? () (gdb) }}} Sigh. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 1 13:02:51 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 01 Oct 2007 13:02:51 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.07fd7cd51ca4c9949d6d45372b9893f9@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): The above mentioned backtrace came from a crash that looked like this on the console: {{{ Child said (2, 35420): <obj) == 0) not true. errno = 9 (Bad file descriptor) >> Cache child died pid=35420 status=0x86 }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Oct 4 09:17:18 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 04 Oct 2007 09:17:18 -0000 Subject: [Varnish] #47: pass mode does not handle POST In-Reply-To: <049.dce81b215711ded70632456b6f336a2b@projects.linpro.no> References: <049.dce81b215711ded70632456b6f336a2b@projects.linpro.no> Message-ID: <058.f9e0f0e628458f8c6778660d95fe6b88@projects.linpro.no> #47: pass mode does not handle POST ----------------------+----------------------------------------------------- Reporter: des | Owner: cecilihf Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Changes (by cecilihf): * owner: des => cecilihf -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Oct 5 08:19:01 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 05 Oct 2007 08:19:01 -0000 Subject: [Varnish] #166: debian/varnish.postinst fails Message-ID: <053.e799974f9ed4b655988a18eb8fa76c07@projects.linpro.no> #166: debian/varnish.postinst fails -----------------------+---------------------------------------------------- Reporter: havardf | Owner: des Type: defect | Status: new Priority: normal | Milestone: Component: packaging | Version: trunk Severity: normal | Keywords: postinst debian packaging -----------------------+---------------------------------------------------- There seems to be two problems with varnish.postinst 1. '/etc/init.d/varnish start' is called twice. dh_installinit is called in rules-script, but its result already exists in .postinst. Might have been introduced with the tag '#DEBHELPER#'? 2. Default path to varnish_storage.bin is not created at installation, hence varnish.postinst will fail, and debian package will remain in a half-configured state. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 11:08:52 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 11:08:52 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.c3b740de8e4c324f85f868a2ca26b9b5@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): I was erroneously using the -s option for gdb. :-( Backtrace here: {{{ root at cache10:/usr/local/varnish/cache10.finn.no# gdb -c varnishd_2007-10-08_12-50-01.core /usr/local/sbin/va rnishd GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `varnishd'. Program terminated with signal 6, Aborted. Reading symbols from /usr/local/lib/libvarnish.so.0...done. Loaded symbols for /usr/local/lib/libvarnish.so.0 Reading symbols from /usr/local/lib/libvarnishcompat.so.0...done. Loaded symbols for /usr/local/lib/libvarnishcompat.so.0 Reading symbols from /usr/local/lib/libvcl.so.0...done. Loaded symbols for /usr/local/lib/libvcl.so.0 Reading symbols from /usr/lib/libthr.so.2...done. Loaded symbols for /usr/lib/libthr.so.2 Reading symbols from /lib/libm.so.4...done. Loaded symbols for /lib/libm.so.4 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from ./bin.fTLfMGTM...done. Loaded symbols for ./bin.fTLfMGTM Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x0000000800c080dc in thr_kill () from /lib/libc.so.6 [New Thread 0x9fa7200 (LWP 100348)] [New Thread 0x9fa7000 (LWP 100347)] [New Thread 0x9fa3e00 (LWP 100346)] [New Thread 0x9fa3c00 (LWP 100343)] [New Thread 0x9fa3800 (LWP 100341)] [New Thread 0x9fa3a00 (LWP 100340)] [New Thread 0x9fa3600 (LWP 100339)] [New Thread 0x9fa3400 (LWP 100338)] [New Thread 0x9fa3200 (LWP 100336)] [New Thread 0x9fa3000 (LWP 100335)] [New Thread 0x9fa2e00 (LWP 100334)] [New Thread 0x9fa2c00 (LWP 100333)] [New Thread 0x9fa2a00 (LWP 100332)] [New Thread 0x9fa2800 (LWP 100331)] [New Thread 0x9fa2600 (LWP 100330)] [New Thread 0x9fa2400 (LWP 100329)] [New Thread 0x9fa2200 (LWP 100328)] [New Thread 0x9fa2000 (LWP 100326)] [New Thread 0x508de00 (LWP 100325)] [New Thread 0x508dc00 (LWP 100322)] [New Thread 0x508da00 (LWP 100321)] [New Thread 0x508d800 (LWP 100320)] [New Thread 0x508d600 (LWP 100319)] [New Thread 0x508d400 (LWP 100317)] [New Thread 0x31efe00 (LWP 100314)] [New Thread 0x31efa00 (LWP 100313)] [New Thread 0x31eac00 (LWP 100311)] [New Thread 0x31e4400 (LWP 100310)] [New Thread 0xbbd000 (LWP 100308)] [New Thread 0x31ea000 (LWP 100307)] [New Thread 0x31ece00 (LWP 100244)] [New Thread 0x31ec400 (LWP 100243)] [New Thread 0x31ec200 (LWP 100177)] [New Thread 0x31ea400 (LWP 100312)] [New Thread 0x31ec800 (LWP 100309)] [New Thread 0x31ec000 (LWP 100306)] [New Thread 0x2ac6c00 (LWP 100305)] [New Thread 0x2ac6600 (LWP 100304)] [New Thread 0x31e6600 (LWP 100303)] [New Thread 0x31e7c00 (LWP 100302)] [New Thread 0x31e7a00 (LWP 100301)] [New Thread 0x31e7400 (LWP 100300)] [New Thread 0x31e7000 (LWP 100299)] [New Thread 0xef6000 (LWP 100298)] [New Thread 0x31e0200 (LWP 100297)] [New Thread 0x1dc3e00 (LWP 100296)] [New Thread 0x31fa000 (LWP 100295)] [New Thread 0xef6e00 (LWP 100293)] [New Thread 0x31e6000 (LWP 100292)] [New Thread 0xb4d000 (LWP 100291)] [New Thread 0xe68c00 (LWP 100290)] [New Thread 0x31ef400 (LWP 100288)] [New Thread 0xf3a200 (LWP 100287)] [New Thread 0x31f3c00 (LWP 100281)] [New Thread 0x31dcc00 (LWP 100275)] [New Thread 0x26bb000 (LWP 100273)] [New Thread 0x31ea200 (LWP 100268)] [New Thread 0x31f3200 (LWP 100266)] [New Thread 0xe68600 (LWP 100264)] [New Thread 0x31e0a00 (LWP 100263)] [New Thread 0xafd400 (LWP 100260)] [New Thread 0x31e1000 (LWP 100246)] [New Thread 0xafda00 (LWP 100242)] [New Thread 0x31f8800 (LWP 100239)] [New Thread 0xef6400 (LWP 100234)] [New Thread 0x31e0c00 (LWP 100231)] [New Thread 0xe33000 (LWP 100230)] [New Thread 0xf3aa00 (LWP 100229)] [New Thread 0x31e4800 (LWP 100227)] [New Thread 0xbbd400 (LWP 100226)] [New Thread 0x31dc000 (LWP 100225)] [New Thread 0x31e4000 (LWP 100223)] [New Thread 0x2a7b200 (LWP 100220)] [New Thread 0xbbdc00 (LWP 100214)] [New Thread 0x31eaa00 (LWP 100211)] [New Thread 0x31e1e00 (LWP 100210)] [New Thread 0x31e1800 (LWP 100209)] [New Thread 0x31f6000 (LWP 100207)] [New Thread 0x31ec600 (LWP 100205)] [New Thread 0x1dc3200 (LWP 100202)] [New Thread 0x31eca00 (LWP 100195)] [New Thread 0x31e3600 (LWP 100194)] [New Thread 0x31e2000 (LWP 100193)] [New Thread 0xf83e00 (LWP 100192)] [New Thread 0xe68800 (LWP 100191)] [New Thread 0xf83000 (LWP 100190)] [New Thread 0x31f6200 (LWP 100186)] [New Thread 0xddc800 (LWP 100179)] [New Thread 0x31f8000 (LWP 100178)] [New Thread 0xf83800 (LWP 100176)] [New Thread 0x31f8200 (LWP 100172)] [New Thread 0xd59400 (LWP 100168)] [New Thread 0x31efc00 (LWP 100167)] [New Thread 0xe68200 (LWP 100166)] [New Thread 0x31e7200 (LWP 100162)] [New Thread 0x1dc3800 (LWP 100160)] [New Thread 0x31e7e00 (LWP 100154)] [New Thread 0xafde00 (LWP 100153)] [New Thread 0xd59c00 (LWP 100151)] [New Thread 0xd59e00 (LWP 100150)] [New Thread 0x31dce00 (LWP 100147)] [New Thread 0x31dca00 (LWP 100286)] [New Thread 0x2a7be00 (LWP 100285)] [New Thread 0x2a7bc00 (LWP 100284)] [New Thread 0x2a7ba00 (LWP 100282)] [New Thread 0xda7800 (LWP 100280)] [New Thread 0x1a53400 (LWP 100279)] [New Thread 0xda7400 (LWP 100278)] [New Thread 0x26bbe00 (LWP 100274)] [New Thread 0x31f3400 (LWP 100272)] [New Thread 0x26bbc00 (LWP 100271)] [New Thread 0x1346c00 (LWP 100270)] [New Thread 0x2ac6e00 (LWP 100269)] [New Thread 0x26bba00 (LWP 100267)] [New Thread 0x1a53000 (LWP 100265)] [New Thread 0x2a7b000 (LWP 100262)] [New Thread 0x31f6e00 (LWP 100259)] [New Thread 0xafdc00 (LWP 100258)] [New Thread 0xe33600 (LWP 100257)] [New Thread 0x2a7b400 (LWP 100256)] [New Thread 0xddc600 (LWP 100255)] [New Thread 0xef6200 (LWP 100254)] [New Thread 0x31e0800 (LWP 100253)] [New Thread 0x26bb400 (LWP 100251)] [New Thread 0xf83c00 (LWP 100250)] [New Thread 0xbbde00 (LWP 100249)] [New Thread 0x1a53c00 (LWP 100248)] [New Thread 0x1dc3c00 (LWP 100245)] [New Thread 0x31dc400 (LWP 100240)] [New Thread 0x31e1600 (LWP 100238)] [New Thread 0xb4d600 (LWP 100236)] [New Thread 0x1346e00 (LWP 100233)] [New Thread 0x31e2e00 (LWP 100224)] [New Thread 0xda7600 (LWP 100222)] [New Thread 0x31dc600 (LWP 100221)] [New Thread 0x1346200 (LWP 100218)] [New Thread 0xddca00 (LWP 100217)] [New Thread 0x31de600 (LWP 100213)] [New Thread 0x31e6200 (LWP 100212)] [New Thread 0x31e6a00 (LWP 100208)] [New Thread 0x31e6400 (LWP 100204)] [New Thread 0xe33e00 (LWP 100203)] [New Thread 0xef6800 (LWP 100201)] [New Thread 0x1a53e00 (LWP 100200)] [New Thread 0xddce00 (LWP 100199)] [New Thread 0xb4de00 (LWP 100198)] [New Thread 0xafd800 (LWP 100197)] [New Thread 0x2ac6400 (LWP 100196)] [New Thread 0xafd600 (LWP 100189)] [New Thread 0x31de200 (LWP 100182)] [New Thread 0x31e2c00 (LWP 100181)] [New Thread 0x31dee00 (LWP 100180)] [New Thread 0xe33c00 (LWP 100173)] [New Thread 0x2a7b800 (LWP 100171)] [New Thread 0xe68400 (LWP 100169)] [New Thread 0x31f6800 (LWP 100165)] [New Thread 0xe33a00 (LWP 100164)] [New Thread 0xbbd200 (LWP 100163)] [New Thread 0x31e0400 (LWP 100161)] [New Thread 0xddcc00 (LWP 100157)] [New Thread 0xddc000 (LWP 100156)] [New Thread 0x31fa200 (LWP 100152)] [New Thread 0x31f8e00 (LWP 100149)] [New Thread 0x31f8c00 (LWP 100148)] [New Thread 0x31f6c00 (LWP 100392)] [New Thread 0x31f6400 (LWP 100387)] [New Thread 0x31f3a00 (LWP 100383)] [New Thread 0x31f3600 (LWP 100381)] [New Thread 0x31ecc00 (LWP 100368)] [New Thread 0x31ea800 (LWP 100358)] [New Thread 0x31e7800 (LWP 100350)] [New Thread 0x31e7600 (LWP 100349)] [New Thread 0x31e6e00 (LWP 100345)] [New Thread 0x31e6c00 (LWP 100344)] [New Thread 0x31e6800 (LWP 100342)] [New Thread 0x31e4e00 (LWP 100337)] [New Thread 0x31e3a00 (LWP 100327)] [New Thread 0x31e3400 (LWP 100324)] [New Thread 0x31e3200 (LWP 100323)] [New Thread 0x31e2800 (LWP 100318)] [New Thread 0x31e2400 (LWP 100316)] [New Thread 0x31e2200 (LWP 100315)] [New Thread 0x31de800 (LWP 100294)] [New Thread 0x31dc800 (LWP 100289)] [New Thread 0x31dc200 (LWP 100283)] [New Thread 0x2ac6200 (LWP 100277)] [New Thread 0x2ac6000 (LWP 100276)] [New Thread 0x26bb200 (LWP 100261)] [New Thread 0x1dc3000 (LWP 100252)] [New Thread 0x1a53600 (LWP 100247)] [New Thread 0x1346a00 (LWP 100241)] [New Thread 0x1346000 (LWP 100235)] [New Thread 0xf83400 (LWP 100228)] [New Thread 0xf3a400 (LWP 100219)] [New Thread 0xef6c00 (LWP 100216)] [New Thread 0xef6a00 (LWP 100215)] [New Thread 0xe68a00 (LWP 100206)] [New Thread 0xddc400 (LWP 100188)] [New Thread 0xddc200 (LWP 100187)] [New Thread 0xda7e00 (LWP 100185)] [New Thread 0xda7c00 (LWP 100184)] [New Thread 0xda7a00 (LWP 100183)] [New Thread 0xd59a00 (LWP 100175)] [New Thread 0xd59800 (LWP 100174)] [New Thread 0xd59000 (LWP 100170)] [New Thread 0xb4da00 (LWP 100159)] [New Thread 0xb4d800 (LWP 100158)] [New Thread 0xb4d200 (LWP 100155)] [New Thread 0xafd000 (LWP 100146)] [New Thread 0x540e00 (LWP 100145)] [New Thread 0x540a00 (LWP 100144)] [New Thread 0x540800 (LWP 100107)] [New Thread 0x540600 (LWP 100065)] [New Thread 0x540000 (LWP 100110)] (gdb) bt #0 0x0000000800c080dc in thr_kill () from /lib/libc.so.6 #1 0x000000080098d01c in raise () from /usr/lib/libthr.so.2 #2 0x0000000800c8716d in abort () from /lib/libc.so.6 #3 0x000000080066ab99 in lbv_assert (func=0x431288 "SES_Delete", file=0x431002 "cache_session.c", line=338, cond=0x4312ba "(sp->obj) == 0", err=2) at assert.c:58 #4 0x000000000041a3f2 in SES_Delete (sp=0x15a3008) at cache_session.c:338 #5 0x0000000000408432 in vca_kev (kp=0x7fffff5fb330) at cache_acceptor_kqueue.c:111 #6 0x00000000004086ed in vca_kqueue_main (arg=0x0) at cache_acceptor_kqueue.c:150 #7 0x000000080098f55b in pthread_create () from /usr/lib/libthr.so.2 #8 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffff5fc000 (gdb) }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 11:18:14 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 11:18:14 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.d5535f3f6b9a651a7611421b243e09aa@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Comment (by anders): Updating to trunk/commit 2084, I get a sig11 core-dump with this back- trace: {{{ (gdb) bt #0 0x0000000800c5c46e in strtoul () from /lib/libc.so.6 #1 0x00000000004147ec in http_DissectResponse (w=0x7fffe331aae0, htc=0x7fffe3318980, hp=0x3dd4038) at cache_http.c:506 #2 0x0000000000411c12 in Fetch (sp=0xe53008) at cache_fetch.c:305 #3 0x000000000040e8b8 in cnt_fetch (sp=0xe53008) at cache_center.c:301 #4 0x000000000040faef in CNT_Session (sp=0xe53008) at steps.h:40 #5 0x0000000000417408 in wrk_do_one (w=0x7fffe331aae0) at cache_pool.c:193 #6 0x00000000004178c8 in wrk_thread (priv=0x85f300) at cache_pool.c:245 #7 0x000000080098f55b in pthread_create () from /usr/lib/libthr.so.2 #8 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffe331b000 }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 11:28:48 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 11:28:48 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.d5868787501778c0f5f84eac2da06a66@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): PHK wanted this additional information, for frame 4: {{{ (gdb) frame 4 #4 0x000000000041a3f2 in SES_Delete (sp=0x15a3008) at cache_session.c:338 338 AZ(sp->obj); (gdb) print *sp $1 = {magic = 741317722, fd = -1, id = 248, xid = 782563328, restarts = 0, wrk = 0x7fffe150bae0, sockaddrlen = 16, mysockaddrlen = 128, sockaddr = 0x15a3438, mysockaddr = 0x15a34b8, addr = "159.171.1.4", '\0' , port = "21500\000\000\000\000\000\000\000\000\000\000", srcaddr = 0x1862700, doclose = 0x0, http = 0x15a31d8, ws = {{s = 0x15a3538 "GET", f = 0x15a38ae "Date: Mon, 08 Oct 2007 10:46:00 GMT", r = 0x0, e = 0x15a7538 '?' ...}}, htc = {{magic = 1041886673, fd = 248, ws = 0x15a30a8, rxbuf = {b = 0x15a3538 "GET", e = 0x15a3876 ""}, pipeline = {b = 0x0, e = 0x0}}}, t_open = 1191840360.5254607, t_req = 1191840389.5482745, t_resp = nan(0x8000000000000), t_end = 1191840360.5254607, step = STP_FETCH, handling = 32, wantbody = 1 '\001', err_code = 200, err_reason = 0x0, list = {vtqe_next = 0x1164008, vtqe_prev = 0x26bc130}, backend = 0xaf9e00, bereq = 0x324e000, obj = 0x8cfed8000, vcl = 0x21016cc9a0, mem = 0x15a3000, workreq = {list = { vtqe_next = 0x0, vtqe_prev = 0x0}, sess = 0x15a3008}, acct = {first = 1191840347.6974971, sess = 1, req = 6, pipe = 0, pass = 0, fetch = 6, hdrbytes = 2035, bodybytes = 176725}, nhashptr = 6, ihashptr = 4, lhashptr = 51, hashptr = 0x15a3878} (gdb) }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 11:33:01 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 11:33:01 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.dab1072602885e98e0810caf71a8fc93@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): And this: {{{ (gdb) print *sp->obj Cannot access memory at address 0x8cfed8000 }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 12:03:46 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 12:03:46 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.3a42bf71093e55482ce1f0e2fb4ffbc6@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): Got a new assert, after updating to trunk/2090 - now trying on a similar (same OS release and processor type) system but which has 8 cores: {{{ Child said (2, 22376): <obj) == 0) not true. errno = 9 (Bad file descriptor) >> }}} The child process was attached to gdb, where I got: {{{ Program received signal SIGABRT, Aborted. [Switching to Thread 0x540e00 (LWP 100250)] 0x0000000800c3273c in thr_kill () at thr_kill.S:2 2 thr_kill.S: No such file or directory. in thr_kill.S Current language: auto; currently asm (gdb) bt #0 0x0000000800c3273c in thr_kill () at thr_kill.S:2 #1 0x00000008009a469f in _thr_send_sig (thread=0x540e00, sig=6) at /usr/src/lib/libthr/thread/thr_kern.c:91 #2 0x0000000800999ff8 in _raise (sig=6) at /usr/src/lib/libthr/thread/thr_syscalls.c:357 #3 0x0000000800ce4216 in abort () at /usr/src/lib/libc/stdlib/abort.c:69 #4 0x0000000800677ba9 in lbv_assert (func=0x42ddc8 "vca_kev", file=0x42dd69 "cache_acceptor_kqueue.c", line=91, cond=0x42de1d "(ss[j]->obj) == 0", err=9) at assert.c:58 #5 0x00000000004082d0 in vca_kev (kp=0x7fffff5fb570) at cache_acceptor_kqueue.c:91 #6 0x00000000004087bd in vca_kqueue_main (arg=0x0) at cache_acceptor_kqueue.c:151 #7 0x000000080099cc71 in thread_start (curthread=0x540e00) at /usr/src/lib/libthr/thread/thr_create.c:246 #8 0x0000000000000000 in ?? () Error accessing memory address 0x7fffff5fc000: Bad address. (gdb) }}} Switching to frame 5, I check ss[j]->obj: {{{ (gdb) frame 5 #5 0x00000000004082d0 in vca_kev (kp=0x7fffff5fb570) at cache_acceptor_kqueue.c:91 91 AZ(ss[j]->obj); (gdb) print ss[j]->obj $1 = (struct object *) 0x80a626000 }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 12:05:50 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 12:05:50 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.a738f0d9d18b9bb6fb72da6be4c720dd@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): PHK wanted this also (I am still in frame 5): {{{ (gdb) print *ss[j]->obj $2 = {magic = 847584578, refcnt = 1, xid = 1619417526, objhead = 0x4780c80, objstore = 0x4781908, ws_o = {{ s = 0x80a626318 "????\004?\220Q?_\225\006?\221>\231?\tb\b?\212?\0205\002?\207??\224????\024?*?K??A%Ex,\226M?\214??\213??\204?\034\214G?\204U???????H|????TsI>g\212?Ij???(4\223\027W?u8 \005T8?\220Q \r\022?c??", f = 0x80a626318 "????\004?\220Q?_\225\006?\221>\231?\tb\b?\212?\0205\002?\207??\224????\024?*?K??A%Ex,\226M?\214??\213??\204?\034\214G?\204U???????H|????TsI>g\212?Ij???(4\223\027W?u8 \005T8?\220Q \r\022?c??", r = 0x0, e = 0x80a62a000 "B\035\2052\001"}}, vary = 0x0, heap_idx = 0, ban_seq = 1, pass = 0, response = 0, valid = 0, cacheable = 0, busy = 1, len = 0, age = 0, entered = 0, ttl = 0, last_modified = 0, http = {{ magic = 1680389577, ws = 0x80a626020, conds = 0 '\0', logtag = 0, status = 0, hd = {{b = 0x0, e = 0x0} }, hdf = '\0' , nhd = 5}}, list = {vtqe_next = 0x0, vtqe_prev = 0x4780c98}, deathrow = {vtqe_next = 0x0, vtqe_prev = 0x0}, store = {vtqh_first = 0x0, vtqh_last = 0x80a6262f0}, waitinglist = { vtqh_first = 0x0, vtqh_last = 0x80a626300}, lru_stamp = 1191844455.703501} }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 12:08:55 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 12:08:55 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.434570fe5bb6a1a1393264b5332db13e@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): More data (still in frame 5): {{{ (gdb) print *ss[j] $3 = {magic = 741317722, fd = 411, id = 411, xid = 1619417526, restarts = 0, wrk = 0x7fffcea76ae0, sockaddrlen = 16, mysockaddrlen = 128, sockaddr = 0x1c29438, mysockaddr = 0x1c294b8, addr = "194.19.70.2", '\0' , port = "13123\000\000\000\000\000\000\000\000\000\000", srcaddr = 0xd36100, doclose = 0x0, http = 0x1c291d8, ws = {{s = 0x1c29538 "GET", f = 0x1c297b8 "nk", r = 0x0, e = 0x1c2d538 ""}}, htc = {{ magic = 1041886673, fd = 411, ws = 0x1c290a8, rxbuf = { b = 0x1c29538 "GET", e = 0x1c29780 "<\225?\001"}, pipeline = {b = 0x0, e = 0x0}}}, t_open = 1191844455.6478961, t_req = 1191844455.703501, t_resp = nan(0x8000000000000), t_end = 1191844455.6478961, step = STP_FETCH, handling = 32, wantbody = 1 '\001', err_code = 200, err_reason = 0x0, list = {vtqe_next = 0x2588008, vtqe_prev = 0x1442130}, backend = 0xafc000, bereq = 0x42e9000, obj = 0x80a626000, vcl = 0x2d817309a0, mem = 0x1c29000, workreq = {list = {vtqe_next = 0x0, vtqe_prev = 0x538090}, sess = 0x1c29008}, acct = {first = 1191844422.4304223, sess = 1, req = 11, pipe = 0, pass = 0, fetch = 9, hdrbytes = 3665, bodybytes = 39831}, nhashptr = 6, ihashptr = 4, lhashptr = 53, hashptr = 0x1c29780} }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 12:17:12 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 12:17:12 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.a8e38071e91bf10cf959c77588a70850@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): More data (still in frame 5): {{{ (gdb) print *ss[j] $3 = {magic = 741317722, fd = 411, id = 411, xid = 1619417526, restarts = 0, wrk = 0x7fffcea76ae0, sockaddrlen = 16, mysockaddrlen = 128, sockaddr = 0x1c29438, mysockaddr = 0x1c294b8, addr = "194.19.70.2", '\0' , port = "13123\000\000\000\000\000\000\000\000\000\000", srcaddr = 0xd36100, doclose = 0x0, http = 0x1c291d8, ws = {{s = 0x1c29538 "GET", f = 0x1c297b8 "nk", r = 0x0, e = 0x1c2d538 ""}}, htc = {{ magic = 1041886673, fd = 411, ws = 0x1c290a8, rxbuf = { b = 0x1c29538 "GET", e = 0x1c29780 "<\225?\001"}, pipeline = {b = 0x0, e = 0x0}}}, t_open = 1191844455.6478961, t_req = 1191844455.703501, t_resp = nan(0x8000000000000), t_end = 1191844455.6478961, step = STP_FETCH, handling = 32, wantbody = 1 '\001', err_code = 200, err_reason = 0x0, list = {vtqe_next = 0x2588008, vtqe_prev = 0x1442130}, backend = 0xafc000, bereq = 0x42e9000, obj = 0x80a626000, vcl = 0x2d817309a0, mem = 0x1c29000, workreq = {list = {vtqe_next = 0x0, vtqe_prev = 0x538090}, sess = 0x1c29008}, acct = {first = 1191844422.4304223, sess = 1, req = 11, pipe = 0, pass = 0, fetch = 9, hdrbytes = 3665, bodybytes = 39831}, nhashptr = 6, ihashptr = 4, lhashptr = 53, hashptr = 0x1c29780} }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 13:37:48 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 13:37:48 -0000 Subject: [Varnish] #156: Varnish 1.1 branch dies with assert error in WS_ReleaseP In-Reply-To: <052.9636620d22b6f890a89073c2cf8ba817@projects.linpro.no> References: <052.9636620d22b6f890a89073c2cf8ba817@projects.linpro.no> Message-ID: <061.36fac5cea26aef7531efaa017b1d0cf6@projects.linpro.no> #156: Varnish 1.1 branch dies with assert error in WS_ReleaseP -----------------------------------+---------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: 1.1 Severity: major | Resolution: Keywords: core dump WS_ReleaseP | -----------------------------------+---------------------------------------- Comment (by anders): Running Varnish/trunk 2090, this happened again: {{{ Child said (2, 36889): <r != NULL) not true. errno = 2 (No such file or directory) }}} Backtrace (child is attached to gdb): {{{ (gdb) bt #0 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 #1 0x00000008009a735d in __thr_umtx_lock (mtx=0xafaf80, id=101364) at /usr/src/lib/libthr/thread/thr_umtx.c:36 #2 0x00000008009a18ec in _thr_umtx_lock (mtx=0xafaf80, id=101364) at thr_umtx.h:62 #3 0x00000008009a249a in mutex_lock_common (curthread=0x3549400, m=0x800e13a80, abstime=0x0) at /usr/src/lib/libthr/thread/thr_mutex.c:527 #4 0x00000008009a2fc3 in _pthread_mutex_lock (m=0x800e13a80) at /usr/src/lib/libthr/thread/thr_mutex.c:796 #5 0x0000000800ce737b in __Bfree_D2A (v=0x2a6b140) at gdtoa_misc.c:98 #6 0x0000000800ce2a1b in __dtoa (d=1191850327.892849, mode=3, ndigits=9, decpt=0x7fff820106bc, sign=0x7fff820106d4, rve=0x7fff820106b0) at gdtoa_dtoa.c:735 #7 0x0000000800cdb49d in __vfprintf (fp=0x7fff820107f0, fmt0=0x42f281 "%u %.9f %.9f %.9f %.9f %.9f", ap=0x7fff82010900) at /usr/src/lib/libc/stdio/vfprintf.c:903 #8 0x0000000800c36451 in vsnprintf ( str=0x7fff82010f9b "2026381664 1191850327.892454863\001?2026378149 1191850325.742480040 1191850326.912809134 0.063056231 1.169573545 0.000755548", n=255, fmt=0x42f281 "%u %.9f %.9f %.9f %.9f %.9f", ap=0x7fff82010900) at /usr/src/lib/libc/stdio/vsnprintf.c:75 #9 0x0000000000428c55 in WSL (w=0x7fff82012ae0, tag=SLT_ReqEnd, id=3823, fmt=0x42f281 "%u %.9f %.9f %.9f %.9f %.9f") at shmlog.c:253 #10 0x000000000040e5fa in cnt_done (sp=0x1028008) at cache_center.c:205 #11 0x000000000040fc29 in CNT_Session (sp=0x1028008) at steps.h:43 #12 0x0000000000417538 in wrk_do_one (w=0x7fff82012ae0) at cache_pool.c:193 #13 0x00000000004179f8 in wrk_thread (priv=0x85b320) at cache_pool.c:245 #14 0x000000080099cc71 in thread_start (curthread=0x3549400) at /usr/src/lib/libthr/thread/thr_create.c:246 #15 0x0000000000000000 in ?? () Error accessing memory address 0x7fff82013000: Bad address. }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 13:48:06 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 13:48:06 -0000 Subject: [Varnish] #156: Varnish 1.1 branch dies with assert error in WS_ReleaseP In-Reply-To: <052.9636620d22b6f890a89073c2cf8ba817@projects.linpro.no> References: <052.9636620d22b6f890a89073c2cf8ba817@projects.linpro.no> Message-ID: <061.8c13fb4f2bc500f22185c2e1f74c9a29@projects.linpro.no> #156: Varnish 1.1 branch dies with assert error in WS_ReleaseP -----------------------------------+---------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: 1.1 Severity: major | Resolution: Keywords: core dump WS_ReleaseP | -----------------------------------+---------------------------------------- Comment (by anders): PHK wanted trace from thread 692: {{{ (gdb) thread 692 [Switching to thread 692 (Thread 0x540e00 (LWP 100362))]#0 0x0000000800c3273c in thr_kill () at thr_kill.S:2 2 thr_kill.S: No such file or directory. in thr_kill.S (gdb) bt #0 0x0000000800c3273c in thr_kill () at thr_kill.S:2 #1 0x00000008009a469f in _thr_send_sig (thread=0x540e00, sig=6) at /usr/src/lib/libthr/thread/thr_kern.c:91 #2 0x0000000800999ff8 in _raise (sig=6) at /usr/src/lib/libthr/thread/thr_syscalls.c:357 #3 0x0000000800ce4216 in abort () at /usr/src/lib/libc/stdlib/abort.c:69 #4 0x0000000800677ba9 in lbv_assert (func=0x432853 "WS_ReleaseP", file=0x43275b "cache_ws.c", line=125, cond=0x43282e "ws->r != NULL", err=2) at assert.c:58 #5 0x0000000000420575 in WS_ReleaseP (ws=0x51420a8, ptr=0x5142538 "GET") at cache_ws.c:125 #6 0x0000000000416a09 in HTC_Rx (htc=0x51420c8) at cache_httpconn.c:143 #7 0x0000000000408443 in vca_kev (kp=0x7fffff5fbc70) at cache_acceptor_kqueue.c:103 #8 0x00000000004087fd in vca_kqueue_main (arg=0x0) at cache_acceptor_kqueue.c:152 #9 0x000000080099cc71 in thread_start (curthread=0x540e00) at /usr/src/lib/libthr/thread/thr_create.c:246 #10 0x0000000000000000 in ?? () Error accessing memory address 0x7fffff5fc000: Bad address. }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 14:41:46 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 14:41:46 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.282abf41f3e8a557309457d91a1c6f25@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): Got this one: {{{ Child said (2, 38243): <vcl) == 0) not true. errno = 2 (No such file or directory) }}} Backtrace looks like this: {{{ (gdb) bt #0 0x0000000800c3273c in thr_kill () at thr_kill.S:2 #1 0x00000008009a469f in _thr_send_sig (thread=0x540e00, sig=6) at /usr/src/lib/libthr/thread/thr_kern.c:91 #2 0x0000000800999ff8 in _raise (sig=6) at /usr/src/lib/libthr/thread/thr_syscalls.c:357 #3 0x0000000800ce4216 in abort () at /usr/src/lib/libc/stdlib/abort.c:69 #4 0x0000000800677ba9 in lbv_assert (func=0x431248 "SES_Delete", file=0x430fc2 "cache_session.c", line=339, cond=0x431289 "(sp->vcl) == 0", err=2) at assert.c:58 #5 0x000000000041a421 in SES_Delete (sp=0x28f0008) at cache_session.c:339 #6 0x0000000000408540 in vca_kev (kp=0x7fffff5fb330) at cache_acceptor_kqueue.c:113 #7 0x00000000004087fd in vca_kqueue_main (arg=0x0) at cache_acceptor_kqueue.c:152 #8 0x000000080099cc71 in thread_start (curthread=0x540e00) at /usr/src/lib/libthr/thread/thr_create.c:246 #9 0x0000000000000000 in ?? () Error accessing memory address 0x7fffff5fc000: Bad address. }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 15:03:03 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 15:03:03 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.e19f626db64526c57e6a250117a26c1c@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): To check for spurious piperead/pipewrite, I use http://phk.freebsd.dk/misc/anordby.patch and log with this command: varnishlog | grep ANORDBY >anordby.log 2>&1 I use this program to check the output: {{{ #! /usr/bin/perl %he=(); while () { $line = $_; if (/ANORDBY (.*) 0x([0-9a-f]+)/) { $cmdtxt = $1; $hex = $2; if (exists $he{"$hex"}) { if ($cmdtxt eq "piperead") { if ($he{"$hex"}{"pipewrite"} == 1) { delete($he{"$hex"}); } else { $he{"$hex"}{"txt"} .= $_; } } elsif ($cmdtxt eq "pipewrite") { if ($he{"$hex"}{"pipewrite"} == 1) { delete($he{"$hex"}); } else { $he{"$hex"}{"txt"} .= $_; } } else { $he{"$hex"}{"txt"} .= $_; } } else { $he{"$hex"}{"txt"} .= $_; if ($cmdtxt eq "piperead") { $he{"$hex"}{"piperead"} = 1; } elsif ($cmdtxt eq "pipewrite") { $he{"$hex"}{"pipewrite"} = 1; } $he{"$hex"}{"txt"} .= $_; } } } foreach $l (keys %he) { print "Address $l =>\n"; print $he{"$l"}{"txt"} . "\n"; } }}} This shows a number of pipewrite pairs, with no read: {{{ Address 1a53008 => 0 Debug "ANORDBY pipewrite 0x1a53008" 0 Debug "ANORDBY pipewrite 0x1a53008" Address 224f008 => 0 Debug "ANORDBY pipewrite 0x224f008" 0 Debug "ANORDBY pipewrite 0x224f008" Address 3231008 => 0 Debug "ANORDBY pipewrite 0x3231008" 0 Debug "ANORDBY pipewrite 0x3231008" (..) }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 16:37:39 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 16:37:39 -0000 Subject: [Varnish] #99: returning pass from vcl_hit crashes the child In-Reply-To: <060.bc6d7510b64f29dc7608b3e7810cf372@projects.linpro.no> References: <060.bc6d7510b64f29dc7608b3e7810cf372@projects.linpro.no> Message-ID: <069.0c28ce4cb08676d3ce777ded4a20a239@projects.linpro.no> #99: returning pass from vcl_hit crashes the child ----------------------------+----------------------------------------------- Reporter: denis at zeno.org | Owner: des Type: defect | Status: closed Priority: normal | Milestone: Component: varnishd | Version: 1.0.3 Severity: normal | Resolution: fixed Keywords: | ----------------------------+----------------------------------------------- Changes (by denis at zeno.org): * status: assigned => closed * resolution: => fixed Comment: this is fixed with change in r1971 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 18:22:43 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 18:22:43 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.ac357597d88cbfdb3cebddf947c36b0b@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Comment (by anders): Updated to trunk/2094, and got another signal 11 core dump. Backtrace from GDB being attached to the child process: {{{ Program received signal SIGSEGV, Segmentation fault. 0x0000000800ca7349 in strtoul (nptr=0x0, endptr=0x0, base=10) at /usr/src/lib/libc/stdlib/strtoul.c:65 65 c = *s++; (gdb) bt #0 0x0000000800ca7349 in strtoul (nptr=0x0, endptr=0x0, base=10) at /usr/src/lib/libc/stdlib/strtoul.c:65 #1 0x0000000000414958 in http_DissectResponse (w=0x7fff8c665ae0, htc=0x7fff8c663980, hp=0x3861038) at cache_http.c:509 #2 0x0000000000411d82 in Fetch (sp=0x66f3008) at cache_fetch.c:305 #3 0x000000000040e998 in cnt_fetch (sp=0x66f3008) at cache_center.c:301 #4 0x000000000040fc5e in CNT_Session (sp=0x66f3008) at steps.h:40 #5 0x00000000004175a8 in wrk_do_one (w=0x7fff8c665ae0) at cache_pool.c:193 #6 0x0000000000417a68 in wrk_thread (priv=0x85b320) at cache_pool.c:245 #7 0x000000080099cc71 in thread_start (curthread=0xc07e00) at /usr/src/lib/libthr/thread/thr_create.c:246 #8 0x00007fff8c466000 in ?? () Error accessing memory address 0x7fff8c666000: Bad address. }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 18:29:50 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 18:29:50 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.32aadf741b5b93e786e04e5b10b4b606@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Comment (by anders): PHK wanted additional information from the latest GDB session: {{{ (gdb) frame 1 #1 0x0000000000414958 in http_DissectResponse (w=0x7fff8c665ae0, htc=0x7fff8c663980, hp=0x3861038) at cache_http.c:509 509 hp->status = (gdb) print *hp $1 = {magic = 1680389577, ws = 0x3861018, conds = 0 '\0', logtag = HTTP_Rx, status = 200, hd = {{b = 0x4306e8 "GET", e = 0x4306eb ""}, { b = 0x66f353c "/mmo/9/113/849/69_1814361711.jpg", e = 0x66f355c ""}, { b = 0x3861280 "HTTP/1.1", e = 0x430542 ""}, {b = 0x0, e = 0x0}, { b = 0x0, e = 0x0}, {b = 0x66f3567 "Accept: */*", e = 0x66f3572 ""}, { b = 0x66f3574 "Referer: http://www.finn.no/finn/viewimage?finnkode=11384969&reference=9/113/849/69_-188942486.jpg&adheading=Dressur%2F+sprang+hest+selges&adTypeId=67&WT.svl=Minibilde", e = 0x66f361b ""}, { b = 0x66f361d "Accept-Language: no", e = 0x66f3630 ""}, { b = 0x66f3632 "Accept-Encoding: gzip, deflate", e = 0x66f3650 ""}, { b = 0x66f3652 "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)", e = 0x66f3695 ""}, {b = 0x66f3697 "Host: cache.finn.no", e = 0x66f36aa ""}, { b = 0x66f36c4 "Cookie: IADVISITOR=135410426; RMID=536d34d144c3aba0; wid=83.109.52.209-909739040.29798010::A285C829BB8ACAB327CB63B2C69459FB; WT_FPC=id=83.109.52.209-909739040.29798010%3A%3AA285C829BB8ACAB327CB63B2C69"..., e = 0x66f38ca ""}, {b = 0x3861993 "X-Varnish: 2125423896", e = 0x38619a8 ""}, {b = 0x38619a9 "X-Forwarded-for: 83.109.9.25", e = 0x38619c5 ""}, {b = 0x0, e = 0x0} }, hdf = '\0' , nhd = 14} }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Oct 9 08:32:00 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 09 Oct 2007 08:32:00 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.9fa329b54a51e3ba0a02a94a1bae7b06@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by anders): For the record. Using this patch with trunk/2095: {{{ --- cache_acceptor_kqueue.c (revision 2088) +++ cache_acceptor_kqueue.c (working copy) @@ -28,9 +28,6 @@ * * $Id$ * - * XXX: We need to pass sessions back into the event engine when they are - * reused. Not sure what the most efficient way is for that. For now - * write the session pointer to a pipe which the event engine monitors. */ #if defined(HAVE_KQUEUE) @@ -141,8 +138,12 @@ assert(n >= 1 && n <= NKEV); nki = 0; for (kp = ke, j = 0; j < n; j++, kp++) { - if (kp->flags & EV_ERROR) + if (kp->flags & EV_ERROR) { +VSL(SLT_Debug, 0, "KQTROUBLE id %p filt %d flags %d fflags %d data %d ptr %p\n" , + (void*)kp->ident, kp->filter, kp->flags, kp->fflags, + kp->data, kp->udata); continue; +} if (kp->filter == EVFILT_TIMER) { dotimer = 1; continue; }}} I got these KQTROUBLE messages: {{{ 0 Debug "KQTROUBLE id 0x212 filt -1 flags 16384 fflags 0 data 9 ptr 0x1a86008%0a" 0 Debug "KQTROUBLE id 0x276 filt -1 flags 16384 fflags 0 data 9 ptr 0x241d008%0a" 0 Debug "KQTROUBLE id 0xd7 filt -1 flags 16384 fflags 0 data 9 ptr 0x4141008%0a" 0 Debug "KQTROUBLE id 0x2f7 filt -1 flags 16384 fflags 0 data 9 ptr 0x2d7f008%0a" 0 Debug "KQTROUBLE id 0x540 filt -1 flags 16384 fflags 0 data 9 ptr 0x2e41008%0a" 0 Debug "KQTROUBLE id 0x1a filt -1 flags 16384 fflags 0 data 9 ptr 0x1cf9008%0a" 0 Debug "KQTROUBLE id 0x78 filt -1 flags 16384 fflags 0 data 9 ptr 0xd16008%0a" 0 Debug "KQTROUBLE id 0x294 filt -1 flags 16384 fflags 0 data 9 ptr 0x118d008%0a" 0 Debug "KQTROUBLE id 0x3de filt -1 flags 16384 fflags 0 data 9 ptr 0x3435008%0a" 0 Debug "KQTROUBLE id 0x40e filt -1 flags 16384 fflags 0 data 9 ptr 0x2941008%0a" 0 Debug "KQTROUBLE id 0x18e filt -1 flags 16384 fflags 0 data 9 ptr 0x3512008%0a" }}} On the console: {{{ Child said (2, 66296): <obj) == 0) not true. errno = 9 (Bad file descriptor) }}} Backtrace: {{{ Program received signal SIGABRT, Aborted. [Switching to Thread 0x541e00 (LWP 100236)] 0x0000000800c3373c in thr_kill () at thr_kill.S:2 2 thr_kill.S: No such file or directory. in thr_kill.S Current language: auto; currently asm (gdb) bt #0 0x0000000800c3373c in thr_kill () at thr_kill.S:2 #1 0x00000008009a569f in _thr_send_sig (thread=0x541e00, sig=6) at /usr/src/lib/libthr/thread/thr_kern.c:91 #2 0x000000080099aff8 in _raise (sig=6) at /usr/src/lib/libthr/thread/thr_syscalls.c:357 #3 0x0000000800ce5216 in abort () at /usr/src/lib/libc/stdlib/abort.c:69 #4 0x0000000800678ba9 in lbv_assert (func=0x4313d8 "SES_Delete", file=0x431152 "cache_session.c", line=338, cond=0x43140a "(sp->obj) == 0", err=9) at assert.c:58 #5 0x000000000041a522 in SES_Delete (sp=0x2b5a008) at cache_session.c:338 #6 0x000000000040856a in vca_kev (kp=0x7fffff5fb350) at cache_acceptor_kqueue.c:109 #7 0x0000000000408889 in vca_kqueue_main (arg=0x0) at cache_acceptor_kqueue.c:152 #8 0x000000080099dc71 in thread_start (curthread=0x541e00) at /usr/src/lib/libthr/thread/thr_create.c:246 #9 0x0000000000000000 in ?? () Error accessing memory address 0x7fffff5fc000: Bad address. (gdb) frame 5 #5 0x000000000041a522 in SES_Delete (sp=0x2b5a008) at cache_session.c:338 338 AZ(sp->obj); Current language: auto; currently c (gdb) print sp->id $1 = 163 }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Oct 9 09:17:13 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 09 Oct 2007 09:17:13 -0000 Subject: [Varnish] #162: Varnish trunk dies with assert error in SES_Delete() In-Reply-To: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> References: <052.06515aaec000ba8f6489d0b0e4ad254c@projects.linpro.no> Message-ID: <061.18739a198c4b7edd6cc5605f1de568f1@projects.linpro.no> #162: Varnish trunk dies with assert error in SES_Delete() ---------------------------------------------------------+------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump SES_Delete cache_session | ---------------------------------------------------------+------------------ Comment (by phk): = A sort of status report = As is appearant from the above, this bug is quite elusive, and I can't really claim to be any closer to its resolution now, than I have been earlier. The exact triggering condition in SES_Delete() is patently impossible from a reading of the source code, and the sanity of the sp->elements considered, this is not a random pointer tango. About the only explanation that makes sense is that a session gets started twice from the acceptor or possibly sent there twice. Other less plausible explanations are locking bugs in the pipe(2) code, hardware cache coherency bugs or memory barrier deficiencies. Based on rather light evidence so far, it seems that using the poll_acceptor either eliminated or at least reduces the frequency drastically, pointing somewhat in the direction of kqueue_acceptor. Despite the poll_acceptor soaking up considerably more CPU time, I think we should continue to run with it for some days, to see if it totally eliminates this problem. If after some days, we find that to be the case, the next step is probably to update the machine from 6.2-R to RELENG_6 head, try the kqueue acceptor and see if the trouble still exists. If it does, a further update to either FreeBSD-current or RELENG_7 might be a good idea. In the meantime I will try to see if I can spot anything in kqueue_acceptor that doesn't work as expected, and to see if I can reproduce the problem in my lab. I may also add some flags and asserts to try to catch sessions which get started more than once. Poul-Henning -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Oct 9 10:29:20 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 09 Oct 2007 10:29:20 -0000 Subject: [Varnish] #167: Varnishd crashes with assert error in RES_WriteObj Message-ID: <052.0d60ec784fa5d17383428842e133f8e7@projects.linpro.no> #167: Varnishd crashes with assert error in RES_WriteObj ----------------------+----------------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Keywords: varnishd core dump RES_WriteObj ----------------------+----------------------------------------------------- I am running Varnish trunk (up to date to commit 2096) with vca_act set to vca_acceptors[1] to use polling instead of kqueue, due to recent kqueue problems discovered in ticket #162. After a short time, Varnish crashed: {{{ Child said (2, 25882): <obj->len) not true. errno = 32 (Broken pipe) }}} My VCL: {{{ backend default { set backend.host = "192.168.0.3"; set backend.port = "80"; } acl purge { "192.168.0.4"/32; } sub vcl_recv { if ((req.http.host ~ "^(cache.finn.no|finn.no|www.finn.no)$") || (req.http.host == "banner.finn.no" && req.url ~ "^/(jsp2|finn/gojsp|daily|board|auximg/papirfly|finn/cacheable|crossdomain.xml)")) { if (req.request == "GET" || req.request == "HEAD") { lookup; } elsif (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } lookup; } else { pipe; } } else { error 403 "Access denied. Contact cacheadmin at finn.no if you have problems."; } } sub vcl_miss { if (req.request == "PURGE") { error 404 "Not in cache."; } else { fetch; } } sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } else { deliver; } } sub vcl_fetch { if (obj.ttl < 86400s) { set obj.ttl = 604800s; } if (obj.http.Cookie) { remove obj.http.Cookie; } if (obj.http.Set-Cookie) { remove obj.http.Set-Cookie; } insert; } }}} Backtrace shows: {{{ Program received signal SIGABRT, Aborted. [Switching to Thread 0x32ffe00 (LWP 100463)] 0x0000000800c080dc in thr_kill () from /lib/libc.so.6 (gdb) bt #0 0x0000000800c080dc in thr_kill () from /lib/libc.so.6 #1 0x000000080098d01c in raise () from /usr/lib/libthr.so.2 #2 0x0000000800c8716d in abort () from /lib/libc.so.6 #3 0x000000080066ab99 in lbv_assert (func=0x430fbc "RES_WriteObj", file=0x430f76 "cache_response.c", line=184, cond=0x430ff0 "u == sp->obj->len", err=32) at assert.c:58 #4 0x0000000000419412 in RES_WriteObj (sp=0x3fc5008) at cache_response.c:184 #5 0x000000000040e353 in cnt_deliver (sp=0x3fc5008) at cache_center.c:162 #6 0x000000000040fc4c in CNT_Session (sp=0x3fc5008) at steps.h:41 #7 0x0000000000417588 in wrk_do_one (w=0x7fffd9aceae0) at cache_pool.c:193 #8 0x0000000000417a48 in wrk_thread (priv=0x85b300) at cache_pool.c:245 #9 0x000000080098f55b in pthread_create () from /usr/lib/libthr.so.2 #10 0x0000000000000000 in ?? () Error accessing memory address 0x7fffd9acf000: Bad address. (gdb) frame 4 #4 0x0000000000419412 in RES_WriteObj (sp=0x3fc5008) at cache_response.c:184 184 assert(u == sp->obj->len); (gdb) print sp->obj->len $1 = 1525 (gdb) print u $2 = 0 }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Oct 9 14:16:38 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 09 Oct 2007 14:16:38 -0000 Subject: [Varnish] #168: varnish crashes when using varnishadmin.pl (from the list) with wrong parameters Message-ID: <060.a25c1357e596340e3b60534af2469250@projects.linpro.no> #168: varnish crashes when using varnishadmin.pl (from the list) with wrong parameters ----------------------------+----------------------------------------------- Reporter: denis at zeno.org | Owner: des Type: defect | Status: new Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Keywords: coredump ----------------------------+----------------------------------------------- when using varnishadmin.pl with "help param.list" as parameter varnishd in debug mode throws the following assertion error: Assert error in mgt_cli_callback(), mgt_cli.c line 352: Condition((close(cp->fdo)) == 0) not true. errno = 9 (Bad file descriptor) Abort trap (core dumped) varnish version is 2096 Oct 9 16:13:56 zeno02-test kernel: pid 25968 (varnishd), uid 0: exited on signal 6 (core dumped) -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Oct 9 17:58:12 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 09 Oct 2007 17:58:12 -0000 Subject: [Varnish] #169: Varnishd crashes with assert error in bes_ClosedFd Message-ID: <052.42606c7ce4f70ce00764b8871fcb46b7@projects.linpro.no> #169: Varnishd crashes with assert error in bes_ClosedFd ----------------------+----------------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Keywords: ----------------------+----------------------------------------------------- I am running Varnish trunk (up to date to commit 2096) with vca_act set to vca_acceptors[1] to use polling instead of kqueue, due to recent kqueue problems discovered in ticket #162. After some time, Varnish crashed: {{{ Child said (2, 882): <fd)) == 0) not true. errno = 57 (Invalid argument) Assert error in bes_ClosedFd(), cache_backend_simple.c line 282: Condition((close(vc->fd)) == 0) not true. errno = 22 (Invalid argument) }}} My VCL: {{{ backend default { set backend.host = "192.168.0.3"; set backend.port = "80"; } acl purge { "192.168.0.4"/32; } sub vcl_recv { if ((req.http.host ~ "^(cache.finn.no|finn.no|www.finn.no)$") || (req.http.host == "banner.finn.no" && req.url ~ "^/(jsp2|finn/gojsp|daily|board|auximg/papirfly|finn/cacheable|crossdomain.xml)")) { if (req.request == "GET" || req.request == "HEAD") { lookup; } elsif (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } lookup; } else { pipe; } } else { error 403 "Access denied. Contact cacheadmin at finn.no if you have problems."; } } sub vcl_miss { if (req.request == "PURGE") { error 404 "Not in cache."; } else { fetch; } } sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } else { deliver; } } sub vcl_fetch { if (obj.ttl < 86400s) { set obj.ttl = 604800s; } if (obj.http.Cookie) { remove obj.http.Cookie; } if (obj.http.Set-Cookie) { remove obj.http.Set-Cookie; } insert; } }}} The backtrace: {{{ Program received signal SIGABRT, Aborted. [Switching to Thread 0xf02f400 (LWP 100556)] 0x0000000800c080dc in thr_kill () from /lib/libc.so.6 (gdb) bt #0 0x0000000800c080dc in thr_kill () from /lib/libc.so.6 #1 0x000000080098d01c in raise () from /usr/lib/libthr.so.2 #2 0x0000000800c8716d in abort () from /lib/libc.so.6 #3 0x000000080066ab99 in lbv_assert (func=0x42ef79 "bes_ClosedFd", file=0x42ed80 "cache_backend_simple.c", line=282, cond=0x42efee "(close(vc->fd)) == 0", err=22) at assert.c:58 #4 0x000000000040d51e in bes_ClosedFd (w=0x7fffce071ae0, vc=0x51fa900) at cache_backend_simple.c:282 #5 0x00000000004097a0 in VBE_ClosedFd (w=0x7fffce071ae0, vc=0x51fa900) at cache_backend.c:291 #6 0x000000000041215c in Fetch (sp=0x24ce008) at cache_fetch.c:385 #7 0x000000000040e978 in cnt_fetch (sp=0x24ce008) at cache_center.c:301 #8 0x000000000040fc3e in CNT_Session (sp=0x24ce008) at steps.h:40 #9 0x0000000000417588 in wrk_do_one (w=0x7fffce071ae0) at cache_pool.c:193 #10 0x0000000000417a48 in wrk_thread (priv=0x85b300) at cache_pool.c:245 #11 0x000000080098f55b in pthread_create () from /usr/lib/libthr.so.2 #12 0x0000000000000000 in ?? () Error accessing memory address 0x7fffce072000: Bad address. (gdb) frame 4 #4 0x000000000040d51e in bes_ClosedFd (w=0x7fffce071ae0, vc=0x51fa900) at cache_backend_simple.c:282 282 AZ(close(vc->fd)); (gdb) print vc->fd $1 = 479 }}} PS: This was while the FreeBSD kernel seemed to be in a bit frozen state, with no/little response. Suddenly it loosened and I could log in. Then I could see this error. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Oct 9 18:32:54 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 09 Oct 2007 18:32:54 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.754411da4f7e0f2e649a2ce3558a5433@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Comment (by anders): Got this same error on another server, which has less cores (two physical with HT - four logical) and a different Intel CPU. Now using trunk/2096 with poll instead of kqueue. Backtrace: {{{ Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x484ce00 (LWP 100375)] 0x0000000800c5c46e in strtoul () from /lib/libc.so.6 (gdb) bt #0 0x0000000800c5c46e in strtoul () from /lib/libc.so.6 #1 0x0000000000414938 in http_DissectResponse (w=0x7fffe4925ae0, htc=0x7fffe4923980, hp=0x550d038) at cache_http.c:509 #2 0x0000000000411d62 in Fetch (sp=0x45fd008) at cache_fetch.c:305 #3 0x000000000040e978 in cnt_fetch (sp=0x45fd008) at cache_center.c:301 #4 0x000000000040fc3e in CNT_Session (sp=0x45fd008) at steps.h:40 #5 0x0000000000417588 in wrk_do_one (w=0x7fffe4925ae0) at cache_pool.c:193 #6 0x0000000000417a48 in wrk_thread (priv=0x85b300) at cache_pool.c:245 #7 0x000000080098f55b in pthread_create () from /usr/lib/libthr.so.2 #8 0x0000000000000000 in ?? () Error accessing memory address 0x7fffe4926000: Bad address. (gdb) frame 1 #1 0x0000000000414938 in http_DissectResponse (w=0x7fffe4925ae0, htc=0x7fffe4923980, hp=0x550d038) at cache_http.c:509 509 hp->status = (gdb) print *hp $1 = {magic = 1680389577, ws = 0x550d018, conds = 0 '\0', logtag = HTTP_Rx, status = 200, hd = {{b = 0x4306c8 "GET", e = 0x4306cb ""}, { b = 0x45fd53c "/mmo/3/113/481/53_1282926909_thumb.jpg", e = 0x45fd562 ""}, {b = 0x550d280 "HTTP/1.1", e = 0x430522 ""}, { b = 0x0, e = 0x0}, {b = 0x0, e = 0x0}, {b = 0x45fd56d "Accept: */*", e = 0x45fd578 ""}, { b = 0x45fd57a "Referer: http://finn.no/finn/car/used/result?CAR_MODEL/MAKE=771&CAR_MODEL/MODEL=1087", e = 0x45fd5ce ""}, { b = 0x45fd5d0 "Accept-Language: no", e = 0x45fd5e3 ""}, { b = 0x45fd5e5 "UA-CPU: x86", e = 0x45fd5f0 ""}, { b = 0x45fd5f2 "Accept-Encoding: gzip, deflate", e = 0x45fd610 ""}, { b = 0x45fd612 "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; IEMB3; IEMB3)", e = 0x45fd671 ""}, { b = 0x45fd673 "Host: cache.finn.no", e = 0x45fd686 ""}, { b = 0x45fd6a0 "Cookie: IAD_CAMPAIGN=_blank; IADVISITOR=336365720; RMID=5100b11d46f205f0; WT_FPC=id=2cfbd1b87fbfe49fdb51190263065921:lv=1191950818906:ss=1191950805078; finn_ld=aeb69f3baeb69f3a; JSESSIONID=tafkZdpJboq"..., e = 0x45fd77c ""}, {b = 0x550dacb "X-Varnish: 484821996", e = 0x550dadf ""}, {b = 0x550dae0 "X-Forwarded-for: 81.0.177.29", e = 0x550dafc ""}, {b = 0x0, e = 0x0} }, hdf = '\0' , nhd = 15} }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Oct 10 12:41:28 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 10 Oct 2007 12:41:28 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.eac5a4853a5dcdc34a271d696f65c585@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Comment (by anders): Now trying trunk/2096 with kqueue debug patch, on 7.0-CURRENT as of october 9. It happens again rather soon: {{{ (gdb) bt #0 0x0000000800c6a784 in strtoul () from /lib/libc.so.7 #1 0x0000000000414ffa in http_DissectResponse (w=0x7ffff0985ae0, htc=0x7ffff0983990, hp=0x2d821f8038) at cache_http.c:509 #2 0x00000000004121c7 in Fetch (sp=0x2d83e8d008) at cache_fetch.c:305 #3 0x000000000040ec92 in cnt_fetch (sp=0x2d83e8d008) at cache_center.c:301 #4 0x000000000040fffc in CNT_Session (sp=0x2d83e8d008) at steps.h:40 #5 0x0000000000417d0d in wrk_do_one (w=0x7ffff0985ae0) at cache_pool.c:193 #6 0x00000000004181b7 in wrk_thread (priv=0x800e111c0) at cache_pool.c:245 #7 0x0000000800992958 in pthread_getprio () from /lib/libthr.so.3 #8 0x00007ffff0786000 in ?? () Cannot access memory at address 0x7ffff0986000 (gdb) frame 1 #1 0x0000000000414ffa in http_DissectResponse (w=0x7ffff0985ae0, htc=0x7ffff0983990, hp=0x2d821f8038) at cache_http.c:509 509 hp->status = (gdb) print *hp $1 = {magic = 1680389577, ws = 0x2d821f8018, conds = 0 '\0', logtag = HTTP_Rx, status = 200, hd = {{b = 0x431a28 "GET", e = 0x431a2b ""}, { b = 0x2d83e8d53c "/mmo/1/110/915/61_-1259745306_tumb.jpg", e = 0x2d83e8d562 ""}, {b = 0x2d821f8280 "HTTP/1.1", e = 0x431882 ""}, { b = 0x0, e = 0x0}, {b = 0x0, e = 0x0}, {b = 0x2d83e8d56d "Accept: */*", e = 0x2d83e8d578 ""}, { b = 0x2d83e8d57a "Referer: http://finn.no/finn/bap/object?finnkode=11091561&sid=xz1abcnERcS077289&WT.svl=Link", e = 0x2d83e8d5d5 ""}, { b = 0x2d83e8d5d7 "Accept-Language: no", e = 0x2d83e8d5ea ""}, { b = 0x2d83e8d5ec "Accept-Encoding: gzip, deflate", e = 0x2d83e8d60a ""}, { b = 0x2d83e8d60c "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", e = 0x2d83e8d64a ""}, {b = 0x2d83e8d64c "Host: cache.finn.no", e = 0x2d83e8d65f ""}, { b = 0x2d83e8d679 "Cookie: IAD_CAMPAIGN=_blank; IADVISITOR=344397518; RMID=57f81d5147062cc0; WT_FPC=id=275309c23e51b22e97e1191583458858:lv=1192011517453:ss=1192007507097; RMFW=011IfHAm4603c8|4A03gK; RMFD=011IfYXaO4033b|"..., e = 0x2d83e8d989 ""}, {b = 0x2d821f87e8 "X-Varnish: 702561589", e = 0x2d821f87fc ""}, {b = 0x2d821f87fd "X-Forwarded-for: 87.248.29.81", e = 0x2d821f881a ""}, {b = 0x0, e = 0x0} }, hdf = '\0' , nhd = 14} }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Oct 10 13:29:35 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 10 Oct 2007 13:29:35 -0000 Subject: [Varnish] #152: Varnish 1.1.1 compile fails on Solaris 10 In-Reply-To: <053.a7b950c8b9db8b2dd5be8558033bb4aa@projects.linpro.no> References: <053.a7b950c8b9db8b2dd5be8558033bb4aa@projects.linpro.no> Message-ID: <062.34b47cde703d2944928668707f4c4324@projects.linpro.no> #152: Varnish 1.1.1 compile fails on Solaris 10 ---------------------+------------------------------------------------------ Reporter: jimlane | Owner: des Type: defect | Status: assigned Priority: normal | Milestone: Varnish 2.0 release Component: build | Version: 1.1.1 Severity: normal | Resolution: Keywords: | ---------------------+------------------------------------------------------ Comment (by des): The {{{-lnsl}}} issue is fixed in r2097, as well as incorrect use of {{{AC_CHECK_LIBS}}}. The {{{NAN}}} issue should be fixed by including {{{}}}, which according to POSIX must define {{{NAN}}}; {{{}}} is non-standard. The {{{statvfs}}} issue should be fixed by reverting an earlier commit to [source:trunk/varnish-cache/bin/varnishd/storage_file.c] where I removed the correct {{{#include}}} because I couldn't remember why it was needed. The {{{compat/daemon.h}}} issue should be fixed by adding {{{daemon(3)}}} from e.g. FreeBSD to libvarnishcompat. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Oct 8 13:43:27 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 08 Oct 2007 13:43:27 -0000 Subject: [Varnish] #156: Varnish 1.1 branch dies with assert error in WS_ReleaseP In-Reply-To: <052.9636620d22b6f890a89073c2cf8ba817@projects.linpro.no> References: <052.9636620d22b6f890a89073c2cf8ba817@projects.linpro.no> Message-ID: <061.9078bffc48e02fc4be961b8f4308fa6c@projects.linpro.no> #156: Varnish 1.1 branch dies with assert error in WS_ReleaseP -----------------------------------+---------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: 1.1 Severity: major | Resolution: Keywords: core dump WS_ReleaseP | -----------------------------------+---------------------------------------- Comment (by anders): PHK said I am tracing the wrong thread? Which tread to check? List of threads follows: {{{ (gdb) info threads 1006 Thread 0x3549400 (LWP 101364) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 1005 Thread 0x3549200 (LWP 101363) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 1004 Thread 0x3549000 (LWP 101362) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 1003 Thread 0x3540e00 (LWP 101361) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 1002 Thread 0x3540c00 (LWP 101360) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 1001 Thread 0x3540200 (LWP 101359) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 1000 Thread 0x3540a00 (LWP 101358) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 999 Thread 0x3540800 (LWP 101357) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 998 Thread 0x3540600 (LWP 101356) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 997 Thread 0x3540400 (LWP 101355) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 996 Thread 0x353fe00 (LWP 101354) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 995 Thread 0x3540000 (LWP 101353) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 994 Thread 0x353fc00 (LWP 101352) 0x0000000800cc944c in writev () at writev.S:2 993 Thread 0x353f800 (LWP 101351) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 992 Thread 0x353fa00 (LWP 101350) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 991 Thread 0x353f600 (LWP 101349) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 990 Thread 0x353f400 (LWP 101348) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 989 Thread 0x353f200 (LWP 101347) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 988 Thread 0x353f000 (LWP 101346) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 987 Thread 0x3538e00 (LWP 101345) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 986 Thread 0x3538c00 (LWP 101344) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 985 Thread 0x3538a00 (LWP 101343) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 984 Thread 0x3538800 (LWP 101342) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 983 Thread 0x3538600 (LWP 101341) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 982 Thread 0x3538400 (LWP 101340) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 981 Thread 0x3538200 (LWP 101339) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 980 Thread 0x352ce00 (LWP 101338) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 979 Thread 0x3538000 (LWP 101337) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 978 Thread 0x352c600 (LWP 101336) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 977 Thread 0x352c000 (LWP 101335) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 976 Thread 0x352cc00 (LWP 101334) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 975 Thread 0x352ca00 (LWP 101333) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 974 Thread 0x352c800 (LWP 101332) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 973 Thread 0x352c200 (LWP 101331) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 972 Thread 0x352c400 (LWP 101330) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 971 Thread 0x3520e00 (LWP 101329) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 970 Thread 0x3520c00 (LWP 101328) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 969 Thread 0x3520a00 (LWP 101327) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 968 Thread 0x3520800 (LWP 101326) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 967 Thread 0x3520600 (LWP 101325) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 966 Thread 0x3520400 (LWP 101324) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 965 Thread 0x3520200 (LWP 101323) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 964 Thread 0x3519e00 (LWP 101322) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 963 Thread 0x3520000 (LWP 101321) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 962 Thread 0x3519c00 (LWP 101320) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 961 Thread 0x3519a00 (LWP 101319) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 960 Thread 0x3519800 (LWP 101318) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 959 Thread 0x3519600 (LWP 101317) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 958 Thread 0x3519400 (LWP 101316) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 957 Thread 0x3519200 (LWP 101315) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 956 Thread 0x3519000 (LWP 101314) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 955 Thread 0x3517e00 (LWP 101313) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 954 Thread 0x3517c00 (LWP 101312) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 953 Thread 0x3517a00 (LWP 101311) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 952 Thread 0x3517800 (LWP 101310) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 951 Thread 0x3517600 (LWP 101309) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 950 Thread 0x3517400 (LWP 101308) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 949 Thread 0x3517200 (LWP 101307) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 948 Thread 0x3517000 (LWP 101306) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 947 Thread 0x3507e00 (LWP 101305) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 946 Thread 0x3507c00 (LWP 101304) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 945 Thread 0x3507a00 (LWP 101303) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 944 Thread 0x3507800 (LWP 101302) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 943 Thread 0x3507600 (LWP 101301) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 942 Thread 0x3507400 (LWP 101300) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 941 Thread 0x3507200 (LWP 101299) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 940 Thread 0x3507000 (LWP 101298) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 939 Thread 0x3505e00 (LWP 101297) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 938 Thread 0x3505c00 (LWP 101296) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 937 Thread 0x3505a00 (LWP 101295) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 936 Thread 0x3505800 (LWP 101294) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 935 Thread 0x3505600 (LWP 101293) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 934 Thread 0x3505400 (LWP 101292) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 933 Thread 0x3505200 (LWP 101291) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 932 Thread 0x3505000 (LWP 101290) 0x0000000800cc944c in writev () at writev.S:2 931 Thread 0x3502e00 (LWP 101289) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 930 Thread 0x3502c00 (LWP 101288) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 929 Thread 0x3502a00 (LWP 101287) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 928 Thread 0x3502800 (LWP 101286) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 927 Thread 0x3502600 (LWP 101285) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 926 Thread 0x3502400 (LWP 101284) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 925 Thread 0x3502200 (LWP 101283) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 924 Thread 0x3502000 (LWP 101282) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 923 Thread 0x3501e00 (LWP 101281) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 922 Thread 0x3501c00 (LWP 101280) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 921 Thread 0x3501a00 (LWP 101279) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 920 Thread 0x3501800 (LWP 101278) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 919 Thread 0x3501600 (LWP 101277) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 918 Thread 0x3501400 (LWP 101276) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 917 Thread 0x3501200 (LWP 101275) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 916 Thread 0x3501000 (LWP 101274) 0x0000000800cc944c in writev () at writev.S:2 915 Thread 0x34ffe00 (LWP 101273) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 914 Thread 0x34ffc00 (LWP 101272) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 913 Thread 0x34ffa00 (LWP 101271) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 912 Thread 0x34ff800 (LWP 101270) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 911 Thread 0x34ff600 (LWP 101269) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 910 Thread 0x34ff400 (LWP 101268) 0x0000000800cc944c in writev () at writev.S:2 909 Thread 0x34ff200 (LWP 101267) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 908 Thread 0x34ff000 (LWP 101266) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 907 Thread 0x34f8e00 (LWP 101265) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 906 Thread 0x34f8c00 (LWP 101264) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 905 Thread 0x34f8a00 (LWP 101263) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 904 Thread 0x34f8800 (LWP 101262) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 903 Thread 0x34f8600 (LWP 101261) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 902 Thread 0x34f8400 (LWP 101260) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 901 Thread 0x34f8200 (LWP 101259) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 900 Thread 0x34f8000 (LWP 101258) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 899 Thread 0x34f2e00 (LWP 101257) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 898 Thread 0x34f2c00 (LWP 101256) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 897 Thread 0x34f2a00 (LWP 101255) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 896 Thread 0x34f2800 (LWP 101254) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 895 Thread 0x34f2600 (LWP 101253) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 894 Thread 0x34f2400 (LWP 101252) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 893 Thread 0x34f2200 (LWP 101251) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 892 Thread 0x34f2000 (LWP 101250) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 891 Thread 0x34e0e00 (LWP 101249) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 890 Thread 0x34e0c00 (LWP 101248) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 889 Thread 0x34e0a00 (LWP 101247) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 888 Thread 0x34e0800 (LWP 101246) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 887 Thread 0x34e0600 (LWP 101245) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 886 Thread 0x34e0400 (LWP 101244) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 885 Thread 0x34e0200 (LWP 101243) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 884 Thread 0x34e0000 (LWP 101242) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 883 Thread 0x34d4e00 (LWP 101241) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 882 Thread 0x34d4c00 (LWP 101240) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 881 Thread 0x34d4a00 (LWP 101239) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 880 Thread 0x34d4800 (LWP 101238) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 879 Thread 0x34d4600 (LWP 101237) 0x0000000800cc944c in writev () at writev.S:2 878 Thread 0x34d4400 (LWP 101236) 0x0000000800cc944c in writev () at writev.S:2 877 Thread 0x34d4200 (LWP 101235) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 876 Thread 0x34d4000 (LWP 101234) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 875 Thread 0x34cec00 (LWP 101233) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 874 Thread 0x34cee00 (LWP 101232) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 873 Thread 0x34cea00 (LWP 101231) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 872 Thread 0x34ce800 (LWP 101230) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 871 Thread 0x34ce600 (LWP 101229) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 870 Thread 0x34ce400 (LWP 101228) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 869 Thread 0x34ce200 (LWP 101227) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 868 Thread 0x34ce000 (LWP 101226) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 867 Thread 0x34bce00 (LWP 101225) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 866 Thread 0x34bcc00 (LWP 101224) 0x0000000800cc944c in writev () at writev.S:2 865 Thread 0x34bca00 (LWP 101223) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 864 Thread 0x34bc800 (LWP 101222) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 863 Thread 0x34bc600 (LWP 101221) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 862 Thread 0x32a5c00 (LWP 101220) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 861 Thread 0x34bc400 (LWP 101219) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 860 Thread 0x34bc200 (LWP 101218) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 859 Thread 0x34bc000 (LWP 101217) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 858 Thread 0x34b5e00 (LWP 101216) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 857 Thread 0x34b5c00 (LWP 101215) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 856 Thread 0x34b5a00 (LWP 101214) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 855 Thread 0x34b5800 (LWP 101213) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 854 Thread 0x34b5400 (LWP 101212) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 853 Thread 0x34b5600 (LWP 101211) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 852 Thread 0x34b5000 (LWP 101210) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 851 Thread 0x34b5200 (LWP 101209) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 850 Thread 0x349ee00 (LWP 101208) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 849 Thread 0x349ec00 (LWP 101207) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 848 Thread 0x349ea00 (LWP 101206) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 847 Thread 0x349e800 (LWP 101205) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 846 Thread 0x349e600 (LWP 101204) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 845 Thread 0x349e400 (LWP 101203) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 844 Thread 0x349e200 (LWP 101202) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 843 Thread 0x349e000 (LWP 101201) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 842 Thread 0x32a5e00 (LWP 101200) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 841 Thread 0x3267e00 (LWP 101199) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 840 Thread 0x3267c00 (LWP 101198) 0x0000000800cc944c in writev () at writev.S:2 839 Thread 0x3267a00 (LWP 101197) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 838 Thread 0x3267800 (LWP 101196) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 837 Thread 0x3267600 (LWP 101195) 0x0000000800cc944c in writev () at writev.S:2 836 Thread 0x3267400 (LWP 101194) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 835 Thread 0x3267200 (LWP 101193) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 834 Thread 0x3267000 (LWP 101192) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 833 Thread 0x3257e00 (LWP 101191) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 832 Thread 0x3257c00 (LWP 101190) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 831 Thread 0x3257800 (LWP 101189) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 830 Thread 0x3257a00 (LWP 101188) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 829 Thread 0x3257400 (LWP 101187) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 828 Thread 0x3257600 (LWP 101186) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 827 Thread 0x3257200 (LWP 101185) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 826 Thread 0x3257000 (LWP 101184) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 825 Thread 0x3244e00 (LWP 101183) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 824 Thread 0x3244c00 (LWP 101182) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 823 Thread 0x3244800 (LWP 101181) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 822 Thread 0x3244a00 (LWP 101180) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 821 Thread 0x3244600 (LWP 101179) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 820 Thread 0x3244400 (LWP 101178) 0x0000000800cc944c in writev () at writev.S:2 819 Thread 0x3244200 (LWP 101177) 0x0000000800cc944c in writev () at writev.S:2 818 Thread 0x3233e00 (LWP 101176) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 817 Thread 0x3244000 (LWP 101175) 0x0000000800ce6bcc in read () at read.S:2 816 Thread 0x3233c00 (LWP 101174) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 815 Thread 0x3233a00 (LWP 101173) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 814 Thread 0x3233800 (LWP 101172) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 813 Thread 0x3233400 (LWP 101171) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 812 Thread 0x3233600 (LWP 101170) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 811 Thread 0x3233200 (LWP 101169) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 810 Thread 0x3233000 (LWP 101168) 0x0000000800cc944c in writev () ---Type to continue, or q to quit--- at writev.S:2 809 Thread 0x2fffe00 (LWP 101167) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 808 Thread 0x2fffc00 (LWP 101166) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 807 Thread 0x2fffa00 (LWP 101165) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 806 Thread 0x2fff800 (LWP 101164) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 805 Thread 0x2fff600 (LWP 101163) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 804 Thread 0x2fff400 (LWP 101162) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 803 Thread 0x2fff200 (LWP 101161) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 802 Thread 0x2fff000 (LWP 101160) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 801 Thread 0x3206e00 (LWP 101159) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 800 Thread 0x3206c00 (LWP 101158) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 799 Thread 0x3206a00 (LWP 101157) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 798 Thread 0x3206800 (LWP 101156) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 797 Thread 0x3206600 (LWP 101155) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 796 Thread 0x3206400 (LWP 101154) 0x0000000000411d68 in Fetch (sp=0x5c02008) at cache_fetch.c:312 795 Thread 0x3206000 (LWP 101153) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 794 Thread 0x3206200 (LWP 101152) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 793 Thread 0x31f0e00 (LWP 101151) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 792 Thread 0x31f0c00 (LWP 101150) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 791 Thread 0x31f0a00 (LWP 101149) 0x0000000800cc944c in writev () at writev.S:2 790 Thread 0x31f0800 (LWP 101148) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 789 Thread 0x31f0600 (LWP 101147) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 788 Thread 0x31f0400 (LWP 101146) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 787 Thread 0x31f0200 (LWP 101145) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 786 Thread 0x31f0000 (LWP 101144) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 785 Thread 0x31dfe00 (LWP 101143) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 784 Thread 0x31dfc00 (LWP 101142) 0x0000000800cc944c in writev () at writev.S:2 783 Thread 0x31dfa00 (LWP 101141) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 782 Thread 0x31df800 (LWP 101140) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 781 Thread 0x31df600 (LWP 101139) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 780 Thread 0x31df400 (LWP 101138) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 779 Thread 0x31df200 (LWP 101137) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 778 Thread 0x31cee00 (LWP 101136) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 777 Thread 0x31df000 (LWP 101135) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 776 Thread 0x31cec00 (LWP 101134) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 775 Thread 0x31cea00 (LWP 101133) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 774 Thread 0x31ce800 (LWP 101132) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 773 Thread 0x31ce600 (LWP 101131) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 772 Thread 0x31ce200 (LWP 101130) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 771 Thread 0x31ce400 (LWP 101129) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 770 Thread 0x31ce000 (LWP 101128) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 769 Thread 0x31bee00 (LWP 101127) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 768 Thread 0x31bea00 (LWP 101126) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 767 Thread 0x31bec00 (LWP 101125) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 766 Thread 0x31be800 (LWP 101124) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 765 Thread 0x31be600 (LWP 101123) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 764 Thread 0x31be400 (LWP 101122) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 763 Thread 0x31be200 (LWP 101121) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 762 Thread 0x31be000 (LWP 101120) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 761 Thread 0x31b1e00 (LWP 101119) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 760 Thread 0x31b1c00 (LWP 101118) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 759 Thread 0x31b1a00 (LWP 101117) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 758 Thread 0x31b1800 (LWP 101116) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 757 Thread 0x31b1600 (LWP 101115) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 756 Thread 0x31b1400 (LWP 101114) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 755 Thread 0x31b1200 (LWP 101113) 0x0000000800cc944c in writev () at writev.S:2 754 Thread 0x31b1000 (LWP 101112) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 753 Thread 0x319fe00 (LWP 101111) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 752 Thread 0x319fc00 (LWP 101110) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 751 Thread 0x319fa00 (LWP 101109) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 750 Thread 0x319f800 (LWP 101108) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 749 Thread 0x319f400 (LWP 101107) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 748 Thread 0x319f600 (LWP 101106) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 747 Thread 0x319f200 (LWP 101105) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 746 Thread 0x319f000 (LWP 101104) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 745 Thread 0x318de00 (LWP 101103) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 744 Thread 0x318dc00 (LWP 101102) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 743 Thread 0x318da00 (LWP 101101) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 742 Thread 0x318d800 (LWP 101100) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 741 Thread 0x318d600 (LWP 101099) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 740 Thread 0x318d200 (LWP 101098) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 739 Thread 0x318d000 (LWP 101097) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 738 Thread 0x318d400 (LWP 101096) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 737 Thread 0x317be00 (LWP 101095) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 736 Thread 0x317bc00 (LWP 101094) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 735 Thread 0x317ba00 (LWP 101093) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 734 Thread 0x317b800 (LWP 101092) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 733 Thread 0x317b600 (LWP 101091) 0x0000000800cc944c in writev () at writev.S:2 732 Thread 0x317b400 (LWP 101090) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 731 Thread 0x317b200 (LWP 101089) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 730 Thread 0x317b000 (LWP 101088) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 729 Thread 0x3173e00 (LWP 101087) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 728 Thread 0x3173c00 (LWP 101086) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 727 Thread 0x3173a00 (LWP 101085) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 726 Thread 0x3173800 (LWP 101084) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 725 Thread 0x3173600 (LWP 101083) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 724 Thread 0x3173400 (LWP 101082) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 723 Thread 0x3173200 (LWP 101081) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 722 Thread 0x3173000 (LWP 101080) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 721 Thread 0x3160e00 (LWP 101079) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 720 Thread 0x3160c00 (LWP 101078) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 719 Thread 0x3160a00 (LWP 101077) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 718 Thread 0x3160600 (LWP 101076) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 717 Thread 0x3160800 (LWP 101075) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 716 Thread 0x3160400 (LWP 101074) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 715 Thread 0x3160000 (LWP 101073) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 714 Thread 0x3160200 (LWP 101072) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 713 Thread 0x3159c00 (LWP 101071) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 712 Thread 0x3159e00 (LWP 101070) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 711 Thread 0x3159a00 (LWP 101069) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 710 Thread 0x3159800 (LWP 101068) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 709 Thread 0x3159600 (LWP 101067) 0x0000000800cc944c in writev () at writev.S:2 708 Thread 0x3159400 (LWP 101066) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 707 Thread 0x3159200 (LWP 101065) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 706 Thread 0x3159000 (LWP 101064) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 705 Thread 0x3156e00 (LWP 101063) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 704 Thread 0x3156c00 (LWP 101062) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 703 Thread 0x3156a00 (LWP 101061) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 702 Thread 0x3156800 (LWP 101060) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 701 Thread 0x3156600 (LWP 101059) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 700 Thread 0x3156400 (LWP 101058) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 699 Thread 0x3156200 (LWP 101057) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 698 Thread 0x3156000 (LWP 101056) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 697 Thread 0x3149e00 (LWP 101055) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 696 Thread 0x540000 (LWP 100106) 0x0000000800c7f6fc in poll () at poll.S:2 695 Thread 0x540600 (LWP 100105) 0x0000000800cc302c in nanosleep () at nanosleep.S:2 694 Thread 0x540800 (LWP 100360) 0x0000000800cc302c in nanosleep () at nanosleep.S:2 693 Thread 0x540a00 (LWP 100361) 0x0000000800cc302c in nanosleep () at nanosleep.S:2 692 Thread 0x540e00 (LWP 100362) 0x0000000800c3273c in thr_kill () at thr_kill.S:2 691 Thread 0xafe000 (LWP 100364) 0x0000000800c6c53c in accept () at accept.S:2 690 Thread 0xafe200 (LWP 100365) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 689 Thread 0xafe400 (LWP 100366) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 688 Thread 0xafe600 (LWP 100367) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 687 Thread 0xafe800 (LWP 100368) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 686 Thread 0xafea00 (LWP 100369) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 685 Thread 0xafec00 (LWP 100370) 0x0000000800cc944c in writev () at writev.S:2 684 Thread 0xafee00 (LWP 100371) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 683 Thread 0xb43000 (LWP 100372) 0x0000000800cc944c in writev () at writev.S:2 682 Thread 0xb43200 (LWP 100373) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 681 Thread 0xb43400 (LWP 100374) 0x0000000800cc944c in writev () at writev.S:2 680 Thread 0xb43600 (LWP 100375) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 679 Thread 0xb43800 (LWP 100376) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 678 Thread 0xb43a00 (LWP 100377) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 677 Thread 0xb43c00 (LWP 100378) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 676 Thread 0xb43e00 (LWP 100379) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 675 Thread 0xb77200 (LWP 100380) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 674 Thread 0xb77000 (LWP 100381) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 673 Thread 0xb77400 (LWP 100382) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 672 Thread 0xb77600 (LWP 100383) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 671 Thread 0xb77800 (LWP 100384) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 670 Thread 0xb77a00 (LWP 100385) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 669 Thread 0xb77c00 (LWP 100386) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 668 Thread 0xb77e00 (LWP 100387) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 667 Thread 0xbab000 (LWP 100388) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 666 Thread 0xbab200 (LWP 100389) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 665 Thread 0xbab400 (LWP 100390) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 664 Thread 0xbab600 (LWP 100391) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 663 Thread 0xbab800 (LWP 100392) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 662 Thread 0xbaba00 (LWP 100393) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 661 Thread 0xbabc00 (LWP 100394) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 660 Thread 0xbabe00 (LWP 100395) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 659 Thread 0xc13000 (LWP 100396) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 658 Thread 0xc13200 (LWP 100397) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 657 Thread 0xc13400 (LWP 100398) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 656 Thread 0xc13600 (LWP 100399) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 655 Thread 0xc13800 (LWP 100400) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 654 Thread 0xc13a00 (LWP 100401) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 653 Thread 0xc13c00 (LWP 100402) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 652 Thread 0xc13e00 (LWP 100403) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 651 Thread 0xc75000 (LWP 100404) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 650 Thread 0xc75200 (LWP 100405) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 649 Thread 0xc75400 (LWP 100406) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 648 Thread 0xc75600 (LWP 100407) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 647 Thread 0xc75800 (LWP 100408) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 646 Thread 0xc75a00 (LWP 100409) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 645 Thread 0xc75c00 (LWP 100410) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 644 Thread 0xc75e00 (LWP 100411) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 643 Thread 0xced000 (LWP 100412) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 642 Thread 0xced200 (LWP 100413) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 641 Thread 0xced400 (LWP 100414) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 640 Thread 0xced600 (LWP 100415) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 639 Thread 0xced800 (LWP 100416) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 638 Thread 0xceda00 (LWP 100417) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 637 Thread 0xcede00 (LWP 100418) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 636 Thread 0xdb5000 (LWP 100419) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 635 Thread 0xdb5200 (LWP 100420) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 634 Thread 0xdb5400 (LWP 100421) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 633 Thread 0xdb5600 (LWP 100422) 0x0000000800cd200c in fcntl () at fcntl.S:2 632 Thread 0xdb5800 (LWP 100423) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 631 Thread 0xdb5a00 (LWP 100424) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 630 Thread 0xdb5c00 (LWP 100425) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 629 Thread 0xdb5e00 (LWP 100426) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 628 Thread 0xde9000 (LWP 100427) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 627 Thread 0xde9200 (LWP 100428) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 626 Thread 0xde9400 (LWP 100429) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 625 Thread 0xde9600 (LWP 100430) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 624 Thread 0xde9800 (LWP 100431) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 623 Thread 0xde9a00 (LWP 100432) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 622 Thread 0xde9c00 (LWP 100433) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 621 Thread 0xde9e00 (LWP 100434) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 620 Thread 0xe6c000 (LWP 100435) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 619 Thread 0xe6c200 (LWP 100436) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 618 Thread 0xe6c400 (LWP 100437) 0x0000000800cc944c in writev () at writev.S:2 617 Thread 0xcedc00 (LWP 100438) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 616 Thread 0xe6c600 (LWP 100439) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 615 Thread 0xe6c800 (LWP 100440) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 614 Thread 0xe6ca00 (LWP 100441) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 613 Thread 0xe6cc00 (LWP 100442) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 612 Thread 0xe6ce00 (LWP 100443) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 611 Thread 0x143a000 (LWP 100444) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 610 Thread 0x143a200 (LWP 100445) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 609 Thread 0x143a400 (LWP 100446) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 608 Thread 0x143a600 (LWP 100447) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 607 Thread 0x143a800 (LWP 100448) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 606 Thread 0x143aa00 (LWP 100449) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 605 Thread 0x143ac00 (LWP 100450) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 604 Thread 0x143ae00 (LWP 100451) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 603 Thread 0x145a000 (LWP 100452) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 602 Thread 0x145a200 (LWP 100453) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 601 Thread 0x145a400 (LWP 100454) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 600 Thread 0x145a600 (LWP 100455) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 599 Thread 0x145a800 (LWP 100456) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 598 Thread 0x145aa00 (LWP 100457) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 597 Thread 0x145ac00 (LWP 100458) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 596 Thread 0x145ae00 (LWP 100459) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 595 Thread 0x14a2000 (LWP 100460) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 594 Thread 0x14a2200 (LWP 100461) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 593 Thread 0x14a2400 (LWP 100462) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 592 Thread 0x14a2600 (LWP 100463) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 591 Thread 0x14a2800 (LWP 100464) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 590 Thread 0x14a2a00 (LWP 100465) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 589 Thread 0x14a2c00 (LWP 100466) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 588 Thread 0x14a2e00 (LWP 100467) 0x0000000800cc944c in writev () at writev.S:2 587 Thread 0x14f6000 (LWP 100468) 0x0000000800cbbe6c in connect () at connect.S:2 586 Thread 0x14f6200 (LWP 100469) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 585 Thread 0x14f6400 (LWP 100470) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 584 Thread 0x14f6600 (LWP 100471) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 583 Thread 0x14f6800 (LWP 100472) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 582 Thread 0x2a3a000 (LWP 100473) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 581 Thread 0x2a3a200 (LWP 100474) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 580 Thread 0x2a3a400 (LWP 100475) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 579 Thread 0x2a3a600 (LWP 100476) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 578 Thread 0x2a3a800 (LWP 100477) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 577 Thread 0x2a3aa00 (LWP 100478) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 576 Thread 0x2a3ac00 (LWP 100479) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 575 Thread 0x2a3ae00 (LWP 100480) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 574 Thread 0x2a45000 (LWP 100481) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 573 Thread 0x2a45200 (LWP 100482) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 572 Thread 0x2a45400 (LWP 100483) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 571 Thread 0x2a45600 (LWP 100484) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 570 Thread 0x2a45800 (LWP 100485) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 569 Thread 0x2a45a00 (LWP 100486) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 568 Thread 0x2a45c00 (LWP 100487) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 567 Thread 0x2a45e00 (LWP 100488) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 566 Thread 0x2a50000 (LWP 100489) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 565 Thread 0x2a50200 (LWP 100490) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 564 Thread 0x2a50400 (LWP 100491) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 563 Thread 0x2a50600 (LWP 100492) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 562 Thread 0x2a50800 (LWP 100493) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 561 Thread 0x2a50a00 (LWP 100494) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 560 Thread 0x2a50c00 (LWP 100495) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 559 Thread 0x2a50e00 (LWP 100496) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 558 Thread 0x2a56000 (LWP 100497) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 557 Thread 0x2a56200 (LWP 100498) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 556 Thread 0x2a56400 (LWP 100499) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 555 Thread 0x2a56600 (LWP 100500) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 554 Thread 0x2a56800 (LWP 100501) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 553 Thread 0x2a56a00 (LWP 100502) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 552 Thread 0x2a56c00 (LWP 100503) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 551 Thread 0x2a56e00 (LWP 100504) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 550 Thread 0x2a66000 (LWP 100505) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 549 Thread 0x2a66200 (LWP 100506) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 548 Thread 0x2a66400 (LWP 100507) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 547 Thread 0x2a66600 (LWP 100508) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 546 Thread 0x2a66800 (LWP 100509) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 545 Thread 0x2a66a00 (LWP 100510) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 544 Thread 0x2a66c00 (LWP 100511) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 543 Thread 0x2a66e00 (LWP 100512) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 542 Thread 0x2a6e000 (LWP 100513) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 541 Thread 0x2a6e200 (LWP 100514) 0x0000000800cc944c in writev () at writev.S:2 540 Thread 0x2a6e400 (LWP 100515) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 539 Thread 0x2a6e600 (LWP 100516) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 538 Thread 0x2a6e800 (LWP 100517) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 537 Thread 0x2a6ea00 (LWP 100518) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 536 Thread 0x2a6ec00 (LWP 100519) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 535 Thread 0x2a6ee00 (LWP 100520) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 534 Thread 0x2aea000 (LWP 100521) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 533 Thread 0x2aea200 (LWP 100522) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 532 Thread 0x2aea600 (LWP 100523) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 531 Thread 0x2aea400 (LWP 100524) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 530 Thread 0x2aea800 (LWP 100525) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 529 Thread 0x2aeaa00 (LWP 100526) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 528 Thread 0x2aeac00 (LWP 100527) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 527 Thread 0x2aeae00 (LWP 100528) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 526 Thread 0x2af6000 (LWP 100529) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 525 Thread 0x2af6200 (LWP 100530) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 524 Thread 0x2af6400 (LWP 100531) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 523 Thread 0x2af6600 (LWP 100532) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 522 Thread 0x2af6800 (LWP 100533) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 521 Thread 0x2af6a00 (LWP 100534) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 520 Thread 0x2af6c00 (LWP 100535) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 519 Thread 0x2af6e00 (LWP 100536) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 518 Thread 0x2b3b000 (LWP 100537) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 517 Thread 0x2b3b200 (LWP 100538) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 516 Thread 0x2b3b400 (LWP 100539) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 515 Thread 0x2b3b600 (LWP 100540) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 514 Thread 0x2b3b800 (LWP 100541) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 513 Thread 0x2b3ba00 (LWP 100542) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 512 Thread 0x2b3bc00 (LWP 100543) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 511 Thread 0x2b3be00 (LWP 100544) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 510 Thread 0x2b67000 (LWP 100545) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 509 Thread 0x2b67200 (LWP 100546) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 508 Thread 0x2b67400 (LWP 100547) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 507 Thread 0x2b67600 (LWP 100548) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 506 Thread 0x2b67800 (LWP 100549) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 505 Thread 0x2b67a00 (LWP 100550) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 504 Thread 0x2b67c00 (LWP 100551) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 503 Thread 0x2b67e00 (LWP 100552) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 502 Thread 0x2b82000 (LWP 100553) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 501 Thread 0x2b82200 (LWP 100554) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 500 Thread 0x2b82400 (LWP 100555) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 499 Thread 0x2b82800 (LWP 100556) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 498 Thread 0x2b82600 (LWP 100557) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 497 Thread 0x2b82a00 (LWP 100558) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 496 Thread 0x2b82c00 (LWP 100559) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 495 Thread 0x2b82e00 (LWP 100560) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 494 Thread 0x2b83000 (LWP 100561) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 493 Thread 0x2b83200 (LWP 100562) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 492 Thread 0x2b83400 (LWP 100563) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 * 491 Thread 0x2b83600 (LWP 100564) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 490 Thread 0x2b83800 (LWP 100565) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 489 Thread 0x2b83a00 (LWP 100566) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 488 Thread 0x2b83e00 (LWP 100567) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 487 Thread 0x2b83c00 (LWP 100568) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 486 Thread 0x2b8e200 (LWP 100569) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 485 Thread 0x2b8e000 (LWP 100570) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 484 Thread 0x2b8e400 (LWP 100571) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 483 Thread 0x2b8e600 (LWP 100572) 0x0000000800cc944c in writev () at writev.S:2 482 Thread 0x2b8e800 (LWP 100573) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 481 Thread 0x2b8ea00 (LWP 100574) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 480 Thread 0x2b8ee00 (LWP 100575) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 479 Thread 0x2b9a000 (LWP 100576) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 478 Thread 0x2b8ec00 (LWP 100577) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 477 Thread 0x2b9a200 (LWP 100578) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 476 Thread 0x2b9a400 (LWP 100579) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 475 Thread 0x2b9a600 (LWP 100580) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 474 Thread 0x2b9a800 (LWP 100581) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 473 Thread 0x2b9aa00 (LWP 100582) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 472 Thread 0x2b9ac00 (LWP 100583) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 471 Thread 0x2b9ae00 (LWP 100584) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 470 Thread 0x2baf000 (LWP 100585) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 469 Thread 0x2baf200 (LWP 100586) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 468 Thread 0x2baf400 (LWP 100587) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 467 Thread 0x2baf600 (LWP 100588) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 466 Thread 0x2baf800 (LWP 100589) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 465 Thread 0x2bafa00 (LWP 100590) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 464 Thread 0x2bafc00 (LWP 100591) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 463 Thread 0x2bafe00 (LWP 100592) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 462 Thread 0x2bf2000 (LWP 100593) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 461 Thread 0x2bf2200 (LWP 100594) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 460 Thread 0x2bf2400 (LWP 100595) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 459 Thread 0x2bf2600 (LWP 100596) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 458 Thread 0x2bf2800 (LWP 100597) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 457 Thread 0x2bf2a00 (LWP 100598) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 456 Thread 0x2bf2c00 (LWP 100599) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 455 Thread 0x2bf2e00 (LWP 100600) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 454 Thread 0x2c08000 (LWP 100601) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 453 Thread 0x2c08200 (LWP 100602) 0x0000000800cc944c in writev () at writev.S:2 452 Thread 0x2c08400 (LWP 100603) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 451 Thread 0x2c08600 (LWP 100604) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 450 Thread 0x2c08800 (LWP 100605) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 449 Thread 0x2c08a00 (LWP 100606) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 448 Thread 0x2c08c00 (LWP 100607) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 447 Thread 0x2c08e00 (LWP 100608) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 446 Thread 0x2c2c000 (LWP 100609) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 445 Thread 0x2c2c200 (LWP 100610) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 444 Thread 0x2c2c400 (LWP 100611) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 443 Thread 0x2c2c600 (LWP 100612) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 442 Thread 0x2c2c800 (LWP 100613) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 441 Thread 0x2c2ca00 (LWP 100614) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 440 Thread 0x2c2cc00 (LWP 100615) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 439 Thread 0x2c2ce00 (LWP 100616) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 438 Thread 0x2c41000 (LWP 100617) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 437 Thread 0x2c41200 (LWP 100618) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 436 Thread 0x2c41400 (LWP 100619) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 435 Thread 0x2c41600 (LWP 100620) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 434 Thread 0x2c41800 (LWP 100621) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 433 Thread 0x2c41a00 (LWP 100622) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 432 Thread 0x2c41c00 (LWP 100623) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 431 Thread 0x2c41e00 (LWP 100624) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 430 Thread 0x28e3000 (LWP 100625) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 429 Thread 0x28e3200 (LWP 100626) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 428 Thread 0x28e3400 (LWP 100627) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 427 Thread 0x28e3800 (LWP 100628) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 426 Thread 0x28e3600 (LWP 100629) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 425 Thread 0x28e3a00 (LWP 100630) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 424 Thread 0x28e3c00 (LWP 100631) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 423 Thread 0x28e3e00 (LWP 100632) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 422 Thread 0x2c7a000 (LWP 100633) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 421 Thread 0x2c7a200 (LWP 100634) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 420 Thread 0x2c7a400 (LWP 100635) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 419 Thread 0x2c7a600 (LWP 100636) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 418 Thread 0x2c7a800 (LWP 100637) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 417 Thread 0x2c7aa00 (LWP 100638) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 416 Thread 0x2c7ac00 (LWP 100639) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 415 Thread 0x2c7ae00 (LWP 100640) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 414 Thread 0x2c8d000 (LWP 100641) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 413 Thread 0x2c8d400 (LWP 100642) 0x0000000800ce6bcc in read () at read.S:2 412 Thread 0x2c8d200 (LWP 100643) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 411 Thread 0x2c8d600 (LWP 100644) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 410 Thread 0x2c8d800 (LWP 100645) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 409 Thread 0x2c8da00 (LWP 100646) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 408 Thread 0x2c8dc00 (LWP 100647) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 407 Thread 0x2c8de00 (LWP 100648) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 406 Thread 0x2c8f000 (LWP 100649) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 405 Thread 0x2c8f200 (LWP 100650) 0x0000000000411d68 in Fetch (sp=0x5617008) at cache_fetch.c:312 404 Thread 0x2c8f400 (LWP 100651) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 403 Thread 0x2c8f600 (LWP 100652) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 402 Thread 0x2c8f800 (LWP 100653) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 401 Thread 0x2c8fa00 (LWP 100654) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 400 Thread 0x2c8fc00 (LWP 100655) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 399 Thread 0x2c92000 (LWP 100656) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 398 Thread 0x2c8fe00 (LWP 100657) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 397 Thread 0x2c92400 (LWP 100658) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 396 Thread 0x2c92200 (LWP 100659) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 395 Thread 0x2c92600 (LWP 100660) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 394 Thread 0x2c92800 (LWP 100661) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 393 Thread 0x2c92a00 (LWP 100662) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 392 Thread 0x2c92c00 (LWP 100663) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 391 Thread 0x2c92e00 (LWP 100664) 0x0000000800cc944c in writev () at writev.S:2 390 Thread 0x2c95000 (LWP 100665) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 389 Thread 0x2c95200 (LWP 100666) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 388 Thread 0x2c95400 (LWP 100667) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 387 Thread 0x2c95600 (LWP 100668) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 386 Thread 0x2c95800 (LWP 100669) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 385 Thread 0x2c95a00 (LWP 100670) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 384 Thread 0x2c95c00 (LWP 100671) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 383 Thread 0x2c95e00 (LWP 100672) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 382 Thread 0x2c98000 (LWP 100673) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 381 Thread 0x2c98200 (LWP 100674) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 380 Thread 0x2c98400 (LWP 100675) 0x0000000800cc944c in writev () at writev.S:2 379 Thread 0x2c98600 (LWP 100676) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 378 Thread 0x2c98800 (LWP 100677) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 377 Thread 0x2c98a00 (LWP 100678) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 376 Thread 0x2c98c00 (LWP 100679) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 375 Thread 0x2c98e00 (LWP 100680) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 374 Thread 0x2c99000 (LWP 100681) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 373 Thread 0x2c99200 (LWP 100682) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 372 Thread 0x2c99400 (LWP 100683) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 371 Thread 0x2c99600 (LWP 100684) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 370 Thread 0x2c99800 (LWP 100685) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 369 Thread 0x2c99a00 (LWP 100686) 0x0000000800cc944c in writev () at writev.S:2 368 Thread 0x2c99c00 (LWP 100687) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 367 Thread 0x2c99e00 (LWP 100688) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 366 Thread 0x2c9e000 (LWP 100689) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 365 Thread 0x2c9e200 (LWP 100690) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 364 Thread 0x2c9e400 (LWP 100691) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 363 Thread 0x2c9e600 (LWP 100692) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 362 Thread 0x2c9e800 (LWP 100693) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 361 Thread 0x2c9ea00 (LWP 100694) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 360 Thread 0x2c9ec00 (LWP 100695) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 359 Thread 0x2c9ee00 (LWP 100696) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 358 Thread 0x2cf3000 (LWP 100697) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 357 Thread 0x2cf3200 (LWP 100698) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 356 Thread 0x2cf3400 (LWP 100699) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 355 Thread 0x2cf3600 (LWP 100700) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 354 Thread 0x2cf3800 (LWP 100701) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 353 Thread 0x2cf3a00 (LWP 100702) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 352 Thread 0x2cf3c00 (LWP 100703) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 351 Thread 0x2cf3e00 (LWP 100704) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 350 Thread 0x2d1f000 (LWP 100705) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 349 Thread 0x2d1f200 (LWP 100706) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 348 Thread 0x2d1f400 (LWP 100707) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 347 Thread 0x2d1f600 (LWP 100708) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 346 Thread 0x2d1f800 (LWP 100709) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 345 Thread 0x2d1fa00 (LWP 100710) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 344 Thread 0x2d1fc00 (LWP 100711) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 343 Thread 0x2d1fe00 (LWP 100712) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 342 Thread 0x2d22000 (LWP 100713) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 341 Thread 0x2d22200 (LWP 100714) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 340 Thread 0x2d22400 (LWP 100715) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 339 Thread 0x2d22600 (LWP 100716) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 338 Thread 0x2d22800 (LWP 100717) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 337 Thread 0x2d22c00 (LWP 100718) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 336 Thread 0x2d22e00 (LWP 100719) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 335 Thread 0x2d22a00 (LWP 100720) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 334 Thread 0x2d24000 (LWP 100721) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 333 Thread 0x2d24200 (LWP 100722) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 332 Thread 0x2d24400 (LWP 100723) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 331 Thread 0x2d24600 (LWP 100724) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 330 Thread 0x2d24800 (LWP 100725) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 329 Thread 0x2d24a00 (LWP 100726) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 328 Thread 0x2d24c00 (LWP 100727) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 327 Thread 0x2d24e00 (LWP 100728) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 326 Thread 0x2d31000 (LWP 100729) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 325 Thread 0x2d31200 (LWP 100730) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 324 Thread 0x2d31400 (LWP 100731) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 323 Thread 0x2d31800 (LWP 100732) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 322 Thread 0x2d31600 (LWP 100733) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 321 Thread 0x2d31a00 (LWP 100734) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 320 Thread 0x2d31c00 (LWP 100735) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 319 Thread 0x2d31e00 (LWP 100736) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 318 Thread 0x2d63000 (LWP 100737) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 317 Thread 0x2d63200 (LWP 100738) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 316 Thread 0x2d63400 (LWP 100739) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 315 Thread 0x2d63600 (LWP 100740) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 314 Thread 0x2d63800 (LWP 100741) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 313 Thread 0x2d63a00 (LWP 100742) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 312 Thread 0x2d63c00 (LWP 100743) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 311 Thread 0x2d63e00 (LWP 100744) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 310 Thread 0x2da3200 (LWP 100745) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 309 Thread 0x2da3000 (LWP 100746) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 308 Thread 0x2da3400 (LWP 100747) 0x0000000800cc944c in writev () at writev.S:2 307 Thread 0x2da3800 (LWP 100748) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 306 Thread 0x2da3a00 (LWP 100749) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 305 Thread 0x2da3c00 (LWP 100750) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 304 Thread 0x2da3e00 (LWP 100751) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 303 Thread 0x2ddc000 (LWP 100752) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 302 Thread 0x2ddc200 (LWP 100753) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 301 Thread 0x2ddc400 (LWP 100754) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 300 Thread 0x2ddc600 (LWP 100755) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 299 Thread 0x2ddc800 (LWP 100756) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 298 Thread 0x2ddca00 (LWP 100757) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 297 Thread 0x2ddce00 (LWP 100758) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 296 Thread 0x2ddcc00 (LWP 100759) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 295 Thread 0x2ddf000 (LWP 100760) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 294 Thread 0x2ddf200 (LWP 100761) 0x0000000800cc944c in writev () ---Type to continue, or q to quit--- at writev.S:2 293 Thread 0x2ddf600 (LWP 100762) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 292 Thread 0x2ddf400 (LWP 100763) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 291 Thread 0x2ddf800 (LWP 100764) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 290 Thread 0x2ddfa00 (LWP 100765) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 289 Thread 0x2ddfc00 (LWP 100766) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 288 Thread 0x2ddfe00 (LWP 100767) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 287 Thread 0x2de5000 (LWP 100768) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 286 Thread 0x2de5200 (LWP 100769) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 285 Thread 0x2de5400 (LWP 100770) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 284 Thread 0x2de5600 (LWP 100771) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 283 Thread 0x2de5800 (LWP 100772) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 282 Thread 0x2de5a00 (LWP 100773) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 281 Thread 0x2de5c00 (LWP 100774) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 280 Thread 0x2de5e00 (LWP 100775) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 279 Thread 0x2de7000 (LWP 100776) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 278 Thread 0x2de7200 (LWP 100777) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 277 Thread 0x2de7400 (LWP 100778) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 276 Thread 0x2de7600 (LWP 100779) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 275 Thread 0x2de7800 (LWP 100780) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 274 Thread 0x2de7a00 (LWP 100781) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 273 Thread 0x2de7c00 (LWP 100782) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 272 Thread 0x2de7e00 (LWP 100783) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 271 Thread 0x2deb000 (LWP 100784) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 270 Thread 0x2deb200 (LWP 100785) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 269 Thread 0x2deb400 (LWP 100786) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 268 Thread 0x2deb600 (LWP 100787) 0x0000000000411d68 in Fetch (sp=0x57d9008) at cache_fetch.c:312 267 Thread 0x2deb800 (LWP 100788) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 266 Thread 0x2deba00 (LWP 100789) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 265 Thread 0x2debc00 (LWP 100790) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 264 Thread 0x2debe00 (LWP 100791) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 263 Thread 0x2e0c000 (LWP 100792) 0x0000000800cc944c in writev () at writev.S:2 262 Thread 0x2e0c200 (LWP 100793) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 261 Thread 0x2e0c400 (LWP 100794) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 260 Thread 0x2e0c600 (LWP 100795) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 259 Thread 0x2e0c800 (LWP 100796) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 258 Thread 0x2e0ca00 (LWP 100797) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 257 Thread 0x2e0cc00 (LWP 100798) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 256 Thread 0x2e0ce00 (LWP 100799) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 255 Thread 0x2e5c000 (LWP 100800) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 254 Thread 0x2e5c200 (LWP 100801) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 253 Thread 0x2e5c400 (LWP 100802) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 252 Thread 0x2e5c600 (LWP 100803) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 251 Thread 0x2e5c800 (LWP 100804) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 250 Thread 0x2e5ca00 (LWP 100805) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 249 Thread 0x2e5cc00 (LWP 100806) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 248 Thread 0x2e5ce00 (LWP 100807) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 247 Thread 0x2e94000 (LWP 100808) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 246 Thread 0x2e94200 (LWP 100809) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 245 Thread 0x2e94400 (LWP 100810) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 244 Thread 0x2e94600 (LWP 100811) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 243 Thread 0x2e94800 (LWP 100812) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 242 Thread 0x2e94a00 (LWP 100813) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 241 Thread 0x2e94c00 (LWP 100814) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 240 Thread 0x2e94e00 (LWP 100815) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 239 Thread 0x2eab000 (LWP 100816) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 238 Thread 0x2eab200 (LWP 100817) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 237 Thread 0x2eab400 (LWP 100818) 0x0000000800cc944c in writev () at writev.S:2 236 Thread 0x2eab600 (LWP 100819) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 235 Thread 0x2eab800 (LWP 100820) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 234 Thread 0x2eaba00 (LWP 100821) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 233 Thread 0x2eabc00 (LWP 100822) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 232 Thread 0x2eabe00 (LWP 100823) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 231 Thread 0x2eae000 (LWP 100824) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 230 Thread 0x2eae200 (LWP 100825) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 229 Thread 0x2eae400 (LWP 100826) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 228 Thread 0x2eae600 (LWP 100827) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 227 Thread 0x2eae800 (LWP 100828) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 226 Thread 0x2eaea00 (LWP 100829) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 225 Thread 0x2eaec00 (LWP 100830) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 224 Thread 0x2eaee00 (LWP 100831) 0x0000000000411d68 in Fetch (sp=0x673d008) at cache_fetch.c:312 223 Thread 0x2ef1000 (LWP 100832) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 222 Thread 0x2ef1200 (LWP 100833) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 221 Thread 0x2ef1400 (LWP 100834) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 220 Thread 0x2ef1600 (LWP 100835) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 219 Thread 0x2ef1800 (LWP 100836) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 218 Thread 0x2ef1a00 (LWP 100837) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 217 Thread 0x2ef1c00 (LWP 100838) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 216 Thread 0x2ef1e00 (LWP 100839) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 215 Thread 0x2f07000 (LWP 100840) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 214 Thread 0x2f07200 (LWP 100841) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 213 Thread 0x2f07400 (LWP 100842) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 212 Thread 0x2f07600 (LWP 100843) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 211 Thread 0x2f07800 (LWP 100844) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 210 Thread 0x2f07a00 (LWP 100845) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 209 Thread 0x2f07c00 (LWP 100846) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 208 Thread 0x2f07e00 (LWP 100847) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 207 Thread 0x2f32000 (LWP 100848) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 206 Thread 0x2f32200 (LWP 100849) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 205 Thread 0x2f32400 (LWP 100850) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 204 Thread 0x2f32600 (LWP 100851) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 203 Thread 0x2f32800 (LWP 100852) 0x0000000800cc944c in writev () at writev.S:2 202 Thread 0x2f32a00 (LWP 100853) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 201 Thread 0x2f32c00 (LWP 100854) 0x0000000800cc944c in writev () at writev.S:2 200 Thread 0x2f32e00 (LWP 100855) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 199 Thread 0x2f3b000 (LWP 100856) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 198 Thread 0x2f3b200 (LWP 100857) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 197 Thread 0x2f3b400 (LWP 100858) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 196 Thread 0x2f3b800 (LWP 100859) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 195 Thread 0x2f3b600 (LWP 100860) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 194 Thread 0x2f3ba00 (LWP 100861) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 193 Thread 0x2f3bc00 (LWP 100862) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 192 Thread 0x2f3be00 (LWP 100863) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 191 Thread 0x2f3e000 (LWP 100864) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 190 Thread 0x2f3e200 (LWP 100865) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 189 Thread 0x2f3e400 (LWP 100866) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 188 Thread 0x2f3e600 (LWP 100867) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 187 Thread 0x2f3e800 (LWP 100868) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 186 Thread 0x2f3ea00 (LWP 100869) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 185 Thread 0x2f3ec00 (LWP 100870) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 184 Thread 0x2f3ee00 (LWP 100871) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 183 Thread 0x2f44000 (LWP 100872) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 182 Thread 0x2f44200 (LWP 100873) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 181 Thread 0x2f44400 (LWP 100874) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 180 Thread 0x2f44600 (LWP 100875) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 179 Thread 0x2f44800 (LWP 100876) 0x0000000800cc944c in writev () at writev.S:2 178 Thread 0x2f44a00 (LWP 100877) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 177 Thread 0x2f44c00 (LWP 100878) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 176 Thread 0x2f44e00 (LWP 100879) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 175 Thread 0x2f45000 (LWP 100880) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 174 Thread 0x2f45200 (LWP 100881) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 173 Thread 0x2f45400 (LWP 100882) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 172 Thread 0x2f45600 (LWP 100883) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 171 Thread 0x2f45800 (LWP 100884) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 170 Thread 0x2f45a00 (LWP 100885) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 169 Thread 0x2f45c00 (LWP 100886) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 168 Thread 0x2f49000 (LWP 100887) 0x0000000800cc944c in writev () at writev.S:2 167 Thread 0x2f45e00 (LWP 100888) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 166 Thread 0x2f49200 (LWP 100889) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 165 Thread 0x2f49400 (LWP 100890) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 164 Thread 0x2f49600 (LWP 100891) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 163 Thread 0x2f49800 (LWP 100892) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 162 Thread 0x2f49a00 (LWP 100893) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 161 Thread 0x2f49c00 (LWP 100894) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 160 Thread 0x2f49e00 (LWP 100895) 0x0000000800cc944c in writev () at writev.S:2 159 Thread 0x2f7a000 (LWP 100896) 0x0000000800cc944c in writev () at writev.S:2 158 Thread 0x2f7a200 (LWP 100897) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 157 Thread 0x2f7a400 (LWP 100898) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 156 Thread 0x2f7a600 (LWP 100899) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 155 Thread 0x2f7a800 (LWP 100900) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 154 Thread 0x2f7aa00 (LWP 100901) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 153 Thread 0x2f7ac00 (LWP 100902) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 152 Thread 0x2f7ae00 (LWP 100903) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 151 Thread 0x2f80000 (LWP 100904) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 150 Thread 0x2f80200 (LWP 100905) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 149 Thread 0x2f80400 (LWP 100906) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 148 Thread 0x2f80600 (LWP 100907) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 147 Thread 0x2f80800 (LWP 100908) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 146 Thread 0x2f80a00 (LWP 100909) 0x0000000800ce6bcc in read () at read.S:2 145 Thread 0x2f80c00 (LWP 100910) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 144 Thread 0x2f80e00 (LWP 100911) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 143 Thread 0x2fac000 (LWP 100912) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 142 Thread 0x2fac200 (LWP 100913) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 141 Thread 0x2fac400 (LWP 100914) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 140 Thread 0x2fac600 (LWP 100915) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 139 Thread 0x2fac800 (LWP 100916) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 138 Thread 0x2faca00 (LWP 100917) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 137 Thread 0x2facc00 (LWP 100918) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 136 Thread 0x2face00 (LWP 100919) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 135 Thread 0x2fdf000 (LWP 100920) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 134 Thread 0x2fdf200 (LWP 100921) 0x000000000040fc8a in CNT_Session ( sp=0x4f60008) at cache_center.c:809 133 Thread 0x2fdf400 (LWP 100922) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 132 Thread 0x2fdf600 (LWP 100923) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 131 Thread 0x2fdf800 (LWP 100924) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 130 Thread 0x2fdfa00 (LWP 100925) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 129 Thread 0x2fdfc00 (LWP 100926) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 128 Thread 0x2fdfe00 (LWP 100927) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 127 Thread 0x2df7000 (LWP 100928) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 126 Thread 0x2df7200 (LWP 100929) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 125 Thread 0x2df7400 (LWP 100930) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 124 Thread 0x2df7600 (LWP 100931) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 123 Thread 0x2df7800 (LWP 100932) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 122 Thread 0x2df7a00 (LWP 100933) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 121 Thread 0x2df7c00 (LWP 100934) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 120 Thread 0x2df7e00 (LWP 100935) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 119 Thread 0x3024000 (LWP 100936) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 118 Thread 0x3024200 (LWP 100937) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 117 Thread 0x3024400 (LWP 100938) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 116 Thread 0x3024600 (LWP 100939) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 115 Thread 0x3024800 (LWP 100940) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 114 Thread 0x3024a00 (LWP 100941) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 113 Thread 0x3024c00 (LWP 100942) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 112 Thread 0x3024e00 (LWP 100943) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 111 Thread 0x303b000 (LWP 100944) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 110 Thread 0x303b200 (LWP 100945) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 109 Thread 0x303b400 (LWP 100946) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 108 Thread 0x303b600 (LWP 100947) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 107 Thread 0x303b800 (LWP 100948) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 106 Thread 0x303ba00 (LWP 100949) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 105 Thread 0x303bc00 (LWP 100950) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 104 Thread 0x3053000 (LWP 100951) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 103 Thread 0x303be00 (LWP 100952) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 102 Thread 0x3053200 (LWP 100953) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 101 Thread 0x3053400 (LWP 100954) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 100 Thread 0x3053600 (LWP 100955) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 99 Thread 0x3053800 (LWP 100956) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 98 Thread 0x3053a00 (LWP 100957) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 97 Thread 0x3053c00 (LWP 100958) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 96 Thread 0x3053e00 (LWP 100959) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 95 Thread 0x3063000 (LWP 100960) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 94 Thread 0x3063200 (LWP 100961) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 93 Thread 0x3063400 (LWP 100962) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 92 Thread 0x3063600 (LWP 100963) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 91 Thread 0x3063800 (LWP 100964) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 90 Thread 0x3063a00 (LWP 100965) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 89 Thread 0x3063c00 (LWP 100966) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 88 Thread 0x3063e00 (LWP 100967) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 87 Thread 0x306e000 (LWP 100968) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 86 Thread 0x306e200 (LWP 100969) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 85 Thread 0x306e400 (LWP 100970) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 84 Thread 0x306e600 (LWP 100971) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 83 Thread 0x306e800 (LWP 100972) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 82 Thread 0x306ea00 (LWP 100973) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 81 Thread 0x306ec00 (LWP 100974) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 80 Thread 0x306ee00 (LWP 100975) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 79 Thread 0x30b0000 (LWP 100976) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 78 Thread 0x30b0200 (LWP 100977) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 77 Thread 0x30b0400 (LWP 100978) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 76 Thread 0x30b0600 (LWP 100979) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 75 Thread 0x30b0800 (LWP 100980) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 74 Thread 0x30b0a00 (LWP 100981) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 73 Thread 0x30b0c00 (LWP 100982) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 72 Thread 0x30b0e00 (LWP 100983) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 71 Thread 0x30b7000 (LWP 100984) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 70 Thread 0x30b7200 (LWP 100985) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 69 Thread 0x30b7400 (LWP 100986) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 68 Thread 0x30b7600 (LWP 100987) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 67 Thread 0x30b7800 (LWP 100988) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 66 Thread 0x30b7a00 (LWP 100989) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 65 Thread 0x30b7c00 (LWP 100990) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 64 Thread 0x30b7e00 (LWP 100991) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 63 Thread 0x30bb200 (LWP 100992) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 62 Thread 0x30bb000 (LWP 100993) 0x0000000800cc944c in writev () at writev.S:2 61 Thread 0x30bb400 (LWP 100994) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 60 Thread 0x30bb600 (LWP 100995) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 59 Thread 0x30bb800 (LWP 100996) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 58 Thread 0x30bba00 (LWP 100997) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 57 Thread 0x30bbc00 (LWP 100998) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 56 Thread 0x30bbe00 (LWP 100999) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 55 Thread 0x30c8000 (LWP 101000) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 54 Thread 0x30c8200 (LWP 101001) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 53 Thread 0x30c8400 (LWP 101002) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 52 Thread 0x30c8600 (LWP 101003) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 51 Thread 0x30c8800 (LWP 101004) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 50 Thread 0x30c8a00 (LWP 101005) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 49 Thread 0x30c8c00 (LWP 101006) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 ---Type to continue, or q to quit--- 48 Thread 0x30c8e00 (LWP 101007) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 47 Thread 0x30d9200 (LWP 101008) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 46 Thread 0x30d9000 (LWP 101009) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 45 Thread 0x30d9400 (LWP 101010) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 44 Thread 0x30d9600 (LWP 101011) memcpy () at /usr/src/lib/libc/amd64/string/bcopy.S:65 43 Thread 0x30d9800 (LWP 101012) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 42 Thread 0x30d9a00 (LWP 101013) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 41 Thread 0x30d9c00 (LWP 101014) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 40 Thread 0x30d9e00 (LWP 101015) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 39 Thread 0x30ec200 (LWP 101016) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 38 Thread 0x30ec000 (LWP 101017) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 37 Thread 0x30ec400 (LWP 101018) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 36 Thread 0x30ec600 (LWP 101019) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 35 Thread 0x30ec800 (LWP 101020) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 34 Thread 0x30eca00 (LWP 101021) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 33 Thread 0x30ecc00 (LWP 101022) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 32 Thread 0x30ece00 (LWP 101023) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 31 Thread 0x3111000 (LWP 101024) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 30 Thread 0x3111200 (LWP 101025) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 29 Thread 0x3111400 (LWP 101026) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 28 Thread 0x3111600 (LWP 101027) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 27 Thread 0x3111800 (LWP 101028) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 26 Thread 0x3111a00 (LWP 101029) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 25 Thread 0x3111c00 (LWP 101030) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 24 Thread 0x3111e00 (LWP 101031) 0x0000000800c3249c in _umtx_op () ---Type to continue, or q to quit--- at _umtx_op.S:2 23 Thread 0x3130000 (LWP 101032) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 22 Thread 0x3130200 (LWP 101033) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 21 Thread 0x3130400 (LWP 101034) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 20 Thread 0x3130600 (LWP 101035) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 19 Thread 0x3130800 (LWP 101036) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 18 Thread 0x3130a00 (LWP 101037) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 17 Thread 0x3130c00 (LWP 101038) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 16 Thread 0x3130e00 (LWP 101039) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 15 Thread 0x3146200 (LWP 101040) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 14 Thread 0x3146000 (LWP 101041) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 13 Thread 0x3146400 (LWP 101042) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 12 Thread 0x3146600 (LWP 101043) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 11 Thread 0x3146800 (LWP 101044) 0x0000000800cc944c in writev () at writev.S:2 10 Thread 0x3146a00 (LWP 101045) 0x0000000800cc944c in writev () at writev.S:2 9 Thread 0x3146c00 (LWP 101046) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 8 Thread 0x3146e00 (LWP 101047) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 7 Thread 0x3149000 (LWP 101048) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 6 Thread 0x3149200 (LWP 101049) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 5 Thread 0x3149400 (LWP 101050) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 4 Thread 0x3149600 (LWP 101051) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 3 Thread 0x3149800 (LWP 101052) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 2 Thread 0x3149a00 (LWP 101053) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 1 Thread 0x3149c00 (LWP 101054) 0x0000000800c3249c in _umtx_op () at _umtx_op.S:2 (gdb) }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Oct 18 08:32:22 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 18 Oct 2007 08:32:22 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.5d75b9d96f131af9f9565ded195e50af@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: major | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Changes (by anders): * priority: normal => high * severity: normal => major Comment: Just to let you know this happens often for me, still, here is another backtrace from today (using trunk 2096 with poll instead of kqueue): {{{ Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x3b62800 (LWP 100202)] 0x0000000800c5c46e in strtoul () from /lib/libc.so.6 (gdb) bt #0 0x0000000800c5c46e in strtoul () from /lib/libc.so.6 #1 0x0000000000414938 in http_DissectResponse (w=0x7ffffd7ecae0, htc=0x7ffffd7ea980, hp=0x6264038) at cache_http.c:509 #2 0x0000000000411d62 in Fetch (sp=0x2fb6008) at cache_fetch.c:305 #3 0x000000000040e978 in cnt_fetch (sp=0x2fb6008) at cache_center.c:301 #4 0x000000000040fc3e in CNT_Session (sp=0x2fb6008) at steps.h:40 #5 0x0000000000417588 in wrk_do_one (w=0x7ffffd7ecae0) at cache_pool.c:193 #6 0x0000000000417a48 in wrk_thread (priv=0x85b300) at cache_pool.c:245 #7 0x000000080098f55b in pthread_create () from /usr/lib/libthr.so.2 #8 0x00007ffffd5ed000 in ?? () Error accessing memory address 0x7ffffd7ed000: Bad address. (gdb) frame 1 #1 0x0000000000414938 in http_DissectResponse (w=0x7ffffd7ecae0, htc=0x7ffffd7ea980, hp=0x6264038) at cache_http.c:509 509 hp->status = (gdb) print *hp $1 = {magic = 1680389577, ws = 0x6264018, conds = 0 '\0', logtag = HTTP_Rx, status = 200, hd = {{b = 0x4306c8 "GET", e = 0x4306cb ""}, { b = 0x2fb653c "/mmo/0/111/587/80_-430841943_thumb.jpg", e = 0x2fb6562 ""}, {b = 0x6264280 "HTTP/1.1", e = 0x430522 ""}, { b = 0x0, e = 0x0}, {b = 0x0, e = 0x0}, {b = 0x2fb656d "Via: 1.1 ISA-01", e = 0x2fb657c ""}, { b = 0x2fb657e "Cookie: IAD_CAMPAIGN=_blank; IADVISITOR=351042594; RMID=947a33c247171810; WT_FPC=id=21f2045fa025b762a611192692091038:lv=1192692114601:ss=1192692091038; didReplyCar26=1", e = 0x2fb6625 ""}, { b = 0x2fb6627 "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; .NET CLR 1.0.3705)", e = 0x2fb668b ""}, { b = 0x2fb668d "Host: cache.finn.no", e = 0x2fb66a0 ""}, { b = 0x2fb66a2 "Accept: */*", e = 0x2fb66ad ""}, { b = 0x2fb66af "Referer: http://www.finn.no/finn/car/used/result", e = 0x2fb66df ""}, {b = 0x2fb66e1 "Accept-Language: no", e = 0x2fb66f4 ""}, {b = 0x6264a05 "X-Varnish: 64188579", e = 0x6264a18 ""}, {b = 0x6264a19 "X-Forwarded-for: 148.122.51.194", e = 0x6264a38 ""}, {b = 0x0, e = 0x0} }, hdf = '\0' , nhd = 14} (gdb) }}} Any ideas? -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Oct 18 11:29:33 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 18 Oct 2007 11:29:33 -0000 Subject: [Varnish] #168: varnish crashes when using varnishadmin.pl (from the list) with wrong parameters In-Reply-To: <060.a25c1357e596340e3b60534af2469250@projects.linpro.no> References: <060.a25c1357e596340e3b60534af2469250@projects.linpro.no> Message-ID: <069.8ae41617883f94903d205d0f50855b09@projects.linpro.no> #168: varnish crashes when using varnishadmin.pl (from the list) with wrong parameters ----------------------------+----------------------------------------------- Reporter: denis at zeno.org | Owner: des Type: defect | Status: new Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Resolution: Keywords: coredump | ----------------------------+----------------------------------------------- Changes (by anders): * cc: anders at fupp.net (added) Comment: This problem occurs for me also. It happens exactly at the time when I disconnect from the connection to the management interface (using telnet): {{{ Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. vcl.load run1 /usr/local/etc/varnish.vcl 200 0 vcl.use run1 200 0 ^] telnet> close Connection closed. }}} Then, this is printed on the console: {{{ Assert error in mgt_cli_callback(), mgt_cli.c line 352: Condition((close(cp->fdo)) == 0) not true. errno = 9 (Bad file descriptor) }}} Tracing the core dump, I get: {{{ Program terminated with signal 6, Aborted. Reading symbols from /usr/local/lib/libvarnish.so.0...done. Loaded symbols for /usr/local/lib/libvarnish.so.0 Reading symbols from /usr/local/lib/libvarnishcompat.so.0...done. Loaded symbols for /usr/local/lib/libvarnishcompat.so.0 Reading symbols from /usr/local/lib/libvcl.so.0...done. Loaded symbols for /usr/local/lib/libvcl.so.0 Reading symbols from /usr/lib/libthr.so.2...done. Loaded symbols for /usr/lib/libthr.so.2 Reading symbols from /lib/libm.so.4...done. Loaded symbols for /lib/libm.so.4 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x0000000800c882dc in kill () from /lib/libc.so.6 [New Thread 0x540000 (LWP 100114)] (gdb) bt #0 0x0000000800c882dc in kill () from /lib/libc.so.6 #1 0x000000080098d02e in raise () from /usr/lib/libthr.so.2 #2 0x0000000800c8716d in abort () from /lib/libc.so.6 #3 0x000000080066ab99 in lbv_assert (func=0x433a00 "mgt_cli_callback", file=0x43318b "mgt_cli.c", line=352, cond=0x433aa9 "(close(cp->fdo)) == 0", err=9) at assert.c:58 #4 0x0000000000423a1c in mgt_cli_callback (e=0x547880, what=1) at mgt_cli.c:352 #5 0x0000000000425288 in ev_schedule_one (evb=0x54d100) at mgt_event.c:454 #6 0x0000000000424c9a in ev_schedule (evb=0x54d100) at mgt_event.c:327 #7 0x0000000000422567 in mgt_run (dflag=2, T_arg=0x7fffffffe9ad "localhost:8080") at mgt_child.c:428 #8 0x000000000042d767 in main (argc=0, argv=0x7fffffffe748) at varnishd.c:559 (gdb) }}} I also use Varnish/trunk 2096. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Oct 18 13:39:41 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 18 Oct 2007 13:39:41 -0000 Subject: [Varnish] #47: pass mode does not handle POST In-Reply-To: <049.dce81b215711ded70632456b6f336a2b@projects.linpro.no> References: <049.dce81b215711ded70632456b6f336a2b@projects.linpro.no> Message-ID: <058.34d321b91b101712209f2469b1a6f193@projects.linpro.no> #47: pass mode does not handle POST ----------------------+----------------------------------------------------- Reporter: des | Owner: cecilihf Type: defect | Status: closed Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: fixed Keywords: | ----------------------+----------------------------------------------------- Changes (by cecilihf): * status: new => closed * resolution: => fixed Comment: Fixed in r2115 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Oct 19 11:10:24 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 19 Oct 2007 11:10:24 -0000 Subject: [Varnish] #104: varnishncsa assertion fails sometimes In-Reply-To: <055.b2f86f700ad9117f39ea76f3dd2e454e@projects.linpro.no> References: <055.b2f86f700ad9117f39ea76f3dd2e454e@projects.linpro.no> Message-ID: <064.f798e2371e7348a9fce168c56b2fe767@projects.linpro.no> #104: varnishncsa assertion fails sometimes -------------------------+-------------------------------------------------- Reporter: dirkgomez | Owner: des Type: defect | Status: closed Priority: normal | Milestone: Component: varnishncsa | Version: 1.0.3 Severity: normal | Resolution: worksforme Keywords: | -------------------------+-------------------------------------------------- Changes (by des): * status: assigned => closed * resolution: => worksforme Comment: no answer from originator -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Oct 19 13:50:30 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 19 Oct 2007 13:50:30 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.6cfc8cd51e01e654ec256df485da700e@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: des Type: defect | Status: assigned Priority: high | Milestone: Component: varnishd | Version: trunk Severity: major | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Changes (by des): * owner: phk => des * status: new => assigned -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Oct 19 13:53:48 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 19 Oct 2007 13:53:48 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.28a0b9f0222af4dce607cf2d442a8a51@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: des Type: defect | Status: closed Priority: high | Milestone: Varnish 1.2 Component: varnishd | Version: trunk Severity: major | Resolution: fixed Keywords: varnishd core dump | --------------------------------+------------------------------------------- Changes (by des): * status: assigned => closed * resolution: => fixed * milestone: => Varnish 1.2 Comment: Fixed in r2151. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Oct 19 13:54:53 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 19 Oct 2007 13:54:53 -0000 Subject: [Varnish] #168: varnish crashes when using varnishadmin.pl (from the list) with wrong parameters In-Reply-To: <060.a25c1357e596340e3b60534af2469250@projects.linpro.no> References: <060.a25c1357e596340e3b60534af2469250@projects.linpro.no> Message-ID: <069.9e051a0f115b2c27a6a133132d046596@projects.linpro.no> #168: varnish crashes when using varnishadmin.pl (from the list) with wrong parameters ----------------------------+----------------------------------------------- Reporter: denis at zeno.org | Owner: des Type: defect | Status: assigned Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Resolution: Keywords: coredump | ----------------------------+----------------------------------------------- Changes (by des): * status: new => assigned -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Oct 19 13:55:00 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 19 Oct 2007 13:55:00 -0000 Subject: [Varnish] #168: varnish crashes when using varnishadmin.pl (from the list) with wrong parameters In-Reply-To: <060.a25c1357e596340e3b60534af2469250@projects.linpro.no> References: <060.a25c1357e596340e3b60534af2469250@projects.linpro.no> Message-ID: <069.d9da0afeed228c77fc45a3d3a4da7e31@projects.linpro.no> #168: varnish crashes when using varnishadmin.pl (from the list) with wrong parameters ----------------------------+----------------------------------------------- Reporter: denis at zeno.org | Owner: des Type: defect | Status: assigned Priority: normal | Milestone: Varnish 1.2 Component: build | Version: trunk Severity: normal | Resolution: Keywords: coredump | ----------------------------+----------------------------------------------- Changes (by des): * milestone: => Varnish 1.2 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Oct 19 13:58:26 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 19 Oct 2007 13:58:26 -0000 Subject: [Varnish] #168: varnish crashes when using varnishadmin.pl (from the list) with wrong parameters In-Reply-To: <060.a25c1357e596340e3b60534af2469250@projects.linpro.no> References: <060.a25c1357e596340e3b60534af2469250@projects.linpro.no> Message-ID: <069.e90513ccf4992b29174dd573270f0806@projects.linpro.no> #168: varnish crashes when using varnishadmin.pl (from the list) with wrong parameters ----------------------------+----------------------------------------------- Reporter: denis at zeno.org | Owner: des Type: defect | Status: closed Priority: normal | Milestone: Varnish 1.2 Component: build | Version: trunk Severity: normal | Resolution: fixed Keywords: coredump | ----------------------------+----------------------------------------------- Changes (by des): * status: assigned => closed * resolution: => fixed Comment: Fixed in r2153. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Oct 19 16:28:05 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 19 Oct 2007 16:28:05 -0000 Subject: [Varnish] #164: Varnishd core dumps because of segmentation violation In-Reply-To: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> References: <052.056d420b6ae1f4a78c1b835427e4222f@projects.linpro.no> Message-ID: <061.e4b4990cf07dc2a85b6a8105ce25227b@projects.linpro.no> #164: Varnishd core dumps because of segmentation violation --------------------------------+------------------------------------------- Reporter: anders | Owner: des Type: defect | Status: reopened Priority: high | Milestone: Varnish 1.2 Component: varnishd | Version: trunk Severity: major | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Changes (by anders): * status: closed => reopened * resolution: fixed => Comment: Tried with r2151 on top of what I have, then got a new assert: {{{ Child said (2, 19251): <> Cache child died pid=19251 status=0x86 }}} Backtrace: {{{ (gdb) bt #0 0x0000000800c0b48c in thr_kill () from /lib/libc.so.7 #1 0x0000000800c9563b in abort () from /lib/libc.so.7 #2 0x000000080066dc7f in lbv_assert (func=Could not find the frame base for "lbv_assert". ) at assert.c:58 #3 0x0000000000414734 in http_dissect_hdrs (w=0x7fffe0d07ae0, hp=0x2d8297d038, fd=1976, p=0x2d8297d2b6 "Server: Apache", t= {b = 0x2d8297d280 "X-Varnish:", e = 0x2d8297d280 "X-Varnish:"}) at cache_http.c:365 #4 0x0000000000414d34 in http_splitline (w=0x7fffe0d07ae0, fd=1976, hp=0x2d8297d038, htc=0x7fffe0d05990, h1=2, h2=3, h3=4) at cache_http.c:462 #5 0x0000000000414fee in http_DissectResponse (w=0x7fffe0d07ae0, htc=0x7fffe0d05990, hp=0x2d8297d038) at cache_http.c:501 #6 0x0000000000412237 in Fetch (sp=0x2d84742008) at cache_fetch.c:305 #7 0x000000000040ed02 in cnt_fetch (sp=0x2d84742008) at cache_center.c:301 #8 0x000000000041006c in CNT_Session (sp=0x2d84742008) at steps.h:40 #9 0x0000000000417d7d in wrk_do_one (w=0x7fffe0d07ae0) at cache_pool.c:193 #10 0x0000000000418227 in wrk_thread (priv=0x800e111e0) at cache_pool.c:245 #11 0x00000008009929a8 in pthread_getprio () from /lib/libthr.so.3 #12 0x0000000000000000 in ?? () Cannot access memory at address 0x7fffe0d08000 }}} So this bug seems not fully fixed yet. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Sat Oct 20 15:20:24 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Sat, 20 Oct 2007 15:20:24 -0000 Subject: [Varnish] #167: Varnishd crashes with assert error in RES_WriteObj In-Reply-To: <052.0d60ec784fa5d17383428842e133f8e7@projects.linpro.no> References: <052.0d60ec784fa5d17383428842e133f8e7@projects.linpro.no> Message-ID: <061.03bbc86334063d1f5d051cb84a94604f@projects.linpro.no> #167: Varnishd crashes with assert error in RES_WriteObj ---------------------------------------------+------------------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump RES_WriteObj | ---------------------------------------------+------------------------------ Comment (by anders): Got this one again, while running trunk/2096 including patches for ticket 168 (changeset r2153) and ticket 164 (changeset 2151): {{{ Child said (2, 51658): <obj->len) not true. errno = 32 (Broken pipe) >> Cache child died pid=51658 status=0x86 }}} The backtrace: {{{ (gdb) bt #0 0x0000000800c0b48c in thr_kill () from /lib/libc.so.7 #1 0x0000000800c9563b in abort () from /lib/libc.so.7 #2 0x000000080066dc7f in lbv_assert (func=Could not find the frame base for "lbv_assert". ) at assert.c:58 #3 0x0000000000419ce9 in RES_WriteObj (sp=0x2d84329008) at cache_response.c:184 #4 0x000000000040e626 in cnt_deliver (sp=0x2d84329008) at cache_center.c:162 #5 0x000000000041007a in CNT_Session (sp=0x2d84329008) at steps.h:41 #6 0x0000000000417d7d in wrk_do_one (w=0x7ffffafd8ae0) at cache_pool.c:193 #7 0x0000000000418227 in wrk_thread (priv=0x800e11200) at cache_pool.c:245 #8 0x00000008009929a8 in pthread_getprio () from /lib/libthr.so.3 #9 0x00007ffffadd9000 in ?? () Cannot access memory at address 0x7ffffafd9000 (gdb) frame 3 #3 0x0000000000419ce9 in RES_WriteObj (sp=0x2d84329008) at cache_response.c:184 184 assert(u == sp->obj->len); }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Sat Oct 20 17:42:35 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Sat, 20 Oct 2007 17:42:35 -0000 Subject: [Varnish] #167: Varnishd crashes with assert error in RES_WriteObj In-Reply-To: <052.0d60ec784fa5d17383428842e133f8e7@projects.linpro.no> References: <052.0d60ec784fa5d17383428842e133f8e7@projects.linpro.no> Message-ID: <061.00c723e49437db118856a23d76ca2a5b@projects.linpro.no> #167: Varnishd crashes with assert error in RES_WriteObj ---------------------------------------------+------------------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump RES_WriteObj | ---------------------------------------------+------------------------------ Comment (by anders): Same thing again, but with different errno: {{{ Child said (2, 899): <obj->len) not true. errno = 14 (Bad address) >> Cache child died pid=899 status=0x86 }}} Backtrace: {{{ (gdb) bt #0 0x0000000800c0b48c in thr_kill () from /lib/libc.so.7 #1 0x0000000800c9563b in abort () from /lib/libc.so.7 #2 0x000000080066dc7f in lbv_assert (func=Could not find the frame base for "lbv_assert". ) at assert.c:58 #3 0x0000000000419ce9 in RES_WriteObj (sp=0x2d848ec008) at cache_response.c:184 #4 0x000000000040e626 in cnt_deliver (sp=0x2d848ec008) at cache_center.c:162 #5 0x000000000041007a in CNT_Session (sp=0x2d848ec008) at steps.h:41 #6 0x0000000000417d7d in wrk_do_one (w=0x7ffff178cae0) at cache_pool.c:193 #7 0x0000000000418227 in wrk_thread (priv=0x800e111e0) at cache_pool.c:245 #8 0x00000008009929a8 in pthread_getprio () from /lib/libthr.so.3 #9 0x00007ffff158d000 in ?? () Cannot access memory at address 0x7ffff178d000 }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Sat Oct 20 21:44:48 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Sat, 20 Oct 2007 21:44:48 -0000 Subject: [Varnish] #170: Segmentation violation in HTC_Read at cache_httpconn.c:195 Message-ID: <052.544b68a3ff13ba083233aa7d256ffb88@projects.linpro.no> #170: Segmentation violation in HTC_Read at cache_httpconn.c:195 --------------------+------------------------------------------------------- Reporter: anders | Owner: des Type: defect | Status: new Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Keywords: varnishd core dump --------------------+------------------------------------------------------- Giving branches/1.2 (up to date to commit 2153) a spin, I get a segfault: Cache child died pid=28417 status=0x8b {{{ (gdb) bt #0 0x0000000800d9cff6 in memcpy () from /lib/libc.so.7 #1 0x0000000000417513 in HTC_Read (htc=0x7fffead55980, d=0x898133000, len=27188) at cache_httpconn.c:195 #2 0x000000000041167c in fetch_straight (sp=0x2d84c5b008, htc=0x7fffead55980, b=0x2d8321b37f "27188") at cache_fetch.c:72 #3 0x000000000041244e in Fetch (sp=0x2d84c5b008) at cache_fetch.c:360 #4 0x000000000040ece2 in cnt_fetch (sp=0x2d84c5b008) at cache_center.c:301 #5 0x000000000041004c in CNT_Session (sp=0x2d84c5b008) at steps.h:40 #6 0x0000000000417ecd in wrk_do_one (w=0x7fffead57ae0) at cache_pool.c:193 #7 0x00000000004183c6 in wrk_thread (priv=0x800f111e0) at cache_pool.c:246 #8 0x0000000800a979a8 in pthread_getprio () from /lib/libthr.so.3 #9 0x00007fffeab58000 in ?? () Cannot access memory at address 0x7fffead58000 (gdb) frame 1 #1 0x0000000000417513 in HTC_Read (htc=0x7fffead55980, d=0x898133000, len=27188) at cache_httpconn.c:195 195 memcpy(p, htc->pipeline.b, l); (gdb) print htc $1 = (struct http_conn *) 0x7fffead55980 (gdb) print *htc $2 = {magic = 1041886673, fd = 20, ws = 0x2d8321b018, rxbuf = { b = 0x2d8321b280 "HTTP/1.1", e = 0x2d8321b3b5 "????"}, pipeline = { b = 0x2d8321b3b5 "????", e = 0x2d8321bde8 ""}} (gdb) info threads 231 Thread 0x800f01120 (LWP 100213) 0x0000000800d4ca4c in poll () from /lib/libc.so.7 230 Thread 0x800f01290 (LWP 100232) 0x0000000800d8248c in nanosleep () from /lib/libc.so.7 229 Thread 0x800f01400 (LWP 100233) 0x0000000800d8248c in nanosleep () from /lib/libc.so.7 228 Thread 0x800f01570 (LWP 100234) 0x0000000800d8248c in nanosleep () from /lib/libc.so.7 227 Thread 0x800f016e0 (LWP 100235) 0x0000000800d8360c in kevent () from /lib/libc.so.7 226 Thread 0x800f01850 (LWP 100237) 0x0000000800d4ca4c in poll () from /lib/libc.so.7 225 Thread 0x2d82007e10 (LWP 100269) 0x0000000800d9d73c in read () from /lib/libc.so.7 224 Thread 0x2d820083d0 (LWP 100295) 0x0000000800d9d73c in read () from /lib/libc.so.7 223 Thread 0x2d8200c8d0 (LWP 100396) 0x0000000800d9d73c in read () from /lib/libc.so.7 222 Thread 0x801b110f0 (LWP 100438) 0x0000000800d9d73c in read () from /lib/libc.so.7 221 Thread 0x801b12680 (LWP 100474) 0x0000000800d9d73c in read () from /lib/libc.so.7 220 Thread 0x2d8619e820 (LWP 100526) 0x0000000800d9d73c in read () from /lib/libc.so.7 219 Thread 0x801b14be0 (LWP 100542) 0x0000000800d9d73c in read () from /lib/libc.so.7 218 Thread 0x801b12db0 (LWP 100272) 0x0000000800d9d73c in read () from /lib/libc.so.7 217 Thread 0x2d820076e0 (LWP 100336) 0x0000000800d9d73c in read () from /lib/libc.so.7 216 Thread 0x2d82008990 (LWP 100364) 0x0000000800d9d73c in read () from /lib/libc.so.7 215 Thread 0x2d8619e3d0 (LWP 100366) 0x0000000800d9d73c in read () from /lib/libc.so.7 214 Thread 0x2d8619d290 (LWP 100374) 0x0000000800d9d73c in read () from /lib/libc.so.7 213 Thread 0x2d83ee1120 (LWP 100391) 0x0000000800d9d73c in read () from /lib/libc.so.7 212 Thread 0x2d8200a650 (LWP 100367) 0x0000000800d9d73c in read () from /lib/libc.so.7 211 Thread 0x2d83ee3230 (LWP 100372) 0x0000000800d9d73c in read () from /lib/libc.so.7 210 Thread 0x2d8619ec70 (LWP 100395) 0x0000000800d9d73c in read () from /lib/libc.so.7 209 Thread 0x2d83ee5ec0 (LWP 100400) 0x0000000800d9d73c in read () from /lib/libc.so.7 208 Thread 0x2d8200aaa0 (LWP 100434) 0x0000000800d9d73c in read () from /lib/libc.so.7 207 Thread 0x2d83ee3f20 (LWP 100440) 0x0000000800d9d73c in read () from /lib/libc.so.7 206 Thread 0x801b14340 (LWP 100441) 0x0000000800d9d73c in read () from /lib/libc.so.7 205 Thread 0x2d82009c40 (LWP 100363) 0x0000000800d9d73c in read () from /lib/libc.so.7 204 Thread 0x2d8619d120 (LWP 100378) 0x0000000800d9d73c in read () from /lib/libc.so.7 203 Thread 0x2d8619d9c0 (LWP 100402) 0x0000000800d8800c in writev () from /lib/libc.so.7 202 Thread 0x2d82007b30 (LWP 100409) 0x0000000800d9d73c in read () from /lib/libc.so.7 201 Thread 0x2d820079c0 (LWP 100243) 0x0000000800d9d73c in read () from /lib/libc.so.7 200 Thread 0x2d83ee6310 (LWP 100244) 0x0000000800d9d73c in read () from /lib/libc.so.7 199 Thread 0x2d82008f50 (LWP 100252) 0x0000000800d9d73c in read () from /lib/libc.so.7 198 Thread 0x801b12ad0 (LWP 100274) 0x0000000800d9d73c in read () from /lib/libc.so.7 197 Thread 0x2d83ee4d80 (LWP 100339) 0x0000000800d9d73c in read () from /lib/libc.so.7 196 Thread 0x2d83ee1290 (LWP 100376) 0x0000000800d9d73c in read () from /lib/libc.so.7 195 Thread 0x2d8200c480 (LWP 100386) 0x0000000800d9d73c in read () from /lib/libc.so.7 194 Thread 0x801b13ef0 (LWP 100432) 0x0000000800d9d73c in read () from /lib/libc.so.7 193 Thread 0x801b141d0 (LWP 100442) 0x0000000800d9d73c in read () from /lib/libc.so.7 192 Thread 0x2d83ee2c70 (LWP 100446) 0x0000000800d8364c in connect () from /lib/libc.so.7 191 Thread 0x2d82007ca0 (LWP 100449) 0x0000000800d9d73c in read () from /lib/libc.so.7 190 Thread 0x2d83ee5340 (LWP 100454) 0x0000000800d9d73c in read () from /lib/libc.so.7 189 Thread 0x2d8200b790 (LWP 100459) 0x0000000800d9d73c in read () from /lib/libc.so.7 188 Thread 0x2d82008c70 (LWP 100469) 0x0000000800d8364c in connect () from /lib/libc.so.7 187 Thread 0x2d83ee44e0 (LWP 100473) 0x0000000800d9d73c in read () from /lib/libc.so.7 186 Thread 0x2d8200b900 (LWP 100477) 0x0000000800d8364c in connect () from /lib/libc.so.7 185 Thread 0x2d83ee6d20 (LWP 100487) 0x0000000800d8800c in writev () from /lib/libc.so.7 184 Thread 0x2d83ee5790 (LWP 100490) 0x0000000800d9d73c in read () from /lib/libc.so.7 183 Thread 0x801b12f20 (LWP 100493) 0x0000000800d9d73c in read () from /lib/libc.so.7 182 Thread 0x801b13c10 (LWP 100495) 0x0000000800d9d73c in read () from /lib/libc.so.7 181 Thread 0x2d83ee2540 (LWP 100498) 0x0000000800d9d73c in read () from /lib/libc.so.7 180 Thread 0x801b10850 (LWP 100502) 0x0000000800d8800c in writev () from /lib/libc.so.7 179 Thread 0x801b10e10 (LWP 100506) 0x0000000800d9cff6 in memcpy () from /lib/libc.so.7 178 Thread 0x2d83ee5900 (LWP 100510) 0x0000000800d9d73c in read () from /lib/libc.so.7 177 Thread 0x2d820080f0 (LWP 100511) 0x0000000800d9d73c in read () from /lib/libc.so.7 176 Thread 0x2d83ee2de0 (LWP 100513) 0x0000000800d9d73c in read () from /lib/libc.so.7 175 Thread 0x2d82008de0 (LWP 100515) 0x0000000800d9d73c in read () from /lib/libc.so.7 174 Thread 0x2d82007850 (LWP 100241) 0x0000000800d9d73c in read () from /lib/libc.so.7 173 Thread 0x2d82008b00 (LWP 100245) 0x0000000800d9d73c in read () from /lib/libc.so.7 172 Thread 0x2d8619d570 (LWP 100265) 0x0000000800d9d73c in read () from /lib/libc.so.7 171 Thread 0x2d8200b620 (LWP 100270) 0x0000000800d9d73c in read () from /lib/libc.so.7 170 Thread 0x2d8200b1d0 (LWP 100333) 0x0000000800d9d73c in read () from /lib/libc.so.7 169 Thread 0x2d82007570 (LWP 100335) 0x0000000800d9d73c in read () from /lib/libc.so.7 168 Thread 0x801b113d0 (LWP 100337) 0x0000000800d9d73c in read () from /lib/libc.so.7 167 Thread 0x2d83ee68d0 (LWP 100338) 0x0000000800d9d73c in read () from /lib/libc.so.7 166 Thread 0x801b109c0 (LWP 100358) 0x0000000800d9d73c in read () from /lib/libc.so.7 165 Thread 0x2d83ee23d0 (LWP 100359) 0x0000000800d9d73c in read () from /lib/libc.so.7 164 Thread 0x801b12960 (LWP 100263) 0x0000000800d9d73c in read () from /lib/libc.so.7 163 Thread 0x801b144b0 (LWP 100264) 0x0000000800d9d73c in read () from /lib/libc.so.7 162 Thread 0x2d83ee2990 (LWP 100268) 0x0000000800d9d73c in read () from /lib/libc.so.7 161 Thread 0x2d83ee20f0 (LWP 100298) 0x0000000800d8364c in connect () from /lib/libc.so.7 160 Thread 0x801b13930 (LWP 100330) 0x0000000800d9d73c in read () from /lib/libc.so.7 159 Thread 0x2d83ee1e10 (LWP 100332) 0x0000000800d9d73c in read () from /lib/libc.so.7 158 Thread 0x2d820086b0 (LWP 100334) 0x0000000800d9d73c in read () from /lib/libc.so.7 157 Thread 0x2d82009ad0 (LWP 100360) 0x0000000800d9d73c in read () from /lib/libc.so.7 156 Thread 0x2d8619d6e0 (LWP 100361) 0x0000000800d9d73c in read () from /lib/libc.so.7 155 Thread 0x801b13200 (LWP 100362) 0x0000000800d8364c in connect () from /lib/libc.so.7 154 Thread 0x2d83ee37f0 (LWP 100365) 0x0000000800d9d73c in read () from /lib/libc.so.7 153 Thread 0x2d8200c5f0 (LWP 100368) 0x0000000800d9d73c in read () from /lib/libc.so.7 152 Thread 0x2d83ee5d50 (LWP 100369) 0x0000000800d9d73c in read () from /lib/libc.so.7 151 Thread 0x2d8200c760 (LWP 100370) 0x0000000800d9d73c in read () from /lib/libc.so.7 150 Thread 0x801b14ec0 (LWP 100371) 0x0000000800d9d73c in read () from /lib/libc.so.7 149 Thread 0x2d83ee2260 (LWP 100373) 0x0000000800d9d73c in read () from /lib/libc.so.7 148 Thread 0x2d8619e540 (LWP 100377) 0x0000000800d9d73c in read () from /lib/libc.so.7 147 Thread 0x2d83ee3db0 (LWP 100379) 0x0000000800d9d73c in read () from /lib/libc.so.7 146 Thread 0x2d8200cbb0 (LWP 100380) 0x0000000800d9d73c in read () from /lib/libc.so.7 145 Thread 0x2d82007290 (LWP 100381) 0x0000000800d9d73c in read () from /lib/libc.so.7 144 Thread 0x2d83ee33a0 (LWP 100382) 0x0000000800d9d73c in read () from /lib/libc.so.7 143 Thread 0x2d83ee2b00 (LWP 100383) 0x0000000800d9d73c in read () from /lib/libc.so.7 142 Thread 0x2d82009960 (LWP 100384) 0x0000000800d9d73c in read () from /lib/libc.so.7 141 Thread 0x2d8200ac10 (LWP 100385) 0x0000000800d9d73c in read () from /lib/libc.so.7 140 Thread 0x2d8619f230 (LWP 100387) 0x0000000800d9d73c in read () from /lib/libc.so.7 139 Thread 0x2d83ee4ef0 (LWP 100388) 0x0000000800d9d73c in read () from /lib/libc.so.7 138 Thread 0x2d83ee30c0 (LWP 100389) 0x0000000800d9d73c in read () from /lib/libc.so.7 137 Thread 0x2d83ee65f0 (LWP 100390) 0x0000000800d9d73c in read () from /lib/libc.so.7 136 Thread 0x2d8619de10 (LWP 100392) 0x0000000800d9d73c in read () from /lib/libc.so.7 135 Thread 0x801b120c0 (LWP 100393) 0x0000000800d9d73c in read () from /lib/libc.so.7 134 Thread 0x801b12510 (LWP 100394) 0x0000000800d9d73c in read () from /lib/libc.so.7 133 Thread 0x2d83ee3ad0 (LWP 100397) 0x0000000800d9d73c in read () from /lib/libc.so.7 132 Thread 0x2d82009680 (LWP 100398) 0x0000000800d9d73c in read () from /lib/libc.so.7 131 Thread 0x2d8200c310 (LWP 100399) 0x0000000800d9d73c in read () from /lib/libc.so.7 130 Thread 0x2d820090c0 (LWP 100401) 0x0000000800d9d73c in read () from /lib/libc.so.7 129 Thread 0x2d83ee51d0 (LWP 100403) 0x0000000800d9d73c in read () from /lib/libc.so.7 128 Thread 0x2d82009f20 (LWP 100404) 0x0000000800d9d73c in read () from /lib/libc.so.7 127 Thread 0x801b14060 (LWP 100405) 0x0000000800d9d73c in read () from /lib/libc.so.7 126 Thread 0x2d82007120 (LWP 100406) 0x0000000800d9d73c in read () from /lib/libc.so.7 125 Thread 0x2d8200ad80 (LWP 100407) 0x0000000800d9d73c in read () from /lib/libc.so.7 124 Thread 0x2d8200b340 (LWP 100408) 0x0000000800d9d73c in read () from /lib/libc.so.7 123 Thread 0x801b11de0 (LWP 100410) 0x0000000800d9d73c in read () from /lib/libc.so.7 * 122 Thread 0x2d8200a090 (LWP 100411) 0x0000000800d9cff6 in memcpy () from /lib/libc.so.7 121 Thread 0x2d8619e6b0 (LWP 100412) 0x0000000800d9d73c in read () from /lib/libc.so.7 120 Thread 0x2d8200a370 (LWP 100425) 0x0000000800d9d73c in read () from /lib/libc.so.7 119 Thread 0x2d820097f0 (LWP 100426) 0x0000000800d9d73c in read () from /lib/libc.so.7 118 Thread 0x2d83ee1400 (LWP 100427) 0x0000000800d9d73c in read () from /lib/libc.so.7 117 Thread 0x2d83ee4aa0 (LWP 100428) 0x0000000800d9d73c in read () from /lib/libc.so.7 116 Thread 0x801b14790 (LWP 100429) 0x0000000800d9d73c in read () from /lib/libc.so.7 115 Thread 0x2d8619dca0 (LWP 100435) 0x0000000800d9d73c in read () from /lib/libc.so.7 114 Thread 0x2d82008820 (LWP 100436) 0x0000000800d9d73c in read () from /lib/libc.so.7 113 Thread 0x2d83ee6bb0 (LWP 100437) 0x0000000800d9d73c in read () from /lib/libc.so.7 112 Thread 0x2d83ee1ca0 (LWP 100439) 0x0000000800d9d73c in read () from /lib/libc.so.7 111 Thread 0x2d8619db30 (LWP 100443) 0x0000000800d9d73c in read () from /lib/libc.so.7 110 Thread 0x801b123a0 (LWP 100444) 0x0000000800d9d73c in read () from /lib/libc.so.7 109 Thread 0x801b106e0 (LWP 100445) 0x0000000800d9d73c in read () from /lib/libc.so.7 108 Thread 0x2d83ee1850 (LWP 100447) 0x0000000800d9d73c in read () from /lib/libc.so.7 107 Thread 0x2d8619ede0 (LWP 100448) 0x0000000800d9d73c in read () from /lib/libc.so.7 106 Thread 0x2d83ee1570 (LWP 100450) 0x0000000800d9d73c in read () from /lib/libc.so.7 105 Thread 0x2d83ee5a70 (LWP 100451) 0x0000000800d9d73c in read () from /lib/libc.so.7 104 Thread 0x2d83ee1b30 (LWP 100452) 0x0000000800d9d73c in read () from /lib/libc.so.7 103 Thread 0x2d83ee61a0 (LWP 100453) 0x0000000800d9d73c in read () from /lib/libc.so.7 102 Thread 0x2d8619eb00 (LWP 100455) 0x0000000800d9d73c in read () from /lib/libc.so.7 101 Thread 0x801b12c40 (LWP 100456) 0x0000000800d9d73c in read () from /lib/libc.so.7 100 Thread 0x2d8200cd20 (LWP 100457) 0x0000000800d9d73c in read () from /lib/libc.so.7 99 Thread 0x2d8200ba70 (LWP 100458) 0x0000000800d9d73c in read () from /lib/libc.so.7 98 Thread 0x2d82009db0 (LWP 100460) 0x0000000800d9d73c in read () from /lib/libc.so.7 97 Thread 0x801b14900 (LWP 100461) 0x0000000800d9d73c in read () from /lib/libc.so.7 96 Thread 0x2d83ee47c0 (LWP 100462) 0x0000000800d8800c in writev () from /lib/libc.so.7 95 Thread 0x2d8200bec0 (LWP 100463) 0x0000000800d9d73c in read () from /lib/libc.so.7 94 Thread 0x801b10b30 (LWP 100464) 0x0000000800d8800c in writev () from /lib/libc.so.7 93 Thread 0x801b134e0 (LWP 100465) 0x0000000800d9d73c in read () from /lib/libc.so.7 92 Thread 0x801b13d80 (LWP 100466) 0x0000000800d9d73c in read () from /lib/libc.so.7 91 Thread 0x2d8200a4e0 (LWP 100467) 0x0000000800d9d73c in read () from /lib/libc.so.7 90 Thread 0x2d82008540 (LWP 100468) 0x0000000800d9d73c in read () from /lib/libc.so.7 89 Thread 0x2d8200c1a0 (LWP 100470) 0x0000000800d9d73c in read () from /lib/libc.so.7 88 Thread 0x801b13650 (LWP 100471) 0x0000000800d8800c in writev () from /lib/libc.so.7 87 Thread 0x801b10f80 (LWP 100472) 0x0000000800d9d73c in read () from /lib/libc.so.7 86 Thread 0x2d82009510 (LWP 100475) 0x0000000800d9d73c in read () from /lib/libc.so.7 85 Thread 0x2d8619d850 (LWP 100476) 0x0000000800d9d73c in read () from /lib/libc.so.7 84 Thread 0x801b10120 (LWP 100478) 0x0000000800d9d73c in read () from /lib/libc.so.7 83 Thread 0x2d83ee5620 (LWP 100479) 0x0000000800d9d73c in read () from /lib/libc.so.7 82 Thread 0x801b11260 (LWP 100480) 0x0000000800d9d73c in read () from /lib/libc.so.7 81 Thread 0x801b11b00 (LWP 100481) 0x0000000800d9d73c in read () from /lib/libc.so.7 80 Thread 0x801b15030 (LWP 100482) 0x0000000800d9d73c in read () from /lib/libc.so.7 79 Thread 0x2d83ee4200 (LWP 100483) 0x0000000800d9d73c in read () from /lib/libc.so.7 78 Thread 0x2d8619e990 (LWP 100484) 0x0000000800d9d73c in read () from /lib/libc.so.7 77 Thread 0x2d83ee4650 (LWP 100485) 0x0000000800d9d73c in read () from /lib/libc.so.7 76 Thread 0x2d83ee6a40 (LWP 100486) 0x0000000800d9d73c in read () from /lib/libc.so.7 75 Thread 0x801b11990 (LWP 100488) 0x0000000800d9d73c in read () from /lib/libc.so.7 74 Thread 0x2d83ee4930 (LWP 100489) 0x0000000800d9d73c in read () from /lib/libc.so.7 73 Thread 0x801b14d50 (LWP 100491) 0x0000000800d9d73c in read () from /lib/libc.so.7 72 Thread 0x2d8619e0f0 (LWP 100492) 0x0000000800d9d73c in read () from /lib/libc.so.7 71 Thread 0x2d8619ef50 (LWP 100494) 0x0000000800d8364c in connect () from /lib/libc.so.7 70 Thread 0x801b12230 (LWP 100496) 0x0000000800d9d73c in read () from /lib/libc.so.7 69 Thread 0x2d83ee2f50 (LWP 100497) 0x0000000800d9d73c in read () from /lib/libc.so.7 68 Thread 0x2d8619e260 (LWP 100499) 0x0000000800d9d73c in read () from /lib/libc.so.7 67 Thread 0x2d8200c030 (LWP 100500) 0x0000000800d9d73c in read () from /lib/libc.so.7 66 Thread 0x801b127f0 (LWP 100501) 0x0000000800d9d73c in read () from /lib/libc.so.7 65 Thread 0x2d83ee6480 (LWP 100503) 0x0000000800d9d73c in read () from /lib/libc.so.7 64 Thread 0x2d82007400 (LWP 100504) 0x0000000800d9d73c in read () from /lib/libc.so.7 63 Thread 0x2d82007f80 (LWP 100505) 0x0000000800d9d73c in read () from /lib/libc.so.7 62 Thread 0x2d82008260 (LWP 100507) 0x0000000800d9d73c in read () from /lib/libc.so.7 61 Thread 0x2d82009230 (LWP 100508) 0x0000000800d9d73c in read () from /lib/libc.so.7 60 Thread 0x2d820093a0 (LWP 100509) 0x0000000800d9d73c in read () from /lib/libc.so.7 59 Thread 0x2d8200a200 (LWP 100512) 0x0000000800d9d73c in read () from /lib/libc.so.7 58 Thread 0x2d8200a7c0 (LWP 100514) 0x0000000800d9d73c in read () from /lib/libc.so.7 57 Thread 0x2d8200a930 (LWP 100516) 0x0000000800d9d73c in read () from /lib/libc.so.7 56 Thread 0x2d8200aef0 (LWP 100517) 0x0000000800d9d73c in read () from /lib/libc.so.7 55 Thread 0x2d8200b060 (LWP 100518) 0x0000000800d9d73c in read () from /lib/libc.so.7 54 Thread 0x2d8200b4b0 (LWP 100519) 0x0000000800d9d73c in read () from /lib/libc.so.7 53 Thread 0x2d8200bbe0 (LWP 100520) 0x0000000800d9d73c in read () from /lib/libc.so.7 52 Thread 0x2d8200bd50 (LWP 100521) 0x0000000800d9d73c in read () from /lib/libc.so.7 51 Thread 0x2d8200ca40 (LWP 100522) 0x0000000800d9d73c in read () from /lib/libc.so.7 50 Thread 0x2d8200ce90 (LWP 100523) 0x0000000800d9d73c in read () from /lib/libc.so.7 49 Thread 0x2d83ee16e0 (LWP 100524) 0x0000000800d9d73c in read () from /lib/libc.so.7 48 Thread 0x2d83ee19c0 (LWP 100525) 0x0000000800d9d73c in read () from /lib/libc.so.7 47 Thread 0x2d83ee1f80 (LWP 100527) 0x0000000800d9d73c in read () from /lib/libc.so.7 46 Thread 0x2d83ee26b0 (LWP 100528) 0x0000000800d9d73c in read () from /lib/libc.so.7 45 Thread 0x2d83ee2820 (LWP 100529) 0x0000000800d9d73c in read () from /lib/libc.so.7 44 Thread 0x2d83ee3510 (LWP 100530) 0x0000000800d9d73c in read () from /lib/libc.so.7 43 Thread 0x2d83ee3680 (LWP 100531) 0x0000000800d9d73c in read () from /lib/libc.so.7 42 Thread 0x2d83ee3960 (LWP 100532) 0x0000000800d9d73c in read () from /lib/libc.so.7 41 Thread 0x2d83ee3c40 (LWP 100533) 0x0000000800d9d73c in read () from /lib/libc.so.7 40 Thread 0x2d83ee4090 (LWP 100534) 0x0000000800d9d73c in read () from /lib/libc.so.7 39 Thread 0x801b10290 (LWP 100535) 0x0000000800d9d73c in read () from /lib/libc.so.7 38 Thread 0x801b10400 (LWP 100536) 0x0000000800d9d73c in read () from /lib/libc.so.7 37 Thread 0x801b10570 (LWP 100537) 0x0000000800d9d73c in read () from /lib/libc.so.7 36 Thread 0x801b10ca0 (LWP 100538) 0x0000000800d9d73c in read () from /lib/libc.so.7 35 Thread 0x801b11540 (LWP 100539) 0x0000000800d9d73c in read () from /lib/libc.so.7 34 Thread 0x801b116b0 (LWP 100540) 0x0000000800d9d73c in read () from /lib/libc.so.7 33 Thread 0x801b11820 (LWP 100541) 0x0000000800d9d73c in read () from /lib/libc.so.7 32 Thread 0x801b11c70 (LWP 100543) 0x0000000800d9d73c in read () from /lib/libc.so.7 31 Thread 0x801b11f50 (LWP 100544) 0x0000000800d9d73c in read () from /lib/libc.so.7 30 Thread 0x801b13090 (LWP 100545) 0x0000000800d9d73c in read () from /lib/libc.so.7 29 Thread 0x801b13370 (LWP 100546) 0x0000000800d9d73c in read () from /lib/libc.so.7 28 Thread 0x801b137c0 (LWP 100547) 0x0000000800d9d73c in read () from /lib/libc.so.7 27 Thread 0x801b13aa0 (LWP 100548) 0x0000000800d9d73c in read () from /lib/libc.so.7 26 Thread 0x801b14620 (LWP 100549) 0x0000000800d9d73c in read () from /lib/libc.so.7 25 Thread 0x801b14a70 (LWP 100550) 0x0000000800d9d73c in read () from /lib/libc.so.7 24 Thread 0x801b151a0 (LWP 100551) 0x0000000800d9d73c in read () from /lib/libc.so.7 23 Thread 0x801b15310 (LWP 100552) 0x0000000800d9d73c in read () from /lib/libc.so.7 22 Thread 0x801b15480 (LWP 100553) 0x0000000800d9d73c in read () from /lib/libc.so.7 21 Thread 0x801b155f0 (LWP 100554) 0x0000000800d9d73c in read () from /lib/libc.so.7 20 Thread 0x801b15760 (LWP 100555) 0x0000000800d9d73c in read () from /lib/libc.so.7 19 Thread 0x801b158d0 (LWP 100556) 0x0000000800d9d73c in read () from /lib/libc.so.7 18 Thread 0x801b15a40 (LWP 100557) 0x0000000800d9d73c in read () from /lib/libc.so.7 17 Thread 0x801b15bb0 (LWP 100558) 0x0000000800d9d73c in read () from /lib/libc.so.7 16 Thread 0x801b15d20 (LWP 100559) 0x0000000800d9d73c in read () from /lib/libc.so.7 15 Thread 0x801b15e90 (LWP 100560) 0x0000000800d9d73c in read () from /lib/libc.so.7 14 Thread 0x2d8edb8120 (LWP 100561) 0x0000000800d9d73c in read () from /lib/libc.so.7 13 Thread 0x2d8edb8290 (LWP 100562) 0x0000000800d9d73c in read () from /lib/libc.so.7 12 Thread 0x2d8edb8400 (LWP 100563) 0x0000000800d9d73c in read () from /lib/libc.so.7 11 Thread 0x2d8edb8570 (LWP 100564) 0x0000000800d9d73c in read () from /lib/libc.so.7 10 Thread 0x2d8edb86e0 (LWP 100565) 0x0000000800d9d73c in read () from /lib/libc.so.7 9 Thread 0x2d8edb8850 (LWP 100566) 0x0000000800d9d73c in read () from /lib/libc.so.7 8 Thread 0x2d8edb89c0 (LWP 100567) 0x0000000800d9d73c in read () from /lib/libc.so.7 7 Thread 0x2d8edb8b30 (LWP 100568) 0x0000000800d9d73c in read () from /lib/libc.so.7 6 Thread 0x2d8edb8ca0 (LWP 100569) 0x0000000800d9d73c in read () from /lib/libc.so.7 5 Thread 0x2d8edb8e10 (LWP 100570) 0x0000000800d9d73c in read () from /lib/libc.so.7 4 Thread 0x2d8edb8f80 (LWP 100571) 0x0000000800d9d73c in read () from /lib/libc.so.7 3 Thread 0x2d8edb90f0 (LWP 100572) 0x0000000800d9d73c in read () from /lib/libc.so.7 2 Thread 0x2d8edb9260 (LWP 100573) 0x0000000800d9d73c in read () from /lib/libc.so.7 1 Thread 0x2d8edb93d0 (LWP 100574) 0x0000000800d9d73c in read () from /lib/libc.so.7 (gdb) }}} OS is FreeBSD/amd 7.0-BETA1 on a SMP, two-CPU system. My VCL: {{{ backend default { set backend.host = "192.168.0.3"; set backend.port = "80"; } acl purge { "192.168.0.4"/32; } sub vcl_recv { if ((req.http.host ~ "^(cache.finn.no|finn.no|www.finn.no)$") || (req.http.host == "banner.finn.no" && req.url ~ "^/(jsp2|finn/gojsp|daily|board|auximg/papirfly|finn/cacheable|crossdomain.xml)")) { if (req.request == "GET" || req.request == "HEAD") { lookup; } elsif (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } lookup; } else { pipe; } } else { error 403 "Access denied. Contact cacheadmin at finn.no if you have problems."; } } sub vcl_miss { if (req.request == "PURGE") { error 404 "Not in cache."; } else { fetch; } } sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } else { deliver; } } sub vcl_fetch { if (obj.ttl < 86400s) { set obj.ttl = 604800s; } if (obj.http.Cookie) { remove obj.http.Cookie; } if (obj.http.Set-Cookie) { remove obj.http.Set-Cookie; } insert; } }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Sat Oct 20 21:47:11 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Sat, 20 Oct 2007 21:47:11 -0000 Subject: [Varnish] #170: Segmentation violation in HTC_Read at cache_httpconn.c:195 In-Reply-To: <052.544b68a3ff13ba083233aa7d256ffb88@projects.linpro.no> References: <052.544b68a3ff13ba083233aa7d256ffb88@projects.linpro.no> Message-ID: <061.df1bf00b8d4c5de15685f57f5464af61@projects.linpro.no> #170: Segmentation violation in HTC_Read at cache_httpconn.c:195 --------------------------------+------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Changes (by anders): * owner: des => phk * component: build => varnishd -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Sun Oct 21 07:38:30 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Sun, 21 Oct 2007 07:38:30 -0000 Subject: [Varnish] #167: Varnishd crashes with assert error in RES_WriteObj In-Reply-To: <052.0d60ec784fa5d17383428842e133f8e7@projects.linpro.no> References: <052.0d60ec784fa5d17383428842e133f8e7@projects.linpro.no> Message-ID: <061.455f9bcea73b8c5d9f7c46e4ed8ca6c0@projects.linpro.no> #167: Varnishd crashes with assert error in RES_WriteObj ---------------------------------------------+------------------------------ Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump RES_WriteObj | ---------------------------------------------+------------------------------ Comment (by anders): Happened again, now with branches/1.2 (up to date to commit 2153 including patches for ticket 168 (changeset r2153) and ticket 164 (changeset 2151)): {{{ Child said (2, 35335): <obj->len) not true. errno = 14 (Bad address) }}} Backtrace: {{{ Program received signal SIGABRT, Aborted. [Switching to Thread 0x2d830026b0 (LWP 100383)] 0x0000000800d1048c in thr_kill () from /lib/libc.so.7 (gdb) bt #0 0x0000000800d1048c in thr_kill () from /lib/libc.so.7 #1 0x0000000800d9a63b in abort () from /lib/libc.so.7 #2 0x000000080066dc7f in lbv_assert (func=Could not find the frame base for "lbv_assert". ) at assert.c:58 #3 0x0000000000419e99 in RES_WriteObj (sp=0x2d84b69008) at cache_response.c:184 #4 0x000000000040e606 in cnt_deliver (sp=0x2d84b69008) at cache_center.c:162 #5 0x000000000041005a in CNT_Session (sp=0x2d84b69008) at steps.h:41 #6 0x0000000000417ecd in wrk_do_one (w=0x7fffe4121ae0) at cache_pool.c:193 #7 0x00000000004183c6 in wrk_thread (priv=0x800f111e0) at cache_pool.c:246 #8 0x0000000800a979a8 in pthread_getprio () from /lib/libthr.so.3 #9 0x00007fffe3f22000 in ?? () Error accessing memory address 0x7fffe4122000: Bad address. (gdb) frame 3 #3 0x0000000000419e99 in RES_WriteObj (sp=0x2d84b69008) at cache_response.c:184 184 assert(u == sp->obj->len); (gdb) print u $1 = 0 (gdb) print sp->obj->len $2 = 31114 }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Sun Oct 21 14:47:08 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Sun, 21 Oct 2007 14:47:08 -0000 Subject: [Varnish] #170: Segmentation violation in HTC_Read at cache_httpconn.c:195 In-Reply-To: <052.544b68a3ff13ba083233aa7d256ffb88@projects.linpro.no> References: <052.544b68a3ff13ba083233aa7d256ffb88@projects.linpro.no> Message-ID: <061.a65a45207a2d4d2c54760874860ab17d@projects.linpro.no> #170: Segmentation violation in HTC_Read at cache_httpconn.c:195 --------------------------------+------------------------------------------- Reporter: anders | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd core dump | --------------------------------+------------------------------------------- Comment (by anders): This happens also with trunk/2096 including patches for ticket 168 (changeset r2153) and ticket 164 (changeset 2151): {{{ Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x2d83dcdca0 (LWP 100612)] 0x0000000800c97ff6 in memcpy () from /lib/libc.so.7 (gdb) bt #0 0x0000000800c97ff6 in memcpy () from /lib/libc.so.7 #1 0x00000000004173c3 in HTC_Read (htc=0x7fffd389b990, d=0x898036000, len=1904) at cache_httpconn.c:182 #2 0x000000000041169c in fetch_straight (sp=0x2d8415b008, htc=0x7fffd389b990, b=0x2d8341d3aa "1904") at cache_fetch.c:72 #3 0x0000000000412380 in Fetch (sp=0x2d8415b008) at cache_fetch.c:331 #4 0x000000000040ed02 in cnt_fetch (sp=0x2d8415b008) at cache_center.c:301 #5 0x000000000041006c in CNT_Session (sp=0x2d8415b008) at steps.h:40 #6 0x0000000000417d7d in wrk_do_one (w=0x7fffd389dae0) at cache_pool.c:193 #7 0x0000000000418227 in wrk_thread (priv=0x800e11200) at cache_pool.c:245 #8 0x00000008009929a8 in pthread_getprio () from /lib/libthr.so.3 #9 0x00007fffd369e000 in ?? () Error accessing memory address 0x7fffd389e000: Bad address. (gdb) frame 1 #1 0x00000000004173c3 in HTC_Read (htc=0x7fffd389b990, d=0x898036000, len=1904) at cache_httpconn.c:182 182 memcpy(p, htc->pipeline.b, l); (gdb) print htc $1 = (struct http_conn *) 0x7fffd389b990 (gdb) print *htc $2 = {magic = 1041886673, fd = 22, ws = 0x2d8341d018, rxbuf = { b = 0x2d8341d280 "HTTP/1.1", e = 0x2d8341d3de "\n\n\n\n\n\n\n