From jesus at omniti.com Mon Aug 11 01:22:06 2008 From: jesus at omniti.com (Theo Schlossnagle) Date: Sun, 10 Aug 2008 21:22:06 -0400 Subject: Fresh patch for Solaris Message-ID: <57710936-1C6F-457E-AA6D-6BB40DB2CF06@omniti.com> http://lethargy.org/~jesus/misc/varnish-solaris-trunk-3071.diff This is cleaned up a bit. Nothing really new, just updated to trunk 3071. Some nice things worth cherry picking (IMHO) err/errx in libvarnishcompat (from NetBSD) replacement of the ifdef'd compile stuff with a define set in configure umem allocator -- should scale much better than malloc Anyway, anyone running Varnish on Solaris, please rev your engines and give this a spin. autogen.sh CC=cc CFLAGS="-mt -m64" ./configure make -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/ From phk at phk.freebsd.dk Mon Aug 11 07:32:49 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 11 Aug 2008 07:32:49 +0000 Subject: Fresh patch for Solaris In-Reply-To: Your message of "Sun, 10 Aug 2008 21:22:06 -0400." <57710936-1C6F-457E-AA6D-6BB40DB2CF06@omniti.com> Message-ID: <32996.1218439969@critter.freebsd.dk> In message <57710936-1C6F-457E-AA6D-6BB40DB2CF06 at omniti.com>, Theo Schlossnagle writes: >http://lethargy.org/~jesus/misc/varnish-solaris-trunk-3071.diff > >This is cleaned up a bit. Nothing really new, just updated to trunk >3071. I'll take a peek at it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From tfheen at linpro.no Mon Aug 11 07:39:36 2008 From: tfheen at linpro.no (Tollef Fog Heen) Date: Mon, 11 Aug 2008 09:39:36 +0200 Subject: Fresh patch for Solaris In-Reply-To: <57710936-1C6F-457E-AA6D-6BB40DB2CF06@omniti.com> (Theo Schlossnagle's message of "Sun, 10 Aug 2008 21:22:06 -0400") References: <57710936-1C6F-457E-AA6D-6BB40DB2CF06@omniti.com> Message-ID: <87k5eovw3r.fsf@linpro.no> ]] Theo Schlossnagle | http://lethargy.org/~jesus/misc/varnish-solaris-trunk-3071.diff | | This is cleaned up a bit. Nothing really new, just updated to trunk | 3071. This seems to try to patch bin/varnishd/storage_umem.c which isn't in the tree, causing it to fail to apply. Could you please update the patch? -- Tollef Fog Heen / Linpro AS t: 21 54 41 73 UNIX is user friendly, it's just picky about who its friends are From phk at phk.freebsd.dk Mon Aug 11 07:42:21 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 11 Aug 2008 07:42:21 +0000 Subject: Fresh patch for Solaris In-Reply-To: Your message of "Mon, 11 Aug 2008 09:39:36 +0200." <87k5eovw3r.fsf@linpro.no> Message-ID: <41889.1218440541@critter.freebsd.dk> In message <87k5eovw3r.fsf at linpro.no>, Tollef Fog Heen writes: >]] Theo Schlossnagle > >| http://lethargy.org/~jesus/misc/varnish-solaris-trunk-3071.diff >| >| This is cleaned up a bit. Nothing really new, just updated to trunk >| 3071. > >This seems to try to patch bin/varnishd/storage_umem.c which isn't in >the tree, causing it to fail to apply. > >Could you please update the patch? Wait a few hours, I'm cherry picking some of the low-hanging fruit -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Aug 11 09:30:39 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 11 Aug 2008 09:30:39 +0000 Subject: Fresh patch for Solaris In-Reply-To: Your message of "Sun, 10 Aug 2008 21:22:06 -0400." <57710936-1C6F-457E-AA6D-6BB40DB2CF06@omniti.com> Message-ID: <1251.1218447039@critter.freebsd.dk> In message <57710936-1C6F-457E-AA6D-6BB40DB2CF06 at omniti.com>, Theo Schlossnagle writes: >http://lethargy.org/~jesus/misc/varnish-solaris-trunk-3071.diff OK, I've picked the obvious stuff out. Various questions about the rest: Doesn't Solaris have fcntl(F_SETLK) as mandated by POSIX ? flock() is not a standardized API, and I havn't seen a system yet which supports it, which doesn't also have fcntl(F_SETLK), so I would rather not mess up the source with a pointless check. Do you know for sure that sendfile on Solaris has no reference to the relevant parts of the file when it returns ? Otherwise it is not safe to use. Why is the extra include of necessary in storage_file.c ? Is there a reason to name the shared object .so instead of .o or is it just cosmetics ? In cache_acceptor.c, please implement the "->pass" function which does the port_send() call. Why the initialization in mgt_child.c ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jesus at omniti.com Mon Aug 11 13:37:39 2008 From: jesus at omniti.com (Theo Schlossnagle) Date: Mon, 11 Aug 2008 09:37:39 -0400 Subject: Fresh patch for Solaris In-Reply-To: <1251.1218447039@critter.freebsd.dk> References: <1251.1218447039@critter.freebsd.dk> Message-ID: On Aug 11, 2008, at 5:30 AM, Poul-Henning Kamp wrote: > In message <57710936-1C6F-457E-AA6D-6BB40DB2CF06 at omniti.com>, Theo > Schlossnagle > writes: > >> http://lethargy.org/~jesus/misc/varnish-solaris-trunk-3071.diff > > OK, I've picked the obvious stuff out. > > Various questions about the rest: > > Doesn't Solaris have fcntl(F_SETLK) as mandated by POSIX ? > flock() is not a standardized API, and I havn't seen a system yet > which supports it, which doesn't also have fcntl(F_SETLK), so I > would rather not mess up the source with a pointless check. Ah, good catch. That wasn't a patch for Solaris. It was for Linux. We've run into some issues with flock being more reliable than fcntl on Linux under high concurrency environments. We ran into this problem on another project and thought to make a preemptive strike in Varnish. However, my patch prefers flock over fcntl, which is clearly wrong as it might do that on BSD*. flock() on Solaris is only available with - lucb (the meager BSD-ish compatibility layer). So, with that patch, Solaris still uses fcntl. (Go POSIX!) The ugly way we did it on the other project (FastXSL) that had this issue is here: http://labs.omniti.com/trac/fastxsl/changeset/43 > > Do you know for sure that sendfile on Solaris has no reference to the > relevant parts of the file when it returns ? Otherwise it is not safe > to use. I asked the internal engineering team at Sun, they said that it has no references. I asked move forcefully and they did a code review (albeit very short) and said again that it had no references. So, at this point, I believe that it is safe to use in Varnish. > > Why is the extra include of necessary in > storage_file.c ? My mistake. I wasn't thorough when I merged up to trunk. No conflict and my lack of attention. Ignore that. > > Is there a reason to name the shared object .so instead of .o or is it > just cosmetics ? The Sun Studio toolchain barfs on the .o. It "knows better" and seems to ignore the request to make it shared. I could get around this with a shell scripts (as you had once suggested), but this change makes it work in all the toolchains I have tried "out of the box" > > In cache_acceptor.c, please implement the "->pass" function which does > the port_send() call. Will do. Not sure how that got reverted actually. > Why the initialization in mgt_child.c ? I believe that is left over from something else. A valgrind warning -- but that could only have been due to other code I added and then removed. It looks clean and safe without initialization. -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/ From jesus at omniti.com Mon Aug 11 14:28:06 2008 From: jesus at omniti.com (Theo Schlossnagle) Date: Mon, 11 Aug 2008 10:28:06 -0400 Subject: Fresh patch for Solaris In-Reply-To: <1251.1218447039@critter.freebsd.dk> References: <1251.1218447039@critter.freebsd.dk> Message-ID: On Aug 11, 2008, at 5:30 AM, Poul-Henning Kamp wrote: > In message <57710936-1C6F-457E-AA6D-6BB40DB2CF06 at omniti.com>, Theo > Schlossnagle > writes: > >> http://lethargy.org/~jesus/misc/varnish-solaris-trunk-3071.diff > > OK, I've picked the obvious stuff out. Next pass: http://lethargy.org/~jesus/misc/varnish-solaris-trunk-3080.diff cleaned up some redundant includes removed the flock stuff (not Solaris related) made the cache_acceptor_ports use ->pass removed unnecessary things (includes, defines and initializations... removed from my patch only) made the umem allocator a "new file" instead of a modified svn copy of the malloc allocator. Patch is about half the size of the previous one. AWESOME progress. This is great. Thanks for your attention phk! -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/ From phk at phk.freebsd.dk Mon Aug 11 14:47:17 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 11 Aug 2008 14:47:17 +0000 Subject: Fresh patch for Solaris In-Reply-To: Your message of "Mon, 11 Aug 2008 10:28:06 -0400." Message-ID: <9684.1218466037@critter.freebsd.dk> In message , Theo Schlossnagle writes: >Next pass: > >http://lethargy.org/~jesus/misc/varnish-solaris-trunk-3080.diff You still have the err.h compat stuff, that's not necessary any more, I removed the two uses of err(). Why is the tcp.c patch necessay ? You have not answered my questions about .so and sendfile ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Mon Aug 11 14:51:50 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 11 Aug 2008 14:51:50 +0000 Subject: Fresh patch for Solaris In-Reply-To: Your message of "Mon, 11 Aug 2008 09:37:39 -0400." Message-ID: <9708.1218466310@critter.freebsd.dk> Sorry for the comment at the end of my prevous email, got your emails in reverse order here :-) >> Doesn't Solaris have fcntl(F_SETLK) as mandated by POSIX ? >Ah, good catch. That wasn't a patch for Solaris. It was for Linux. >We've run into some issues with flock being more reliable than fcntl >on Linux under high concurrency environments. We're not even remotely close to concurrency, so I'd prefer to stick with fcntl until we see any actual problems. >> Is there a reason to name the shared object .so instead of .o or is it >> just cosmetics ? > >The Sun Studio toolchain barfs on the .o. It "knows better" [...] Ok, fair enough. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jesus at omniti.com Mon Aug 11 15:11:03 2008 From: jesus at omniti.com (Theo Schlossnagle) Date: Mon, 11 Aug 2008 11:11:03 -0400 Subject: Fresh patch for Solaris In-Reply-To: <9684.1218466037@critter.freebsd.dk> References: <9684.1218466037@critter.freebsd.dk> Message-ID: <853AF395-BB67-478B-AC34-C685E35D0910@omniti.com> Next pass: http://lethargy.org/~jesus/misc/varnish-solaris-trunk-3080-2.diff On Aug 11, 2008, at 10:47 AM, Poul-Henning Kamp wrote: > You still have the err.h compat stuff, that's not necessary any more, > I removed the two uses of err(). Didn't catch that. Fixed. > Why is the tcp.c patch necessay ? FIONBIO isn't available by default as it is a BSD thing. You're "supposed" to use fcntl on Solaris. fcntl is slower and ioctl _will_ work, but the define of FIONBIO is in sys/filio.h. You get it if you turn on BSD compatibility when including ioctl.h, but we don't want the other stuff that comes with that. So far, that single line of code is the only one requiring BSD_COMP on Solaris side, so I'm hesitant to -DBSD_COMP the whole thing. > You have not answered my questions about .so and sendfile ? .o -> .so is not purely cosmetic. There is an issue with the Sun Studio toolchain that makes .o not work. sendfile on Solaris should be safe. When the call returns no bits should be referenced at all. -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/ From jesus at omniti.com Wed Aug 13 19:19:51 2008 From: jesus at omniti.com (Theo Schlossnagle) Date: Wed, 13 Aug 2008 15:19:51 -0400 Subject: waitpid EINTR issue in varnishd Message-ID: <72AB9D1A-F485-4DF0-A445-4C375F25075B@omniti.com> On Solaris, I am occasionally seeing a sigchild due to the completion of the vcl compile that fires during the waitpid() call causing waitpid to -1 errno EINTR. Here is a patch that addresses that condition. -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: vcc_waitpid.patch Type: application/octet-stream Size: 660 bytes Desc: not available URL: From jesus at omniti.com Wed Aug 13 19:26:07 2008 From: jesus at omniti.com (Theo Schlossnagle) Date: Wed, 13 Aug 2008 15:26:07 -0400 Subject: Two varnishtest failures on Solaris Message-ID: I am getting two failures in the test suite using varnishtest on Solaris. 8091 with the Solaris patch applied: http://lethargy.org/~jesus/misc/varnish-solaris-trunk-3091.diff Two failing tests are c00015 and v00006 ; ./varnishtest -v tests/c00015.vtc # TEST tests/c00015.vtc starting # TEST Test switching VCLs ## s1 Starting server ### s1 listen on 127.0.0.1:9080 (fd 3) ## v1 Launch ### v1 CMD: cd ../varnishd && ./varnishd -d -d -n /tmp/__v1 -a '127.0.0.1:9081' -T 127.0.0.1:9001 ## s1 Started on 127.0.0.1:9080 ### v1 opening CLI connection #### v1 debug| storage_file: filename: ./varnish.egaGks (unlinked) size 100 MB.\n #### v1 debug| Using old SHMFILE\n #### v1 debug| Notice: locking SHMFILE in core failed: Not owner #### v1 debug| \n #### v1 debug| Debugging mode, enter "start" to start child\n ### v1 CLI connection fd = 7 #### v1 CLI TX| vcl.inline vcl1 "backend s1 { .host = \"127.0.0.1\"; .port = \"9080\"; }\n \n" #### v1 CLI RX| VCL compiled. ### v1 CLI STATUS 200 #### v1 CLI TX| vcl.use vcl1 ### v1 CLI STATUS 200 ## v1 Start #### v1 CLI TX| start #### v1 debug| child ( #### v1 debug| 9319) Started\n ### v1 CLI STATUS 200 #### v1 debug| Child (9319) said Closed fds: 3 5 7 8 11 12 14 15\n #### v1 CLI TX| debug.xid 1000 #### v1 debug| Child (9319) said Child starts\n #### v1 debug| Child (9319) said managed to mmap 105201664 bytes of 105201664\n #### v1 debug| Child (9319) said Ready\n #### v1 CLI RX| XID is 1000\n ### v1 CLI STATUS 200 #### v1 CLI TX| vcl.inline vcl2 "backend s1 { .host = \"127.0.0.1\"; .port = \"9080\"; }\n\n\tsub vcl_recv {\n\t\tpass;\n\t} \n" #### v1 CLI RX| VCL compiled. ### v1 CLI STATUS 200 #### v1 CLI TX| vcl.use vcl2 ### v1 CLI STATUS 200 #### v1 CLI TX| vcl.list #### v1 CLI RX| available 0 vcl1\n #### v1 CLI RX| active 0 vcl2\n ### v1 CLI STATUS 200 ## v1 CLI 200 #### v1 CLI TX| vcl.use vcl1 ### v1 CLI STATUS 200 ## v1 CLI 200 ## c1 Starting client #### v1 CLI TX| vcl.use vcl2 ## c1 Started ### c1 Connect to 127.0.0.1:9081 ### v1 CLI STATUS 200 ## v1 CLI 200 ## c2 Starting client #### v1 CLI TX| vcl.use vcl1 ## c2 Started ### c2 Connect to 127.0.0.1:9081 ### c1 Connected to 127.0.0.1:9081 fd is 12 #### c1 txreq| GET /foo HTTP/1.1\r\n #### c1 txreq| \r\n ### c1 rxresp ### v1 CLI STATUS 200 ## v1 CLI 200 ## c3 Starting client #### v1 CLI TX| vcl.show vcl2 ### c2 Connected to 127.0.0.1:9081 fd is 13 #### c2 txreq| GET /foo HTTP/1.1\r\n #### c2 txreq| \r\n ### c2 rxresp ## c3 Started ### c3 Connect to 127.0.0.1:9081 ### c3 Connected to 127.0.0.1:9081 fd is 14 #### c3 txreq| GET /foo HTTP/1.1\r\n #### c3 txreq| \r\n ### c3 rxresp #### v1 CLI RX| backend s1 { .host = "127.0.0.1"; .port = "9080"; }\n #### v1 CLI RX| \n #### v1 CLI RX| \tsub vcl_recv {\n #### v1 CLI RX| \t\tpass;\n #### v1 CLI RX| \t}\n ### v1 CLI STATUS 200 ## v1 CLI 200 #### v1 CLI TX| vcl.discard vcl2 #### v1 debug| unlink ./vcl.hndxcCNb.so\n ### v1 CLI STATUS 200 ### s1 Accepted socket fd is 11 ## v1 CLI 200 #### v1 CLI TX| vcl.list ### s1 rxreq #### v1 CLI RX| active 1 vcl1\n ### v1 CLI STATUS 200 ## v1 CLI 200 # RESETTING after tests/c00015.vtc ## s1 Waiting for server #### s1 rxhdr| GET /foo HTTP/1.1\r\n #### s1 rxhdr| X-Varnish: 1001\r\n #### s1 rxhdr| X-Forwarded-For: 127.0.0.1\r\n #### s1 rxhdr| Host: 127.0.0.1\r\n #### s1 rxhdr| \r\n #### s1 http[ 0] | GET #### s1 http[ 1] | /foo #### s1 http[ 2] | HTTP/1.1 #### s1 http[ 3] | X-Varnish: 1001 #### s1 http[ 4] | X-Forwarded-For: 127.0.0.1 #### s1 http[ 5] | Host: 127.0.0.1 #### s1 EXPECT req.url (/foo) == /foo (/foo) match #### s1 txresp| HTTP/1.1 200 Ok\r\n #### s1 txresp| Content-Length: 6\r\n #### s1 txresp| \r\n #### s1 txresp| foobar\r\n ### s1 rxreq #### c2 rxhdr| HTTP/1.1 200 Ok\r\n #### c2 rxhdr| Content-Length: 6\r\n #### c2 rxhdr| Date: Wed, 13 Aug 2008 19:21:13 GMT\r\n #### c2 rxhdr| X-Varnish: 1001\r\n #### c2 rxhdr| Age: 0\r\n #### c2 rxhdr| Via: 1.1 varnish\r\n #### c2 rxhdr| Connection: keep-alive\r\n #### c2 rxhdr| \r\n #### c2 http[ 0] | HTTP/1.1 #### c2 http[ 1] | 200 #### c2 http[ 2] | Ok #### c2 http[ 3] | Content-Length: 6 #### c2 http[ 4] | Date: Wed, 13 Aug 2008 19:21:13 GMT #### c2 http[ 5] | X-Varnish: 1001 #### c2 http[ 6] | Age: 0 #### c2 http[ 7] | Via: 1.1 varnish #### c2 http[ 8] | Connection: keep-alive #### c2 body| foobar #### c2 EXPECT resp.status (200) == 200 (200) match ---- c2 EXPECT resp.http.content-length (6) == 7 (7) failed ; ./varnishtest -v tests/v00006.vtc # TEST tests/v00006.vtc starting # TEST VCL: Test backend retirement ## s1 Starting server ### s1 listen on 127.0.0.1:9080 (fd 3) ## v1 Launch ### v1 CMD: cd ../varnishd && ./varnishd -d -d -n /tmp/__v1 -a '127.0.0.1:9081' -T 127.0.0.1:9001 -p thread_pools=1 -w1,1,300 ## s1 Started on 127.0.0.1:9080 ### v1 opening CLI connection #### v1 debug| storage_file: filename: ./varnish.2jaqEs (unlinked) size 100 MB.\n #### v1 debug| Using old SHMFILE\n #### v1 debug| Notice: locking SHMFILE in core failed: #### v1 debug| Not owner\n #### v1 debug| Debugging mode, enter "start" to start child\n ### v1 CLI connection fd = 7 #### v1 CLI TX| vcl.inline vcl1 "\n\tbackend b1 {\n\t\t.host = \"localhost\";\n\t\t.port = \"9080\";\n\t}\n" #### v1 CLI RX| VCL compiled. ### v1 CLI STATUS 200 #### v1 CLI TX| vcl.use vcl1 ### v1 CLI STATUS 200 ## v1 Start #### v1 CLI TX| start #### v1 debug| child ( #### v1 debug| 9477) Started\n #### v1 debug| Child (9477) said Closed fds: 3 5 7 8 11 12 14 15\n ### v1 CLI STATUS 200 #### v1 CLI TX| debug.xid 1000 #### v1 debug| Child (9477) said Child starts\n #### v1 debug| Child ( #### v1 debug| 9477 #### v1 debug| ) said managed to mmap 105172992 bytes of 105172992\n #### v1 debug| Child (9477) said #### v1 debug| Ready\n #### v1 CLI RX| XID is 1000\n ### v1 CLI STATUS 200 ## c1 Starting client ## s1 Waiting for server ## c1 Started ### c1 Connect to 127.0.0.1:9081 ### c1 Connected to 127.0.0.1:9081 fd is 12 #### c1 txreq| GET / HTTP/1.1\r\n #### c1 txreq| \r\n ### c1 rxresp ### s1 Accepted socket fd is 11 ### s1 rxreq #### s1 rxhdr| GET / HTTP/1.1\r\n #### s1 rxhdr| X-Varnish: 1001\r\n #### s1 rxhdr| X-Forwarded-For: 127.0.0.1\r\n #### s1 rxhdr| Host: localhost\r\n #### s1 rxhdr| \r\n #### s1 http[ 0] | GET #### s1 http[ 1] | / #### s1 http[ 2] | HTTP/1.1 #### s1 http[ 3] | X-Varnish: 1001 #### s1 http[ 4] | X-Forwarded-For: 127.0.0.1 #### s1 http[ 5] | Host: localhost #### s1 txresp| HTTP/1.1 200 Ok\r\n #### s1 txresp| \r\n ### s1 shutting fd 11 ## s1 Ending ## c1 Waiting for client #### c1 rxhdr| HTTP/1.1 200 Ok\r\n #### c1 rxhdr| Content-Length: 0\r\n #### c1 rxhdr| Date: Wed, 13 Aug 2008 19:25:40 GMT\r\n #### c1 rxhdr| X-Varnish: 1001\r\n #### c1 rxhdr| Age: 0\r\n #### c1 rxhdr| Via: 1.1 varnish\r\n #### c1 rxhdr| Connection: keep-alive\r\n #### c1 rxhdr| \r\n #### c1 http[ 0] | HTTP/1.1 #### c1 http[ 1] | 200 #### c1 http[ 2] | Ok #### c1 http[ 3] | Content-Length: 0 #### c1 http[ 4] | Date: Wed, 13 Aug 2008 19:25:40 GMT #### c1 http[ 5] | X-Varnish: 1001 #### c1 http[ 6] | Age: 0 #### c1 http[ 7] | Via: 1.1 varnish #### c1 http[ 8] | Connection: keep-alive ### c1 Closing fd 12 ## c1 Ending ## v1 as expected: n_backend (1) == 1 ## v1 as expected: n_vcl_avail (1) == 1 ## v1 as expected: n_vcl_discard (0) == 0 ## s2 Starting server ### s2 listen on 127.0.0.1:9180 (fd 3) ## s2 Started on 127.0.0.1:9180 #### v1 CLI TX| vcl.inline vcl2 "\n\tbackend b2 {\n\t\t.host = \"localhost\";\n\t\t.port = \"9180\";\n\t}\n" #### v1 CLI RX| VCL compiled. ### v1 CLI STATUS 200 #### v1 CLI TX| vcl.use vcl2 ### v1 CLI STATUS 200 ## v1 as expected: n_backend (2) == 2 ## v1 as expected: n_vcl_avail (2) == 2 ## v1 as expected: n_vcl_discard (0) == 0 #### v1 CLI TX| debug.backend #### v1 CLI RX| 4ff790 b1 1\n #### v1 CLI RX| 4ff850 b2 1\n ### v1 CLI STATUS 200 ## v1 CLI 200 #### v1 CLI TX| vcl.list #### v1 CLI RX| available 1 vcl1\n #### v1 CLI RX| active 0 vcl2\n ### v1 CLI STATUS 200 ## v1 CLI 200 #### v1 CLI TX| vcl.discard vcl1 #### v1 debug| unlink ./vcl.ORk8t3RP.so\n ### v1 CLI STATUS 200 ## v1 CLI 200 ## v1 as expected: n_backend (2) == 2 ## v1 as expected: n_vcl_avail (1) == 1 ## v1 as expected: n_vcl_discard (1) == 1 ## c1 Starting client #### v1 CLI TX| debug.backend ## c1 Started ### c1 Connect to 127.0.0.1:9081 #### v1 CLI RX| 4ff790 b1 1\n #### v1 CLI RX| 4ff850 b2 1\n ### v1 CLI STATUS 200 ## v1 CLI 200 #### v1 CLI TX| vcl.list #### v1 CLI RX| discarded 1 vcl1\n #### v1 CLI RX| active 0 vcl2\n ### v1 CLI STATUS 200 ## v1 CLI 200 ---- v1 Not true: n_backend (2) == 1 (1) -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/ From aotto at mosso.com Thu Aug 14 16:58:19 2008 From: aotto at mosso.com (Adrian Otto) Date: Thu, 14 Aug 2008 09:58:19 -0700 Subject: Bug in 1.1.2: Multiple headers stripped (oops) Message-ID: <19F24EDD-BC7A-4216-B390-102961663E4A@mosso.com> Hello, I've found, and reproduced an apparent bug in varnish 1.1.2. It happens when handling a Connection header sent from the client browser. The HTTP/1.1 protocol requires that any header names listed in the Connection header be removed before the proxy forwards the request to the backend web server. That works. It removes the specified header. However, it also removes the NEXT header that follows the specified one. This can lead to substantial trouble if the header following the specified one is the Host header. It results in a 404 result from the backend web server because varnish sends a Host header with the IP address of the backend web server rather than the original Host name that was sent by the client. This results in the incorrect document being fetched (and cached) from the backend server. I have provided two examples of the problem below. I need to do some additional work to set up my dev environment to start working on a patch, but if anyone else already has a patch for this, or can easily produce one, I's certainly appreciate any assistance you are willing to offer in the mean time. Thanks, Adrian Otto Here is an example: Notice that the Connection header from the client browser specifies that the TE header should be removed. However, once the request is proxied to the backend web server, the Host header has changed. 13 SessionOpen c 10.3.0.30 49408 0 ExpBan 121045864 was banned 13 ReqStart c 10.3.0.30 49408 121045866 13 RxRequest c GET 13 RxURL c /spacer_thin.gif 13 RxProtocol c HTTP/1.1 13 RxHeader c Referer: http://www.example.com/carchart/index.htm 13 RxHeader c Accept-Language: en-US,en;q=0.9 13 RxHeader c Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0 13 RxHeader c Cookie: X-Mapping- caklakng=BB7BF84977E04EAA5A5B5A914FFEE25F 13 RxHeader c X-Cluster-Client-Ip: 10.2.0.200 13 RxHeader c Cookie2: $Version=1 13 RxHeader c Connection: Keep-Alive, TE 13 RxHeader c Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- xbitmap, */*;q=0.1 13 RxHeader c TE: deflate, gzip, chunked, identity, trailers 13 RxHeader c Host: www.example.com 13 RxHeader c Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1 13 RxHeader c User-Agent: Opera/9.51 (Windows NT 5.1; U; en) 13 VCL_call c recv 13 VCL_return c lookup 13 VCL_call c hash 13 VCL_return c hash 13 VCL_call c miss 13 VCL_return c fetch 16 BackendOpen b default 10.0.0.100 34140 10.4.0.40 80 16 BackendXID b 121045866 13 Backend c 16 default 16 TxRequest b GET 16 TxURL b /spacer_thin.gif 16 TxProtocol b HTTP/1.1 16 TxHeader b Referer: http://www.example.com/carchart/index.htm 16 TxHeader b Accept-Language: en-US,en;q=0.9 16 TxHeader b Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0 16 TxHeader b X-Cluster-Client-Ip: 10.2.0.200 16 TxHeader b Cookie2: $Version=1 16 TxHeader b Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- xbitmap, */*;q=0.1 16 TxHeader b Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1 16 TxHeader b User-Agent: Opera/9.51 (Windows NT 5.1; U; en) 16 TxHeader b X-Varnish: 121045866 16 TxHeader b X-Forwarded-for: 10.3.0.30 16 TxHeader b Host: 10.4.0.40 16 RxProtocol b HTTP/1.1 16 RxStatus b 404 16 RxResponse b Not Found 16 RxHeader b Date: Thu, 14 Aug 2008 16:41:57 GMT 16 RxHeader b Server: Apache/2.0.52 (CentOS) 16 RxHeader b Content-Length: 290 16 RxHeader b Connection: close 16 RxHeader b Content-Type: text/html; charset=iso-8859-1 13 ObjProtocol c HTTP/1.1 13 ObjStatus c 404 13 ObjResponse c Not Found 13 ObjHeader c Date: Thu, 14 Aug 2008 16:41:57 GMT 13 ObjHeader c Server: Apache/2.0.52 (CentOS) 13 ObjHeader c Content-Type: text/html; charset=iso-8859-1 16 BackendClose b default 13 TTL c 121045866 RFC 1800 1218732117 1218732117 0 0 0 13 VCL_call c fetch 13 VCL_return c insert 13 Length c 290 13 VCL_call c deliver 13 VCL_return c deliver 13 TxProtocol c HTTP/1.1 13 TxStatus c 404 13 TxResponse c Not Found 13 TxHeader c Server: Apache/2.0.52 (CentOS) 13 TxHeader c Content-Type: text/html; charset=iso-8859-1 13 TxHeader c Content-Length: 290 13 TxHeader c Date: Thu, 14 Aug 2008 16:41:57 GMT 13 TxHeader c X-Varnish: 121045866 13 TxHeader c Age: 0 13 TxHeader c Via: 1.1 varnish 13 TxHeader c Connection: keep-alive 13 ReqEnd c 121045866 1218732117.032603979 1218732117.123506069 0.003006935 0.090833902 0.000068188 0 StatAddr 10.3.0.30 0 84 2 2 0 0 1 522 333 0 ExpKill 121045864 -1218732117 13 SessionClose c timeout 13 StatSess c 10.3.0.30 49408 0 1 1 0 0 1 233 290 13 SessionOpen c 10.3.0.30 49409 13 ReqStart c 10.3.0.30 49409 121045867 13 RxRequest c GET 13 RxURL c /spacer_thin.gif 13 RxProtocol c HTTP/1.1 13 RxHeader c Referer: http://www.example.com/carchart/index.htm 13 RxHeader c Accept-Language: en-US,en;q=0.9 13 RxHeader c Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0 13 RxHeader c Cookie: X-Mapping- caklakng=BB7BF84977E04EAA5A5B5A914FFEE25F 13 RxHeader c X-Cluster-Client-Ip: 10.2.0.200 13 RxHeader c Cookie2: $Version=1 13 RxHeader c Connection: Keep-Alive, TE 13 RxHeader c Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- xbitmap, */*;q=0.1 13 RxHeader c Host: www.example.com 13 RxHeader c TE: deflate, gzip, chunked, identity, trailers 13 RxHeader c Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1 13 RxHeader c User-Agent: Opera/9.51 (Windows NT 5.1; U; en) 13 VCL_call c recv 13 VCL_return c lookup 13 VCL_call c hash 13 VCL_return c hash 13 Hit c 121045866 13 VCL_call c hit 13 VCL_return c deliver 13 Length c 290 13 VCL_call c deliver 13 VCL_return c deliver 13 TxProtocol c HTTP/1.1 13 TxStatus c 404 13 TxResponse c Not Found 13 TxHeader c Server: Apache/2.0.52 (CentOS) 13 TxHeader c Content-Type: text/html; charset=iso-8859-1 13 TxHeader c Content-Length: 290 13 TxHeader c Date: Thu, 14 Aug 2008 16:43:18 GMT 13 TxHeader c X-Varnish: 121045867 121045866 13 TxHeader c Age: 81 13 TxHeader c Via: 1.1 varnish 13 TxHeader c Connection: keep-alive 13 ReqEnd c 121045867 1218732198.320590973 1218732198.320696115 0.003068924 0.000036001 0.000069141 0 StatAddr 10.3.0.30 0 165 3 3 0 0 1 766 623 13 SessionClose c timeout 13 StatSess c 10.3.0.30 49409 0 1 1 0 0 0 244 290 0 CLI Rd "url.purge" "/spacer_thin.gif" 0 CLI Wr 0 200 PURGE /spacer_thin.gif Here is another example that illustrates the same problem, but actually fetches the correct document from the backend web server. If the Host header is relocated to BEFORE the TE header, the Accept- Charset header vanishes instead, and the Host header goes through unmodified. This actually works, but we don't want to strip headers that should still be there. 13 SessionOpen c 10.3.0.30 49410 0 ExpBan 121045866 was banned 13 ReqStart c 10.3.0.30 49410 121045868 13 RxRequest c GET 13 RxURL c /spacer_thin.gif 13 RxProtocol c HTTP/1.1 13 RxHeader c Referer: http://www.example.com/carchart/index.htm 13 RxHeader c Accept-Language: en-US,en;q=0.9 13 RxHeader c Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0 13 RxHeader c Cookie: X-Mapping- caklakng=BB7BF84977E04EAA5A5B5A914FFEE25F 13 RxHeader c X-Cluster-Client-Ip: 10.2.0.200 13 RxHeader c Cookie2: $Version=1 13 RxHeader c Connection: Keep-Alive, TE 13 RxHeader c Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- xbitmap, */*;q=0.1 13 RxHeader c Host: www.example.com 13 RxHeader c TE: deflate, gzip, chunked, identity, trailers 13 RxHeader c Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1 13 RxHeader c User-Agent: Opera/9.51 (Windows NT 5.1; U; en) 13 VCL_call c recv 13 VCL_return c lookup 13 VCL_call c hash 13 VCL_return c hash 13 VCL_call c miss 13 VCL_return c fetch 16 BackendOpen b default 10.0.0.100 39798 10.4.0.40 80 16 BackendXID b 121045868 13 Backend c 16 default 16 TxRequest b GET 16 TxURL b /spacer_thin.gif 16 TxProtocol b HTTP/1.1 16 TxHeader b Referer: http://www.example.com/carchart/index.htm 16 TxHeader b Accept-Language: en-US,en;q=0.9 16 TxHeader b Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0 16 TxHeader b X-Cluster-Client-Ip: 10.2.0.200 16 TxHeader b Cookie2: $Version=1 16 TxHeader b Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- xbitmap, */*;q=0.1 16 TxHeader b Host: www.example.com 16 TxHeader b User-Agent: Opera/9.51 (Windows NT 5.1; U; en) 16 TxHeader b X-Varnish: 121045868 16 TxHeader b X-Forwarded-for: 10.3.0.30 16 RxProtocol b HTTP/1.1 16 RxStatus b 200 16 RxResponse b OK 16 RxHeader b Date: Thu, 14 Aug 2008 16:43:35 GMT 16 RxHeader b Server: Apache/2.0.52 (CentOS) 16 RxHeader b Last-Modified: Fri, 02 Feb 2007 02:57:20 GMT 16 RxHeader b ETag: "40d5-2b-808f4400" 16 RxHeader b Accept-Ranges: bytes 16 RxHeader b Content-Length: 43 16 RxHeader b Connection: close 16 RxHeader b Content-Type: image/gif 13 ObjProtocol c HTTP/1.1 13 ObjStatus c 200 13 ObjResponse c OK 13 ObjHeader c Date: Thu, 14 Aug 2008 16:43:35 GMT 13 ObjHeader c Server: Apache/2.0.52 (CentOS) 13 ObjHeader c Last-Modified: Fri, 02 Feb 2007 02:57:20 GMT 13 ObjHeader c ETag: "40d5-2b-808f4400" 13 ObjHeader c Content-Type: image/gif 16 BackendClose b default 13 TTL c 121045868 RFC 1800 1218732215 1218732215 0 0 0 13 VCL_call c fetch 13 VCL_return c insert 13 Length c 43 13 VCL_call c deliver 13 VCL_return c deliver 13 TxProtocol c HTTP/1.1 13 TxStatus c 200 13 TxResponse c OK 13 TxHeader c Server: Apache/2.0.52 (CentOS) 13 TxHeader c Last-Modified: Fri, 02 Feb 2007 02:57:20 GMT 13 TxHeader c ETag: "40d5-2b-808f4400" 13 TxHeader c Content-Type: image/gif 13 TxHeader c Content-Length: 43 13 TxHeader c Date: Thu, 14 Aug 2008 16:43:35 GMT 13 TxHeader c X-Varnish: 121045868 13 TxHeader c Age: 0 13 TxHeader c Via: 1.1 varnish 13 TxHeader c Connection: keep-alive 13 ReqEnd c 121045868 1218732215.300252914 1218732215.390639067 0.005183935 0.090335131 0.000051022 0 StatAddr 10.3.0.30 0 182 4 4 0 0 2 1043 666 0 ExpKill 121045866 -1218732216 13 SessionClose c timeout 13 StatSess c 10.3.0.30 49410 0 1 1 0 0 1 277 43 From phk at phk.freebsd.dk Fri Aug 15 08:32:16 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 15 Aug 2008 08:32:16 +0000 Subject: waitpid EINTR issue in varnishd (& testcases) In-Reply-To: Your message of "Wed, 13 Aug 2008 15:19:51 -0400." <72AB9D1A-F485-4DF0-A445-4C375F25075B@omniti.com> Message-ID: <1952.1218789136@critter.freebsd.dk> I have committed the EINTR patch and, I hope, fixed the two testcases, please check this and send me your latest solaris patch. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jesus at omniti.com Fri Aug 15 23:11:12 2008 From: jesus at omniti.com (Theo Schlossnagle) Date: Fri, 15 Aug 2008 19:11:12 -0400 Subject: waitpid EINTR issue in varnishd (& testcases) In-Reply-To: <1952.1218789136@critter.freebsd.dk> References: <1952.1218789136@critter.freebsd.dk> Message-ID: On Aug 15, 2008, at 4:32 AM, Poul-Henning Kamp wrote: > > I have committed the EINTR patch and, I hope, fixed the two testcases, > please check this and send me your latest solaris patch. This looks good and now the full test suite passes. Attached is the Solaris patch. -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: varnish-cache-3095-solaris.patch Type: application/octet-stream Size: 15373 bytes Desc: not available URL: -------------- next part -------------- From tfheen at linpro.no Mon Aug 18 07:43:06 2008 From: tfheen at linpro.no (Tollef Fog Heen) Date: Mon, 18 Aug 2008 09:43:06 +0200 Subject: waitpid EINTR issue in varnishd (& testcases) In-Reply-To: (Theo Schlossnagle's message of "Fri, 15 Aug 2008 19:11:12 -0400") References: <1952.1218789136@critter.freebsd.dk> Message-ID: <87zlnakbud.fsf@linpro.no> ]] Theo Schlossnagle | On Aug 15, 2008, at 4:32 AM, Poul-Henning Kamp wrote: | | > | > I have committed the EINTR patch and, I hope, fixed the two testcases, | > please check this and send me your latest solaris patch. | | This looks good and now the full test suite passes. | | Attached is the Solaris patch. Seems to build fine for me; I have a failure in varnishtest which is there regardless of your patch or not. (Linux, Ubuntu 8.04, x86_64) -- Tollef Fog Heen / Linpro AS t: 21 54 41 73 UNIX is user friendly, it's just picky about who its friends are From aotto at mosso.com Mon Aug 18 18:20:23 2008 From: aotto at mosso.com (Adrian Otto) Date: Mon, 18 Aug 2008 11:20:23 -0700 Subject: Bug in 1.1.2: Multiple headers stripped (oops) In-Reply-To: <19F24EDD-BC7A-4216-B390-102961663E4A@mosso.com> References: <19F24EDD-BC7A-4216-B390-102961663E4A@mosso.com> Message-ID: Hello, I found the reason why I hit this bug, but I need some tips to understand how to debug what's happening in the VCL code. First, I'll explain the bug: 1) My default.vcl is configured to strip the Cookie header in vcl_recv like this: sub vcl_recv { if(req.http.Cookie) { remove req.http.Cookie; lookup; } } 2) The Connection header sent from the client browser specified that the "TE" header should be stripped, and was so marked in position 13 of http_DoConnection::hp->hdf, indicating that header at index 13 should be skipped when headers are copied into the backend request. 3) The VCL code ran to strip the cookie header, removed it, and shifted the other elements in the array UP one position. The Host header followed TE at position/index 14, and became position/index 13. 4) The code for copying/morphing the client request to the backend ran, and skipped position/index 13, which was the Host header. 5) Varnish kindly noticed that there was no Host header present in the backend connection, and added one of it's own using the IP address of the backend server as the content of that header. 6) The incorrect document was fetched for the backend because the Host header was not correct. I debugged the problem and found that the client HTTP headers change in this section of code: Run till exit from #0 WSL (w=0x43c02c50, tag=SLT_VCL_call, id=13, fmt=0x42c8b5 "%s") at shmlog.c:248 0x00000000004183e3 in VCL_recv_method (sp=0x17bab748) at ../../ include/vcl_returns.h:37 37 VCL_MET_MAC(recv,RECV,(VCL_RET_ERROR|VCL_RET_PASS| VCL_RET_PIPE|VCL_RET_LOOKUP)) (gdb) list 32 #define VCL_RET_KEEP (1 << 9) 33 #define VCL_RET_MAX 10 34 #endif 35 36 #ifdef VCL_MET_MAC 37 VCL_MET_MAC(recv,RECV,(VCL_RET_ERROR|VCL_RET_PASS| VCL_RET_PIPE|VCL_RET_LOOKUP)) 38 VCL_MET_MAC(pipe,PIPE,(VCL_RET_ERROR|VCL_RET_PIPE)) 39 VCL_MET_MAC(pass,PASS,(VCL_RET_ERROR|VCL_RET_PASS)) 40 VCL_MET_MAC(hash,HASH,(VCL_RET_HASH)) 41 VCL_MET_MAC(miss,MISS,(VCL_RET_ERROR|VCL_RET_PASS| VCL_RET_FETCH)) (gdb) bt #0 VCL_recv_method (sp=0x17b955b8) at ../../include/vcl_returns.h:37 #1 0x000000000040aa43 in cnt_recv (sp=0x17b955b8) at cache_center.c:730 #2 0x000000000040af61 in CNT_Session (sp=0x17b955b8) at steps.h:34 #3 0x000000000041389b in wrk_do_one (w=0x43c02c50) at cache_pool.c:194 #4 0x0000000000413dd3 in wrk_thread (priv=0x17b7cc60) at cache_pool.c:248 #5 0x00000037d50062f7 in start_thread () from /lib64/libpthread.so.0 #6 0x0000003190cce85d in clone () from /lib64/libc.so.6 (gdb) print VCL_recv_method::sp->http->hd $43 = {[0] = {b = 0x17babc50 "GET", e = 0x17babc53 ""}, [1] = {b = 0x17babc54 "/spacer_thin.gif", e = 0x17babc64 ""}, [2] = {b = 0x17babc65 "HTTP/1.1", e = 0x17babc6d ""}, [3] = {b = 0x0, e = 0x0}, [4] = {b = 0x0, e = 0x0}, [5] = {b = 0x17babc6f "Referer: http://www.motioncars.com/carchart/ index.htm", e = 0x17babca4 ""}, [6] = {b = 0x17babca6 "Accept-Language: en-US,en;q=0.9", e = 0x17babcc5 ""}, [7] = {b = 0x17babcc7 "Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0", e = 0x17babcfe ""}, [8] = {b = 0x17babd00 "Cookie: X-Mapping- caklakng=BB7BF84977E04EAA5A5B5A914FFEE25F", e = 0x17babd3b ""}, [9] = {b = 0x17babd3d "X-Cluster-Client-Ip: 122.53.96.247", e = 0x17babd5f ""}, [10] = {b = 0x17babd61 "Cookie2: $Version=1", e = 0x17babd74 ""}, [11] = {b = 0x17babd76 "Connection: Keep-Alive, TE", e = 0x17babd90 ""}, [12] = {b = 0x17babd92 "Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- xbitmap, */*;q=0.1", e = 0x17babe0f ""}, [13] = {b = 0x17babe11 "TE: deflate, gzip, chunked, identity, trailers", e = 0x17babe3f ""}, [14] = {b = 0x17babe41 "Host: www.motioncars.com", e = 0x17babe59 ""}, [15] = {b = 0x17babe5b "Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1", e = 0x17babe8d ""}, [16] = {b = 0x17babe8f "User-Agent: Opera/9.51 (Windows NT 5.1; U; en)", e = 0x17babebd ""}, [17] = {b = 0x0, e = 0x0} } (gdb) print VCL_recv_method::sp->http->hdf $44 = '\0' , "\001", '\0' (gdb) step vcl_handlingname (u=2) at cache_vcl.c:288 288 switch (u) { (gdb) print VCL_recv_method::sp->http->hdf $45 = '\0' , "\001", '\0' (gdb) print VCL_recv_method::sp->http->hd $46 = {[0] = {b = 0x17babc50 "GET", e = 0x17babc53 ""}, [1] = {b = 0x17babc54 "/spacer_thin.gif", e = 0x17babc64 ""}, [2] = {b = 0x17babc65 "HTTP/1.1", e = 0x17babc6d ""}, [3] = {b = 0x0, e = 0x0}, [4] = {b = 0x0, e = 0x0}, [5] = {b = 0x17babc6f "Referer: http://www.motioncars.com/carchart/ index.htm", e = 0x17babca4 ""}, [6] = {b = 0x17babca6 "Accept-Language: en-US,en;q=0.9", e = 0x17babcc5 ""}, [7] = {b = 0x17babcc7 "Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0", e = 0x17babcfe ""}, [8] = {b = 0x17babd3d "X-Cluster-Client-Ip: 122.53.96.247", e = 0x17babd5f ""}, [9] = {b = 0x17babd61 "Cookie2: $Version=1", e = 0x17babd74 ""}, [10] = {b = 0x17babd76 "Connection: Keep-Alive, TE", e = 0x17babd90 ""}, [11] = {b = 0x17babd92 "Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- xbitmap, */*;q=0.1", e = 0x17babe0f ""}, [12] = {b = 0x17babe11 "TE: deflate, gzip, chunked, identity, trailers", e = 0x17babe3f ""}, [13] = {b = 0x17babe41 "Host: www.motioncars.com", e = 0x17babe59 ""}, [14] = {b = 0x17babe5b "Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1", e = 0x17babe8d ""}, [15] = {b = 0x17babe8f "User-Agent: Opera/9.51 (Windows NT 5.1; U; en)", e = 0x17babebd ""}, [16] = {b = 0x17babe8f "User-Agent: Opera/9.51 (Windows NT 5.1; U; en)", e = 0x17babebd ""}, [17] = {b = 0x0, e = 0x0} } (gdb) bt #0 vcl_handlingname (u=2) at cache_vcl.c:288 #1 0x0000000000418407 in VCL_recv_method (sp=0x17bab748) at ../../ include/vcl_returns.h:37 #2 0x000000000040aa43 in cnt_recv (sp=0x17bab748) at cache_center.c:730 #3 0x000000000040af61 in CNT_Session (sp=0x17bab748) at steps.h:34 #4 0x000000000041389b in wrk_do_one (w=0x43c02c50) at cache_pool.c:194 #5 0x0000000000413dd3 in wrk_thread (priv=0x17b7cc60) at cache_pool.c:248 #6 0x00000037d50062f7 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003190cce85d in clone () from /lib64/libc.so.6 (gdb) list 283 284 static const char * 285 vcl_handlingname(unsigned u) 286 { 287 288 switch (u) { 289 #define VCL_RET_MAC(a, b, c,d) case VCL_RET_##b: return(#a); 290 #define VCL_RET_MAC_E(a, b, c,d) case VCL_RET_##b: return(#a); 291 #include "vcl_returns.h" 292 #undef VCL_RET_MAC Notice that sp->http->hd[8] gets removed from the array and other items shifted up... apparently somewhere after "VCL_recv_method (sp=0x17bab748) at ../../include/vcl_returns.h:37" and before "vcl_handlingname (u=2) at cache_vcl.c:288". However, the sp->http- >hdf variable is not adjusted accordingly. That's the bug. When it removes the header, it should also adjust the sp->http->hdf variable so that the header indexes marked for skipping with HDF_FILTER still refer to the correct indexes in the sp->http->hd array after it's modified. Another fix might be to adjust whatever code is behind "remove" in vcl_recv so that it simply sets HDF_FILTER in sp->http->hdf for the header to be deleted, instead of actually removing it. I'm not sure why two different methods are used for this, so I'm not sure which fix is better. Note that the workaround is to adjust the configuration to eliminate all "remove" statements in vcl_recv. If no headers are removed, then the bug is avoided. Nobody should use the "remove" configuration directive in the vcl_recv section of the config file until this is fixed, or this bug may crop up. Any header might vanish, depending on how the client browser forms it's request. My workaround was to configure a proxying firewall (before varnish in our topology) to strip out the Cookie header instead, eliminating the need for varnish to check/remove the header. I'd like to actually submit a patch for this, but I'm having difficulty understanding where to find the actual C code that corresponds to the VCL code responsible for "remove" in vcl_recv so I can fix it. My gdb is not showing it to me, so I'm not sure what code to patch. Can anyone offer me a clue here for how to find the correct code to fix? Thanks, Adrian On Aug 14, 2008, at 9:58 AM, Adrian Otto wrote: > Hello, > > I've found, and reproduced an apparent bug in varnish 1.1.2. It > happens when handling a Connection header sent from the client > browser. The HTTP/1.1 protocol requires that any header names > listed in the Connection header be removed before the proxy > forwards the request to the backend web server. That works. It > removes the specified header. However, it also removes the NEXT > header that follows the specified one. This can lead to substantial > trouble if the header following the specified one is the Host > header. It results in a 404 result from the backend web server > because varnish sends a Host header with the IP address of the > backend web server rather than the original Host name that was sent > by the client. This results in the incorrect document being fetched > (and cached) from the backend server. > > I have provided two examples of the problem below. I need to do > some additional work to set up my dev environment to start working > on a patch, but if anyone else already has a patch for this, or can > easily produce one, I's certainly appreciate any assistance you are > willing to offer in the mean time. > > Thanks, > > Adrian Otto > > Here is an example: > > Notice that the Connection header from the client browser specifies > that the TE header should be removed. However, once the request is > proxied to the backend web server, the Host header has changed. > > 13 SessionOpen c 10.3.0.30 49408 > 0 ExpBan 121045864 was banned > 13 ReqStart c 10.3.0.30 49408 121045866 > 13 RxRequest c GET > 13 RxURL c /spacer_thin.gif > 13 RxProtocol c HTTP/1.1 > 13 RxHeader c Referer: http://www.example.com/carchart/ > index.htm > 13 RxHeader c Accept-Language: en-US,en;q=0.9 > 13 RxHeader c Accept-Encoding: deflate, gzip, x-gzip, > identity, *;q=0 > 13 RxHeader c Cookie: X-Mapping- > caklakng=BB7BF84977E04EAA5A5B5A914FFEE25F > 13 RxHeader c X-Cluster-Client-Ip: 10.2.0.200 > 13 RxHeader c Cookie2: $Version=1 > 13 RxHeader c Connection: Keep-Alive, TE > 13 RxHeader c Accept: text/html, application/xml;q=0.9, > application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- > xbitmap, */*;q=0.1 > 13 RxHeader c TE: deflate, gzip, chunked, identity, trailers > 13 RxHeader c Host: www.example.com > 13 RxHeader c Accept-Charset: iso-8859-1, utf-8, utf-16, > *;q=0.1 > 13 RxHeader c User-Agent: Opera/9.51 (Windows NT 5.1; U; en) > 13 VCL_call c recv > 13 VCL_return c lookup > 13 VCL_call c hash > 13 VCL_return c hash > 13 VCL_call c miss > 13 VCL_return c fetch > 16 BackendOpen b default 10.0.0.100 34140 10.4.0.40 80 > 16 BackendXID b 121045866 > 13 Backend c 16 default > 16 TxRequest b GET > 16 TxURL b /spacer_thin.gif > 16 TxProtocol b HTTP/1.1 > 16 TxHeader b Referer: http://www.example.com/carchart/ > index.htm > 16 TxHeader b Accept-Language: en-US,en;q=0.9 > 16 TxHeader b Accept-Encoding: deflate, gzip, x-gzip, > identity, *;q=0 > 16 TxHeader b X-Cluster-Client-Ip: 10.2.0.200 > 16 TxHeader b Cookie2: $Version=1 > 16 TxHeader b Accept: text/html, application/xml;q=0.9, > application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- > xbitmap, */*;q=0.1 > 16 TxHeader b Accept-Charset: iso-8859-1, utf-8, utf-16, > *;q=0.1 > 16 TxHeader b User-Agent: Opera/9.51 (Windows NT 5.1; U; en) > 16 TxHeader b X-Varnish: 121045866 > 16 TxHeader b X-Forwarded-for: 10.3.0.30 > 16 TxHeader b Host: 10.4.0.40 > 16 RxProtocol b HTTP/1.1 > 16 RxStatus b 404 > 16 RxResponse b Not Found > 16 RxHeader b Date: Thu, 14 Aug 2008 16:41:57 GMT > 16 RxHeader b Server: Apache/2.0.52 (CentOS) > 16 RxHeader b Content-Length: 290 > 16 RxHeader b Connection: close > 16 RxHeader b Content-Type: text/html; charset=iso-8859-1 > 13 ObjProtocol c HTTP/1.1 > 13 ObjStatus c 404 > 13 ObjResponse c Not Found > 13 ObjHeader c Date: Thu, 14 Aug 2008 16:41:57 GMT > 13 ObjHeader c Server: Apache/2.0.52 (CentOS) > 13 ObjHeader c Content-Type: text/html; charset=iso-8859-1 > 16 BackendClose b default > 13 TTL c 121045866 RFC 1800 1218732117 1218732117 0 0 0 > 13 VCL_call c fetch > 13 VCL_return c insert > 13 Length c 290 > 13 VCL_call c deliver > 13 VCL_return c deliver > 13 TxProtocol c HTTP/1.1 > 13 TxStatus c 404 > 13 TxResponse c Not Found > 13 TxHeader c Server: Apache/2.0.52 (CentOS) > 13 TxHeader c Content-Type: text/html; charset=iso-8859-1 > 13 TxHeader c Content-Length: 290 > 13 TxHeader c Date: Thu, 14 Aug 2008 16:41:57 GMT > 13 TxHeader c X-Varnish: 121045866 > 13 TxHeader c Age: 0 > 13 TxHeader c Via: 1.1 varnish > 13 TxHeader c Connection: keep-alive > 13 ReqEnd c 121045866 1218732117.032603979 > 1218732117.123506069 0.003006935 0.090833902 0.000068188 > 0 StatAddr 10.3.0.30 0 84 2 2 0 0 1 522 333 > 0 ExpKill 121045864 -1218732117 > 13 SessionClose c timeout > 13 StatSess c 10.3.0.30 49408 0 1 1 0 0 1 233 290 > 13 SessionOpen c 10.3.0.30 49409 > 13 ReqStart c 10.3.0.30 49409 121045867 > 13 RxRequest c GET > 13 RxURL c /spacer_thin.gif > 13 RxProtocol c HTTP/1.1 > 13 RxHeader c Referer: http://www.example.com/carchart/ > index.htm > 13 RxHeader c Accept-Language: en-US,en;q=0.9 > 13 RxHeader c Accept-Encoding: deflate, gzip, x-gzip, > identity, *;q=0 > 13 RxHeader c Cookie: X-Mapping- > caklakng=BB7BF84977E04EAA5A5B5A914FFEE25F > 13 RxHeader c X-Cluster-Client-Ip: 10.2.0.200 > 13 RxHeader c Cookie2: $Version=1 > 13 RxHeader c Connection: Keep-Alive, TE > 13 RxHeader c Accept: text/html, application/xml;q=0.9, > application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- > xbitmap, */*;q=0.1 > 13 RxHeader c Host: www.example.com > 13 RxHeader c TE: deflate, gzip, chunked, identity, trailers > 13 RxHeader c Accept-Charset: iso-8859-1, utf-8, utf-16, > *;q=0.1 > 13 RxHeader c User-Agent: Opera/9.51 (Windows NT 5.1; U; en) > 13 VCL_call c recv > 13 VCL_return c lookup > 13 VCL_call c hash > 13 VCL_return c hash > 13 Hit c 121045866 > 13 VCL_call c hit > 13 VCL_return c deliver > 13 Length c 290 > 13 VCL_call c deliver > 13 VCL_return c deliver > 13 TxProtocol c HTTP/1.1 > 13 TxStatus c 404 > 13 TxResponse c Not Found > 13 TxHeader c Server: Apache/2.0.52 (CentOS) > 13 TxHeader c Content-Type: text/html; charset=iso-8859-1 > 13 TxHeader c Content-Length: 290 > 13 TxHeader c Date: Thu, 14 Aug 2008 16:43:18 GMT > 13 TxHeader c X-Varnish: 121045867 121045866 > 13 TxHeader c Age: 81 > 13 TxHeader c Via: 1.1 varnish > 13 TxHeader c Connection: keep-alive > 13 ReqEnd c 121045867 1218732198.320590973 > 1218732198.320696115 0.003068924 0.000036001 0.000069141 > 0 StatAddr 10.3.0.30 0 165 3 3 0 0 1 766 623 > 13 SessionClose c timeout > 13 StatSess c 10.3.0.30 49409 0 1 1 0 0 0 244 290 > 0 CLI Rd "url.purge" "/spacer_thin.gif" > 0 CLI Wr 0 200 PURGE /spacer_thin.gif > > > Here is another example that illustrates the same problem, but > actually fetches the correct document from the backend web server. > If the Host header is relocated to BEFORE the TE header, the Accept- > Charset header vanishes instead, and the Host header goes through > unmodified. This actually works, but we don't want to strip headers > that should still be there. > > 13 SessionOpen c 10.3.0.30 49410 > 0 ExpBan 121045866 was banned > 13 ReqStart c 10.3.0.30 49410 121045868 > 13 RxRequest c GET > 13 RxURL c /spacer_thin.gif > 13 RxProtocol c HTTP/1.1 > 13 RxHeader c Referer: http://www.example.com/carchart/ > index.htm > 13 RxHeader c Accept-Language: en-US,en;q=0.9 > 13 RxHeader c Accept-Encoding: deflate, gzip, x-gzip, > identity, *;q=0 > 13 RxHeader c Cookie: X-Mapping- > caklakng=BB7BF84977E04EAA5A5B5A914FFEE25F > 13 RxHeader c X-Cluster-Client-Ip: 10.2.0.200 > 13 RxHeader c Cookie2: $Version=1 > 13 RxHeader c Connection: Keep-Alive, TE > 13 RxHeader c Accept: text/html, application/xml;q=0.9, > application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- > xbitmap, */*;q=0.1 > 13 RxHeader c Host: www.example.com > 13 RxHeader c TE: deflate, gzip, chunked, identity, trailers > 13 RxHeader c Accept-Charset: iso-8859-1, utf-8, utf-16, > *;q=0.1 > 13 RxHeader c User-Agent: Opera/9.51 (Windows NT 5.1; U; en) > 13 VCL_call c recv > 13 VCL_return c lookup > 13 VCL_call c hash > 13 VCL_return c hash > 13 VCL_call c miss > 13 VCL_return c fetch > 16 BackendOpen b default 10.0.0.100 39798 10.4.0.40 80 > 16 BackendXID b 121045868 > 13 Backend c 16 default > 16 TxRequest b GET > 16 TxURL b /spacer_thin.gif > 16 TxProtocol b HTTP/1.1 > 16 TxHeader b Referer: http://www.example.com/carchart/ > index.htm > 16 TxHeader b Accept-Language: en-US,en;q=0.9 > 16 TxHeader b Accept-Encoding: deflate, gzip, x-gzip, > identity, *;q=0 > 16 TxHeader b X-Cluster-Client-Ip: 10.2.0.200 > 16 TxHeader b Cookie2: $Version=1 > 16 TxHeader b Accept: text/html, application/xml;q=0.9, > application/xhtml+xml, image/png, image/jpeg, image/gif, image/x- > xbitmap, */*;q=0.1 > 16 TxHeader b Host: www.example.com > 16 TxHeader b User-Agent: Opera/9.51 (Windows NT 5.1; U; en) > 16 TxHeader b X-Varnish: 121045868 > 16 TxHeader b X-Forwarded-for: 10.3.0.30 > 16 RxProtocol b HTTP/1.1 > 16 RxStatus b 200 > 16 RxResponse b OK > 16 RxHeader b Date: Thu, 14 Aug 2008 16:43:35 GMT > 16 RxHeader b Server: Apache/2.0.52 (CentOS) > 16 RxHeader b Last-Modified: Fri, 02 Feb 2007 02:57:20 GMT > 16 RxHeader b ETag: "40d5-2b-808f4400" > 16 RxHeader b Accept-Ranges: bytes > 16 RxHeader b Content-Length: 43 > 16 RxHeader b Connection: close > 16 RxHeader b Content-Type: image/gif > 13 ObjProtocol c HTTP/1.1 > 13 ObjStatus c 200 > 13 ObjResponse c OK > 13 ObjHeader c Date: Thu, 14 Aug 2008 16:43:35 GMT > 13 ObjHeader c Server: Apache/2.0.52 (CentOS) > 13 ObjHeader c Last-Modified: Fri, 02 Feb 2007 02:57:20 GMT > 13 ObjHeader c ETag: "40d5-2b-808f4400" > 13 ObjHeader c Content-Type: image/gif > 16 BackendClose b default > 13 TTL c 121045868 RFC 1800 1218732215 1218732215 0 0 0 > 13 VCL_call c fetch > 13 VCL_return c insert > 13 Length c 43 > 13 VCL_call c deliver > 13 VCL_return c deliver > 13 TxProtocol c HTTP/1.1 > 13 TxStatus c 200 > 13 TxResponse c OK > 13 TxHeader c Server: Apache/2.0.52 (CentOS) > 13 TxHeader c Last-Modified: Fri, 02 Feb 2007 02:57:20 GMT > 13 TxHeader c ETag: "40d5-2b-808f4400" > 13 TxHeader c Content-Type: image/gif > 13 TxHeader c Content-Length: 43 > 13 TxHeader c Date: Thu, 14 Aug 2008 16:43:35 GMT > 13 TxHeader c X-Varnish: 121045868 > 13 TxHeader c Age: 0 > 13 TxHeader c Via: 1.1 varnish > 13 TxHeader c Connection: keep-alive > 13 ReqEnd c 121045868 1218732215.300252914 > 1218732215.390639067 0.005183935 0.090335131 0.000051022 > 0 StatAddr 10.3.0.30 0 182 4 4 0 0 2 1043 666 > 0 ExpKill 121045866 -1218732216 > 13 SessionClose c timeout > 13 StatSess c 10.3.0.30 49410 0 1 1 0 0 1 277 43 -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Tue Aug 19 19:37:11 2008 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 19 Aug 2008 19:37:11 +0000 Subject: Bug in 1.1.2: Multiple headers stripped (oops) In-Reply-To: Your message of "Mon, 18 Aug 2008 11:20:23 MST." Message-ID: <69886.1219174631@critter.freebsd.dk> Hi Adrian, Good catch. I created a ticket (#292, now closed) for this because we number our regression tests by ticket number. Fixed in #3110. Poul-Henning >I found the reason why I hit this bug, but I need some tips to >understand how to debug what's happening in the VCL code. First, I'll >explain the bug: > >1) My default.vcl is configured to strip the Cookie header in >vcl_recv like this: > >sub vcl_recv { > if(req.http.Cookie) { > remove req.http.Cookie; > lookup; > } >} > >2) The Connection header sent from the client browser specified that >the "TE" header should be stripped, and was so marked in position 13 >of http_DoConnection::hp->hdf, indicating that header at index 13 >should be skipped when headers are copied into the backend request. > >3) The VCL code ran to strip the cookie header, removed it, and >shifted the other elements in the array UP one position. The Host >header followed TE at position/index 14, and became position/index 13. > >4) The code for copying/morphing the client request to the backend >ran, and skipped position/index 13, which was the Host header. > >5) Varnish kindly noticed that there was no Host header present in >the backend connection, and added one of it's own using the IP >address of the backend server as the content of that header. > >6) The incorrect document was fetched for the backend because the >Host header was not correct. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jesus at omniti.com Thu Aug 21 01:37:40 2008 From: jesus at omniti.com (Theo Schlossnagle) Date: Wed, 20 Aug 2008 21:37:40 -0400 Subject: commit 3113 all good Message-ID: <6B52A944-8725-41F2-91DD-FDD49490B325@omniti.com> Trunk past 3113 compiles out of the box on Solaris 10+ and passes all the varnishtests. Thanks! -- Theo Schlossnagle Esoteric Curio -- http://lethargy.org/ OmniTI Computer Consulting, Inc. -- http://omniti.com/ From ingvar at linpro.no Fri Aug 22 06:03:05 2008 From: ingvar at linpro.no (Ingvar Hagelund) Date: Fri, 22 Aug 2008 08:03:05 +0200 Subject: make check reports 2 errors on fedora9/i386 Message-ID: <48AE5699.1010701@linpro.no> Started with a clean svn co, just ./autogen.sh; ./configure; make -j4; make check. See attatched log from make check. This is on fedora9/i386 on Intel E6550. I did not see this on my home box running same OS (fedora9/i386) on AMD. Ingvar -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: makecheck-fedora9-i386-intelE6550-svn-3120-2008-08-22.log URL: From ingvar at linpro.no Fri Aug 22 07:20:36 2008 From: ingvar at linpro.no (Ingvar Hagelund) Date: Fri, 22 Aug 2008 09:20:36 +0200 Subject: make check reports 2 errors on fedora9/i386 In-Reply-To: <48AE5699.1010701@linpro.no> References: <48AE5699.1010701@linpro.no> Message-ID: <48AE68C4.1030308@linpro.no> * Ingvar Hagelund > Started with a clean svn co, just ./autogen.sh; ./configure; make -j4; > make check. > > See attatched log from make check. This is on fedora9/i386 on Intel E6550. Did the same. A clean svn checkout, but added -v to varnishtest, and run autogen, configure, make, make check. Now, only 1 test failed(!) Ingvar -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: makecheck-v-fedora9-i386-intelE6550-svn-3120-2008-08-22.log URL: