From phk at phk.freebsd.dk Tue Dec 1 09:08:05 2009 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 01 Dec 2009 09:08:05 +0000 Subject: varnish bottleneck? In-Reply-To: Your message of "Fri, 27 Nov 2009 09:12:10 +0800." <4B0F276A.9000909@gmail.com> Message-ID: <31719.1259658485@critter.freebsd.dk> In message <4B0F276A.9000909 at gmail.com>, ll writes: >if (req.request == "POST"){ >pipe; >} Use pass, not pipe. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From varnish-bugs at projects.linpro.no Tue Dec 1 16:05:40 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 01 Dec 2009 16:05:40 -0000 Subject: [Varnish] #593: Add VCL syntax check option Message-ID: <053.9b98b16d697afce6b64a60f696e3af9c@projects.linpro.no> #593: Add VCL syntax check option ---------------------+------------------------------------------------------ Reporter: mperham | Type: enhancement Status: new | Priority: normal Milestone: | Component: build Version: trunk | Severity: normal Keywords: | ---------------------+------------------------------------------------------ Apache and nginx both have the "-t" option which will check the syntax of config files and exit, which can be used to prevent deployment of invalid configurations. I'd like to see a similar feature in Varnish. It looks like "-t" is already taken so I would suggest "-c" or '-C' for 'Check configuration'. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 1 18:38:53 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 01 Dec 2009 18:38:53 -0000 Subject: [Varnish] #547: Varnish core dumps frequently In-Reply-To: <053.bb3ab0292d73880f2e8477a3aa3e3c6f@projects.linpro.no> References: <053.bb3ab0292d73880f2e8477a3aa3e3c6f@projects.linpro.no> Message-ID: <062.8e7a3bc053022765202192c0ff2d9dc9@projects.linpro.no> #547: Varnish core dumps frequently ---------------------+------------------------------------------------------ Reporter: victori | Owner: Type: defect | Status: new Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Resolution: Keywords: | ---------------------+------------------------------------------------------ Comment (by victori): Fixed... in lib/libvarnish/tcp.c / TCP_linger(int sock, int linger) change AZ(setsockopt(sock, SOL_SOCKET, SO_LINGER, &lin, sizeof lin)); to setsockopt(sock, SOL_SOCKET, SO_LINGER, &lin, sizeof lin); This fixues the issue on solaris. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 2 07:53:16 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 02 Dec 2009 07:53:16 -0000 Subject: =?utf-8?b?W1Zhcm5pc2hdICM1OTQ6IEVTSSBmcm9tIEFwYWNoZTIgYmFja2Vu?= =?utf-8?b?ZCB3aXRoIG1vZF9kZWZsYXRlIHJldHVybnMgTm8gRVNJIHByb2Nl?= =?utf-8?b?c3NpbmcsIGZpcnN0IGNoYXIgbm90IOKAmDwgJw==?= Message-ID: <050.308fb7601729eace2249e2b04bf1efa8@projects.linpro.no> #594: ESI from Apache2 backend with mod_deflate returns No ESI processing, first char not ?< ' ----------------------+----------------------------------------------------- Reporter: cd34 | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 2.0 Severity: normal | Keywords: ----------------------+----------------------------------------------------- Varnish 2.0.3/2.0.4/2.0.5 Debian GNU 3.1/Testing Apache2.2.14 When using Varnish with esi includes, if the backend has mod_deflate enabled, the ESI include request appears to have the request sent through with Accept-Encoding: gzip,deflate (or whatever the browser requesting the page that contains the esi:include has sent) Temporarily disabling mod_deflate allows ESI to work correctly. Setting -p esi_syntax=0x1 on the command line doesn't appear to work either. If I leave mod_deflate on, and do the request manually without Accept-Encode, the ESI include works. I believe eliminating the Accept-Encoding header from the request when doing an ESI include would probably fix the issue. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 2 12:22:20 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 02 Dec 2009 12:22:20 -0000 Subject: [Varnish] #217: varnishtop freeze In-Reply-To: <049.a5bcd9b08e2b26a91152e804621bfcd7@projects.linpro.no> References: <049.a5bcd9b08e2b26a91152e804621bfcd7@projects.linpro.no> Message-ID: <058.ba3751a746928a642ba9a51d3ec96bf2@projects.linpro.no> #217: varnishtop freeze ------------------------+--------------------------------------------------- Reporter: akv | Owner: phk Type: defect | Status: reopened Priority: normal | Milestone: Component: varnishtop | Version: trunk Severity: normal | Resolution: Keywords: | ------------------------+--------------------------------------------------- Changes (by adam): * status: closed => reopened * resolution: worksforme => Comment: I am experiencing this as well with varnish 2.0.5. It's not happening all the time, though. Sometimes, varnishtop runs fine until I quit. But other times it freezes after zero or one updates (either nothing happens, or I get one update. I have not gotten more than one update when this happens), using 100% CPU. It has to be kill(1)ed, after which the terminal needs to be reset in order to echo properly (which may suggest that some screen-updating-loop is running indefinitely for some reason, leaving the terminal in a weird state when killed). It seems pretty random. I have experienced several freezes in a row, or none for several runs and everything in between. A race, perhaps? I am running this on Linux 2.6.18-6-vserver-amd64 (it's a virtual server, yes, running on a Debian physical amd64). -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 2 22:22:28 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 02 Dec 2009 22:22:28 -0000 Subject: [Varnish] #525: insane number of vbe_conn structs In-Reply-To: <051.57ac8b981935811a2f2e039ef0763cb9@projects.linpro.no> References: <051.57ac8b981935811a2f2e039ef0763cb9@projects.linpro.no> Message-ID: <060.2df9b23d9ac26be32ad6c1c2afb18c48@projects.linpro.no> #525: insane number of vbe_conn structs --------------------+------------------------------------------------------- Reporter: drmac | Owner: Type: defect | Status: closed Priority: normal | Milestone: Component: build | Version: 2.0 Severity: normal | Resolution: invalid Keywords: | --------------------+------------------------------------------------------- Changes (by tfheen): * status: new => closed * resolution: => invalid Comment: The vbe_conns counter is unlocked, so it will sometimes underflow (from 0 to 2^64-1 or -5 or so). This is perfectly normal and nothing to worry about. We don't want to add locks to this as it would slow varnish slightly down. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 09:09:30 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 09:09:30 -0000 Subject: [Varnish] #574: [2.0.5] Comparing two headers In-Reply-To: <051.d599e745e9d07ae7dacdc9310dbe5e5b@projects.linpro.no> References: <051.d599e745e9d07ae7dacdc9310dbe5e5b@projects.linpro.no> Message-ID: <060.9e2651e3560f4a3b8b5044fc31280344@projects.linpro.no> #574: [2.0.5] Comparing two headers -------------------------+-------------------------------------------------- Reporter: mikko | Owner: tfheen Type: enhancement | Status: closed Priority: normal | Milestone: Component: varnishd | Version: 2.0 Severity: normal | Resolution: fixed Keywords: | -------------------------+-------------------------------------------------- Changes (by tfheen): * status: new => closed * resolution: => fixed Comment: This was fixed in r4304 which is in 2.0.5. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 09:15:03 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 09:15:03 -0000 Subject: [Varnish] #464: Varnish won't build on solaris In-Reply-To: <055.a00768cebb44c580e3934ceafb255b7a@projects.linpro.no> References: <055.a00768cebb44c580e3934ceafb255b7a@projects.linpro.no> Message-ID: <064.30a9710c4ff5c40cc3cbe1a12bfe9050@projects.linpro.no> #464: Varnish won't build on solaris -----------------------+---------------------------------------------------- Reporter: joevandyk | Owner: tfheen Type: defect | Status: closed Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Resolution: fixed Keywords: | -----------------------+---------------------------------------------------- Changes (by tfheen): * status: new => closed * resolution: => fixed Comment: This should be fixed with 2.0.5. We require C99 which defines isfinite, and the NAN, __BEGIN_DECLS and __END_DECLS are gone. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 09:18:05 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 09:18:05 -0000 Subject: [Varnish] #585: ESI produces wrong backend requests (2.0.5) In-Reply-To: <053.0e4d27e966162965c92c334184ab9afc@projects.linpro.no> References: <053.0e4d27e966162965c92c334184ab9afc@projects.linpro.no> Message-ID: <062.f90929be5512eb7c1ba70f14d291ce0a@projects.linpro.no> #585: ESI produces wrong backend requests (2.0.5) ----------------------+----------------------------------------------------- Reporter: ttmails | Owner: tfheen Type: defect | Status: closed Priority: normal | Milestone: Component: varnishd | Version: 2.0 Severity: normal | Resolution: fixed Keywords: | ----------------------+----------------------------------------------------- Changes (by tfheen): * status: new => closed * resolution: => fixed Comment: (In [4379]) Merge bits of r4351 to fix off-by-one error in ESI parsing in 2.0.5 Fixes #585 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 10:02:59 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 10:02:59 -0000 Subject: [Varnish] #343: Varnish crashes frequently with restart in vcl_fetch In-Reply-To: <050.2e50ed37b95b065838f8bf3e8675abf9@projects.linpro.no> References: <050.2e50ed37b95b065838f8bf3e8675abf9@projects.linpro.no> Message-ID: <059.c596fb7e5ea8a39ae9275769d31e8b82@projects.linpro.no> #343: Varnish crashes frequently with restart in vcl_fetch ----------------------+----------------------------------------------------- Reporter: noah | Owner: phk Type: defect | Status: closed Priority: normal | Milestone: Component: varnishd | Version: 2.0 Severity: normal | Resolution: fixed Keywords: | ----------------------+----------------------------------------------------- Changes (by tfheen): * status: new => closed * resolution: => fixed Comment: No response from submitter, so closing this bug. Please reopen if you are able to reproduce this with a newer varnish version. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 10:07:47 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 10:07:47 -0000 Subject: [Varnish] #217: varnishtop freeze In-Reply-To: <049.a5bcd9b08e2b26a91152e804621bfcd7@projects.linpro.no> References: <049.a5bcd9b08e2b26a91152e804621bfcd7@projects.linpro.no> Message-ID: <058.d7a31fa1f89cab6970910b2acdca40cc@projects.linpro.no> #217: varnishtop freeze ------------------------+--------------------------------------------------- Reporter: akv | Owner: phk Type: defect | Status: reopened Priority: normal | Milestone: Component: varnishtop | Version: trunk Severity: normal | Resolution: Keywords: | ------------------------+--------------------------------------------------- Comment (by tfheen): If you can get us a backtrace when it locks up, that would be most appreciated. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 10:45:10 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 10:45:10 -0000 Subject: [Varnish] #559: Missing varnishd options in documentation In-Reply-To: <054.d95de5af4b2fe8fb6aa84ac6634632f1@projects.linpro.no> References: <054.d95de5af4b2fe8fb6aa84ac6634632f1@projects.linpro.no> Message-ID: <063.20d63994758bd52ac3d1982a0ac8cc05@projects.linpro.no> #559: Missing varnishd options in documentation ---------------------------+------------------------------------------------ Reporter: walraven | Owner: Type: defect | Status: closed Priority: normal | Milestone: Component: documentation | Version: trunk Severity: normal | Resolution: fixed Keywords: | ---------------------------+------------------------------------------------ Comment (by tfheen): (In [4380]) Merge r4354: Document -C option in usage. Fixes #559 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 11:02:47 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 11:02:47 -0000 Subject: [Varnish] #584: Backend probing half-close issue In-Reply-To: <049.6294e4c2987b9bdaaa94b2de81cedc73@projects.linpro.no> References: <049.6294e4c2987b9bdaaa94b2de81cedc73@projects.linpro.no> Message-ID: <058.6e5e9f040c2f5b064e510b24ec4a2cb9@projects.linpro.no> #584: Backend probing half-close issue ----------------------+----------------------------------------------------- Reporter: phk | Owner: phk Type: defect | Status: closed Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: fixed Keywords: | ----------------------+----------------------------------------------------- Comment (by tfheen): (In [4381]) Merge r4356, r4357: Don't half-close probes, add .expected_response r4356: Don't halfclose the backend polling TCP connection after sending the request, some backends gets confused by this. Add a ".status" to backend polling, to configure the expected HTTP status code for a good poll. Fixes #584 r4357: .expected_response is a better idea than .status Tip o' the hat to: tfheen -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 11:14:34 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 11:14:34 -0000 Subject: [Varnish] #593: Add VCL syntax check option In-Reply-To: <053.9b98b16d697afce6b64a60f696e3af9c@projects.linpro.no> References: <053.9b98b16d697afce6b64a60f696e3af9c@projects.linpro.no> Message-ID: <062.c94834dae8f6685a3b0ea2d8a9907d37@projects.linpro.no> #593: Add VCL syntax check option -------------------------+-------------------------------------------------- Reporter: mperham | Owner: Type: enhancement | Status: closed Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Resolution: worksforme Keywords: | -------------------------+-------------------------------------------------- Changes (by phk): * status: new => closed * resolution: => worksforme Comment: -C already does that. It spits out the compiled C-code, but only if the VCl code can be compiled. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 11:26:54 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 11:26:54 -0000 Subject: [Varnish] #591: Extra .Ed causes vcl(7) to be rendered incorrectly In-Reply-To: <051.39f66d59f7216559410a389214027784@projects.linpro.no> References: <051.39f66d59f7216559410a389214027784@projects.linpro.no> Message-ID: <060.9f00627ad29b8ec818ba90335cde1047@projects.linpro.no> #591: Extra .Ed causes vcl(7) to be rendered incorrectly ---------------------------+------------------------------------------------ Reporter: fgsch | Owner: Type: defect | Status: closed Priority: normal | Milestone: Component: documentation | Version: trunk Severity: minor | Resolution: fixed Keywords: | ---------------------------+------------------------------------------------ Changes (by phk): * status: new => closed * resolution: => fixed Comment: (In [4383]) Remove spurious .Ed Fixes #591 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 11:28:24 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 11:28:24 -0000 Subject: =?utf-8?b?UmU6IFtWYXJuaXNoXSAjNTk0OiBFU0kgZnJvbSBBcGFjaGUyIGJh?= =?utf-8?b?Y2tlbmQgd2l0aCBtb2RfZGVmbGF0ZSByZXR1cm5zIE5vIEVTSSBw?= =?utf-8?b?cm9jZXNzaW5nLCBmaXJzdCBjaGFyIG5vdCDigJg8ICc=?= In-Reply-To: <050.308fb7601729eace2249e2b04bf1efa8@projects.linpro.no> References: <050.308fb7601729eace2249e2b04bf1efa8@projects.linpro.no> Message-ID: <059.d0985509862232cf7713f78df5a0c042@projects.linpro.no> #594: ESI from Apache2 backend with mod_deflate returns No ESI processing, first char not ?< ' ----------------------+----------------------------------------------------- Reporter: cd34 | Owner: phk Type: defect | Status: closed Priority: normal | Milestone: Component: varnishd | Version: 2.0 Severity: normal | Resolution: wontfix Keywords: | ----------------------+----------------------------------------------------- Changes (by phk): * status: new => closed * resolution: => wontfix Comment: Varnish does not support uncompressing content from the backend for ESI processing. Disabling mod_deflate is the right thing to do. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 11:46:44 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 11:46:44 -0000 Subject: [Varnish] #590: Assert error in ESI_Parse(), cache_esi.c line 822: Condition(st->len < st->space) not true. In-Reply-To: <050.18b8c0fcb73b96bb78d3b132f35e6485@projects.linpro.no> References: <050.18b8c0fcb73b96bb78d3b132f35e6485@projects.linpro.no> Message-ID: <059.1a222e67020c6c636be5a8173961ad0c@projects.linpro.no> #590: Assert error in ESI_Parse(), cache_esi.c line 822: Condition(st->len < st->space) not true. ------------------------------+--------------------------------------------- Reporter: kali | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: major | Resolution: Keywords: esi parser crash | ------------------------------+--------------------------------------------- Comment (by phk): Uhm which assert fails ? Do you have a backtrace ? -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 11:54:52 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 11:54:52 -0000 Subject: [Varnish] #547: Varnish core dumps frequently In-Reply-To: <053.bb3ab0292d73880f2e8477a3aa3e3c6f@projects.linpro.no> References: <053.bb3ab0292d73880f2e8477a3aa3e3c6f@projects.linpro.no> Message-ID: <062.7d51444ed29fbd377e0da9649c6b43d6@projects.linpro.no> #547: Varnish core dumps frequently ---------------------+------------------------------------------------------ Reporter: victori | Owner: Type: defect | Status: closed Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Resolution: fixed Keywords: | ---------------------+------------------------------------------------------ Changes (by phk): * status: new => closed * resolution: => fixed Comment: (In [4384]) Don't panic in TCP_linger() if the other end closed on us. Fixes #547 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 3 17:56:42 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 03 Dec 2009 17:56:42 -0000 Subject: [Varnish] #590: Assert error in ESI_Parse(), cache_esi.c line 822: Condition(st->len < st->space) not true. In-Reply-To: <050.18b8c0fcb73b96bb78d3b132f35e6485@projects.linpro.no> References: <050.18b8c0fcb73b96bb78d3b132f35e6485@projects.linpro.no> Message-ID: <059.f197c47e3c34d801e697a50f6b2363fd@projects.linpro.no> #590: Assert error in ESI_Parse(), cache_esi.c line 822: Condition(st->len < st->space) not true. ------------------------------+--------------------------------------------- Reporter: kali | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: major | Resolution: Keywords: esi parser crash | ------------------------------+--------------------------------------------- Comment (by kali): I can provide the file that triggers the crash every time it is ESI_parse()d, but I'd prefer not to attach it here. Please ask if you want it emailed somewhere. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Dec 4 09:21:23 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 04 Dec 2009 09:21:23 -0000 Subject: [Varnish] #595: segfault in 2.0.5 Message-ID: <060.071ff4a8be3f869dbe7ea4923288a338@projects.linpro.no> #595: segfault in 2.0.5 ---------------------------------+------------------------------------------ Reporter: maheshollalwar | Type: defect Status: new | Priority: high Milestone: Varnish 2.1 release | Component: build Version: 2.0 | Severity: major Keywords: | ---------------------------------+------------------------------------------ Hi, varnishd restarts frequently with segfault. Below is the error log Dec 4 14:06:54 cdnbt4 kernel: varnishd[48308]: segfault at 00000004b1400000 rip 00002b8d9820bb6a rsp 0000000055f878e0 error 4 Dec 4 08:36:54 cdnbt4 varnishd[43836]: Child (47311) died signal=11 Dec 4 08:36:54 cdnbt4 varnishd[43836]: child (48820) Started Dec 4 08:36:54 cdnbt4 varnishd[43836]: Child (48820) said Closed fds: 4 5 6 10 11 13 14 Dec 4 08:36:54 cdnbt4 varnishd[43836]: Child (48820) said Child starts Dec 4 08:36:54 cdnbt4 varnishd[43836]: Child (48820) said managed to mmap 64424509440 bytes of 64424509440 Dec 4 08:36:54 cdnbt4 varnishd[43836]: Child (48820) said Ready Thanks. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Dec 4 10:20:39 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 04 Dec 2009 10:20:39 -0000 Subject: [Varnish] #596: Grace doesn't work with directors Message-ID: <054.139989b819dddd6218854e8e65384337@projects.linpro.no> #596: Grace doesn't work with directors ----------------------+----------------------------------------------------- Reporter: zviratko | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 2.0 Severity: normal | Keywords: ----------------------+----------------------------------------------------- Grace doesn't kick in when the backend is a director. Director is probably always considered healthy. Suggestion: consider director sick when all its backends are sick. Sorry if it is fixed in trunk, my version is varnish-2.0.4-5~bpo50+1 (debian lenny with backports) -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Dec 4 14:38:42 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 04 Dec 2009 14:38:42 -0000 Subject: [Varnish] #597: Update VCLExampleRemovingSomeCookies Message-ID: <054.acb22eb98659883402070d542dc800c4@projects.linpro.no> #597: Update VCLExampleRemovingSomeCookies ----------------------+----------------------------------------------------- Reporter: zviratko | Type: documentation Status: new | Priority: normal Milestone: | Component: documentation Version: trunk | Severity: normal Keywords: | ----------------------+----------------------------------------------------- IMO a better version: cookies "foo" and "bar" are removed if (req.http.Cookie) { set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *[foo|bar]=[^;]+;? *", "\1"); set req.http.Cookie = regsub(req.http.Cookie, "[; ]*$", ""); if (req.http.Cookie == "") { remove req.http.Cookie; } } -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Dec 7 13:56:10 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 07 Dec 2009 13:56:10 -0000 Subject: [Varnish] #596: Grace doesn't work with directors In-Reply-To: <054.139989b819dddd6218854e8e65384337@projects.linpro.no> References: <054.139989b819dddd6218854e8e65384337@projects.linpro.no> Message-ID: <063.d72e78e47b776e5a2ff2a49585741581@projects.linpro.no> #596: Grace doesn't work with directors ----------------------+----------------------------------------------------- Reporter: zviratko | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 2.0 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by zviratko): I am sorry, PEBKAC... please close/delete/destroy this ticket... -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Dec 7 14:32:46 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 07 Dec 2009 14:32:46 -0000 Subject: [Varnish] #596: Grace doesn't work with directors In-Reply-To: <054.139989b819dddd6218854e8e65384337@projects.linpro.no> References: <054.139989b819dddd6218854e8e65384337@projects.linpro.no> Message-ID: <063.130caaede8bb830a8e6198e2b1e7d295@projects.linpro.no> #596: Grace doesn't work with directors ----------------------+----------------------------------------------------- Reporter: zviratko | Owner: phk Type: defect | Status: closed Priority: normal | Milestone: Component: varnishd | Version: 2.0 Severity: normal | Resolution: worksforme Keywords: | ----------------------+----------------------------------------------------- Changes (by kristian): * status: new => closed * resolution: => worksforme -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 8 10:01:42 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 08 Dec 2009 10:01:42 -0000 Subject: [Varnish] #598: SIGSEGV due to null pointer dereference in SES_Delete - VSL Message-ID: <051.df1099a9170809cedd00f10db1ac3f3c@projects.linpro.no> #598: SIGSEGV due to null pointer dereference in SES_Delete - VSL -------------------+-------------------------------------------------------- Reporter: slink | Type: defect Status: new | Priority: normal Milestone: | Component: build Version: trunk | Severity: normal Keywords: | -------------------+-------------------------------------------------------- On most Solaris Versions except very recent ones, *printf don't check for null %s arguments. When WRK_QueueSession calls SES_Delete when WRK_Queue has failed (which is the case when the chosen pool's queue is full), varnish will crash on Solaris: > ::stack libc.so.1`strlen+0x40() libc.so.1`vsnprintf+0x51() VSL+0x285() SES_Delete+0x2f3() WRK_QueueSession+0x87() vca_acct+0x44e() libc.so.1`_thr_setup+0x5b() libc.so.1`_lwp_start() > ::status debugging core file of varnishd (64-bit) from BTO1W02CAS0S file: /coremedia/cache/varnish/sbin/varnishd initial argv: /coremedia/cache/varnish/sbin/varnishd -a 0.0.0.0:80 -T localhost:6082 -p rush_ threading model: multi-threaded status: process terminated by SIGSEGV (Segmentation Fault) > ::regs %rax = 0x0000000000000020 %r8 = 0x0000000000000000 %rbx = 0x0000000000000000 %r9 = 0x0000000000449148 %rcx = 0x0000000000000073 %r10 = 0x0000000000000073 %rdx = 0xfffffd7ff2611d18 %r11 = 0x0000000000000246 %rsi = 0x0000000000000000 %r12 = 0x0000000000000000 %rdi = 0x0000000000000000 %r13 = 0x000000000044914a %r14 = 0x0000000000000000 %r15 = 0x0000000000000000 %cs = 0x004b %fs = 0x01bb %gs = 0x0000 %ds = 0x0043 %es = 0x0043 %ss = 0x0043 %rip = 0xfffffd7fff054b70 libc.so.1`strlen+0x40 %rbp = 0xfffffd7ff2611bf0 %rsp = 0xfffffd7ff2610cb8 %rflags = 0x00010246 id=0 vip=0 vif=0 ac=0 vm=0 rf=1 nt=0 iopl=0x0 status= -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 8 10:04:01 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 08 Dec 2009 10:04:01 -0000 Subject: [Varnish] #598: SIGSEGV due to null pointer dereference in SES_Delete - VSL In-Reply-To: <051.df1099a9170809cedd00f10db1ac3f3c@projects.linpro.no> References: <051.df1099a9170809cedd00f10db1ac3f3c@projects.linpro.no> Message-ID: <060.d045951dfa7360f92d7b5b99d75b31b9@projects.linpro.no> #598: SIGSEGV due to null pointer dereference in SES_Delete - VSL --------------------+------------------------------------------------------- Reporter: slink | Owner: Type: defect | Status: new Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Resolution: Keywords: | --------------------+------------------------------------------------------- Comment (by slink): Sorry for having missed wiki formatting > ::stack libc.so.1`strlen+0x40() libc.so.1`vsnprintf+0x51() VSL+0x285() SES_Delete+0x2f3() WRK_QueueSession+0x87() vca_acct+0x44e() libc.so.1`_thr_setup+0x5b() libc.so.1`_lwp_start() > ::status debugging core file of varnishd (64-bit) from XXX file: /coremedia/cache/varnish/sbin/varnishd initial argv: /coremedia/cache/varnish/sbin/varnishd -a 0.0.0.0:80 -T localhost:6082 -p rush_ threading model: multi-threaded status: process terminated by SIGSEGV (Segmentation Fault) > ::regs %rax = 0x0000000000000020 %r8 = 0x0000000000000000 %rbx = 0x0000000000000000 %r9 = 0x0000000000449148 %rcx = 0x0000000000000073 %r10 = 0x0000000000000073 %rdx = 0xfffffd7ff2611d18 %r11 = 0x0000000000000246 %rsi = 0x0000000000000000 %r12 = 0x0000000000000000 %rdi = 0x0000000000000000 %r13 = 0x000000000044914a %r14 = 0x0000000000000000 %r15 = 0x0000000000000000 %cs = 0x004b %fs = 0x01bb %gs = 0x0000 %ds = 0x0043 %es = 0x0043 %ss = 0x0043 %rip = 0xfffffd7fff054b70 libc.so.1`strlen+0x40 %rbp = 0xfffffd7ff2611bf0 %rsp = 0xfffffd7ff2610cb8 %rflags = 0x00010246 id=0 vip=0 vif=0 ac=0 vm=0 rf=1 nt=0 iopl=0x0 status= %gsbase = 0xffffffff80000000 %fsbase = 0xfffffd7ffeecea00 %trapno = 0xe %err = 0x4 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 8 10:04:38 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 08 Dec 2009 10:04:38 -0000 Subject: [Varnish] #598: SIGSEGV due to null pointer dereference in SES_Delete - VSL In-Reply-To: <051.df1099a9170809cedd00f10db1ac3f3c@projects.linpro.no> References: <051.df1099a9170809cedd00f10db1ac3f3c@projects.linpro.no> Message-ID: <060.7894af9f8f42e8d2446f829a7d277c98@projects.linpro.no> #598: SIGSEGV due to null pointer dereference in SES_Delete - VSL --------------------+------------------------------------------------------- Reporter: slink | Owner: Type: defect | Status: new Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Resolution: Keywords: | --------------------+------------------------------------------------------- Comment (by slink): (sorry was not working as I expected again. won't bother, I guess you'll get it anyway) -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 8 10:11:02 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 08 Dec 2009 10:11:02 -0000 Subject: [Varnish] #100: Setting the TTL does not adjust Expires: header In-Reply-To: <049.f779265ea82196cf4acaa40d47ee6ca5@projects.linpro.no> References: <049.f779265ea82196cf4acaa40d47ee6ca5@projects.linpro.no> Message-ID: <058.024aecf142c14e2de262fdf5b80c3c67@projects.linpro.no> #100: Setting the TTL does not adjust Expires: header ----------------------+----------------------------------------------------- Reporter: des | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by slink): I'd suggest this bug should be obsoleted by RFE 225 http://varnish.projects.linpro.no/ticket/225 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 8 11:45:54 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 08 Dec 2009 11:45:54 -0000 Subject: [Varnish] #599: WRK_Queue should prefer thread pools with idle threads / improve thread pool loadbalancing Message-ID: <051.42995626b1efb20bdc787f921a085f93@projects.linpro.no> #599: WRK_Queue should prefer thread pools with idle threads / improve thread pool loadbalancing -------------------------+-------------------------------------------------- Reporter: slink | Owner: phk Type: enhancement | Status: new Priority: high | Milestone: Component: varnishd | Version: trunk Severity: normal | Keywords: -------------------------+-------------------------------------------------- The algorithm implemented in WRK_Queue basically was so far: * Choose a worker pool round robin * Dispatch the request on that pool * find an idle thread OR * put the request on a queue OR * fail of the queue is full (reached ovfl_max) This algorithm is probably good enough for many cases, but I noticed that it can have a negative impact in particular during startup. Threads for the pools are created sequentially (in wrk_herder_thread), so shortly after startup, some pools may get hit by requests when they don't have any threads yet. I noticed this because overflowing pools would trigger the issue documented in #598. Here's a snapshot of this situation in Solaris mdb: {{{ > 0x0000000000464ee0/D varnishd`nwq: varnishd`nwq: 4 > wq/p varnishd`wq: varnishd`wq: 0x483f50 ## w'queues > 0x483f50,4/p 0x483f50: 0x507160 0x506ed0 0x506f30 0x506f90 struct wq { 82 unsigned magic; 83 #define WQ_MAGIC 0x606658fa 84 struct lock mtx; 85 struct workerhead idle; 86 VTAILQ_HEAD(, workreq) overflow; 87 unsigned nthr; 88 unsigned nqueue; 89 unsigned lqueue; 90 uintmax_t ndrop; 91 uintmax_t noverflow; 92 }; > 0x507160,60::dump -e 507160: 606658fa 00000000 005359a0 00000000 507170: c3c3fe00 fffffd7f f03d0e30 fffffd7f 507180: 00000000 00000000 00507180 00000000 507190: 00000177 00000000 00000000 00000000 177 thr 5071a0: 00000000 00000000 00000051 00000000 51 overflow 5071b0: 00507150 00000000 00000000 00000000 > 0x506ed0,60::dump -e 506ed0: 606658fa 00000000 005359f0 00000000 506ee0: b9d6ee00 fffffd7f c1a38e30 fffffd7f 506ef0: 00000000 00000000 00506ef0 00000000 506f00: 00000050 00000000 00000000 00000000 50 thr 506f10: 00000000 00000000 000001cf 00000000 1cf noverflow 506f20: 00000051 00000000 00000000 00000000 > 0x506f30,60::dump -e 506f30: 606658fa 00000000 00535a40 00000000 506f40: 00000000 00000000 00506f40 00000000 506f50: 007b65e8 00000000 0292e778 00000000 506f60: 00000000 00000201 00000000 00000000 0 thr 201 nqueue 506f70: 00000001 00000000 00000201 00000000 1 drop 201 noverflow 506f80: 00000061 00000000 00000000 00000000 > 0x506f90,60::dump -e 506f90: 606658fa 00000000 00535a90 00000000 506fa0: 00000000 00000000 00506fa0 00000000 506fb0: 007baf08 00000000 0285e218 00000000 506fc0: 00000000 00000201 00000000 00000000 0 thr 201 nqueue 506fd0: 00000000 00000000 00000201 00000000 201 noverflow 506fe0: 00506f80 00000000 00000000 00000000 }}} Notice that {{{wq[2]}}} and {{{wq[3]}}} have their nqueues saturated and no idle threads while {{{wq[0]}}} and {{{wq[1]}}} probably have idle threads by now. I am suggesting the following changes to WRK_Queue: * Improve the round-robin selection on MP systems by using a volatile static (still avoiding additional locking overhead for the round robin state) * First check all pools for idle threads (starting with the pool selected by round-robin to remain in O(1) for the normal case) * Only queue a request if there exists no pool with idle threads, and queue where the queue is shortest * Fail only if all queues are full I'll attach a diff with my suggested solution. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 9 12:36:00 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 09 Dec 2009 12:36:00 -0000 Subject: [Varnish] #582: Varhish crash with no error code In-Reply-To: <054.6288692149fe22877a0c7692d2babcec@projects.linpro.no> References: <054.6288692149fe22877a0c7692d2babcec@projects.linpro.no> Message-ID: <063.5fff1e2aa0491110a6b27f46d78871fa@projects.linpro.no> #582: Varhish crash with no error code ------------------------------------------------------------------------------------------+ Reporter: doserror | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd (varnish-2.0.4) Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS | ------------------------------------------------------------------------------------------+ Comment (by doserror): Hello, Yes, it does serve traffic. The proxy is in production for those sites : http://www.tv2east.dk/ http://www.tvmidtvest.dk/ , and the problem seems to be more complex. Even though the child process crashes the proxy continue its execution, but the cache is gone. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 9 12:43:42 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 09 Dec 2009 12:43:42 -0000 Subject: [Varnish] #582: Varhish crash with no error code In-Reply-To: <054.6288692149fe22877a0c7692d2babcec@projects.linpro.no> References: <054.6288692149fe22877a0c7692d2babcec@projects.linpro.no> Message-ID: <063.2e155fbe7c0ca18cdbe5bd7cd8fc102b@projects.linpro.no> #582: Varhish crash with no error code ------------------------------------------------------------------------------------------+ Reporter: doserror | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: varnishd (varnish-2.0.4) Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS | ------------------------------------------------------------------------------------------+ Comment (by doserror): Also there are no core dumps. But I will try running it in debug mode soon. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 9 13:14:16 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 09 Dec 2009 13:14:16 -0000 Subject: [Varnish] #600: error in man 7 vcl Message-ID: <054.e30f20b95ab9ac98a1fc2ed91b795c36@projects.linpro.no> #600: error in man 7 vcl ----------------------+----------------------------------------------------- Reporter: zviratko | Type: defect Status: new | Priority: normal Milestone: | Component: documentation Version: 2.0 | Severity: normal Keywords: | ----------------------+----------------------------------------------------- man vcl states: Use req.hash += req.http.Set-Cookie or similar to include the Set-Cookie HTTP header in the hash string should read: Use req.hash += req.http.Cookie or similar to include the Cookie HTTP header in the hash string -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 9 13:16:45 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 09 Dec 2009 13:16:45 -0000 Subject: [Varnish] #600: error in man 7 vcl In-Reply-To: <054.e30f20b95ab9ac98a1fc2ed91b795c36@projects.linpro.no> References: <054.e30f20b95ab9ac98a1fc2ed91b795c36@projects.linpro.no> Message-ID: <063.9dd6fd05c26b9a124001353ed11c0287@projects.linpro.no> #600: error in man 7 vcl ---------------------------+------------------------------------------------ Reporter: zviratko | Owner: Type: defect | Status: new Priority: normal | Milestone: Component: documentation | Version: 2.0 Severity: normal | Resolution: Keywords: | ---------------------------+------------------------------------------------ Comment (by zviratko): while there: sub vcl_fetch { if (obj.http.Set-Cookie) { insert; } should probably be sub vcl_fetch { if (obj.http.Set-Cookie) { deliver; } not to confuse anyone... -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Dec 14 08:10:58 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 14 Dec 2009 08:10:58 -0000 Subject: [Varnish] #600: error in man 7 vcl In-Reply-To: <054.e30f20b95ab9ac98a1fc2ed91b795c36@projects.linpro.no> References: <054.e30f20b95ab9ac98a1fc2ed91b795c36@projects.linpro.no> Message-ID: <063.1686b5d9fe9dd24b99dd730f60b98f12@projects.linpro.no> #600: error in man 7 vcl ---------------------------+------------------------------------------------ Reporter: zviratko | Owner: Type: defect | Status: closed Priority: normal | Milestone: Component: documentation | Version: 2.0 Severity: normal | Resolution: fixed Keywords: | ---------------------------+------------------------------------------------ Changes (by tfheen): * status: new => closed * resolution: => fixed Comment: (In [4385]) Fix Set-Cookie vs Cookie confusion and old keywords. Fixes #600 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Dec 14 14:43:51 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 14 Dec 2009 14:43:51 -0000 Subject: [Varnish] #601: adding the X-Forwarded-For Header should be optional Message-ID: <050.553044ee4ef2849ec7a5fff487fbf81e@projects.linpro.no> #601: adding the X-Forwarded-For Header should be optional -------------------------+-------------------------------------------------- Reporter: Elfe | Owner: phk Type: enhancement | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Keywords: -------------------------+-------------------------------------------------- At the moment an X-Forwarded-For Header is added no matter what. It would be nice to be able to disable the extra header that is inserted by Varnish. This would decrease the issues when deploying Varnish to existing setups. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 16 08:59:42 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 16 Dec 2009 08:59:42 -0000 Subject: [Varnish] #577: 64bit Catch 22 on Solaris (gcc _builtin_xxx functions) In-Reply-To: <054.ec05b545ae9e3cb7a5d7d7f3ddfd3c8d@projects.linpro.no> References: <054.ec05b545ae9e3cb7a5d7d7f3ddfd3c8d@projects.linpro.no> Message-ID: <063.52aebd25a4f21c62e69ef9e86e163611@projects.linpro.no> #577: 64bit Catch 22 on Solaris (gcc _builtin_xxx functions) ----------------------+----------------------------------------------------- Reporter: whocares | Owner: Type: defect | Status: closed Priority: normal | Milestone: Component: build | Version: 2.0 Severity: normal | Resolution: fixed Keywords: | ----------------------+----------------------------------------------------- Comment (by tfheen): (In [4399]) Merge r4349: Hide GCC specific backtrace() compat function under a #ifdef. We do not want to be dependent on GCC. Fixes #577 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 16 09:55:39 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 16 Dec 2009 09:55:39 -0000 Subject: [Varnish] #572: Fix for: Create worker thread failed 11 Resource temporarily unavailable (Thread Problem on Linux) In-Reply-To: <054.6c0a8b1caae77bb470ba56faa0e87948@projects.linpro.no> References: <054.6c0a8b1caae77bb470ba56faa0e87948@projects.linpro.no> Message-ID: <063.645463d164f52baa03e4d14670748e5b@projects.linpro.no> #572: Fix for: Create worker thread failed 11 Resource temporarily unavailable (Thread Problem on Linux) -------------------------+-------------------------------------------------- Reporter: whocares | Owner: phk Type: enhancement | Status: closed Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: fixed Keywords: | -------------------------+-------------------------------------------------- Comment (by tfheen): (In [4402]) Merge r4352: Add a parameter to set the workerthread stacksize. On 32 bit systems, it may be necessary to tweak this down to get high numbers of worker threads squeezed into the address-space. I have no idea how much stack-space a worker thread normally uses, so no guidance is given, and we default to the system default. Fixes #572 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 16 11:05:07 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 16 Dec 2009 11:05:07 -0000 Subject: [Varnish] #591: Extra .Ed causes vcl(7) to be rendered incorrectly In-Reply-To: <051.39f66d59f7216559410a389214027784@projects.linpro.no> References: <051.39f66d59f7216559410a389214027784@projects.linpro.no> Message-ID: <060.ac30d4d63630b146a97468f0308f3a21@projects.linpro.no> #591: Extra .Ed causes vcl(7) to be rendered incorrectly ---------------------------+------------------------------------------------ Reporter: fgsch | Owner: Type: defect | Status: closed Priority: normal | Milestone: Component: documentation | Version: trunk Severity: minor | Resolution: fixed Keywords: | ---------------------------+------------------------------------------------ Comment (by tfheen): (In [4407]) Merge r4383: Remove spurious .Ed Fixes #591 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 16 11:14:47 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 16 Dec 2009 11:14:47 -0000 Subject: [Varnish] #547: Varnish core dumps frequently In-Reply-To: <053.bb3ab0292d73880f2e8477a3aa3e3c6f@projects.linpro.no> References: <053.bb3ab0292d73880f2e8477a3aa3e3c6f@projects.linpro.no> Message-ID: <062.cc0ac37a35a986307c60133fdb3e7436@projects.linpro.no> #547: Varnish core dumps frequently ---------------------+------------------------------------------------------ Reporter: victori | Owner: Type: defect | Status: closed Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Resolution: fixed Keywords: | ---------------------+------------------------------------------------------ Comment (by tfheen): (In [4408]) Merge r4384: Don't panic in TCP_linger() if the other end closed on us. Fixes #547 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 16 11:21:18 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 16 Dec 2009 11:21:18 -0000 Subject: [Varnish] #600: error in man 7 vcl In-Reply-To: <054.e30f20b95ab9ac98a1fc2ed91b795c36@projects.linpro.no> References: <054.e30f20b95ab9ac98a1fc2ed91b795c36@projects.linpro.no> Message-ID: <063.99800f300cccf0ade44aa6a3b779d43d@projects.linpro.no> #600: error in man 7 vcl ---------------------------+------------------------------------------------ Reporter: zviratko | Owner: Type: defect | Status: closed Priority: normal | Milestone: Component: documentation | Version: 2.0 Severity: normal | Resolution: fixed Keywords: | ---------------------------+------------------------------------------------ Comment (by tfheen): (In [4409]) Merge r4385: Fix Set-Cookie vs Cookie confusion and old keywords. Fixes #600 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Wed Dec 16 20:03:52 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Wed, 16 Dec 2009 20:03:52 -0000 Subject: [Varnish] #602: server.hostname and server.identity are not documented in vcl(7) Message-ID: <051.2c451a342587525e715b0d0ec03b0b1f@projects.linpro.no> #602: server.hostname and server.identity are not documented in vcl(7) -------------------+-------------------------------------------------------- Reporter: fgsch | Type: documentation Status: new | Priority: normal Milestone: | Component: documentation Version: trunk | Severity: minor Keywords: | -------------------+-------------------------------------------------------- server.hostname and server.identity are not documented in vcl(7). -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 17 17:07:35 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 17 Dec 2009 17:07:35 -0000 Subject: [Varnish] #603: admin interface gets confused (2.0.5) Message-ID: <050.d9000138163c8a83adad7c52189c3c8c@projects.linpro.no> #603: admin interface gets confused (2.0.5) ----------------------+----------------------------------------------------- Reporter: knan | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 2.0 Severity: normal | Keywords: ----------------------+----------------------------------------------------- We use automated scripts to load and discard vcls via varnishadm. After a few days uptime, one of four nodes got its admin interface confused in an interesting way - vcl.list output mixed in with help text, and the other way around, not consistently. Probably other commands as well. varnishadm randomly reports error 106 on vcl.load and vcl.discard. varnish 2.0.5 built with debian files from svn, updated debian lenny x86-64 {{{ prod31:~# telnet localhost 6082 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. help 200 449 help [command] ping [timestamp] status start stop stats vcl.load vcl.inline vcl.use vcl.discard vcl.list vcl.show param.show [-l] [] param.set quit available 29 boot available 0 prod2009-12-15-152016 available 0 prod2009-12-15-152411 available 61 prod2009-12-15-153329 active 108 prod2009-12-17-113718 vcl.list 200 110 purge.url purge.hash purge [&& ]... purge.list }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 17 18:46:39 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 17 Dec 2009 18:46:39 -0000 Subject: [Varnish] #604: n_vbe_conn corrupted Message-ID: <051.43ce56a29404685d4382fdd7e6e174e7@projects.linpro.no> #604: n_vbe_conn corrupted -------------------+-------------------------------------------------------- Reporter: fpapa | Type: defect Status: new | Priority: normal Milestone: | Component: build Version: trunk | Severity: normal Keywords: | -------------------+-------------------------------------------------------- trunk-4384. varnishstats shows 18446744069414584320 . . N struct vbe_conn (0xFFFFFFFF00000000); sometimes it's 18446744069414584321. snapshot: {{{ 2+05:21:11 xxx Hitrate ratio: 10 100 1000 Hitrate avg: 0.3909 0.4132 0.4175 927071 6.00 4.83 Client connections accepted 2023349 15.00 10.53 Client requests received 845112 5.00 4.40 Cache hits 1165479 10.00 6.07 Cache misses 1164466 9.00 6.06 Backend conn. success 1013 0.00 0.01 Backend conn. failures 1163681 10.00 6.06 Fetch with Length 570 0.00 0.00 Fetch pre HTTP/1.1 closed 9586 . . N struct sess_mem 9572 . . N struct sess 19795 . . N struct object 19809 . . N struct objectcore 19792 . . N struct objecthead 18446744069414584320 . . N struct vbe_conn 20 . . N worker threads 27 0.00 0.00 N worker threads created 94 0.00 0.00 N overflowed work requests 1 . . N backends 497918 . . N expired objects 2073 . . N LRU nuked objects 98114 . . N LRU moved objects 1939149 15.00 10.10 Objects sent with write 927071 7.00 4.83 Total Sessions 2023349 15.00 10.53 Total Requests 1164251 10.00 6.06 Total fetch 549018241 4406.74 2858.41 Total header bytes 2033215786 17440.91 10585.75 Total body bytes 34414 0.00 0.18 Session Closed 1994544 15.00 10.38 Session Linger 1991892 15.00 10.37 Session herd 116392415 876.15 605.99 SHM records 8291566 64.01 43.17 SHM writes 1284 0.00 0.01 SHM MTX contention 48 0.00 0.00 SHM cycles through buffer 1617028 13.00 8.42 SMA allocator requests 39589 . . SMA outstanding allocations 96192022 . . SMA outstanding bytes 3885145393 . . SMA bytes allocated 3788953371 . . SMA bytes free 8 0.00 0.00 SMS allocator requests 200 . . SMS bytes allocated 200 . . SMS bytes freed 1164464 9.00 6.06 Backend requests made 1 0.00 0.00 N vcl total 1 0.00 0.00 N vcl available 1 . . N total active purges 1 0.00 0.00 N new purges added }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Dec 21 05:59:58 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 21 Dec 2009 05:59:58 -0000 Subject: [Varnish] #605: Unix socket support Message-ID: <051.8e447d3e209fa5e928bb8ae0061b16e6@projects.linpro.no> #605: Unix socket support -------------------+-------------------------------------------------------- Reporter: ash2k | Type: enhancement Status: new | Priority: normal Milestone: | Component: build Version: trunk | Severity: normal Keywords: | -------------------+-------------------------------------------------------- Hello! Please add unix socket support for backend connections. It would be very useful if varnish and backend(s) are on the same box. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 24 19:24:19 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 24 Dec 2009 19:24:19 -0000 Subject: [Varnish] #606: Varnish 2.0.6 -- fix for ESI src= parsing error -- causes a "bogus" character in the URL and a potential memory clobber Message-ID: <049.070c9154bde6d4b5a3ec933268fa40fe@projects.linpro.no> #606: Varnish 2.0.6 -- fix for ESI src= parsing error -- causes a "bogus" character in the URL and a potential memory clobber -------------------+-------------------------------------------------------- Reporter: niz | Type: defect Status: new | Priority: high Milestone: | Component: build Version: trunk | Severity: normal Keywords: | -------------------+-------------------------------------------------------- Hi, I thought I would report a bug and what I believe is the fix for an ESI src= parsing error in Varnish. In cache_vrt_esi.c the terminating '\0' for the "val" part of "attr=val" is not set as the last character of the allocated string but rather the first byte beyond the allocated string. This causes an extra bogus character to remain in the string and it also clobbers the first byte beyond what was allocated. The symptom on our system was a periodic extra character in url part of the " which generated a 404 page not found which made it look like the ESI src wasn't resolved. If, however, the last byte memory was already a '\0' or a blank then it would work fine. The fix is a one line change to this code (in cache_vrt_esi.c)... if ( val.b != val.e ) { s = Tlen(val) + 1; c = WS_Alloc(ws, s); memcpy(c, val.b, Tlen(val)); val.b = c; val.e = val.b + s; /* note: s length already includes the space of '\0' */ *val.e = '\0'; /* <=== this should be "*(val.e-1) = '\0';" */ } I hope this helps. Best, /j -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Dec 25 00:02:59 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 25 Dec 2009 00:02:59 -0000 Subject: [Varnish] #607: Varnish 2.0.6 -- fixes for ESI tag parsing when xmlns:esi attribute present; plus issues with "& " evaluation in URLs Message-ID: <049.16465d721c8397862566be8a6d9abbc1@projects.linpro.no> #607: Varnish 2.0.6 -- fixes for ESI tag parsing when xmlns:esi attribute present; plus issues with "&" evaluation in URLs -------------------+-------------------------------------------------------- Reporter: niz | Type: defect Status: new | Priority: high Milestone: | Component: build Version: trunk | Severity: normal Keywords: esi | -------------------+-------------------------------------------------------- I fixed two minor problems we when varnish was parsing the following ESI tag: {{{ }}} The first problem is that Varnish will only allow alpha-numeric attribute names, so "xmlns:esi" appears to be illegal. In cache_vrt_esi.c, changing: {{{ if (!isalnum(*in->b)) { to... if (!isalnum(*in->b) && (strchr(":_-.", *in->b) == NULL)) { }}} fixes this. It also allows '_', '-' and '.' characters in the attribute name. The second problem is that the XML character entity references (i.e. "&") do not get changed in to their actual representation. So where I was expecting query parameters "?mode=id&id=12034" I get "?mode=id&id=12034". If the ESI included URL is going through varnish I can do a vcl regsub() to change "&" to "&" -- but if it is going to an external URL things won't work. I fixed this problem by processing XML character entity references in cache_vrt_esi.c -- which I think is the best solution. I have attached the diffs to cache_vrt_esi.c for both these fixes. I tested them and I believe they are safe but this is first time I've looked at this code and so I suppose I could have done something dumb. Best,[[BR]] /j -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Dec 25 00:08:29 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 25 Dec 2009 00:08:29 -0000 Subject: [Varnish] #608: Varnish 2.0.6 -- fixes for ESI tag parsing when xmlns:esi attribute present; plus issues with "& " evaluation in URLs Message-ID: <049.34aecf43a7f89853387dac29c9b47292@projects.linpro.no> #608: Varnish 2.0.6 -- fixes for ESI tag parsing when xmlns:esi attribute present; plus issues with "&" evaluation in URLs -------------------+-------------------------------------------------------- Reporter: niz | Type: defect Status: new | Priority: high Milestone: | Component: build Version: trunk | Severity: normal Keywords: esi | -------------------+-------------------------------------------------------- I fixed two minor problems we when varnish was parsing the following ESI tag: {{{ }}} The first problem is that Varnish will only allow alpha-numeric attribute names, so "xmlns:esi" appears to be illegal. In cache_vrt_esi.c, changing: {{{ if (!isalnum(*in->b)) { to... if (!isalnum(*in->b) && (strchr(":_-.", *in->b) == NULL)) { }}} fixes this. It also allows '_', '-' and '.' characters in the attribute name. The second problem is that the XML character entity references (i.e. "&") do not get changed in to their actual representation. So where I was expecting query parameters "?mode=id&id=12034" I get "?mode=id&id=12034". If the ESI included URL is going through varnish I can do a vcl regsub() to change "&" to "&" -- but if it is going to an external URL things won't work. I fixed this problem by processing XML character entity references in cache_vrt_esi.c -- which I think is the best solution. I have attached the diffs to cache_vrt_esi.c for both these fixes. I tested them and I believe they are safe but this is first time I've looked at this code and so I suppose I could have done something dumb. Best,[[BR]] /j -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Sun Dec 27 14:19:50 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Sun, 27 Dec 2009 14:19:50 -0000 Subject: [Varnish] #609: use more efficient implementation for TIM_mono on Solaris (if gethrtime() exists) Message-ID: <051.337d8b55936583877f6dd99d9c0e7293@projects.linpro.no> #609: use more efficient implementation for TIM_mono on Solaris (if gethrtime() exists) -------------------------+-------------------------------------------------- Reporter: slink | Owner: phk Type: enhancement | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Keywords: -------------------------+-------------------------------------------------- I noticed that on (Open)Solaris, clock_gettime(CLOCK_MONOTONIC, ..) falls into a syscall, while neither clock_gettime(CLOCK_REALTIME, ..) nor gethrtime() need one. References: * http://cvs.opensolaris.org/source/xref/onnv/onnv- gate/usr/src/lib/libc/amd64/sys/__clock_gettime.s#33 * http://cvs.opensolaris.org/source/s?defs=__clock_gettime&project=/onnv So, for the time being (until this is changed in (Open)Solaris), we save a systemcall by using gethrtime() for TIM_mono. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Sun Dec 27 14:24:37 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Sun, 27 Dec 2009 14:24:37 -0000 Subject: [Varnish] #609: use more efficient implementation for TIM_mono on Solaris (if gethrtime() exists) In-Reply-To: <051.337d8b55936583877f6dd99d9c0e7293@projects.linpro.no> References: <051.337d8b55936583877f6dd99d9c0e7293@projects.linpro.no> Message-ID: <060.7d8ed1c360a6d7c1c689090d35a30dde@projects.linpro.no> #609: use more efficient implementation for TIM_mono on Solaris (if gethrtime() exists) -------------------------+-------------------------------------------------- Reporter: slink | Owner: phk Type: enhancement | Status: new Priority: normal | Milestone: Component: varnishd | Version: trunk Severity: normal | Resolution: Keywords: | -------------------------+-------------------------------------------------- Comment (by slink): I should add that I've also added a simple time delta test to time.c to avoid regressions like by getting an order of magnitude wrong etc. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 29 16:44:25 2009 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 29 Dec 2009 16:44:25 -0000 Subject: [Varnish] #610: rushing too often may overflow session workspace Message-ID: <051.2c223427ba1546d13326db79182cfdb2@projects.linpro.no> #610: rushing too often may overflow session workspace ----------------------+----------------------------------------------------- Reporter: slink | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: 2.0 Severity: major | Keywords: ----------------------+----------------------------------------------------- (see also the discussion in http://projects.linpro.no/pipermail/varnish-dev/2009-December/002348.html ) When very many sessions are waiting for a busy object from a slow backend, varnish may panic in HSH_prepare with a message similar to the following {{{ varnishd[7518]: [ID 733861 local0.error] Child (8742) Panic message: Missing errorhandling code in HSH_Prepare(), cache_hash.c line 187: Condition((p) != 0) not true. errno = 32 (Broken pipe) thread = (cache-worker)sp = 10bc7038 { fd = 2322, id = 2322, xid = 441755642, client = X.X.X.X:61981, step = STP_LOOKUP, handling = hash, ws = 10bc70a8 { overflow id = "sess", {s,f,r,e} = {10bc7938,,+16340,0,+16384}, }, worker = fffffd7e5d504e00 { }, vcl = { srcname = { "input", "Default", }, }, }, }}} This panic message is, in many cases, a symptom of too small a session workspace (see also #472, #551, #560). For the issue documented here to apply, the session workspace should be inspected: {{{ (gdb) x /52x sp->ws->e-1000 0x10bcb550: 0x00000000 0x00000000 0x10bc794f 0x00000000 0x10bcb560: 0x10bc7950 0x00000000 0x10bc7bcb 0x00000000 0x10bcb570: 0x10bc7bd6 0x00000000 0x00000000 0x00000000 0x10bcb580: 0x00000000 0x00000000 0x00000000 0x00000000 0x10bcb590: 0x10bc794f 0x00000000 0x10bc7950 0x00000000 0x10bcb5a0: 0x10bc7bcb 0x00000000 0x10bc7bd6 0x00000000 0x10bcb5b0: 0x00000000 0x00000000 0x00000000 0x00000000 0x10bcb5c0: 0x00000000 0x00000000 0x10bc794f 0x00000000 0x10bcb5d0: 0x10bc7950 0x00000000 0x10bc7bcb 0x00000000 0x10bcb5e0: 0x10bc7bd6 0x00000000 0x00000000 0x00000000 0x10bcb5f0: 0x00000000 0x00000000 0x00000000 0x00000000 0x10bcb600: 0x10bc794f 0x00000000 0x10bc7950 0x00000000 0x10bcb610: 0x10bc7bcb 0x00000000 0x10bc7bd6 0x00000000 }}} If this issue applies, the workspace will be clobbered with a sequence of hash pointers: {{{ (gdb) print sp->nhashptr $8 = 6 (gdb) print *sp->hashptr $14 = 0x10bc794f "/" (gdb) print *(sp->hashptr+1) $15 = 0x10bc7950 "" (gdb) print *(sp->hashptr+2) $16 = 0x10bc7bcb "www.XXXX.de" (gdb) print *(sp->hashptr+3) $17 = 0x10bc7bd6 "" (gdb) print *(sp->hashptr+4) $18 = 0x0 (gdb) print *(sp->hashptr+5) $19 = 0x0 (gdb) print *(sp->hashptr+6) $20 = 0x0 }}} In the case, the reason for the panic is that the session repeatedly got rushed (scheduled onto a thread) while the object it was waiting for was still busy. While this should, in principle, only add additional computational overhead for checking if the object being waited for has become available in the cache, a side effect of the particular (efficient) memory management used for workspaces causes session workspace exhaustion: When we determine in cnt_lookup that the requested object is busy, we've already allocated memory from the session workspace, so if we enter cnt_lookup very often, we will eventually clobber the session workspace and panic: {{{ #!c if (sp->obj == NULL) { HSH_Prepare(sp, sp->vcl->nhashcount); VCL_hash_method(sp); assert(sp->handling == VCL_RET_HASH); } o = HSH_Lookup(sp); if (o == NULL) { /* * We lost the session to a busy object, disembark the * worker thread. The hash code to restart the session, * still in STP_LOOKUP, later when the busy object isn't. * * XXX When getting here, we have allocated space for * sp->hashptr on sp->http->ws using HSH_Prepare, which we never * return until the whole session object is destroyed. We would * want to use WS_Release on the space allocated by HSH_Prepare, * but VCL_hash_method(sp) could have allocated space and used * (refenced) it anywhere in the session object - so unless we * changed memory management into something less efficient, we * can't safely return the space. * * This imposes a practical limit on the number of restarts * */ return (1); } }}} The comment explains why I think this cant be fixed easily. I will attach a suggested fix for varnish 2.0.5 which should reduce rushes to those cases when they actually make sense. I haven't looked at the current trunk yet, but as far as I remember, the relevant functions have been changed siginificantly. Thank you for this great product, Nils -- Ticket URL: Varnish The Varnish HTTP Accelerator