From varnish-bugs at projects.linpro.no Sun Dec 2 15:36:54 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Sun, 02 Dec 2007 15:36:54 -0000 Subject: [Varnish] #181: osx leopard: unknown sendfile() implementation In-Reply-To: <053.fe4dde1a1b9387be05d3390e48119f02@projects.linpro.no> References: <053.fe4dde1a1b9387be05d3390e48119f02@projects.linpro.no> Message-ID: <062.8198663823b8cb01d6d7de5b366854eb@projects.linpro.no> #181: osx leopard: unknown sendfile() implementation ---------------------+------------------------------------------------------ Reporter: batlogg | Owner: des Type: defect | Status: new Priority: normal | Milestone: Component: build | Version: 1.1.1 Severity: normal | Resolution: Keywords: | ---------------------+------------------------------------------------------ Comment (by supergiantrobot): I need to test this fix, which I am doing now, but do get a clean build, I added the lines... {{{ #if defined(__APPLE__) #define __FreeBSD__ #endif }}} to the top of cache_pool.c. Also, sendfile() on Mac OS X is different. Line 161 of the file should be replaced with ... {{{ off_t length = len; if (sendfile(fd, *w->wfd, off, &length, &sfh, 0) != 0) }}} Also, look at the solution to Ticket #118 -- its change must be made, too, before varnishd can run. More soon... I can submit a Mac OS X-specific patch once I check things out. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 4 11:25:20 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 04 Dec 2007 11:25:20 -0000 Subject: [Varnish] #184: After a url purge the page takes about 20 sek or time out before it is viewed. Message-ID: <058.88bcec527e068e486073fafb55a2d4d2@projects.linpro.no> #184: After a url purge the page takes about 20 sek or time out before it is viewed. --------------------------+------------------------------------------------- Reporter: krkomvarnish | Owner: des Type: defect | Status: new Priority: high | Milestone: Component: build | Version: 1.1.1 Severity: major | Keywords: Purge --------------------------+------------------------------------------------- we have use a php script to purge an url in varnish. after a purge the webpage take a lot longer time before it is downloaded to the client. if we restart the service the speed increases and it is stable until a new purge. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Dec 7 14:44:41 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 07 Dec 2007 14:44:41 -0000 Subject: [Varnish] #185: Repport bytes (really) transfered into statistics Message-ID: <050.27efe119bed31e21588d4686f12eaa60@projects.linpro.no> #185: Repport bytes (really) transfered into statistics -------------------------+-------------------------------------------------- Reporter: Gwen | Owner: des Type: enhancement | Status: new Priority: normal | Milestone: Component: build | Version: trunk Severity: minor | Keywords: -------------------------+-------------------------------------------------- When a file transfert is abort the number of bytes really sent to client is not saved. For some statistics reasons, this information can be useful -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Dec 7 17:45:17 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 07 Dec 2007 17:45:17 -0000 Subject: [Varnish] #186: No logging data on quick aborts with varnishlog or varnishncsa Message-ID: <052.35a6a62e76ff9bcab197bf597959229a@projects.linpro.no> #186: No logging data on quick aborts with varnishlog or varnishncsa -------------------------+-------------------------------------------------- Reporter: rinogo | Owner: des Type: defect | Status: new Priority: normal | Milestone: Component: varnishncsa | Version: 1.1.1 Severity: normal | Keywords: -------------------------+-------------------------------------------------- "We've also noticed that if the file transfer is aborted before about 2MB is transfered, the file transfer isn't logged at all to access.log. Is this by design? For the same reasons stated in #1, we need to find a reason to log these transfers as well. On a side note, I tried using Squid (yuck) and found that it satisfied point 1, but very strangely also had the same 'problem' described in point 2. (Although the 'logging threshold' was lower, around 1MB or so) So weird." -From message to varnish-misc, 12/7/07 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Fri Dec 7 17:52:10 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Fri, 07 Dec 2007 17:52:10 -0000 Subject: [Varnish] #187: Insufficient logging data - actual bytes transferred needed Message-ID: <052.e24a3b4d3f02f7eafd5b154cbc7cbed9@projects.linpro.no> #187: Insufficient logging data - actual bytes transferred needed -------------------------+-------------------------------------------------- Reporter: rinogo | Owner: phk Type: enhancement | Status: new Priority: normal | Milestone: Component: varnishlog | Version: 1.1.1 Severity: normal | Keywords: -------------------------+-------------------------------------------------- " > varnishncsa seems to only log the size (in bytes) of the file being > transfered. We deal a lot with large files (30MB-100MB), so many of our > file transfers are cut short. We need to be able to log exactly how > many bytes were actually transfered. Is this possible? I think you can't get this information for the moment. In your case, file transfer is made by RES_WriteObj() (cache_response.c) with a call to WRK_Sendfile() (cache_pool.c). If you look into thins function : if (WRK_Flush(w) == 0 && sendfile(*w->wfd, fd, &off, len) != len) w->werr++; The number of bytes copied (returned by sendfile) is not saved :( With a little patch you can save this value and use it for statistics. Gwen" -From message to varnish-misc, 12/7/07 -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Sun Dec 9 21:56:56 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Sun, 09 Dec 2007 21:56:56 -0000 Subject: [Varnish] #78: URLS without trailing / direct to backend . In-Reply-To: <052.898d4b1c1c82444eefb273439c7b645c@projects.linpro.no> References: <052.898d4b1c1c82444eefb273439c7b645c@projects.linpro.no> Message-ID: <061.59f61a9c76b3dc67ba08eff2502f6e14@projects.linpro.no> #78: URLS without trailing / direct to backend . ----------------------+----------------------------------------------------- Reporter: kevinm | Owner: des Type: defect | Status: assigned Priority: high | Milestone: Component: varnishd | Version: Severity: major | Resolution: Keywords: | ----------------------+----------------------------------------------------- Comment (by nanor): Hi, I got the same issue.. Do you use nginx as backend http server ? regards -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Mon Dec 10 22:49:06 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Mon, 10 Dec 2007 22:49:06 -0000 Subject: [Varnish] #188: thread pileup Message-ID: <054.501398c9432dffc43debd3778be2327e@projects.linpro.no> #188: thread pileup ----------------------+----------------------------------------------------- Reporter: steinove | Owner: des Type: defect | Status: new Priority: high | Milestone: Component: build | Version: 1.1.1 Severity: normal | Keywords: ----------------------+----------------------------------------------------- On centos 4.4, 2.6.9 smp kernel, varnish 1.1.1 When number of requests gets high enough (5000-6000 req/s), suddenly threads starts piling up to the max, cpu goes to 100%, server load goes to 1000 in a few secs. This results in a huge number of overflowed work requests and dropped work requests. DES has more logs of the problem. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 11 11:24:50 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 11 Dec 2007 11:24:50 -0000 Subject: [Varnish] #188: thread pileup In-Reply-To: <054.501398c9432dffc43debd3778be2327e@projects.linpro.no> References: <054.501398c9432dffc43debd3778be2327e@projects.linpro.no> Message-ID: <063.7565bb20ed872a67472293a44cd30064@projects.linpro.no> #188: thread pileup ----------------------+----------------------------------------------------- Reporter: steinove | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: 1.1.1 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Changes (by phk): * owner: des => phk * component: build => varnishd Comment: My best diagnosis, is that the backend is very slow for some particular popular object. One thread picks it up from the backend and a LOT of requests gets parked on the busy object until it comes back from the backend. Once it comes back, all the parked requests gets reinjected and causes the thread explosion. Some kind of pacing algorithm is obviously called for, the trick is to find one that doesn't need too many carefully tweaked numbers to work generally and reliably. Thinking... /phk -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 11 12:23:26 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 11 Dec 2007 12:23:26 -0000 Subject: [Varnish] #188: thread pileup In-Reply-To: <054.501398c9432dffc43debd3778be2327e@projects.linpro.no> References: <054.501398c9432dffc43debd3778be2327e@projects.linpro.no> Message-ID: <063.f680be91c3e0c37fd9ac871e7acd7c84@projects.linpro.no> #188: thread pileup ----------------------+----------------------------------------------------- Reporter: steinove | Owner: phk Type: defect | Status: assigned Priority: high | Milestone: Component: varnishd | Version: 1.1.1 Severity: normal | Resolution: Keywords: | ----------------------+----------------------------------------------------- Changes (by phk): * status: new => assigned Comment: Commit 2294 is an attempt to handle this. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 11 15:55:39 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 11 Dec 2007 15:55:39 -0000 Subject: [Varnish] #189: C{ ... }C blocks not allowed outside sub, acl or backend Message-ID: <050.3378c91a54b1c5aa906fae97008af2b2@projects.linpro.no> #189: C{ ... }C blocks not allowed outside sub, acl or backend --------------------+------------------------------------------------------- Reporter: adam | Owner: des Type: defect | Status: new Priority: normal | Milestone: Component: build | Version: trunk Severity: normal | Keywords: --------------------+------------------------------------------------------- It should be possible to include C{ ... }C blocks outside sub-, acl- or backend-blocks, as well as inside them. As it is now, e.g. #include'ing stuff is nigh-on impossible, as the scope is invariably inside some function or method in the generated code. -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 18 09:23:30 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 18 Dec 2007 09:23:30 -0000 Subject: [Varnish] #190: VCL Sample Message-ID: <057.c1af45dfcb09cdebdbb846ab6cba109a@projects.linpro.no> #190: VCL Sample ---------------------------+------------------------------------------------ Reporter: michael.lee | Owner: des Type: documentation | Status: new Priority: normal | Milestone: Component: build | Version: 1.1.1 Severity: normal | Keywords: ---------------------------+------------------------------------------------ We noticed that the default VCL Sample in VCL Man page is incorrect. It does not work when cookies are present. Looks like in order to cache dynamic contents when cookies are present, we need customize the vcl_hash() subroutine as well. Is it possible to include some well-tested vcl samples on wiki and VCL Man page? Here is an example inspired by this article - http://www.ipersec.com/index.php/2007/11/15/varnish-simple-and-fast-http- acceleration/ {{{ backend default { set backend.host = "localhost"; set backend.port = "81"; } sub vcl_recv { if (req.request != "GET" && req.request != "HEAD") { pipe; } if (req.http.Expect) { pipe; } if (req.http.Authenticate) { pass; } //ignore and pass no-cache documents if (req.http.Cache-Control ~ "no-cache") { pass; } /* lookup objects from cache even when cookies are present */ lookup; } sub vcl_pipe { pipe; } sub vcl_pass { pass; } sub vcl_hash { /* * The hash subroutine is important * It defines the "name" pattern of * cache objects * We will use url+host by default. * In this way, different machines may * share the same cache object. */ set req.hash += req.url; set req.hash += req.http.host; /* * You can also append cookie * to the hash. In this way, * varnish will fetch/insert a cache object for each * machine. */ //set req.hash += req.http.cookie; hash; } sub vcl_hit { if (!obj.cacheable) { pass; } deliver; } sub vcl_miss { fetch; } sub vcl_fetch { if (!obj.valid) { error; } if (!obj.cacheable) { pass; } /* ignore set-cookie response */ if(obj.http.Set-Cookie){ pass; } /* ignores no-cache documents */ if(obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" || obj.http.Cache-Control ~ "private"){ pass; } /* insert documents to cache even when cookies are present */ insert; } sub vcl_deliver { deliver; } sub vcl_timeout { discard; } sub vcl_discard { discard; } }}} -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 18 09:25:12 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 18 Dec 2007 09:25:12 -0000 Subject: [Varnish] #190: VCL Sample In-Reply-To: <057.c1af45dfcb09cdebdbb846ab6cba109a@projects.linpro.no> References: <057.c1af45dfcb09cdebdbb846ab6cba109a@projects.linpro.no> Message-ID: <066.46319db889ece66658e18d534edbf75b@projects.linpro.no> #190: VCL Sample ---------------------------+------------------------------------------------ Reporter: michael.lee | Owner: des Type: documentation | Status: new Priority: normal | Milestone: Component: build | Version: 1.1.1 Severity: normal | Resolution: Keywords: | ---------------------------+------------------------------------------------ Comment (by michael.lee): The vcl_hash() subroutine is not documented in VCL Man page. Is it possible to include some samples and description? -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 18 09:27:43 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 18 Dec 2007 09:27:43 -0000 Subject: [Varnish] #190: VCL Sample In-Reply-To: <057.c1af45dfcb09cdebdbb846ab6cba109a@projects.linpro.no> References: <057.c1af45dfcb09cdebdbb846ab6cba109a@projects.linpro.no> Message-ID: <066.8888fce51d1fd4947da63aaacecfbf55@projects.linpro.no> #190: VCL Sample ---------------------------+------------------------------------------------ Reporter: michael.lee | Owner: des Type: documentation | Status: new Priority: normal | Milestone: Component: build | Version: 1.1.1 Severity: normal | Resolution: Keywords: | ---------------------------+------------------------------------------------ Comment (by michael.lee): I assigned this ticket to "build" component, can someone help me assign it to "documentation" component? Thanks -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Thu Dec 20 06:25:21 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Thu, 20 Dec 2007 06:25:21 -0000 Subject: [Varnish] #191: Varnish won't cache if apache keepalive is set to on Message-ID: <057.afbf7964f5f8580f04cb9d3e7d90da6b@projects.linpro.no> #191: Varnish won't cache if apache keepalive is set to on -------------------------+-------------------------------------------------- Reporter: michael.lee | Owner: phk Type: defect | Status: new Priority: high | Milestone: Component: varnishd | Version: 1.1.1 Severity: normal | Keywords: keepalive -------------------------+-------------------------------------------------- We found when apache keep alive is on, some contents never got cache. Here is the vcl file we used. Please note that we enabled varnish to cache contents even when cookies are present. I don't know if it makes any difference or not. But I'm just giving you enough information to help you address the root cause. {{{ backend default { set backend.host = "localhost"; set backend.port = "81"; } sub vcl_recv { if (req.request != "GET" && req.request != "HEAD") { pipe; } if (req.http.Expect) { pipe; } if (req.http.Authenticate) { pass; } if (req.http.Cache-Control ~ "no-cache") { pass; } lookup; } sub vcl_pipe { pipe; } sub vcl_pass { pass; } sub vcl_hash { set req.hash += req.url; set req.hash += req.http.host; //set req.hash += req.http.cookie; hash; } sub vcl_hit { if (!obj.cacheable) { pass; } deliver; } sub vcl_miss { fetch; } sub vcl_fetch { if (!obj.valid) { error; } if (!obj.cacheable) { pass; } if(obj.http.Set-Cookie){ pass; } if(obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~ "no-cache" || obj.http.Cache-Control ~ "private"){ pass; } insert; } sub vcl_deliver { deliver; } sub vcl_timeout { discard; } sub vcl_discard { discard; } }}} We found when apache keepalive is set to "on", dynamic contents can't be cached. We have set the Cache-control, Pragram and Expires headers correctly for dynamic contents. It works perfect when apache keepalive set to "off". I don't know if this ticket relates to ticket #55 or not Please help to investigate. Many thanks in advance, Michael -- Ticket URL: Varnish The Varnish HTTP Accelerator From varnish-bugs at projects.linpro.no Tue Dec 25 05:38:14 2007 From: varnish-bugs at projects.linpro.no (Varnish) Date: Tue, 25 Dec 2007 05:38:14 -0000 Subject: [Varnish] #192: varnishd 1.1.2 died in VBE_ClosedFd() Message-ID: <051.3035b4ae37c243d40690ad0cd9c173a8@projects.linpro.no> #192: varnishd 1.1.2 died in VBE_ClosedFd() ----------------------+----------------------------------------------------- Reporter: jnlin | Owner: phk Type: defect | Status: new Priority: normal | Milestone: Component: varnishd | Version: 1.1.2 Severity: normal | Keywords: ----------------------+----------------------------------------------------- Running varnishd-1.1.2 on FreeBSD/amd64 7.0-BETA2 (about 200 requests per second) Child said (2, 4709): <fd)) == 0) not true. errno = 54 (Connection reset by peer) >> And the child restarted, so the cache hit ratio is low. -- Ticket URL: Varnish The Varnish HTTP Accelerator