From martin at varnish-software.com Tue Jun 5 08:08:11 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 5 Jun 2012 10:08:11 +0200 Subject: Updated persistent patches Message-ID: Hi, Attached is an updated patch set for the persistence changes I have been working on. Compared to last submission this includes some bug fixes for both new and existing bugs, some polish on the patch set, and stuff to avoid holding the lock while syncing to disk. Regards, Martin Blix Grydeland -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: persistent.tar.gz Type: application/x-gzip Size: 18173 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Jun 6 06:40:18 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 06 Jun 2012 06:40:18 +0000 Subject: Updated persistent patches In-Reply-To: Your message of "Tue, 05 Jun 2012 10:08:11 +0200." Message-ID: <46892.1338964818@critter.freebsd.dk> In message , Martin Blix Grydeland writes: ---------------------- Subject: [PATCH 01/21] Add a EXP_NukeLRU() function to nuke an entire LRU + struct objcore *oc_array[10]; [...] + while (n < 10) { That's a no-go, #define or sizeof please. You should place both the BINHEAP_NOIDX asserts in the first loop, (see EXP_NukeOne) it does nothing good outside the lock in the second loop. ---------------------- Subject: [PATCH 02/21] Add smp_copy_sign() and smp_reapply_sign() utility functions I can clearly see the utility of these functions, but it bothers me a lot that encapsulating this functionality further obscures the fact that when we use signatures as data-containers we have no idea if we overwrite the allocation the represent. Adding a "max_length" field to struct smp_sign is undesirable, as we don't want that included in the signature (it's only relevant while creating the signature, not while checking it). I guess this is really a manifestation of the fact that smp_sign should not be used as a space-handle in the first place, so what we really should do is add: struct smp_space { uint64_t first; uint64_t len; uint64_t last; struct smp_sign sign; } (This is a runtime structure, it's never stored, so it should *not* go in "include/persistent.h") And use that as our generic handle for "stick things into this space" So we'd have: void smp_copy_space(struct smp_space *dst, struct smp_space *src) { assert(dst->first + src->len < dst->last); memcpy(dst->first, src->first, src->len); dst->len = src->len; smp_reset_sign(&dst->sign); smp_append_sign(&dst->sign, dst->len) } ---------------------- Subject: [PATCH 03/21] Sync the complete sign area on smp_sync_sign Subject: [PATCH 21/21] Round to page sizes on smp_sync_sign() The issue with msync is that both the pointer an length should be properly rounded to pages before we call it, otherwise it might fail. We should wrap this in a "smp_msync" function, rather than pollute all msync calls with it. Notice that pagesize is the correct thing to round to, not sc->granularity. We should clarify the meaning of the smp_sign->length field in include/persistent.h: Does it include struct smp_sign or not ? ---------------------- Subject: [PATCH 04/21] Free the LRU object and set free_offset when dropping empty segments in smp_close_seg() Looks correct. (When do we close an empty segment, other than shutdown ?) ---------------------- Subject: [PATCH 05/21] Don't exit on full silo Are you sure you have all the AN(sc->cur_seg) the change of semantics requires ? For clarity, always keep left in sync with cur_seg: if (left < extra + min_size) { if (sc->cur_seg != NULL) smp_close_seg(sc, sc->cur_seg); smp_new_seg(sc); if (sc->cur_seg != NULL) left = smp_spaceleft(sc, sc->cur_seg); >>> else >>> left = 0; } In both the next two cases, you should consider truncation of the new segment, it's silly to waste a full segment if the overflow is just one page: + if (smp_segend(&tmpsg) > sc->mediasize) + tmpsg.p.offset = sc->ident->stuff[SMP_SPC_STUFF]; + sg = VTAILQ_FIRST(&sc->segments); + if (sg != NULL && sg->p.offset >= tmpsg.p.offset) { + if (smp_segend(&tmpsg) > sg->p.offset) + return; + assert(smp_segend(&tmpsg) <= sg->p.offset); } (The last if might be clearer if you swapped the direction of the compare to tmpsg <= sg) The return should be commented as failure + ALLOC_OBJ(sg, SMP_SEG_MAGIC); + AN(sg); Instead of AN, you should just return failure. We should handle malloc failures gracefully where we can easily do so. sg->p.offset = IRNUP(sc, sg->p.offset); sg->p.length = IRNDN(sc, sg->p.length); Isn't there a risk that the offset is rounded up, but the length not rounded down, so we overstep by one page ? I don't think it can happen, but an assert to make sure is probably a good idea. ---------------------- Subject: [PATCH 06/21] smp_thread() stopping printf("Silo completely loaded\n"); All things considered, we should probably VSL these and live with it. SLT_PersistentEvent or something. + if (sg != NULL && sg != sc->cur_seg && sg->nobj == 0) { smp_save_segs(sc); } Loose {} + VTIM_sleep(1); We should avoid whole-second sleeps, they have a bad tendency to get things to run in lockstep. Ideally each silo should sleep for 1+random-small-delta, but at least make it different from 1.0 + while (1) + VTIM_sleep(1); + NEEDLESS_RETURN(NULL); You should simply let the tread die, and pthread_join() it from smp_close(). In either case, it is silly that we close the silos serially, you should add a "signal_close()" method to stevedore, and call any which is non-NULL as the first thing in STV_close(), so that we can sed the SC_STOP on all silos before we wait for the first one to do so. ---------------------- Subject: [PATCH 11/21] Generalize the ban reporting to the stevedores I appreciate the generality of STV_NewBan(), but what is the point of calling BAN_Spec, rather than passing the ban+len as arguments ? ---------------------- Subject: [PATCH 12/21] Also report dropped bans to the stevedores. Looks ok. ---------------------- Subject: [PATCH 14/21] Add consistency checks between the ban lists at startup I wonder if it would make more sense to not write here, and instead wait until everything is loaded and then write new (and possibly shorter) ban lists to both segments ? (Ref: recent talk about pruning bans) ---------------------- Subject: [PATCH 15/21] Add the concept of an extra area to the beginning of smp signs. Subject: [PATCH 16/21] Store the ban timestamp in the smp ban lists Subject: [PATCH 17/21] Keep an offset on the ban lists (in an extra signed I'm not quite sure how this should be done, but I'm sure its not this way. There is really only one bit of info you really need to record, and that is the timestamp of the oldest non-removed ban, and as far as I can see, you only need to store it on silo close, and the right way to store it, might be to rewrite the live parts of the ban-lists at that time. There is certainly no need to record the bantimes in the segment (0016), since they are already there in the ban-spec. non-mentioned patches are not reviewed. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jammy.linux at gmail.com Wed Jun 6 07:35:36 2012 From: jammy.linux at gmail.com (Jammy) Date: Wed, 6 Jun 2012 15:35:36 +0800 Subject: fix timezone processing in libvarnish Message-ID: <36EBC1DF-2A1D-41FA-9A81-152FD0A87CB0@gmail.com> These days, we met with some Expire headers with following format which are not supported by varnish yet. Sun, 06 Nov 1994 08:49:37 +0000 Also it seems that strptime was broken dealing with %z & %Z, so I create the patch to support timezone spec defined in http://www.ietf.org/rfc/rfc822.txt zone = "UT" / "GMT" ; Universal Time ; North American : UT / "EST" / "EDT" ; Eastern: - 5/ - 4 / "CST" / "CDT" ; Central: - 6/ - 5 / "MST" / "MDT" ; Mountain: - 7/ - 6 / "PST" / "PDT" ; Pacific: - 8/ - 7 / 1ALPHA ; Military: Z = UT; ; A:-1; (J not used) ; M:-12; N:+1; Y:+12 / ( ("+" / "-") 4DIGIT ) ; Local differential ; hours+min. (HHMM) 1ALPHA format is easy to add, too, but I think it might not be used widely, so I leave it unsupported with performance consideration. Test cases: + tst("Sunday, 06-Nov-94 08:49:37 CST", 784090177); + tst("Sunday, 06-Nov-94 08:49:37 PST", 784082977); + tst("Sun, 06 Nov 1994 08:49:37 +0000", 784111777); + tst("Sun, 06 Nov 1994 08:49:37 -0000", 784111777); + tst("Sun, 06 Nov 1994 08:49:37 +0100",784115377); + tst("Sun, 06 Nov 1994 08:49:37 -0100",784108177); + tst("Sun, 06 Nov 1994 08:49:37 +0010", 784112377); + tst("Sun, 06 Nov 1994 08:49:37 -0010", 784111177); Please review the change, thanks ---------------------------------- Best wishes, Jammy -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-fix-timezone-process.patch Type: application/octet-stream Size: 4851 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed Jun 6 12:04:11 2012 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 06 Jun 2012 12:04:11 +0000 Subject: fix timezone processing in libvarnish In-Reply-To: Your message of "Wed, 06 Jun 2012 15:35:36 +0800." <36EBC1DF-2A1D-41FA-9A81-152FD0A87CB0@gmail.com> Message-ID: <48204.1338984251@critter.freebsd.dk> In message <36EBC1DF-2A1D-41FA-9A81-152FD0A87CB0 at gmail.com>, Jammy writes: >These days, we met with some Expire headers with following format which = >are not supported by varnish yet. >Sun, 06 Nov 1994 08:49:37 +0000 There is no requirement to support this format, see RFC2616, section 3.3.1, and I'm not particularly keen on the multi-pass way your patch solves it, since I have an item on my list that we should write something faster than strftime() which does a lot of work we don't need. (Belive it or not, but at high loads strftime() shows up in profiling!) If you would like to take a stab at that, you are more than welcome. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jammy.linux at gmail.com Thu Jun 7 01:40:20 2012 From: jammy.linux at gmail.com (Jammy) Date: Thu, 7 Jun 2012 09:40:20 +0800 Subject: fix timezone processing in libvarnish In-Reply-To: <48204.1338984251@critter.freebsd.dk> References: <48204.1338984251@critter.freebsd.dk> Message-ID: Yep, the multi-pass way is not *fast*, but for the common cases, aka. GMT is fine, sure, I should adjust "GMT" to be the number one in the array. I will think of your idea against replacing strptime as well. Glad to receive the feedback from you. Thanks On 6 Jun, 2012, at 8:04 PM, Poul-Henning Kamp wrote: > In message <36EBC1DF-2A1D-41FA-9A81-152FD0A87CB0 at gmail.com>, Jammy writes: > >> These days, we met with some Expire headers with following format which = >> are not supported by varnish yet. >> Sun, 06 Nov 1994 08:49:37 +0000 > > There is no requirement to support this format, see RFC2616, section > 3.3.1, and I'm not particularly keen on the multi-pass way your > patch solves it, since I have an item on my list that we should > write something faster than strftime() which does a lot of work we > don't need. > > (Belive it or not, but at high loads strftime() shows up in > profiling!) > > If you would like to take a stab at that, you are more than > welcome. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. ---------------------------------- Best wishes, Jammy -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Thu Jun 7 11:03:37 2012 From: slink at schokola.de (Nils Goroll) Date: Thu, 07 Jun 2012 13:03:37 +0200 Subject: [PATCH] Drop old grace objects when we have a new object that matches the Vary In-Reply-To: <1325820122-7794-1-git-send-email-github@bsdchicks.com> References: <1325820122-7794-1-git-send-email-github@bsdchicks.com> Message-ID: <4FD08A89.9010300@schokola.de> Hi Doc and All, unfortunately, I don't have time for a closer look at this atm, but I'd like to point out that I had posted a patch following an alternative approach in 2010-11: https://www.varnish-cache.org/lists/pipermail/varnish-dev/2010-November/006600.html At the time, the idea was to expire alternative (unneeded) graced objects during lookup when we pass by anyway. I have not maintained the patch since then, but you might want to reconsider the alternative approach. At the time I thought it was really simple and straight forward. Cheers, Nils From varnish-dev at projects.linpro.no Thu Jun 7 17:04:20 2012 From: varnish-dev at projects.linpro.no (varnish-dev at projects.linpro.no) Date: Thu, 7 Jun 2012 18:04:20 +0100 Subject: fw: work Message-ID: <5194859813.Z32BBU8G355404@hcsnx.ifgpcdpwccr.ru> Great opportunity! Our company is a financial consulting company that meets with complicated problems that often have no easy answers. We are seeking motivated candidates with a broad range of backgrounds to help us solve such critical business dilemmas. We are currently looking to recruit for a part-time position a person with varied experience and a strong desire to work. Candidates having 2-3 hours of spare time a day and holding European citizenship are preferred. We offer a competitive salary based on experience, free choice of working hours and fully paid training during a trial period. The duties of Regional Representative will involve working with our clients providing appropriate service, updating database, assisting with certain company?s activities. We promise that with us you will grow personally and professionally, you will definitely gain experience and what is really important you will have a significant income. This is your opportunity to do more, learn more, and be more. Most importantly, you will have the opportunity to become part of one of the largest and fastest growing companies around the world. To express your interest in the above vacancy please provide us your contact information such as: First and Last name Your Email address Phone number. our contact: resum at quintcareerseu.com,and we answer as soon as possible. Our HR representative will conduct a prior interview with you over the phone and provide with more detailed information regarding this position. We have exactly what you are looking for! From cdevecchi at gmail.com Mon Jun 18 14:31:14 2012 From: cdevecchi at gmail.com (Claudio Devecchi) Date: Mon, 18 Jun 2012 11:31:14 -0300 Subject: Varnish Backend Conditional Message-ID: Hi varnish-devs I had compiled varnish-ims (http://www.varnish-cache.org/project/repository) because I need so much the funcionality to check on backend if the object had changed before a I evict my cache, and I need to receive a 304 if nothing happened with it. Everything is fine and working well, but I having a big problem with varnishncsa, when I configure to log like apache take a look at the log. It is writing this way: 172.24.2.41 - - [15/Jun/2012:19:13:42 -0300] "- http://localhost-HTTP/1.0" 0 "-" "-" And the correct should be: 172.22.2.41 - - [13/Jun/2012:16:06:00 -0300] "GET http://test.com/test.jpgHTTP/1.1" 200 0 "-" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:12.0) Gecko/20100101 Firefox/12.0" Could you help me to fix this problem please? Tks Claudio -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish-dev at projects.linpro.no Wed Jun 20 15:36:08 2012 From: varnish-dev at projects.linpro.no (varnish-dev at projects.linpro.no) Date: Wed, 20 Jun 2012 16:36:08 +0100 Subject: Looking for remote assistants, paid $ 100 per hour helping other people Message-ID: <4FE1E8F7.402060@projects.linpro.no> We invite you to work in the remote assistant position. This work takes 2-3 hours per week and requires absolutely no investment. The essence of this work for incoming client requests in your city. The starting salary is about 2500 EUR per month + bonuses. You get paid your salary every 2 weeks and your bonuses after fulfilling each task! We guarantee work for everyone. But we accept applications this week only! Therefore, you should write a request right now. And you will start earning money, starting from next week. Please indicate in the request: Your name: Your email address: City of residence: Please send the request to my email Clayton at workineurop.com,and I will answer you personally as soon as possible Sincerely, Clayton Bergman From kristian at varnish-software.com Fri Jun 22 08:44:06 2012 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Fri, 22 Jun 2012 10:44:06 +0200 Subject: experimental-ims and varnishncsa Message-ID: <20120622084406.GN3661@freud.kly.no> Hi Geoff, I ran across a bug in the experimental-ims branch that's evident in varnishncsa: experimental-ims: 172.24.2.41 - - [15/Jun/2012:19:13:42 -0300] "- http://localhost-HTTP/1.0" 0 "-" "-" vanilla: 172.22.2.41 - - [13/Jun/2012:16:06:00 -0300] "GET http://test.com/test.jpgHTTP/1.1" 200 0 "-" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:12.0) I've tracked it down to the commit f7dbf53eb01d26170f6c4bafd3f0a50d00444dc5 which is a PHK-commit. Any chance you could take a look? Given that it's obviously a merge-related bug it seems somewhat time consuming to track down without intimate knowledge of the IMS-branch. - Kristian From martin at varnish-software.com Fri Jun 22 12:23:57 2012 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 22 Jun 2012 14:23:57 +0200 Subject: Thread creation rate problems in Varnish (3.0 and trunk) Message-ID: Kristian and I have been looking into the algorithms governing thread creation in Varnish a bit lately, in order to explain some situations where Varnish is starved on number of threads but new threads are not created as fast as expected. The type of workload where problems have been seen is typically workloads where the connections are very long lived. As connections come in and when the idle threads has been exhausted, the throughput is starved on new threads not being created quickly enough. In tailored test setups we have seen delays in excess of 5 seconds before a new thread is created, while the queue length is increasing and we are far from max threads. >From looking at the algorithms we have discovered some problems. Posting them to the list to open it up for discussion around how it should be resolved. 1. There is no guarantee that a signal on the herder_cond will actually cause a thread to be created. As the herder runs without any locking, any signal sent while it is busy (e.g. creating a thread) is lost, and it will go back to sleep without ever dealing with the second signal. The busy time includes the thread_pool_add_delay wait time, increasing the likelihood of this happening. This applies both to 3.0 and trunk, although the problem is somewhat less visible in trunk because the herder_cond sleep time is limited by thread_pool_purge_delay (from the merge of the add and remove thread herders). 2. There is a mechanism in the thread breeder that looks at the queue length of the last run, and will only create a thread if the queue length has increased in the mean time. I guess the thought behind this is to stop creating threads when we are in a positive trend and the queue length is shortening. This mechanism does not work very well when the connections are long lived and already executing threads are likely to continue to be busy serving the same connection, as it limits the number of threads created to always be one regardless of queue length. Possibly the test should be a larger_than_or_equal, or this limiting factor should go away. This problem exists both in 3.0 and trunk. 3. The thread herder is not locking the data structures when determining whether a thread should be created or not, opening memory races on the queue lengths and possibly choosing not to add a thread even when there is a queue. Specific to trunk and the new acceptor code, these problems open up a situation where the acceptor task is queued for a later thread, and the breeder is signaled but that signal is lost and the acceptor is left not running. Thus no new connections are accepted and no new tasks are queued, so the breeder isn't signaled and no threads are created (queue length is not increasing). This scenario probably accounts for the longest periods of no threads created during testing. Regards, Martin Blix Grydeland -- Martin Blix Grydeland Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Fri Jun 22 17:19:41 2012 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 22 Jun 2012 19:19:41 +0200 Subject: experimental-ims and varnishncsa In-Reply-To: <20120622084406.GN3661@freud.kly.no> References: <20120622084406.GN3661@freud.kly.no> Message-ID: <4FE4A92D.5060000@uplex.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 6/22/12 10:44 AM, Kristian Lyngstol wrote: > > I ran across a bug in the experimental-ims branch that's evident in > varnishncsa: > > experimental-ims: > > 172.24.2.41 - - [15/Jun/2012:19:13:42 -0300] "- > http://localhost-HTTP/1.0" 0 "-" "-" > > vanilla: > > 172.22.2.41 - - [13/Jun/2012:16:06:00 -0300] "GET > http://test.com/test.jpgHTTP/1.1" 200 0 "-" "Mozilla/5.0 (X11; > Ubuntu; Linux i686; rv:12.0) > > I've tracked it down to the commit > f7dbf53eb01d26170f6c4bafd3f0a50d00444dc5 which is a PHK-commit. > > Any chance you could take a look? Given that it's obviously a > merge-related bug it seems somewhat time consuming to track down > without intimate knowledge of the IMS-branch. Sure, I'll have a look on the weekend. Thanks for figuring out the commit, makes the search much easier. Best, Geoff - -- UPLEX Systemoptimierung Schwanenwik 24 22087 Hamburg http://uplex.de/ Mob: +49-176-63690917 -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJP5KksAAoJEOUwvh9pJNUR8wwQAJ/a6romF14mNxdciOp64Lgb QbChkYdl21wpOcUoW5lcjI9XSIy0jL9EbsQm+sKYQ9V+KJQg8h7uWuRR2/8qwWhn M+xvuqTh3lQ0v+/DjV/W6pTXxW6HItaO+Lc46lXRsnxCbfa+XXG4xOiwmR9kZRmx uYyGXg1+Y17RUIBzWbJPk1SDbJUDgoYNaZlFIOARJeBBSSu+Xq5Y9ZxLFYTPiTcQ sEmCZ8XT/tSeiFCXNYcIUMdwF+c0nhz3edRGqf9OLqp7eeaiyVgYc4htzWPhUSNR H+LYjm3fLIFxWswBaAUqq8PKDKJ1TUpSeLq5ucURhX2TMsHswFQsQdUNnad2aQv8 +BgERJPPrrm5zMdy6cxxANZ0cpyqkX4eN2JNyM72HMfh7BKMYwrg7LMb+zo03rLO RGiuKcjmI93COgcOWTacRZAqvfCgUh8vBQ7z0FZCzJBxJzbzF7OLkv1pOIyZoypW UwspTKMFOylZkluXRRiwgecVIsLNyd8d0fxSn0p8ridHr62DDo9EtbTk8bOuLdKd JPfYc8gMJN0hAGkCh3sH6d8D2TWsOT/2k4rJlyldO0/rxYPaTSbyb/b7Ll9nExfH qYw1L7xw1Qrz1Cg5KC2A2y8UZSdsQOrpJxjAuJpaaYh7wT0zZQUIT8FANZyVN4DN Ju0OLjNaX/xycGqpouN0 =6u83 -----END PGP SIGNATURE-----